Why 99% Accuracy Isn’t Enough for Production AI
Three models, two different jobs. When accuracy is almost the same, the more important question is: does the model know when it might be wrong?
Large language models are accurate most of the time. In insurance claims, that isn’t enough: an adjuster also needs to know which answers not to trust. Most model comparisons report accuracy and stop there. We looked at a second number: how sure each model was when it was wrong.
We compared OpenAI Decisions and TypeSafe’s Jev, two small models built for quick decisions, with GPT-6 Astra, a large reasoning model. We gave them an easy task (what type of document is this claim page?) and a harder one (does a pet’s medical history describe the visit on the invoice?).
The two tasks gave very different answers.
What is this page?
We used 400 claim pages, about half PDFs and half phone photos. Each system chose one of seven types: invoice, receipt, estimate, medical record, claim form, other document, or not a document.
Most pages are obvious. Some, like a registration form with handwritten vaccine dates, aren’t. Those are the pages where we want a model to hesitate.
| Metric | Decisions | Jev | Astra |
|---|---|---|---|
| Correct, out of 400 | 397 | 398 | 395 |
| Mistakes made while sure | 0 | 0 | 5 |
| Pages held back for a person | 6 | 6 | 1 |
| Cost per 1,000 pages | ~$0.28 | ~$0.22 | ~$51 |
All three were right roughly 99% of the time. But the shape of the mistakes was very different. Decisions’ 3 errors had confidence scores of 0.59, 0.61 and 0.75; Jev’s 2 were at 0.63 and 0.69. All 5 of Astra’s errors had a confidence score of 1.0.
A 0.8 cutoff caught every mistake made by Decisions and Jev, while routing just 6 of 400 pages per model to a person.
For Astra, confidence required a workaround: it doesn’t expose log probabilities, so we asked the same question five times and treated the fraction of matching answers as confidence. The problem is that agreement measures stability, not correctness. A model can settle on a wrong reading and repeat it every time.
Is the billed visit in the medical records?
This task means reading a pet’s medical history, sometimes tens of thousands of words, to locate the note that matches the invoice date, then judging how urgent that visit was.
Of the 95 evaluated claims, 31 had long histories.
Accuracy changed when the task changed.
Fig. 01 / 400 pages, 95 claims| Metric | Decisions | Jev | Astra |
|---|---|---|---|
| Visit in the records: correct | 90 / 95 | 85 / 95 | 94 / 95 |
| Urgency: correct | 77 / 95 | 74 / 95 | 92 / 95 |
| Urgency mistakes made while sure | 3 | 8 | 3 |
| Urgency answers held back for a person | 36 | 34 | 0 |
| Long histories: visit found | 30 / 31 | 23 / 31 | 31 / 31 |
| Cost per 100 claims | $0.17 | $0.08 | $24.37 |
Here the result flipped. Astra correctly judged urgency in 92 of 95 claims. Decisions managed 77, and Jev 74. Jev also missed the billed visit in 8 of the 31 longest histories, often with high confidence that it was absent.
Decisions was more cautious, holding back 36 urgency answers for review. Astra held back none. It was much more accurate on this task, but its agreement-based confidence still didn’t distinguish its 3 urgency mistakes from its correct answers.
What did “sure” mean?
We counted a prediction as sure at confidence 0.8 or above. Decisions and Jev provide scores over the answer options. Astra does not expose logprobs, so we used five identical questions and assigned each answer its vote share: four or five matching answers count as sure.
The Astra voting workaround
The code below is a shortened excerpt from the benchmark. Same prompt, same options, five separate runs:
votes = []
for _ in range(samples):
body = post(f"{OPENAI_URL}/responses", ...) # same prompt/options
...
votes.append(json.loads(text)["answer"])
return {v: votes.count(v) / samples for v in set(votes)}This is a repeatability score, not necessarily a calibrated probability of correctness. The benchmark implementation is available in the benchmark script ↗.
There isn’t one “best” model here.
For quick classifications, they matched the large model’s accuracy at a fraction of the cost, and their uncertainty was useful for routing ambiguous pages to a human.
Finding one visit in a long medical history requires retrieval and judgment. Astra was much stronger on urgency, though its five-run agreement didn’t reveal its mistakes.
For costly decisions, evaluate not just how often the model is correct, but how often a wrong answer crosses the auto-approval threshold.
Giving the fast models only the notes near the invoice date might solve much of the long-history gap. This is a hypothesis we haven’t tested yet.
When not to use each kind of model
Small decision models: Avoid relying on them for open-ended tasks that require multi-step reasoning, synthesizing extensive context, or connecting evidence across sources. They work best when the decision is narrow and well defined.
Reasoning chat models: Avoid making them the default for simple, high-volume decisions where speed, cost, and reliable uncertainty signals matter. Getting the same answer repeatedly does not make it correct.
A few important limitations
Both datasets are small. They show an interesting pattern, but don’t establish a definitive ranking between close models. We developed the option wording while working with Decisions, which may favor it. Urgency is also a judgment call, and agreement between two reference models is not the same as independently verified ground truth.
We haven’t tested whether a date-based retrieval step would improve the small models’ performance on long histories. And because Astra’s “confidence” here comes from repeated outputs rather than native probabilities, its score is not directly comparable to the other two systems’ confidence distributions.
An adjuster doesn’t just need a model that’s right 99% of the time. They need a reliable way to find the 1% worth reviewing.
All three systems, prompts, and evaluation details: View the benchmark script on GitHub ↗.