Connected Claims USAFind us at booth 502, Signia by Hilton Orlando, Nov 3-4Booth 502 · Orlando, Nov 3-4Booth 502 · Connected ClaimsBook a meeting
Blog›Guides
GuidesNovember 14, 2026·12 min read·Pankaj Dhariwal, CEO·Updated November 16, 2026

Claims AI accuracy is a measurement with an expiry date

GPT-4 went from 84% to 51% on identical questions in three months. A sepsis model fell from 0.729 to 0.525 AUC over a decade. An accuracy figure is a measurement, and measurements expire.

PD
Pankaj Dhariwal · CEO and Co-founder
November 14, 2026·12 min read·Updated November 16, 2026
The numbers behind this
84% to 51%GPT-4 accuracy on identical prime-number questions, March to June 2023Chen, Zaharia and Zou, Harvard Data Science Review 6.2, 2024. Same prompts, same task, same product name. The same study found GPT-4 improved on multi-hop reasoning in that window
0.729 to 0.525AUC of an RNN built on Epic sepsis features, degrading over a decadeYang, Karstens, Ross and Yala, arXiv:2203.16452, March 2022. A research replication of commercial approaches on the public MIMIC-IV dataset, not an audit of a shipped product, and a clinical model, not a claims model
3.45%Sentences omitted by an LLM summarization pipeline, against 1.47% hallucinatedAsgari et al., npj Digital Medicine 8:274, 2025. 1,712 of 49,590 transcript sentences omitted, 286 of them major; 191 of 12,999 summary sentences hallucinated, 84 of them major. Clinician-annotated, one pipeline
0US rules setting a re-test cadence or an accuracy floor for P&C claims AIThe NAIC model bulletin, adopted in roughly thirty jurisdictions, uses should throughout and states that its goal is not to prescribe specific practices. NY DFS Circular Letter No. 7 asks for at least annual drift testing, scoped to underwriting and pricing

A published claims AI accuracy number is a measurement with an expiry date. Nobody is lying when they print it. A figure can be exactly true on the day it was taken and still tell a carrier nothing about what the same system does months later.

Three things move in between. The claims mix drifts away from the training data. The model underneath is replaced on a schedule the carrier does not set. And the output an adjuster reads is a summary that gives no account of what it left out.

So the durable questions are what re-establishes the number after the model changes, and what the claim file shows when the output was wrong. Neither is a percentage. Why accuracy is also the wrong headline metric for a business case is in the right ROI metrics for investigation AI.

The AI claims processing accuracy number in the deck has a date on it

Answer

How accurate is AI in claims processing?

There is no industry figure. Published claims run wide and rarely carry a method: Shift Technology's September 2025 launch release reported early adopters seeing greater than 99% accuracy in claims assessment, with no test set, measurement window or line of business stated beside it. Accuracy is answerable only as of a date.

When Shift Technology launched Shift Claims on September 16, 2025, the release reported that early adopters see "greater than 99% accuracy in claims assessment". That is a real, dated, public figure, more than most of this market offers.

This is not a criticism of Shift. The point is structural and applies to every vendor including us. Fourteen months later the sentence is still accurate about the past and silent about the present.

Vendors already say the demo number is the cheap part. The clearest statement sits in a sponsored thought-leadership column SiftMed published in Claims Journal on November 13, 2026.

When you build, the first version comes fast and the demos look great. You will see 80 or 85 percent accuracy early and it will feel close to done, but the bigger investment is in everything after that, and it's a recurring cost.

Amir Tavafi, Chief AI Officer, SiftMed, in SiftMed's sponsored Claims Journal column, November 13, 2026

That is one vendor executive's own unaudited estimate, not an industry benchmark. The same column names four ways a result decays after the demo, and three of them are not about medical records at all. They move a coverage position, a fraud finding and a recovery demand exactly as they move a medical summary.

01Four decay modes

A validated number stops being true in four ordinary ways

The taxonomy is a medical-record vendor's. Three of the four modes reach every stage of a claim, not only the medical file.

The four decay modes, and what each one moves
Decay mode, as publishedWhat movesMedical records are messyInput qualitySummaries fall behind as records arriveFreshnessThe models keep changingVersion churnSecurity and governance stay on your deskYour burden
List items quoted verbatim from the SiftMed column, November 13, 2026. The tags are ours.
Our extensionThree of the four outlive the medical file: they reach the coverage position, the fraud finding and the demand.

Nobody is lying. A number can be true on the day it was measured and tell you nothing about today.

Taxonomy and list items from SiftMed's sponsored thought-leadership column in Claims Journal, bylined Kelsey Lomond, Head of Marketing, SiftMed, November 13, 2026. The tags and the claims translation are ours. No published study measures recursive summary-of-a-summary loss, so that mode is a mechanism consistent with the omission and position-bias evidence, not a measured effect.

Drift is measurable, and it has been measured

Answer

What is model drift in insurance?

The NAIC defines it. In its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023, model drift means the decay of a model's performance over time arising from underlying changes between the data used to train the model and the data on which it is deployed.

The headline effect size comes from a 2022 research replication of commercial sepsis predictors: an RNN built on Epic features degrading from a 0.729 AUC to a 0.525 AUC over a decade. On a binary task 0.525 is close to a coin flip. Nobody touched the code; the world around it moved.

Carry the caveat: this is a research replication of a clinical model, not an audit of a shipped product. The mechanism transfers, the magnitude does not.

A second study points the same way on a different task. Nestor and colleagues, writing for the Machine Learning for Health workshop at NeurIPS in 2018, found "a deterioration of 0.3 AUC when evaluating mortality prediction on data from 10 years later".

The subtle finding matters most. A 2025 review of degradation detection in medical AI reports a study of seven acute kidney injury models over nine years where discrimination stayed stable while calibration degraded. Discrimination is the metric a vendor publishes. Calibration is what a triage threshold runs on.

02Measured decay

Three independent domains, three windows, one shape

Each row is a model whose measured performance fell without anyone changing its code.

Measured decay in four published findings
What was measuredChangeWindowSepsis RNN on Epic features, AUC0.729 to 0.525a decadeMortality prediction, AUC-0.310 yearsGPT-4, prime vs composite84% to 51%3 monthsSeven AKI models, calibrationdegraded9 years
In the AKI review, discrimination stayed stable while calibration degraded. The published metric held.

A number that was true when it was measured and false later, with nothing in between to tell you.

Sources: Yang, Karstens, Ross and Yala, arXiv:2203.16452 (2022), an RNN replicating Epic sepsis features on MIMIC-IV; Nestor et al., ML4H at NeurIPS (2018), mortality prediction evaluated on data from 10 years later; Chen, Zaharia and Zou, Harvard Data Science Review 6.2 (2024); Guan, Bates and Zhou, arXiv:2506.17442 (2025), reporting a study of seven acute kidney injury models over nine years. All four are clinical or general-purpose models, not claims models.

The claims equivalents are not exotic. A new program or line changes the inputs. A CAT season changes the outcome rate. A fee-schedule revision or a repair-network change shifts the relationship between them.

Model churn: the substrate moves on a published schedule

Answer

What happens to accuracy when the AI vendor changes its model?

Behavior changes, sometimes downward, on tasks nobody re-tested. GPT-4 answered identical prime-number questions at 84% accuracy in March 2023 and 51% in June 2023. The carrier changed nothing. A published accuracy figure survives a model swap only if somebody re-runs the measurement on the new version.

Even a perfectly stable claims mix would not hold the number, because the model underneath changes on dates the carrier does not pick. The Harvard Data Science Review study was contested when it appeared, with commentators arguing the prime-number probe was narrow. Cite it for what is uncontested: behavior changes across versions.

Version change is a schedule, published by the model provider. OpenAI's deprecations page lists the shutdown dates.

Shutdown dateModels retiredNotice date
October 23, 2026gpt-3.5-turbo variants, gpt-4 variants, o1 seriesApril 22, 2026
December 11, 2026gpt-5 series, o3 seriesJune 11, 2026
January 14, 2027GPT-4o variants, GPT-4.1 variantsOctober 14, 2026

A carrier signing a three-year claims AI contract in 2026 is signing for a system whose substrate has published shutdown dates inside the contract term. So ask: show me your last three model migrations and the evaluation you re-ran on each one.

Hesper runs on the same substrate and our models change too. What has to be true is that a changed output is detectable at the level of the single claim, by the adjuster holding that file, on the day it happens. That comes from citation, not from a percentage.

The failure you cannot see is omission, not hallucination

Answer

Is hallucination the main accuracy risk in AI claim summaries?

No. Omission is more common and harder to catch. A 2025 clinician-annotated study of one LLM summarization pipeline measured omissions in 3.45% of source sentences against hallucinations in 1.47%, and 286 of those omissions were classified major. A hallucination is present and wrong; an omission leaves nothing behind.

Omission is exactly what a bare summary hides. The study, by Asgari and colleagues in npj Digital Medicine 8:274 in 2025, found omission more than twice as common as hallucination.

Across 12,999 sentences in 450 clinical notes, 191 contained hallucinations, a rate of 1.47%, and 84 of those (44%) were classified major. Across 49,590 sentences in the source consultation transcripts, 1,712 were omitted, a rate of 3.45%, of which 286 (16.7%) were major.

03Omission vs hallucination

The more common failure is the one a summary cannot show you

Omitted sentences ran more than twice as common as hallucinated ones, and an omission leaves nothing on the page to catch.

Sentences omitted1,712Across 450 consultation transcripts
÷
Sentences in the source transcripts49,590
=
Omission rate3.45%Against a 1.47% hallucination rate
Classified major by clinicians
286omitted sentences classified major, 16.7% of omissions
84hallucinated sentences classified major, 44% of them
The gapOnly one of the two is visible

A hallucinated sentence is on the page and wrong, so a careful reader can catch it. An omitted sentence is absent, and no amount of reading the summary will surface it.

You cannot proofread an absence. You can only check it against the source document.

Source: Asgari et al., A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation, npj Digital Medicine 8:274 (2025). Clinician-annotated, one LLM document-generation pipeline on 450 clinical notes and their consultation transcripts, not a claims study. Rates varied across the authors' prompting experiments, so they are not a constant.

Do not quote those as the rates of claims AI. This is a clinical-documentation study on one pipeline, and the rates moved substantially across the prompting experiments the authors ran. That is precisely why a single published figure cannot stand in for a system.

Long claim files make it worse. Liu and colleagues, in "Lost in the Middle: How Language Models Use Long Contexts", found performance "significantly degrades when models must access relevant information in the middle of long contexts". A contradiction on page 180 of a 400-page file is more likely to be dropped than the same one on page 2.

This is why investigation-grade evidence is a different product from a good summary. When every asserted fact carries the document, page and line it came from, a reader can check what is on the page and go looking for what is not. The decision stays with the adjuster: see which claims decisions stay human.

What a regulator can ask for, and what it does not require

Answer

Does any US rule require insurers to re-test claims AI for accuracy?

No. The NAIC model bulletin is guidance, adopted state by state, and it says its goal is not to prescribe specific practices or documentation requirements. What it does is put claim administration in scope, expect ongoing monitoring, and tell insurers a department may request drift-evaluation documentation, including for a vendor's model.

Raise this with a vendor before signing. In an investigation or market conduct action a department may request "documentation related to validation, testing, and auditing, including evaluation of Model Drift", and the same demand is listed for third-party AI systems. The bulletin also contemplates contract terms that "provide audit rights and/or entitle the Insurer to receive audit reports by qualified auditing entities".

No US rule sets a re-test cadence for P&C claims AI

The bulletin disclaims it in terms: "The goal of the bulletin is not to prescribe specific practices or to prescribe specific documentation requirements." Anyone writing that regulators now require annual revalidation of claims AI is wrong.

The nearest thing to a stated cadence sits outside claims. New York DFS Insurance Circular Letter No. 7, issued July 11, 2024, asks for "a description of testing conducted at least annually to assess the output of AIS models". That circular covers underwriting and pricing. It does not cover claims.

Related reading: NAIC AI risk evaluation v5.0 and what the claim-file rules asked for before AI. Hesper logs every agent action with its sources, its reasoning and its timestamps, so a reconstruction is a query rather than a project.

The test set ages faster than the model

Answer

Why do vendors not simply re-run a golden test set after every change?

Because the test set that tells you the truth has to look like today's claims, and the records that look most like today's claims carry the most identifiers. HIPAA Safe Harbor removes the dates below year, the geography below state level and the account and device identifiers that carry the drift signal. So the cheapest test set is the oldest one.

The obvious fix is a held-out test set, re-run on every change. It is rare because in a bodily injury or workers' compensation claims file the test set is protected health information, governed by 45 CFR 164.514.

A test set only tells the truth if it resembles today's claims. Safe Harbor strips most of what makes a file look like today. Expert determination keeps more and costs money and time every time the set is refreshed. So the cheapest test set is the oldest one.

The cheapest test set a carrier can lawfully keep is the one that looks least like the claims arriving today. Safe Harbor removes the dates, the geography below state level and the account and device identifiers that carry the drift signal.

Governance and drift pull in opposite directions, and that, not vendor indifference, is why re-validation is rare. Building the harness is a discipline of its own, covered in how to grade an AI investigator.

Five ways a vendor can evidence AI claims processing accuracy over time

Answer

What should a carrier ask for instead of an accuracy percentage?

Evidence that survives a model change. Five forms exist and they rank: a one-off demo number, a published aggregate figure, per-output citation to source documents, a versioned eval harness re-run on every model change, and a reconstructable claim file. The first two expire. The last three keep working.

Rank the evidence forms by what survives a model change. There are five, and they are less alternatives than a ladder.

Evidence formWhat it provesWhat it does not proveSurvives a model swapWho can check it independently
One-off demo accuracy, on a vendor-chosen sample, on one dateThe system can work on a curated sample on one dayField performance, performance on your claims mix, performance next quarterNo. Nothing re-establishes itNobody. The sample is not disclosed
Published aggregate accuracy, a figure in a release or a deckThat the vendor will put a number in public with its name on it, which most will notScope, test set, measurement window or line of business. Shift's greater than 99% accuracy in claims assessment, September 16, 2025, carries none of these, and it is one of the more transparent examples on the marketOnly if it is re-published after each change, which no vendor currently commits toNobody, unless the methodology is published beside it
Per-output citation to source documents, every asserted fact linked to its pageAny single output can be checked against the record today, by the person relying on it, without trusting an aggregateAggregate accuracy. It says nothing about how often the system is wrong overallYes. The check is per output and version-independentThe adjuster, an internal auditor, a capacity partner, opposing counsel, a DOI examiner
Versioned eval harness with a held-out test set, re-run on every model changeThe number was re-established on this version rather than on an ancestorThat the test set still resembles today's claims mix, which is the HIPAA problem aboveYes, by construction. That is its purposeThe carrier's model-risk function, and an examiner invoking the bulletin's drift-documentation language
Reconstructable claim file: what the system did on this claim, on this date, with which inputs and sourcesWhat happened on the one claim now under examination, years laterAnything about the populationYes. The file is fixed once writtenA DOI examiner, a reinsurer, a capacity partner, a court

The cost ordering explains the market. A demo number is free. A published figure costs a press release. The bottom three rows cost architecture, a standing team and retention infrastructure, and they keep working after a measurement expires. More in what AI you can defend actually requires.

Hesper publishes no accuracy percentage, because an undated, un-scoped percentage is not evidence. What we offer instead is per-decision citation and a reconstructable trail, with built-in fraud detection. The same cited evidence file serves the coverage position, the fraud finding and the subrogation demand.

An accuracy percentage tells you what happened on a test set, on a date. A citation tells you whether this file is right, today. Only one of the two still works after the model changes.

Hesper AI product research

The six questions to put in the RFP

Answer

What questions should a claims team ask a vendor about accuracy?

Six. When was your published figure measured, on what test set, in which lines. How many times did the model change in twelve months, and what did you re-run. Does every asserted fact cite its page. Can you reconstruct one claim from eighteen months ago. Will the contract give us audit rights.

Every one of these is answerable on a call. A vendor who can only answer the first has a slide.

  1. What date was your published accuracy figure measured, on what test set, and in which lines of business?
  2. How many times has the underlying model changed in the last twelve months, and what did you re-run on each change?
  3. Does every asserted fact in an output link to the page it came from, or only the summary as a whole?
  4. Can you produce, for a single claim closed eighteen months ago, what the system did, with which inputs and which sources?
  5. If my examiner asks for drift-evaluation documentation for your model, can you hand it over, and will you say so in the contract?
  6. What is in your test set, how old is it, and how was it de-identified?

None of this argues against buying claims AI. It argues against buying a number. Question four is the one a market conduct examiner will effectively ask years later, about a file nobody remembers.

Key takeaways

  • A published claims AI accuracy figure is a measurement with an expiry date: it describes the system as it stood on the day it was measured.
  • Decay has effect sizes in the literature: an RNN on Epic sepsis features fell from 0.729 to 0.525 AUC over a decade, and GPT-4 went from 84% to 51% on identical prime-number questions in three months.
  • Omission is the failure a summary hides: one clinician-annotated study measured omissions in 3.45% of source sentences against 1.47% hallucinations.
  • No US rule sets a re-test cadence or an accuracy floor for P&C claims AI, so ask for per-output citation, a versioned eval harness re-run on every model change, and a claim file that reconstructs years later.

Frequently asked questions

There is no industry accuracy figure. Published claims rarely carry a methodology. Shift Technology's September 16, 2025 launch release reported early adopters seeing greater than 99% accuracy in claims assessment, with no test set, measurement window or line of business stated. The more useful question is what re-establishes the figure after the model changes.

Not as a rule with a cadence. The model bulletin is guidance adopted jurisdiction by jurisdiction, and it says its goal is not to prescribe specific practices. It does put claim administration in scope, and a department may request documentation including evaluation of model drift. The same request is listed for a third-party vendor's model, which is why audit rights belong in the contract.

No. An accuracy percentage with no date, no test set, no measurement window and no line of business is not evidence. What we publish instead is the mechanism: every asserted fact cites the document, page or line it came from, every agent action is logged with its sources, reasoning and timestamps, and the same cited evidence file carries the coverage position, the fraud finding and the recovery demand.

On the Hesper platform

Hesper AI runs every stage of a claim, from first notice of loss to subrogation recovery, with investigation-grade evidence behind every decision.

The platformFNOL intakeClaims triageFraud investigationSubrogation & recovery

Keep reading

← More articles on the Hesper AI blog

See Hesper AI on your documents

Request a demo and we'll run an analysis on your real document samples.