A published claims AI accuracy number is a measurement with an expiry date. Nobody is lying when they print it. A figure can be exactly true on the day it was taken and still tell a carrier nothing about what the same system does months later.
Three things move in between. The claims mix drifts away from the training data. The model underneath is replaced on a schedule the carrier does not set. And the output an adjuster reads is a summary that gives no account of what it left out.
So the durable questions are what re-establishes the number after the model changes, and what the claim file shows when the output was wrong. Neither is a percentage. Why accuracy is also the wrong headline metric for a business case is in the right ROI metrics for investigation AI.
The AI claims processing accuracy number in the deck has a date on it
Answer
How accurate is AI in claims processing?
There is no industry figure. Published claims run wide and rarely carry a method: Shift Technology's September 2025 launch release reported early adopters seeing greater than 99% accuracy in claims assessment, with no test set, measurement window or line of business stated beside it. Accuracy is answerable only as of a date.
When Shift Technology launched Shift Claims on September 16, 2025, the release reported that early adopters see "greater than 99% accuracy in claims assessment". That is a real, dated, public figure, more than most of this market offers.
This is not a criticism of Shift. The point is structural and applies to every vendor including us. Fourteen months later the sentence is still accurate about the past and silent about the present.
Vendors already say the demo number is the cheap part. The clearest statement sits in a sponsored thought-leadership column SiftMed published in Claims Journal on November 13, 2026.
That is one vendor executive's own unaudited estimate, not an industry benchmark. The same column names four ways a result decays after the demo, and three of them are not about medical records at all. They move a coverage position, a fraud finding and a recovery demand exactly as they move a medical summary.
A validated number stops being true in four ordinary ways
The taxonomy is a medical-record vendor's. Three of the four modes reach every stage of a claim, not only the medical file.
Nobody is lying. A number can be true on the day it was measured and tell you nothing about today.
Drift is measurable, and it has been measured
Answer
What is model drift in insurance?
The NAIC defines it. In its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023, model drift means the decay of a model's performance over time arising from underlying changes between the data used to train the model and the data on which it is deployed.
The headline effect size comes from a 2022 research replication of commercial sepsis predictors: an RNN built on Epic features degrading from a 0.729 AUC to a 0.525 AUC over a decade. On a binary task 0.525 is close to a coin flip. Nobody touched the code; the world around it moved.
Carry the caveat: this is a research replication of a clinical model, not an audit of a shipped product. The mechanism transfers, the magnitude does not.
A second study points the same way on a different task. Nestor and colleagues, writing for the Machine Learning for Health workshop at NeurIPS in 2018, found "a deterioration of 0.3 AUC when evaluating mortality prediction on data from 10 years later".
The subtle finding matters most. A 2025 review of degradation detection in medical AI reports a study of seven acute kidney injury models over nine years where discrimination stayed stable while calibration degraded. Discrimination is the metric a vendor publishes. Calibration is what a triage threshold runs on.
Three independent domains, three windows, one shape
Each row is a model whose measured performance fell without anyone changing its code.
A number that was true when it was measured and false later, with nothing in between to tell you.
The claims equivalents are not exotic. A new program or line changes the inputs. A CAT season changes the outcome rate. A fee-schedule revision or a repair-network change shifts the relationship between them.
Model churn: the substrate moves on a published schedule
Answer
What happens to accuracy when the AI vendor changes its model?
Behavior changes, sometimes downward, on tasks nobody re-tested. GPT-4 answered identical prime-number questions at 84% accuracy in March 2023 and 51% in June 2023. The carrier changed nothing. A published accuracy figure survives a model swap only if somebody re-runs the measurement on the new version.
Even a perfectly stable claims mix would not hold the number, because the model underneath changes on dates the carrier does not pick. The Harvard Data Science Review study was contested when it appeared, with commentators arguing the prime-number probe was narrow. Cite it for what is uncontested: behavior changes across versions.
Version change is a schedule, published by the model provider. OpenAI's deprecations page lists the shutdown dates.
A carrier signing a three-year claims AI contract in 2026 is signing for a system whose substrate has published shutdown dates inside the contract term. So ask: show me your last three model migrations and the evaluation you re-ran on each one.
Hesper runs on the same substrate and our models change too. What has to be true is that a changed output is detectable at the level of the single claim, by the adjuster holding that file, on the day it happens. That comes from citation, not from a percentage.
The failure you cannot see is omission, not hallucination
Answer
Is hallucination the main accuracy risk in AI claim summaries?
No. Omission is more common and harder to catch. A 2025 clinician-annotated study of one LLM summarization pipeline measured omissions in 3.45% of source sentences against hallucinations in 1.47%, and 286 of those omissions were classified major. A hallucination is present and wrong; an omission leaves nothing behind.
Omission is exactly what a bare summary hides. The study, by Asgari and colleagues in npj Digital Medicine 8:274 in 2025, found omission more than twice as common as hallucination.
Across 12,999 sentences in 450 clinical notes, 191 contained hallucinations, a rate of 1.47%, and 84 of those (44%) were classified major. Across 49,590 sentences in the source consultation transcripts, 1,712 were omitted, a rate of 3.45%, of which 286 (16.7%) were major.
The more common failure is the one a summary cannot show you
Omitted sentences ran more than twice as common as hallucinated ones, and an omission leaves nothing on the page to catch.
A hallucinated sentence is on the page and wrong, so a careful reader can catch it. An omitted sentence is absent, and no amount of reading the summary will surface it.
You cannot proofread an absence. You can only check it against the source document.
Do not quote those as the rates of claims AI. This is a clinical-documentation study on one pipeline, and the rates moved substantially across the prompting experiments the authors ran. That is precisely why a single published figure cannot stand in for a system.
Long claim files make it worse. Liu and colleagues, in "Lost in the Middle: How Language Models Use Long Contexts", found performance "significantly degrades when models must access relevant information in the middle of long contexts". A contradiction on page 180 of a 400-page file is more likely to be dropped than the same one on page 2.
This is why investigation-grade evidence is a different product from a good summary. When every asserted fact carries the document, page and line it came from, a reader can check what is on the page and go looking for what is not. The decision stays with the adjuster: see which claims decisions stay human.
What a regulator can ask for, and what it does not require
Answer
Does any US rule require insurers to re-test claims AI for accuracy?
No. The NAIC model bulletin is guidance, adopted state by state, and it says its goal is not to prescribe specific practices or documentation requirements. What it does is put claim administration in scope, expect ongoing monitoring, and tell insurers a department may request drift-evaluation documentation, including for a vendor's model.
Raise this with a vendor before signing. In an investigation or market conduct action a department may request "documentation related to validation, testing, and auditing, including evaluation of Model Drift", and the same demand is listed for third-party AI systems. The bulletin also contemplates contract terms that "provide audit rights and/or entitle the Insurer to receive audit reports by qualified auditing entities".
No US rule sets a re-test cadence for P&C claims AI
The bulletin disclaims it in terms: "The goal of the bulletin is not to prescribe specific practices or to prescribe specific documentation requirements." Anyone writing that regulators now require annual revalidation of claims AI is wrong.
The nearest thing to a stated cadence sits outside claims. New York DFS Insurance Circular Letter No. 7, issued July 11, 2024, asks for "a description of testing conducted at least annually to assess the output of AIS models". That circular covers underwriting and pricing. It does not cover claims.
Related reading: NAIC AI risk evaluation v5.0 and what the claim-file rules asked for before AI. Hesper logs every agent action with its sources, its reasoning and its timestamps, so a reconstruction is a query rather than a project.
The test set ages faster than the model
Answer
Why do vendors not simply re-run a golden test set after every change?
Because the test set that tells you the truth has to look like today's claims, and the records that look most like today's claims carry the most identifiers. HIPAA Safe Harbor removes the dates below year, the geography below state level and the account and device identifiers that carry the drift signal. So the cheapest test set is the oldest one.
The obvious fix is a held-out test set, re-run on every change. It is rare because in a bodily injury or workers' compensation claims file the test set is protected health information, governed by 45 CFR 164.514.
A test set only tells the truth if it resembles today's claims. Safe Harbor strips most of what makes a file look like today. Expert determination keeps more and costs money and time every time the set is refreshed. So the cheapest test set is the oldest one.
The cheapest test set a carrier can lawfully keep is the one that looks least like the claims arriving today. Safe Harbor removes the dates, the geography below state level and the account and device identifiers that carry the drift signal.
Governance and drift pull in opposite directions, and that, not vendor indifference, is why re-validation is rare. Building the harness is a discipline of its own, covered in how to grade an AI investigator.
Five ways a vendor can evidence AI claims processing accuracy over time
Answer
What should a carrier ask for instead of an accuracy percentage?
Evidence that survives a model change. Five forms exist and they rank: a one-off demo number, a published aggregate figure, per-output citation to source documents, a versioned eval harness re-run on every model change, and a reconstructable claim file. The first two expire. The last three keep working.
Rank the evidence forms by what survives a model change. There are five, and they are less alternatives than a ladder.
The cost ordering explains the market. A demo number is free. A published figure costs a press release. The bottom three rows cost architecture, a standing team and retention infrastructure, and they keep working after a measurement expires. More in what AI you can defend actually requires.
Hesper publishes no accuracy percentage, because an undated, un-scoped percentage is not evidence. What we offer instead is per-decision citation and a reconstructable trail, with built-in fraud detection. The same cited evidence file serves the coverage position, the fraud finding and the subrogation demand.
The six questions to put in the RFP
Answer
What questions should a claims team ask a vendor about accuracy?
Six. When was your published figure measured, on what test set, in which lines. How many times did the model change in twelve months, and what did you re-run. Does every asserted fact cite its page. Can you reconstruct one claim from eighteen months ago. Will the contract give us audit rights.
Every one of these is answerable on a call. A vendor who can only answer the first has a slide.
- What date was your published accuracy figure measured, on what test set, and in which lines of business?
- How many times has the underlying model changed in the last twelve months, and what did you re-run on each change?
- Does every asserted fact in an output link to the page it came from, or only the summary as a whole?
- Can you produce, for a single claim closed eighteen months ago, what the system did, with which inputs and which sources?
- If my examiner asks for drift-evaluation documentation for your model, can you hand it over, and will you say so in the contract?
- What is in your test set, how old is it, and how was it de-identified?
None of this argues against buying claims AI. It argues against buying a number. Question four is the one a market conduct examiner will effectively ask years later, about a file nobody remembers.
Key takeaways
- A published claims AI accuracy figure is a measurement with an expiry date: it describes the system as it stood on the day it was measured.
- Decay has effect sizes in the literature: an RNN on Epic sepsis features fell from 0.729 to 0.525 AUC over a decade, and GPT-4 went from 84% to 51% on identical prime-number questions in three months.
- Omission is the failure a summary hides: one clinician-annotated study measured omissions in 3.45% of source sentences against 1.47% hallucinations.
- No US rule sets a re-test cadence or an accuracy floor for P&C claims AI, so ask for per-output citation, a versioned eval harness re-run on every model change, and a claim file that reconstructs years later.