AI hallucination is usually explained with a vague hand-wave, "sometimes it just makes things up." That explanation is not wrong, but it undersells how well understood the actual mechanism now is, and how real the consequences have already been.
The incidents are no longer rare or hypothetical
A public legal tracker had identified 1,598 US court cases involving AI-fabricated citations as of June 2026, up from around 200 a year earlier, with 496 involving licensed attorneys. The penalties have escalated fast. In one case, Couvrette v. Wisnovsky, two attorneys were ordered to pay a combined $110,204.38 in fines after submitting briefs with 15 nonexistent cases and 8 fabricated quotations. In another, a Sixth Circuit panel sanctioned two attorneys $15,000 each, plus full opposing legal fees across three appeals, for briefs containing 24 or more fake citations.
This is not limited to law. Deloitte Australia delivered a government-commissioned review in July 2025 for a fee of roughly AU$440,000. It contained fabricated academic citations, a nonexistent study, and an invented quote attributed to a federal court judge, some references even carried a technical artifact ("oaicite") that revealed exactly where the text had come from. Deloitte issued a partial refund in October 2025, one of Australia's first documented refunds tied to undisclosed AI use in a government contract.
Why this actually happens, from OpenAI's own research
OpenAI published a paper in 2025 titled "Why Language Models Hallucinate" that reframes the problem entirely. The core finding: hallucination is not a mysterious glitch, it is a predictable, almost mathematically inevitable consequence of how these models are trained and graded. During training, a model learns to predict statistically likely text. That works well for grammar and common patterns, but for a specific, low-frequency fact, like a particular person's exact birthday, there is no pattern to lean on, so the model produces a plausible-sounding guess instead of a "correct" answer, because it has no built-in mechanism to distinguish "I know this" from "this sounds right."
The more damaging part is how models get evaluated. Standard benchmarks score answers as simply right or wrong, which means saying "I don't know" is graded the same as being flatly incorrect. That incentive structure pushes models toward confident guessing rather than honest uncertainty, because guessing improves the score even when it increases real-world hallucination. OpenAI's paper describes this plainly as models being optimized to be good test-takers, not to be reliably truthful.
Independent benchmark data confirms this is not improving as fast as expected
Vectara's hallucination leaderboard, an independent benchmark that tests models on summarizing real articles, expanded its test set in November 2025 to 7,700 harder articles across ten domains. On the updated, harder dataset, every current top-tier reasoning model, including GPT-5, Claude Sonnet 4.5, and Grok-4, exceeded a 10% hallucination rate. Notably, models built specifically to "think" more before answering did not reliably hallucinate less than models that answer directly, undercutting the assumption that more reasoning automatically means more accuracy.
What actually helps
Two shipped, named product features are worth knowing about. Google's "Grounding with Google Search" connects Gemini to live search results at answer time, and Google reports this cuts hallucinations by 40% compared to an ungrounded model. Anthropic's Claude Citations API grounds Claude's claims in direct quotes from source documents, and Anthropic reports this took source hallucinations in evaluated use from 10% down to 0%. Both are examples of the same underlying fix: grounding the model in real, retrievable source material at the moment it answers, rather than relying purely on what it memorized during training.
The takeaway
Hallucination is not a bug waiting for a future model update to fix, it is a structural side effect of how these systems are trained and scored. The realistic response is not distrust of every AI answer, it is treating specific numbers, quotes, and citations as claims to verify, especially for anything with real legal, financial, or professional consequences, exactly the category of use where Deloitte and dozens of law firms have already been burned.
