Large language models keep making things up because we keep rewarding them when they guess. That, in essence, is the core of model hallucination: systems confidently generate false statements rather than admit uncertainty, driven by next-word pretraining and accuracy-centric evaluations that prefer an answer over an “I don’t know.” Fresh analyses from OpenAI (Sept 5, 2025) and a companion arXiv preprint formalize the mechanics—and the fix—using stark statistics that expose why today’s benchmarks incentivize confident errors over calibrated abstention [1][2].

Key Takeaways

– shows abstention 52% vs 1%, accuracy 22% vs 24%, yet errors 26% vs 75%; abstention slashes hallucinations despite similar accuracy [1] – reveals a lower bound: hallucination rate tracks facts appearing exactly 1 time in training, by Good–Turing frequency estimates [3] – demonstrates consistency-based calibration across 9 datasets improves confidence estimates as sample count rises and reasoning steps are included [5] – indicates accuracy-only leaderboards favor 1% abstainers over 52% abstainers, despite roughly 3x higher error rates (75% vs 26%) [1] – suggests “persona vectors” steer hallucination or sycophancy traits across 2 model families (Qwen, Llama), enabling inference-time mitigation without retraining [4]

Why accuracy-centric benchmarks fuel model hallucination

Most large models are trained to predict the next token, not to calibrate uncertainty or abstain in ambiguous contexts. When we later evaluate them only on task accuracy, we implicitly rank the systems that answer everything—even when unsure—above those that decline low-confidence prompts. OpenAI’s analysis argues this accuracy-first incentive structure is a root cause of model hallucination because it rewards guessing and hides the cost of overconfident errors [1].

In a concrete comparison, OpenAI reports that a high-abstention model (gpt-5-thinking-mini) declines to answer 52% of the time, records 22% accuracy, and has a 26% error rate. A low-abstention baseline (o4-mini) abstains only 1%, hits 24% accuracy, yet produces a 75% error rate. Benchmarks that stare only at accuracy (22% vs 24%) would rank the low-abstention model higher while ignoring nearly threefold more errors (75% vs 26%) [1].

Statistical inevitability of model hallucination

A 2023 theoretical result makes the problem even starker: if a language model is well-calibrated, it must hallucinate on low-frequency facts. Kalai and Vempala show a lower bound in which the hallucination rate mirrors the fraction of facts that appear exactly once in training—a Good–Turing-style estimate of the long tail. The implication is unavoidable: pretraining alone cannot purge hallucinations that stem from singletons and rare events [3].

The 2025 preprint extends this perspective, formalizing how the statistical pressures of next-word prediction and accuracy-centric scoring jointly produce hallucinations. It argues that facts seen only once create inevitable error rates and that the remedy is socio-technical: redesign leaderboards and product metrics to reward calibrated abstention rather than indiscriminate guessing [2].

Abstention vs guessing: what the metrics hide

The OpenAI example illuminates the triad that matters: accuracy, abstention, and errors. The high-abstention model’s 52% abstain rate means it answers fewer questions, but when it does, it avoids many wrong claims, yielding a 26% error rate overall. By contrast, the 1% abstainer answers nearly everything—and accumulates a 75% error rate. In coverage terms, one model attempts roughly 48% of prompts, the other 99%, but the latter floods outputs with incorrect assertions [1].

If your leaderboard scores accuracy alone, you miss this trade-off. Two systems with 22–24% accuracy can impose wildly different costs on users when one generates nearly three times as many falsehoods. This is why OpenAI and others call for metrics that penalize confident errors and explicitly credit safe abstentions. Users and developers need dashboards that surface abstention and error rates alongside accuracy, not hidden beneath it [1][2].

The formal case for reworking benchmarks to curb model hallucination

The arXiv preprint by Kalai, Nachum, Vempala and colleagues formalizes how current evaluations nudge models toward risky behavior. Accuracy-only scoring, when paired with next-word pretraining, induces models to produce an answer even when posterior uncertainty is high—precisely when a calibrated system should decline. The authors propose leaderboard reforms: penalize confident wrong answers and reward selective abstention to better align competition with real-world reliability [2].

OpenAI’s argument lands similarly, but grounds it in empirical contrasts that product teams can replicate. If a benchmark had penalized the 75% error profile more heavily than the 26% one, the high-abstention model would have ranked higher despite marginally lower accuracy. This inversion is the point: we must prefer fewer, more trustworthy answers over more, riskier guesses, and benchmarks should make that preference explicit [1].

Statistical inevitability meets product reality

Theory tells us that long-tail facts—those appearing exactly once—will keep tripping models, even if we scale pretraining. In practice, that means product deployments should assume persistent hallucination pressure on rare entities, edge cases, and niche domains. The 2023 lower bound highlights that mitigation must come after pretraining: calibration, abstention, retrieval, or human escalation, rather than hoping another epoch of training extinguishes the tail [3].

The 2025 preprint urges socio-technical fixes, not just model tweaks. Changing leaderboard incentives, adding abstention-aware scoring, and incorporating calibrated confidence into UX flows can reduce downstream harm without claiming to “solve” hallucination. In other words, reliable systems are designed to say “I don’t know” when the distribution says they should [2].

Steering personas to curb model hallucination

Not all hallucinations are purely statistical; some reflect behavioral tendencies like sycophancy or a disposition to “fill in the blank.” Anthropic’s “persona vectors” research, reported by VentureBeat, identifies activation-space directions corresponding to traits such as hallucination or sycophancy. By adding or subtracting these vectors at inference time, researchers steered behavior in Qwen and Llama variants, pointing to post-hoc controls that can dampen persona-driven errors without retraining [4].

Persona steering is not a panacea, but it complements calibration and abstention. A system that both detects a “hallucination-prone” persona activation and applies a steering vector, while also abstaining under high uncertainty, will better align with safety and reliability goals—especially on prompts that coax models into overconfident improvisation [4].

Calibration by sample consistency: gains and trade-offs

Confidence estimation matters as much as content. Lyu et al. propose “consistency-based calibration”: sample multiple generations and derive confidence from their agreement. Across nine reasoning datasets, this approach improved calibration; performance strengthened as sample count rose and when intermediate explanations (chain-of-thought) were included. In short, consensus among samples is a useful signal for when to trust an answer—or abstain [5].

Two caveats stand out. First, instruction-tuning can make calibration harder, a reminder that making models helpful may inadvertently skew their confidence signals. Second, multi-sample inference costs more compute. Teams can offset this by sampling only on high-stakes prompts, or by using agreement thresholds to automatically abstain or escalate when samples diverge, tightening control over model hallucination in production workflows [5].

What to change now to reduce model hallucination

Rework benchmarks to penalize confident wrong answers and credit calibrated abstention. A leaderboard that ranks a 1% abstainer with a 75% error rate above a 52% abstainer with 26% errors is sending the wrong signal to builders and buyers. Make abstention and error rates first-class metrics alongside accuracy, not hidden footnotes [1].

Adopt socio-technical design patterns. For research competitions and internal evals, publish accuracy with coverage, abstention, and error profiles, and encourage participants to optimize for lower total error—not just higher hit rates. For product UX, expose uncertainty, abstain when confidence is low, and route ambiguous cases to retrieval or human review. This aligns winning strategies with the realities of the long tail and the mathematics of rare facts [2][3].

Integrate calibration and inference-time steering. Use consistency-based confidence from multiple samples where stakes justify compute, and abstain when samples disagree. Layer persona-vector steering to tamp down sycophancy or improvisational tendencies that inflate model hallucination on adversarial or suggestive prompts. These steps do not “solve” hallucination, but they measurably reduce its prevalence where it matters [4][5].

Finally, accept that pretraining scale alone won’t fix the long tail. The Good–Turing bound tells us that facts seen exactly once in training will remain a persistent failure mode for calibrated models. Treat hallucination mitigation as an ongoing engineering discipline—metrics, UX, calibration, and governance—rather than a one-off research milestone [3].

Sources:

[1] OpenAI – Why language models hallucinate: https://openai.com/index/why-language-models-hallucinate

[2] arXiv (preprint) – Why Language Models Hallucinate: https://arxiv.org/abs/2509.04664 [3] arXiv – Calibrated Language Models Must Hallucinate: https://arxiv.org/abs/2311.14648

[4] VentureBeat – New ‘persona vectors’ from Anthropic let you decode and direct an LLM’s personality: https://venturebeat.com/ai/new-persona-vectors-from-anthropic-let-you-decode-and-direct-an-llms-personality [5] arXiv – Calibrating Large Language Models with Sample Consistency: https://arxiv.org/abs/2402.13904 TARGET_KEYWORDS: [model hallucination, LLM hallucination rate, calibrated abstention, Good-Turing estimate, singleton facts, 52% abstention, 1% abstention, 26% error rate, 75% error rate, 22% accuracy, 24% accuracy, accuracy-only metrics, benchmark penalties, leaderboard scoring, consistency calibration, nine datasets, persona vectors, Qwen Llama tests, instruction-tuning calibration, next-word pretraining] FOCUS_KEYWORDS: [model hallucination, hallucination error rate 75%, abstention 52% vs 1%, accuracy 22% vs 24%, Good–Turing hallucination bound, sample consistency calibration, persona vectors hallucination] SEMANTIC_KEYWORDS: [confidence calibration, abstention rate, coverage vs precision, long-tail distribution, rare events, expected error, overconfidence penalty, evaluation metrics, retrieval augmentation, chain-of-thought sampling, inference-time steering, benchmark reform] LONG_TAIL_KEYWORDS: [why do LLMs hallucinate Good-Turing, model hallucination 52% vs 1% abstention, accuracy 22 vs 24 with 75% errors, penalize confident errors benchmarks, calibrated abstention leaderboard design, sample consistency nine datasets, persona vectors reduce hallucinations, instruction tuning hurts calibration, OpenAI error rate 26 vs 75] FEATURED_SNIPPET: Model hallucination persists because accuracy-only benchmarks reward guessing over abstention. OpenAI shows two models with similar accuracy (22% vs 24%) but vastly different error rates (26% vs 75%) driven by abstention behavior (52% vs 1%). Reforming metrics to penalize confident errors, rewarding calibrated abstention, and adding consistency-based confidence can measurably reduce false outputs without retraining [1][2][5].

Image generated by DALL-E 3