LLMs Hallucinate and That's a Benchmarking Problem
The Weekly Salt #85
This week, we review:
On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?
Towards a Unified View of Large Language Model Post-Training
Repositories (full list of curated repositories here):
⭐Why Language Models Hallucinate
The paper Why Language Models Hallucinate argues that hallucinations, plausible but false outputs generated by LLMs, are not mysterious accidents but natural outcomes of how these systems are trained and evaluated. The authors show that even with error-free training data, statistical pressures inherent in pre-training lead to generative errors similar to misclassifications in supervised learning. Hallucinations are especially common in cases where no clear pattern exists in the data, such as arbitrary facts (e.g., birthdays appearing only once in training). Their analysis ties hallucinations to computational learning theory, making them predictable rather than anomalous.
Post-training, which aims to refine models and reduce such errors, often fails to eliminate hallucinations because of evaluation incentives. Current benchmarks overwhelmingly use binary scoring systems that penalize uncertainty and abstentions (such as answering “I don’t know”). As a result, models are effectively rewarded for bluffing, producing confident but incorrect answers, since guessing maximizes scores under such evaluation schemes. This mirrors human test-taking behavior, where students often guess when unsure to improve scores, even if it results in overconfident mistakes.
The authors propose that hallucinations can only be meaningfully reduced if the evaluation ecosystem changes. Instead of seeking a “perfect hallucination benchmark,” they argue for modifying mainstream evaluations to reward expressions of uncertainty and penalize overconfident falsehoods less severely. By introducing explicit confidence thresholds and scoring systems that give partial credit for abstention, evaluations can encourage models to truthfully express uncertainty. This would reduce the incentive for guessing and, over time, yield more trustworthy AI systems.
Overall, the paper reframes hallucinations as a socio-technical issue: they stem as much from misaligned incentives in evaluation as from statistical limitations in training. By grounding hallucinations in established learning theory and connecting them to test-taking dynamics, the authors highlight that suppression of hallucinations is possible, not by demanding perfect accuracy, but by redesigning benchmarks to align with trustworthiness. This shift, they argue, could pave the way toward AI systems better calibrated to acknowledge uncertainty rather than fabricate plausible falsehoods.
In other words, that’s mainly an evaluation problem.
On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
This paper examines whether widely used benchmarks such as MMLU, ARC-C, and HellaSwag reliably reflect the real-world robustness of LLMs. While benchmarks provide standardized, fixed question wordings that enable fair comparisons, they fail to account for the linguistic variability of natural use cases, where the same query may be expressed in multiple ways. To test this, the authors systematically paraphrased more than 53,000 benchmark questions across six datasets, producing over 260,000 reworded variants, and evaluated 34 state-of-the-art LLMs on them.
The results show that while model rankings remain stable across paraphrased benchmarks (Kendall’s τ > 0.9), absolute performance drops significantly. Between 15% and 30% of paraphrased questions elicited different answers from the same model, underscoring sensitivity to surface-level wording changes. Interestingly, weaker models often displayed high consistency by repeating the same (frequently wrong) answers, while stronger models achieved both higher accuracy and greater robustness. This duality highlights that consistency alone is not a reliable measure of reasoning ability.
The study also uncovers evidence of data contamination in older benchmarks, where models perform disproportionately well on original formulations compared to paraphrased ones, suggesting memorization rather than genuine generalization. Newer benchmarks appear less affected, possibly because they are less likely to have leaked into training data or are better designed against shallow shortcuts.
That’s a very interesting paper, but there is one big issue in their methodology: The paraphrases were generated by an LLM. I suspect that the conclusion could be different if they were produced by humans or with other LLMs
Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?
This paper argues that today’s LLMs are brittle when instructions conflict with the conventions they absorbed during SFT/RLHF (e.g., “do not use bullet points,” “answer incorrectly on purpose,” or “write code without comments”). The authors call this bias cognitive inertia: models tend to replicate standardized formats and correctness norms favored by annotators, which boosts fluency but hurts flexibility on atypical or counterintuitive requests.
To measure this, they introduce Inverse IFEval, a benchmark of 1,012 high-quality, bilingual (EN/ZH) items spanning 23 domains and eight “inverse” instruction types: Question Correction, Intentional Textual Flaws, Code without Comments, Counter-Conventional Formatting, Deliberately Incorrect Answers, Instructional Induction, Mid-turn Instruction Modification, and Counterfactual Answering. The dataset is built via a human-in-the-loop pipeline (expert seeds → large-scale LLM generation → automatic filtering → expert verification), and scored with an optimized LLM-as-a-Judge setup whose judging accuracy is tuned to ~98%.
Experiments across many open and closed models show sizable declines relative to conventional instruction benchmarks (like IFEval). Larger models and “thinking” variants (deliberate/CoT-style inference) perform better; smaller, purely instruction-tuned “non-thinking” models fare notably worse, suggesting that reflective reasoning helps override trained conventions. Performance varies by category (e.g., models are relatively stronger on Counterfactual Answering and weaker on Question Correction), exposing distinct failure modes.
Towards a Unified View of Large Language Model Post-Training
This paper proposes a unified theoretical framework for post-training LLMs, reconciling two dominant paradigms: SFT, which relies on curated offline demonstrations, and RL, which leverages online rollouts with reward signals. While these approaches are often treated as separate or sequential (e.g., “SFT then RL”), the authors show that they are in fact special cases of a single optimization objective. They derive a Unified Policy Gradient Estimator (UPGE), which expresses the gradients of diverse post-training methods within the same formulation, decomposed into four interchangeable components: stabilization mask, reference policy denominator, advantage estimate, and likelihood gradient.
Building on this unified view, the authors introduce Hybrid Post-Training (HPT), a practical algorithm that dynamically integrates SFT and RL. HPT adjusts the weighting of each signal based on real-time performance feedback: when the model struggles, it leans on SFT to exploit demonstration data; when competence improves, it shifts toward RL to encourage exploration. Unlike fixed-ratio or multi-stage pipelines, HPT adapts throughout training, providing both stability and flexibility. This mechanism aims to maximize the complementary strengths of exploitation (SFT) and exploration (RL) without sacrificing reasoning patterns or efficiency.
Extensive experiments on six math reasoning benchmarks (AIME24/25, AMC, MATH-500, Minerva, OlympiadBench) and two out-of-distribution suites (ARC-c, GPQA-Diamond) validate HPT’s effectiveness across different model families (Qwen2.5, LLaMA3.1). HPT consistently outperforms strong baselines, including SFT-only, GRPO-only, SFT→RL pipelines, and mixed-policy methods like LUFFY and SRFT. Notably, on Qwen2.5-Math-7B, HPT improves AIME24 accuracy by nearly 7 points over the best competitor, while also achieving superior Pass@k scores, suggesting that it enhances both accuracy and exploratory breadth. Smaller backbones (1.5B scale) also benefit, showing that HPT is effective even when RL alone fails.
The authors further analyze training dynamics, showing that HPT balances exploration and exploitation, stabilizes entropy and response length, avoids catastrophic forgetting, and internalizes long-form reasoning patterns from demonstrations. Ablation studies confirm the importance of the gating mechanism: a purely SFT-heavy regime underperforms, while a dynamic balance yields the best outcomes. In conclusion, the paper reframes SFT and RL not as conflicting paradigms but as complementary signals under a unified gradient estimator, and demonstrates that adaptive integration via HPT is both theoretically principled and empirically superior for advancing LLM reasoning and generalization.
They released their code here: