Verifying, Morphing, and Reader-Testing LLMs
The Weekly Salt #123
This week, we review:
LLM-as-a-Verifier: A General-Purpose Verification Framework
Morphing into Hybrid Attention Models
AI translation of literary texts is “fine”, but readers still prefer human translations
Google also finally published the Gemma 4 Technical Report. I’ll publish a complete review next week!
LLM-as-a-Verifier: A General-Purpose Verification Framework
The paper’s core motivation is that LLM progress has mostly come from scaling generation: pretraining, post-training, and test-time sampling. The authors argue that verification, choosing which generated solution is actually correct, is an underdeveloped scaling axis. Their context is agentic systems, where multiple sampled trajectories often contain at least one correct solution, but current judges are too coarse to reliably pick it. On Terminal-Bench V2, they show that an oracle verifier could reach 98.9% by selecting the best sampled trajectory, while standard LM judges suffer from tied/discrete scores and trained reward models may not generalize across domains.
Their method, LLM-as-a-Verifier, turns an LLM judge into a more fine-grained verifier without additional training. Instead of asking the model to output a single discrete score, they extract the model’s probability distribution over scoring tokens and compute an expected continuous score. They then scale verification along three dimensions: score granularity, repeated evaluations, and criteria decomposition. The continuous rewards are converted into pairwise preferences using a Bradley–Terry formulation.
For selecting the best trajectory among many candidates, they introduce a Probabilistic Pivot Tournament: first run a ring pass to reduce positional bias, select promising pivots, then compare non-pivots against those pivots rather than doing all pairwise comparisons. This reduces the ranking cost from quadratic in the number of candidates to a much smaller pivot-based budget while concentrating compute on plausible winners.
Results are strong across domains: the method reaches 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, 87.4% preference accuracy on RoboRewardBench, and 73.3% on MedAgentBench. It also improves score separation on Terminal-Bench as granularity increases, correlates with task progress in code-generation trajectories, and can be used as a dense reward for RL, improving sample efficiency on LIBERO and MATH.
Morphing into Hybrid Attention Models
This paper addresses the long-context efficiency problem of Transformers. Full softmax attention gives strong retrieval and matching ability, but its computation grows quadratically with sequence length and its KV cache grows with context length.
Linear attention and state-space models are cheaper, but often lose recall-sensitive performance. Hybrid attention models try to keep a few full-attention layers and convert the rest to linear attention, but the key unresolved question is which layers should remain full attention. Existing methods mostly use fixed placement rules or score layers independently, ignoring how layers interact when converted together.
The proposed method, FlashMorph, reframes layer selection as a budget-constrained subset optimization problem. It first builds a “morphable” model by giving each full-attention layer a trained linear-attention branch, learned through hidden-state alignment with the original Transformer. Then it freezes both the original full-attention model and the linear branches, and optimizes only a small set of layerwise gates that interpolate between full and linear attention.
A key methodological detail is that the gates are optimized on synthetic long-context retrieval examples, not general language modeling data, because the authors want the selection signal to stress long-range information access. The objective combines hidden-state alignment with a linearization regularizer that pushes the model toward linear attention whenever possible. After optimization, the highest-gate layers are kept as full attention under the budget, the rest become linear attention, and the resulting hybrid model is finalized with logits distillation and long-context finetuning.
Results show that FlashMorph preserves strong long-context retrieval while drastically reducing layer-selection cost. On Qwen3-1.7B, it maintains perfect NIAH-Single-1 accuracy across 32K–256K contexts and improves harder NIAH settings while using only 20M layer-selection tokens. It also preserves general zero-shot commonsense performance, improves recall-heavy tasks, gives long-context speedups such as 2.81× prefill at 256K and 2.07× decode at 512K, and cuts selection cost to 2.1 GPU hours versus 15.4 for HALO, 1071.8 for KL-LS, and 2561.3 for PostNAS.
AI translation of literary texts is “fine”, but readers still prefer human translations
This paper studies a practical and cultural question: as AI-assisted literary translation enters publishing, do readers experience AI-translated novels the same way they experience human translations?
(Clearly not for me. I still feel like I can spot AI when used in translation and then I don’t read it the same. I’m wondering more often whether this is really translated correctly...)
The authors argue that standard MT evaluation, fluency, adequacy, automatic metrics, or short-segment judgmentsm misses what matters for literature: immersion, voice, rhythm, emotional effect, and sustained reading experience.
The method is a reader-centered evaluation dataset and protocol called LAIT. The authors compare recently published professional human translations with machine translations of 15 recent novels originally in French, Polish, and Japanese, all translated into English. They use roughly 8K-word excerpts, chosen from recent 2025–2026 publications to reduce training-data contamination risk. The MT side is produced with an agentic LLM pipeline: source analysis and style guidance, chunk-level translation and review, acceptance gates, revision loops, and excerpt-level consistency/literary review.
For evaluation, they recruit 15 avid readers. Each book is evaluated by two readers. Readers first do immersive reading of whole excerpts, then after a one-day break perform close reading of aligned ~300-word HT–MT chunks. The protocol collects ratings, direct preferences, explanations, AI-origin guesses, and span-level annotations of good or poor wording. This gives 30 excerpt-level comparisons, 772 chunk-level comparisons, about 1K reader comments, and 7.2K span annotations.
The main result is nuanced: readers often find MT readable and sometimes prefer it, but overall they still prefer HT, especially under close reading. HT wins 19/30 excerpt-level comparisons and 522/772 chunk-level comparisons; HT also gets higher ratings for acceptability and smoothness. MT is not simply bad: about one-third of chunk choices favor MT, and readers cannot reliably detect it, guessing correctly only 17/30 times after comparing both versions. However, MT has more unstable weak spots, with more negative highlights and greater chunk-to-chunk variability, and automatic metrics, including LLM-as-judge methods, fail to recover reader preferences and tend to favor MT.