This week, we review:

• LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

• Distilled Reinforcement Learning for LLM Post-training

• DeepLoop: Depth Scaling for Looped Transformers

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

This paper replaces scalar reward optimization for open-ended tasks with Experiential Learning, where the evaluator acts as a coach rather than a judge. Given a prompt, a policy response, and task-specific rubrics, the coach produces reusable textual guidance about strengths, weaknesses, and response strategies. That guidance conditions a teacher model, whose token distributions are distilled into the policy using on-policy samples. The feedback model and the additional context are not required at inference time.

The method is tested with Qwen3-8B and OLMo-3-7B policies, using either the initial policy or GPT-4o as the coach.

Training uses 7,500 WildChat-IF prompts, while evaluation covers held-out WildChat prompts and four unseen open-ended benchmarks. Experiential Learning generally outperforms rubric-based RL, especially on AlpacaEval and WildBench, although it does not win every model–benchmark combination.

Rubric-based reinforcement learning trains a model using rewards derived from an explicit evaluation rubric. The rubric lists criteria such as correctness, completeness, relevance, style, or safety. An evaluator scores each response against those criteria, and the model is updated to produce responses that receive higher scores.

In LLM post-training, the evaluator is often another language model. Unlike textual coaching, rubric-based RL usually compresses the evaluation into one or several numeric rewards, so the policy learns which outputs score well but receives less detailed information about how to improve them.

An additional analysis finds that RL obtains larger training-set gains but transfers less effectively, consistent with optimization against idiosyncrasies of the scalar evaluator.

The central contribution is a practical way to preserve information that a scalar score discards, rather than a demonstration that textual feedback has a measurable information advantage of the stated magnitude. The bandwidth comparison is explicitly heuristic, and most results depend on model-based rubrics and evaluation. Iteratively updating the teacher also improves the target-domain score while substantially degrading out-of-distribution instruction following; mixing general-domain distillation data only partially recovers that loss.

Distilled Reinforcement Learning for LLM Post-training

Distilled RL incorporates a teacher’s token-level preferences directly into policy-gradient training instead of adding a separate distillation loss. For student-generated trajectories with positive advantage, teacher-to-student likelihood ratios redistribute credit among tokens. Teacher weighting is disabled for negative trajectories, where it would otherwise push the student away from teacher-preferred tokens, and sequence-level normalization prevents the teacher ratios from globally weakening or amplifying the update. The teacher therefore guides successful trajectories without becoming an unconditional imitation target.

Experiments train three student models on DAPO-17K and compare against GRPO, on-policy distillation, and a direct combination of the two.

The largest result is in cross-family distillation: the DeepSeek-R1-Distill-Qwen-1.5B student reaches a 40.00 average across ten mathematical benchmarks, versus 36.86 for RL and 35.27 for conventional distillation. Gains are smaller but consistent for Qwen3-4B and Qwen3-1.7B. Pass@16 and knowledge-intensive evaluations are mostly competitive, and ablations identify the negative-sample reset as the most important component, with its removal reducing average pass@1 by 6.39 to 8.81 points.

The method still assumes that the fixed teacher provides useful token preferences whenever the student has produced a successful answer. Because the teacher is only evaluated along student trajectories, training does not establish whether it can solve each problem itself. A student may also eventually outperform the teacher on particular examples. Teacher-competence gating or occasional teacher rollouts would be needed to make the supervision reliable in that regime.

DeepLoop: Depth Scaling for Looped Transformers

DeepLoop addresses optimization in Transformers that repeatedly apply the same physical blocks to obtain greater effective depth. Standard depth-scaling rules treat residual branches at different layers as independently parameterized, but looped models accumulate updates from repeated visits to shared parameters.

The paper models this effect through the alignment of visit-wise updates and derives a more conservative square-root residual and initialization scaling rule for the case where repeated visits remain aligned. The architecture otherwise remains a Post-LN looped Transformer.

At GPT-2 small and medium scales, DeepLoop is effectively neutral when each block is used once and reduces validation loss when blocks are revisited three, five, or seven times. On the medium model with seven loops, average downstream accuracy rises from 52.95% to 53.88% in zero-shot evaluation and from 54.62% to 55.20% in one-shot evaluation. Applying the same scaling to a hierarchical recurrent reasoner improves two-vote ARC-AGI-1 accuracy from 36.50% to 39.75%, with gains across all tested voting budgets.

The language-model comparisons use single runs at 124M and 350M parameters, so the relatively small loss and downstream differences are not yet statistically characterized. The ARC-AGI experiment includes a stronger multi-seed control, but it tests one recurrent architecture and dataset. The analysis also relies on an alignment coefficient that is not directly measured. Determining its behavior at larger scale, under other normalization schemes, and during longer training is necessary before treating the proposed exponent as a general rule for recurrent-depth models.

Google’s Gemma 4 technical report is surprisingly brief. Compared with the documentation released alongside many competing open-weight models, it leaves out much of the information that researchers and practitioners would normally expect, particularly about pre-training, data composition, optimization, and post-training.

That makes the report disappointing as a source of training transparency. It is still useful, however, because Gemma 4 introduces several architectural and deployment-oriented ideas worth examining in detail. The family combines a conventional decoder-only Transformer backbone with techniques designed to reduce memory use, improve long-context inference, support low-precision deployment, and accelerate generation through speculative decoding.

This article focuses primarily on the language-modeling components of Gemma 4.

I will examine its Transformer architecture, parameter allocation, local and global attention hierarchy, KV-cache optimizations, per-layer embeddings, quantization-aware training, speculative drafter, tokenizer, and the limited information Google provides about the training recipe.

For a more performance-oriented analysis covering accuracy, inference speed, token efficiency, memory consumption, and comparisons with models such as Qwen3.5/3.6, see my separate benchmark article in The Kaitchup:

The Kaitchup – AI on a Budget

Gemma 4 31B vs Qwen3.5 27B: Inference Speed, Token-Efficiency, Accuracy, and Memory Consumption

Qwen3.5 27B has established itself as one of the strongest LLMs under 100B parameters, delivering top-tier results across many tasks and even outperforming some much larger MoE models. As we showed in a previous article, it is also highly robust to quantization, especially when the attention layers are preserved. As a result, it quickly became the default choice for local AI…

Read more

4 months ago · 32 likes · 3 comments · Benjamin Marie

The common Transformer backbone

All Gemma 4 models are very standard causal, decoder-only Transformers. Text and modality-derived embeddings enter one autoregressive sequence, and the model generates one token at a time.

The residual blocks use both pre-normalization and post-normalization with RMSNorm. In a standard pre-norm Transformer, the input to attention or the feed-forward network is normalized before the sublayer, while the residual stream itself remains comparatively unconstrained. Adding a second normalization after the sublayer gives the model another point at which to regulate activation magnitude. This is relevant when combining long sequences, low-precision arithmetic and heterogeneous modality embeddings, all of which can produce less predictable activation distributions.

Read more

This week, we review:

• LLM-as-a-Verifier: A General-Purpose Verification Framework

• Morphing into Hybrid Attention Models

• AI translation of literary texts is “fine”, but readers still prefer human translations

Google also finally published the Gemma 4 Technical Report. I’ll publish a complete review next week!

LLM-as-a-Verifier: A General-Purpose Verification Framework

The paper’s core motivation is that LLM progress has mostly come from scaling generation: pretraining, post-training, and test-time sampling. The authors argue that verification, choosing which generated solution is actually correct, is an underdeveloped scaling axis. Their context is agentic systems, where multiple sampled trajectories often contain at least one correct solution, but current judges are too coarse to reliably pick it. On Terminal-Bench V2, they show that an oracle verifier could reach 98.9% by selecting the best sampled trajectory, while standard LM judges suffer from tied/discrete scores and trained reward models may not generalize across domains.

Their method, LLM-as-a-Verifier, turns an LLM judge into a more fine-grained verifier without additional training. Instead of asking the model to output a single discrete score, they extract the model’s probability distribution over scoring tokens and compute an expected continuous score. They then scale verification along three dimensions: score granularity, repeated evaluations, and criteria decomposition. The continuous rewards are converted into pairwise preferences using a Bradley–Terry formulation.

For selecting the best trajectory among many candidates, they introduce a Probabilistic Pivot Tournament: first run a ring pass to reduce positional bias, select promising pivots, then compare non-pivots against those pivots rather than doing all pairwise comparisons. This reduces the ranking cost from quadratic in the number of candidates to a much smaller pivot-based budget while concentrating compute on plausible winners.

Results are strong across domains: the method reaches 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, 87.4% preference accuracy on RoboRewardBench, and 73.3% on MedAgentBench. It also improves score separation on Terminal-Bench as granularity increases, correlates with task progress in code-generation trajectories, and can be used as a dense reward for RL, improving sample efficiency on LIBERO and MATH.

Morphing into Hybrid Attention Models

This paper addresses the long-context efficiency problem of Transformers. Full softmax attention gives strong retrieval and matching ability, but its computation grows quadratically with sequence length and its KV cache grows with context length.

Linear attention and state-space models are cheaper, but often lose recall-sensitive performance. Hybrid attention models try to keep a few full-attention layers and convert the rest to linear attention, but the key unresolved question is which layers should remain full attention. Existing methods mostly use fixed placement rules or score layers independently, ignoring how layers interact when converted together.

The proposed method, FlashMorph, reframes layer selection as a budget-constrained subset optimization problem. It first builds a “morphable” model by giving each full-attention layer a trained linear-attention branch, learned through hidden-state alignment with the original Transformer. Then it freezes both the original full-attention model and the linear branches, and optimizes only a small set of layerwise gates that interpolate between full and linear attention.

A key methodological detail is that the gates are optimized on synthetic long-context retrieval examples, not general language modeling data, because the authors want the selection signal to stress long-range information access. The objective combines hidden-state alignment with a linearization regularizer that pushes the model toward linear attention whenever possible. After optimization, the highest-gate layers are kept as full attention under the budget, the rest become linear attention, and the resulting hybrid model is finalized with logits distillation and long-context finetuning.

Results show that FlashMorph preserves strong long-context retrieval while drastically reducing layer-selection cost. On Qwen3-1.7B, it maintains perfect NIAH-Single-1 accuracy across 32K–256K contexts and improves harder NIAH settings while using only 20M layer-selection tokens. It also preserves general zero-shot commonsense performance, improves recall-heavy tasks, gives long-context speedups such as 2.81× prefill at 256K and 2.07× decode at 512K, and cuts selection cost to 2.1 GPU hours versus 15.4 for HALO, 1071.8 for KL-LS, and 2561.3 for PostNAS.

AI translation of literary texts is “fine”, but readers still prefer human translations

This paper studies a practical and cultural question: as AI-assisted literary translation enters publishing, do readers experience AI-translated novels the same way they experience human translations?

(Clearly not for me. I still feel like I can spot AI when used in translation and then I don’t read it the same. I’m wondering more often whether this is really translated correctly...)

The authors argue that standard MT evaluation, fluency, adequacy, automatic metrics, or short-segment judgmentsm misses what matters for literature: immersion, voice, rhythm, emotional effect, and sustained reading experience.

The method is a reader-centered evaluation dataset and protocol called LAIT. The authors compare recently published professional human translations with machine translations of 15 recent novels originally in French, Polish, and Japanese, all translated into English. They use roughly 8K-word excerpts, chosen from recent 2025–2026 publications to reduce training-data contamination risk. The MT side is produced with an agentic LLM pipeline: source analysis and style guidance, chunk-level translation and review, acceptance gates, revision loops, and excerpt-level consistency/literary review.

For evaluation, they recruit 15 avid readers. Each book is evaluated by two readers. Readers first do immersive reading of whole excerpts, then after a one-day break perform close reading of aligned ~300-word HT–MT chunks. The protocol collects ratings, direct preferences, explanations, AI-origin guesses, and span-level annotations of good or poor wording. This gives 30 excerpt-level comparisons, 772 chunk-level comparisons, about 1K reader comments, and 7.2K span annotations.

The main result is nuanced: readers often find MT readable and sometimes prefer it, but overall they still prefer HT, especially under close reading. HT wins 19/30 excerpt-level comparisons and 522/772 chunk-level comparisons; HT also gets higher ratings for acceptability and smoothness. MT is not simply bad: about one-third of chunk choices favor MT, and readers cannot reliably detect it, guessing correctly only 17/30 times after comparing both versions. However, MT has more unstable weak spots, with more negative highlights and greater chunk-to-chunk variability, and automatic metrics, including LLM-as-judge methods, fail to recover reader preferences and tend to favor MT.

This week, we review:

• Rethinking the Role of Efficient Attention in Hybrid Architectures

• Variable-Width Transformers

• The Hidden Power of Scaling Factor in LoRA Optimization

Rethinking the Role of Efficient Attention in Hybrid Architectures

The paper asks a mechanistic question behind modern hybrid LLMs: in architectures mixing full attention with efficient modules such as SWA (e.g., GPT-OSS), Lightning Attention, Mamba-2 (e.g., Nemotron 3), or Gated DeltaNet (e.g., Qwen3.5), do the efficient modules actually provide long-context capability, or do they merely change how full-attention layers learn?

Hybrid attention is now common for long-context efficiency, but prior work mostly reports end metrics or isolated ablations rather than explaining training dynamics and capability emergence.

Methodologically, they run a controlled scaling-law study over a full-attention baseline and six layer-wise hybrids: SWA with 128/512/2048 windows plus three recurrent mixers. Models span S1–S5, up to 665M total parameters, are pretrained at 16K context, and are assessed with validation loss for short-context modeling and LongPPL as a continuous proxy for long-context capability. The key empirical pattern is that validation-loss curves largely overlap across architectures, while LongPPL differs strongly in low-data regimes but converges with sufficient training.

The central mechanistic claim is that efficient attention is not the primary carrier of long-range information. Full attention is. They support this with inference-time receptive-field restriction, where constraining full attention hurts LongPPL much more than constraining efficient modules, and with layer-wise NIAH probing, where long-range information gains concentrate in full-attention layers.

They then explain different convergence speeds via “Large-Window Laziness”: large SWA windows satisfy much of the local next-token signal, weakening gradients that would otherwise force full-attention layers to develop retrieval heads. Gradient-influence profiling and checkpoint-level retrieval-head tracing show SWA-2048 has slower retrieval-head sharpening and slower Q/K convergence than smaller-window or recurrent hybrids.

The proposed design implication is to optimize the hybrid around full-attention retrieval rather than making efficient modules “stronger.” Their concrete intervention is SWA-128-NoPE: use small-window SWA to keep pressure on full attention, and apply NoPE only in full-attention layers to improve long-range retrieval.

Evaluation is intentionally secondary here: SWA-128-NoPE improves RULER/NIAH and LongBench at 16K/32K while keeping short-context averages essentially unchanged, but the authors note the experiments remain sub-billion-scale and may not fully transfer to frontier-scale training recipes.

Variable-Width Transformers

The paper targets a largely implicit design assumption in decoder-only Transformers: once a global hidden size is chosen, every block gets the same residual width and roughly the same capacity. The authors argue this is not obviously optimal because different depths perform different computational roles, and because parameter-matched variable-width models can have lower average per-layer width, reducing attention-side compute and KV-cache footprint.

The proposed architecture, > <former, uses a variable-width, X-shaped depth profile: early and late layers are wide, while middle layers form a bottleneck. The key implementation choice is to keep a fixed global residual stream, equal to the widest layer, while each Transformer block reads from and writes to only a layer-specific slice. Coordinates inactive in a narrow block bypass it and are copied forward. When width expands again, previously active coordinates are restored rather than learned through projection. This makes width changes parameter-free and preserves a skip-path interpretation instead of inserting learned residual adapters.

Width schedules are generated geometrically and controlled mainly by bottleneck location and bottleneck width. The authors sweep several global shapes and find the X-shaped profile works best, then use a ratio-based recipe with the bottleneck around three quarters through depth and about 30% of the baseline hidden width. Ablations indicate the carry-forward residual mechanism is important: zero padding is worse, and learned projection for expanded dimensions is worse still at the tested scale.

Evaluation is on dense LMs from 200M to 2B parameters and a 3B-total/1B-active MoE, trained on DCLM with constant-width baselines. > <former improves language-modeling loss across tested sizes while using lower pretraining FLOPs and lower average layer width; loss-matched scaling fits imply about 22% fewer FLOPs and about 15% lower average width. Zero-shot downstream results are strongest on perplexity metrics; NLU accuracy is mostly favorable for the 2B dense model and mixed for MoE.

The Hidden Power of Scaling Factor in LoRA Optimization

One more paper to discuss LoRA’s alpha.

The paper argues that LoRA’s scaling factor, alpha, has been mischaracterized as just another way to scale the learning rate. The authors’ motivation is that common LoRA practice fixes alpha through simple rank-tied heuristics, then compensates with unusually large learning rates. They claim this hides the real optimization bottleneck. Their empirical sweeps suggest that LoRA’s low-rank parameterization suppresses the effective curvature spectrum, making the landscape smoother but also leaving standard settings underpowered.

The core analysis is a “Signal-Drift” view of LoRA optimization. They decompose LoRA’s induced weight-space dynamics into a task-aligned signal term and a structural drift term caused by the bilinear adapter parameterization. In this framing, the learning rate and alpha are not interchangeable: increasing the learning rate scales both useful signal and destabilizing drift, while increasing alpha preferentially restores task-aligned curvature and preserves a better signal-to-drift profile under adaptive optimizers.

The proposed method, LoRA-α, is minimal: keep the standard small full-finetuning-style learning rate, and set alpha much larger than conventional LoRA defaults using either an empirical LLM-oriented sublinear rank rule or an analytic layer-wise rule derived from curvature alignment at initialization. The practical goal is to decouple capacity restoration from optimizer step-size tuning: alpha handles the curvature/signal restoration, while the learning rate remains conservative. Any remaining tuning is confined to a narrow scaling multiplier around the proposed alpha.

Evaluation is broad but secondary to the paper’s argument: they test encoder, decoder, diffusion, and vision-language models across supervised, contrastive, flow-matching, and RL settings. LoRA-α generally improves over vanilla LoRA and other LoRA variants, with gains stronger at higher ranks. In long-context reasoning SFT it is reported as the closest LoRA-style approximation to full fine-tuning.

This week, we review:

• ⭐How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs

• Dynamic Linear Attention

• Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation

⭐How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs

The paper introduces FlowTracer, a token-level credit assignment method for reinforcement learning on LLM reasoning tasks. The problem it targets is that RLVR and GRPO-style training usually apply outcome rewards broadly across generated tokens, even though only some tokens materially affect the final answer. FlowTracer instead uses the model’s own attention structure to estimate which generated tokens route information from the prompt toward the answer region. It represents the sequence as a token graph, derives directed edge strengths from aggregated attention, conditions the graph on the final answer span, and suppresses attention paths that do not reach that answer region.

Technically, the method converts raw attention into an answer-targeted flow graph. Raw attention is treated as a local token-to-token interaction signal, but the authors argue that it is insufficient by itself because many attention paths end in irrelevant branches or overemphasize tokens near the answer. FlowTracer reweights the graph so that effective influence is conserved through intermediate tokens and only answer-reaching paths are retained. It then propagates flow from question tokens to answer tokens and scores each token by throughput, treating high-throughput tokens as routing hubs in the reasoning trace.

The authors use these throughput scores to modify GRPO training. After response sampling, they run one extra forward pass to extract middle-layer attention maps, compute flow scores, select the top 40% high-flow tokens, and give those tokens a larger policy-update weight. In their implementation, the high-flow token multiplier is fixed at 1.5. Their qualitative and intervention analyses suggest that high-flow tokens often include structural delimiters, repeated symbols, variables, punctuation, or aggregation points rather than ordinary fluent filler. When they mask attention from the top 20% high-flow tokens on GSM8K, outputs change more often than when masking random or low-flow tokens, supporting the claim that the selected tokens have stronger causal influence on the final answer.

Experiments compare FlowTracer with GRPO and token-prioritization heuristics based on random selection, entropy, gradients, correlation, and attention scores. The main evaluations use Qwen3-4B and Qwen3-8B on math benchmarks including AIME24, AIME25, AMC23, MATH500, and OlympiadBench, with additional tests on Countdown and CrossThinkQA; supplementary results use Llama-3.1-8B and Llama-3.2-3B. Reported gains are consistent but moderate in most settings, with larger gains in some long-context and puzzle-solving cases. The paper also reports that middle-layer attention works better than early, late, or all-layer aggregation, that hard top-40% selection is more stable than continuous weighting, and that the added computation is about 2.1% to 4.5% per training step.

The stated limitations are that the method assumes a localized answer region, remains tied to outcome rewards, and may require adaptation for very long contexts or open-ended tool-use trajectories.

Dynamic Linear Attention

The paper proposes Dynamic Linear Attention (DLA), a memory modeling framework for long-context language models using linear attention. Its starting point is that standard self-attention has quadratic cost, while linear attention reduces cost but often compresses too much history into a limited state representation.

The authors focus on multi-state linear attention, where past tokens are summarized into multiple memory states, and argue that prior methods such as Log-Linear Attention use fixed merging schedules that do not adapt to uneven information density across a sequence. This can merge semantically important transitions into coarse summaries too early and accumulate errors over long contexts.

DLA replaces fixed state construction with information-aware dynamic state merging. For each incoming token representation, the method computes a lightweight score measuring how much it differs from the most recent memory state. Tokens with low variation are merged into the current state, while tokens with larger representational drift start a new state. The intended effect is to keep finer resolution around semantic changes and compress locally redundant spans more aggressively. The paper’s analysis frames this as reducing within-block heterogeneity during summarization, which is the main source of approximation error in blockwise memory compression.

The second component is capacity-bounded memory modeling. DLA maintains a chronological cache of memory states with a fixed maximum size. Each state tracks both how many tokens it summarizes and an aggregate information score. When the cache reaches capacity, the method merges the adjacent pair with the lowest information density, preserving temporal order while freeing space for newer states. At decoding time, the model reads from these stored states using learned query-dependent weights, so the cache remains bounded while still allowing the model to emphasize more relevant states.

The authors evaluate DLA by pretraining variants of Mamba-2-780M and Gated DeltaNet-1.3B from scratch on 50B tokens with 16K sequence length, using the same maximum state count as Log-Linear Attention. Evaluation covers 16 datasets: commonsense reasoning, in-context retrieval, RULER, and LongBench. The reported results show that DLA improves over vanilla and Log-Linear variants on these evaluations, including gains on retrieval-heavy long-context tasks and LongBench categories such as QA, summarization, and few-shot learning. Efficiency tests on a single A100 report higher throughput and lower memory use than the Log-Linear variant, though both multi-state approaches remain heavier than vanilla Mamba-2 because they cache multiple summary states. Ablations indicate that both dynamic state merging and the bounded-memory component contribute to the final results, and that moderate changes to cache size and merge threshold have limited impact.

Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation

The paper studies whether long teacher reasoning traces can be compressed before supervised knowledge distillation to reduce training and inference cost.

The proposed Compress-Distill pipeline first generates verified-correct reasoning traces from large teacher models, then rewrites those traces with separate instruction-tuned compressor models, and finally fine-tunes student models on either raw traces, compressed traces, or answer-only outputs. The two teachers are Qwen3.5-397B-A17B and gpt-oss-120B; the compressors are Llama-3.3-70B-Instruct and Ministral-3-14B-Instruct-2512. The authors report about 283k correct traces per teacher, with compressed traces reduced to 8.6–21.0% of original character length depending on teacher and compressor.

Experimentally, the study covers two teachers, four student models, LoRA and full fine-tuning, raw and compressed trace sources, answer-only ablations, and additional length-matched truncation ablations. The students include Qwen3.5-0.8B-Base, Qwen3.5-9B-Base, Llama-3.1-8B, and gpt-oss-20B. Training uses next-token prediction on assistant tokens, sample packing, BF16, FlashAttention 2, and a one-epoch setup; evaluation uses greedy decoding with an 8,192-token cap across in-distribution reasoning datasets and out-of-distribution reasoning and knowledge benchmarks.

The main result is an accuracy-efficiency trade-off. Compressed traces reduce training tokens to 12–30% of raw traces, reduce wall-clock training time by 2.0–7.6x, and shorten inference outputs by roughly 3–19x, with smaller gains for the already-shorter gpt-oss teacher traces. However, raw teacher traces give the highest overall downstream accuracy in every evaluated teacher, student, and training-method configuration. For example, under full fine-tuning, Qwen3.5-9B trained from Qwen raw traces reaches 0.866 overall accuracy, compared with 0.834 for Llama-70B-compressed traces and 0.817 for Ministral-14B-compressed traces; gpt-oss-20B trained from gpt-oss raw traces reaches 0.844, compared with 0.776 and 0.767 for the two compressed variants.

This week, we review:

• ⭐Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling

• Less is More: Early Stopping Rollout for On-Policy Distillation

• Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

⭐Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling

This paper studies a bottleneck in parallel test-time scaling: multiple reasoning branches often rediscover the same intermediate information because they do not communicate until final answer aggregation.

The proposed method, Collaborative Parallel Thinking (CPT), keeps the branches separate but periodically extracts compact intermediate findings, deduplicates them into a query-level shared pool, and broadcasts selected entries back into the context for subsequent decoding.

CPT is training-free and uses the same policy model both for reasoning and for extracting information units. Sharing is not active throughout the whole run: the method first lets branches explore independently, starts broadcasting when marginal new information drops, and stops synchronization when further sharing appears unlikely to help. This design targets the latency cost of redundant exploration while trying to avoid early collapse of branch diversity.

Experiments on HMMT and AIME math benchmarks with Qwen3-Thinking models compare CPT against parallel sampling, DeepConf, and LeaP across rollout budgets. CPT improves the accuracy–latency frontier in these settings, with gains in both individual branch accuracy and majority-vote accuracy. The analysis suggests that its benefit comes from reducing duplicate intermediate discoveries rather than only making branches agree earlier.

The main caveat is implementation cost. CPT shares information through prompt-context updates, which can require re-prefilling and add FLOPs even when generated-token usage and wall-clock latency improve. The idea is most compelling where parallel decoding is already available and latency matters more than raw FLOPs.

Less is More: Early Stopping Rollout for On-Policy Distillation

The paper studies a failure mode in on-policy distillation, where a student generates rollouts and a teacher provides token-level supervision on those rollouts.

The authors argue that late-token supervision can become unreliable because the teacher is conditioned on a prefix produced by the student rather than by itself. As the student prefix drifts away from the teacher’s preferred trajectories, the teacher increasingly behaves like a next-token completer rather than a corrective evaluator.

The proposed fix, Early Stopping Rollout: generate and train only on the first part of the student response, typically the first 100 tokens, while leaving the rest of the on-policy distillation loop unchanged. Across math, coding, and function-calling experiments, this truncated-rollout version usually matches or outperforms full-rollout distillation, with the largest gains in cross-generation and cross-family teacher–student pairs where full-rollout training is less stable. The method also reduces generation and training cost substantially because it avoids long student rollouts.

The analysis suggests that early tokens carry disproportionate value because they often encode framing, planning, and commitment to a solution strategy. Training only on this early window can still reduce teacher–student divergence at later positions, an effect the authors call cascading alignment. They also report that the student can converge to a useful supported sub-mode of the teacher distribution rather than copying the teacher’s dominant behavior, which may explain cases where the distilled student exceeds the teacher on the reported metrics.

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

This work studies the learnable scale vectors in normalization layers of LLMs, a small parameter group that is usually treated as an implementation detail. The central result is that these vectors matter primarily through optimization, not expressivity: in Pre-Norm Transformers they can be absorbed into adjacent linear maps, yet removing them worsens pre-training loss and token efficiency.

The authors analyze how scale vectors act as lightweight, state-dependent preconditioners for nearby linear mappings. This framing leads to several design choices: using branch-specific scale vectors, changing where scale vectors are placed around linear maps, reparameterizing their magnitude and direction, and applying weight decay differently depending on whether a scale vector sits before or after a linear map.

Experiments validate these choices in dense and MoE Llama-style models from 0.12B to 2B parameters, under long-token pre-training budgets and across AdamW, Muon, and warmup-stable-decay schedules. The combined strategy consistently lowers terminal validation loss relative to tuned baselines, with small wall-clock and memory overhead.

This week, we review:

• ⭐Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

• Steered LLM Activations are Non-Surjective

• STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

⭐Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

On-policy distillation (OPD) is framed here as efficient because it finds useful parameter-update directions early, rather than merely because it supplies denser supervision.

The central observation is that OPD quickly settles into a stable trajectory toward the final model. At the module level, updates concentrate on reasoning-relevant regions while avoiding low-return changes in embeddings and less sensitive layers. At the update-direction level, the change in weights is more concentrated in low-rank subspaces, and those dominant directions align early with the final update direction.

The method, EffOPD, periodically extrapolates along the current update direction, uses a small validation set to choose how far to move, and rejects extrapolations that hurt validation performance. It adds no new trainable modules and is intended to be dropped into existing OPD pipelines. Reported results show about 3× average training acceleration across model sizes from 1.5B to 32B and tasks including math reasoning and code generation, with final performance close to standard OPD.

Steered LLM Activations are Non-Surjective

Activation steering directly modifies a model’s internal activations to induce a behavior.

This work asks whether those steered internal states could also be produced by some ordinary text prompt. It frames the question as prompt reachability: given a fixed model, do steered activations have any preimage under the model’s normal forward pass from discrete prompts?

The main result is negative under standard assumptions.

Because prompts form a discrete set while residual-stream activations live in a continuous space, the naturally reachable activation states occupy only a sparse subset of possible states. Adding a steering vector almost surely moves the model off that prompt-reachable set. The argument covers random steering vectors, common difference-of-means steering vectors, and even adversarial vectors that force a one-step match but almost surely fail to preserve the match over a sequence.

The experiments are consistent with the theory. Natural activations can be inverted back to their original prompts, but steered activations cannot be inverted in the same way across Llama, Qwen, and Gemma models. When the closest prompt tokens are forced, they usually reconstruct the original prompt rather than a prompt that reproduces the steered behavior. Many-shot in-context examples can sometimes produce similar surface behavior, such as bypassing refusals, but their internal activations move farther from the steered trajectories rather than closer.

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

STALE studies a failure mode in long-term LLM-agent memory: a later observation can make an earlier user-state belief invalid without explicitly correcting it.

This is more demanding than fact retrieval because the agent must infer the current state, reject stale premises embedded in user queries, and adapt downstream behavior accordingly. The benchmark contains 400 expert-validated conflict scenarios, 1,200 probes, more than 100 everyday topics, and contexts up to 150K tokens.

The evaluation separates three capabilities: detecting outdated beliefs, resisting queries that assume outdated beliefs are still true, and applying the updated state in ordinary downstream tasks. Current frontier LLMs and memory frameworks perform poorly, with the best evaluated model reaching only 55.2% overall accuracy. The main pattern is that models can often retrieve the newer evidence but fail to adjudicate between old and new beliefs, especially when the conflict is indirect or hidden inside the user’s phrasing.

CUPMem reframes memory as current-state management rather than retrieval. It performs write-time consolidation, marks or replaces stale beliefs, searches state slots that may be structurally affected by a new observation, and restricts generation to active state. The result points to an important design lesson: persistent agents need explicit revision and dependency handling, not just larger memory stores or better retrieval.

The useful contribution is the evaluation target. Robust agent memory should be measured by whether the system maintains a coherent and up-to-date model of the user under implicit change.

DeepSeek-V4 is a million-token-context model family.

The paper introduces two MoE models: DeepSeek-V4-Pro, with 1.6T total parameters and 49B activated per token, and DeepSeek-V4-Flash, with 284B total parameters and 13B activated per token.

• deepseek-ai/DeepSeek-V4-Pro

• deepseek-ai/DeepSeek-V4-Flash

Both are built around the same core goal: make very long context practical without paying the full cost of vanilla attention.

In constrast with DeepSeek V3, the main architectural changes are hybrid compressed attention and a new residual-stream mechanism called Manifold-Constrained Hyper-Connections, or mHC.

The short version: DeepSeek-V4 is still a Transformer-style MoE model, but its attention layers no longer treat the past as a flat list of all previous tokens. Instead, the model stores compressed summaries of the past, selectively retrieves some of them, and keeps a small exact local window for recent tokens. That is the key architectural idea.

In this article, let’s review how it works, particularly focusing on the attention module and its efficiency, and its training recipe.

Read more

This week, we review:

• ⭐Why Fine-Tuning Encourages Hallucinations and How to Fix It

• Temporally Extended Mixture-of-Experts Models

• Micro Language Models Enable Instant Responses

DeepSeek-V4 was released last week. While its accuracy doesn’t particularly stand out, the public report is highly insightful and introduces a number of new techniques. I’m preparing a full review.

⭐Why Fine-Tuning Encourages Hallucinations and How to Fix It

The work frames SFT-induced hallucination as a continual-learning problem: when a model is fine-tuned on facts it did not previously know, the same updates that add those facts can degrade facts it already knew.

The experiments separate task learning from factual learning by training on known and unknown QA pairs while evaluating held-out known facts. The key finding is that ordinary SFT first improves task-format behavior, then begins acquiring new facts, and that second phase coincides with a substantial drop on previously known held-out facts. When unknown facts are removed from training, the degradation largely disappears.

The proposed fixes depend on whether the fine-tuning should actually teach new facts. When factual updates are not needed, restricting trainable parameters can preserve task performance while suppressing factual plasticity. In the main experiments, updating attention while freezing other groups reduces forgetting because the model learns the QA behavior without absorbing many new facts. When factual learning is required, the paper uses self-distillation: after an initial task-adaptation phase, a frozen snapshot regularizes the continuing model’s output distribution. This keeps new-fact learning close to standard SFT while reducing held-out factual degradation from about 15% to about 3%.

The mechanism study argues against a purely behavioral explanation and against simple global capacity limits. Synthetic facts with name-like entity keys cause forgetting that grows with scale, while UUID-style keys cause little forgetting even at very large scale, despite equivalent supervision and successful learning. The interpretation is localized interference: new facts whose entity representations overlap with existing ones perturb nearby factual representations. Self-distillation appears to work because it limits this representational drift rather than merely shrinking weight movement.

The practical implication is that “fine-tune less” is too blunt. For alignment, formatting, privacy, or domain adaptation that should not rewrite facts, constrain factual plasticity. For updates that must add knowledge, use an explicit stability objective and evaluate not only target-domain gains but regression on previously reliable facts, especially facts that are semantically close to the new training data.

Temporally Extended Mixture-of-Experts Models

This paper addresses a practical weakness in sparse MoE LLMs: expert routing changes so often that offloading and prefetching become hard to use effectively.

The proposed fix is to make expert selection persistent over multiple tokens. Each MoE layer keeps an active expert mask, and a small controller decides whether to keep that mask or switch to a new one. The normal router then selects experts only from the active mask. This casts expert loading as a temporally extended decision rather than a per-token choice.

The implementation adds a per-layer controller with a termination head, value heads, and a selection head initialized from the router. Training uses an option-critic objective with a deliberation cost that penalizes switching, plus self-distillation against the frozen base MoE and LoRA updates to the model.

Experiments are run on gpt-oss-20b, a 24-layer MoE with 32 experts per layer and top-4 routing, trained on Nemotron Post-Training Dataset v2 and evaluated on 200-question subsets of MATH, MMLU, and MMMLU. With 16 allowed experts, the lowest deliberation-cost setting reaches 64.0% on MATH, 72.5% on MMLU, and 59.5% on MMMLU, compared with base-model scores of 71.5%, 79.5%, and 67.5%.

The main caveat is that the memory and latency benefits are argued from the routing structure, while end-to-end offloading measurements are left for future systems work. The deliberation cost is also a tunable hyperparameter rather than a measured hardware cost. The evaluation is narrow: one base model, three benchmark families, 200 samples each, and independent per-layer options rather than a cross-layer switching plan that would better match real expert loading.

Micro Language Models Enable Instant Responses

This work proposes a device-cloud split for latency-sensitive assistants on wearables and other constrained devices. A very small on-device language model generates only the first few words of a response, while a much larger cloud model continues from that prefix. The key design choice is to treat the local model as a response initiator rather than a full assistant, and to prompt the cloud model as a continuation model rather than a respondent starting from scratch.

The local models are decoder-only Transformers in the 8M–30M parameter range, trained on cleaned chat-style data. The strongest variants remain far below today’s “small LLM” scale but are competitive with several larger compact baselines on short dialogue-opening evaluations. In the collaborative setup, committing 4–8 words works best: it gives the cloud model enough semantic direction while keeping the correction rate low. Longer local prefixes degrade handoff quality and increase the chance that the cloud model must repair a bad start.

The empirical case is strongest on perceived latency and handoff feasibility. On Orange Pi hardware, the 28M model reaches first token in 45 ms and four words in 55 ms, while also using less energy per token than a larger small-model baseline. In a 15-person user study, collaborative responses were rated equivalent to standalone cloud responses about half the time, preferred 28% of the time, and dispreferred 22.7% of the time. The authors also test explicit, natural, and humor-based recovery strategies for bad prefixes; users prefer integrated recovery over visible correction, though the safer choice depends on deployment context.

The main limitation is that this is not a general replacement for on-device reasoning. The method depends on short, everyday prompts, a cloud model that can follow continuation instructions, and a UI that can exploit partial output without making failures feel jarring. The paper is most useful as a systems framing: for many assistant interactions, reducing perceived time-to-first-response may matter more than moving the whole model onto the device.

This week, we review:

• ⭐Accelerating Speculative Decoding with Block Diffusion Draft Trees

• IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

⭐Accelerating Speculative Decoding with Block Diffusion Draft Trees

DDTree is a speculative decoding method for block diffusion drafters. Starting from DFlash, which can predict an entire draft block in one pass but still verifies only a single drafted trajectory, DDTree uses the same one-pass outputs to build a compact tree of candidate continuations. The main idea is that block diffusion already exposes useful uncertainty over future positions, and the missed opportunity in vanilla DFlash is collapsing that uncertainty into just one path.

A one-pass block diffusion drafter does not provide autoregressive, path-conditioned probabilities. It provides per-position marginals. DDTree defines its selection objective on the factorized distribution induced by those marginals, then uses a best-first heap procedure to choose the tree that maximizes expected accepted prefix length under a fixed node budget. Verification is still done in a single target-model forward pass with ancestor-only attention, so the method preserves cheap drafting while expanding the set of continuations that can be accepted in each round.

DDTree improves over vanilla DFlash in all 60 dataset-model-temperature settings reported, covering reasoning, code, and instruction benchmarks. At temperature 0, Qwen3-8B goes from 5.56× to 7.52× on MATH-500, from 4.84× to 6.90× on HumanEval, and from 4.78× to 6.75× on GSM8K; the 30B coder model also improves consistently, including 6.09× to 8.22× on HumanEval.

IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

This paper is about inference-time KV-cache management for long-context LLMs under tight GPU memory.

The main claim is that existing page-based or offloading methods lose accuracy because they keep pages in token order, so query-relevant tokens end up scattered across many pages. IceCache changes the unit of organization: it clusters semantically similar key embeddings and stores them together, then retrieves pages by searching a hierarchical index rather than scanning the cache in sequence order. In effect, it turns page selection into an approximate nearest-neighbor lookup over grouped tokens, which is meant to improve recall of useful cache entries while reducing transfer waste between CPU and GPU.

During prefill, transformed key embeddings are inserted into a per-head hierarchical structure called a DCI-tree, whose leaf nodes map to physical KV pages. During decoding, the current query searches that tree to find the most relevant nodes/pages, and those pages are fetched for sparse attention. The system also overlaps index maintenance with GPU work so that retrieval and offloading do not fully sit on the critical path.

The empirical result is a fairly strong accuracy-efficiency tradeoff: on LongBench, the paper reports that a 256-token budget preserves about 99% of full-KV accuracy, and on Llama 3.1 8B a 64-token budget reaches 47.8 average score, slightly above PQCache at 47.3 despite PQCache using four times the budget. It also reports 100% passkey retrieval accuracy from 10k to 100k words across budgets 64, 128, and 256, plus 47.4 on GSM8K CoT at a 10% budget versus 48.2 for full KV.

This week, we review:

• ⭐TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

• LightThinker++: From Reasoning Compression to Memory Management

• Reasoning Shift: How Context Silently Shortens LLM Reasoning

⭐TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

TriAttention is a KV-cache compression method for long-chain reasoning under RoPE.

Its central argument is that recent-query attention is a weak basis for eviction because post-RoPE queries keep rotating with position, so only a very small observation window is actually representative. The paper moves the analysis to the pre-RoPE space and reports that many heads have query and key vectors concentrated around stable, non-zero centers. From that, it derives a distance-dependent attention preference and uses it to score cached keys by predicted future usefulness, with Q/K norms added as a fallback signal when concentration is weaker. In practice, the cache is pruned by retaining the top-scoring keys at periodic intervals during decoding.

Empirically, the method is tested on reasoning-oriented models and benchmarks rather than broad language modeling: Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and GPT-OSS-20B, with AIME24, AIME25, and MATH 500 at generation lengths up to 32k tokens (which is quite short for more recent models like Qwen3.5/Gemma 4).

The result pattern is consistent: TriAttention is materially better than prior compression baselines at the same cache budget, and in several settings it stays relatively close to full attention.

LightThinker++: From Reasoning Compression to Memory Management

LightThinker++ extends the original LightThinker from step compression to explicit memory management.

The base idea in LightThinker is to compress intermediate reasoning into compact semantic representations and discard the raw chain, reducing what must remain in context. The new claim is that this is not enough for harder problems: once compression becomes irreversible, the model can hit logical bottlenecks. LightThinker++ reframes efficient reasoning as active context management, adding explicit memory primitives so the model can control what is kept, compressed, and reused during inference.

Instead of only teaching the model how to summarize thought traces, the authors train it to manage memory deliberately through a trajectory synthesis pipeline designed to produce purposeful memory actions. In practice, the paper presents LightThinker++ as a reversible and adaptive alternative to one-shot thought compression, aimed at both standard reasoning workloads and longer agentic interaction loops where context growth is the core failure mode.

For the original LightThinker, the paper reports a 70% reduction in peak token usage and a 26% reduction in inference time with little accuracy loss. For LightThinker++, under the same context budget, peak token usage drops by 69.9% while accuracy increases by 2.42%. In long-horizon agentic settings, the system is reported to keep a roughly stable memory footprint beyond 80 rounds, with a 60%–70% reduction in token footprint and an average performance gain of 14.8% across the tested scenarios.

Reasoning Shift: How Context Silently Shortens LLM Reasoning

Do reasoning models solve the same problem the same way once that problem is embedded in extra context?

The answer is largely no.

Across three settings, an irrelevant long prefix, a multi-turn chat with independent earlier tasks, and a prompt that bundles two independent problems, the models tend to produce much shorter reasoning traces than they do when the target problem is presented alone. Context changes how much explicit reasoning the model performs, even when the task itself is unchanged.

The main empirical result is a consistent compression of reasoning length, often substantial. On IMOAnswerBench, the reported reduction reaches roughly 50% in some non-isolated settings, with accompanying accuracy drops in the subtask and long-input conditions for all four evaluated reasoning models.

On MATH500, adding even a few hundred irrelevant tokens already shortens reasoning, and scaling the distractor prefix further pushes the reduction toward 50%. The effect is present in both thinking and non-thinking modes for Qwen3.5-27B, but it is much stronger in thinking mode, which supports the paper’s framing that long-context conditions specifically suppress deliberative behavior rather than merely shortening responses in general.

This week, we review:

• ⭐Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

⭐Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

The paper studies a specific failure mode of self-distillation in mathematical reasoning. Its main claim is that when the teacher is conditioned on rich side information, especially full solutions, the student learns a shorter and more confident reasoning style that suppresses explicit signs of uncertainty such as checking, reconsidering, or branching.

The authors argue that these signals are functionally useful in math: they help the model keep alternative hypotheses alive and recover from bad intermediate steps. In their controlled generation study, increasing teacher-side information sharply reduces both trace length and uncertainty markers, for example, on DeepSeek-R1-Distill-Qwen-7B, unguided generation averages about 13k tokens and 182.5 epistemic markers, while full-solution guidance drops this to about 1.9k tokens and 8.8 markers.

The central empirical result is that this compression can damage reasoning even when the distilled targets are correct.

In off-policy SFT, the authors build two 800-example datasets of correct trajectories: one from unguided long-form reasoning and one from short solution-guided reasoning. Training on the long unguided traces leaves performance roughly stable, but training on the short guided traces causes large drops across benchmarks: for DeepSeek-R1-Distill-Qwen-7B, AIME24 falls from 54.79 to 20.21, AIME25 from 37.92 to 12.71, AMC23 from 89.06 to 57.03, and MATH500 from 92.19 to 65.52.

In on-policy training, the same pattern shows up across DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, and OLMo3-7B-Instruct: SDPO shortens responses more aggressively than GRPO, suppresses uncertainty markers more strongly, and can reduce out-of-distribution accuracy by about 40 percent on AIME24 in the strongest reported case. The effect is weaker when the teacher context is made less informative, such as removing the <think> content from the provided solution.

For post-training practice, the useful takeaway is to treat uncertainty expression as a capability rather than as formatting waste.

MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

This paper proposes MSA, a latent-memory transformer design for contexts that are far beyond standard long-context LLM ranges. It replaces dense self-attention over the full memory with document-level sparse attention: the model routes to a small set of relevant documents, loads compressed KV state for those documents, and then generates from that assembled sparse context.

The retrieval step is part of the model rather than a separate pipeline, so the architecture is trained end to end instead of using a decoupled retriever-reader stack. A second key idea is document-wise RoPE, which resets positional indexing per document so training on shorter contexts can generalize to very large document banks at inference. The paper’s claim is that this combination gives near-linear scaling in memory size while keeping retrieval fidelity much more stable than standard long-context baselines.

Empirically, the paper is strongest on robustness under extreme scale.

The authors build MSA on a Qwen3-4B backbone, use continual pre-training on a 158.95B-token corpus, and evaluate both long-context QA and needle-in-a-haystack retrieval. On RULER NIAH, it holds 94.84% accuracy at 1M tokens, whereas the unmodified Qwen3-4B baseline collapses to 24.69% at 1M.

What seems most important here is not just the headline context length, but the way the system treats memory as a native latent-state operation instead of bolting retrieval on top.

This week, we review:

• ⭐Efficient Exploration at Scale

• Efficient Reasoning with Balanced Thinking

• Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing

The week of “efficiency,” it seems.

⭐Efficient Exploration at Scale

The paper proposes an online RLHF pipeline that updates the reward model and policy incrementally as preference data arrives, instead of collecting a fixed batch and retraining offline. The core idea is to combine on-policy data collection with uncertainty-aware query selection: they keep a point-estimate reward model for training the policy, add an epistemic neural network head to model reward uncertainty, and choose response pairs for labeling by maximizing the variance of predicted choice probabilities across ensemble particles.

A smaller but important detail is the “affirmative nudge,” a constant positive offset in the policy update that is introduced to avoid the training collapse they say appears in prior online RLHF variants.

Empirically, the comparison is against offline RLHF, periodic RLHF, and a simpler online RLHF baseline, all built on Gemma 9B. Feedback is not from humans but from a Gemini 1.5 Pro–based preference simulator trained on human data, using about 200K training prompts and win rate against the SFT baseline as the main metric. In that setup, the uncertainty-guided method reaches roughly the same win rate at about 20K choices that offline RLHF reaches only after more than 200K choices, while plain online RLHF and periodic RLHF help but do not close the gap. The paper’s main contribution is therefore not a new preference objective so much as a systems-level argument that RLHF becomes much more label-efficient when reward learning, policy improvement, and exploration are all kept online.

What seems most solid here is the local claim: online, uncertainty-guided exploration materially improves data efficiency in their simulator. The broader scaling claim should be read more carefully. The headline 1,000x figure is not measured directly; it comes from extrapolating fitted curves, and the entire study depends on a single simulator-based evaluation loop rather than fresh human labels.

So the paper is strongest as evidence that RLHF data efficiency is being bottlenecked by exploration and update schedules, and weaker as evidence that the same scaling law will survive real annotators, different prompt mixes, or stronger base models.

Efficient Reasoning with Balanced Thinking

This paper presents ReBalance, a training-free test-time control method for reasoning models. Instead of trying only to shorten chain-of-thought, it treats inefficient reasoning as a balance problem between overthinking and underthinking.

The mechanism is simple in outline: use stepwise confidence and confidence variance to detect which regime the model is in, build a steering vector from hidden-state prototypes associated with those regimes, and then modulate that vector during decoding so the model trims redundant detours when it is wavering and explores more when it is prematurely confident. The steering vector and control surface are fit once per backbone from a small seen set of 500 sampled MATH problems, then reused across evaluation tasks, with no extra forward passes beyond ordinary decoding.

The main empirical result is that this control scheme usually improves efficiency without paying the usual accuracy tax. On six math benchmarks, the authors report gains of up to 7.0 Pass@1 points and token reductions of up to 52.3% relative to the base model. Representative examples are DeepSeek-R1-Distill-Qwen-1.5B on MATH-500, which moves from 79.6 to 83.0 while cutting tokens from 4516 to 3474, and QwQ-32B on MATH-500, which goes from 94.8 to 95.2 with tokens reduced from 4535 to 3662. The effect is not limited to math: with the same math-derived steering and control surface, the method is reported to transfer across science, commonsense, and code tasks while shortening traces by as much as 29.9%.

Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing

This paper is about getting multi-token decoding out of a standard autoregressive LLM without retraining it.

The method appends synthetic mask-token embeddings to the prompt, uses the frozen model to predict several future tokens in parallel, organizes those guesses into a speculative tree, then verifies them against the base model so decoding stays lossless. The interesting claim is that the model already contains enough latent structure for this to work: later decoder layers push the mask-token states toward the states of the true future tokens, and the algorithm turns that into a practical inference scheme rather than a training objective.

Empirically, the paper evaluates Llama 3 and Qwen3 models on SpecBench against other training-free baselines such as Prompt Lookup Decoding, STAND, and Lookahead Decoding.

Higher accepted-token counts per model call and higher throughput at matched block complexity, with especially clear gains on the Llama models. For example, on Llama 3.1-8B-Instruct, average accepted length rises from 1.38 to 1.62 at BC=30 and from 1.51 to 1.71 at BC=60 relative to LADE, while throughput rises from 32.6 to 38.9 and from 35.6 to 40.5 tokens per second; the method also reduces forward calls more than the baselines. The paper also argues that the right tree shape depends on task type: open-ended tasks tend to like a single, wider mask-token probe, while more constrained tasks can benefit from deeper probing with two masks.

The paper shows that frozen models can be probed more effectively than current training-free decoders assume, but it does not compare against trained multi-token systems such as auxiliary-head or draft-model methods, and some tasks still favor cache-based baselines. It is also evaluated in a fairly controlled setup: single-GPU measurements, fixed generation length, and a simplified tree policy that expands only the top path at deeper levels.

This week, we review:

• ⭐Attention Residuals

• LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

• FlashSampling: Fast and Memory-Efficient Exact Sampling

⭐Attention Residuals

Standard PreNorm residual connections implicitly sum all prior layer outputs with equal weight.

As depth increases, this unweighted accumulation grows the hidden-state magnitude and reduces the relative influence of any single layer, making early features harder to reuse and pushing later layers to “shout” to be heard.

The authors (the team behind the KIMI models) propose Attention Residuals (AttnRes): instead of always adding the previous state, each layer forms its input by taking a softmax-weighted mixture over earlier layer outputs (plus the embedding). The weighting is content-dependent through the keys (the earlier outputs), while keeping the query extremely lightweight (a learned per-layer vector) and normalizing keys to prevent large-magnitude layers from dominating.

Full AttnRes requires access to many prior activations, which becomes expensive under activation recomputation and pipeline parallelism because those states must be retained and communicated. To make the idea scalable, Block AttnRes groups layers into a small number of blocks, summarizes each completed block into a single representation, and runs depth-wise attention over these block summaries (and the current within-block partial sum).

The report adds system-level mechanisms to keep overhead small: caching block representations across pipeline stages to avoid repeated transfers, and a two-phase inference schedule that batches “inter-block” attention once per block and then merges it with sequential within-block updates efficiently; for long-context prefilling, it shards block caches across tensor-parallel devices to control memory.

Empirically, the depth-wise attention residual improves validation loss consistently across model sizes, and the block approximation recovers most of the gain while staying close to standard training and inference costs. Integrated into a large MoE Transformer pre-trained on a trillion-scale token budget, the method improves downstream results broadly, with the biggest gains appearing on multi-step reasoning and code generation rather than pure recall.

Diagnostics show output magnitudes stop drifting upward with depth and gradient norms become more evenly distributed across blocks, suggesting less “early-layer overtraining” and less reliance on ever-larger late-layer updates. Ablations also support the core claim that selective, input-conditioned depth mixing matters: fixed mixing over depth largely fails to help, sliding-window variants underperform, and relatively small block counts capture most of the benefit.

LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

LookaheadKV tackles a recurring bottleneck in long-context LLM inference: KV caches make decoding efficient, but prompt KV grows linearly with context and becomes a memory and latency constraint.

Prior eviction methods either rely on cheap attention heuristics that degrade quality under tight cache budgets, or “look ahead” by explicitly generating a draft response to estimate which prompt tokens will matter, improving accuracy but adding substantial prefilling overhead.

The paper’s central idea is to keep the accuracy benefits of future-aware eviction while removing the need to generate any draft tokens at inference time.

The method augments a frozen transformer with two lightweight, trainable components used only during prefill:

1. A small set of learned “lookahead” soft tokens appended to the prompt whose attention queries are trained to mimic the model’s future response attention toward the prompt

2. A selectively activated LoRA variant that applies only to these lookahead tokens so normal token processing remains unchanged.

Training is supervised by “ground-truth” importance scores computed from the model’s own offline-generated responses, and the added parameter count stays below 0.5% across several Llama and Qwen model sizes.

In evaluation on LongBench, RULER, LongProc (HTML→TSV), and MT-Bench, LookaheadKV is consistently competitive with or better than both heuristic baselines (e.g., SnapKV-style methods) and draft-based future-approximation baselines, while keeping time-to-first-token overhead near the cheap heuristics and far below draft-generation approaches across context lengths (including generalization beyond the training window).

FlashSampling: Fast and Memory-Efficient Exact Sampling

Large-vocabulary decoding turns “sample one token” into a systems problem: typical stacks write the LM-head logits to HBM, run additional kernels (temperature/masking, normalization, sampling), then discard the logits. FlashSampling targets that overhead by fusing exact categorical sampling into the LM-head matmul so the full logits tensor is never materialized in HBM. The kernel only writes a small per-tile summary and finishes with a lightweight reduction.

The core observation is that exact sampling can be implemented without forming probabilities: add independent Gumbel noise to each logit and return the index of the largest perturbed value. FlashSampling exploits this by computing the matmul output tile-by-tile on chip, applying deterministic transforms (e.g., temperature, bias, masking), injecting Gumbel noise, and keeping only the best candidate per row within each vocabulary tile. A second-stage reduction over tiles selects the final sample. Exactness is argued in two layers: the fused kernel is pathwise exact because a global maximum over the vocabulary is the maximum of tile-local maxima, while online and tensor-parallel variants remain exact in distribution via a hierarchical factorization that samples a group first using its total log-mass, then samples within the chosen group—avoiding all-gather of full logits across ranks.

In end-to-end vLLM integration, the paper reports up to 19% lower time-per-output-token on the tested models, with larger gains when the LM head accounts for a larger share of decode time.

They released their code here:

• GitHub: FlashSampling/FlashSampling

This week, we review:

• ⭐Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs

• Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

• On-Policy Self-Distillation for Reasoning Compression

⭐Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs

Reasoning-style LLMs are usually sold on the promise of multi-step problem solving, but this paper by Google Research argues their most underappreciated trick is much simpler: they make closed-book factual recall work better even when the question is single-hop and doesn’t “need” a derivation.

Using hybrid models where the same weights can be run with reasoning either enabled or suppressed, the authors show consistent gains on two closed-book QA benchmarks when they probe not just top-1 accuracy but the coverage you get by sampling many answers. The gap between reasoning-on and reasoning-off widens as you sample more, which is a strong signal that reasoning isn’t merely re-ranking already-accessible answers but unlocking answers that are effectively out of reach in the non-reasoning mode.

The paper then does something I wish more mechanistic work did: it treats “reasoning” as an intervention target and tries to separate what’s compute from what’s content.

One mechanism is a computational buffer effect: even replacing the full trace with a meaningless filler repeated to match the original length improves results, but only up to a point and never fully matches true reasoning traces.

The second mechanism is more semantic and more interesting: factual priming. Real traces for simple questions rarely contain genuine step-by-step logic. Instead, they spill related facts, candidate entities, and quasi-retrieval scaffolding. If you extract those factual statements and feed them back as context, while turning reasoning off, you recover most of the benefit, which makes “generative self-retrieval” a pretty accurate mental model of what’s happening.

So, if the intermediate facts are wrong, the final answer is more likely to be wrong too, even after controlling for question difficulty. The authors build a fact-extraction and verification pipeline (including filtering out statements that simply restate the question or explicitly resolve it) and show that you can cash out the analysis into an inference-time strategy: sample multiple traces, prefer ones that actually contain factual statements, and prefer ones whose intermediate facts verify cleanly. In their simulation, this kind of selection moves expected accuracy from 27.9 to 31.3 on SimpleQA-Verified and from 56.9 to 59.8 on EntityQuestions.

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

1.58-bit LLMs have been a research topic for a while. So far, they haven’t really found their target use cases, but Microsoft keeps studying them.

Sparse-BitNet argues that extremely low-bit, ternary LLMs (the “1.58-bit” BitNet style where weights land in {-1, 0, +1}) play unusually well with semi-structured N:M sparsity, better than standard full-precision baselines under the same structured pruning constraints. The core claim is empirical but intuitive: ternary training already drives a large fraction of weights toward exact zeros and creates a magnitude landscape where “keep the top-N per block” hurts less, so the model tolerates higher structured sparsity before quality falls off a cliff.

In this paper, masks are recomputed every step from the continuous, high-precision master weights (not from the ternary weights), and the forward path is “quantize then apply the N:M mask” so the deployed weights satisfy the exact sparse layout. The key stability trick is to not gate gradients with the mask: masked weights still get updated via straight-through estimators, letting connectivity keep evolving instead of freezing early.

On Qwen2.5 models from 0.5B to 3B trained on roughly 50B tokens, their main 6:8 setting shows consistently smaller incremental loss from sparsifying the ternary models than from sparsifying BF16, both in perplexity and in a small battery of zero-shot accuracy benchmarks. When they sweep more aggressive N:8 patterns, the ternary models cross “practical collapse” later than BF16 under comparable structured constraints. On the systems side, they report end-to-end throughput gains using a custom 6:8 sparse kernel, modest at small shapes, topping out around 1.30x in their larger prefill/decode settings, and they show that switching from dense to sparse late leaves a persistent quality gap, so you don’t get the best of both worlds “for free” by pruning at the end.

On-Policy Self-Distillation for Reasoning Compression

Reasoning-tuned LLMs often “think out loud” far past the point of usefulness: they restate, hedge, branch, and re-verify even on easy prompts.

This paper’s thesis is that a lot of that verbosity is not only redundant, it actively increases the chance of going off the rails. Their method, OPSDC, takes the same model, runs it once with a conciseness instruction to get a “teacher” distribution, runs it normally to get a “student,” and trains the student on its own rollouts to match the teacher token-by-token using reverse-KL.

No ground-truth answers, no explicit token budgets, no difficulty estimator, just on-policy self-distillation with a periodically refreshed teacher so compression can keep ratcheting.

On Qwen3-8B/14B, OPSDC cuts reasoning tokens heavily while raising accuracy on MATH-500 (roughly 56–59% shorter, with large absolute accuracy gains; the 14B model jumps from about 70% to about 86%). On AIME 2024 it improves the 14B model by about 10 points with ~41% compression, while on the harder AIME 2025 it still compresses ~35% but gives back ~5 points.

The method is also meaningfully difficulty-adaptive “for free” (more compression on easier sets, less on harder ones) and keeps output entropy stable (contrasting with length-penalized RL that tends to suppress exploratory tokens).

This week, we review:

• ⭐Reinforcement-aware Knowledge Distillation for LLM Reasoning

• Recursive Think-Answer Process for LLMs and VLMs

⭐Reinforcement-aware Knowledge Distillation for LLM Reasoning

Reinforcement-learning post-training has become the go-to way to squeeze better multi-step reasoning out of LLMs, but it leaves you with an awkward deployment problem: the models that learn best are often too expensive to run.

This paper tackles the practical version of that problem, distilling an RL-trained “reasoning” teacher into a smaller student while the student itself is being trained with RL, and argues that most knowledge-distillation recipes were built for supervised fine-tuning, not for an on-policy learning loop where the student’s rollout distribution keeps drifting.

The authors pin the failure mode on two things: teacher traces that stop matching what the student is currently doing, and a teacher-student divergence penalty that fights reward maximization unless you babysit the loss weights.

Their proposal, RL-aware distillation (RLAD): don’t bolt “imitate the teacher” onto RL as a separate objective. Make imitation part of the same trust-region machinery that already keeps policy updates sane.

The main mechanism, Trust Region Ratio Distillation (TRRD), replaces the usual teacher-student divergence regularizer with a clipped likelihood-ratio objective whose anchor is a mixture of the teacher and the student’s previous policy.

In plain terms, the teacher doesn’t get a constant vote but it gets influence only as the current advantage signal supports moving in that direction, and the update is bounded relative to the mixture anchor rather than blindly pulled toward teacher behavior everywhere. The paper also emphasizes that the teacher is used to score student rollouts (log-probabilities), not to generate additional trajectories, which matters for cost.

On long-context math, RLAD consistently beats plain GRPO and a KL-style distillation baseline: for a Qwen3-8B-Base student with a 32B teacher at 30K context, average score rises from 61.0 to 66.5, with notable jumps on the harder AIME benchmarks and steadier validation dynamics. Batch latency increases by roughly 12% versus GRPO in their setup, and RLAD and the KL-style baseline cost about the same when both must consult a large teacher.

Recursive Think-Answer Process for LLMs and VLMs

Think–Answer “reasoner” models have a familiar failure mode: they visibly doubt themselves mid-generation, yet still commit to a single-shot output that’s wrong.

This paper’s stance is that sampling-and-reranking is an external patch, not a fix, because the model never learns a policy for “am I confident enough to stop?” The proposed Recursive Think–Answer Process (R-TAP) turns that question into something you can optimize directly: train the model to run multiple internal reasoning cycles when needed, and to terminate early when the trajectory looks reliable.

R-TAP’s trick is to introduce a separate confidence generator during training. It takes the question plus the model’s current Think–Answer pair and outputs a continuous confidence score.

It’s first trained as a correctness classifier using multiple sampled solutions per question. Then the base LLM/VLM is reinforcement-tuned on short recursive trajectories (multiple Think–Answer pairs, each conditioned on the previous ones) with two confidence-shaped incentives:

• reward trajectories where confidence increases from one cycle to the next

• reward stopping on a final cycle whose confidence clears a fixed threshold

Standard “format/correctness/length” rewards are kept alongside this, but confidence is the new steering wheel: it pushes the policy toward self-correction behavior without bolting on a test-time verifier.

Empirically, the method is presented as broadly “drop-in” across both language-only and vision-language backbones. They show consistent accuracy gains on a mix of math, knowledge, and coding benchmarks for LLMs, and on multimodal reasoning suites for VLMs.

Ablations argue that you need both the “confidence increases” signal and the “confident final answer” signal to get the full lift. I also like that the paper tries to measure behavioral change, not just benchmark deltas: it tracks the drop in “Oops”-style self-correction tokens and reports that this correlates with reduced inference time and diminished reliance on majority-vote self-consistency, suggesting the policy is becoming more stable per sample, not merely luckier with more draws.

Zhipu AI (Z.ai) and researchers at Tsinghua University have published the technical report detailing GLM-5, a large language model (LLM) they position as a bridge from “vibe coding”, i.e., short, prompt-driven programming, to longer-horizon “agentic engineering,” where the model is expected to plan, call tools, navigate repositories and iteratively fix its own work.

→ Technical report: GLM-5: from Vibe Coding to Agentic Engineering

Under the hood, the technical report argues that the leap is less about a single benchmark trick and more about systems engineering: scaling a mixture-of-experts (MoE) backbone, swapping dense attention for a content-aware sparse mechanism at long context, and then running a staged alignment pipeline that includes asynchronous reinforcement learning (RL) designed for long rollouts.

The result is a model trained on a 28.5-trillion-token budget, with a maximum training context extended to 200K tokens, and a post-training pipeline that tries to keep “reasoning sharpness” while teaching the model to behave like an agent.

In this article, I’ll take a deep dive into the report’s key technical details, with an emphasis on the model architecture and training pipeline. Z.ai shares an unusually rich set of implementation notes that should be genuinely useful for future model development. In terms of information density and novelty, I’d place this report somewhere between Qwen’s typical technical write-ups, which often emphasize evaluation while offering fewer training specifics, and DeepSeek AI’s reports, which are more method-heavy and frequently introduce ideas that others later adopt.

Read more

This week, we review:

• ⭐The Art of Efficient Reasoning: Data, Reward, and Optimization

• Does Your Reasoning Model Implicitly Know When to Stop Thinking?

⭐The Art of Efficient Reasoning: Data, Reward, and Optimization

This work dissects what actually happens when you use RL to make chain-of-thought reasoning shorter without trashing accuracy.

The authors focus on a simple but widely used recipe: GRPO on a DeepSeek-R1–distilled Qwen backbone, trained on DeepScaleR math prompts with a truncation-style reward that only pays out for correct answers under a target length.

They argue that efficient reasoning training reliably follows two phases: first, a “length adaptation” phase where the model rapidly compresses its rollouts to meet the token budget, and second, a slower “reasoning refinement” phase where accuracy is recovered within that shorter-length regime. To make this visible, they track length distributions conditioned on correctness and sweep performance across inference budgets from 2k to 32k tokens on math and code benchmarks like AIME’25, MATH-500, and LiveCodeBench.

The most consequential finding is about data difficulty. If you train under a length-aware reward only on very hard math problems, the policy can collapse: entropy spikes, rollouts shrink aggressively, and downstream metrics drop, because the signal is dominated by length penalties on mostly incorrect trajectories. In contrast, training on an “easy” subset yields stable entropy, smooth convergence to the target length, and essentially no loss, sometimes even a modest gain, on tough benchmarks such as AIME25. In other words, dense positive reward is more important than matching the difficulty of your target eval set, and the induced “short-answer” bias even transfers reasonably from math to code.

They then poke at the rest of the recipe: rollouts, reward shaping on negatives, and off-policy tricks. Increasing the number of rollouts per prompt noticeably accelerates length adaptation and improves asymptotic math performance, although gains on LiveCodeBench are minor and compute cost climbs quickly. Masking or down-weighting different subsets of negative rollouts leads to distinct failure modes, including ultra-short but low-quality outputs, so you cannot just zero out half the trajectories and hope for the best. Off-policy training with moderately stale rollouts can match or slightly beat on-policy early in training, but high staleness produces entropy blow-ups and length rebound, so the recommendation is essentially “on-policy, many rollouts, easy prompts.”

Applying these guidelines to multiple Qwen3 models from 0.6B to 30B roughly halves average response length on AIME25 while preserving, or for smaller models, clearly improving, Mean@8 and Pass@8.

Does Your Reasoning Model Implicitly Know When to Stop Thinking?

The paper starts from the now-familiar observation that RLVR-style training encourages very long chains of thought that are only loosely correlated with correctness. On math benchmarks, models often arrive at the right answer early and then keep talking: the Ratio of First Correct Step metric shows that, in a majority of correct solutions, the answer appears well before the final step, and additional post-training does little to fix this under standard pass@1 sampling.

To probe whether the model “knows” when it should have stopped, the authors introduce TSearch, a beam-style exploration over partial reasoning traces scored by average prefix log-likelihood, with a special focus on the probability assigned to a terminator token. As the exploration width increases, TSearch with this prefix score finds shorter reasoning paths and higher accuracy, whereas a variant that only looks at next-token scores collapses to short but bad outputs, evidence that concise, high-confidence solutions exist in the model but are invisible to greedy decoding.

For deployment, TSearch is turned into SAGE (Self-Aware Guided Efficient Reasoning), which operates at the level of reasoning “steps” (segments of CoT) instead of individual tokens. At each step it samples multiple candidate steps from the model, extends the current beams, scores the resulting prefixes by average log-likelihood, and accepts a completion as soon as a sampled step ends with, effectively doing step-wise beam search guided by the model’s own stopping confidence.

A degraded version that removes this structured exploration underperforms SAGE in both pass@1 and length, showing that the benefit is not just early truncation. Across several math benchmarks and three LRMs, SAGE improves pass@1 while roughly halving token usage compared to standard random sampling, with stronger models on harder datasets seeing larger accuracy gains and weaker models on easier datasets seeing larger compression.

The same idea is then folded into training: SAGE-RL simply replaces two of the eight rollouts per question in GRPO/GSPO-style RLVR with SAGE samples, which consistently yields higher pass@1 (about two points on average) and substantially better token efficiency than both plain RLVR and prior efficiency-oriented baselines.

In my view, the most interesting claim is not that yet another sampling heuristic works, but that the model’s log-probability landscape already encodes a fairly sharp notion of “I’m done” around the token, and that beam-style exploration plus a simple prefix score is enough to expose it. That said, the evidence is firmly grounded in math problems with verifiable rewards, explicit step boundaries, and a designated terminator; how much of this carries over to fuzzier domains like tool-use planning or open-ended writing is an open question.

This week, we review:

• ⭐ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces

Only one paper this week. I’m working on a full review of the GLM-5 technical report that I plan to publish next week.

⭐ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces

The paper looks at why latent “soft thinking” sometimes helps reasoning models and sometimes quietly hurts them.

The authors start from an empirical observation: under latent-only reasoning, trajectories that end in wrong answers actually exhibit fewer low-confidence steps than those that end in correct ones. They argue that when the model’s token distribution is flat, the soft embedding is an average over multiple incompatible continuations, injecting noise into the hidden state. As this noise accumulates, the model can become confidently wrong even though it rarely “admits” uncertainty along the way.

To address this, ThinkRouter, proposed by the authors, is a purely inference-time mechanism that decides, at each thinking step, whether to reason in discrete token space or in latent space based on the maximum next-token probability.

When the model is confident enough (above a routing threshold), it uses a Soft Thinking–style latent step: take the top-j tokens under the renormalized distribution and form a probability-weighted embedding that is appended to the context as a “soft token.” When confidence is low (below the threshold), the method instead samples a single discrete token and appends its embedding, avoiding the mixture of many weak options. A simple “Cold Stop” heuristic monitors the distribution to emit an end-of-thinking token, after which the final answer is generated in the usual discrete space. The only task-specific hyperparameter is the routing threshold, selected via a small grid search over a handful of validation examples per benchmark.

Across several STEM math and coding benchmarks (AIME 2024/2025, GPQA Diamond, HumanEval, MBPP) and models ranging from 1.7B to 32B parameters (Qwen3 variants and a gpt-oss model), ThinkRouter consistently outperforms standard chain-of-thought decoding, Soft Thinking, and a random routing baseline, with average Pass@1 gains of up to ~20 points and generation-length reductions of up to ~15%.

This week, we review:

• ⭐On the Optimal Reasoning Length for RL-Trained Language Models

• Context Compression via Explicit Information Transmission

• Large Language Model Reasoning Failures

⭐On the Optimal Reasoning Length for RL-Trained Language Models

How long should the model think?

The authors study RL fine-tuning of two math-reasoning models, a bare Qwen3-1.7B base model and a distill of a strong reasoning model, DeepSeek-R1-Distill-Qwen-1.5B, under several length-control objectives (GRPO/DAPO-style baselines plus RLOO-LP, ALP, and DRPO). They evaluate on AIME 2024/2025, AMC, and MATH-500, looking at both accuracy and average output tokens. The main result is that there is no universal “more is better” rule: the base model benefits monotonically from longer chains of thought, while the already-strong distilled model has a sweet spot at intermediate lengths, with both shorter and longer outputs hurting performance.

On the optimization side, for Qwen3-1.7B-Base, making the RL objective care about brevity consistently harms reasoning performance. The model appears to need long, exploratory rollouts during RL just to acquire coherent multi-step patterns. In contrast, for DeepSeek-R1-Distill-Qwen-1.5B, moderate penalties improve the efficiency–performance tradeoff: they trim output length while maintaining or slightly improving accuracy, whereas very strong penalties again degrade results.

All experiments share a DAPO-style RL setup, with fairly heavy compute (8 GPUs for 72 hours) and long context limits (up to 16k response tokens), so the observed length effects are not artifacts of trivial truncation.

To unpack why the distilled model has an optimal intermediate length, the authors extend earlier “test-time scaling” analyses to RL-trained policies.

Instead of just tracking entropy, they decompose behavior into mode accuracy (is the most common answer correct?), entropy over answers, and mode share (how much mass sits on the most common answer). In the long-output regime, mode accuracy stays flat or even improves, but entropy rises and mode share falls: the model’s distribution moves closer to the right answer but spreads probability across too many options, and accuracy drops. In the short-output regime, both mode accuracy and mode share are low and entropy high, suggesting the model is both thinking too little and scattering its guesses.

So, in short, long outputs fail by dispersion, short outputs fail by under-thinking.

Context Compression via Explicit Information Transmission

The paper tackles long-context inference for LLMs through “soft” context compression, where a long input is mapped to a small set of continuous vectors that the decoder then conditions on.

Existing soft methods treat the LLM itself as the compressor: they add special compression tokens and let self-attention iteratively pull information into them. The authors argue this has two structural problems: information in early layers gets overwritten as it flows up the stack, and different compression tokens compete independently, leading to redundant coverage of some spans and neglect of others. They frame these as distribution mismatch across layers and a lack of globally coordinated allocation of compression capacity.

Their alternative, ComprExIT, freezes the base LLM and reinterprets compression as explicit information transmission over its hidden states. First, a depth-wise step builds “token anchors” at each position by gating and mixing layer representations, emphasizing early and mid layers while down-weighting late, generation-specialized ones. Second, a width-wise step aggregates anchors into a small number of “slots” via an entropy-regularized optimal transport plan that balances how much each token contributes to each slot under locality constraints. A small MLP then maps the resulting slots into the decoder’s input space. The compression module adds about 1% parameters to Llama-3.2 1B/3B backbones and is trained in two phases: generic next-token prediction on SlimPajama, followed by supervised QA fine-tuning, with a fixed compression ratio.

On six in-domain QA benchmarks (SQuAD, NewsQA, TriviaQA, SearchQA, HotpotQA, NQ), ComprExIT consistently beats strong soft compression baselines like ICAE, 500x, and Activation Beacon, improving average F1 by around 7–10 points depending on model size, while approaching or even exceeding prompt-tuned models that see the full context. The gains become larger in out-of-domain QA and in the harder setting where the compression module only sees language-modeling supervision and no task labels, where competing methods often collapse toward near-random performance on the 1B model. Ablations show that both depth-wise layer aggregation and globally optimized allocation matter: removing either causes sizable drops and leads to more redundant, low-rank allocation patterns across compression tokens.

Large Language Model Reasoning Failures

The survey proposes a two-axis taxonomy: along one axis, reasoning is split into informal (cognitive and social), formal (logic, math, code), and embodied (physical and spatial reasoning from text-only setups through 2D vision to 3D, interactive environments). Along the other axis, failures are grouped as fundamental (rooted in architecture or training), application-specific (e.g., theory of mind or robotics), and robustness failures (instabilities under small perturbations). The authors then populate this grid with a large collection of empirical results, matching each failure type with hypothesized causes and mitigation strategies.

Across informal reasoning, the survey highlights deficits that look uncomfortably like missing executive functions: limited working memory, weak inhibitory control, and poor cognitive flexibility, together with fragile abstract reasoning. LLMs also mirror and amplify human cognitive biases, confirmation, framing, anchoring, order effects, and social/affective biases, and their behavior in social reasoning tasks (theory of mind, emotional understanding, moral and norm reasoning) is brittle under rephrasings, language changes, or more realistic multi-turn interactions.

In formal reasoning, the paper emphasizes compositional failures, instability on perturbed benchmarks, and surprisingly persistent problems in counting and basic arithmetic, tracing them not just to prompting but to tokenization, positional encoding, and the next-token objective itself.

The taxonomy makes clear that many celebrated gains in “reasoning” sit on top of unresolved, cross-cutting fundamental failures, like working-memory limits, bias inherited from data and architecture, and the absence of robust internal world models.