Restoring KV Caches, Reducing Overthinking, and Diagnosing ALiBi
The Weekly Salt #125
This week, we review:
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
Query-agnostic KV-cache eviction compresses a context once so that the resulting cache can serve multiple future queries. Under small memory budgets, retaining only a selected subset of the original key-value pairs can remove information needed later.
RestoreKV supplements the retained cache with a small set of learned restore tokens. During context prefill, these tokens attend to the full cache in one LoRA-adapted pass and produce a compact, context-specific restore cache. The adapters are then disabled for querying and decoding.
Training uses self-distillation from the frozen full-cache model and updates 0.4% of its parameters without task-specific tuning.
The method is evaluated across four model backbones, four long-context benchmarks, and five existing eviction methods.
On Qwen3-4B, it improves 59 of 60 budget-matched comparisons. With a 5% KV budget, adding RestoreKV raises KVzip’s RULER-4K score from 38.2 to 73.2. Combined with KVzip+, it reaches 86.4 RULER accuracy at 16× compression, with less than 0.5% additional one-time cache-construction overhead in the reported 32K-context evaluation.
Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
Long reasoning traces often contain hedging, abandoned approaches, and self-contradiction that consume tokens without improving the final answer.
The paper argues that this behavior cannot be explained by length alone: after controlling for response length, incorrect traces still contain more unproductive self-reflection.
Its proposed proxy for identifying useful reflection is the sequence of intermediate answer commitments. By checking whether each candidate answer is correct, training can estimate whether the reasoning that follows moves toward or away from the target without requiring manually annotated reasoning steps.
DASH, or Drift Aware advantage SHaping, uses this signal to assign credit at the segment level rather than treating the entire trace as a single unit.
On competition-level mathematics benchmarks where overthinking is common, DASH reports an average accuracy of 59.45%, compared with 58.1% for Dr.GRPO and 56.95% for GRPO. The trained models also show fewer unproductive reflection patterns and more useful self-correction.
The method depends on tasks with known ground-truth answers and identifiable answer commitments inside the trace.
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
ALiBi adds position-dependent linear biases to attention scores. At sufficiently large distances, these biases can push attention weights below the representable range of the floating-point format, causing them to become zero.
The affected attention heads then lose access to parts of the context. The paper distinguishes this numerical failure from the more general decline that occurs when a model is evaluated beyond its training context and confirms that the issue also appears in pretrained models using ALiBi.
Experiments with 148-million-parameter decoder models show that the failure can significantly damage token-retrieval performance while producing only small changes on standard decoder benchmarks.
The authors evaluate four training-time mitigations, both separately and in combination. Replacing linear distance scaling with log-scaled distances gives the most consistent improvement on passkey retrieval. At the same time, the default ALiBi slopes remain competitive, particularly on needle-in-a-haystack tests.
The mismatch between retrieval failures and conventional benchmark results is the main practical concern in my opinion: aggregate language-model evaluations may not reveal that some heads have lost numerical access to distant tokens.