Rich Feedback Outperforms Scalar Rewards on Open-Ended Tasks
The Weekly Salt #124
This week, we review:
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
Distilled Reinforcement Learning for LLM Post-training
DeepLoop: Depth Scaling for Looped Transformers
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
This paper replaces scalar reward optimization for open-ended tasks with Experiential Learning, where the evaluator acts as a coach rather than a judge. Given a prompt, a policy response, and task-specific rubrics, the coach produces reusable textual guidance about strengths, weaknesses, and response strategies. That guidance conditions a teacher model, whose token distributions are distilled into the policy using on-policy samples. The feedback model and the additional context are not required at inference time.
The method is tested with Qwen3-8B and OLMo-3-7B policies, using either the initial policy or GPT-4o as the coach.
Training uses 7,500 WildChat-IF prompts, while evaluation covers held-out WildChat prompts and four unseen open-ended benchmarks. Experiential Learning generally outperforms rubric-based RL, especially on AlpacaEval and WildBench, although it does not win every model–benchmark combination.
Rubric-based reinforcement learning trains a model using rewards derived from an explicit evaluation rubric. The rubric lists criteria such as correctness, completeness, relevance, style, or safety. An evaluator scores each response against those criteria, and the model is updated to produce responses that receive higher scores.
In LLM post-training, the evaluator is often another language model. Unlike textual coaching, rubric-based RL usually compresses the evaluation into one or several numeric rewards, so the policy learns which outputs score well but receives less detailed information about how to improve them.
An additional analysis finds that RL obtains larger training-set gains but transfers less effectively, consistent with optimization against idiosyncrasies of the scalar evaluator.
The central contribution is a practical way to preserve information that a scalar score discards, rather than a demonstration that textual feedback has a measurable information advantage of the stated magnitude. The bandwidth comparison is explicitly heuristic, and most results depend on model-based rubrics and evaluation. Iteratively updating the teacher also improves the target-domain score while substantially degrading out-of-distribution instruction following; mixing general-domain distillation data only partially recovers that loss.
Distilled Reinforcement Learning for LLM Post-training
Distilled RL incorporates a teacher’s token-level preferences directly into policy-gradient training instead of adding a separate distillation loss. For student-generated trajectories with positive advantage, teacher-to-student likelihood ratios redistribute credit among tokens. Teacher weighting is disabled for negative trajectories, where it would otherwise push the student away from teacher-preferred tokens, and sequence-level normalization prevents the teacher ratios from globally weakening or amplifying the update. The teacher therefore guides successful trajectories without becoming an unconditional imitation target.
Experiments train three student models on DAPO-17K and compare against GRPO, on-policy distillation, and a direct combination of the two.
The largest result is in cross-family distillation: the DeepSeek-R1-Distill-Qwen-1.5B student reaches a 40.00 average across ten mathematical benchmarks, versus 36.86 for RL and 35.27 for conventional distillation. Gains are smaller but consistent for Qwen3-4B and Qwen3-1.7B. Pass@16 and knowledge-intensive evaluations are mostly competitive, and ablations identify the negative-sample reset as the most important component, with its removal reducing average pass@1 by 6.39 to 8.81 points.
The method still assumes that the fixed teacher provides useful token preferences whenever the student has produced a successful answer. Because the teacher is only evaluated along student trajectories, training does not establish whether it can solve each problem itself. A student may also eventually outperform the teacher on particular examples. Teacher-competence gating or occasional teacher rollouts would be needed to make the supervision reliable in that regime.
DeepLoop: Depth Scaling for Looped Transformers
DeepLoop addresses optimization in Transformers that repeatedly apply the same physical blocks to obtain greater effective depth. Standard depth-scaling rules treat residual branches at different layers as independently parameterized, but looped models accumulate updates from repeated visits to shared parameters.
The paper models this effect through the alignment of visit-wise updates and derives a more conservative square-root residual and initialization scaling rule for the case where repeated visits remain aligned. The architecture otherwise remains a Post-LN looped Transformer.
At GPT-2 small and medium scales, DeepLoop is effectively neutral when each block is used once and reduces validation loss when blocks are revisited three, five, or seven times. On the medium model with seven loops, average downstream accuracy rises from 52.95% to 53.88% in zero-shot evaluation and from 54.62% to 55.20% in one-shot evaluation. Applying the same scaling to a hierarchical recurrent reasoner improves two-vote ARC-AGI-1 accuracy from 36.50% to 39.75%, with gains across all tested voting budgets.
The language-model comparisons use single runs at 124M and 350M parameters, so the relatively small loss and downstream differences are not yet statistically characterized. The ARC-AGI experiment includes a stronger multi-seed control, but it tests one recurrent architecture and dataset. The analysis also relies on an alignment coefficient that is not directly measured. Determining its behavior at larger scale, under other normalization schemes, and during longer training is necessary before treating the proposed exponent as a general rule for recurrent-depth models.