Literature
Display
← All papers

RAGEN-2: Reasoning Collapse in Agentic RL

Zihan Wang · Chi Gui · Xing Jin · Qineng Wang · Licheng Liu · Kangrui Wang · Shiqi Chen · Linjie Li · Zhengyuan Yang · Pingyue Zhang · Yiping Lu · Jiajun Wu · Li Fei-Fei · Lijuan Wang · Yejin Choi · Manling Li

Preprint posted · arXiv (preprint) ↗

AI agents15 min text · CondensedOriginal paper ↗
Contents

Overview

RAGEN-2 examines a failure of agent reinforcement learning: varied language can become insensitive to the particular problem. It pairs a cross-input diagnostic with reward-variance filtering and retains the full mathematical appendix.

At a glance

Reasoning can remain fluent and varied while converging to reusable templates across different inputs. RAGEN-2 diagnoses this with prompt–reasoning compatibility scores and proposes filtering updates toward prompts with informative reward differences. Results support the intervention across several agent tasks, algorithms and model sizes, with explicit exceptions when reward variance mainly reflects noise. The measurements are operational proxies for input dependence, rather than direct measurements of reasoning correctness or causal faithfulness.

1 Introduction

Ordinary reward and entropy monitoring can miss a policy that produces different-looking versions of the same generic rationale. Building on RAGEN, this work separates diversity within an input from differences across inputs. Its contribution is a diagnosis of template collapse, a gradient signal-to-noise interpretation, and an inexpensive update filter based on within-prompt reward variance. The empirical argument combines training curves, cross-run correlations and filtering interventions; the appendices state the assumptions behind the accompanying bounds.

Refer to caption
Figure 1: Left: input-driven reasoning adapts to the current state; templated reasoning produces nearly identical responses across different inputs. Right: four reasoning regimes characterized along two axes: conditional entropy H⁡(Z∣X)H(Z\mid X) (within-input diversity) and mutual information I⁡(X,Z)I(X;Z) (input dependence). Details in Section 2.

2 Template Collapse in Multi-turn Agent RL

The proposed failure mode concerns the relationship between an input and its generated reasoning. A model may preserve token variability without preserving that relationship.

2.1 Setup and Preliminaries

At each interaction turn the agent sees an observation, produces reasoning and an executable action, and receives reward. The diagnostic input includes the complete available history; the measured reasoning excludes action tokens and boundary markers. Policy-gradient objectives combine reward-weighted updates with KL and entropy regularizers. These regularizers do not weight trajectories by reward differences, although their gradients can still depend on the prompt.

Equation
ℒ(θ)=𝔼x,τ[A(τ,x)]−λKLDKL(πθ∥πref)+λHH(πθ),\mathcal{L}(\theta)=\mathbb{E}_{x,\tau}\big[A(\tau,x)\big]-\lambda_{\mathrm{KL}}D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})+\lambda_{H}H(\pi_{\theta}),

2.2 Rethinking Reasoning Collapse from an Information-Theoretic Lens

The entropy decomposition distinguishes within-input randomness from cross-input information. High values on both axes describe diverse, input-sensitive reasoning; high randomness with low input dependence describes template collapse. Low randomness can instead be either input-sensitive compression or complete collapse. Thus shorter or more deterministic reasoning alone does not establish that a model has stopped responding to its input.

Equation
H⁡(Z)=I⁡(X,Z)+H⁡(Z∣X),H(Z)=I(X;Z)+H(Z\mid X),

2.3 Mutual Information Proxy Family

For each sampled reasoning trace, teacher-forced scoring compares its likelihood under its own prompt and every other prompt in the batch. Retrieval accuracy asks whether the original prompt ranks first; chance is 1/P. A continuous proxy compares matched likelihood with a uniform prompt-mixture likelihood, with per-token normalization and optional z-score/EMA smoothing. First-turn and trajectory-sampled variants are retained in Table 1. The reported strongest correlation with task performance is +0.39, while entropy correlations are −0.11 to −0.14. These scores depend on the scorer and candidate prompts; they are not exact Shannon MI. Reusing rollouts avoids additional sampling, but constructing the cross-score matrix still requires likelihood evaluations.

Table 1: MI proxy family. All variants are derived from in-batch cross-scoring of reasoning traces against prompts, using matched (per-token log-prob under the true prompt) and marginal (per-token log-prob under the uniform prompt mixture) as base quantities. First-turn variants use only the first agent turn; trajectory variants sample across all turns.
Type Proxy Formula Notes
Discrete Retrieval-Acc 1P​G∑i,k𝟏[argmaxj𝐋i,k,j=i]\frac{1}{PG}\sum_{i,k}\mathbf{1}[\arg\max_{j}\mathbf{L}_{i,k,j}=i] Chance level 1/P1/P under template collapse
bi lilin k Recall@kk 1P​G∑i,k𝟏[i∈top-kj(𝐋i,k,j)]\frac{1}{PG}\sum_{i,k}\mathbf{1}[i\in\mathrm{top}\text{-}k_{j}(\mathbf{L}_{i,k,j})] k∈{2,4,8}k\in\{2,4,8\}
Continuous (raw) MI-Est 1P​G​∑i,k(matchedi,k−marginali,k)\frac{1}{PG}\sum_{i,k}(\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}) Per-token; approaches 0 under collapse
MI-Seq-Est 1P​G​∑i,k(𝐋i,k,i−log⁡1P​∑je𝐋i,k,j)\frac{1}{PG}\sum_{i,k}\bigl(\mathbf{L}_{i,k,i}-\log\tfrac{1}{P}\sum_{j}e^{\mathbf{L}_{i,k,j}}\bigr) Per-sequence; no length normalization
Continuous (z-score) MI-ZScore 1P​G​∑i,kmatchedi,k−marginali,kσbatch+ϵ\frac{1}{PG}\sum_{i,k}\frac{\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}}{\sigma_{\mathrm{batch}}+\epsilon} Normalized by current-batch marginal std
MI-ZScore-EMA 1P​G​∑i,kmatchedi,k−marginali,kσEMA+ϵ\frac{1}{PG}\sum_{i,k}\frac{\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}}{\sigma_{\mathrm{EMA}}+\epsilon} σEMA(t)=α​σEMA(t−1)+(1−α)​σbatch(t)\sigma_{\mathrm{EMA}}^{(t)}=\alpha\,\sigma_{\mathrm{EMA}}^{(t-1)}+(1{-}\alpha)\,\sigma_{\mathrm{batch}}^{(t)}
Equations 1
matchedi,k=𝐋i,k,i|Zi,k|,marginali,k=1|Zi,k|​log⁡1P​∑jexp⁡(𝐋i,k,j),\mathrm{matched}_{i,k}=\frac{\mathbf{L}_{i,k,i}}{|Z_{i,k}|},\qquad\mathrm{marginal}_{i,k}=\frac{1}{|Z_{i,k}|}\log\frac{1}{P}\sum_{j}\exp(\mathbf{L}_{i,k,j}), (1)
Acc=1P​G∑i=1P∑k=1G𝕀[i=argmaxj𝐋i,k,j].\mathrm{Acc}=\frac{1}{PG}\sum_{i=1}^{P}\sum_{k=1}^{G}\mathbb{I}\Big[i=\arg\max_{j}\,\mathbf{L}_{i,k,j}\Big].
I^​(X,Z)=1P​G​∑i=1P∑k=1G(matchedi,k−marginali,k),\widehat{I}(X;Z)=\frac{1}{PG}\sum_{i=1}^{P}\sum_{k=1}^{G}\Big(\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}\Big),
Original PDF visual retaining vector text, complete panel arrangement and native embedded image detail.
Figure 2: Schematic Signal-to-Noise Ratio (SNR) view of RL updates. Left: total gradient decomposes into task gradient (sharpens with higher within-input reward variance) and regularization gradient. Right: high reward variance yields strong task gradient and better convergence (high SNR); low reward variance makes regularization gradient dominate, producing erratic updates and input-agnostic reasoning (low SNR).

3 The Mechanism of Template Collapse: A Signal-to-Noise Ratio (SNR) View

The proposed mechanism is an imbalance between reward-discriminative gradients and other update components. When rewards barely distinguish trajectories for a prompt, regularization and sampling noise can have greater relative influence. This motivates selecting update data by reward variation rather than regulating output entropy alone.

3.1 Observing Signal-Noise Imbalance in RL Gradients

Figure 3 groups prompts into six reward-variance buckets for PPO and GRPO. Task-gradient norms increase with variance, while regularizer norms vary much less. The authors interpret this as weak task signal in low-variance groups. The caption also notes substantial low-variance task gradients, so the observation should not be reduced to a claim that every sampled gradient vanishes or that norm alone measures useful signal.

Original figure
Figure 3: Prompts sorted into six equal-sized reward-variance buckets Q1–Q6. We find: (a) Task gradient norm increases monotonically with bucket RV; (b) When RV near 0, substantial task gradients persist despite carrying almost no useful signal; (c) Regularizer gradient norm (KL + entropy) is flat across buckets. This directly supports the SNR mechanism under both algorithms.

3.2 Formalizing the SNR Mechanism via Gradient Decomposition

With a conditional-mean baseline, Cauchy–Schwarz bounds the expected task-gradient norm by the square root of reward variance times a score-function factor. The update is then conceptually decomposed into task signal, task noise and regularization. This supports a low-variance weakness argument under the stated assumptions; it does not establish that high variance guarantees useful signal or that regularizers universally force MI to zero. Appendices H and K make the magnitude and dominance claims more precise.

Equations
gtask​(x)=1G​∑gAg​∇θ​log⁡πθ​(τg∣x).g_{\mathrm{task}}(x)=\frac{1}{G}\sum_{g}A_{g}\,\nabla_{\theta}\log\pi_{\theta}(\tau_{g}\mid x).
|gtask​(x)|≤Var^​(R∣X=x)⋅C.|g_{\mathrm{task}}(x)|\leq\sqrt{\widehat{\mathrm{Var}}(R\mid X=x)}\cdot C.
Table 2: Three-noise decomposition of the policy update gradient.
Component Source Level Ctrl. Mitigation
gsignalg_{\text{signal}} Meaningful reward differences across same-prompt trajectories Prompt No SNR-Aware Filtering
gtask-noiseg_{\text{task-noise}} Sampling and environment stochasticity Prompt No Filter high-noise prompts
gregg_{\text{reg}} Uniform contraction per chain, independent of input (KL, entropy) Chain Yes Tune λKL\lambda_{\text{KL}}, λent\lambda_{\text{ent}}
Refer to caption
Figure 4: SNR-Aware Filtering workflow. At each training iteration: (1) rollout generation collects trajectories; (2) within-prompt reward variance is computed as SNR proxy; (3) prompts are ranked by RV and top-p fraction retained, policy update performed only on high-signal subset. This filtering loop can prevent updating on noisy rollouts and requires no additional models/rollouts beyond standard RL.

3.3 SNR-Aware Filtering: Prioritizing High-Signal Updates

Sample several trajectories per prompt, estimate their return variance, and sort prompt groups from highest to lowest variance. Top-p filtering retains the smallest prefix containing a chosen fraction of total variance mass, then updates only on those groups. A setting of 0.9 therefore targets 90% of variance mass, not necessarily 90% of prompts. Selection happens after rollout collection, so it changes gradient contributions without changing that iteration’s sampling distribution. The score is cheap but conflates task-dependent variation with environmental or evaluation noise.

Equations
Var^​(R∣X)\displaystyle\widehat{\mathrm{Var}}(R\mid X) =1G−1​∑g=1G(Rg​(X)−R¯​(X))2,\displaystyle=\frac{1}{G-1}\sum_{g=1}^{G}\big(R_{g}(X)-\overline{R}(X)\big)^{2},
R¯​(X)\displaystyle\overline{R}(X) =1G​∑g=1GRg​(X).\displaystyle=\frac{1}{G}\sum_{g=1}^{G}R_{g}(X).
Var^​(R∣X=xσ⁡(1))≥Var^​(R∣X=xσ⁡(2))≥⋯≥Var^​(R∣X=xσ⁡(P)),\widehat{\mathrm{Var}}(R\mid X=x_{\sigma(1)})\geq\widehat{\mathrm{Var}}(R\mid X=x_{\sigma(2)})\geq\cdots\geq\widehat{\mathrm{Var}}(R\mid X=x_{\sigma(P)}),
τ=ρ​∑i=1PVar^​(R∣X=xi),\tau=\rho\sum_{i=1}^{P}\widehat{\mathrm{Var}}(R\mid X=x_{i}),
S={σ⁡(1),…,σ⁡(k∗)},where ​k∗=min⁡{k:∑j=1kVar^​(R∣X=xσ⁡(j))≥τ}.S=\left\{\sigma(1),\dots,\sigma(k^{*})\right\},\quad\text{where }k^{*}=\min\left\{k:\sum_{j=1}^{k}\widehat{\mathrm{Var}}(R\mid X=x_{\sigma(j)})\geq\tau\right\}.
Table 3: Summary of the features of the environments used.
Task Stochastic Multi-turn State Reward
Sokoban ✗ ✓ Grid Dense
FrozenLake ✓ ✓ Grid Binary
MetaMathQA ✗ ✓ Text Dense
Countdown ✗ ✗ Text Binary
SearchQA ✗ ✓ Text Dense
WebShop ✗ ✓ Text Dense
DeepCoder ✗ ✗ Text Dense

4 Experiments

Experiments first examine collapse diagnostics, then compare filtering strategies, model/algorithm variants and interventions intended to probe the proposed mechanism. Reported peak validation performance should be distinguished from final-checkpoint performance.

4.1 Experimental Testbed

The testbed combines grid planning, navigation, mathematics, search, shopping and programming. Most controlled algorithm/model comparisons use four core tasks; additional plots cover search, web interaction and code generation.

Environments and tasks.

Sokoban stresses irreversible box pushes; FrozenLake adds sparse rewards and stochastic transitions. MetaMathQA tests mathematical answers, and Countdown constructs arithmetic expressions. SearchQA requires iterative retrieval, WebShop requires selecting products that satisfy instructions, and DeepCoder rewards executable solutions against tests. These tasks differ in observability, horizon and reward structure, so a shared filtering rule need not have the same effect everywhere.

Training and evaluation setup.

The main setup uses Qwen2.5-3B with veRL/HybridFlow and PPO, DAPO, GRPO or Dr. GRPO, for up to 400 iterations. A default iteration collects 128 trajectories as eight prompts with sixteen samples each. Filtering reduces the update batch, and the paper scales loss by the keep-rate parameter to keep step sizes comparable. Appendix B specifies validation, optimization and early stopping.

4.2 Template Collapse as a Consistent Failure Mode

In the illustrated unfiltered runs, retrieval-based input dependence falls before task performance while reasoning entropy remains high or spikes. Filtering better preserves the diagnostic and task success. Additional length plots show compression in several settings, but length itself cannot distinguish efficient input-sensitive reasoning from generic templates; the input-dependence measurement is the central evidence.

Original figure
Figure 5: Training dynamics under different intervention strategies. (a) Task success rate, (b) MI proxy (retrieval accuracy), and (c) reasoning entropy. Without filtering, MI degrades early while entropy spikes, signaling template collapse. Filtering effectively mitigates the decline in retrieval accuracy, with top-p SNR-Aware filtering best preserving both task performance and reasoning diversity.
Refer to caption
Figure 6: Comparison of filtering strategies showing Top-pp consistently outperforming Top-kk and no-filter baselines across four environments.
Table 4: SNR-Aware Filtering results (%) across algorithms, model scales, types, and modalities. Each cell reports baseline peak with filter delta in parentheses; Qwen2.5-VL-3B includes text (T) and image (V) inputs. Filtering improves average score across all variants.
Experiment Variants Sokoban FrozenLake MetaMathQA Countdown Average
Baseline
PPO (Schulman et al., 2017b),
Qwen2.5-3B (Qwen Team, 2024) 12.9 (+16.0) 67.0 (+10.9) 92.6 (+0.6) 97.9 (+0.0) 67.6 (+6.9)
Algorithm
DAPO (Yu et al., 2025) 16.2 (+5.1) 66.8 (+2.1) 90.8 (+2.8) 95.7 (+1.6) 67.4 (+2.9)
GRPO (Shao et al., 2024) 12.1 (+9.0) 70.9 (-3.0) 91.2 (+1.2) 95.7 (+2.2) 67.5 (+3.7)
Dr. GRPO (Liu et al., 2025) 12.1 (-0.4) 23.2 (+0.6) 91.2 (+1.4) 96.5 (+1.4) 55.8 (+0.8)
Model Scale (PPO)
Qwen2.5-0.5B (Qwen Team, 2024) 3.3 (+22.9) 19.5 (+0.0) 10.0 (-0.2) 23.0 (-0.7) 14.0 (+5.5)
Qwen2.5-1.5B (Qwen Team, 2024) 17.0 (+6.2) 36.5 (+1.6) 80.3 (+7.0) 56.6 (+1.6) 47.6 (+4.1)
Qwen2.5-7B (Qwen Team, 2024) 42.4 (+4.9) 85.0 (-0.6) 84.0 (+11.7) 97.7 (+0.3) 77.3 (+4.1)
Model Type
Qwen2.5-3B-Instruct (Qwen Team, 2024) 22.5 (+14.2) 83.6 (+2.3) 91.2 (+0.4) 96.3 (-0.6) 73.4 (+4.1)
Llama3.2-3B (Meta Llama, 2024) 24.4 (+18.8) 84.6 (-0.2) 86.1 (+3.7) 99.2 (-1.2) 73.6 (+5.3)
Modality (Input Type)
Qwen2.5-VL-3B (T) (Bai et al., 2025) 53.0 (+6.0) 16.0 (+53.5) - - 34.5 (+29.8)
Qwen2.5-VL-3B (V) (Bai et al., 2025) 65.0 (+12.0) 19.5 (+59.5) - - 42.3 (+35.8)
Refer to caption
Figure 7: Reasoning length decline across eight environments, showing systematic compression as a behavioral signature of template collapse.

4.3 SNR-Aware Filtering Consistently Improves Performance

Table 4 reports higher average peak performance for every listed model/algorithm configuration, with individual task exceptions. Qwen2.5-3B PPO rises from 67.6% average by 6.9 percentage points; Sokoban rises from 12.9% by 16.0 points. Some cells decline, including Dr. GRPO on Sokoban and Qwen2.5-7B on FrozenLake. Figure 6’s actual panels cover Sokoban, SearchQA, WebShop and DeepCoder. At a fixed 128-rollout budget, Table 5 reports 26–41% shorter filtered update steps and under 0.1% overhead for variance calculation. This measures filtering cost, not the full cost of every MI cross-scoring diagnostic.

Table 5: Sweep over prompt batch size PP and group size GG (trajectories per prompt) on Sokoban (Qwen2.5-3B). NF = no filtering; F = SNR-Aware Filtering (ρ=0.9\rho{=}0.9). Total rollout budget is fixed at 128 trajectories across all configurations.
PP (prompts) ×\times GG (traj/prompt) Task Perf. (%) Step Time (s) VRAM (GB)
NF F Δ\Delta NF F Δ\Delta% NF F Δ\Delta
128×1128\times 1 23.6 – – 89.8 – – 201.80 – –
64×264\times 2 18.8 27.3 +8.6+8.6 91.8 64.9 −29%-29\% 201.39 201.83 +0.44+0.44
32×432\times 4 24.2 27.4 +3.2+3.2 89.8 52.6 −41%-41\% 202.11 201.67 −0.44-0.44
8×168\times 16 15.6 23.6 +8.0+8.0 89.2 65.9 −26%-26\% 201.54 201.90 +0.36+0.36

5 Analysis

The analysis asks whether the diagnostic separates interventions and whether controlled selection/noise experiments support the proposed signal-quality explanation.

5.1 MI Diagnoses Collapse Better Than Entropy Across All Interventions

Entropy-strength, KL-strength and filter-mass sweeps trace different trajectories in performance–diagnostic space. The strongest reported MI-family correlation is +0.39; some MI variants are weaker or near zero, while entropy measures are slightly negative. This favors the selected proxy within these runs, without making it a universally calibrated predictor or validating the paper’s informal “twice as reliable” phrasing. KL anchoring can preserve properties of a reference policy without improving them.

Refer to caption
Figure 8: Spearman correlations showing MI-family metrics positively predict performance while entropy metrics are near-zero or negative.
Refer to caption
Figure 9: In FrozenLake, median success rates for both Top-p filtering (orange) and no filtering (gray) decrease as environment stochasticity increases from 0% to 100%. SNR-Aware Filtering maintains a clear advantage from 0% to 50% stochasticity, but the gap closes at 80%–100%, where high transition noise weakens reward variance as an informative signal proxy.

5.2 Does SNR Mechanism Really Interpret Agent RL?

Keeping only the highest reward-variance quartile yields 21.1% Sokoban success versus 11.0% for the lowest; the intermediate task scores are not strictly monotone. In FrozenLake the filtering advantage largely disappears at 80–100% transition noise. A prompt-level filter reaches 23.6% versus 12.9% without filtering and 16.8% for the trajectory-selection baseline. The source’s trajectory counts and MI-proxy units are not fully consistent, so these results support an intervention effect without resolving every causal alternative. Cross-prompt variance heterogeneity also matters: Table 8 reports a −5-point FrozenLake GRPO case when variance is relatively uniform. As more prompts yield nearly identical rewards, adaptive selection tightens. High format validity can coexist with low input dependence; neither formatting, entropy nor response length substitutes for a content-sensitive diagnostic.

Table 6: Quartile ablation on Sokoban (Qwen2.5-3B, P=8P{=}8, G=16G{=}16, keeping 25% of prompts per step). Task performance and MI degrade monotonically from Q1 to Q4.
Quartile RV Range Task Perf (%) MI Proxy Entropy
Q1 (highest RV) [4.4–5.6] 21.1 0.95 2.02
Q2 [1.5–4.2] 19.5 0.93 1.53
Q3 [0.0–0.2] 10.7 0.81 1.41
Q4 (lowest RV) [0.0–0.1] 11.0 0.73 1.87
Table 7: Trajectory-level vs. prompt-level filtering on Sokoban (Qwen2.5-3B). Prompt-level SNR-Aware Filtering provides the largest gains; trajectory-level filtering confirms that the benefit is not due to prompt-distribution bias.
Method Prompts Used Traj/Update Task Perf (%) MI Proxy
No filter 8/8 128 12.9 0.83
Prompt-level RV (ρ=0.9\rho{=}0.9) 3.2/8 50.6 23.6 1.80
Trajectory-level 8/8 64 16.8 0.20
Table 8: Per-setting RV statistics and filtering effectiveness. Std/Mean of RV predicts whether SNR-Aware Filtering helps: high ratio means bimodal RV and effective filtering; low ratio means uniform RV and random discarding.
Setting Filter Δ\Delta P G RV Mean RV Std RV Var RV Min RV Max Std/Mean
Sokoban, 14B +4.6%+4.6\% 8 8 2.24 2.88 8.32 0.10 6.00 1.29
Sokoban, 3B +3.2%+3.2\% 32 4 2.49 2.89 8.35 0.05 6.52 1.16
FrozenLake, 3B (GRPO) −5.0%-5.0\% 32 8 0.54 0.18 0.03 0.22 0.76 0.33
Refer to caption
Figure 10: Effective kept ratio and zero-variance prompt count, showing adaptive selection pressure as variance collapses during training.
Refer to caption
Figure 11: Prompt-level reward distribution across training phases, showing RV collapse as prompts shift toward uniform reward structures.
Refer to caption
Figure 12: Format validity versus MI and entropy diagnostics, showing that high validity does not guarantee high input dependence.

6 Related Work

Related work distinguishes language-level rationale degeneration, action-policy concentration and collapse under repeated self-training. Lexical or embedding diversity does not necessarily measure whether differences follow the input. Retrieval-style matching probes that relationship, whereas reasoning faithfulness asks whether a rationale causally supports the decision. Filtering complements standard stabilization and credit-assignment methods by changing which sampled groups contribute updates.

Original PDF visual retaining vector text, complete panel arrangement and native embedded image detail.
Figure 13: Training dynamics under three interventions. For each setting, we choose two checkpoints (steps 10/400) and connect them into a trajectory (arrows point to later steps). Color intensity indicates weaker to stronger intervention.

7 Conclusions and Limitations

The practical takeaway is to monitor input dependence alongside reward and entropy, and to test whether reward-variance filtering improves a particular training regime. Limitations include coupling between task and regularizer gradients, single-agent experiments, possible gaming of the variance criterion, sparse/noisy rewards, reduced exploration under aggressive filtering, and task-dependent keep-mass tuning. The theoretical upper bounds and empirical proxies explain specific risks; they do not guarantee that retained reasoning is correct or faithful.

8 Acknowledgements

The authors acknowledge Yuxiang Lin for infrastructure and environment support, and Kyunghyun Cho for manuscript discussions.

Appendix Contents

The appendices provide extended context, experimental settings, diagnostic plots, filtering definitions and mathematical derivations. All original displayed formulas are retained below.

Appendix A Extended Related Work

The extended review separates three questions: what degenerates during closed-loop training, how to measure input-sensitive reasoning, and how stabilization methods interact with weak reward signals.

Reasoning collapse and policy degeneracy in closed-loop LM and agent RL training.

Repeated optimization on self-generated trajectories can narrow both language patterns and behavioral choices. Rationale templating may appear while external behavior still earns reward, so language-level collapse and policy-level degeneration need not coincide. Multi-turn interaction adds opportunities for cross-task templates and mechanical later-turn behavior.

Evaluating reasoning diversity, input dependence, and reasoning faithfulness.

Lexical overlap, distributional distance and token uncertainty measure output variation but may miss input-independent templates. Prompt reconstruction and matching provide a different diagnostic axis. Input sensitivity remains distinct from causal faithfulness: a rationale can mention the correct input without being the mechanism that produced the action.

Stabilizing multi-turn Agent RL under closed-loop sampling.

KL control, entropy bonuses, clipping, reward shaping, curricula, replay and intermediate supervision address related stability problems. When trajectories receive indistinguishable rewards, these mechanisms may still leave little directional task signal. The proposed filter acts on that data-selection problem and can coexist with those other controls.

Appendix B Detailed Experimental Settings

The experimental details specify reward functions, horizons, optimization and stopping rules. Several descriptions differ across sections, so those discrepancies are noted rather than silently harmonized.

B.1 Environments and Tasks

Sokoban gives +1 for placing a box, −1 for removing it, +10 for completion and −0.1 per action. FrozenLake’s stated default executes intended actions with 98% probability and rewards only reaching the goal. MetaMathQA describes exact-match correctness with halved reward after each retry; Countdown distinguishes correct, well-formed incorrect and malformed expressions. DeepCoder uses test outcomes, SearchQA rewards answers/search efficiency, and WebShop scores instruction–product matching. The appendix’s numerical Countdown example reuses a number despite the stated once-only rule; the notes retain the rule without adopting that example.

B.2 Training and Evaluation Setup

Interactive grid tasks use up to five turns with two actions each. This setup section assigns one turn to both Countdown and MetaMathQA, despite the earlier description of MetaMathQA retries. Actor/critic learning rates are 10⁻⁶/10⁻⁵, update batch size is 32, per-GPU minibatch size four, and entropy coefficient 0.001; malformed output incurs −0.1. Early stopping triggers after five persistently low-variance iterations or five validation checkpoints below 1% success. Evaluation uses 512 fixed prompts and temperature 0.5. The formal top-p rule is cumulative variance mass despite this section’s loose description as a prompt fraction.

Appendix C Filtering Ablation Results

Sokoban ablations vary token sampling, filter metric and retained variance direction. Table 9 reports peak success with diagnostics at that checkpoint, plus whether training ever collapses below 1% validation success. Reward-variance selection improves success from 0.17 to 0.38 in its comparison block; retaining the largest rather than smallest variance groups gives 0.44 versus 0.29 in a separate block. Entropy- and length-based alternatives can preserve one diagnostic while failing on stability or task success. Sampling parameters and filtering interact, so rows from different blocks do not form one controlled ranking.

Table 9: Ablation results for sampling strategies, filtering metrics, and keep strategies. Values in parentheses denote the change relative to the corresponding baseline in each block. A crossmark in the Stable column indicates training collapse.
EXPERIMENT SETUP TASK PERF MI PROXY ENTROPY STABLE
Sampling Strategies
Top-p = 1.0 (Baseline) 0.17 0.54 2.76 ✗
Top-p = 0.9 0.38 (+0.20) 0.84 (+0.29) 1.64 (-1.12) ✓
Top-p = 0.5 0.29 (+0.12) 0.83 (+0.29) 1.88 (-0.88) ✓
Min-p = 0.05 0.42 (+0.25) 0.67 (+0.13) 1.64 (-1.12) ✓
Min-p = 0.2 0.45 (+0.27) 0.36 (-0.18) 3.01 (+0.26) ✓
Top-k = 0.25 0.22 (+0.05) 0.86 (+0.32) 1.28 (-1.48) ✓
Top-k = 0.5 0.44 (+0.27) 0.89 (+0.35) 1.47 (-1.29) ✓
Filtering Metrics
No Filter (Baseline) 0.17 0.54 2.76 ✗
Reward Variance 0.38 (+0.20) 0.84 (+0.29) 1.64 (-1.12) ✓
Reward Sum 0.24 (+0.07) 0.80 (+0.26) 4.18 (+1.42) ✗
Entropy 0.20 (+0.02) 0.41 (-0.14) 2.20 (-0.56) ✗
Entropy Variance 0.23 (+0.06) 0.70 (+0.16) 2.94 (+0.18) ✗
Length 0.16 (-0.02) 0.91 (+0.36) 1.65 (-1.10) ✗
Keep Strategies
Keep Largest (Baseline) 0.44 0.89 1.47 ✓
Keep Smallest 0.29 (-0.15) 0.47 (-0.42) 5.31 (+3.84) ✓

Appendix D Additional Experimental Visualizations

The supplementary plots show how several cross-scoring diagnostics and conditional entropy evolve under the same interventions.

D.1 MI Proxy Metrics During Training

Figure 14 includes the raw MI estimate, normalized/smoothed score, above-chance retrieval and retrieval at different candidate counts. The curves support the distinction between input dependence and entropy, with top-p filtering generally retaining stronger retrieval. The original trajectories are retained because individual metrics and regularization settings do not behave identically.

Original figure
Figure 14: Six MI proxy metrics over training steps. (a) MI Estimate, (b) MI Z-Score (EMA), (c) Retrieval Above Chance, (d) Retrieval Accuracy @4, (e) Retrieval Accuracy @8, (f) Conditional Entropy H⁡(Z|X)H(Z|X). Consistent with Figure 5, all proxies confirm that without filtering, MI degrades early, signaling reasoning collapse. Filtering effectively preserves information retention across all metrics, with top-pp SNR-aware filtering best maintaining reasoning diversity throughout training.

Appendix E Notation and basic identities

The notation distinguishes the policy’s true distribution from the fixed-scorer quantities used to approximate it.

E.1 Random variables and distributions

A rollout consists of a prompt, sampled reasoning trajectory and reward. Subtracting the conditional mean reward yields a within-prompt zero-mean advantage; the policy score is the gradient of log likelihood and has zero expectation under regularity conditions. Reward variance measures observed spread, which may come from meaningful trajectory differences or evaluation noise.

Equations
ξ=(x,z,r),\xi=(x,z,r),
x∼P(X),z∼πθ(⋅∣x),r=R(z;x),x\sim P(X),\qquad z\sim\pi_{\theta}(\cdot\mid x),\qquad r=R(z;x),
A⁡(z,x):=R⁡(z,x)−b⁡(x).A(z;x):=R(z;x)-b(x).
𝔼⁡[A⁡(Z,x)∣X=x]=𝔼⁡[R⁡(Z,x)∣X=x]−b⁡(x)=0.\mathbb{E}[A(Z;x)\mid X=x]=\mathbb{E}[R(Z;x)\mid X=x]-b(x)=0.
s⁡(z,x):=∇θ​log​πθ​(z∣x).s(z;x):=\nabla_{\theta}\log\pi_{\theta}(z\mid x).
𝔼z∼πθ(⋅∣x)[s(z;x)]=∇θ∫πθ(z∣x)dz=0.\mathbb{E}_{z\sim\pi_{\theta}(\cdot\mid x)}[\,s(z;x)\,]=\nabla_{\theta}\int\pi_{\theta}(z\mid x)\,dz=0.
RV(x):=Var(R(Z;x)∣X=x),Z∼πθ(⋅∣x).\mathrm{RV}(x):=\mathrm{Var}(R(Z;x)\mid X=x),\qquad Z\sim\pi_{\theta}(\cdot\mid x).

E.2 Entropy and mutual information

Conditional entropy measures average variability for a fixed input. Mutual information measures how far each conditional reasoning distribution differs from the marginal distribution over reasoning. These are properties of the full policy-induced distribution.

Equations
H(Z∣X):=𝔼x∼P⁡(X)[H(Z∣X=x)]=−𝔼x∼P(X),z∼πθ(⋅∣x)[logπθ(z∣x)].H(Z\mid X):=\mathbb{E}_{x\sim P(X)}\!\big[H(Z\mid X=x)\big]=-\mathbb{E}_{x\sim P(X),\,z\sim\pi_{\theta}(\cdot\mid x)}\!\big[\log\pi_{\theta}(z\mid x)\big].
I(X;Z):=𝔼x∼P(X),z∼πθ(⋅∣x)[logπθ​(z∣x)pθ​(z)],pθ(z):=𝔼x∼P⁡(X)[πθ(z∣x)].I(X;Z):=\mathbb{E}_{x\sim P(X),\,z\sim\pi_{\theta}(\cdot\mid x)}\!\left[\log\frac{\pi_{\theta}(z\mid x)}{p_{\theta}(z)}\right],\qquad p_{\theta}(z):=\mathbb{E}_{x\sim P(X)}\big[\pi_{\theta}(z\mid x)\big].

Decomposition identity (Shannon quantities).

The Shannon identity relates true marginal entropy, conditional entropy and MI. Scorer-defined likelihood quantities are measured under a possibly different distribution and need not equal these Shannon quantities. Their diagnostic interpretation depends on how well the scorer represents the evaluated policy.

Equations 2
H⁡(Z)=H⁡(Z∣X)+I⁡(X,Z),H(Z)\;=\;H(Z\mid X)\;+\;I(X;Z), (2)
𝒟^q:=NLL^q​(Z∣X)+I^q​(X,Z),\widehat{\mathcal{D}}_{q}:=\widehat{\mathrm{NLL}}_{q}(Z\mid X)+\widehat{I}_{q}(X;Z),

Interpretation for reasoning diversity.

A fall in conditional entropy means greater within-prompt concentration; a fall in MI means weaker cross-prompt discrimination. Tracking both prevents treating all compression as the same failure mode. Low reward variance is one proposed condition under which reward-agnostic updates gain influence.

Appendix F Scorer-based Proxies for Reasoning Diversity

This appendix formalizes matched and mixture likelihoods using a fixed scorer and a sampled collection of prompt groups.

F.1 Setup and notation

For each of P prompts, sample G trajectories and score every trajectory under every prompt. The matched score uses the originating prompt; the mixture uses a log-sum-exp over the uniform prompt set. Both depend on the scorer and batch composition. Appendix F fixes the scorer, whereas the main description writes current-policy likelihoods; the paper does not fully reconcile those conventions. Neither finite-batch proxy should be treated as exact MI without additional assumptions.

Equations
zi,k∼πθ(⋅∣xi),k=1,…,G.z_{i,k}\sim\pi_{\theta}(\cdot\mid x_{i}),\qquad k=1,\dots,G.
ℓi​(z):=log⁡q⁡(z∣xi).\ell_{i}(z)\;:=\;\log q(z\mid x_{i}).
ℓmix​(z):=log⁡(1P​∑j=1Pexp⁡(ℓj​(z)))=log⁡(1P​∑j=1Pq⁡(z∣xj)).\ell_{\mathrm{mix}}(z)\;:=\;\log\!\left(\frac{1}{P}\sum_{j=1}^{P}\exp(\ell_{j}(z))\right)=\log\!\left(\frac{1}{P}\sum_{j=1}^{P}q(z\mid x_{j})\right).

Appendix G Formal Definition of the Filtering Operator

A filter is specified by a grouping key, a statistic computed within each group, and a selection mask. Multiplying the per-sample loss by that mask removes rejected groups from gradient computation. The displayed definition is general; reward variance and prompt grouping are the paper’s primary instantiation.

Equations
u=g⁡(ξ).u=g(\xi).
ℬu:={ξ∈ℬ:g⁡(ξ)=u}.\mathcal{B}_{u}:=\{\xi\in\mathcal{B}:g(\xi)=u\}.
m(u):=𝟏{ϕ(ℬu)≥τ}.m(u):=\mathbf{1}\{\phi(\mathcal{B}_{u})\geq\tau\}.
ℒfilt​(θ)=1|ℬ|​∑ξ∈ℬm⁡(g⁡(ξ))​Lθ​(ξ).\mathcal{L}_{\mathrm{filt}}(\theta)\;=\;\frac{1}{|\mathcal{B}|}\sum_{\xi\in\mathcal{B}}m\!\big(g(\xi)\big)\,L_{\theta}(\xi).

Remark (post-sampling).

Filtering occurs after sampling. It leaves the already-collected rollout distribution intact while changing the distribution of samples used for optimization.

Remark (normalization).

Normalizing by all samples, retained samples or retained groups changes gradient scale even when the selection mask is identical. This is relevant when comparing learning rates and effective batch sizes.

G.1 Filtering Strategy Variants

The variants share prompt groups and return-variance scores; their difference is how the kept subset is selected.

Top-p (nucleus-style) filtering.

Top-p sorts by descending variance and keeps the smallest prefix reaching a chosen share of total variance mass. The number of groups adapts to the distribution. Zero-signal batch rejection requires the implementation’s handling of zero groups and numerical thresholds; it does not follow automatically from the positive-total-mass formula alone.

Equation
k∗=min⁡{k:∑j=1kVar^​(R∣X=xσ⁡(j))≥ρ​∑i=1NVar^​(R∣X=xi)}.k^{*}=\min\left\{k:\sum_{j=1}^{k}\widehat{\mathrm{Var}}(R\mid X=x_{\sigma(j)})\geq\rho\sum_{i=1}^{N}\widehat{\mathrm{Var}}(R\mid X=x_{i})\right\}.

Top-k (proportional) filtering.

Top-k retains a fixed fraction of groups by rank. It controls update count directly but can retain weak groups when little useful variation exists anywhere in the batch.

Equation
S={σ⁡(1),…,σ⁡(k)}.S=\{\sigma(1),\dots,\sigma(k)\}.

Min-p (threshold) filtering.

Min-p keeps groups whose variance exceeds a chosen fraction of the maximum. It adapts the count to a relative quality threshold, but a fluctuating maximum can make selection unstable.

Equations
τ=p⋅maxi⁡Var^​(R∣X=xi),\tau=p\cdot\max_{i}\widehat{\mathrm{Var}}(R\mid X=x_{i}),
S={i:Var^​(R∣X=xi)≥τ}.S=\left\{i:\widehat{\mathrm{Var}}(R\mid X=x_{i})\geq\tau\right\}.

Reverse top-p (low-variance) filtering.

Reverse top-p prioritizes low-variance groups as a diagnostic intervention. Its role is to test whether the high-variance direction matters, rather than merely reducing the update batch.

Implementation notes.

The implementation can exclude exactly zero-variance groups before selection and uses epsilon 0.01 for cumulative-threshold checks. These details affect degenerate batches and are separate from the idealized operator.

Appendix H RV Controls Task-Signal Magnitude and SNR

The bounds connect observed reward variation to expected task-gradient magnitude and sampling noise under explicit assumptions. They provide upper bounds and illustrative cases, not a guarantee that high-variance prompts carry useful gradients.

H.1 Setup

Fix a prompt and sample trajectories from its conditional policy. Use the conditional mean reward as baseline, so the second moment of advantage equals reward variance. The score-function norm supplies the additional factor in the gradient bound.

H.2 Assumption

Reward decomposes into a trajectory-dependent conditional mean and noise with zero conditional mean. The assumed conditional noise variance depends on the prompt but not the sampled trajectory, while the policy score is a measurable function of prompt and trajectory. These conditions support the variance decomposition that follows.

Equations
R(z;x)=μ(x,z)+ε,μ(x,z):=𝔼[R(z;x)∣x,z],R(z;x)=\mu(x,z)+\varepsilon,\qquad\mu(x,z):=\mathbb{E}[R(z;x)\mid x,z],
𝔼[ε∣x,z]=0,Var(ε∣x,z)=σ2(x)≥0.\mathbb{E}[\varepsilon\mid x,z]=0,\qquad\mathrm{Var}(\varepsilon\mid x,z)=\sigma^{2}(x)\geq 0.

H.3 Task-gradient magnitude is RV-controlled

The expected task-gradient norm is at most √RV times the root second moment of the score. The proof applies Cauchy–Schwarz to every unit-vector projection and takes the supremum. Low variance limits expected gradient magnitude when the score factor is controlled; the bound supplies no positive lower bound for high-variance groups.

Equations
‖gtask​(x)‖≤RV⁡(x)​𝔼⁡[‖s⁡(z,x)‖2∣X=x].\|g_{\mathrm{task}}(x)\|\;\leq\;\sqrt{\mathrm{RV}(x)}\,\sqrt{\mathbb{E}[\|s(z;x)\|^{2}\mid X=x]}.
gtask​(x)=𝔼⁡[A​s∣X=x].g_{\mathrm{task}}(x)=\mathbb{E}[A\,s\mid X=x].
|⟨u,gtask​(x)⟩|=|𝔼⁡[A⁡⟨u,s⟩∣X=x]|≤𝔼⁡[A2∣X=x]​𝔼⁡[⟨u,s⟩2∣X=x],\big|\langle u,g_{\mathrm{task}}(x)\rangle\big|=\big|\mathbb{E}[A\,\langle u,s\rangle\mid X=x]\big|\leq\sqrt{\mathbb{E}[A^{2}\mid X=x]}\;\sqrt{\mathbb{E}[\langle u,s\rangle^{2}\mid X=x]},
|⟨u,gtask​(x)⟩|≤𝔼⁡[A2∣X=x]​𝔼⁡[‖s‖2∣X=x].\big|\langle u,g_{\mathrm{task}}(x)\rangle\big|\leq\sqrt{\mathbb{E}[A^{2}\mid X=x]}\;\sqrt{\mathbb{E}[\|s\|^{2}\mid X=x]}.
‖gtask​(x)‖≤𝔼⁡[A2∣X=x]​𝔼⁡[‖s‖2∣X=x].\|g_{\mathrm{task}}(x)\|\leq\sqrt{\mathbb{E}[A^{2}\mid X=x]}\;\sqrt{\mathbb{E}[\|s\|^{2}\mid X=x]}.
𝔼⁡[A2∣X=x]=Var⁡(R∣X=x)=RV⁡(x).\mathbb{E}[A^{2}\mid X=x]=\mathrm{Var}(R\mid X=x)=\mathrm{RV}(x).

H.4 SNR is upper bounded by RV and reward noise

For G independent rollouts, the estimator’s mean-squared error has a floor from conditional reward noise. Combining that floor with the magnitude bound gives SNR ≤ √(G·RV)/σ. The result is vacuous when σ is zero. It formalizes a noise limitation under the stated model, rather than equating observed variance with signal.

Equations
g^task​(x):=1G​∑k=1GAk​sk,Ak:=A⁡(zk,x),sk:=s⁡(zk,x),\widehat{g}_{\mathrm{task}}(x):=\frac{1}{G}\sum_{k=1}^{G}A_{k}\,s_{k},\qquad A_{k}:=A(z_{k};x),\ s_{k}:=s(z_{k};x),
SNR⁡(x):=‖gtask​(x)‖𝔼⁡[‖g^task​(x)−gtask​(x)‖2∣X=x].\mathrm{SNR}(x):=\frac{\|g_{\mathrm{task}}(x)\|}{\sqrt{\mathbb{E}\big[\|\widehat{g}_{\mathrm{task}}(x)-g_{\mathrm{task}}(x)\|^{2}\mid X=x\big]}}.
SNR⁡(x)≤G⋅RV⁡(x)σ⁡(x).\mathrm{SNR}(x)\;\leq\;\sqrt{G}\cdot\frac{\sqrt{\mathrm{RV}(x)}}{\sigma(x)}.
g^=1G​∑k=1GAk​sk,g=𝔼⁡[A​s∣x],\widehat{g}\;=\;\frac{1}{G}\sum_{k=1}^{G}A_{k}s_{k},\qquad g\;=\;\mathbb{E}[As\mid x],
g^−g=1G​∑k=1G(Yk−g).\widehat{g}-g=\frac{1}{G}\sum_{k=1}^{G}(Y_{k}-g).
𝔼⁡[‖g^−g‖2∣x]\displaystyle\mathbb{E}\!\left[\|\widehat{g}-g\|^{2}\mid x\right] =1G2​𝔼​[‖∑k=1G(Yk−g)‖2|x]\displaystyle=\frac{1}{G^{2}}\mathbb{E}\!\left[\left\|\sum_{k=1}^{G}(Y_{k}-g)\right\|^{2}\Bigm|x\right]
=1G2​∑k=1G𝔼⁡[‖Yk−g‖2∣x]+1G2​∑k≠ℓ𝔼⁡[⟨Yk−g,Yℓ−g⟩∣x]\displaystyle=\frac{1}{G^{2}}\sum_{k=1}^{G}\mathbb{E}\!\left[\|Y_{k}-g\|^{2}\mid x\right]+\frac{1}{G^{2}}\sum_{k\neq\ell}\mathbb{E}\!\left[\langle Y_{k}-g,\;Y_{\ell}-g\rangle\mid x\right]
=1G2​∑k=1G𝔼⁡[‖Yk−g‖2∣x]\displaystyle=\frac{1}{G^{2}}\sum_{k=1}^{G}\mathbb{E}\!\left[\|Y_{k}-g\|^{2}\mid x\right]
=1G​𝔼​[‖A​s−g‖2∣x].\displaystyle=\frac{1}{G}\,\mathbb{E}\!\left[\|As-g\|^{2}\mid x\right].
A=R−b⁡(x)=(μ−𝔼⁡[μ∣x])+ε=:Aμ+ε.A\;=\;R-b(x)\;=\;(\mu-\mathbb{E}[\mu\mid x])+\varepsilon\;=:\;A_{\mu}+\varepsilon.
A​s−g=(Aμ​s−g)+ε​s,As-g=(A_{\mu}s-g)+\varepsilon s,
‖A​s−g‖2=‖Aμ​s−g‖2+‖ε​s‖2+2​⟨Aμ​s−g,ε​s⟩.\|As-g\|^{2}=\|A_{\mu}s-g\|^{2}+\|\varepsilon s\|^{2}+2\langle A_{\mu}s-g,\;\varepsilon s\rangle.
𝔼[⟨Aμs−g,εs⟩∣x]=𝔼[𝔼[⟨Aμs−g,εs⟩∣x,z]|x]=𝔼[⟨Aμs−g,s⟩𝔼[ε∣x,z]|x]=0,\mathbb{E}\!\left[\langle A_{\mu}s-g,\;\varepsilon s\rangle\mid x\right]=\mathbb{E}\!\left[\mathbb{E}\!\left[\langle A_{\mu}s-g,\;\varepsilon s\rangle\mid x,z\right]\Bigm|x\right]=\mathbb{E}\!\left[\langle A_{\mu}s-g,\;s\rangle\,\mathbb{E}[\varepsilon\mid x,z]\Bigm|x\right]=0,
𝔼⁡[‖A​s−g‖2∣x]≥𝔼⁡[‖ε​s‖2∣x].\mathbb{E}\!\left[\|As-g\|^{2}\mid x\right]\;\geq\;\mathbb{E}\!\left[\|\varepsilon s\|^{2}\mid x\right].
𝔼⁡[‖g^−g‖2∣x]≥1G​𝔼​[‖ε​s‖2∣x].\mathbb{E}\!\left[\|\widehat{g}-g\|^{2}\mid x\right]\;\geq\;\frac{1}{G}\,\mathbb{E}\!\left[\|\varepsilon s\|^{2}\mid x\right].
𝔼⁡[‖ε​s‖2∣x]\displaystyle\mathbb{E}\!\left[\|\varepsilon s\|^{2}\mid x\right] =𝔼[𝔼[ε2∥s∥2∣x,z]|x]\displaystyle=\mathbb{E}\!\left[\mathbb{E}\!\left[\varepsilon^{2}\|s\|^{2}\mid x,z\right]\Bigm|x\right]
=𝔼[∥s∥2𝔼[ε2∣x,z]|x]\displaystyle=\mathbb{E}\!\left[\|s\|^{2}\,\mathbb{E}\!\left[\varepsilon^{2}\mid x,z\right]\Bigm|x\right]
=𝔼⁡[‖s‖2​σ2​(x)|x]=σ2​(x)​𝔼​[‖s‖2∣x],\displaystyle=\mathbb{E}\!\left[\|s\|^{2}\,\sigma^{2}(x)\Bigm|x\right]=\sigma^{2}(x)\,\mathbb{E}\!\left[\|s\|^{2}\mid x\right],
𝔼⁡[‖g^−g‖2∣x]≥1G​σ2​(x)​𝔼​[‖s‖2∣x].\mathbb{E}\!\left[\|\widehat{g}-g\|^{2}\mid x\right]\;\geq\;\frac{1}{G}\,\sigma^{2}(x)\,\mathbb{E}\!\left[\|s\|^{2}\mid x\right].
‖g‖=‖𝔼⁡[A​s∣x]‖≤RV⁡(x)​𝔼⁡[‖s‖2∣x].\|g\|=\left\|\mathbb{E}[As\mid x]\right\|\leq\sqrt{\mathrm{RV}(x)}\;\sqrt{\mathbb{E}\!\left[\|s\|^{2}\mid x\right]}.
SNR⁡(x)≤RV⁡(x)​𝔼⁡[‖s‖2∣x]1G​σ2​(x)​𝔼​[‖s‖2∣x]=G⋅RV⁡(x)σ⁡(x).∎\mathrm{SNR}(x)\leq\frac{\sqrt{\mathrm{RV}(x)}\sqrt{\mathbb{E}[\|s\|^{2}\mid x]}}{\sqrt{\frac{1}{G}\sigma^{2}(x)\mathbb{E}[\|s\|^{2}\mid x]}}=\sqrt{G}\cdot\frac{\sqrt{\mathrm{RV}(x)}}{\sigma(x)}.\qed

H.5 Low-SNR updates induce parameter drift

An illustrative independent zero-mean update process has expected squared displacement η²Tv after T steps. The proof expands the sum and removes cross terms using independence. This shows how noise can accumulate without a directional mean; it does not itself prove template formation in a trained language model.

Equations
θt+1=θt+η​ξt,\theta_{t+1}=\theta_{t}+\eta\,\xi_{t},
𝔼⁡[‖θT−θ0‖2]=η2​T​v.\mathbb{E}\big[\|\theta_{T}-\theta_{0}\|^{2}\big]=\eta^{2}\,T\,v.
θT−θ0=η​∑t=0T−1ξt.\theta_{T}-\theta_{0}=\eta\sum_{t=0}^{T-1}\xi_{t}.
‖θT−θ0‖2=η2​‖∑t=0T−1ξt‖2=η2​(∑t=0T−1‖ξt‖2+2​∑0≤i<j≤T−1⟨ξi,ξj⟩).\|\theta_{T}-\theta_{0}\|^{2}=\eta^{2}\left\|\sum_{t=0}^{T-1}\xi_{t}\right\|^{2}=\eta^{2}\left(\sum_{t=0}^{T-1}\|\xi_{t}\|^{2}+2\sum_{0\leq i<j\leq T-1}\langle\xi_{i},\xi_{j}\rangle\right).
𝔼⁡⟨ξi,ξj⟩=⟨𝔼⁡[ξi],𝔼⁡[ξj]⟩=0,i≠j.\mathbb{E}\langle\xi_{i},\xi_{j}\rangle=\left\langle\mathbb{E}[\xi_{i}],\,\mathbb{E}[\xi_{j}]\right\rangle=0,\qquad i\neq j.
𝔼⁡[‖θT−θ0‖2]=η2​∑t=0T−1𝔼⁡[‖ξt‖2]=η2​T​v,\mathbb{E}\big[\|\theta_{T}-\theta_{0}\|^{2}\big]=\eta^{2}\sum_{t=0}^{T-1}\mathbb{E}[\|\xi_{t}\|^{2}]=\eta^{2}\,T\,v,

Appendix I Template Mixing Reduces Input Dependence

Mix each prompt-conditional distribution with the same prompt-independent template distribution at weight α. Joint convexity of KL yields Iα ≤ (1−α)I: this particular mixing operation can only contract input dependence. The result assumes the shared-template mixture rather than proving that every regularizer implements it. The source’s closing continuity remark concerns the separate KL-closeness result in Appendix L.

Equations
pα​(z∣x):=(1−α)​p​(z∣x)+α​q​(z),pα​(z):=(1−α)​p​(z)+α​q​(z).p_{\alpha}(z\mid x):=(1-\alpha)p(z\mid x)+\alpha q(z),\qquad p_{\alpha}(z):=(1-\alpha)p(z)+\alpha q(z).
Iα​(X,Z)≤(1−α)​I​(X,Z).I_{\alpha}(X;Z)\;\leq\;(1-\alpha)\,I(X;Z).
KL(pα(⋅∣x)∥pα(⋅))\displaystyle\mathrm{KL}\!\big(p_{\alpha}(\cdot\mid x)\,\|\,p_{\alpha}(\cdot)\big) =𝔼z∼pα(⋅∣x)[logpα​(z∣x)pα​(z)]\displaystyle=\mathbb{E}_{z\sim p_{\alpha}(\cdot\mid x)}\left[\log\frac{p_{\alpha}(z\mid x)}{p_{\alpha}(z)}\right]
=𝔼z∼pα(⋅∣x)[logpα(z∣x)−logpα(z)].\displaystyle=\mathbb{E}_{z\sim p_{\alpha}(\cdot\mid x)}\big[\log p_{\alpha}(z\mid x)-\log p_{\alpha}(z)\big].
Iα(X;Z)=𝔼x[KL(pα(⋅∣x)∥pα(⋅))].I_{\alpha}(X;Z)=\mathbb{E}_{x}\Big[\mathrm{KL}\!\big(p_{\alpha}(\cdot\mid x)\,\|\,p_{\alpha}(\cdot)\big)\Big].
KL((1−α)a+αb∥(1−α)c+αd)≤(1−α)KL(a∥c)+αKL(b∥d).\mathrm{KL}\!\big((1-\alpha)a+\alpha b\,\|\,(1-\alpha)c+\alpha d\big)\leq(1-\alpha)\mathrm{KL}(a\|c)+\alpha\,\mathrm{KL}(b\|d).
pα(⋅∣x)=(1−α)p(⋅∣x)+αq,pα(⋅)=(1−α)p(⋅)+αq,p_{\alpha}(\cdot\mid x)=(1-\alpha)p(\cdot\mid x)+\alpha q,\qquad p_{\alpha}(\cdot)=(1-\alpha)p(\cdot)+\alpha q,
KL(pα(⋅∣x)∥pα(⋅))\displaystyle\mathrm{KL}\!\big(p_{\alpha}(\cdot\mid x)\,\|\,p_{\alpha}(\cdot)\big) ≤(1−α)KL(p(⋅∣x)∥p(⋅))+αKL(q∥q)\displaystyle\leq(1-\alpha)\mathrm{KL}\!\big(p(\cdot\mid x)\,\|\,p(\cdot)\big)+\alpha\,\mathrm{KL}(q\|q)
=(1−α)KL(p(⋅∣x)∥p(⋅)).\displaystyle=(1-\alpha)\mathrm{KL}\!\big(p(\cdot\mid x)\,\|\,p(\cdot)\big).
𝔼x[KL(pα(⋅∣x)∥pα(⋅))]≤(1−α)𝔼x[KL(p(⋅∣x)∥p(⋅))].\mathbb{E}_{x}\Big[\mathrm{KL}\!\big(p_{\alpha}(\cdot\mid x)\,\|\,p_{\alpha}(\cdot)\big)\Big]\leq(1-\alpha)\,\mathbb{E}_{x}\Big[\mathrm{KL}\!\big(p(\cdot\mid x)\,\|\,p(\cdot)\big)\Big].
Iα​(X,Z)≤(1−α)​I​(X,Z),I_{\alpha}(X;Z)\leq(1-\alpha)\,I(X;Z),

Appendix J Filtering Reduces Gradient-Estimation MSE

Filtering changes both estimator noise and the target mean gradient. The derivation separates these two effects.

J.1 Setup

Each prompt group has a mean gradient plus independent zero-mean estimation noise with a specified second moment. A retained subset contains n such groups. The analysis treats that subset and its target gradient as given.

Equation
g^i=gi+εi,𝔼⁡[εi]=0,𝔼​‖εi‖2=σi2,\widehat{g}_{i}\;=\;g_{i}+\varepsilon_{i},\qquad\mathbb{E}[\varepsilon_{i}]=0,\qquad\mathbb{E}\|\varepsilon_{i}\|^{2}=\sigma_{i}^{2},

J.2 Unfiltered vs. filtered estimators

For a fixed kept subset, the averaged estimator is unbiased for that subset’s mean and has MSE equal to the sum of kept noise variances divided by n². The proof uses independence to remove cross terms. Dropping high-noise groups can help, but fewer groups also change the denominator; this identity alone does not guarantee an improvement for every selection rule or for data-dependent selection using the same noisy samples.

Equations
g¯^:=1P​∑i=1Pg^i,g¯:=1P​∑i=1Pgi.\widehat{\bar{g}}:=\frac{1}{P}\sum_{i=1}^{P}\widehat{g}_{i},\qquad\bar{g}:=\frac{1}{P}\sum_{i=1}^{P}g_{i}.
g¯^S:=1n​∑i∈Sg^i,g¯S:=1n​∑i∈Sgi.\widehat{\bar{g}}_{S}:=\frac{1}{n}\sum_{i\in S}\widehat{g}_{i},\qquad\bar{g}_{S}:=\frac{1}{n}\sum_{i\in S}g_{i}.
𝔼​‖g¯^S−g¯S‖2=1n2​∑i∈Sσi2.\mathbb{E}\big\|\widehat{\bar{g}}_{S}-\bar{g}_{S}\big\|^{2}\;=\;\frac{1}{n^{2}}\sum_{i\in S}\sigma_{i}^{2}.
𝔼⁡[g^i]=gi.\mathbb{E}[\widehat{g}_{i}]=g_{i}.
𝔼⁡[g¯^S]=1n​∑i∈S𝔼⁡[g^i]=1n​∑i∈Sgi=g¯S.\mathbb{E}[\widehat{\bar{g}}_{S}]=\frac{1}{n}\sum_{i\in S}\mathbb{E}[\widehat{g}_{i}]=\frac{1}{n}\sum_{i\in S}g_{i}=\bar{g}_{S}.
g¯^S−g¯S=1n​∑i∈S(g^i−gi)=1n​∑i∈Sεi.\widehat{\bar{g}}_{S}-\bar{g}_{S}=\frac{1}{n}\sum_{i\in S}(\widehat{g}_{i}-g_{i})=\frac{1}{n}\sum_{i\in S}\varepsilon_{i}.
𝔼​‖g¯^S−g¯S‖2\displaystyle\mathbb{E}\big\|\widehat{\bar{g}}_{S}-\bar{g}_{S}\big\|^{2} =1n2​𝔼​‖∑i∈Sεi‖2\displaystyle=\frac{1}{n^{2}}\,\mathbb{E}\left\|\sum_{i\in S}\varepsilon_{i}\right\|^{2}
=1n2​(∑i∈S𝔼​‖εi‖2+∑i,j∈Si≠j𝔼⁡⟨εi,εj⟩).\displaystyle=\frac{1}{n^{2}}\left(\sum_{i\in S}\mathbb{E}\|\varepsilon_{i}\|^{2}+\sum_{\begin{subarray}{c}i,j\in S\\ i\neq j\end{subarray}}\mathbb{E}\langle\varepsilon_{i},\varepsilon_{j}\rangle\right).
𝔼⁡⟨εi,εj⟩=⟨𝔼⁡[εi],𝔼⁡[εj]⟩=0,\mathbb{E}\langle\varepsilon_{i},\varepsilon_{j}\rangle=\left\langle\mathbb{E}[\varepsilon_{i}],\,\mathbb{E}[\varepsilon_{j}]\right\rangle=0,
𝔼​‖g¯^S−g¯S‖2=1n2​∑i∈S𝔼​‖εi‖2=1n2​∑i∈Sσi2.\mathbb{E}\big\|\widehat{\bar{g}}_{S}-\bar{g}_{S}\big\|^{2}=\frac{1}{n^{2}}\sum_{i\in S}\mathbb{E}\|\varepsilon_{i}\|^{2}=\frac{1}{n^{2}}\sum_{i\in S}\sigma_{i}^{2}.

Remark (bias relative to the original objective).

The filtered estimator generally targets a different mean gradient from the full batch. Its unbiasedness for the retained objective should not be described as unbiasedness for the original training objective.

Appendix K Reward-Agnostic Regularizers and Update Dominance

The regularizer analysis concerns reward-agnostic gradients: they do not use advantage weights. This is weaker than saying the gradients are identical for every input.

K.1 Setup

Write the total expected update as the task gradient plus KL/entropy regularizer gradients. Define a dominance ratio from the regularizer norm relative to the sum of task and regularizer norms. Prompt dependence is explicit in the appendix’s notation.

Equations
gtask​(x):=𝔼⁡[(R⁡(z,x)−b⁡(x))​s​(z,x)∣X=x],s⁡(z,x):=∇θ​log​πθ​(z∣x).g_{\mathrm{task}}(x)\;:=\;\mathbb{E}\!\left[(R(z;x)-b(x))\,s(z;x)\mid X=x\right],\qquad s(z;x):=\nabla_{\theta}\log\pi_{\theta}(z\mid x).
greg​(x):=λKL​gKL​(x)+λent​gent​(x),g_{\mathrm{reg}}(x):=\lambda_{\mathrm{KL}}\,g_{\mathrm{KL}}(x)+\lambda_{\mathrm{ent}}\,g_{\mathrm{ent}}(x),
gtotal​(x)=gtask​(x)+greg​(x).g_{\mathrm{total}}(x)=g_{\mathrm{task}}(x)+g_{\mathrm{reg}}(x).
ρ⁡(x):=‖greg​(x)‖‖gtask​(x)‖+‖greg​(x)‖∈[0,1].\rho(x):=\frac{\|g_{\mathrm{reg}}(x)\|}{\|g_{\mathrm{task}}(x)\|+\|g_{\mathrm{reg}}(x)\|}\in[0,1].

K.2 Low-RV prompts amplify regularizer influence

The task-gradient upper bound induces a lower bound on regularizer dominance. If regularizer and score-function norms change slowly compared with reward variance, smaller variance makes the regularizer’s relative influence larger. This is a conditional influence statement; the resulting direction of change in input dependence requires further assumptions or empirical evidence.

Equations
‖gtask​(x)‖≤RV⁡(x)​𝔼⁡[‖s‖2∣X=x].\|g_{\mathrm{task}}(x)\|\leq\sqrt{\mathrm{RV}(x)}\;\sqrt{\mathbb{E}[\|s\|^{2}\mid X=x]}.
ρ⁡(x)=‖greg​(x)‖‖gtask​(x)‖+‖greg​(x)‖\rho(x)=\frac{\|g_{\mathrm{reg}}(x)\|}{\|g_{\mathrm{task}}(x)\|+\|g_{\mathrm{reg}}(x)\|}
ρ⁡(x)≥‖greg​(x)‖‖greg​(x)‖+RV⁡(x)​𝔼⁡[‖s‖2∣X=x].\rho(x)\;\geq\;\frac{\|g_{\mathrm{reg}}(x)\|}{\|g_{\mathrm{reg}}(x)\|+\sqrt{\mathrm{RV}(x)}\sqrt{\mathbb{E}[\|s\|^{2}\mid X=x]}}.

Appendix L KL-Closeness to the Base Implies MI-Closeness

On finite prompt and reasoning alphabets, uniform KL-closeness to a reference implies MI-closeness. The proof bounds joint KL, applies Pinsker to total variation, then uses entropy continuity for joint and marginal distributions. The bound depends on alphabet size and may be loose for language spaces. Staying near a reference can preserve its input dependence, but does not imply increasing it.

Equations
Pθ​(x,z):=P⁡(x)​πθ​(z∣x),P0​(x,z):=P⁡(x)​π0​(z∣x).P_{\theta}(x,z):=P(x)\pi_{\theta}(z\mid x),\qquad P_{0}(x,z):=P(x)\pi_{0}(z\mid x).
supx∈𝒳KL(πθ(⋅∣x)∥π0(⋅∣x))≤ε,\sup_{x\in\mathcal{X}}\mathrm{KL}\!\left(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{0}(\cdot\mid x)\right)\leq\varepsilon,
|Iθ​(X,Z)−I0​(X,Z)|≤f⁡(ε).\big|I_{\theta}(X;Z)-I_{0}(X;Z)\big|\leq f(\varepsilon).
KL(Pθ(X,Z)∥P0(X,Z))=𝔼x∼P[KL(πθ(⋅∣x)∥π0(⋅∣x))].\mathrm{KL}\!\left(P_{\theta}(X,Z)\,\|\,P_{0}(X,Z)\right)=\mathbb{E}_{x\sim P}\!\left[\mathrm{KL}\!\left(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{0}(\cdot\mid x)\right)\right].
KL(Pθ(X,Z)∥P0(X,Z))≤ε.\mathrm{KL}\!\left(P_{\theta}(X,Z)\,\|\,P_{0}(X,Z)\right)\leq\varepsilon.
‖Pθ​(X,Z)−P0​(X,Z)‖TV≤12KL(Pθ(X,Z)∥P0(X,Z))≤ε2=:δ.\|P_{\theta}(X,Z)-P_{0}(X,Z)\|_{\mathrm{TV}}\leq\sqrt{\tfrac{1}{2}\,\mathrm{KL}\!\left(P_{\theta}(X,Z)\,\|\,P_{0}(X,Z)\right)}\leq\sqrt{\tfrac{\varepsilon}{2}}=:\delta.
|Hθ​(X,Z)−H0​(X,Z)|≤δ​log⁡(|𝒳|​|𝒵|−1)+h2​(δ),\big|H_{\theta}(X,Z)-H_{0}(X,Z)\big|\leq\delta\log(|\mathcal{X}||\mathcal{Z}|-1)+h_{2}(\delta),
‖Pθ​(Z)−P0​(Z)‖TV≤δ,\|P_{\theta}(Z)-P_{0}(Z)\|_{\mathrm{TV}}\leq\delta,
|Hθ​(Z)−H0​(Z)|≤δ​log⁡(|𝒵|−1)+h2​(δ)≤δ​log⁡(|𝒳|​|𝒵|−1)+h2​(δ).\big|H_{\theta}(Z)-H_{0}(Z)\big|\leq\delta\log(|\mathcal{Z}|-1)+h_{2}(\delta)\leq\delta\log(|\mathcal{X}||\mathcal{Z}|-1)+h_{2}(\delta).
|Iθ​(X,Z)−I0​(X,Z)|\displaystyle\big|I_{\theta}(X;Z)-I_{0}(X;Z)\big| =|(Hθ​(Z)−H0​(Z))−(Hθ​(X,Z)−H0​(X,Z))|\displaystyle=\big|(H_{\theta}(Z)-H_{0}(Z))-(H_{\theta}(X,Z)-H_{0}(X,Z))\big|
≤|Hθ​(Z)−H0​(Z)|+|Hθ​(X,Z)−H0​(X,Z)|\displaystyle\leq\big|H_{\theta}(Z)-H_{0}(Z)\big|+\big|H_{\theta}(X,Z)-H_{0}(X,Z)\big|
≤2​(δ​log⁡(|𝒳|​|𝒵|−1)+h2​(δ)).\displaystyle\leq 2\Big(\delta\log(|\mathcal{X}||\mathcal{Z}|-1)+h_{2}(\delta)\Big).
f⁡(ε):=2​(δ​log⁡(|𝒳|​|𝒵|−1)+h2​(δ)),δ:=ε2,f(\varepsilon):=2\Big(\delta\log(|\mathcal{X}||\mathcal{Z}|-1)+h_{2}(\delta)\Big),\qquad\delta:=\sqrt{\tfrac{\varepsilon}{2}},

Appendix M Decomposing Changes in Input Dependence

The change in true MI equals the change in marginal entropy minus the change in conditional entropy. Therefore, if within-input diversity increases more than total diversity, MI decreases. An entropy bonus may create this imbalance, but the algebraic identity does not say that it always does; the paper presents that connection as a plausible mechanism.

Equations
Δin:=Hθ​(Z∣X)−H0​(Z∣X),Δmarg:=Hθ​(Z)−H0​(Z).\Delta_{\mathrm{in}}:=H_{\theta}(Z\mid X)-H_{0}(Z\mid X),\qquad\Delta_{\mathrm{marg}}:=H_{\theta}(Z)-H_{0}(Z).
Iθ​(X,Z)−I0​(X,Z)=Δmarg−Δin.I_{\theta}(X;Z)-I_{0}(X;Z)=\Delta_{\mathrm{marg}}-\Delta_{\mathrm{in}}.
Iθ​(X,Z)≤I0​(X,Z)−γ,I_{\theta}(X;Z)\leq I_{0}(X;Z)-\gamma,
Iθ​(X,Z)−I0​(X,Z)\displaystyle I_{\theta}(X;Z)-I_{0}(X;Z) =(Hθ​(Z)−H0​(Z))−(Hθ​(Z∣X)−H0​(Z∣X))\displaystyle=\big(H_{\theta}(Z)-H_{0}(Z)\big)-\big(H_{\theta}(Z\mid X)-H_{0}(Z\mid X)\big)
=Δmarg−Δin.\displaystyle=\Delta_{\mathrm{marg}}-\Delta_{\mathrm{in}}.

Appendix N GRPO Normalization Amplifies Noise at Low RV

The displayed normalization divides advantage by √RV. Under the reward-noise assumptions, the normalized gradient estimator has a variance floor proportional to σ²/(K·RV), times the score second moment. Low reward variance can thus amplify the relative noise contribution. The bound is zero when σ is zero and describes the idealized population normalization, rather than all practical stabilized finite-sample implementations.

Equations
A~(z;x):=A⁡(z,x)RV⁡(x),A(z;x):=R(z;x)−b(x),b(x):=𝔼z∼πθ(⋅∣x)[R(z;x)].\widetilde{A}(z;x):=\frac{A(z;x)}{\sqrt{\mathrm{RV}(x)}},\qquad A(z;x):=R(z;x)-b(x),\qquad b(x):=\mathbb{E}_{z\sim\pi_{\theta}(\cdot\mid x)}[R(z;x)].
g^GRPO​(x):=1K​∑k=1KA~k​sk,gGRPO​(x):=𝔼⁡[A~​s∣X=x],\widehat{g}_{\mathrm{GRPO}}(x):=\frac{1}{K}\sum_{k=1}^{K}\widetilde{A}_{k}\,s_{k},\qquad g_{\mathrm{GRPO}}(x):=\mathbb{E}[\widetilde{A}\,s\mid X=x],
𝔼⁡[‖g^GRPO​(x)−gGRPO​(x)‖2∣X=x]≥1K⋅σ2​(x)RV⁡(x)​𝔼​[‖s‖2∣X=x].\mathbb{E}\!\left[\big\|\widehat{g}_{\mathrm{GRPO}}(x)-g_{\mathrm{GRPO}}(x)\big\|^{2}\mid X=x\right]\;\geq\;\frac{1}{K}\cdot\frac{\sigma^{2}(x)}{\mathrm{RV}(x)}\;\mathbb{E}[\|s\|^{2}\mid X=x].

Appendix O Core Author Contributions

Zihan Wang led work across conceptualization, infrastructure, analysis, experiments and writing. Chi Gui, Xing Jin and Qineng Wang contributed ideas, software, experiments and presentation; Licheng Liu focused on formal analysis and theory alongside experiments and writing.

Concept demonstration

Open image ↗