Overview
The overview contrasts static prompt-response training with trajectories containing environment observations, reasoning, actions and feedback. All original framework diagrams, result plots, finding boxes, prompts and rollout examples are retained.
At a glance
RAGEN implements StarPO, a framework for reinforcement learning over multi-turn language-agent trajectories. Experiments use Bandit, Sokoban, FrozenLake and WebShop. Baseline training can lose reward diversity and task performance; StarPO-S combines high-variability prompt filtering, critic-based optimization and gradient-shaping changes to improve stability. Diverse initial states, several responses per state, moderate action batches and fresh rollouts often improve performance. Explicit reasoning tags alone do not ensure faithful or useful reasoning: traces can shorten or misdescribe the state while actions still earn reward. Fine-grained reasoning rewards are proposed future work rather than a demonstrated solution.
1 Introduction
Interactive agents must act repeatedly under feedback, unlike a single static completion. RAGEN provides controlled environments and a training loop to study how optimization, rollout construction and reward signals affect those agents. The symbolic tasks isolate particular difficulties; WebShop Yao et al. (2022a) adds language grounding and longer contexts.
The paper calls recurring loss of behavioral diversity and performance an “Echo Trap.” Its stabilization methods and generalization tests provide empirical guidance for these settings, not a universal theory of agent collapse. Most experiments fine-tune small, already pretrained Qwen models. “Self-evolution” refers to updating them with generated trajectories and task rewards; it does not mean training a language model from scratch or autonomous modification of its training system.
2 Framework
StarPO is a trajectory-level abstraction that can use several familiar policy optimizers; RAGEN supplies rollout execution and evaluation.
2.1 The MDP Formulation for Agent Training
The agent observes a state/history, generates an action, and receives the next observation and reward from a possibly stochastic environment. A finite sequence of these transitions forms the optimization material. State can include the interaction history, allowing the formulation to represent information absent from the current observation. The environment’s dynamics remain external to the policy.
Equations 1
| (1) |
2.2 StarPO: Reinforcing Reasoning via Trajectory-Level Optimization
The objective maximizes expected cumulative trajectory reward. Autoregressive token probabilities parameterize the agent’s generated reasoning and actions. Treating the rollout as one unit enables trajectory rewards and token-level updates, but does not by itself solve long-horizon credit assignment or make the observed environment transitions trainable policy outputs. The original objective-comparison box is preserved.
Equation 2
| (2) |
Original source text
Trajectory-Level Objective in StarPO vs. Previous Methods
Previous methods (e.g., PPO (Schulman et al., 2017), GRPO (Shao et al., 2024a)):
Equation
StarPO (ours):
Equation
2.2.2 Optimization Procedure: Learning from Reasoning-Interaction Trajectories
For each initial state, the policy generates multiple trajectories. Each turn can contain a reasoning block and a sequence of executable actions; the environment supplies feedback. A loop samples P states and N trajectories per state, then updates with batch size E; the stated update count is LPN/E over L loops. The framework allows on-policy collection or older-policy data conceptually, while the reported study does not establish a replay-buffer implementation or benefit.
Equation 3
| (3) |
2.2.3 Modular Optimization Strategies
PPO Schulman et al. (2017) uses a critic to estimate token-level advantages and clips probability-ratio updates. GRPO Shao et al. (2024a) normalizes each trajectory’s return against its group and assigns the resulting advantage across its tokens. The full objectives remain visible. A critic changes the variance and credit-assignment tradeoff; the experiments do not show one optimizer dominating across all environments.
Equations 4–6
| (4) |
| (5) |
| (6) |
2.3 The RAGEN System
RAGEN supports structured multi-turn rollouts, configurable rewards, environment extensions and policy updates. Its main contribution is experimental infrastructure and diagnosis, alongside the StarPO-S modifications.
3 Experiment Setup
Four tasks probe distinct sources of difficulty, with validation success and training diagnostics tracked across update iterations.
3.1 Environments and Tasks
Bandit is a one-decision stochastic reward problem; Sokoban has deterministic but irreversible pushes; FrozenLake has stochastic motion and sparse success reward; WebShop requires searching and selecting products matching text instructions. These are controlled agent tasks, not robotics or unconstrained live-web deployment.
3.2 Training Settings
Main symbolic experiments fine-tune Qwen2.5-Instruct-0.5B; WebShop uses 3B. Defaults sample eight prompts with 16 trajectories each, up to five turns and ten total actions. PPO uses GAE with γ = λ = 1; Adam and an entropy coefficient of 0.001 support optimization. Invalid response formatting incurs −0.1. Runs last 100–200 rollout/update iterations, with task and variant differences detailed in the appendix.
3.3 Evaluation Metrics
Success is evaluated on 256 fixed prompts with temperature 0.5. Other diagnostics include token entropy, within-prompt reward standard deviation, output length and gradient norm. These are different quantities: in a stochastic environment, reward variation can reflect transition noise even when the policy’s behavior is identical; longer text does not establish better reasoning.
4 Experimental Results and Findings
The main experiments examine instability, stabilization, rollout design and the relationship between visible reasoning and task reward.
4.1 Multi-turn Agent RL Training Introduces New Instability Pattern
Vanilla PPO/GRPO often improve then collapse on Bandit and Sokoban. PPO lasts longer in those examples, while GRPO is more stable on FrozenLake; both can learn WebShop. The proposed explanation involving critic noise and language priors is plausible but not causally isolated. Bandit examples shift from varied hypotheses to repeated templates, motivating the Echo Trap description and comparison with recursive-data collapse Shumailov et al. (2024).
Reward standard deviation sometimes falls before performance, while gradient spikes accompany failure and entropy trajectories can become erratic. These plots show useful correlated diagnostics. They do not establish a universally predictive threshold, prove that a gradient spike causes collapse, or demonstrate that recovery is impossible. Several finding boxes use stronger language than the evidence supports; their original content remains available.
Original source text
Original source text
Finding 2: Model collapse in agent RL is reflected as "Echo Trap" over training
We find that early-stage agent respond with diverse symbolic reasoning, but collapse into deterministic, repetitive templates after training. Models converge to fixed phrasing, indicating that RL may reinforce superficial patterns instead of general reasoning and forms an "Echo Trap" that hinders long-term generalization.
Original source text
Finding 3: Collapse follows similar dynamics and can be anticipated by indicators
Reward standard deviation and entropy often fluctuate before performance degrades, while gradient norm spikes typically mark the point of irreversible collapse. These metrics provide early indicators and motivate the need for stabilization strategies.
4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping
StarPO-S ranks initial states by the standard deviation of returns from repeated rollouts and retains only the highest-variability fraction for updates. The default keeps 25%, though the best fraction depends on the task. This resembles uncertainty sampling Settles (2009), but return variability is not a calibrated measure of epistemic uncertainty and can reward intrinsically noisy states.
Filtering delays collapse in the plotted PPO runs and reduces update work. It still requires generating rollouts to estimate their variability; speed gains should not be read as avoiding all discarded-sample generation costs. A tested run without collapse is evidence over its finite horizon, not a guarantee. GRPO benefits less consistently.
The stabilized recipe also removes the KL penalty and uses asymmetric probability clipping inspired by DAPO Yu et al. (2025). PPO’s critic remains part of the preferred symbolic-task setup. The combined system improves the reported curves, while appendix ablations show that individual changes need not eliminate late collapse. Turn-aware masking and bi-level GAE are additional variants rather than required components of every reported run.
Equation 7
| (7) |
Original source text
Finding 4: Filtering low-variance trajectories improves stability and efficiency
Training on high-variance prompts delays or eliminates collapse in multi-turn RL. StarPO-S improves performance and reduces update steps by discarding low-information rollouts, especially under PPO. This aligns with active learning principles, where uncertain examples offer the most informative learning signals.
4.3 Generating Useful Trajectories for RL Training
With a fixed rollout batch, four responses per prompt give stronger held-out-vocabulary and FrozenLake transfer than the default 16: 25.39% and 21.48% in Table 1. This is not an across-the-board monotonic diversity result: 32 responses achieve slightly higher in-distribution Sokoban accuracy (21.09% versus 20.70%). One or two responses weaken FrozenLake transfer, supporting a balance between state variety and within-state comparisons.
Allowing five or six actions per turn generally helps the tested Sokoban variants. Six reaches 33.59% on Sokoban, 31.64% with new symbols and 6.64% on the larger board; FrozenLake peaks at five actions (21.09%) and falls at six. Seven actions degrade several outcomes. These budgets change the interaction structure, so they are not a universal optimum for long-horizon agents.
Online-k reuses one rollout batch for k updates. Fresh Online-1 data often learns faster and transfers better than reuse for five or ten updates, though curves overlap and individual endpoints vary. More frequent collection also changes sampling cost; no equal-compute universal ranking is established.
| Response Per Prompt | SingleSokoban | SokobanNewVocab | FrozenLake |
|---|---|---|---|
| 32 | 21.09% | 20.22% | 17.97% |
| 16 | 20.31% | 21.48% | 19.53% |
| 8 | 20.31% | 19.53% | 17.19% |
| 4 | 20.70% | 25.39% | 21.48% |
| 2 | 19.92% | 25.00% | 12.50% |
| 1 | 19.53% | 22.27% | 12.50% |
| Max Actions / Turn | Sokoban | SokobanNewVocab | LargeSokoban | FrozenLake |
|---|---|---|---|---|
| 1 | 12.11% | 13.67% | 1.17% | 11.72% |
| 2 | 16.41% | 21.09% | 3.52% | 18.36% |
| 3 | 19.53% | 19.53% | 1.95% | 20.88% |
| 4 | 26.95% | 26.95% | 5.08% | 20.70% |
| 5 | 28.13% | 25.78% | 6.25% | 21.09% |
| 6 | 33.59% | 31.64% | 6.64% | 18.36% |
| 7 | 22.27% | 28.52% | 3.91% | 19.53% |
| Train on Bandit | Train on Sokoban | |||||
|---|---|---|---|---|---|---|
| Bandit | Bandit-Rev | FrozenLake | LargeSokoban | Sokoban | SokobanNewVocab | |
| StarPO-S | 100.00 | 67.58 | 19.92 | 2.34 | 21.48 | 18.75 |
| NoThink | 81.25 | 56.25 | 19.53 | 2.73 | 20.73 | 26.17 |
Original source text
Finding 5: Task diversity, action budget, and rollout frequency affect rollout quality
Diverse task instances enable better policy contrast and generalization across environments. Moderate action budgets provide enough planning space and avoid the noise introduced by overly long sequences. Up-to-date rollouts ensure optimization targets remain aligned with current policy behavior.
4.4 Reasoning Improves Generalization but Fades in Multi-Turn Settings Without Fine-Grained Rewards
Reasoning-format training improves held-out symbolic Bandit choices: 100.00 versus 81.25% in the aligned setting and 67.58 versus 56.25% under reversed reward associations. These results indicate an effect of the training/output format; they do not independently verify that the written explanations are faithful computations.
Sokoban-trained models show mixed effects. NoThink performs better on the larger board (2.73 versus 2.34%) and new vocabulary (26.17 versus 18.75%), while the reasoning variant is slightly better on original Sokoban and FrozenLake transfer. Table 3 incorrectly bolds NoThink’s 20.73 over the larger 21.48 original-Sokoban value; all original cells remain unchanged.
Trace lengths generally shrink during training, and examples contain incorrect plans despite successful actions. Length is only a proxy; shortening alone does not prove reasoning disappeared. A formatting penalty encourages tags but cannot assess the correctness of thoughts. The paper argues for process-sensitive or partial-correctness rewards, which it does not evaluate as a complete remedy.
| Bandit | Sokoban | |||||
|---|---|---|---|---|---|---|
| Step | Original | NoThink | Reverse | Reverse NoThink | Original | NoThink |
| 0 | 66.0 | 12.7 | 68.8 | 12.7 | 307.1 | 68.6 |
| 100 | 25.6 | 12.4 | 33.7 | 13.0 | 104.6 | 55.3 |
| 200 | 17.6 | 12.4 | 30.7 | 13.0 | 89.5 | 60.0 |
Original source text
Finding 6: Reasoning fails to emerge without meticulous reward design
While symbolic reasoning can emerge in simple, single-turn tasks under weak supervision, it fails to persist in multi-turn environments without the reward design explicitly encouraging interpretable intermediate reasoning steps. We observe that even with structured prompts, reasoning gradually decays during training if the reward signal focuses only on final outcomes. This suggests that without meticulous reward shaping, agents may tend to collapse into shortcut behaviors that bypass reasoning altogether.
5 Related Work
The work connects standard policy optimization, group-relative and leave-one-out baselines, structured reasoning prompts, process rewards and search, alongside interactive and multi-agent architectures. Its distinctive emphasis is empirical training behavior under multi-turn environmental feedback rather than a wholly new policy-gradient estimator.
6 Conclusions and Limitations
RAGEN makes these learning dynamics inspectable and provides a practical stabilization recipe for small agent tasks. Scope remains limited: modest models and task spaces, short rollout horizons, no multimodal experiments, no systematic replay-buffer study, and limited reported uncertainty across training seeds. Larger-scale deployment and a reliable reward for reasoning quality remain open. The successful SFT comparison in Appendix G is an important counterpoint to claims about reward-only self-improvement.
Acknowledgements
The authors acknowledge DeepSeek, veRL, TinyZero and many contributors for infrastructure, conceptual inspiration and discussions.
Appendix A Background of Reinforcement Learning
The appendix reviews expected-reward optimization, old/new policy probability ratios, clipped PPO objectives and generalized advantage estimation Schulman et al. (2018). GAE uses discounted temporal-difference residuals with γ and λ controlling the estimator. GRPO instead normalizes sampled returns within a group, avoiding a learned value model. The original equations, including KL-regularized background objectives, are retained; these should be distinguished from StarPO-S’s KL-free configuration.
Equations 8–14
| (8) |
| (9) |
| (10) |
| (11) |
| (12) |
| (13) |
| (14) |
Appendix B Extended Related Work
Extended related work covers actor-critic optimization, process rewards, Monte Carlo tree search, STaR-style rationale bootstrapping, and simplified RL recipes. Agent systems span reasoning/action loops, planning, multi-agent cooperation, web and GUI interaction, coding, games and embodied tasks. This literature supplies context; RAGEN’s own experiments do not evaluate all those application domains.
Appendix C Detailed Experimental Settings
The appendix details task rewards, implementation settings and the meaning of each diagnostic.
C.1 Environments and Tasks
The symbolic tasks expose different feedback structures, while WebShop supplies language-rich context. The paper describes symbolic tasks as decoupled from pretraining, but the deliberately semantic Bandit arm names explicitly exploit language priors; that claim should be restricted to the more abstract grid settings.
Bi-Arm Bandits.
The safe Bandit arm yields 0.15 deterministically; the risky arm yields Bernoulli(0.25), with higher expected return (0.25) but less frequent positive payoff. “Success” means choosing the higher-expected-value arm, not receiving a positive reward on a particular draw. Semantic name pairs and their reversals test transfer of learned associations.
Sokoban rewards boxes on targets, penalizes moving them off, adds +10 for completion and charges −0.1 per action. FrozenLake executes the intended direction with probability 1/3 and a perpendicular direction with total probability 2/3; only reaching the goal gives +1. WebShop uses search, click and purchase actions to satisfy a product request. Their reward scales and stochasticity differ substantially.
C.2 Training and Evaluation Settings
Training uses H100/A100 GPUs, veRL, vLLM, FSDP, Ray and XFORMERS. Defaults are eight prompt groups, 16 trajectories each, update batch 32 and per-GPU minibatch four. Agents receive at most five actions per turn and ten per episode; evaluation stops after five turns or ten actions. WebShop runs for 100 iterations rather than 200 because long contexts are expensive.
Adam uses (0.9, 0.999) moments. Entropy regularization is 0.001; vanilla StarPO has a 0.001 KL penalty with the k1 estimator, while the stabilized configuration removes that training penalty and monitors KL afterward. The generic appendix statement naming 0.5B should not override the main experiment’s 3B WebShop model. Prompt templates separately allow “up to 10 actions,” exposing a prompt-versus-runtime budget distinction.
C.3 Evaluation Metrics
Success measures task completion or the correct Bandit arm. Entropy is average token uncertainty; in-group standard deviation measures return dispersion; output length measures verbosity; gradient norm diagnoses update magnitude. The appendix’s wording about where all non-success metrics are computed is ambiguous, and plots explicitly label some as training signals. None directly measures the factual validity or causal usefulness of reasoning.
Appendix D Results on Larger Models and Various Optimization Algorithms
The additional comparisons separate trained 3B/7B scaling, zero-shot large models, optimization ablations and turn-aware updates.
Scaling Effects.
Larger models help most on Bandit and WebShop in the reported curves; grid tasks gain less consistently. WebShop’s 7B run exceeds memory on four H100s, so its trained comparison uses 3B. Attributing the pattern to language priors is an interpretation, not a controlled pretraining-overlap study.
Frontier Model Performance.
Zero-shot GPT-4o scores 27.73% on Sokoban and 26.56% on FrozenLake; Qwen2.5-72B scores 19.53% and 23.83%. The trained 0.5B model scores 20.70% and 21.48%. Thus it is comparable in scale of performance but below GPT-4o on both tasks, and it receives task-specific RL data that the zero-shot baselines do not. The parameter-count comparison is known for Qwen, not for undisclosed GPT-4o. This is not evidence of general capability parity.
| Model | Sokoban (%) | FrozenLake (%) | Response Length | Effective Action (%) |
|---|---|---|---|---|
| GPT-4o | 27.73 | 26.56 | 190.3 / 132.1 | 70.0 / 88.7 |
| Qwen2.5-72B | 19.53 | 23.83 | 179.4 / 107.3 | 67.8 / 87.3 |
| 0.5B (ours) | 20.70 | 21.48 | 89.5 / 110.6 | 79.2 / 80.7 |
Gradient Shaping.
Removing KL loosens the anchor to the reference policy. Asymmetric clipping uses lower ε = 0.2 and upper ε = 0.28, allowing a larger increase than decrease in action probability before clipping. Both improve some peaks or delay decline, but Figure 11 still shows late collapse. These are practical ablations rather than stability guarantees.
Response Masking and Bi-Level GAE.
Response masking and bi-level GAE improve some Sokoban curves on 0.5B, suggesting that turn-aware credit assignment and selective token updates matter. The paper refers to the cited method for implementation details; it does not fully derive either variant here.
Appendix E When Does Uncertainty-Based Filtering Help?
Filtering helps most when many prompt groups have uniformly good or uniformly bad returns and thus little useful within-group contrast. WebShop already has more return diversity, so filtering may add less. Figure 13 supports a task-dependent association, but environmental randomness also contributes to variance. Selecting high-variance groups can change the training distribution and is not guaranteed to favor the most learnable or important instances.
Appendix F Case Study: The Emergence of Echo Trap with RL
The complete Bandit examples contrast varied initial hypotheses with repetitive statements at step 150. They illustrate one form of template concentration, without measuring how often it occurs or whether every repeated rationale is incorrect. Figure 14 preserves the corresponding response-length curves, including the longer reversed-Bandit traces.
| Step | Model Output |
|---|---|
| 0 | Dragon is often associated with patience, strength, and reliability, while Phoenix is associated with speed, agility, and creativity. |
| 0 | The symbolic meaning of Dragon and Phoenix’s names suggests that they may have different reward distributions... Therefore, I think Phoenix is more likely to give higher rewards on average. |
| 150 | My initial intention is to pick arm Dragon. My first action is to go for arm Dragon. |
| 150 | My initial intention is to pick arm Dragon, so my first action is to go for arm Dragon. |
| 150 | My initial intention is to choose Arm Dragon. After analyzing both arms, my intention was to choose Arm Dragon. |
Appendix G Comparing Agent RL with Supervised Fine-Tuning
Supervised fine-tuning uses BFS-generated trajectories: 1,000 training and 100 test examples, search depth up to 100, LoRA rank 64/α 32, learning rate 10⁻⁴ and batch 128. The model predicts the next action in a conversational format. Reported SFT success is 74.6% on Sokoban and 23% on FrozenLake, versus 20.3% and 21.8% for StarPO-S. This strong Sokoban result limits claims about reward-only training’s effectiveness, although demonstrations provide supervision unavailable to the RL-only comparison and the appendix does not establish an equal-data/equal-compute study.
Appendix H Efficient Training with Low-Rank Adaptation (LoRA)
A parameter-efficient RL variant is evaluated as an alternative to full-parameter updates.
Motivation.
LoRA Hu et al. (2021) uses rank 64 and α 64 on all linear projections, with actor and critic learning rates increased tenfold. Over an 80-minute comparison, reported device-memory occupancy falls from about 48% to 23%, GPU utilization from 34% to 14%, and relative power draw from 22% to 12%. These occupancy/utilization percentages are not direct FLOP counts or total energy-to-solution measurements.
The text claims performance parity but prints approximately “0.2%” success, inconsistent with the roughly 0.2 fraction/20% reported elsewhere. The intended unit is not resolved. Hardware-normalized percentages, changed learning rates and the absence of uncertainty estimates limit conclusions about broadly equal quality or universal resource savings.
Appendix I PPO Failure Mode in Frozen Lake
PPO’s earlier FrozenLake collapse is attributed to difficult value estimation under stochastic transitions; GRPO avoids learning a critic. This is a hypothesis supported by differing learning curves, not an isolated critic-error experiment or evidence that value methods generally fail in stochastic tasks.
Appendix J Prompt Templates
All model templates, grid vocabularies and action instructions are retained as original PDF panels with expandable source text.
J.1 Bi-Arm Bandit Environment Prompts
The Bandit template asks the agent to infer reward tendencies from symbolic arm names, choose one arm and respond in answer tags, with a 200-word/token limit. Its assistant prefix opens a think block even though the system wording primarily specifies answers. The original literal delimiters and name placeholders are preserved.
Original source text
Model Templates
<|im_start|>[system]:
{prompt}
You’re a helpful assistant. You always respond by giving your answer in <answer>...</answer>. Max response length: 200 words (tokens).
<|im_end|>
<|im_start|>[user]:
{prompt}
You are playing a bandit game. Goal: Maximize your total reward by choosing which arm to pull.
Game Rules:
1. There are 2 arms, named name_a and name_b
2. Each arm has its own reward distribution, related to their names.
3. Analyze the symbolic meaning of each arm’s name to guess how their reward distribution might behave.
4. Based on the symbolic meaning of their names, which arm do you think is more likely to give higher rewards on average? Choose between name_a and name_b, and output like <answer> name_a </answer> or <answer> name_b </answer>.
<|im_end|>
<|im_start|>assistant
<think>
J.2 Sokoban Environment Prompts
The Sokoban template explains pushing but not pulling, lists symbols for walls, floor, targets, boxes and players, and separates action sequences with ||. The instruction box allows ten actions, while the experimental runtime typically caps five per turn. Original wording and symbols remain unchanged.
Original source text
Model Templates
<|im_start|>system
{prompt}
You’re a helpful assistant. You always respond by first wrapping your thoughts in <think>...</think>, then giving your answer in <answer>...</answer>. Max response length: 200 words (tokens).
<|im_end|>
<|im_start|>user
{prompt}
You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets. When you are right next to a box, you can push it by moving in the same direction. You cannot push a box through a wall, and you cannot pull a box. The answer should be a sequence of actions, like <answer>Right || Right || Up</answer>
<|im_end|>
<|im_start|>assistant
<think>
Original source text
Grid Representation
The meaning of each symbol in the state is:
#: wall, _: empty, O: target, : box on target, X: box, P: player, S: player on target
Original source text
Instruction Template
Your available actions are:
Up, Down, Left, Right
You can make up to 10 actions, separated by the action separator " || "
J.3 FrozenLake Environment Prompts
FrozenLake’s template warns about slips, defines the grid symbols and uses the same structured thought/answer format. It retains the original awkward wording “Forbid the hole” and the stated action budget. Figure 15’s full three-turn counterexample is preserved alongside the prompt material.
Original source text
Model Templates
<|im_start|>system
{prompt}
You’re a helpful assistant. You always respond by first wrapping your thoughts in <think>...</think>, then giving your answer in <answer>...</answer>. Max response length: 200 words (tokens).
<|im_end|>
<|im_start|>user
{prompt}
You are solving the FrozenLake puzzle. Forbid the whole and go to the target. You may move to the unintended direction due to the slippery ice. Example answer format: <think>To forbid the hole and go to the target, I should go left then go up.</think><answer>Left || Up</answer>
<|im_end|>
<|im_start|>assistant
<think>
Original source text
Grid Representation
The meaning of each symbol in the state is:
P: player, _: empty, O: hole, G: goal, X: player in hole, : player on goal
Original source text
Instruction Template
Your available actions are:
Left, Down, Right, Up
You can make up to 10 actions, separated by the action separator " || "
Original source text
Sokoban – Spurious Reasoning Case (3 Turns)
Turn 1:
State:
###### #___O# #__X_# ###P## ###### ######
Output: <think>I’ve reached the target on the right. I need to move right and push the target.</think> <answer>Right || Right || Up</answer> Reward: -0.3
Turn 2:
State:
###### #__XO# #__P_# ###_## ###### ######
Output: <think>Now I need to move left and push the box on the left.</think> <answer>Left || Left || Up</answer> Reward: -0.3
Turn 3:
State:
###### #P_XO# #____# ###_## ###### ######
Output: <think>Now I need to move right and push the box on the right.</think> <answer>Right || Right || Up</answer> Reward: +1.0
Appendix K Generalization Evaluation Environments
New-vocabulary Sokoban changes symbolic cell labels while retaining spatial structure. LargerSokoban expands the board from 6 × 6 with one box to 8 × 8 with two, increasing planning complexity. FrozenLake transfer changes the task dynamics. These held-out settings test distinct shifts; none alone establishes broad out-of-distribution agent competence.
Appendix L Case Study: Spurious Reward from Incorrect Reasoning
The three-turn Sokoban example earns a final positive reward despite statements such as pushing a target and inconsistent directions. It demonstrates that successful action outcomes need not validate the associated explanation. The example’s displayed final reward is +1, while the environment description gives +10 completion reward; the reporting/normalization difference is not explained. Finer reward supervision is motivated but not implemented as a validated fix here.
Appendix M Extended Case Studies
All eight model/task cases remain intact: two Bandit, two Sokoban, two FrozenLake and two WebShop panels. The larger Bandit example is more elaborate but still speculates from names rather than estimating empirical reward distributions. Sokoban examples contain heuristic or incorrect plans. The 7B FrozenLake case reaches the goal through a favorable slip despite an unsuitable command, illustrating noisy credit assignment.
WebShop contrasts a 0.5B loop with a successful 3B sequence; the surrounding appendix prose incorrectly calls all larger cases 7B, whereas the actual panel and caption specify 3B. Those WebShop examples reach turns seven or nine, exceeding the nominal five-turn evaluation cap, so they should not be treated as literal demonstrations of the stated default evaluation protocol. Case selection is qualitative, and longer rationales are not evidence of faithful internal reasoning.
Original source text
0.5B Bandit
Prompt: Choose the better arm based on symbolic meaning: Dragon vs. Phoenix.
Model Output:
<think>Dragon has a higher symbolic meaning compared to Phoenix, suggesting it might have a more significant impact on one’s character or fate.</think
<answer>Dragon</answer>
Reward: +1.0
Original source text
7B Bandit
Prompt: Choose the better arm based on symbolic meaning: Dragon vs. Phoenix.
Model Output:
<think>Based on the symbolic meanings of the names "Dragon" and "Phoenix," we can make educated guesses about their reward distributions. The name "Dragon" often symbolizes something powerful, majestic, and long-lived, which could imply a higher average reward and a more stable reward distribution. On the other hand, the name "Phoenix" is often associated with rebirth, transformation, and resilience after destruction or adversity. This could suggest a higher variability in rewards, with the potential for very high rewards but also the risk of lower rewards.
Given these interpretations, the Dragon might be more likely to give higher rewards on average, as its name suggests a more powerful and stable source of rewards. However, it’s important to note that this is based on symbolic meanings and not on empirical data. In the bandit game, we will need to gather evidence through pulling the arms to make a more informed decision.</think>
<answer>Dragon</answer>
Reward: +1.0
Original source text
0.5B Sokoban – Multi-Turn Rollout
Turn 1:
State:
###### #__### #O#### #__XP# #____# ######
Output:
<think>Pushing all boxes to targets.</think>
<answer>Left || Left || Down</answer>
Reward: -0.3
Turn 2:
State:
###### #__### #O#### #X___# #_P__# ######
Output:
<think>Clearing a wall to make progress.</think>
<answer>Right || Down || Left</answer>
Reward: -0.3
Turn 3:
State:
###### #__### #O#### #X___# #_P__# ######
Output:
<think>Pushing another box.</think>
<answer>Left || Down || Up</answer>
Reward: -0.3
Original source text
7B Sokoban – Multi-Turn Rollout
Turn 1:
State:
###### ##_P_# ##_X_# ##__O# ###__# ######
Output:
<think>First, move the box to target.</think>
<answer>Up || Right || Up</answer>
Reward: -0.3
Turn 2:
State:
###### ##__P# ##_X_# ##__O# ###__# ######
Output:
<think>Now move second box to target.</think>
<answer>Left || Down || Left || Down</answer>
Reward: -0.4
Turn 3:
State:
###### ##___# ##___# ##PXO# ###__# ######
Output:
<think>Finally, move last box to target.</think>
<answer>Right || Up || Right || Up</answer>
Reward: +1.0
Original source text
0.5B FrozenLake – Multi-Turn Rollout
State:
_P_G ____ O_OO O___
Model Output (repeated each turn):
<think>Forbid the hole and go to the target, I should go left then go up.</think>
<answer>Left || Up</answer>
Reward: 0.0 (repeated for 4 turns)
Final Turn (Turn 5):
State:
P__G ____ O_OO O___
Model Output:
<think>Forbid the hole and go to the target, I should go left then go up.</think>
<answer>Left || Up</answer>
Reward: 0.0
Original source text
7B FrozenLake – Multi-Turn Rollout
Turn 1:
State:
____ _P__ OGO_ ____
Model Output:
<think>Fill holes, then go target.</think>
<answer>Left || Up</answer>
Reward: +1.0 (due to slipping into goal)
Note: although the action was incorrect, the agent reached the goal due to stochastic slipping.
Original source text
0.5B WebShop – Multi-Turn Rollout
Turn 1:
State:
Instruction: Find me super soft decorative pillows for living room with color: beige latte, and size: 20”x20”, and price lower than 30.00 dollars.We must buy a product within 10 actions. It doesn’t have to match perfectly with description.
Search term should not include details like size, …
Model Output:
<think>To find a super soft decorative pillows for the living room with a beige latte color and a size of 20”x20”, I will start with a search for pillows with a beige color and a size of 20”x20”.</think>
<answer>search[bisque pillow with beige color 20”x20’]</answer>
Reward: 0.0
Turn 2:
State:
Instruction: [SEP] Find me super soft decorative pillows for living room with color: beige latte, and size: 20”x20”, and price lower than 30.00 dollars [SEP] Back to Search [SEP] Page 1 (Total results: 50) [SEP] Next > [SEP] B08L2ZDWN2 [SEP] PAVILIA Decorative Sherpa Throw Pillow Covers, … click[b08qvdnjc7], click[b07p24zmxk], click[b07n33bpq6], click[b09nrf2qgd],
Model Output:
<think>Searching for pillows with beige color and 20”x20” size, and price lower than $30.</think>
<answer>click[b08l2zdwn2]</answer>
Reward: 0.0
Turn 3:
State:
Instruction: [SEP] Find me super soft decorative pillows for living room with color: beige latte, and size: 20”x20”, and price lower than 30.00 dollars [SEP] Back to Search [SEP] < Prev [SEP] color [SEP] beige latte [SEP] black [SEP] blush pink [SEP] dusty blue [SEP] emerald green [SEP] … grey], click[mustard yellow], click[teal blue], click[white], click[18”x18”],
Model Output:
<think>Searching for pillows with beige color and 20”x20” size, and price lower than $30.</think>
<answer>click[beige latte]</answer>
Reward: 0.0
…
Turn 9:
State:
Instruction: [SEP] Find me super soft decorative pillows for living room with color: beige latte, and size: 20”x20”, and price lower than 30.00 dollars [SEP] Back to Search [SEP] < Prev [SEP] color [SEP] beige latte [SEP] black [SEP] blush pink [SEP] dusty blue [SEP] emerald green [SEP] … grey], click[mustard yellow], click[teal blue], click[white], click[18”x18”],
Model Output:
<think>Searching for pillows with beige color and 20”x20” size, and price lower than $30.</think>
<answer>click[black]</answer>
Reward: 0.0
Original source text
3B WebShop – Multi-Turn Rollout
Turn 1:
State:
Instruction: Find me wall mounted, white item, easy clean tv stands for living room with color: white+brown, and price lower than 130.00 dollars.We must buy a product within 10 actions. It doesn’t have to match perfectly with description.
Search term should not include details like size, …
Model Output:
<think>Let’s start by searching for the items within the given criteria and narrow down the options. The search term will be ’white+brown tv stand living room’ to find wall-mounted, easy-clean TV stands with the specified colors and in the living room category.</think>
<answer>search[white+brown tv stand living room]</answer>
Reward: 0.0
Turn 2:
State:
Instruction: [SEP] Find me wall mounted, white item, easy clean tv stands for living room with color: white+brown, and price lower than 130.00 dollars [SEP] Back to Search [SEP] Page 1 (Total results: 50) [SEP] Next > [SEP] B09F6SCW47 [SEP] hongxinq Floating TV Stand with Storage Shelves, … click[b079n3vlrj], click[b0924pklq4], click[b09lhm4wqs], click[b08k7ldm7q],
Model Output:
<think>After reviewing the search results, the first item that seems to match the criteria is the ’Floating TV Stand with Storage Shelves, Chipboard Wall Mounted Hanging Television Cabinet’ from ’B09F6SCW47’. It is white with brown accents, fits the living room and bedroom category, and is priced at $90.79, which is lower than the $130.00 limit. It’s also the closest match in terms of color and style. Since it’s the first matching option, I will prioritize clicking on it to move forward with the purchase. However, I’ll make sure to click [size] and then [color] if they are available and required to finalize the purchase.</think>
<answer>click[b09f6scw47]</answer>
Reward: 0.0
…
Turn 7:
State:
Instruction: [SEP] Find me wall mounted, white item, easy clean tv stands for living room with color: white+brown, and price lower than 130.00 dollars [SEP] Back to Search [SEP] < Prev [SEP] color [SEP] brown [SEP] concrete gray [SEP] grey [SEP] white [SEP] white+brown [SEP] hongxinq … click[buy now], click[brown], click[concrete gray], click[grey], click[white],
Model Output:
<think>With the ’Floating TV Stand with Storage Shelves, Chipboard Wall Mounted Hanging Television Cabinet’ from ’B09F6SCW47’ confirmed to have the correct white and brown color options and the required size of 12Inx11.8Inx35.4In, I will now proceed to click the ’buy now’ button to finalize the purchase. Given that we have 3 actions left and the product matches the criteria closely, I will click on ’buy now’ to complete the transaction. This ensures that we stay within the action limit and make a purchase that closely aligns with the specified requirements.</think>
<answer>click[buy now]</answer>
Reward: 1.0