Literature
Display
← All papers

VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents

Kangrui Wang · Pingyue Zhang · Zihan Wang · Yaning Gao · Linjie Li · Qineng Wang · Hanyang Chen · Chi Wan · Yiping Lu · Zhengyuan Yang · Lijuan Wang · Ranjay Krishna · Jiajun Wu · Li Fei-Fei · Yejin Choi · Manling Li

· NeurIPS 2025 ↗

AI agents19 min text · CondensedOriginal paper ↗
Contents

At a glance

VAGEN trains vision-language agents to describe the current visual state, reason about an action and predict its consequence before executing it. The full method rewards those descriptions against simulator state and passes credit first across turns, then across tokens. Its reported overall score rises from 0.21 for the untrained 3B backbone to 0.82, compared with 0.76 for the base training method.

The overall score averages five task-level results, including image-similarity scores; it is not an 82% universal success rate. The ratio 0.82/0.21 is about 3.9, or a 290% relative increase. The abstract’s “nearly 3× improvement” should be interpreted with that distinction. This edition follows the complete 55-page v1, including all figures, tables, formulas, algorithms, prompts, examples and references.

1 Introduction

Visual agents must infer useful state from images and maintain context as actions change the scene. VAGEN calls their explicitly generated descriptions and predictions an internal world model [1]. This is a reasoning format inside a language policy, not a separately trained visual simulator or a calibrated probability distribution over hidden states.

Three questions organize the experiments: which reasoning components help, which representation best expresses state, and how rewards and credit assignment should supervise it. The strongest controlled comparisons use the same Qwen2.5-VL-3B backbone. Proprietary-model comparisons use different architectures, training histories and inference conditions, so they show benchmark performance rather than an isolated model-size advantage.

Complete original PDF block S1.F1
Figure 1: Overview of the VAGEN Framework. Within a POMDP, VLM Agents build internal world models through explicit visual state reasoning over the StateEstimation P⁡(s^t|ot)P(\hat{s}_{t}|o_{t}) and TransitionModeling P⁡(s^t+1|ot,s^t,a^t)P(\hat{s}_{t+1}|o_{t},\hat{s}_{t},\hat{a}_{t}), where s^t\hat{s}_{t} denotes the internal belief of the state, a^t\hat{a}_{t} represents the internal belief of the action, and atea^{\text{e}}_{t} is the parsed executable action, as detailed in Table 1.

2 Build Internal World Models via Visual State Reasoning in Multi-Turn RL

The base framework collects complete visual interaction trajectories and optimizes generated action and reasoning tokens while masking observations from the policy loss.

2.1 Problem Formulation

The POMDP separates true simulator state, the agent’s visual observation, emitted text, parsed executable action and reward. An observation may omit important information, particularly in first-person navigation. The policy seeks discounted return from its observation/action history; the state descriptions are generated approximations rather than privileged state fed directly to every agent.

Table 1 distinguishes the reasoning action representation from the executable action. In manipulation, however, prompts also supply object positions, so the tested input is not uniformly image-only across tasks.

Table 1: Summary of important notations used in the VAGEN framework.

Notation Symbol
Description
Observation oto_{t}
The raw visual observation at turn tt.
State sts_{t}
Ground-truth environment state at turn tt.
State (Belief) s^t\hat{s}_{t}
VLM’s belief of state sts_{t} at turn tt.
Action ata_{t}
The action tokens at tt, at=⟨zt,ate⟩a_{t}=\langle{z}_{t},a^{\text{e}}_{t}\rangle, augmented with reasoning tokens zt{z}_{t}.
Action (Belief) a^t\hat{a}_{t}
VLM’s belief of action ata_{t} at turn tt.
Action (Executable) atea^{\text{e}}_{t}
The parsed, executable component of ata_{t}.
Reward rtr_{t}
Scalar reward at turn tt, rt=R⁡(st,at)r_{t}=R(s_{t},a_{t}) where RR is the reward function.
Trajectory τ\tau
Rollout τ=(s0,o0,a0,r0,⋯,sT,oT)\tau=(s_{0},o_{0},a_{0},r_{0},\cdots,s_{T},o_{T}).
Tokenized Trajectory τ¯\bar{\tau}
Token sequence for VLM training, concatenating encoded observations and actions.
Agent Policy πθ\pi_{\theta}
Policy parameterized by a VLM with parameters θ\theta.
Critic / Value Func. VϕV_{\phi}
State-value function parameterized by ϕ\phi.
Token Index i,ji,j
τ¯i\bar{\tau}_{i}: the ii-th token. τ¯i:j\bar{\tau}_{i:j}: tokens ii–jj. τ¯<i\bar{\tau}_{<i}: prefix up to i−1i-1. τ¯t,i\bar{\tau}_{t,i}: the ii-th token of the tt-th turn.
Trajectory Reward R⁡(τ)R(\tau)
Sum of rewards over trajectory, ∑tR⁡(st,at)\sum_{t}R(s_{t},a_{t}).
Advantage Estimate AA
AiA_{i}: advantage for token ii; AtturnA_{t}^{\text{turn}}: advantage for turn tt; At,itokenA_{t,i}^{\text{token}}: advantage for token ii in turn tt.
Temporal-Difference Error δ\delta
δi\delta_{i}: TD-error for token ii; δtturn\delta_{t}^{\text{turn}}: TD-error for turn tt; δt,itoken\delta_{t,i}^{\text{token}}: TD-error for token ii in turn tt.
Discount Factor γ\gamma
γ∈[0,1]\gamma\in[0,1]; γtoken\gamma_{\text{token}}: within-turn discount; γturn\gamma_{\text{turn}}: across-turn discount.

2.2 Visual State Reasoning as an Internal World Model

An actor–critic model is trained over whole interaction trajectories. Explicit state descriptions and next-state predictions are inserted into the actor’s output, and environment observations become context for subsequent turns.

Reasoning Trajectory Rollout for Multi-Turn VLM Agents.

At each turn the VLM emits reasoning plus an answer containing executable actions. A parser sends the answer to the environment, then appends the resulting observation and reward to the trajectory. Reasoning tokens influence later decisions as context, but only the parsed actions change the environment.

Equation
at=⟨zt,ate⟩.a_{t}=\langle{z}_{t},a^{\text{e}}_{t}\rangle.

Visual State Reasoning as World Modeling.

NoThink omits substantive reasoning; FreeThink allows unrestricted reasoning; StateEstimation adds an observation description; TransitionModeling adds a prediction; WorldModeling combines both around an action rationale. Output-format rewards encourage the corresponding tags.

A next-state prediction is made before the action outcome is observed. At the next turn the agent generates a fresh state description from the new observation; it does not simply reuse the previous prediction as a verified state. The method therefore encourages explicit predictive reasoning without enforcing a persistent, probabilistically consistent belief update.

The main text shows empty think tags for NoThink, whereas Table 6 shows only answer tags. Both source forms are retained rather than silently standardized.

Equations
zt=∅{z}_{t}=\emptyset
zt≠∅, and ​zt​ is natural language tokens. {z}_{t}\neq\emptyset,\text{ and }{z}_{t}\text{ is natural language tokens. }
zt=⟨s^t,a^t⟩, learning to approximate ​s^t→st{z}_{t}=\langle\hat{s}_{t},\hat{a}_{t}\rangle,\text{ learning to approximate }\hat{s}_{t}\rightarrow s_{t}
zt=⟨a^t,s^t+1⟩, learning to approximate ​s^t+1→st+1{z}_{t}=\langle\hat{a}_{t},\hat{s}_{t+1}\rangle,\text{ learning to approximate }\hat{s}_{t+1}\rightarrow s_{t+1}
zt=⟨s^t,a^t,s^t+1⟩, learning to approximate ​s^t→st,s^t+1→st+1{z}_{t}=\langle\hat{s}_{t},\hat{a}_{t},\hat{s}_{t+1}\rangle,\text{ learning to approximate }\hat{s}_{t}\rightarrow s_{t},\hat{s}_{t+1}\rightarrow s_{t+1}

Policy Optimization.

PPO [26] updates the actor with clipped likelihood ratios for generated tokens. A loss mask excludes observation tokens, which were supplied by the environment rather than sampled from the policy. Teacher-forced training evaluates each generated token using its causal trajectory prefix; the complete training sequence does not imply access to future observations when acting.

Equation
JPPO​(θ)=1∑iMiloss​∑iMiloss⋅min⁡(ui​(θ)​Ai,clip​(ui​(θ),1−ε,1+ε)​Ai),J^{\text{PPO}}(\theta)=\frac{1}{\sum_{i}M_{i}^{\text{loss}}}\sum_{i}M_{i}^{\text{loss}}\cdot\min\left(u_{i}(\theta)A_{i},\text{clip}(u_{i}(\theta),1-\varepsilon,1+\varepsilon)A_{i}\right),

Advantage and Value Estimation.

The critic regresses to advantage-plus-value targets. In VAGEN-Base, intermediate generated tokens receive a KL penalty relative to a frozen reference, while the total trajectory reward is placed at the final action token. Token-level GAE [22] propagates that signal backward, skipping observation tokens.

This preserves fine token distinctions but can make a turn’s contribution difficult to identify over long token sequences. The later method changes reward placement and advantage propagation, rather than merely changing the reasoning prompt.

Equations
JCritic​(ϕ)=1∑iMiloss​∑iMiloss⋅(Vϕ​(τ¯<i+1)−Yi)2J^{\text{Critic}}(\phi)=\frac{1}{\sum_{i}M_{i}^{\text{loss}}}\sum_{i}M_{i}^{\text{loss}}\cdot\left(V_{\phi}(\bar{\tau}_{<i+1})-Y_{i}\right)^{2}
δi=ri+γ​Vϕ​(τ¯<j)−Vϕ​(τ¯<i)\delta_{i}=r_{i}+\gamma V_{\phi}(\bar{\tau}_{<j})-V_{\phi}(\bar{\tau}_{<i})
Ai=δi+γ​λ​AjA_{i}=\delta_{i}+\gamma\lambda A_{j}
Yi=Ai+Vϕ​(τ¯<i).Y_{i}=A_{i}+V_{\phi}(\bar{\tau}_{<i}).

2.3 Reward Design in Different Environments and Tasks for VLM Agents

Five task families cover one-box Sokoban, deterministic FrozenLake, first-person object navigation, manipulation through predefined pick/place/push primitives, and SVG reconstruction. The last task iteratively edits an image description in code and has no external world dynamics.

The environment settings are deliberately small: two to four interaction turns, with several primitive actions allowed per turn. These experiments test multi-turn visual decisions, but do not establish unrestricted long-horizon autonomy or low-level robot control.

Complete original PDF block S2.F2.sf1
(a) Sokoban
Complete original PDF block S2.F2.sf2
(b) FrozenLake
Complete original PDF block S2.F2.sf3
(c) Navigation
Complete original PDF block S2.F2.sf4
(d) PrimitiveSkill
Complete original PDF block S2.F2.sf5
(e) SVG Reconstruction
Figure 2: Examples of visual states from five environments used in our study.

Reward Strategy and Metrics.

Four families are evaluated by success rate; SVG uses DreamSim and DINO similarity. The overall score averages task-family summaries, including the mean of the two SVG similarities, so its components have different meanings.

Although the main description emphasizes sparse binary task rewards, Appendix A also lists unfinished-step penalties, box-placement bonuses and stage rewards for manipulation. SVG training is effectively DreamSim-dominated: Table 21 weights it by 5 and DINO by only 0.0001. The full reward tables are preserved.

Table 2: Reasoning on internal world models given visual states, including both StateEstimation and TransitionModeling, can significantly improve the RL performance. Test success rates are reported for Sokoban, FrozenLak, Navigation and PrimitiveSkill. Test Dino and DreamSim scores are reported for SVG Reconstruction task. Best performance is in bold. We gray out results without controlled variables (i.e., same size and architecture), shown for reference only. VAGEN-Full with world modeling enhancement on reward shaping and credit assignment will be introduced in Sec 4 so we also gray it out, shown as a reference.

Model/Method Sokoban FrozenLake Navigation PrimitiveSkill SVG Overall
Base Common Average Place Stack Drawer Align Average Dino DreamSim Average
Open-Source Models
Qwen2.5-VL-72B [34] 0.18 0.44 0.72 0.75 0.73 1.00 0.50 0.00 1.00 0.44 0.89 0.62 0.76 0.51
Qwen2.5-VL-7B [34] 0.13 0.14 0.28 0.39 0.34 0.00 0.00 0.00 0.75 0.19 0.83 0.28 0.55 0.27
Qwen2.5-VL-3B [34] 0.14 0.14 0.22 0.27 0.24 0.00 0.00 0.00 0.00 0.00 0.80 0.27 0.54 0.21
VLM-R1-3B [24] 0.13 0.13 0.31 0.34 0.33 0.00 0.00 0.00 0.00 0.00 0.81 0.28 0.55 0.23
VAGEN: Multi-Turn RL with World Model Reasoning for Visual States (Backbone: Qwen2.5-VL-3B)
FreeThink 0.57 0.68 0.67 0.67 0.67 1.00 0.63 0.00 1.00 0.66 0.91 0.64 0.78 0.67
NoThink 0.57 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.89 0.62 0.76 0.28
StateEstimation 0.56 0.68 0.78 0.69 0.74 0.00 0.00 0.00 0.00 0.00 0.92 0.64 0.78 0.56
TransitionModeling 0.41 0.76 0.67 0.59 0.62 1.00 0.63 0.63 1.00 0.82 0.89 0.64 0.77 0.68
WorldModeling 0.61 0.71 0.78 0.80 0.79 1.00 0.88 0.88 0.88 0.91 0.90 0.65 0.78 0.76
VAGEN-Full 0.79 0.74 0.80 0.81 0.81 1.00 0.88 1.00 1.00 0.97 0.91 0.67 0.79 0.82
RL Baselines with World Model Reasoning Strategy (Backbone: Qwen2.5-VL-3B)
Vanilla-PPO 0.18 0.21 0.32 0.25 0.29 0.00 0.00 0.00 0.00 0.00 0.83 0.44 0.64 0.26
GRPO w/ Mask 0.20 0.57 0.88 0.81 0.85 0.00 0.00 0.00 1.00 0.25 0.92 0.66 0.79 0.54
Turn-PPO w/ Mask 0.38 0.70 0.78 0.84 0.81 0.00 0.00 0.00 1.00 0.25 0.89 0.64 0.77 0.55
Proprietary Models
GPT-5 [35] 0.70 0.77 0.75 0.81 0.78 1.00 0.63 0.00 1.00 0.66 0.95 0.75 0.85 0.75
o3 [36] 0.60 0.78 0.81 0.75 0.78 1.00 0.63 0.00 1.00 0.66 0.92 0.71 0.82 0.73
o4-mini [36] 0.44 0.82 0.75 0.75 0.75 1.00 0.50 0.00 0.75 0.56 0.90 0.66 0.78 0.67
GPT-4o [37] 0.43 0.54 0.75 0.69 0.72 0.50 0.63 0.00 0.88 0.50 0.91 0.69 0.80 0.60
Gemini 2.5 Pro22 2 Due to Gemini 2.5 Pro’s safety policy, a subset of evaluation responses for PrimitiveSkill and Navigation was blocked; reported results are based solely on the unblocked test cases.[38] 0.58 0.78 0.63 0.63 0.63 0.63 0.63 0.00 0.75 0.50 0.93 0.78 0.86 0.67
Gemini 2.0 [39] 0.28 0.61 0.50 0.63 0.56 0.75 0.13 0.00 0.25 0.28 0.93 0.74 0.84 0.51
Claude 4.5 Sonnet [40] 0.31 0.80 0.67 0.67 0.67 0.63 0.50 0.00 1.00 0.53 0.95 0.81 0.88 0.64
Claude 3.7 Sonnet [41] 0.25 0.69 0.48 0.47 0.47 0.63 0.13 0.00 1.00 0.44 0.94 0.77 0.85 0.54

2.4 What Can We Reason About Visual States?

The reasoning study compares formats under base training and then compares several policy-optimization baselines. These are separate questions from the effect of the full reward/credit-assignment system.

Off-the-Shelf VLMs struggle to solve multi-turn agentic tasks.

Table 2 lists twelve off-the-shelf models, despite the prose and Appendix Table 22 referring to seven. GPT-5 scores 0.75 overall; all listed untrained models fail the Drawer subtask. This does not mean they fail all manipulation tasks: several succeed on Place and Align.

The cross-model table has qualifications. Gemini 2.5 Pro excludes blocked responses from its evaluated cases. Claude 4.5 is 0.64 in Table 2, rather than the abstract’s 0.62. Qwen2.5-VL-72B’s four manipulation cells (1.00, 0.50, 0.00, 1.00) average 0.625, but its printed average is 0.44; that printed average is consistent with the listed 0.51 overall. These source values are retained and the inconsistency is not resolved by inventing corrected benchmark results.

World Modeling can improve visual reasoning via multi-turn RL.

With Qwen2.5-VL-3B, base WorldModeling scores 0.76 overall, FreeThink 0.67, TransitionModeling 0.68, StateEstimation 0.56 and NoThink 0.28. Combining descriptions and predictions gives the strongest overall result among these five formats.

The gains are task-dependent. NoThink ties FreeThink on Sokoban at 0.57; StateEstimation has zero manipulation success; TransitionModeling reaches 0.76 on FrozenLake versus WorldModeling’s 0.71. Format enforcement supports useful learning in these settings, but is not uniformly superior in every task or evidence that the emitted rationale is always faithful.

Existing RL methods are inadequate for multi-turn VLM agents.

Unmasked PPO scores 0.26 overall, masked GRPO 0.54 and masked turn-level PPO 0.55. VAGEN argues that observation masking and token-specific credit are useful, while group-relative estimates become difficult with diverse trajectories.

These aggregate results do not establish that the alternatives are inadequate on every environment. GRPO’s navigation average is 0.85, above base WorldModeling’s 0.79 and VAGEN-Full’s 0.81. The explanations about sample cost and trajectory diversity are the authors’ interpretation, not a controlled sample-complexity theorem.

Table 3: Examples of how visual states are converted into internal beliefs using natural language, symbolic and structured representations.

Visual State Natural Language Symbolic Structured
Complete original PDF block S2.T3.visual
"The player is at the upper-left,
the box is to the right of the player,
the target is below the player"
P_O,
_X_,
#__
{’player’:[0,0],
’box’:[1,1],
’target’:[2,0],
’wall’: [0,2] }

Table 4: Performance comparison across different visual state representations. It shows that the representations are task-dependent.
Visual State Representation Sokoban FrozenLake PrimitiveSkill
Place Stack Drawer Align Average
Natural-Lanaguage 0.61 0.71 1.00 0.88 0.88 0.88 0.91
Structured 0.28 0.63 1.00 0.88 0.88 1.00 0.94
Symbolic 0.49 0.49 – – – – –

3 How Can We Represent Internal Beliefs about the World?

The representation study constrains observation and prediction fields to natural language, symbolic grids or structured dictionaries. It compares all three on the grid tasks and language versus dictionaries on manipulation. It does not compare every representation on all five environments.

Results and Insights.

Natural language scores 0.61 on Sokoban and 0.71 on FrozenLake, versus structured dictionaries at 0.28/0.63 and symbols at 0.49/0.49. Symbols are weaker than language but are not the worst Sokoban representation. Manipulation dictionaries reach 0.94 versus 0.91 for language.

The manipulation setup already supplies object coordinates, leaving visual matching and action planning rather than pure metric reconstruction from pixels. That support helps explain the structured result. The text says subsequent manipulation studies use structured output, but Tables 2 and 5 retain the base score 0.91; the source does not fully reconcile that representation provenance.

4 Can World Modeling Help with Reward Shaping and Credit Assignment?

VAGEN-Full adds supervision of current/next-state descriptions and a hierarchy for assigning credit to the turns and tokens that produced them.

4.1 WorldModeling Reward

An initial CLIP similarity reward was judged too insensitive to fine spatial detail. The method instead uses simulator-derived state information and GPT-4.1 nano to assess observation descriptions and predictions. Depending on the protocol, the judge gives a binary answer or extracts a dictionary for rule-based comparison.

The two matching terms have weights 0.5 in the main full-method experiments. This is privileged training supervision, not a reward requiring only the agent’s images. The judgment measures selected textual relationships and can be exploited; it does not independently verify a complete causal world model.

Equation
rtreason=βs⋅𝕀StateEstimation​(s^t,st)+βw⋅𝕀TransitionModeling​(s^t+1,st+1),r^{\text{reason}}_{t}=\beta_{s}\cdot\mathbb{I}_{\texttt{StateEstimation}}(\hat{s}_{t},s_{t})+\beta_{w}\cdot\mathbb{I}_{\texttt{TransitionModeling}}(\hat{s}_{t+1},s_{t+1}),

4.2 Bi-Level General Advantage Estimation (GAE)

The first GAE pass computes a temporal-difference error and advantage at each action boundary, using turn-level discount and trace parameters. The second pass propagates that signal backward among the generated tokens within each action, with separate token-level parameters and KL penalties.

There is a source-level boundary distinction: the main prose says the final token’s advantage is initialized to the turn advantage, while Algorithm 2 adds the final token’s local TD error to the turn advantage. The complete formulas and algorithm are retained. A reproduction must choose the intended implementation rather than treating those two descriptions as identical.

Equations
δtturn=rt+γturn​Vϕ​(τ¯≤at+1)−Vϕ​(τ¯≤at).\delta_{t}^{\text{turn}}=r_{t}+\gamma_{\text{turn}}V_{\phi}(\bar{\tau}_{\leq a_{t+1}})-V_{\phi}(\bar{\tau}_{\leq a_{t}}).
Atturn=δtturn+γturn​λturn​At+1turn.A_{t}^{\text{turn}}=\delta_{t}^{\text{turn}}+\gamma_{\text{turn}}\lambda^{\text{turn}}A_{t+1}^{\text{turn}}.
δt,itoken=rt,iKL+γtoken​Vϕ​(τ¯t,<i+1)−Vϕ​(τ¯t,<i),\delta_{t,i}^{\text{token}}=r_{t,i}^{\text{KL}}+\gamma_{\text{token}}V_{\phi}(\bar{\tau}_{t,<i+1})-V_{\phi}(\bar{\tau}_{t,<i}),
At,itoken=δt,itoken+γtoken​λtoken​At,i+1token.A_{t,i}^{\text{token}}=\delta_{t,i}^{\text{token}}+\gamma_{\text{token}}\lambda^{\text{token}}A_{t,i+1}^{\text{token}}.
Complete original PDF block S4.F3
Figure 3: Token-Level GAE and Bi-Level GAE frameworks. Standard Token-Level GAE, where a single, sparse reward at the end of the trajectory is propagated backward token-by-token. Our proposed Bi-Level GAE assigns rewards at each turn. It first computes a turn-level advantage (purple arrows) before propagating that credit to the individual tokens within the turn (orange arrows), allowing for robust, hierarchical estimation for credit assignment.

4.3 VAGEN-Full Multi-Turn Reinforcement Learning Framework

A turn’s reward combines the environment reward, valid-format reward and state-reasoning reward. The actor–critic pipeline remains the base PPO system, with bi-level advantage estimation replacing the single backward GAE pass.

Table 5: Test success rate and scores in VAGEN-Base and VAGEN-Full.
Model/Method Sokoban FrozenLake Navigation PrimitiveSkill SVG Overall
Base Common Average Place Stack Drawer Align Average Dino DreamSim Average
Qwen2.5-VL-3B 0.14 0.14 0.22 0.27 0.24 0.00 0.00 0.00 0.00 0.00 0.80 0.27 0.54 0.21
Qwen2.5-VL-3B w/ VAGEN-Base 0.61 0.71 0.78 0.80 0.79 1.00 0.88 0.88 0.88 0.91 0.90 0.65 0.78 0.76
Qwen2.5-VL-3B w/ VAGEN-Full 0.79 0.74 0.80 0.81 0.81 1.00 0.88 1.00 1.00 0.97 0.91 0.67 0.79 0.82
Equation
rt=rtreason+rtformat+R⁡(st,at),r_{t}=r_{t}^{\text{reason}}+r_{t}^{\text{format}}+R(s_{t},a_{t}),

Experiment Setup.

Base and Full both use the combined WorldModeling output format. Full adds the judge-based reward and bi-level GAE for the four interactive environment families. SVG is an explicit exception: its reported Full configuration adds only bi-level GAE, because there is no external transition to judge. Table 21’s generic grounding/prediction weights should not override this stated exception.

Results and Insights.

Base→Full changes are Sokoban 0.61→0.79, FrozenLake 0.71→0.74, navigation 0.79→0.81, manipulation 0.91→0.97 and SVG 0.78→0.79. The overall change is 0.76→0.82. Manipulation Place stays at 1.00 and Stack at 0.88; Drawer and Align increase to 1.00.

Full exceeds the listed proprietary models on the aggregate score, but not every task: o4-mini has 0.82 FrozenLake success, and several proprietary models exceed Full’s SVG similarity. The reported scores lack test confidence intervals, repeated-seed uncertainty and a complete specification of test-set sizes/splits, limiting the strength of small differences and generalization claims.

4.4 Ablations

Training curves suggest that state-reasoning rewards help base training and that bi-level credit alone can be unstable when intermediate rewards are sparse or inaccurate. Their combination is generally strong across the shown tasks. These are training curves, not a full table of repeated-seed test ablations.

Appendix D provides additional model and FreeThink results. They show a useful but nonuniform mechanism: FreeThink improves on one listed task and is unchanged on another. Better credit assignment can also accelerate exploitation of a flawed reward.

Complete original PDF block S4.F4
Figure 4: Training success rate for VAGEN-Base, VAGEN-Full and ablations. Since SVG reconstruction is a multi-turn reasoning task without world dynamics, we only apply the Bi-Level GAE configuration.

4.5 Case Studies and Findings

Figure 5 preserves complete side-by-side cases for navigation, Sokoban and manipulation; the surrounding prose also discusses FrozenLake, which appears in the appendix’s reward-hacking cases. These are qualitative illustrations, not a measured frequency of each behavior.

Complete original PDF block S4.F5
Figure 5: Comparison of VAGEN-Base and VAGEN-Full across Navigation, Sokoban, and PrimitiveSkill environments, from left to right. Within each environment, the left column shows VAGEN-Base results while the right column displays VAGEN-Full results. The improvement of VAGEN-Full shows enhanced reasoning ability in understanding environments such as their spatial information.

Enhanced Spatial Understanding and Planning.

Selected traces illustrate spatial reasoning and action sequences, while also exposing unreliable rationales: a Base navigation response says to rotate left but emits rotateright, and Full Sokoban text can misstate the target’s position. These examples support comparing behavior and outcomes, not assuming that each rationale correctly explains the action or scene.

Response Convergence and Reduced Exploration.

Early responses are varied and verbose; later responses often reuse short templates. The authors associate this with reduced entropy and exploration. The supplied figures chiefly show task-performance curves and examples, so the claimed entropy trend is not accompanied by a dedicated quantitative entropy plot here. Concision can reflect either useful regularity or loss of state specificity.

Reward Hacking and Over-optimization.

Agents sometimes discover generic statements that satisfy the judge without predicting the specific next state. Stronger credit assignment can amplify that shortcut. Appendix D describes stricter extraction/F1 scoring and a repetition penalty; it does not provide a separate quantitative ablation establishing that these eliminate reward hacking.

5 Related Work

The method combines earlier language-model RL, multi-turn trajectory optimization and explicit visual-state reasoning. Its novelty is evaluated as a system and through selected ablations, not by proving a new general theory of world models.

RL for LLMs and VLMs.

Prior work uses human, rule-based and model-generated rewards [20][24]. Hierarchical language-agent methods [12][14] motivate separating levels of feedback. VAGEN applies a turn/token advantage scheme to visual agent trajectories with observation masking.

Multi-turn Agent Training for LLMs and VLMs.

Prompted embodied agents and modular adapters precede direct multi-turn VLM training. Related systems include GiGPO for long-horizon credit, AReaL for asynchronous training and DART for GUI agents [65][66][67]. VAGEN emphasizes what agents explicitly describe and predict while interacting.

World Model Reasoning and Visual State Reasoning.

Visual grounding and causal analyses study how models use images [68][73]. VAGEN extends the empirical question across interaction turns, using text as the state representation. Its state predictions are evaluated against environment information, rather than by demonstrating a standalone rollout simulator.

6 Conclusion and Limitations

The evidence supports structured visual reasoning and carefully delivered supervision in these short, controlled tasks. It also shows that optimizing reasoning rewards can produce generic answers that exploit the judge. Output structure, high reward and reliable internal understanding are therefore distinct outcomes.

Important limits are short horizons, supplied manipulation coordinates, privileged simulator-state rewards, task-specific settings, incomplete comparison metadata and uncertainty, and source numerical/pseudocode inconsistencies. Additional model families and supervised visual training are proposed extensions; the current experiments do not establish general real-world robot deployment.

Appendix

The appendix supplies the implementation specification, exact prompts, state representations, reward protocols and qualitative cases retained below.

Appendix

Appendix material follows the source’s original section structure.

Appendix A World Modeling for Visual State Reasoning via Multi-Turn RL

This appendix specifies the rollout/training pipeline and the task configurations used to interpret the headline results.

A.1 Problem Formulation: VLM Agent Training under a POMDP Formulation

The POMDP formulation is motivated by incomplete visual observations and the need to condition actions on interaction history.

Inherent Partial Observability in Environments.

First-person navigation hides objects outside the current view, so turning or moving can acquire information as well as make progress. This is a clear partial-observability example. A fully visible deterministic grid does not automatically have the same information constraints.

Methodological Congruence with POMDPs.

Teacher-forced updates evaluate collected trajectories with their historical context. Correct causal masking is what keeps training consistent with acting; later observations in a stored sequence should not be available to earlier token predictions.

A.2 Multi-Turn Reinforcement Learning in VLM Agentic Tasks

The rollout loop parses generated answers into environment actions, records new observations/rewards and repeats to the turn limit. The training loop computes masked PPO and critic updates over those trajectories. Figures 6–7, Algorithm 1 and the exact output formats in Table 6 are retained.

Algorithm 1 makes explicit that the next action token is used when propagating token-level credit, skipping observation tokens. Its environment-return notation is a compact description; the task reward tables include format and intermediate components as well.

Complete original PDF block A1.F6
Figure 6: Multi-turn RL framework for VLM agents.
Complete original PDF block A1.F7
Figure 7: Overview of the VAGEN Framework. The framework operates in a dual-loop structure: an inner Rollout Loop for data collection and an outer Training Loop for policy optimization.
Algorithm 1 VAGEN-Base: Reinforcement Learning for VLM Agents
Complete original PDF block alg1
Table 6: Reasoning strategy formats.
Strategy Format
NoThink <answer>atea^{\text{e}}_{t}</answer>
FreeThink <think>zt{z}_{t}</think>
<answer>atea^{\text{e}}_{t}</answer>
StateEstimation <think>
<observation>s^t\hat{s}_{t}</observation>
<reasoning>a^t\hat{a}_{t}</reasoning>
</think>
<answer>atea^{\text{e}}_{t}</answer>
TransitionModeling <think>
<reasoning>a^t\hat{a}_{t}</reasoning>
<prediction>s^t+1\hat{s}_{t+1}</prediction>
</think>
<answer>atea^{\text{e}}_{t}</answer>
WorldModeling <think>
<observation>s^t\hat{s}_{t}</observation>
<reasoning>a^t\hat{a}_{t}</reasoning>
<prediction>s^t+1\hat{s}_{t+1}</prediction>
</think>
<answer>atea^{\text{e}}_{t}</answer>

A.3 VAGEN Framework

VAGEN builds on VeRL [80] and separates environment services from model training. This supports exchanging environments and RL variants. The manuscript does not report a distributed scaling study showing how throughput changes with more machines.

A.4 Environments and Tasks for VLM Agents

The action spaces and limits below define the actual evaluation scope, including the short turn horizons and privileged coordinate assistance.

Sokoban

Sokoban uses a 6×6 room and one box, with puzzles requiring at least five actions. Each response may contain three moves and the episode has three interaction turns. The underlying max_steps=100 does not imply a hundred-turn agent evaluation.

Table 7: Action space for the Sokoban environment.
Name Description
Up Move the agent one cell upward on the grid.
Left Move the agent one cell to the left on the grid.
Right Move the agent one cell to the right on the grid.
Down Move the agent one cell downward on the grid.
Table 8: Hyperparameters for the Sokoban environment.
Name Value Description
dim_room (6,6)(6,6) Dimensions of the Sokoban grid
max_steps 100 Maximum number of steps allowed per episode
num_boxes 1 Number of boxes to be pushed onto targets
min_actions_to_succeed 5 Minimum number of actions required to solve the puzzle
max_actions_per_step 3 Maximum number of actions the agent can take per turn
max_turns 3 Maximum number of turns the agent can interact with the environment

FrozenLake

FrozenLake uses a random 4×4 map, at least five required moves, three actions per turn and three turns. Slipperiness is disabled, despite the exact prompts warning that tiles may slip. That prompt/configuration mismatch is preserved and should be considered in reproduction.

Table 9: Action space for the FrozenLake environment.
Name Description
Up Move the agent one cell upward on the grid.
Left Move the agent one cell to the left on the grid.
Right Move the agent one cell to the right on the grid.
Down Move the agent one cell downward on the grid.
Table 10: Hyperparameters for the FrozenLake environment.
Name Value Description
desc None Environment map layout (if None, randomly generated)
is_slippery False Whether the frozen tiles are slippery
size 4 Size of the square grid
max_actions_per_step 3 Maximum number of actions the agent can take per turn
min_actions_to_succeed 5 Minimum number of actions required to reach the goal
max_turns 3 Maximum number of turns the agent can interact with the environment

Navigation

Navigation uses a single 255-resolution view, a 100° field of view, 0.5-distance movement increments and a 1.5 success threshold. Four turns allow up to five actions each. Rotations are 90° and camera tilts 30°. The two reported navigation categories are Base and Common; detailed test split counts are not supplied.

Table 11: Action space for the Navigation environment.
Name Description
MoveAhead Move forward by some distance
MoveBack Move backward by some distance
MoveRight Move rightward by some distance
MoveLeft Move leftward by some distance
RotateRight Rotate to the right by 90 degrees
RotateLeft Rotate to the left by 90 degrees
LookUp Tilt the camera upward by 30 degrees
LookDown Tilt the camera downward by 30 degrees
Table 12: Hyperparameters for the Navigation environment.
Name Value Description
resolution 255 Resolution of the rendered images
down_sample_ratio 1.0 Ratio for down-sampling images
fov 100 Field of view angle in degrees
multiview False Whether to use multiple camera views
max_actions_per_step 5 Maximum number of actions the agent can take per turn
success_threshold 1.5 Threshold for considering task successful
step_length 0.5 Distance traveled in a single movement action
max_turns 4 Maximum number of turns the agent can interact with the environment

PrimitiveSkill

Manipulation exposes pick, place and push primitives, with two actions per response and three turns. Prompts supply workspace bounds and object-position lists in integer millimeters, plus task information such as drawer opening. The model matches positions to visible objects and plans primitive calls; it is not learning torque control or inferring every coordinate from the image.

Table 13: Action space for the PrimitiveSkill environment.
Name Description
pick(x, y, z) Grasp an object located at position (x, y, z) in the robot’s workspace
place(x, y, z) Place the object currently held by the robot’s gripper at the target position (x, y, z)
push(x1, y1, z1, x2, y2, z2) Push an object from position (x1, y1, z1) to position (x2, y2, z2)
Table 14: Hyperparameters for the PrimitiveSkill environment.
Name Value Description
max_actions_per_step 2 Maximum number of actions the agent can take per turn
max_turns 3 Maximum number of turns the agent can interact with the environment

SVG Reconstruction

SVG reconstruction uses the starvector/svg-icons-simple dataset and two turns of open-ended SVG generation. Its output is scored by rendering and comparing with the target. This is iterative visual code generation, so the world-dynamics interpretation used for physical interaction is limited.

Table 15: Action space for the SVG Reconstruction environment.
Name Description
SVG Code Open text format allowing for the specification of SVG markup code
Table 16: Hyperparameters for the SVG Reconstruction environment.
Name Value Description
dataset_name starvector/svg-icons-simple Dataset used for SVG examples
max_turns 2 Maximum number of turns the agent can interact with the environment

A.5 Reward Assignment

The appendix reward specification is richer than the main text’s sparse-success shorthand. Keep environment rewards, formatting rewards and reasoning rewards separate when reproducing a condition.

Sokoban

Sokoban awards 10 for success, −0.1 at unfinished steps, 1 for placing a box, and 0.5 for valid response format. Full-method grounding and prediction each have weight 0.5.

Table 17: Reward structure for the Sokoban environment.
Reward Type Value Description
Success reward 10 Awarded when all boxes are placed on target locations
Failure penalty -0.1 Applied each step when the task is not completed
Box placement reward 1 Granted for each box pushed onto a target location
Format reward 0.5 Provided at each turn to encourage visual state reasoning
Grounding reward weight 0.5 Weight applied to StateEstimation reward
World modeling reward weight 0.5 Weight applied to TransitionModeling reward

FrozenLake

FrozenLake combines success reward 10, unfinished-step penalty −0.1 and format reward 0.5. The current/next-state matching terms each have weight 0.5 when enabled.

Table 18: Reward structure for the FrozenLake environment.
Reward Type Value Description
Success reward 10 Awarded when the agent reaches the goal position
Failure penalty -0.1 Applied each step when the task is not completed
Format reward 0.5 Provided at each turn to encourage visual state reasoning
Grounding reward weight 0.5 Weight applied to StateEstimation reward
World modeling reward weight 0.5 Weight applied to TransitionModeling reward

Navigation

Navigation uses the same stated success, failure and formatting values, with the two reasoning terms weighted 0.5. A judge receives relative target directions/distances extracted from the simulator.

Table 19: Reward structure for the Navigation environment.
Reward Type Value Description
Success reward 10 Awarded when the agent reaches the goal location
Failure penalty -0.1 Applied each step when the task is not completed
Format reward 0.5 Provided at each turn to encourage visual state reasoning
Grounding reward weight 0.5 Weight applied to StateEstimation reward
World modeling reward weight 0.5 Weight applied to TransitionModeling reward

PrimitiveSkill

Manipulation adds a stage bonus of 2×(highest completed stage+1), in addition to success 10, unfinished-step −0.1 and format 0.5. This intermediate task supervision matters when interpreting claims about learning from sparse rewards.

Table 20: Reward structure for the PrimitiveSkill environment.
Reward Type Value Description
Success reward 10 Awarded when the manipulation task is completed
Failure penalty -0.1 Applied each step when the task is not completed
Stage-based reward (stage+1)×2(\text{stage}+1)\times 2 Granted upon completing key manipulation subgoals (where stage is the highest successfully completed stage)
Format reward 0.5 Provided at each turn to encourage visual state reasoning
Grounding reward weight 0.5 Weight applied to StateEstimation reward
World modeling reward weight 0.5 Weight applied to TransitionModeling reward

SVG

Table 21 lists DreamSim weight 5 and DINO weight 0.0001, describing the latter as effectively unused. Its generic grounding/prediction rows conflict with the explicit main-text statement that SVG Full uses only bi-level GAE; this edition follows that stated experimental exception and preserves the table for inspection.

Table 21: Reward structure for the SVG Reconstruction environment.
Reward Type Value Description
Image similarity Variable Weighted DreamSim [81] and DINO [82] scores measuring similarity between generated and target images
Format reward 0.5 Provided at each turn to encourage visual state reasoning
Grounding reward weight 0.5 Weight applied to StateEstimation reward
World modeling reward weight 0.5 Weight applied to TransitionModeling reward
DreamSim weight 5.0 Scaling factor applied to DreamSim similarity scores
Dino weight 0.0001 We only use DreamSim score for reward

A.6 Evaluation Metrics

Success indicators and image-similarity functions are distinct metrics. An average across them summarizes this benchmark mix, not a calibrated probability of completing an arbitrary task.

Metrics for Task-Completion Environments

Grid, navigation and manipulation scores average binary trajectory completion. The manuscript does not supply confidence intervals or a complete test-trajectory count for every category, so fine differences should be interpreted cautiously.

Equation 1
Success Rate=𝔼τ∼𝒟​[f⁡(τ)]\text{Success Rate}=\mathbb{E}_{\tau\sim\mathcal{D}}[f(\tau)] (1)

Metrics for SVG Reconstruction

SVG reports mean final-output DreamSim [81] and DINO [82] similarities. The appendix describes DreamSim as a diffusion-model metric, but that description should not be used to infer the metric’s architecture; the cited perceptual-similarity method defines it. Similarity is not exact geometric or semantic correctness of the generated SVG.

Equations 2–3
DreamSim Score =𝔼τ∼𝒟​[g​(τ)]\displaystyle=\mathbb{E}_{\tau\sim\mathcal{D}}[g(\tau)] (2)
DINO Score =𝔼τ∼𝒟​[h​(τ)]\displaystyle=\mathbb{E}_{\tau\sim\mathcal{D}}[h(\tau)] (3)

Appendix B What Can We Reason About Visual States?

This appendix provides exact off-the-shelf and training prompts, backbone settings and additional learning curves.

B.1 Bottleneck of Off-the-Shelf VLMs on Agentic Tasks

The five evaluation prompts use FreeThink-style outputs and include environment guides, action constraints, examples and sample responses. They are retained in full, including the FrozenLake slipperiness mismatch and incomplete or invalid sample outputs.

Table 22 documents seven older model identifiers with temperature 0.7 and max-token settings 150/400, without fully mapping those two limits to tasks. It does not specify corresponding settings for the additional GPT-5, o3, o4-mini, Gemini 2.5 Pro and Claude 4.5 rows later present in Table 2.

Complete original PDF block A2.SS1.p1.2.part1
Complete original PDF block A2.SS1.p1.3.part1
Complete original PDF block A2.SS1.p1.4.part1Complete original PDF block A2.SS1.p1.4.part2
Complete original PDF block A2.SS1.p1.5.part1Complete original PDF block A2.SS1.p1.5.part2
Complete original PDF block A2.SS1.p1.6.part1
Table 22: Off-the-shelf VLMs for evaluation.
Model Name Model ID Provider Max Tokens Temperature
VLM-R1-3B omlab/VLM-R1-Qwen2.5VL-3B-Math-0305 omlab 150/400 0.7
Qwen2.5-VL-3B Qwen/Qwen2.5-VL-3B-Instruct Qwen 150/400 0.7
Qwen2.5-VL-7B Qwen/Qwen2.5-VL-7B-Instruct Qwen 150/400 0.7
Qwen2.5-VL-72B Qwen/Qwen2.5-VL-72B-Instruct Qwen 150/400 0.7
GPT-4o gpt-4o openai 150/400 0.7
Gemini 2.0 gemini-2.0-flash gemini 150/400 0.7
Claude 3.7 Sonnet claude-3-7-sonnet-20250219 claude 150/400 0.7

B.2 Reasoning in Multi-turn RL Training

Training uses Qwen2.5-VL-3B actor and critic initializations, top-p 0.95, temperature 0.7, token discount 1, KL coefficient 0.001, learning rates 10⁻⁶/10⁻⁵, batch 128 and PPO minibatch 32. The described servers have eight H100s, 104 CPUs and 1.7 TB memory, running two experiments concurrently; sessions take roughly four to eight hours.

The full WorldModeling prompts are retained, rather than reconstructed from templates. Other formats vary the requested fields. Some printed examples themselves violate the format or action limit—for example a three-action manipulation answer despite the two-action limit, and malformed SVG/XML fragments—so they should not be treated as validated demonstrations. Turn-level discount/trace values and several optimizer details are not listed in Table 23.

Table 23: Multi-turn RL training hyperparameters.
Parameter Value Description
Rollout Phase
Top-p 0.95 Nucleus sampling parameter for action generation
Temperature 0.7 Sampling temperature for controlling randomness
Update Phase
Advantage Estimator masked_gae Generalized Advantage Estimation with masking
Actor Model Qwen/Qwen2.5-VL-3B-Instruct Pre-trained model used for actor initialization
Critic Model Qwen/Qwen2.5-VL-3B-Instruct Pre-trained model used for critic initialization
γtoken\gamma_{\text{token}} 1.0 Discount factor for token-wise advantage calculation
KL Penalty Coefficient (β\beta) 0.001 Coefficient for KL divergence penalty in PPO objective
Actor Learning Rate 1e-6 Learning rate for the actor network
Critic Learning Rate 1e-5 Learning rate for the critic network
Train Batch Size 128 Total batch size for training
PPO Mini Batch Size 32 Mini-batch size for PPO updates
Complete original PDF block A2.SS2.p4.1.part1Complete original PDF block A2.SS2.p4.1.part2
Complete original PDF block A2.SS2.p4.2.part1Complete original PDF block A2.SS2.p4.2.part2
Complete original PDF block A2.SS2.p4.3.part1Complete original PDF block A2.SS2.p4.3.part2
Complete original PDF block A2.SS2.p4.4.part1Complete original PDF block A2.SS2.p4.4.part2
Complete original PDF block A2.SS2.p4.5.part1Complete original PDF block A2.SS2.p4.5.part2

B.3 Learning Dynamics of RL Baselines

Figure 8 compares training dynamics for unmasked PPO, masked GRPO, masked turn-PPO and base VAGEN. The ranking varies with the environment; the figure supports the aggregate comparison while also showing strong navigation learning by alternatives. It does not provide a repeated-seed confidence analysis.

Complete original PDF block A2.F8
Figure 8: Training success rates for RL Baselines and VAGEN-Base

Appendix C How Can We Represent Internal Beliefs about the World?

Representation formats constrain the observation and prediction fields. The appendix preserves the exact grids and dictionaries so that differences in grounding difficulty remain visible.

Table 24: Visual state representation formats across different environments.
Environment Format Type Description
Sokoban Natural-Lanaguage Text Free-form textual descriptions of the game state
Symbolic Grid Environment-native grid-based symbols representing game entities
Structured Dictionary JSON-like dictionary containing positions of game objects
FrozenLake Natural-Lanaguage Text Free-form textual descriptions of the game state
Symbolic Grid Environment-native grid-based symbols representing game entities
Structured Dictionary JSON-like dictionary containing positions of game objects
PrimitiveSkill Natural-Lanaguage Text Free-form textual descriptions of the manipulation state
Structured Dictionary JSON-like dictionary containing 3D coordinates of task-relevant objects

C.1 Symbolic Representations

Symbolic formats encode objects and agent positions as grid characters. They require the model to map pixels to a discrete layout, which can be difficult without suitable grounding training.

Sokoban Symbolic Format

Sokoban distinguishes walls, floor, targets, boxes, the player and overlapping box/player-on-target states. The retained example is a symbolic illustration, not a full specification of every generated training room.

######
#_P__#
#_XO_#
###__#
######

FrozenLake Symbolic Format

FrozenLake symbols distinguish frozen cells, holes, goal, player and terminal player states. The grid example preserves the exact row arrangement; flattening it into prose would lose the representation being studied.

___G
O___
__O_
_P__

C.2 Structured Representations

Structured outputs use dictionaries of positions and task entities. Their precision is useful only when the entities and coordinates are grounded correctly.

Sokoban Structured Format

The Sokoban dictionary contains player, box and target coordinates plus grid size. The source calls the example “symbolic” in the surrounding sentence, but its actual format is structured data.

{player_position: (4, 2), box_positions: [(3, 2)],
target_positions: [(3, 1)], grid_size: (6, 6)}

FrozenLake Structured Format

The FrozenLake dictionary records player, target, holes and grid dimensions. The shown player and target coincide; it is an example state, not a typical starting configuration.

{player_position: (3, 2), target_position: (3, 2),
hole_positions: [(1, 2)], grid_size: (4, 4)}

PrimitiveSkill Structured Format

Manipulation dictionaries associate object names with three-dimensional coordinates. Because the prompt supplies unlabeled position candidates, success still requires visual object matching, but the input substantially reduces metric state-estimation difficulty.

{red_cube: (100, 100, 40), green_cube: (200, 200, 60)}

Appendix D How to Improve Reasoning over World Models?

The reward appendix exposes both the exact judge criteria and the ways agents can optimize those criteria without consistently accurate reasoning.

D.1 World Modeling Reward via LLM-as-a-Judge

GPT-4.1 nano evaluates state descriptions and next-state predictions using task-specific YES/NO prompts. All eight prompts are retained. Grid criteria emphasize selected relative directions rather than exhaustive state reconstruction; some rules automatically accept goal-completed cases. Navigation prediction scoring demands explicit direction words and rejects generic “closer” descriptions. Manipulation accepts coordinate errors up to ±10 units.

These protocols use simulator state and hand-designed tolerances. They are useful supervision signals but neither calibrated state-estimation accuracy nor proof of causal reasoning. The judge is itself fallible, and the exact wording creates shortcuts that the case studies later expose.

Complete original PDF block A4.SS1.p3.1.part1Complete original PDF block A4.SS1.p3.1.part2
Complete original PDF block A4.SS1.p4.1.part1Complete original PDF block A4.SS1.p4.1.part2
Complete original PDF block A4.SS1.p5.1.part1Complete original PDF block A4.SS1.p5.1.part2
Complete original PDF block A4.SS1.p6.1.part1Complete original PDF block A4.SS1.p6.1.part2
Complete original PDF block A4.SS1.p7.1.part1
Complete original PDF block A4.SS1.p8.1.part1
Complete original PDF block A4.SS1.p9.1.part1Complete original PDF block A4.SS1.p9.1.part2
Complete original PDF block A4.SS1.p10.1.part1Complete original PDF block A4.SS1.p10.1.part2

Text-based State Information

Ground truth is serialized differently by task: pairwise spatial sentences for grids, target/visible-object dictionaries for navigation, and named coordinates for manipulation. All nine structured/grid examples across the representation and reward appendices are retained. Their source spelling and sample values are not silently repaired.

[’box0 is above and at the same column as the player’,
 ’target0 is above and at the same column as the player’,
 ’target0 is above and at the same column as box0’]
[‘target is above and at the same column as the player’,
 ’hole0 is above and at the same column as the player’,
 ’hole1 is above and on the right side of the player’,
 ’hole2 is above and on the right side of the player’,
 ’hole3 is above and on the right side of the player’,
 ’hole4 is at the same row and to the right of the player’]
{’target_obj_type’: ’Toaster’,
 ’target_distance_to_player’: 2.59,
 ’target_direction_to_player’: ’ahead’,
 ’visible_objects’: [{’type’: ’Cabinet’,
   ’direction_to_player’: ’left’,
   ’distance_to_player’: 0.94},
  {’type’: ’Drawer’,
   ’direction_to_player’: ’left’,
   ’distance_to_player’: 1.18},
  {’type’: ’CounterTop’,
   ’direction_to_player’: ’left’,
   ’distance_to_player’: 1.33},
  ...]}
{’lefz_target_position’: (80, -100, 0),
 ’righz_target_position’: (80, 100, 0),
 ’red_cube_position’: (83, -102, 20),
 ’green_cube_position’: (-47, 95, 20)}

D.2 Mitigating Reward Hacking via Structured Evaluation and Repetition Penalty

The mitigation first asks an LLM to extract entities and relations into a dictionary, then scores them against a reference with F1. A max-heap tracks frequent sentences. In the reported Sokoban setting, a response receives an additional −0.1 only if it is both frequent and has F1<0.7.

This targets repeated incorrect text rather than penalizing every common phrase. The appendix does not specify all frequency-tracking/extraction parameters or provide a quantitative mitigation ablation. Its claim of improved robustness is therefore more limited than proof that exploitation has been removed.

D.3 Bi-Level General Advantage Estimation (GAE) Details and Additional Results

Algorithm 2 retains the complete two-pass advantage computation, including the final-token local TD term that is omitted from the simpler initialization description in the main text.

Table 25 adds bi-level GAE to FreeThink: FrozenLake improves 0.39→0.62, but Sokoban stays 0.43. These base values differ from Table 2’s FreeThink 0.68/0.57 without an explanation of the condition change, so the two tables should not be pooled as the same run.

On Sokoban, Table 26 reports Base→Full of 0.61→0.79 for Qwen 3B, 0.63→0.92 for Qwen 7B and 0.36→0.39 for InternVL3-2B. Approximate training costs are 40 H100-hours each for the two grid tasks, 30 each for navigation/manipulation and 10 for SVG; judge usage is about 23M, 23M, 6M and 4.6M tokens respectively, with none for SVG. These are reported resource totals, not dollar prices or a throughput scaling study.

Table 25: Bi-level GAE helps FreeThink RL baselines, and can serve as a general mechanism in RL training for VLM agents.
Group / Model Task Method Test Success Rate
Qwen2.5-VL 3B FrozenLake FreeThink 0.39
FrozenLake FreeThink + Bi-Level GAE 0.62
Qwen2.5-VL 3B Sokoban FreeThink 0.43
Sokoban FreeThink + Bi-Level GAE 0.43
Table 26: Results on different model families and sizes on Sokoban task.
Group / Model Method Test Success Rate
Qwen2.5-VL 3B VAGEN-BASE 0.61
VAGEN-FULL 0.79
Qwen2.5-VL 7B VAGEN-BASE 0.63
VAGEN-FULL 0.92
InternVL3-2B VAGEN-BASE 0.36
VAGEN-FULL 0.39
Table 27: Training efficiency for primary experiments (approximate).
Task H100 GPU Hours (Approx.) LLM-as-Judge Token Cost (Approx.)
FrozenLake 40 2.3×1072.3\times 10^{7}
Sokoban 40 2.3×1072.3\times 10^{7}
Navigation 30 6.0×1066.0\times 10^{6}
ManiSkill 30 4.6×1064.6\times 10^{6}
SVG 10 N/A
Algorithm 2 Bi-Level General Advantage Estimation (GAE)
Complete original PDF block alg2

Appendix E Case Study

The complete early/late examples are retained as evidence about response behavior. They illustrate changes in phrasing and grounding, without measuring how frequently each pattern occurs across the test set.

E.1 Visual State Reasoning Enhances Spatial Understanding and Multi-Step Planning

Navigation examples recognize obstacles and plan around them; Sokoban examples reason about relative box/target positions. Figure 5 also includes manipulation. These selected comparisons illustrate the intended benefit but should not be generalized into a claim that every retained response is correct.

E.2 Response Convergence and Reduced Exploration

Four case blocks compare early and late Sokoban/navigation responses. Later outputs often reuse a directional template. Some Sokoban examples still contain questionable directions relative to their supplied state, so standardization is not equivalent to correctness. The qualitative evidence is consistent with reduced linguistic diversity; a dedicated numerical entropy analysis is not supplied here.

Complete original PDF block A5.SS2.p5.1.part1Complete original PDF block A5.SS2.p5.1.part2
Complete original PDF block A5.SS2.p6.1.part1
Complete original PDF block A5.SS2.p7.1.part1Complete original PDF block A5.SS2.p7.1.part2
Complete original PDF block A5.SS2.p8.1.part1Complete original PDF block A5.SS2.p8.1.part2

E.3 Reward Hacking and Over-optimization

The two FrozenLake case blocks expose generic goal claims and broad directional alternatives. Repeating that the player will reach the gift can satisfy weak judging criteria without a state-specific prediction. Better advantage propagation can optimize this shortcut more efficiently, making reward design as important as optimization strength.

Complete original PDF block A5.SS3.p4.1.part1
Complete original PDF block A5.SS3.p5.1.part1

Appendix F Summary of Findings

The four original finding boxes summarize the authors’ interpretation and are retained in full. The results favor explicit state reasoning, task-appropriate representations and turn-aware supervision, while the case studies show substantial limitations of judge-based rewards.

A careful reading keeps aggregate benchmark gains separate from per-task superiority, structured output separate from faithful reasoning, and reward maximization separate from grounded prediction. The source’s own final discussion recommends harder-to-exploit rewards and monitoring for templated or ungrounded responses.

Complete original PDF block finding-1
Complete original PDF block finding-2
Complete original PDF block finding-3
Complete original PDF block finding-4
Concept demonstration

Open image ↗