At a glance
VAGEN trains vision-language agents to describe the current visual state, reason about an action and predict its consequence before executing it. The full method rewards those descriptions against simulator state and passes credit first across turns, then across tokens. Its reported overall score rises from 0.21 for the untrained 3B backbone to 0.82, compared with 0.76 for the base training method.
The overall score averages five task-level results, including image-similarity scores; it is not an 82% universal success rate. The ratio 0.82/0.21 is about 3.9, or a 290% relative increase. The abstract’s “nearly 3× improvement” should be interpreted with that distinction. This edition follows the complete 55-page v1, including all figures, tables, formulas, algorithms, prompts, examples and references.
1 Introduction
Visual agents must infer useful state from images and maintain context as actions change the scene. VAGEN calls their explicitly generated descriptions and predictions an internal world model [1]. This is a reasoning format inside a language policy, not a separately trained visual simulator or a calibrated probability distribution over hidden states.
Three questions organize the experiments: which reasoning components help, which representation best expresses state, and how rewards and credit assignment should supervise it. The strongest controlled comparisons use the same Qwen2.5-VL-3B backbone. Proprietary-model comparisons use different architectures, training histories and inference conditions, so they show benchmark performance rather than an isolated model-size advantage.
2 Build Internal World Models via Visual State Reasoning in Multi-Turn RL
The base framework collects complete visual interaction trajectories and optimizes generated action and reasoning tokens while masking observations from the policy loss.
2.1 Problem Formulation
The POMDP separates true simulator state, the agent’s visual observation, emitted text, parsed executable action and reward. An observation may omit important information, particularly in first-person navigation. The policy seeks discounted return from its observation/action history; the state descriptions are generated approximations rather than privileged state fed directly to every agent.
Table 1 distinguishes the reasoning action representation from the executable action. In manipulation, however, prompts also supply object positions, so the tested input is not uniformly image-only across tasks.
Notation
Symbol
Observation
State
State (Belief)
Action
Action (Belief)
Action (Executable)
Reward
Trajectory
Tokenized Trajectory
Agent Policy
Critic / Value Func.
Token Index
Trajectory Reward
Advantage Estimate
Temporal-Difference Error
Discount Factor
2.2 Visual State Reasoning as an Internal World Model
An actor–critic model is trained over whole interaction trajectories. Explicit state descriptions and next-state predictions are inserted into the actor’s output, and environment observations become context for subsequent turns.
Reasoning Trajectory Rollout for Multi-Turn VLM Agents.
At each turn the VLM emits reasoning plus an answer containing executable actions. A parser sends the answer to the environment, then appends the resulting observation and reward to the trajectory. Reasoning tokens influence later decisions as context, but only the parsed actions change the environment.
Equation
Visual State Reasoning as World Modeling.
NoThink omits substantive reasoning; FreeThink allows unrestricted reasoning; StateEstimation adds an observation description; TransitionModeling adds a prediction; WorldModeling combines both around an action rationale. Output-format rewards encourage the corresponding tags.
A next-state prediction is made before the action outcome is observed. At the next turn the agent generates a fresh state description from the new observation; it does not simply reuse the previous prediction as a verified state. The method therefore encourages explicit predictive reasoning without enforcing a persistent, probabilistically consistent belief update.
The main text shows empty think tags for NoThink, whereas Table 6 shows only answer tags. Both source forms are retained rather than silently standardized.
Equations
Policy Optimization.
PPO [26] updates the actor with clipped likelihood ratios for generated tokens. A loss mask excludes observation tokens, which were supplied by the environment rather than sampled from the policy. Teacher-forced training evaluates each generated token using its causal trajectory prefix; the complete training sequence does not imply access to future observations when acting.
Equation
Advantage and Value Estimation.
The critic regresses to advantage-plus-value targets. In VAGEN-Base, intermediate generated tokens receive a KL penalty relative to a frozen reference, while the total trajectory reward is placed at the final action token. Token-level GAE [22] propagates that signal backward, skipping observation tokens.
This preserves fine token distinctions but can make a turn’s contribution difficult to identify over long token sequences. The later method changes reward placement and advantage propagation, rather than merely changing the reasoning prompt.
Equations
2.3 Reward Design in Different Environments and Tasks for VLM Agents
Five task families cover one-box Sokoban, deterministic FrozenLake, first-person object navigation, manipulation through predefined pick/place/push primitives, and SVG reconstruction. The last task iteratively edits an image description in code and has no external world dynamics.
The environment settings are deliberately small: two to four interaction turns, with several primitive actions allowed per turn. These experiments test multi-turn visual decisions, but do not establish unrestricted long-horizon autonomy or low-level robot control.
Reward Strategy and Metrics.
Four families are evaluated by success rate; SVG uses DreamSim and DINO similarity. The overall score averages task-family summaries, including the mean of the two SVG similarities, so its components have different meanings.
Although the main description emphasizes sparse binary task rewards, Appendix A also lists unfinished-step penalties, box-placement bonuses and stage rewards for manipulation. SVG training is effectively DreamSim-dominated: Table 21 weights it by 5 and DINO by only 0.0001. The full reward tables are preserved.
Model/Method
Sokoban
FrozenLake
Navigation
PrimitiveSkill
SVG
Overall
Base
Common
Average
Place
Stack
Drawer
Align
Average
Dino
DreamSim
Average
Open-Source Models
Qwen2.5-VL-72B [34]
0.18
0.44
0.72
0.75
0.73
1.00
0.50
0.00
1.00
0.44
0.89
0.62
0.76
0.51
Qwen2.5-VL-7B [34]
0.13
0.14
0.28
0.39
0.34
0.00
0.00
0.00
0.75
0.19
0.83
0.28
0.55
0.27
Qwen2.5-VL-3B [34]
0.14
0.14
0.22
0.27
0.24
0.00
0.00
0.00
0.00
0.00
0.80
0.27
0.54
0.21
VLM-R1-3B [24]
0.13
0.13
0.31
0.34
0.33
0.00
0.00
0.00
0.00
0.00
0.81
0.28
0.55
0.23
VAGEN: Multi-Turn RL with World Model Reasoning for Visual States (Backbone: Qwen2.5-VL-3B)
FreeThink
0.57
0.68
0.67
0.67
0.67
1.00
0.63
0.00
1.00
0.66
0.91
0.64
0.78
0.67
NoThink
0.57
0.09
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.89
0.62
0.76
0.28
StateEstimation
0.56
0.68
0.78
0.69
0.74
0.00
0.00
0.00
0.00
0.00
0.92
0.64
0.78
0.56
TransitionModeling
0.41
0.76
0.67
0.59
0.62
1.00
0.63
0.63
1.00
0.82
0.89
0.64
0.77
0.68
WorldModeling
0.61
0.71
0.78
0.80
0.79
1.00
0.88
0.88
0.88
0.91
0.90
0.65
0.78
0.76
VAGEN-Full
0.79
0.74
0.80
0.81
0.81
1.00
0.88
1.00
1.00
0.97
0.91
0.67
0.79
0.82
RL Baselines with World Model Reasoning Strategy (Backbone: Qwen2.5-VL-3B)
Vanilla-PPO
0.18
0.21
0.32
0.25
0.29
0.00
0.00
0.00
0.00
0.00
0.83
0.44
0.64
0.26
GRPO w/ Mask
0.20
0.57
0.88
0.81
0.85
0.00
0.00
0.00
1.00
0.25
0.92
0.66
0.79
0.54
Turn-PPO w/ Mask
0.38
0.70
0.78
0.84
0.81
0.00
0.00
0.00
1.00
0.25
0.89
0.64
0.77
0.55
Proprietary Models
GPT-5 [35]
0.70
0.77
0.75
0.81
0.78
1.00
0.63
0.00
1.00
0.66
0.95
0.75
0.85
0.75
o3 [36]
0.60
0.78
0.81
0.75
0.78
1.00
0.63
0.00
1.00
0.66
0.92
0.71
0.82
0.73
o4-mini [36]
0.44
0.82
0.75
0.75
0.75
1.00
0.50
0.00
0.75
0.56
0.90
0.66
0.78
0.67
GPT-4o [37]
0.43
0.54
0.75
0.69
0.72
0.50
0.63
0.00
0.88
0.50
0.91
0.69
0.80
0.60
Gemini 2.5 Pro22
2
Due to Gemini 2.5 Pro’s safety policy, a subset of evaluation responses for PrimitiveSkill and Navigation was blocked; reported results are based solely on the unblocked test cases.[38]
0.58
0.78
0.63
0.63
0.63
0.63
0.63
0.00
0.75
0.50
0.93
0.78
0.86
0.67
Gemini 2.0
[39]
0.28
0.61
0.50
0.63
0.56
0.75
0.13
0.00
0.25
0.28
0.93
0.74
0.84
0.51
Claude 4.5 Sonnet [40]
0.31
0.80
0.67
0.67
0.67
0.63
0.50
0.00
1.00
0.53
0.95
0.81
0.88
0.64
Claude 3.7 Sonnet [41]
0.25
0.69
0.48
0.47
0.47
0.63
0.13
0.00
1.00
0.44
0.94
0.77
0.85
0.54
2.4 What Can We Reason About Visual States?
The reasoning study compares formats under base training and then compares several policy-optimization baselines. These are separate questions from the effect of the full reward/credit-assignment system.
Off-the-Shelf VLMs struggle to solve multi-turn agentic tasks.
Table 2 lists twelve off-the-shelf models, despite the prose and Appendix Table 22 referring to seven. GPT-5 scores 0.75 overall; all listed untrained models fail the Drawer subtask. This does not mean they fail all manipulation tasks: several succeed on Place and Align.
The cross-model table has qualifications. Gemini 2.5 Pro excludes blocked responses from its evaluated cases. Claude 4.5 is 0.64 in Table 2, rather than the abstract’s 0.62. Qwen2.5-VL-72B’s four manipulation cells (1.00, 0.50, 0.00, 1.00) average 0.625, but its printed average is 0.44; that printed average is consistent with the listed 0.51 overall. These source values are retained and the inconsistency is not resolved by inventing corrected benchmark results.
World Modeling can improve visual reasoning via multi-turn RL.
With Qwen2.5-VL-3B, base WorldModeling scores 0.76 overall, FreeThink 0.67, TransitionModeling 0.68, StateEstimation 0.56 and NoThink 0.28. Combining descriptions and predictions gives the strongest overall result among these five formats.
The gains are task-dependent. NoThink ties FreeThink on Sokoban at 0.57; StateEstimation has zero manipulation success; TransitionModeling reaches 0.76 on FrozenLake versus WorldModeling’s 0.71. Format enforcement supports useful learning in these settings, but is not uniformly superior in every task or evidence that the emitted rationale is always faithful.
Existing RL methods are inadequate for multi-turn VLM agents.
Unmasked PPO scores 0.26 overall, masked GRPO 0.54 and masked turn-level PPO 0.55. VAGEN argues that observation masking and token-specific credit are useful, while group-relative estimates become difficult with diverse trajectories.
These aggregate results do not establish that the alternatives are inadequate on every environment. GRPO’s navigation average is 0.85, above base WorldModeling’s 0.79 and VAGEN-Full’s 0.81. The explanations about sample cost and trajectory diversity are the authors’ interpretation, not a controlled sample-complexity theorem.
Visual State
Natural Language
Symbolic
Structured
| Visual State Representation | Sokoban | FrozenLake | PrimitiveSkill | ||||
| Place | Stack | Drawer | Align | Average | |||
| Natural-Lanaguage | 0.61 | 0.71 | 1.00 | 0.88 | 0.88 | 0.88 | 0.91 |
| Structured | 0.28 | 0.63 | 1.00 | 0.88 | 0.88 | 1.00 | 0.94 |
| Symbolic | 0.49 | 0.49 | – | – | – | – | – |
3 How Can We Represent Internal Beliefs about the World?
The representation study constrains observation and prediction fields to natural language, symbolic grids or structured dictionaries. It compares all three on the grid tasks and language versus dictionaries on manipulation. It does not compare every representation on all five environments.
Results and Insights.
Natural language scores 0.61 on Sokoban and 0.71 on FrozenLake, versus structured dictionaries at 0.28/0.63 and symbols at 0.49/0.49. Symbols are weaker than language but are not the worst Sokoban representation. Manipulation dictionaries reach 0.94 versus 0.91 for language.
The manipulation setup already supplies object coordinates, leaving visual matching and action planning rather than pure metric reconstruction from pixels. That support helps explain the structured result. The text says subsequent manipulation studies use structured output, but Tables 2 and 5 retain the base score 0.91; the source does not fully reconcile that representation provenance.
4 Can World Modeling Help with Reward Shaping and Credit Assignment?
VAGEN-Full adds supervision of current/next-state descriptions and a hierarchy for assigning credit to the turns and tokens that produced them.
4.1 WorldModeling Reward
An initial CLIP similarity reward was judged too insensitive to fine spatial detail. The method instead uses simulator-derived state information and GPT-4.1 nano to assess observation descriptions and predictions. Depending on the protocol, the judge gives a binary answer or extracts a dictionary for rule-based comparison.
The two matching terms have weights 0.5 in the main full-method experiments. This is privileged training supervision, not a reward requiring only the agent’s images. The judgment measures selected textual relationships and can be exploited; it does not independently verify a complete causal world model.
Equation
4.2 Bi-Level General Advantage Estimation (GAE)
The first GAE pass computes a temporal-difference error and advantage at each action boundary, using turn-level discount and trace parameters. The second pass propagates that signal backward among the generated tokens within each action, with separate token-level parameters and KL penalties.
There is a source-level boundary distinction: the main prose says the final token’s advantage is initialized to the turn advantage, while Algorithm 2 adds the final token’s local TD error to the turn advantage. The complete formulas and algorithm are retained. A reproduction must choose the intended implementation rather than treating those two descriptions as identical.
Equations
4.3 VAGEN-Full Multi-Turn Reinforcement Learning Framework
A turn’s reward combines the environment reward, valid-format reward and state-reasoning reward. The actor–critic pipeline remains the base PPO system, with bi-level advantage estimation replacing the single backward GAE pass.
| Model/Method | Sokoban | FrozenLake | Navigation | PrimitiveSkill | SVG | Overall | ||||||||
| Base | Common | Average | Place | Stack | Drawer | Align | Average | Dino | DreamSim | Average | ||||
| Qwen2.5-VL-3B | 0.14 | 0.14 | 0.22 | 0.27 | 0.24 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.80 | 0.27 | 0.54 | 0.21 |
| Qwen2.5-VL-3B w/ VAGEN-Base | 0.61 | 0.71 | 0.78 | 0.80 | 0.79 | 1.00 | 0.88 | 0.88 | 0.88 | 0.91 | 0.90 | 0.65 | 0.78 | 0.76 |
| Qwen2.5-VL-3B w/ VAGEN-Full | 0.79 | 0.74 | 0.80 | 0.81 | 0.81 | 1.00 | 0.88 | 1.00 | 1.00 | 0.97 | 0.91 | 0.67 | 0.79 | 0.82 |
Equation
Experiment Setup.
Base and Full both use the combined WorldModeling output format. Full adds the judge-based reward and bi-level GAE for the four interactive environment families. SVG is an explicit exception: its reported Full configuration adds only bi-level GAE, because there is no external transition to judge. Table 21’s generic grounding/prediction weights should not override this stated exception.
Results and Insights.
Base→Full changes are Sokoban 0.61→0.79, FrozenLake 0.71→0.74, navigation 0.79→0.81, manipulation 0.91→0.97 and SVG 0.78→0.79. The overall change is 0.76→0.82. Manipulation Place stays at 1.00 and Stack at 0.88; Drawer and Align increase to 1.00.
Full exceeds the listed proprietary models on the aggregate score, but not every task: o4-mini has 0.82 FrozenLake success, and several proprietary models exceed Full’s SVG similarity. The reported scores lack test confidence intervals, repeated-seed uncertainty and a complete specification of test-set sizes/splits, limiting the strength of small differences and generalization claims.
4.4 Ablations
Training curves suggest that state-reasoning rewards help base training and that bi-level credit alone can be unstable when intermediate rewards are sparse or inaccurate. Their combination is generally strong across the shown tasks. These are training curves, not a full table of repeated-seed test ablations.
Appendix D provides additional model and FreeThink results. They show a useful but nonuniform mechanism: FreeThink improves on one listed task and is unchanged on another. Better credit assignment can also accelerate exploitation of a flawed reward.
4.5 Case Studies and Findings
Figure 5 preserves complete side-by-side cases for navigation, Sokoban and manipulation; the surrounding prose also discusses FrozenLake, which appears in the appendix’s reward-hacking cases. These are qualitative illustrations, not a measured frequency of each behavior.
Enhanced Spatial Understanding and Planning.
Selected traces illustrate spatial reasoning and action sequences, while also exposing unreliable rationales: a Base navigation response says to rotate left but emits rotateright, and Full Sokoban text can misstate the target’s position. These examples support comparing behavior and outcomes, not assuming that each rationale correctly explains the action or scene.
Response Convergence and Reduced Exploration.
Early responses are varied and verbose; later responses often reuse short templates. The authors associate this with reduced entropy and exploration. The supplied figures chiefly show task-performance curves and examples, so the claimed entropy trend is not accompanied by a dedicated quantitative entropy plot here. Concision can reflect either useful regularity or loss of state specificity.
Reward Hacking and Over-optimization.
Agents sometimes discover generic statements that satisfy the judge without predicting the specific next state. Stronger credit assignment can amplify that shortcut. Appendix D describes stricter extraction/F1 scoring and a repetition penalty; it does not provide a separate quantitative ablation establishing that these eliminate reward hacking.
5 Related Work
The method combines earlier language-model RL, multi-turn trajectory optimization and explicit visual-state reasoning. Its novelty is evaluated as a system and through selected ablations, not by proving a new general theory of world models.
RL for LLMs and VLMs.
Prior work uses human, rule-based and model-generated rewards [20][24]. Hierarchical language-agent methods [12][14] motivate separating levels of feedback. VAGEN applies a turn/token advantage scheme to visual agent trajectories with observation masking.
Multi-turn Agent Training for LLMs and VLMs.
Prompted embodied agents and modular adapters precede direct multi-turn VLM training. Related systems include GiGPO for long-horizon credit, AReaL for asynchronous training and DART for GUI agents [65][66][67]. VAGEN emphasizes what agents explicitly describe and predict while interacting.
World Model Reasoning and Visual State Reasoning.
Visual grounding and causal analyses study how models use images [68][73]. VAGEN extends the empirical question across interaction turns, using text as the state representation. Its state predictions are evaluated against environment information, rather than by demonstrating a standalone rollout simulator.
6 Conclusion and Limitations
The evidence supports structured visual reasoning and carefully delivered supervision in these short, controlled tasks. It also shows that optimizing reasoning rewards can produce generic answers that exploit the judge. Output structure, high reward and reliable internal understanding are therefore distinct outcomes.
Important limits are short horizons, supplied manipulation coordinates, privileged simulator-state rewards, task-specific settings, incomplete comparison metadata and uncertainty, and source numerical/pseudocode inconsistencies. Additional model families and supervised visual training are proposed extensions; the current experiments do not establish general real-world robot deployment.
Appendix
The appendix supplies the implementation specification, exact prompts, state representations, reward protocols and qualitative cases retained below.
Appendix
Appendix material follows the source’s original section structure.
Appendix A World Modeling for Visual State Reasoning via Multi-Turn RL
This appendix specifies the rollout/training pipeline and the task configurations used to interpret the headline results.
A.1 Problem Formulation: VLM Agent Training under a POMDP Formulation
The POMDP formulation is motivated by incomplete visual observations and the need to condition actions on interaction history.
Inherent Partial Observability in Environments.
First-person navigation hides objects outside the current view, so turning or moving can acquire information as well as make progress. This is a clear partial-observability example. A fully visible deterministic grid does not automatically have the same information constraints.
Methodological Congruence with POMDPs.
Teacher-forced updates evaluate collected trajectories with their historical context. Correct causal masking is what keeps training consistent with acting; later observations in a stored sequence should not be available to earlier token predictions.
A.2 Multi-Turn Reinforcement Learning in VLM Agentic Tasks
The rollout loop parses generated answers into environment actions, records new observations/rewards and repeats to the turn limit. The training loop computes masked PPO and critic updates over those trajectories. Figures 6–7, Algorithm 1 and the exact output formats in Table 6 are retained.
Algorithm 1 makes explicit that the next action token is used when propagating token-level credit, skipping observation tokens. Its environment-return notation is a compact description; the task reward tables include format and intermediate components as well.
| Strategy | Format |
| NoThink | <answer></answer> |
| FreeThink | <think></think> |
| <answer></answer> | |
| StateEstimation | <think> |
| <observation></observation> | |
| <reasoning></reasoning> | |
| </think> | |
| <answer></answer> | |
| TransitionModeling | <think> |
| <reasoning></reasoning> | |
| <prediction></prediction> | |
| </think> | |
| <answer></answer> | |
| WorldModeling | <think> |
| <observation></observation> | |
| <reasoning></reasoning> | |
| <prediction></prediction> | |
| </think> | |
| <answer></answer> |
A.3 VAGEN Framework
VAGEN builds on VeRL [80] and separates environment services from model training. This supports exchanging environments and RL variants. The manuscript does not report a distributed scaling study showing how throughput changes with more machines.
A.4 Environments and Tasks for VLM Agents
The action spaces and limits below define the actual evaluation scope, including the short turn horizons and privileged coordinate assistance.
Sokoban
Sokoban uses a 6×6 room and one box, with puzzles requiring at least five actions. Each response may contain three moves and the episode has three interaction turns. The underlying max_steps=100 does not imply a hundred-turn agent evaluation.
| Name | Description |
| Up | Move the agent one cell upward on the grid. |
| Left | Move the agent one cell to the left on the grid. |
| Right | Move the agent one cell to the right on the grid. |
| Down | Move the agent one cell downward on the grid. |
| Name | Value | Description |
| dim_room | Dimensions of the Sokoban grid | |
| max_steps | 100 | Maximum number of steps allowed per episode |
| num_boxes | 1 | Number of boxes to be pushed onto targets |
| min_actions_to_succeed | 5 | Minimum number of actions required to solve the puzzle |
| max_actions_per_step | 3 | Maximum number of actions the agent can take per turn |
| max_turns | 3 | Maximum number of turns the agent can interact with the environment |
FrozenLake
FrozenLake uses a random 4×4 map, at least five required moves, three actions per turn and three turns. Slipperiness is disabled, despite the exact prompts warning that tiles may slip. That prompt/configuration mismatch is preserved and should be considered in reproduction.
| Name | Description |
| Up | Move the agent one cell upward on the grid. |
| Left | Move the agent one cell to the left on the grid. |
| Right | Move the agent one cell to the right on the grid. |
| Down | Move the agent one cell downward on the grid. |
| Name | Value | Description |
| desc | None | Environment map layout (if None, randomly generated) |
| is_slippery | False | Whether the frozen tiles are slippery |
| size | 4 | Size of the square grid |
| max_actions_per_step | 3 | Maximum number of actions the agent can take per turn |
| min_actions_to_succeed | 5 | Minimum number of actions required to reach the goal |
| max_turns | 3 | Maximum number of turns the agent can interact with the environment |
Navigation
Navigation uses a single 255-resolution view, a 100° field of view, 0.5-distance movement increments and a 1.5 success threshold. Four turns allow up to five actions each. Rotations are 90° and camera tilts 30°. The two reported navigation categories are Base and Common; detailed test split counts are not supplied.
| Name | Description |
| MoveAhead | Move forward by some distance |
| MoveBack | Move backward by some distance |
| MoveRight | Move rightward by some distance |
| MoveLeft | Move leftward by some distance |
| RotateRight | Rotate to the right by 90 degrees |
| RotateLeft | Rotate to the left by 90 degrees |
| LookUp | Tilt the camera upward by 30 degrees |
| LookDown | Tilt the camera downward by 30 degrees |
| Name | Value | Description |
| resolution | 255 | Resolution of the rendered images |
| down_sample_ratio | 1.0 | Ratio for down-sampling images |
| fov | 100 | Field of view angle in degrees |
| multiview | False | Whether to use multiple camera views |
| max_actions_per_step | 5 | Maximum number of actions the agent can take per turn |
| success_threshold | 1.5 | Threshold for considering task successful |
| step_length | 0.5 | Distance traveled in a single movement action |
| max_turns | 4 | Maximum number of turns the agent can interact with the environment |
PrimitiveSkill
Manipulation exposes pick, place and push primitives, with two actions per response and three turns. Prompts supply workspace bounds and object-position lists in integer millimeters, plus task information such as drawer opening. The model matches positions to visible objects and plans primitive calls; it is not learning torque control or inferring every coordinate from the image.
| Name | Description |
| pick(x, y, z) | Grasp an object located at position (x, y, z) in the robot’s workspace |
| place(x, y, z) | Place the object currently held by the robot’s gripper at the target position (x, y, z) |
| push(x1, y1, z1, x2, y2, z2) | Push an object from position (x1, y1, z1) to position (x2, y2, z2) |
| Name | Value | Description |
| max_actions_per_step | 2 | Maximum number of actions the agent can take per turn |
| max_turns | 3 | Maximum number of turns the agent can interact with the environment |
SVG Reconstruction
SVG reconstruction uses the starvector/svg-icons-simple dataset and two turns of open-ended SVG generation. Its output is scored by rendering and comparing with the target. This is iterative visual code generation, so the world-dynamics interpretation used for physical interaction is limited.
A.5 Reward Assignment
The appendix reward specification is richer than the main text’s sparse-success shorthand. Keep environment rewards, formatting rewards and reasoning rewards separate when reproducing a condition.
Sokoban
Sokoban awards 10 for success, −0.1 at unfinished steps, 1 for placing a box, and 0.5 for valid response format. Full-method grounding and prediction each have weight 0.5.
| Reward Type | Value | Description |
| Success reward | 10 | Awarded when all boxes are placed on target locations |
| Failure penalty | -0.1 | Applied each step when the task is not completed |
| Box placement reward | 1 | Granted for each box pushed onto a target location |
| Format reward | 0.5 | Provided at each turn to encourage visual state reasoning |
| Grounding reward weight | 0.5 | Weight applied to StateEstimation reward |
| World modeling reward weight | 0.5 | Weight applied to TransitionModeling reward |
FrozenLake
FrozenLake combines success reward 10, unfinished-step penalty −0.1 and format reward 0.5. The current/next-state matching terms each have weight 0.5 when enabled.
| Reward Type | Value | Description |
| Success reward | 10 | Awarded when the agent reaches the goal position |
| Failure penalty | -0.1 | Applied each step when the task is not completed |
| Format reward | 0.5 | Provided at each turn to encourage visual state reasoning |
| Grounding reward weight | 0.5 | Weight applied to StateEstimation reward |
| World modeling reward weight | 0.5 | Weight applied to TransitionModeling reward |
Navigation
Navigation uses the same stated success, failure and formatting values, with the two reasoning terms weighted 0.5. A judge receives relative target directions/distances extracted from the simulator.
| Reward Type | Value | Description |
| Success reward | 10 | Awarded when the agent reaches the goal location |
| Failure penalty | -0.1 | Applied each step when the task is not completed |
| Format reward | 0.5 | Provided at each turn to encourage visual state reasoning |
| Grounding reward weight | 0.5 | Weight applied to StateEstimation reward |
| World modeling reward weight | 0.5 | Weight applied to TransitionModeling reward |
PrimitiveSkill
Manipulation adds a stage bonus of 2×(highest completed stage+1), in addition to success 10, unfinished-step −0.1 and format 0.5. This intermediate task supervision matters when interpreting claims about learning from sparse rewards.
| Reward Type | Value | Description |
| Success reward | 10 | Awarded when the manipulation task is completed |
| Failure penalty | -0.1 | Applied each step when the task is not completed |
| Stage-based reward | Granted upon completing key manipulation subgoals (where stage is the highest successfully completed stage) | |
| Format reward | 0.5 | Provided at each turn to encourage visual state reasoning |
| Grounding reward weight | 0.5 | Weight applied to StateEstimation reward |
| World modeling reward weight | 0.5 | Weight applied to TransitionModeling reward |
SVG
Table 21 lists DreamSim weight 5 and DINO weight 0.0001, describing the latter as effectively unused. Its generic grounding/prediction rows conflict with the explicit main-text statement that SVG Full uses only bi-level GAE; this edition follows that stated experimental exception and preserves the table for inspection.
| Reward Type | Value | Description |
| Image similarity | Variable | Weighted DreamSim [81] and DINO [82] scores measuring similarity between generated and target images |
| Format reward | 0.5 | Provided at each turn to encourage visual state reasoning |
| Grounding reward weight | 0.5 | Weight applied to StateEstimation reward |
| World modeling reward weight | 0.5 | Weight applied to TransitionModeling reward |
| DreamSim weight | 5.0 | Scaling factor applied to DreamSim similarity scores |
| Dino weight | 0.0001 | We only use DreamSim score for reward |
A.6 Evaluation Metrics
Success indicators and image-similarity functions are distinct metrics. An average across them summarizes this benchmark mix, not a calibrated probability of completing an arbitrary task.
Metrics for Task-Completion Environments
Grid, navigation and manipulation scores average binary trajectory completion. The manuscript does not supply confidence intervals or a complete test-trajectory count for every category, so fine differences should be interpreted cautiously.
Equation 1
| (1) |
Metrics for SVG Reconstruction
SVG reports mean final-output DreamSim [81] and DINO [82] similarities. The appendix describes DreamSim as a diffusion-model metric, but that description should not be used to infer the metric’s architecture; the cited perceptual-similarity method defines it. Similarity is not exact geometric or semantic correctness of the generated SVG.
Equations 2–3
| DreamSim Score | (2) | |||
| DINO Score | (3) |
Appendix B What Can We Reason About Visual States?
This appendix provides exact off-the-shelf and training prompts, backbone settings and additional learning curves.
B.1 Bottleneck of Off-the-Shelf VLMs on Agentic Tasks
The five evaluation prompts use FreeThink-style outputs and include environment guides, action constraints, examples and sample responses. They are retained in full, including the FrozenLake slipperiness mismatch and incomplete or invalid sample outputs.
Table 22 documents seven older model identifiers with temperature 0.7 and max-token settings 150/400, without fully mapping those two limits to tasks. It does not specify corresponding settings for the additional GPT-5, o3, o4-mini, Gemini 2.5 Pro and Claude 4.5 rows later present in Table 2.
| Model Name | Model ID | Provider | Max Tokens | Temperature |
| VLM-R1-3B | omlab/VLM-R1-Qwen2.5VL-3B-Math-0305 | omlab | 150/400 | 0.7 |
| Qwen2.5-VL-3B | Qwen/Qwen2.5-VL-3B-Instruct | Qwen | 150/400 | 0.7 |
| Qwen2.5-VL-7B | Qwen/Qwen2.5-VL-7B-Instruct | Qwen | 150/400 | 0.7 |
| Qwen2.5-VL-72B | Qwen/Qwen2.5-VL-72B-Instruct | Qwen | 150/400 | 0.7 |
| GPT-4o | gpt-4o | openai | 150/400 | 0.7 |
| Gemini 2.0 | gemini-2.0-flash | gemini | 150/400 | 0.7 |
| Claude 3.7 Sonnet | claude-3-7-sonnet-20250219 | claude | 150/400 | 0.7 |
B.2 Reasoning in Multi-turn RL Training
Training uses Qwen2.5-VL-3B actor and critic initializations, top-p 0.95, temperature 0.7, token discount 1, KL coefficient 0.001, learning rates 10⁻⁶/10⁻⁵, batch 128 and PPO minibatch 32. The described servers have eight H100s, 104 CPUs and 1.7 TB memory, running two experiments concurrently; sessions take roughly four to eight hours.
The full WorldModeling prompts are retained, rather than reconstructed from templates. Other formats vary the requested fields. Some printed examples themselves violate the format or action limit—for example a three-action manipulation answer despite the two-action limit, and malformed SVG/XML fragments—so they should not be treated as validated demonstrations. Turn-level discount/trace values and several optimizer details are not listed in Table 23.
| Parameter | Value | Description |
| Rollout Phase | ||
| Top-p | 0.95 | Nucleus sampling parameter for action generation |
| Temperature | 0.7 | Sampling temperature for controlling randomness |
| Update Phase | ||
| Advantage Estimator | masked_gae | Generalized Advantage Estimation with masking |
| Actor Model | Qwen/Qwen2.5-VL-3B-Instruct | Pre-trained model used for actor initialization |
| Critic Model | Qwen/Qwen2.5-VL-3B-Instruct | Pre-trained model used for critic initialization |
| 1.0 | Discount factor for token-wise advantage calculation | |
| KL Penalty Coefficient () | 0.001 | Coefficient for KL divergence penalty in PPO objective |
| Actor Learning Rate | 1e-6 | Learning rate for the actor network |
| Critic Learning Rate | 1e-5 | Learning rate for the critic network |
| Train Batch Size | 128 | Total batch size for training |
| PPO Mini Batch Size | 32 | Mini-batch size for PPO updates |
B.3 Learning Dynamics of RL Baselines
Figure 8 compares training dynamics for unmasked PPO, masked GRPO, masked turn-PPO and base VAGEN. The ranking varies with the environment; the figure supports the aggregate comparison while also showing strong navigation learning by alternatives. It does not provide a repeated-seed confidence analysis.
Appendix C How Can We Represent Internal Beliefs about the World?
Representation formats constrain the observation and prediction fields. The appendix preserves the exact grids and dictionaries so that differences in grounding difficulty remain visible.
| Environment | Format | Type | Description |
| Sokoban | Natural-Lanaguage | Text | Free-form textual descriptions of the game state |
| Symbolic | Grid | Environment-native grid-based symbols representing game entities | |
| Structured | Dictionary | JSON-like dictionary containing positions of game objects | |
| FrozenLake | Natural-Lanaguage | Text | Free-form textual descriptions of the game state |
| Symbolic | Grid | Environment-native grid-based symbols representing game entities | |
| Structured | Dictionary | JSON-like dictionary containing positions of game objects | |
| PrimitiveSkill | Natural-Lanaguage | Text | Free-form textual descriptions of the manipulation state |
| Structured | Dictionary | JSON-like dictionary containing 3D coordinates of task-relevant objects |
C.1 Symbolic Representations
Symbolic formats encode objects and agent positions as grid characters. They require the model to map pixels to a discrete layout, which can be difficult without suitable grounding training.
Sokoban Symbolic Format
Sokoban distinguishes walls, floor, targets, boxes, the player and overlapping box/player-on-target states. The retained example is a symbolic illustration, not a full specification of every generated training room.
###### #_P__# #_XO_# ###__# ######
FrozenLake Symbolic Format
FrozenLake symbols distinguish frozen cells, holes, goal, player and terminal player states. The grid example preserves the exact row arrangement; flattening it into prose would lose the representation being studied.
___G O___ __O_ _P__
C.2 Structured Representations
Structured outputs use dictionaries of positions and task entities. Their precision is useful only when the entities and coordinates are grounded correctly.
Sokoban Structured Format
The Sokoban dictionary contains player, box and target coordinates plus grid size. The source calls the example “symbolic” in the surrounding sentence, but its actual format is structured data.
{player_position: (4, 2), box_positions: [(3, 2)],
target_positions: [(3, 1)], grid_size: (6, 6)}
FrozenLake Structured Format
The FrozenLake dictionary records player, target, holes and grid dimensions. The shown player and target coincide; it is an example state, not a typical starting configuration.
{player_position: (3, 2), target_position: (3, 2),
hole_positions: [(1, 2)], grid_size: (4, 4)}
PrimitiveSkill Structured Format
Manipulation dictionaries associate object names with three-dimensional coordinates. Because the prompt supplies unlabeled position candidates, success still requires visual object matching, but the input substantially reduces metric state-estimation difficulty.
{red_cube: (100, 100, 40), green_cube: (200, 200, 60)}
Appendix D How to Improve Reasoning over World Models?
The reward appendix exposes both the exact judge criteria and the ways agents can optimize those criteria without consistently accurate reasoning.
D.1 World Modeling Reward via LLM-as-a-Judge
GPT-4.1 nano evaluates state descriptions and next-state predictions using task-specific YES/NO prompts. All eight prompts are retained. Grid criteria emphasize selected relative directions rather than exhaustive state reconstruction; some rules automatically accept goal-completed cases. Navigation prediction scoring demands explicit direction words and rejects generic “closer” descriptions. Manipulation accepts coordinate errors up to ±10 units.
These protocols use simulator state and hand-designed tolerances. They are useful supervision signals but neither calibrated state-estimation accuracy nor proof of causal reasoning. The judge is itself fallible, and the exact wording creates shortcuts that the case studies later expose.
Text-based State Information
Ground truth is serialized differently by task: pairwise spatial sentences for grids, target/visible-object dictionaries for navigation, and named coordinates for manipulation. All nine structured/grid examples across the representation and reward appendices are retained. Their source spelling and sample values are not silently repaired.
[’box0 is above and at the same column as the player’, ’target0 is above and at the same column as the player’, ’target0 is above and at the same column as box0’]
[‘target is above and at the same column as the player’, ’hole0 is above and at the same column as the player’, ’hole1 is above and on the right side of the player’, ’hole2 is above and on the right side of the player’, ’hole3 is above and on the right side of the player’, ’hole4 is at the same row and to the right of the player’]
{’target_obj_type’: ’Toaster’,
’target_distance_to_player’: 2.59,
’target_direction_to_player’: ’ahead’,
’visible_objects’: [{’type’: ’Cabinet’,
’direction_to_player’: ’left’,
’distance_to_player’: 0.94},
{’type’: ’Drawer’,
’direction_to_player’: ’left’,
’distance_to_player’: 1.18},
{’type’: ’CounterTop’,
’direction_to_player’: ’left’,
’distance_to_player’: 1.33},
...]}
{’lefz_target_position’: (80, -100, 0),
’righz_target_position’: (80, 100, 0),
’red_cube_position’: (83, -102, 20),
’green_cube_position’: (-47, 95, 20)}
D.2 Mitigating Reward Hacking via Structured Evaluation and Repetition Penalty
The mitigation first asks an LLM to extract entities and relations into a dictionary, then scores them against a reference with F1. A max-heap tracks frequent sentences. In the reported Sokoban setting, a response receives an additional −0.1 only if it is both frequent and has F1<0.7.
This targets repeated incorrect text rather than penalizing every common phrase. The appendix does not specify all frequency-tracking/extraction parameters or provide a quantitative mitigation ablation. Its claim of improved robustness is therefore more limited than proof that exploitation has been removed.
D.3 Bi-Level General Advantage Estimation (GAE) Details and Additional Results
Algorithm 2 retains the complete two-pass advantage computation, including the final-token local TD term that is omitted from the simpler initialization description in the main text.
Table 25 adds bi-level GAE to FreeThink: FrozenLake improves 0.39→0.62, but Sokoban stays 0.43. These base values differ from Table 2’s FreeThink 0.68/0.57 without an explanation of the condition change, so the two tables should not be pooled as the same run.
On Sokoban, Table 26 reports Base→Full of 0.61→0.79 for Qwen 3B, 0.63→0.92 for Qwen 7B and 0.36→0.39 for InternVL3-2B. Approximate training costs are 40 H100-hours each for the two grid tasks, 30 each for navigation/manipulation and 10 for SVG; judge usage is about 23M, 23M, 6M and 4.6M tokens respectively, with none for SVG. These are reported resource totals, not dollar prices or a throughput scaling study.
| Group / Model | Task | Method | Test Success Rate |
| Qwen2.5-VL 3B | FrozenLake | FreeThink | 0.39 |
| FrozenLake | FreeThink + Bi-Level GAE | 0.62 | |
| Qwen2.5-VL 3B | Sokoban | FreeThink | 0.43 |
| Sokoban | FreeThink + Bi-Level GAE | 0.43 |
| Group / Model | Method | Test Success Rate |
| Qwen2.5-VL 3B | VAGEN-BASE | 0.61 |
| VAGEN-FULL | 0.79 | |
| Qwen2.5-VL 7B | VAGEN-BASE | 0.63 |
| VAGEN-FULL | 0.92 | |
| InternVL3-2B | VAGEN-BASE | 0.36 |
| VAGEN-FULL | 0.39 |
| Task | H100 GPU Hours (Approx.) | LLM-as-Judge Token Cost (Approx.) |
| FrozenLake | 40 | |
| Sokoban | 40 | |
| Navigation | 30 | |
| ManiSkill | 30 | |
| SVG | 10 | N/A |
Appendix E Case Study
The complete early/late examples are retained as evidence about response behavior. They illustrate changes in phrasing and grounding, without measuring how frequently each pattern occurs across the test set.
E.1 Visual State Reasoning Enhances Spatial Understanding and Multi-Step Planning
Navigation examples recognize obstacles and plan around them; Sokoban examples reason about relative box/target positions. Figure 5 also includes manipulation. These selected comparisons illustrate the intended benefit but should not be generalized into a claim that every retained response is correct.
E.2 Response Convergence and Reduced Exploration
Four case blocks compare early and late Sokoban/navigation responses. Later outputs often reuse a directional template. Some Sokoban examples still contain questionable directions relative to their supplied state, so standardization is not equivalent to correctness. The qualitative evidence is consistent with reduced linguistic diversity; a dedicated numerical entropy analysis is not supplied here.
E.3 Reward Hacking and Over-optimization
The two FrozenLake case blocks expose generic goal claims and broad directional alternatives. Repeating that the player will reach the gift can satisfy weak judging criteria without a state-specific prediction. Better advantage propagation can optimize this shortcut more efficiently, making reward design as important as optimization strength.
Appendix F Summary of Findings
The four original finding boxes summarize the authors’ interpretation and are retained in full. The results favor explicit state reasoning, task-appropriate representations and turn-aware supervision, while the case studies show substantial limitations of judge-based rewards.
A careful reading keeps aggregate benchmark gains separate from per-task superiority, structured output separate from faithful reasoning, and reward maximization separate from grounded prediction. The source’s own final discussion recommends harder-to-exploit rewards and monitoring for templated or ungrounded responses.