Overview
This edition follows the expanded arXiv v2 dated 31 March 2026. The paper distinguishes full-benchmark evaluation from MindCube-Tiny experiments, and randomized object ordering from a separate fixed-order ablation.
At a glance
MindCube evaluates whether vision-language models can integrate a few views and answer questions about unseen spatial relationships. Its reported 21,154 questions cover camera rotation, movement around objects and occluded layouts. Strong general models remain well below the human comparison.
On the 1,050-question MindCube-Tiny subset, Qwen2.5-VL-3B improves from 37.81% without training to 57.81% after supervised map-then-reason training, then 61.33% after reinforcement learning. The corresponding gains are 20.00 and 23.52 percentage points. These results support structured intermediate representations; they do not by themselves establish human-like internal mental models.
1 Introduction
Partial views require maintaining object identity across viewpoints, inferring what lies outside the current image and simulating hypothetical movement. MindCube emphasizes these demands rather than direct single-image spatial recognition. Its taxonomy separates movement patterns, perspective changes and queried relations.
The study compares extra intermediate views, free-form verbal reasoning and JSON cognitive maps. Merely supplying more visual evidence or a ground-truth map gives limited gains for the tested frozen model. Training the model to generate a simple map and then reason over it works better. Generated text is an observable scaffold, not direct evidence of a faithful internal cognitive process.
2 MindCube Benchmark and Evaluation
The benchmark uses selected, approximately orthogonal views and template-generated multiple-choice questions. Full-benchmark model comparisons and the smaller diagnostic experiments use different sample compositions.
2.1 MindCube Benchmark
Rotation uses a nearly stationary rotating camera; Among observes a central object against different surroundings; Around tests occlusion and conversion between lateral and depth relationships. Annotation records objects, orientation, visibility and viewpoint changes before generating questions and distractors.
The reported total is 21,154 questions across 3,268 images and 976 groups. Table 1’s three setting totals are 1,081, 18,204 and 1,869 questions. Its group counts sum to 975, one fewer than the prose total. Among dominates the pooled score, so overall accuracy does not weight the three settings equally.
| Method | Overall | Rotation | Among | Around |
| Baseline | ||||
| Random (chance) | 32.35 | 36.36 | 32.29 | 30.66 |
| Random (frequency) | 33.02 | 38.30 | 32.66 | 35.79 |
| Open-Weight Multi Image Models | ||||
| LLaVA-Onevision-7B Li et al. (2024a) | 47.43 | 36.45 | 48.42 | 44.09 |
| LLaVA-Video-Qwen-7B Zhang et al. (2024d) | 41.96 | 35.71 | 43.55 | 30.12 |
| LongVA-7B Zhang et al. (2024c) | 29.46 | 35.89 | 29.55 | 24.88 |
| mPLUG-Owl3-7B-241101 Ye et al. (2024) | 44.85 | 37.84 | 47.11 | 26.91 |
| InternVL3-8B Zhu et al. (2025) | 37.50 | 26.00 | 42.03 | 36.00 |
| Qwen2.5-VL-7B-Instruct Bai et al. (2025) | 29.26 | 38.76 | 29.50 | 21.35 |
| Qwen2.5-VL-3B-Instruct Bai et al. (2025) | 33.21 | 37.37 | 33.26 | 30.34 |
| DeepSeek-VL2-Small Lu et al. (2024) | 47.62 | 37.00 | 50.38 | 26.91 |
| Gemma-3-12B-it Team et al. (2025) | 46.67 | 38.39 | 48.38 | 34.63 |
| Mantis-8B (SigLip) Jiang et al. (2024) | 41.05 | 37.65 | 40.23 | 50.99 |
| Proprietary Models | ||||
| GPT-5-2025-08-07 OpenAI (2025) | 47.59 | 93.33 | 34.17 | 41.63 |
| Gemini-2.5-pro-2025-06 Team (2025) | 47.05 | 85.50 | 25.95 | 38.40 |
| Claude-4-Sonnet-20250514 Anthropic (2025) | 44.75 | 48.42 | 44.21 | 47.62 |
| Spatial Models | ||||
| RoboBrain Ji et al. (2025) | 37.38 | 35.80 | 38.28 | 29.53 |
| SpaceMantis Chen et al. (2024a) | 22.81 | 37.65 | 21.26 | 29.32 |
| Spatial-MLLM Wu et al. (2025a) | 32.06 | 38.39 | 20.92 | 32.82 |
| Space-Qwen Chen et al. (2024a) | 33.28 | 38.02 | 33.71 | 26.32 |
2.2 Evaluation on MindCube
Across seventeen evaluated models, the best reported overall score is DeepSeek-VL2-Small’s 47.62%, versus 32.35% random chance. GPT-5 reaches 93.33% on Rotation but 34.17% on Among; no model leads all settings. Spatial finetuning is not uniformly beneficial.
The broad “near-random” description masks substantial model/category variation. The table also does not establish a uniform proprietary-over-open advantage: its best overall result is open weight. The human study uses MindCube-Tiny rather than the entire benchmark.
3 Which Scaffolds Best Guide Spatial Mental Modeling?
Frozen-model experiments ask whether an external representation or requested output format helps without updating weights. The three alternatives are illustrated in Figure 3.
1. View Interpolation. Interpolating between sparse views introduces perceptual continuity, echoing the process of mental animation (Hegarty, 1992) and supporting internal transformation such as imagined rotation. This structure scaffolds the dynamic updating capability of spatial mental models. Figure 3 shows a one-frame inserting example that replaces the original question images.
2. Augmented Cognitive Map. A cognitive map is a 2D schematic representation of object layouts in space. Such maps resemble Tversky’s cognitive collages (Tversky, 1993), and they capture locally coherent but fragmented structures. Recent studies (Yang et al., 2024; Yeh et al., 2025) on VLM-based spatial intelligence typically adopt a plain form that only encodes object positions in a top-down view. We propose an augmented variant that incorporates discrete views, with both objects and views annotated by position and orientation, thereby approaching the relational consistency of spatial mental models.
3. Free Form Reasoning. Open-ended, step-by-step natural language reasoning offers a procedural approximation of how spatial models are constructed and queried. While less rigid than map-like structures, such reasoning reflects the inferential function of spatial mental models, especially under ambiguous or incomplete observations (Tversky et al., 1994).
3.1 Data Structures as Cognitive Scaffolds for Spatial Mental Models
Intermediate views offer perceptual continuity. Cognitive maps encode an allocentric arrangement, with an augmented variant adding camera positions/orientations and a plain variant retaining objects. Free-form reasoning asks for observations, cross-view integration and question-directed inference.
These representations are inspired by cognitive maps and cognitive collages. They operationalize a useful computational scaffold rather than a neurological model of human spatial thought.
3.2 Experiment Setup
The controlled study uses Qwen2.5-VL-3B-Instruct on MindCube-Tiny: 600 Among, 250 Around and 200 Rotation questions. Table 2 retains ten input/output configurations; the frozen comparison omits map-only output because models tend to mix mapping and reasoning.
Evaluation measures answer accuracy plus map validity, coverage, directional/facing similarity and rotation-aligned isomorphism. A valid JSON structure need not describe a correct scene. Input ground-truth maps and self-generated maps are distinct conditions.
| Name | What the model receives (input) | What the model produces (output) |
| Raw-QA | Raw views + question text | Direct answer |
| VI-1 | Raw views + 1 interpolated view + question text | Direct answer |
| VI-2 | Raw views + 2 interpolated views + question text | Direct answer |
| FFR | Raw views + question text | Free-form reasoning answer |
| Aug-CGMap-In | Augmented cognitive map (objects + camera) + question text | Direct answer |
| Aug-CGMap-Out | Raw views + question text | Augmented cognitive map answer |
| Plain-CGMap-Out | Raw views + question text | Plain cognitive map answer |
| Aug-CGMap-FFR-Out | Raw views + question text | Augmented cognitive map + free-form reasoning answer |
| Plain-CGMap-FFR-Out | Raw views + question text | Plain cognitive map + free-form reasoning answer |
| CGMap-In-FFR-Out | Augmented cognitive map (objects + camera) + question text | Free-form reasoning answer |
3.3 Do Scaffolds Improve Spatial Mental Modeling Without Training?
Raw answering reaches 37.81%; one interpolated view gives 37.90%, and two give 37.81%. Supplying an augmented map without reasoning drops to 32.00%. Free-form reasoning gives 40.48%, plain generated maps plus reasoning 41.33%, and a supplied map plus reasoning 41.43%. The last number is not self-generated-map performance, despite one prose comparison suggesting otherwise.
Map format compliance is much stronger than geometric correctness. The main Table 3 chart and Appendix E.2 report inconsistent graph-metric values; both original presentations are retained. The evidence supports modest prompting gains, not reliably accurate spatial reconstructions.
| Config. | Overall | Rotation | Among | Around |
| Raw-QA | 37.81 | 34.00 | 36.00 | 45.20 |
| VI-1 | 37.90 | 35.50 | 37.33 | 41.20 |
| VI-2 | 37.81 | 35.50 | 36.50 | 42.80 |
| Aug-CGMap-In | 32.00 | 35.00 | 30.50 | 33.20 |
| FFR | 40.48 | 32.00 | 36.00 | 58.00 |
| Aug-CGMap-FFR-Out | 40.57 | 21.00 | 43.00 | 50.40 |
| Plain-CGMap-FFR-Out | 41.33 | 25.00 | 39.67 | 58.40 |
| CGMap-In-FFR-Out | 41.43 | 37.00 | 41.67 | 44.40 |
• Explicit reasoning is crucial for improving performance.
• Reasoning acts as a necessary mechanism to ground spatial structure in frozen settings.
• Passive structures (like maps as input) alone and visual continuity offer little benefit.
4 Can We Train for the Emergence of Spatial Mental Models via VLMs’ Use of Scaffolds?
Training tests whether supervised maps and reasoning chains provide stronger guidance than prompting alone, and whether reinforcement learning improves the resulting policies.
4.1 Designing a Robust Experimental Framework
The study constructs 10,000 training QA pairs with rule-based cognitive maps and manually designed reasoning templates. Maps orient the annotated front view upward. All main configurations retain the same base model and Tiny evaluation set.
Supervised finetuning compares direct answers, map-only outputs, reasoning-only outputs and joint map/reasoning outputs. Reinforcement learning uses VAGEN with GRPO, comparing base-model initialization against task-specific supervised checkpoints. The reward favors valid output format and a correct final answer.
4.2 Do the Emergence of Spatial Mental Models Truly Benefit from Explicit Training?
Direct-answer supervised finetuning reaches 52.67%, already 14.86 points above the frozen baseline. Reasoning alone gives 55.43%; plain map-only output 54.29%; plain map followed by reasoning 57.81%. Thus the joint approach adds 5.14 points over direct-answer finetuning, rather than attributing the whole 20-point gain to mapping.
Map-only training yields more geometrically exact maps but lower answer accuracy. Plain map-only isomorphism is 45.52%, versus 35.33% for map-then-reason. This dissociation supports evaluating both structure and task utility; it does not prove that every generated map is causally used.
| Config. | MindCube-Tiny QA Accuracy (%) | Generated Cognitive Map (%) | |||||
| Overall | Rotation | Among | Around | Overall Sim. | Isom. Rate | ||
| SFT | Raw-QA | 52.67 | 34.50 | 52.50 | 67.60 | – | – |
| FFR | 55.43 | 36.00 | 57.17 | 66.80 | – | – | |
| Aug-CGMap-Out | 52.48 | 30.00 | 52.17 | 71.20 | 61.28 | 13.90 | |
| Plain-CGMap-Out | 54.29 | 32.00 | 53.67 | 73.60 | 78.18 | 45.52 | |
| Aug-CGMap-FFR-Out | 54.29 | 41.50 | 52.33 | 69.20 | 61.92 | 15.33 | |
| Plain-CGMap-FFR-Out | 57.81 | 36.50 | 61.17 | 66.80 | 74.18 | 35.33 | |
| RL | RL-FFR (from scratch) | 49.52 | 26.50 | 51.50 | 63.20 | – | – |
| RL-Aug-CGMap-FFR-Out (from scratch) | 52.48 | 36.00 | 51.50 | 68.00 | 55.71 | 0.00 | |
| RL-Plain-CGMap-FFR-Out (from scratch) | 50.86 | 34.00 | 50.50 | 65.20 | 29.59 | 6.67 | |
| RL-FFR (from SFT) | 59.14 | 31.50 | 66.00 | 64.80 | – | – | |
| RL-Aug-CGMap-FFR-Out (from SFT) | 60.86 | 36.00 | 66.00 | 68.40 | 62.48 | 16.95 | |
| RL-Plain-CGMap-FFR-Out (from SFT) | 61.33 | 29.50 | 69.17 | 68.00 | 73.36 | 35.33 | |
• Joint cogmap and reasoning setting yields optimal performance through synergistic effects.
• Neither map generation nor reasoning alone matches the performance of the joint approach.
4.3 Can Reinforcement Learning Further Refine Spatial Mental Models?
Reinforcement learning from the base model reaches 49.52–52.48%, depending on format. Starting from supervised checkpoints reaches 59.14% for reasoning alone, 60.86% for augmented map/reasoning and 61.33% for plain map/reasoning.
The best joint model gains 3.52 points over its supervised predecessor. Its Rotation score falls from 36.50% to 29.50%, while Among improves markedly. Overall improvement therefore does not mean improvement in every spatial category.
4.4 Effect of Object Presentation Order in Cognitive Map Supervision
Primary results randomize object order in map supervision. Fixed spatial order supplies a predictable sequence and produces much higher reconstruction scores: plain map-only isomorphism rises from 45.52% to 89.05%. The source interprets this as a potential shortcut, rather than independent evidence of deeper spatial understanding.
Under fixed order, plain map-then-reason reaches 60.76% with supervised training; both map/reasoning RL variants reach 70.67%. These larger results belong to the separate fixed-order condition. The main 61.33% headline uses randomized order.
| Config. | Overall | Rotation | Among | Around | Overall Sim. | Isom. Rate | |
| SFT | Raw-QA | 52.28 | 34.50 | 52.50 | 66.00 | – | – |
| FFR | 53.52 | 36.00 | 54.67 | 64.80 | – | – | |
| Aug-CGMap-Out | 54.19 | 35.50 | 53.17 | 71.60 | 74.30 | 43.24 | |
| Plain-CGMap-Out | 54.38 | 35.50 | 53.50 | 71.60 | 91.73 | 89.05 | |
| Aug-CGMap-FFR-Out | 55.24 | 49.50 | 52.50 | 66.40 | 75.27 | 46.00 | |
| Plain-CGMap-FFR-Out | 60.76 | 47.50 | 62.33 | 67.60 | 88.79 | 73.81 | |
| RL | RL-Aug-CGMap-FFR-Out (from SFT) | 70.67 | 53.00 | 76.83 | 70.00 | 85.53 | 58.86 |
| RL-Plain-CGMap-FFR-Out (from SFT) | 70.67 | 48.00 | 79.17 | 68.40 | 85.79 | 71.52 |
5 Related Works
The work connects spatial-cognition benchmarks with structured maps, reasoning supervision and multiview representations. Prior spatial-memory studies, coordinate-aware prompting, geometry-aware pretraining and reconstruction methods offer complementary approaches.
MindCube specifically stresses hidden relations and viewpoint consistency. Success on its canonical, template-based layouts remains narrower than general three-dimensional reconstruction, navigation or embodied deployment.
6 Conclusion
Constructing a simple map and then reasoning about it gives the strongest main training result. Supervised initialization is particularly useful before reinforcement learning. Extra views or passive map input alone are much less reliable for the tested model.
The strongest evidence concerns this benchmark, model and training setup. Generated explanations, aggregate accuracy and imperfect map similarity cannot alone establish human-equivalent mental representations or faithful reasoning.
Ethics Statement
The authors report using anonymized public data and self-collected scenes without personal identifiers, and compensating annotators above local minimum wages. They acknowledge environmental/geographic coverage limits, computational cost and potential dual use.
Reproducibility Statement
The paper supplies project, code, dataset and checkpoint links and states that processing/training/evaluation code accompanies the supplementary material. Reproduction should track the exact version and experimental condition because several appendix counts and settings differ from the main presentation.
Acknowledgments
Support is credited to Stanford HAI, ONR grants N00014-23-1-2355, N00014-22-1-2740 and N00014-21-1-2801, and DSO National Laboratories agreement DSOCO25017.
Appendix
The complete appendix covers dataset construction, evaluation, representations and metrics, all prompts/responses, and supervised and reinforcement-learning details. Original figures and technical examples remain available below.
Appendix A The Use of Large Language Models
The authors disclose using Gemini 2.5 Pro and GPT-5 for editing and literature-search assistance, with human review and responsibility for the final content.
Appendix B MindCube Benchmark
The benchmark appendix specifies image selection, annotation, question taxonomy and examples. These constraints define what its accuracy measures can support.
B.1 Details for Data Collection and Annotation
ARKitScenes camera trajectories are filtered for linear motion or near-orthogonal rotation, followed by manual checks for sharp images and recognizable objects. Among groups from DL3DV-10K and WildRGB-D require distinct surroundings and recoverable background objects. Around selects a reference front view with sufficient scene visibility.
Annotations cover front/back and left/right relations, optional grouping of objects into semantic entities, intrinsic object orientation and four visibility levels. Questions combine those labels through templates, retaining their provenance in IDs and metadata.
The image-count breakdown is inconsistent: 2,302 public-source plus 400 self-collected images totals 2,702, not the stated 3,268. The text also mentions four dynamics while the primary benchmark reports three settings. These source discrepancies are not silently reconciled.
B.2 Details of our MindCube Benchmark
The three view settings and four label dimensions organize the benchmark’s intended reasoning demands. Their canonical orientations simplify interpretation and constrain generalization.
B.2.1 Three kinds of invisibility settings
Rotation captures two to four orthogonal directions from a nearly fixed point. Around asks models to maintain object permanence and convert left/right into front/back across occlusion. Among relates surrounding objects through a shared central anchor across four views.
These are controlled visibility problems. The stated requirement for “true 3D reasoning” is an interpretation of task design, not a proof that no alternative strategy can solve the questions.
B.2.2 Label taxonomy
Visual patterns are linear or rectilinear. Hypothetical dynamics include translation, rotation, simultaneous changes and sequential changes described in text. Relation queries distinguish agent–agent, agent–object and object–object relations; perspective labels distinguish the current view, another observer’s visibility and another observer’s relational frame.
The detailed tables show particularly weak object–object performance and substantial model/category differences. They retain older model variants such as GPT-4o and InternVL2, whereas the main table includes GPT-5 and InternVL3; the rows should not be treated as one identical evaluation roster.
| Model | Overall | Object Arrangement | Perspective Taking | |||
| Linear | Perp. | Self | Level1 | Level2 | ||
| LLaVA-Video-7B-Qwen2 | ||||||
| Mantis(SigLip) | 50.99 | |||||
| GPT-4o | ||||||
| Qwen2.5-VL-3B-Instruct | ||||||
| LongVA-7B | 39.83 | |||||
| Qwen2.5-VL-7B-Instruct | ||||||
| deepseek-vl2-small | 47.62 | 49.63 | 48.32 | |||
| Robobrain | ||||||
| Claude-sonnet-4 | 45.32 | |||||
| Space-Mantis | ||||||
| InternVL2-8B | 64.56 | |||||
| Space-Qwen | ||||||
| LLaVA-Onevision-7B | 47.43 | |||||
| Spatial-MLLM | ||||||
| mPLUG-Owl3-7B | 44.85 | |||||
| Model | Relation Pattern | Viewpoint Dynamics | ||||
| A-A | A-O | O-O | Rotation | Meanwhile | Sequence | |
| LLaVA-Video-7B-Qwen2 | ||||||
| Mantis(SigLip) | ||||||
| GPT-4o | 49.30 | |||||
| Qwen2.5-VL-3B-Instruct | ||||||
| LongVA-7B | ||||||
| Qwen2.5-VL-7B-Instruct | ||||||
| deepseek-vl2-small | 68.27 | 87.13 | ||||
| Robobrain | ||||||
| Claude-sonnet-4 | 48.42 | 34.76 | ||||
| Space-Mantis | ||||||
| InternVL2-8B | ||||||
| Space-Qwen | ||||||
| LLaVA-Onevision-7B | 29.79 | |||||
| Spatial-MLLM | ||||||
| mPLUG-Owl3-7B | ||||||
B.3 Examples
The three complete examples illustrate Among and two Around configurations, including original answer options, visibility assumptions and marked correct answers.
Appendix C Evaluation on MindCube
This appendix retains the exact evaluation prompts, caption baselines, human comparison, controlled perturbations and failure examples.
C.1 Prompt Templates for Evaluation
The baseline prompt requests one option only, with no reasoning explanation. Its exact wording is preserved for reproducibility.
Evaluation Prompt Prefix
Based on these images, answer the question based on this rule: You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ‘A. above’ from ‘ A. above B. under C. front D. behind.’, your response should only be ‘A. above’.
The Question is:
C.2 Details in text only evaluation
Replacing images with brief or spatially detailed captions reduces accuracy for three tested models. In this table, LLaVA-OneVision drops from 47.08% with images to 34.17% with brief captions or 35.92% with dense captions.
The captioning and answer prompts are preserved. This demonstrates information loss for the tested caption pipeline; it does not show that every possible textual scene representation is inferior. These baseline numbers differ from the main full-benchmark table.
Prompt for Brief Captioning
Describe this image briefly.
Prompt for Dense Captioning
Describe this image in detail, specifially focusing on the spatial relationship between objects.
Text-only evaluation Prompt Prefix
You need to gather information about each image based on the descriptions I provide below, and answer the given questions using those textual descriptions, without directly viewing the images.
Image 1: <Caption 1>
…
Image N: <Caption N>
| Model | Brief (T) | Dense (T) | Original (I) |
| RoboBrain-8B | 33.92% 7.83% | 35.58% 6.17% | 41.75% |
| LLaVA-OneVision-7B | 34.17% 12.91% | 35.92% 11.16% | 47.08% |
| Qwen2.5-VL-7B-Instruct | 27.00% 5.33% | 28.75% 3.58% | 32.33% |
C.3 Human Evaluation
Five annotators independently answer the Tiny benchmark, averaging 94.55% with a 94.20–94.77% range, compared with the reported GPT-4o score of 36.54%. This is a large gap on the sampled tasks, not a claim about all human or model spatial abilities.
C.4 Evaluation Setup
The evaluation roster spans native multi-image models, interleaved image/text models, proprietary APIs and spatially finetuned variants. Model versions differ between the main and some appendix tables, so comparisons must retain their specific row labels.
C.5 Analysis in settings
Controlled Around and Among tests probe sensitivity to visibility, camera height, appearance and reference-object count.
C.5.1 Around
The source reports a 50.7% decrease between fully visible and fully occluded conditions, a 28.7% drop across the mostly-visible/mostly-occluded boundary, and 24.8% higher accuracy from elevated views. It gives p < 0.01 for the first comparison but does not provide enough protocol/statistical detail here to reconstruct those estimates.
Illustrated failures suggest reliance on visible correspondences and local proximity. That diagnosis is plausible, but selected explanations do not uniquely identify the model’s internal mechanism.
C.5.2 Among
Attribute invariance changes the central object’s appearance without moving the surrounding layout. Quantity sensitivity adds central reference objects. Paired answers are grouped as both correct, both wrong, or one correct and one wrong.
SpaceQwen is reported to have roughly 40% inconsistency, and extra reference objects do not reliably help. The plot legend uses “Consistent Changes/Wrong Way/Inconsistent Changes,” while the prose defines correctness-based categories; the original plot and definitions are retained without conflating those labels.
C.6 Failure case analysis
Selected GPT-4o responses misidentify which views correspond or reverse the subject of a “behind” relation. The examples motivate global scene integration, but do not measure the prevalence of those failure mechanisms.
Appendix D Data Structures as Cognitive Scaffolds, Evaluation Metrics, and Input-Output Configurations
This section defines the representations and graph metrics, then provides complete prompts for each input/output condition.
D.1 Data Structures as Cognitive Scaffolds
The cognitive-science motivation treats spatial models as useful, sometimes incomplete representations that support transformations and inference. The computational study tests view interpolation, structured maps and verbal reasoning as observable approximations.
D.1.1 View Interpolation for Dynamic Updating
One or two intermediate frames are inserted between consecutive input views. Frames are selected heuristically from source videos when available; otherwise Stable Virtual Camera synthesizes them. Thus the condition can mix retrieved and generated imagery, with potential synthesis artifacts.
D.1.2 Cognitive Maps for Integrated Spatial Layouts
Maps use a 10×10 top-down grid, from [0,0] at the upper left to [9,9] at the lower right. Object records contain positions and optional facing labels; augmented maps additionally encode camera positions and directions. “Inner/outer” denote orientation perpendicular to the grid.
Plain maps omit camera records. JSON is an interface for text-generating models and a measurable output representation, rather than a claim to reproduce neural cognitive maps or metric-accurate 3D geometry.
D.1.3 Free Form Reasoning
Reasoning templates proceed through individual-view observation, recognition of shared objects across views, and question-directed spatial inference or imagined movement. These are supervised/generated explanations; their presence alone does not establish faithfulness to the model’s computation.
D.2 Evaluation Metrics
Answer accuracy measures task completion; graph metrics separately assess whether the intermediate map matches the annotated scene.
D.2.1 QA Accuracy
QA accuracy is the fraction of questions with the correct final answer, expressed as a percentage. It does not score whether a supporting explanation is correct.
Equation
D.2.2 Graph Metrics for Cognitive Maps
Graph evaluation checks format, aligns permitted rotations, compares object relations and orientations, and tests exact relational agreement. Syntax validity and geometric accuracy are separate outcomes.
Notation.
Each map contains named objects with two-dimensional positions and optional facing labels. Shared objects form the intersection used by several similarity terms.
Extended directional relation.
The directional relation uses the dominant coordinate difference to assign left/right/up/down, with vertical direction winning equal-magnitude ties. Near-coincident objects can receive inner/outer labels using facing annotations and a 0.5 distance threshold. The original piecewise definition and relation-matrix equation are retained.
Equations
Coverage.
Coverage is the fraction of ground-truth objects retrieved. Extra generated objects are not penalized by this term alone.
Equation
Directional similarity.
Directional similarity measures agreement over defined ground-truth relations between shared objects. Missing-object coverage should therefore be read alongside this relation score.
Equations
Facing similarity.
Facing similarity compares orientations for shared objects whose ground-truth facing is defined. The printed formula does not specify the empty-denominator convention.
Equations
Overall similarity.
Overall similarity weights directional agreement by 0.7 and facing agreement by 0.3. Coverage is reported separately; it does not appear in this printed weighted formula.
Equation
Rotation-invariant isomorphism.
Isomorphism requires complete relation-matrix agreement after one of the permitted rotations: right-angle turns about z and a right-angle turn about x or y. This is stricter than high average similarity, but remains a test of the chosen relational representation.
Equation
Graph validity.
A valid map must parse as JSON, meet field constraints and contain at least one valid object position. Such minimal validity does not imply complete object coverage or a correct layout.
D.3 Prompts for All Input-Output Configurations
All original prompt instructions, image examples, JSON templates and answer-format requirements follow. The specified object categories and viewpoint descriptors are part of the experimental input.
D.3.1 Example for Raw-QA
Raw-QA requests only a tagged multiple-choice answer from the four views.
Example Prompt for Raw-QA
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.2 Example for FFR
FFR adds a step-by-step explanation inside think tags before the final tagged answer.
Example Prompt for FFR: Free-Form Reasoning
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
Please do step by step reasoning first, then give your final answer. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should be this format: ’<think>(replace with your reasoning here)</think><answer>A. Above</answer>’.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.3 Example for VI-1 and VI-2
VI-1 and VI-2 expand four original viewpoints into eight or twelve images, updating the question’s image indices while retaining direct answering.
Prompt for VI-1: View Interpolation with 1 Frame
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these 8 images showing the white jar from different viewpoints (from front (image 1) to left (image 3), from left (image 3) to back (image 5), from back (image 5) to right (image 7), from right (image 7) back to front (image 1)), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 7, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
Prompt for VI-2: View Interpolation with 2 Frames
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these 12 images showing the white jar from different viewpoints (from front (image 1) to left (image 4), from left (image 4) to back (image 7), from back (image 7) to right (image 10), from right (image 10) back to front (image 1)), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 10, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.4 Example for Aug-CGMap-In
Aug-CGMap-In supplies the annotated object and camera map as additional context. The prompt still describes examining the images; the supplied map is privileged structured information.
Prompt for Aug-CGMap-In: Grounded Augmented Cognitive Map as Input
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. Also, we provide you a cognitive map that shows the general layout for the scene. Please use the cognitive map to reason and answer the question.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Cognitive Map Format]
We provide you a 2D grid map of the scene that is related to the question you should answer. Below is the description of the map:
- The map uses a 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- The map is shown in the bird’s view
- Directions are defined as:
* up = towards the top of the grid (decreasing y-value)
* right = towards the right of the grid (increasing x-value)
* down = towards the bottom of the grid (increasing y-value)
* left = towards the left of the grid (decreasing x-value)
* inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
* outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- "objects" lists all important items in the scene with their positions
- "facing" indicates which direction an object is oriented towards (when applicable)
- "views" represents the different camera viewpoints in the scene
Below is the cognitive map of the scene related to the question. Please use it to reason and answer the question.
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [5, 5]},
{"name": "bed sheet with a floral pattern",
"position": [5, 8]},
{"name": "white headboard", "position": [2, 5]},
{"name": "clothes rack", "position": [5, 2]},
{"name": "table with cups on it", "position": [8, 5]}
],
"views": [
{"name": "Image 1", "position": [5, 6], "facing": "up"},
{"name": "Image 2", "position": [4, 5], "facing": "right"},
{"name": "Image 3", "position": [5, 4], "facing": "down"},
{"name": "Image 4", "position": [6, 5], "facing": "left"}
]
}
‘‘‘
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.5 Example for Aug-CGMap-Out
Aug-CGMap-Out gives object categories and asks for a generated object/camera map followed by the answer, without explicitly requesting a reasoning chain.
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
- outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations accordingly
- Include camera positions and directions for each view
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your answer**:
‘‘‘json
{
"objects": [
{"name": "object_name", "position": [x, y],
"facing": "direction"},
{"name": "object_without_orientation", "position": [x, y]}
],
"views": [
{"name": "View/Image 1", "position": [x, y],
"facing": "direction"},
{"name": "View/Image 2", "position": [x, y],
"facing": "direction"}
]
}
‘‘‘
2. Next, provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above".
3. In general, your response’s format should be like "Based on my observation, the answer is:
<cogmap>(Replace with your cogmap here)</cogmap><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.6 Example for Plain-CGMap-Out
Plain-CGMap-Out requests only the object map and final answer. The original schema/example includes placeholders and comments and is preserved as a prompt template, not executable JSON.
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Object positions must maintain accurate relative spatial relationships
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:
‘‘‘json
{
"object_category_1": {"position": [x, y]},
"object_category_2": {"position": [x, y],
"facing": "direction"},
# if the object is asked for orientation
...
}
‘‘‘
2. Next, provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is:
<cogmap>(Replace with your cogmap here)</cogmap><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.7 Example for Plain-CGMap-FFR-Out
Plain-CGMap-FFR-Out requests an object map, then detailed reasoning, then the selected answer. This is the main successful training configuration.
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Object positions must maintain accurate relative spatial relationships
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:
‘‘‘json
{
"object_category_1": {"position": [x, y]},
"object_category_2": {"position": [x, y],
"facing": "direction"},
# if the object is asked for orientation
...
}
‘‘‘
2. Next, please also provide your reasons step by step in details, then provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is:
<cogmap>(Replace with your cogmap here)</cogmap><think>(Replace with your reasoning here)</think><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.8 Example for Aug-CGMap-FFR-Out
The augmented joint prompt additionally requires camera positions and directions. Its extra fields increase the mapping burden.
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
- outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations accordingly
- Include camera positions and directions for each view
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:
‘‘‘json
{
"objects": [
{"name": "object_name", "position": [x, y],
"facing": "direction"},
{"name": "object_without_orientation", "position": [x, y]}
],
"views": [
{"name": "View/Image 1", "position": [x, y],
"facing": "direction"},
{"name": "View/Image 2", "position": [x, y],
"facing": "direction"}
]
}
‘‘‘
2. Next, please also provide your reasons step by step in details, then provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is:
<cogmap>(Replace with your cogmap here)</cogmap><think>(Replace with your reasoning here)</think><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
D.3.9 Example for CGMap-In-FFR-Out
CGMap-In-FFR-Out supplies a ground-truth augmented map and requests reasoning before the answer, separating map use from map generation.
[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. Also, we provide you a cognitive map that shows the general layout for the scene. Please use the cognitive map to reason and answer the question.
[Answer Instruction]
Please do step by step reasoning first, then give your final answer. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should be this format: ’<think>(replace with your reasoning here)</think><answer>A. Above</answer>’.
[Cognitive Map Format]
We provide you a 2D grid map of the scene that is related to the question you should answer. Below is the description of the map:
- The map uses a 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- The map is shown in the bird’s view
- Directions are defined as:
* up = towards the top of the grid (decreasing y-value)
* right = towards the right of the grid (increasing x-value)
* down = towards the bottom of the grid (increasing y-value)
* left = towards the left of the grid (decreasing x-value)
* inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
* outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- "objects" lists all important items in the scene with their positions
- "facing" indicates which direction an object is oriented towards (when applicable)
- "views" represents the different camera viewpoints in the scene
Below is the cognitive map of the scene related to the question. Please use it to reason and answer the question.
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [5, 5]},
{"name": "bed sheet with a floral pattern",
"position": [5, 8]},
{"name": "white headboard", "position": [2, 5]},
{"name": "clothes rack", "position": [5, 2]},
{"name": "table with cups on it", "position": [8, 5]}
],
"views": [
{"name": "Image 1", "position": [5, 6], "facing": "up"},
{"name": "Image 2", "position": [4, 5], "facing": "right"},
{"name": "Image 3", "position": [5, 4], "facing": "down"},
{"name": "Image 4", "position": [6, 5], "facing": "left"}
]
}
‘‘‘
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard
Appendix E Which Scaffolds Best Guide Spatial Thinking in Unchanged VLMs?
Frozen-model examples expose the distinction between formatting a map and constructing useful geometry. The complete outputs, including errors and source color markings, are retained.
E.1 VLM Response Examples for Configurations in Section D.3
On the illustrated jar question, several direct-answer conditions select the correct option, while generated maps contain out-of-range coordinates, collapsed camera positions or missing objects. Some accompanying explanations contradict their final answers. These cases illustrate failures rather than representative success rates.
C. Bed sheet with a floral pattern
B. Clothes rack
C. Bed sheet with a floral pattern
<think>To determine what is to the left of the white jar from the viewpoint presented in image 4, let’s analyze the scene:
- In image 4, the white jar is placed on a table covered with a pink tablecloth featuring heart patterns.
- The table has a glass top and is positioned against a wall.
- To the left of the jar, there is a piece of furniture that appears to be a bed or a bed frame.
Given this information, the object to the left of the white jar is most likely a bed or a bed frame. </think>
<answer>C. Bed sheet with a floral pattern</answer>
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [150, 446],
"facing": "left"},
{"name": "bed sheet with a floral pattern",
"position": [0, 317],
"facing": "left"},
{"name": "clothes rack", "position": [0, 108],
"facing": "left"},
{"name": "table with cups on it", "position": [188, 200],
"facing": "left"},
{"name": "white headboard", "position": [188, 317],
"facing": "left"}
],
"views": [
{"name": "Image 1", "position": [0, 0], "facing": "left"},
{"name": "Image 2", "position": [0, 0], "facing": "left"},
{"name": "Image 3", "position": [0, 0], "facing": "left"},
{"name": "Image 4", "position": [0, 0], "facing": "left"}
]
}
‘‘‘
<CogMap>
<Your cognitive map>
<Reasoning>
From the viewpoint presented in image 4, we can see that the white jar is positioned in the foreground, facing left. The bed sheet with a floral pattern is located directly behind the white jar, covering the lower part of the image. The clothes rack is visible on the left side of the image, slightly above the bed sheet. The table with cups on it is situated further back, to the right of the white jar. The white headboard is also visible, but it is not as prominent as the other objects due to its position behind the bed sheet.
Therefore, my answer is A. Table with cups on it.
</Answer>
Therefore, my answer is A. Table with cups on it.
‘‘‘json
{
"white_jar": {"position": [150, 440], "facing": "left"},
"table_with_cups_on_it": {"position": [187, 260],
"facing": "right"}
}
‘‘‘
<CogMap>
<Your cognitive map>
<Reasoning>
From the viewpoint presented in image 4, we can see that the white jar is positioned on the bed sheet with a floral pattern. The table with cups on it is located to the right of the white jar, as indicated by its position in the image. The clothes rack is not visible in this particular view, and the white headboard is not directly related to the white jar or the table with cups on it. Therefore, the correct answer is A. Table with cups on it.
<Answer>
Therefore, my answer is A. Table with cups on it.
C. Bed sheet with a floral pattern
E.2 Additional Graph Metrics for Generated Graphs
Appendix Table 5 reports validity of 92.57% for augmented and 94.95% for plain maps, but exact isomorphism of only 0.10% and 7.43%. Augmented maps have higher weighted similarity, 51.12% versus 37.44%, despite lower exact agreement.
These values differ from the main Table 3 chart, which shows similarities 41.14/36.86 and isomorphism 11.45/16.64. The source does not explain the discrepancy; all values remain visible in their original contexts.
E.3 Further Analysis on View Interpolation
The denser-view analysis excludes Rotation because the source data do not support more than five interpolation steps. On the remaining settings, plain map/reasoning with raw views scores 47.41%, versus the direct-answer interpolation peak of 46.47%. These figures are not the full Tiny scores.
View density effects vary by model: Qwen2.5-VL-7B improves substantially at four inserted views, while Qwen3-VL-8B is inconsistent. GPT-5 and Qwen3-VL-235B generally decline as more views are added. The study demonstrates no universal benefit from additional frames, rather than universal harm.
| Method (Qwen2.5VL-3B) | VI-0 (Raw) | VI-1 | VI-2 | VI-3 | VI-4 | VI-5 | VI-6 | VI-7 |
| Plain-CGMap-FFR-Out | 47.41 | 44.94 | 44.59 | 43.18 | 44.35 | 43.41 | 42.82 | 45.28 |
| RawQA | 43.76 | 46.47 | 45.53 | 44.94 | 44.35 | 44.24 | 45.18 | 44.59 |
| FFR | 45.53 | 43.71 | 43.83 | 41.65 | 41.00 | 40.36 | 40.89 | 40.77 |
E.4 Explicit Reasoning with Visual-of-Thought
A separate tool-use experiment renders explicit spatial sketches and returns them to the model for interpretation.
Explicit Visual Sketching via External Tools.
The model either receives a rendered ground-truth grid or generates drawing coordinates for Matplotlib, then reasons from the resulting image. Overall accuracy rises from 41.43% to 42.10% for supplied maps, and from 41.33% to 43.13% for generated-map reasoning.
The benefit is not uniform by setting: some Rotation or Around comparisons decline. This tool-mediated result extends the representation study; it is distinct from the main supervised map-then-reason policy.
| Configuration | Map Modality | Overall | Rotation | Among | Around |
| Raw-QA | - | 37.81 | 34.00 | 36.00 | 45.20 |
| CGMap-In (Original) | Text (JSON) | 41.43 | 37.00 | 41.67 | 44.40 |
| Img-CGMap-In (New) | Visual (Grid) | 42.10 | 31.50 | 44.17 | 45.60 |
| Plain-CGMap (Original) | Text (JSON) | 41.33 | 25.00 | 39.67 | 58.40 |
| Img-CGMap (New) | Visual (Grid) | 43.13 | 32.75 | 41.13 | 55.06 |
Appendix F Can We Teach VLMs to Build and Leverage Spatial Representations?
Supervised experiments use annotated layouts and templated reasoning chains to teach structured outputs. The appendix supplies data-generation rules, training settings and ablations.
F.1 Supervised Fine-Tuning Data Curation
Maps and reasoning chains provide complementary supervision: a schematic scene representation and a procedure for answering a question about it.
F.1.1 Cognitive Map Generation
A script reads setting labels and scene metadata, places objects on a canonical 10×10 grid, adds optional facing and camera records, and serializes the map as JSON. Around uses linear object layouts, Among a central cross-like arrangement, and Rotation a central camera with changing orientation. The code also handles a translation setting.
These targets encode relational templates rather than measured metric reconstructions. Algorithm 1 preserves the full Require/Ensure specification and 25 numbered steps.
F.1.2 Free-Form Reasoning Generation
Manually designed templates describe visible objects, establish shared scene identity and apply the query’s perspective change or hypothetical motion. Figures 14–16 retain both template placeholders and filled examples for all three principal settings.
F.2 Detailed Experimental Setup
Table 10 specifies 10,000 QA pairs, three epochs, full-parameter tuning, learning rate 10⁻⁵, cosine scheduling and effective batch size 256 on two H100 GPUs. Images use 784–90,000 pixels and sequences up to 8,192 tokens, with BF16, AdamW and ZeRO Stage 3.
The text reconciles a four-GPU script default with the claimed two-GPU runs. However, the later hyperparameter table calls batch 512 the original configuration. These details are inconsistent, so the edition preserves both instead of presenting one as fully resolved.
| Parameter | Value |
| Dataset size | 10,000 QA pairs |
| Epochs | 3 |
| Learning rate | 1e-5 |
| Scheduler | Cosine |
| Fine-tuning type | Full-parameter |
| Batch Size | 256 |
| GPUs used | 2 × NVIDIA H100 |
| Max image resolution | 90,000 pixels |
| Min image resolution | 784 pixels |
| Model Max Length | 8192 tokens |
| Weight Decay | 0 |
| Warmup Ratio | 0.03 |
| Max Grad Norm | 1 |
| Precision | BF16 |
| Optimizer | AdamW |
F.3 VLM Response Examples After SFT for Configurations in Section D.3
The complete supervised outputs show that an apparently correct coordinate map can coexist with a wrong final answer or contradictory verbal reasoning. In the jar example, augmented map-only answers correctly, whereas several reasoning variants answer incorrectly. These selected examples qualify broad claims that more explanation necessarily improves an individual prediction.
<answer>D. White headboard</answer>
Based on my observation, the answer is:
<cogmap>
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [5, 5]},
{"name": "bed sheet with a floral pattern",
"position": [5, 8]},
{"name": "white headboard", "position": [2, 5]},
{"name": "clothes rack", "position": [5, 2]},
{"name": "table with cups on it", "position": [8, 5]}
],
"views": [
{"name": "Image 1", "position": [5, 6], "facing": "up"},
{"name": "Image 2", "position": [4, 5], "facing": "right"},
{"name": "Image 3", "position": [5, 4], "facing": "down"},
{"name": "Image 4", "position": [6, 5], "facing": "left"}
]
}
’’’
</cogmap><answer>C. Bed sheet with a floral pattern</answer>
Based on my observation, the answer is:
<cogmap>
‘‘‘json
{
"white jar": {"position": [5, 5]},
"bed sheet with a floral pattern": {"position": [5, 8]},
"white headboard": {"position": [2, 5]},
"clothes rack": {"position": [5, 2]},
"table with cups on it": {"position": [8, 5]}
}
’’’
</cogmap><answer>B. Clothes rack </answer>
<think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the bed sheet with a floral pattern. In image 3, I can see white jar in front of the clothes rack. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view bed sheet with a floral pattern behind white jar in the second view, it implies that in the first view, bed sheet with a floral pattern is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As clothes rack is observed behind white jar in the third view, it means that in the first view, clothes rack is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: table with cups on it is to the right of white jar, bed sheet with a floral pattern is to my behind, clothes rack is to the left of white jar. So the answer is B. Clothes rack</think><answer>B. Clothes rack</answer>
Based on my observation, the answer is:
<cogmap>
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [5, 5]},
{"name": "bed sheet with a floral pattern",
"position": [5, 8]},
{"name": "white headboard", "position": [2, 5]},
{"name": "clothes rack", "position": [5, 2]},
{"name": "table with cups on it", "position": [8, 5]}
],
"views": [
{"name": "Image 1", "position": [5, 6], "facing": "up"},
{"name": "Image 2", "position": [4, 5], "facing": "right"},
{"name": "Image 3", "position": [5, 4], "facing": "down"},
{"name": "Image 4", "position": [6, 5], "facing": "left"}
]
}
’’’
</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the clothes rack. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view clothes rack behind white jar in the second view, it implies that in the first view, clothes rack is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, bed sheet with a floral pattern is to my behind, table with cups on it is to the left of white jar. So the answer is A. Table with cups on it.</think><answer>A. Table with cups on it</answer>
Based on my observation, the answer is:
<cogmap>
‘‘‘json
{
"white jar": {"position": [5, 5]},
"bed sheet with a floral pattern": {"position": [5, 8]},
"white headboard": {"position": [2, 5]},
"clothes rack": {"position": [5, 2]},
"table with cups on it": {"position": [8, 5]}
}
’’’
</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the clothes rack. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view clothes rack behind white jar in the second view, it implies that in the first view, clothes rack is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, bed sheet with a floral pattern is to my behind, table with cups on it is to the left of white jar. So the answer is A. Table with cups on it.</think><answer>A. Table with cups on it</answer>
F.4 Detailed Graph Metric Results for SFT Graph-Related Experiments
The four training curves separate graph convergence from answer accuracy. Map-only training learns structure quickly, while joint plain-map/reasoning training reaches the highest final QA score despite lower exact isomorphism. The curves do not include run-to-run uncertainty estimates.
F.5 Which Part of VLM is the Bottleneck for Spatial Understanding?
At step 57 in Raw-QA training, updating only the language model gives 51.43%, versus 52.67% for all modules and 37.81% for vision-only tuning or no tuning. This supports language-side adaptation in this experiment.
It does not establish that perception is never a bottleneck, nor does the table quantify the claimed computational saving. The conclusion is limited to the tested model, checkpoint and task.
| Training Method | Overall | Rotation | Among | Around |
| Raw-QA (no fine-tuning) | 37.81 | 34.00 | 36.00 | 45.20 |
| Freeze LLM (Vision Encoder Only) | 37.81 | 30.50 | 37.00 | 45.60 |
| Freeze Vision Encoder (LLM Only) | 51.43 | 34.00 | 50.00 | 68.80 |
| Tune All Parts | 52.67 | 34.50 | 52.50 | 67.60 |
F.6 Branching from Raw-QA SFT Checkpoint
A separate two-stage experiment starts from a direct-answer checkpoint and continues with scaffold inputs while still requesting only the final option. Its own baseline is 46.36%; map-input and augmented-map/reasoning conditions reach 49.00%, and reasoning reaches 46.82%.
These numbers use a different baseline from the main supervised table and do not outperform its 52.67% Raw-QA result. The stated curriculum benefit should be interpreted within this separate protocol.
| Method | Overall | Rotation | Among | Around |
| Raw-QA | 46.36 | 33.50 | 51.20 | 46.75 |
| Aug-CGMap-In | 49.00 | 35.50 | 53.20 | 50.50 |
| FF Rsn | 46.82 | 37.00 | 50.60 | 47.00 |
| Aug-CGMap-FFR-Out | 49.00 | 37.00 | 53.20 | 49.75 |
F.7 Hyperparameter Tuning Results
The tuning sweep prefers learning rate 10⁻⁵, batch 512 and warmup 0.03 among seven tested choices, reaching 52.67%. The PDF’s learning-rate multipliers 1, 2 and 4 are restored where HTML omitted them. Batch 512 here conflicts with Table 10’s 256; no broader optimum is established.
F.8 Effect of Removing Viewpoint Descriptors
Removing textual viewpoint descriptors tests possible language shortcuts. Frozen plain map/reasoning reaches 42.67%, versus Raw-QA’s 38.57%. Under supervised training, the best observed scores are 54.0% and 52.8%, a smaller 1.2-point gap.
The supervised table selects each configuration’s best checkpoint at different steps, rather than one common final checkpoint. The advantage persists in this test but is substantially narrower than the primary supervised result.
| Config. | Overall | Rotation | Among | Around |
| Raw-QA | 38.57 | 34.00 | 36.83 | 46.40 |
| Aug-CGMap-In | 32.00 | 34.50 | 30.50 | 33.60 |
| FFR | 39.14 | 32.50 | 33.83 | 57.20 |
| Aug-CGMap-FFR-Out | 38.95 | 19.50 | 41.00 | 49.60 |
| Plain-CGMap-FFR-Out | 42.67 | 28.50 | 39.83 | 60.80 |
| CGMap-In-FFR-Out | 40.67 | 34.50 | 42.67 | 40.80 |
Appendix G Can Reinforcement Learning Further Refine Spatial Thought Processes?
The reinforcement-learning appendix details initialization, rollout limits and sparse reward components, plus complete illustrative outputs.
G.1 Detailed Experimental Setup
Each RL configuration trains for half an epoch with batch 32, prompt/response limits of 1,024/512 tokens, eight trajectories per rollout and one turn. Reported actor and critic learning rates are 10⁻⁶ and 10⁻⁵. Valid format earns +1 and a correct answer +5.
The appendix enumerates only three configurations, while the main table reports six, including plain-map variants and reasoning initialized from SFT. “From scratch” here means from the pretrained base VLM without task SFT, not random weights.
G.2 RL Reward Design Ablation
The combined reward gives 49.52%, answer-only reward 50.76%, and format-only reward 45.05%, compared with a frozen 40.57% baseline in this ablation. Answer correctness matters more than format reward; the combined choice is not the highest-scoring tested reward.
G.3 VLM Response Examples After RL for Configurations in Section D.3
The retained examples contrast wrong free-form and base-initialized augmented-map answers with a correct answer after supervised initialization. The uninitialized map uses out-of-grid coordinates and repeated camera positions; the supervised map uses the intended schema. One selected successful output does not demonstrate general map faithfulness.
<think>From image 4, the white jar is positioned on a surface with a floral-patterned bed sheet. To the left of the jar, there is a table with a pink tablecloth featuring heart shapes and a Hello Kitty design. There are also cups on the table. Therefore, the object to the left of the white jar is the table with the cups.</think><answer>A. Table with cups on it</answer>
<cogmap>
{
"objects": [
{"name": "white jar", "position": [265, 436],
"facing": "left"}
],
"views": [
{"name": "View/Image 1", "position": [265, 436],
"facing": "left"},
{"name": "View/Image 2", "position": [265, 436],
"facing": "left"},
{"name": "View/Image 3", "position": [265, 436],
"facing": "left"},
{"name": "View/Image 4", "position": [265, 436],
"facing": "left"}
]
}
<Your cognitive map></cogmap>
<think>The white jar is located in the center of the 10x10 grid, facing to the left. The clothes rack is to the left of the white jar, as seen in the image 2 view. The bed sheet with a floral pattern is located to the right of the white jar, as seen in the image 1 view. The white headboard is not visible in any of the provided images.
Based on the provided images, the white jar is located in the center of the 10x10 grid, facing to the left. The clothes rack is to the left of the white jar, as seen in the image 2 view. The bed sheet with a floral pattern is located to the right of the white jar, as seen in the image 1 view. The white headboard is not visible in any of the provided images.</think><answer>A. Table with cups on it.</answer>
Based on my observation, the answer is:
<cogmap>
‘‘‘json
{
"objects": [
{"name": "white jar", "position": [5, 5]},
{"name": "bed sheet with a floral pattern",
"position": [5, 8]},
{"name": "white headboard", "position": [2, 5]},
{"name": "clothes rack", "position": [5, 2]},
{"name": "table with cups on it", "position": [8, 5]}
],
"views": [
{"name": "Image 1", "position": [5, 6], "facing": "up"},
{"name": "Image 2", "position": [4, 5], "facing": "right"},
{"name": "Image 3", "position": [5, 4], "facing": "down"},
{"name": "Image 4", "position": [6, 5], "facing": "left"}
]
}
’’’
</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the clothes rack. In image 2, I can see white jar in front of the table with cups on it. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view table with cups on it behind white jar in the second view, it implies that in the first view, table with cups on it is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, table with cups on it is to my behind, bed sheet with a floral pattern is to the left of white jar. So the answer is C. Bed sheet with a floral pattern.</think><answer>C. Bed sheet with a floral pattern</answer>