Literature
Display
← All papers

MindCube: Spatial Mental Modeling from Limited Views

Qineng Wang · Baiqiao Yin · Pingyue Zhang · Jianshu Zhang · Kangrui Wang · Zihan Wang · Jieyu Zhang · Keshigeyan Chandrasegaran · Han Liu · Ranjay Krishna · Saining Xie · Li Fei-Fei · Jiajun Wu · Manling Li

· ICLR 2026 ↗

Computer vision18 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the expanded arXiv v2 dated 31 March 2026. The paper distinguishes full-benchmark evaluation from MindCube-Tiny experiments, and randomized object ordering from a separate fixed-order ablation.

At a glance

MindCube evaluates whether vision-language models can integrate a few views and answer questions about unseen spatial relationships. Its reported 21,154 questions cover camera rotation, movement around objects and occluded layouts. Strong general models remain well below the human comparison.

On the 1,050-question MindCube-Tiny subset, Qwen2.5-VL-3B improves from 37.81% without training to 57.81% after supervised map-then-reason training, then 61.33% after reinforcement learning. The corresponding gains are 20.00 and 23.52 percentage points. These results support structured intermediate representations; they do not by themselves establish human-like internal mental models.

1 Introduction

Partial views require maintaining object identity across viewpoints, inferring what lies outside the current image and simulating hypothetical movement. MindCube emphasizes these demands rather than direct single-image spatial recognition. Its taxonomy separates movement patterns, perspective changes and queried relations.

The study compares extra intermediate views, free-form verbal reasoning and JSON cognitive maps. Merely supplying more visual evidence or a ground-truth map gives limited gains for the tested frozen model. Training the model to generate a simple map and then reason over it works better. Generated text is an observable scaffold, not direct evidence of a faithful internal cognitive process.

Complete original main1 graphic.
Figure 1: Top: VLMs cannot maintain a coherent mental model when evaluating on the MindCube benchmark. Bottom: We study how we can help build spatial mental models through external (scaling of views, cognitive map input) and internal strategies (fine-tuning, cognitive map elicitation). We find joint cognitive map and reasoning setting yields the highest gain (+23.52%+23.52\%).
Complete original main2 graphic.
Figure 2: MindCube taxonomy and examples. Left: Three camera movement patterns (Rotation, Around, Among) with corresponding spatial QA examples. Right: Four-dimensional taxonomy categorizing MindCube questions types.

2 MindCube Benchmark and Evaluation

The benchmark uses selected, approximately orthogonal views and template-generated multiple-choice questions. Full-benchmark model comparisons and the smaller diagnostic experiments use different sample compositions.

2.1 MindCube Benchmark

Rotation uses a nearly stationary rotating camera; Among observes a central object against different surroundings; Around tests occlusion and conversion between lateral and depth relationships. Annotation records objects, orientation, visibility and viewpoint changes before generating questions and distractors.

The reported total is 21,154 questions across 3,268 images and 976 groups. Table 1’s three setting totals are 1,081, 18,204 and 1,869 questions. Its group counts sum to 975, one fewer than the prose total. Among dominates the pooled score, so overall accuracy does not weight the three settings equally.

Table 1: Left: MindCube data statistics. The number next to the setting (Rotation, Among, Around) means the total QA pairs. Numbers next to each dataset (e.g., Arkitscenes) mean QA pairs/image groups. For example, “865/53” for Arkitscenes in Rotation means 865 QA pairs and 53 image groups from it. Right: Performance of VLMs on MindCube. Dark blue indicates the best result among all models and light blue indicates the second best result among all models.
Complete original table1-graphic graphic.
Method Overall Rotation Among Around
Baseline
Random (chance) 32.35 36.36 32.29 30.66
Random (frequency) 33.02 38.30 32.66 35.79
Open-Weight Multi Image Models
LLaVA-Onevision-7B Li et al. (2024a) 47.43 36.45 48.42 44.09
LLaVA-Video-Qwen-7B Zhang et al. (2024d) 41.96 35.71 43.55 30.12
LongVA-7B Zhang et al. (2024c) 29.46 35.89 29.55 24.88
mPLUG-Owl3-7B-241101 Ye et al. (2024) 44.85 37.84 47.11 26.91
InternVL3-8B Zhu et al. (2025) 37.50 26.00 42.03 36.00
Qwen2.5-VL-7B-Instruct Bai et al. (2025) 29.26 38.76 29.50 21.35
Qwen2.5-VL-3B-Instruct Bai et al. (2025) 33.21 37.37 33.26 30.34
DeepSeek-VL2-Small Lu et al. (2024) 47.62 37.00 50.38 26.91
Gemma-3-12B-it Team et al. (2025) 46.67 38.39 48.38 34.63
Mantis-8B (SigLip) Jiang et al. (2024) 41.05 37.65 40.23 50.99
Proprietary Models
GPT-5-2025-08-07 OpenAI (2025) 47.59 93.33 34.17 41.63
Gemini-2.5-pro-2025-06 Team (2025) 47.05 85.50 25.95 38.40
Claude-4-Sonnet-20250514 Anthropic (2025) 44.75 48.42 44.21 47.62
Spatial Models
RoboBrain Ji et al. (2025) 37.38 35.80 38.28 29.53
SpaceMantis Chen et al. (2024a) 22.81 37.65 21.26 29.32
Spatial-MLLM Wu et al. (2025a) 32.06 38.39 20.92 32.82
Space-Qwen Chen et al. (2024a) 33.28 38.02 33.71 26.32

2.2 Evaluation on MindCube

Across seventeen evaluated models, the best reported overall score is DeepSeek-VL2-Small’s 47.62%, versus 32.35% random chance. GPT-5 reaches 93.33% on Rotation but 34.17% on Among; no model leads all settings. Spatial finetuning is not uniformly beneficial.

The broad “near-random” description masks substantial model/category variation. The table also does not establish a uniform proprietary-over-open advantage: its best overall result is open weight. The human study uses MindCube-Tiny rather than the entire benchmark.

3 Which Scaffolds Best Guide Spatial Mental Modeling?

Frozen-model experiments ask whether an external representation or requested output format helps without updating weights. The three alternatives are illustrated in Figure 3.

Complete original main3 graphic.
Figure 3: Grounded examples of our three data structures that approximate spatial mental models.

1. View Interpolation. Interpolating between sparse views introduces perceptual continuity, echoing the process of mental animation (Hegarty, 1992) and supporting internal transformation such as imagined rotation. This structure scaffolds the dynamic updating capability of spatial mental models. Figure 3 shows a one-frame inserting example that replaces the original question images.

2. Augmented Cognitive Map. A cognitive map is a 2D schematic representation of object layouts in space. Such maps resemble Tversky’s cognitive collages (Tversky, 1993), and they capture locally coherent but fragmented structures. Recent studies (Yang et al., 2024; Yeh et al., 2025) on VLM-based spatial intelligence typically adopt a plain form that only encodes object positions in a top-down view. We propose an augmented variant that incorporates discrete views, with both objects and views annotated by position and orientation, thereby approaching the relational consistency of spatial mental models.

3. Free Form Reasoning. Open-ended, step-by-step natural language reasoning offers a procedural approximation of how spatial models are constructed and queried. While less rigid than map-like structures, such reasoning reflects the inferential function of spatial mental models, especially under ambiguous or incomplete observations (Tversky et al., 1994).

3.1 Data Structures as Cognitive Scaffolds for Spatial Mental Models

Intermediate views offer perceptual continuity. Cognitive maps encode an allocentric arrangement, with an augmented variant adding camera positions/orientations and a plain variant retaining objects. Free-form reasoning asks for observations, cross-view integration and question-directed inference.

These representations are inspired by cognitive maps and cognitive collages. They operationalize a useful computational scaffold rather than a neurological model of human spatial thought.

3.2 Experiment Setup

The controlled study uses Qwen2.5-VL-3B-Instruct on MindCube-Tiny: 600 Among, 250 Around and 200 Rotation questions. Table 2 retains ten input/output configurations; the frozen comparison omits map-only output because models tend to mix mapping and reasoning.

Evaluation measures answer accuracy plus map validity, coverage, directional/facing similarity and rotation-aligned isomorphism. A valid JSON structure need not describe a correct scene. Input ground-truth maps and self-generated maps are distinct conditions.

Table 2: Input–output configurations used in all experiments. The suffix “-In” means the cognitive map is given to the model as input, whereas “-Out” means the cognitive map is predicted as an intermediate output before answering. “Aug” indicates maps with object and camera annotations; “Plain” indicates maps without these augmentations. VI = View Interpolation, CGMap = Cognitive Map, FFR = Free-form reasoning. Figure 3 shows visual examples of input structures.
Name What the model receives (input) What the model produces (output)
Raw-QA Raw views + question text Direct answer
VI-1 Raw views + 1 interpolated view + question text Direct answer
VI-2 Raw views + 2 interpolated views + question text Direct answer
FFR Raw views + question text Free-form reasoning →\rightarrow answer
Aug-CGMap-In Augmented cognitive map (objects + camera) + question text Direct answer
Aug-CGMap-Out Raw views + question text Augmented cognitive map →\rightarrow answer
Plain-CGMap-Out Raw views + question text Plain cognitive map →\rightarrow answer
Aug-CGMap-FFR-Out Raw views + question text Augmented cognitive map + free-form reasoning →\rightarrow answer
Plain-CGMap-FFR-Out Raw views + question text Plain cognitive map + free-form reasoning →\rightarrow answer
CGMap-In-FFR-Out Augmented cognitive map (objects + camera) + question text Free-form reasoning →\rightarrow answer

3.3 Do Scaffolds Improve Spatial Mental Modeling Without Training?

Raw answering reaches 37.81%; one interpolated view gives 37.90%, and two give 37.81%. Supplying an augmented map without reasoning drops to 32.00%. Free-form reasoning gives 40.48%, plain generated maps plus reasoning 41.33%, and a supplied map plus reasoning 41.43%. The last number is not self-generated-map performance, despite one prose comparison suggesting otherwise.

Map format compliance is much stronger than geometric correctness. The main Table 3 chart and Appendix E.2 report inconsistent graph-metric values; both original presentations are retained. The evidence supports modest prompting gains, not reliably accurate spatial reconstructions.

Table 3: Left: QA accuracy (%) of Qwen2.5-VL-3B-Instruct on the MindCube-Tiny benchmark under different configs for frozen VLMs. Right: Graph metrics for two cog map output settings.
Config. Overall Rotation Among Around
Raw-QA 37.81 34.00 36.00 45.20
VI-1 37.90↑\uparrow 35.50 37.33 41.20
VI-2 37.81−- 35.50 36.50 42.80
Aug-CGMap-In 32.00↓\downarrow 35.00 30.50 33.20
FFR 40.48↑\uparrow 32.00 36.00 58.00
Aug-CGMap-FFR-Out 40.57↑\uparrow 21.00 43.00 50.40
Plain-CGMap-FFR-Out 41.33↑\uparrow 25.00 39.67 58.40
CGMap-In-FFR-Out 41.43↑\uparrow 37.00 41.67 44.40
Complete original table3-chart graphic.
Key Takeaways: Scaffolding Spatial Mental Models in Frozen VLMs

• Explicit reasoning is crucial for improving performance.

• Reasoning acts as a necessary mechanism to ground spatial structure in frozen settings.

• Passive structures (like maps as input) alone and visual continuity offer little benefit.

4 Can We Train for the Emergence of Spatial Mental Models via VLMs’ Use of Scaffolds?

Training tests whether supervised maps and reasoning chains provide stronger guidance than prompting alone, and whether reinforcement learning improves the resulting policies.

4.1 Designing a Robust Experimental Framework

The study constructs 10,000 training QA pairs with rule-based cognitive maps and manually designed reasoning templates. Maps orient the annotated front view upward. All main configurations retain the same base model and Tiny evaluation set.

Supervised finetuning compares direct answers, map-only outputs, reasoning-only outputs and joint map/reasoning outputs. Reinforcement learning uses VAGEN with GRPO, comparing base-model initialization against task-specific supervised checkpoints. The reward favors valid output format and a correct final answer.

Complete original main4 graphic.
Figure 4: SFT per 5 step training performance on task accuracy and graph metrics.

4.2 Do the Emergence of Spatial Mental Models Truly Benefit from Explicit Training?

Direct-answer supervised finetuning reaches 52.67%, already 14.86 points above the frozen baseline. Reasoning alone gives 55.43%; plain map-only output 54.29%; plain map followed by reasoning 57.81%. Thus the joint approach adds 5.14 points over direct-answer finetuning, rather than attributing the whole 20-point gain to mapping.

Map-only training yields more geometrically exact maps but lower answer accuracy. Plain map-only isomorphism is 45.52%, versus 35.33% for map-then-reason. This dissociation supports evaluating both structure and task utility; it does not prove that every generated map is causally used.

Table 4: QA accuracy (%) and cognitive map generation quality of Qwen2.5-VL-3B-Instruct under both SFT and RL on MindCube-Tiny. FFR refers to free-form reasoning. Bolded means the best within that training category (SFT or RL).
Config. MindCube-Tiny QA Accuracy (%) Generated Cognitive Map (%)
Overall Rotation Among Around Overall Sim. Isom. Rate
SFT Raw-QA 52.67 34.50 52.50 67.60 – –
FFR 55.43↑\uparrow 36.00 57.17 66.80 – –
Aug-CGMap-Out 52.48 30.00 52.17 71.20 61.28 13.90
Plain-CGMap-Out 54.29↑\uparrow 32.00 53.67 73.60 78.18 45.52
Aug-CGMap-FFR-Out 54.29↑\uparrow 41.50 52.33 69.20 61.92 15.33
Plain-CGMap-FFR-Out 57.81↑\uparrow 36.50 61.17 66.80 74.18 35.33
RL RL-FFR (from scratch) 49.52 26.50 51.50 63.20 – –
RL-Aug-CGMap-FFR-Out (from scratch) 52.48 36.00 51.50 68.00 55.71 0.00
RL-Plain-CGMap-FFR-Out (from scratch) 50.86 34.00 50.50 65.20 29.59 6.67
RL-FFR (from SFT) 59.14 31.50 66.00 64.80 – –
RL-Aug-CGMap-FFR-Out (from SFT) 60.86 36.00 66.00 68.40 62.48 16.95
RL-Plain-CGMap-FFR-Out (from SFT) 61.33 29.50 69.17 68.00 73.36 35.33
Key Takeaways: Explicit Training for the Emergence of Spatial Mental Models

• Joint cogmap and reasoning setting yields optimal performance through synergistic effects.

• Neither map generation nor reasoning alone matches the performance of the joint approach.

4.3 Can Reinforcement Learning Further Refine Spatial Mental Models?

Reinforcement learning from the base model reaches 49.52–52.48%, depending on format. Starting from supervised checkpoints reaches 59.14% for reasoning alone, 60.86% for augmented map/reasoning and 61.33% for plain map/reasoning.

The best joint model gains 3.52 points over its supervised predecessor. Its Rotation score falls from 36.50% to 29.50%, while Among improves markedly. Overall improvement therefore does not mean improvement in every spatial category.

Key Takeaways: Reinforcement Learning for the Emergence of Spatial Mental Models

• Combining cognitive maps with reasoning consistently improves all learning outcomes.

• Starting from scratch, RL provides only marginal gains for spatial reasoning; its true power is unlocked when building upon a strong SFT foundation.

4.4 Effect of Object Presentation Order in Cognitive Map Supervision

Primary results randomize object order in map supervision. Fixed spatial order supplies a predictable sequence and produces much higher reconstruction scores: plain map-only isomorphism rises from 45.52% to 89.05%. The source interprets this as a potential shortcut, rather than independent evidence of deeper spatial understanding.

Under fixed order, plain map-then-reason reaches 60.76% with supervised training; both map/reasoning RL variants reach 70.67%. These larger results belong to the separate fixed-order condition. The main 61.33% headline uses randomized order.

Table 5: SFT and RL results under fixed spatial object order. The map-then-reason approach (Plain-CGMap-FFR-Out) remains the best-performing configuration, consistent with the randomized-order results in Table 4.
Config. Overall Rotation Among Around Overall Sim. Isom. Rate
SFT Raw-QA 52.28 34.50 52.50 66.00 – –
FFR 53.52↑\uparrow 36.00 54.67 64.80 – –
Aug-CGMap-Out 54.19↑\uparrow 35.50 53.17 71.60 74.30 43.24
Plain-CGMap-Out 54.38↑\uparrow 35.50 53.50 71.60 91.73 89.05
Aug-CGMap-FFR-Out 55.24↑\uparrow 49.50 52.50 66.40 75.27 46.00
Plain-CGMap-FFR-Out 60.76↑\uparrow 47.50 62.33 67.60 88.79 73.81
RL RL-Aug-CGMap-FFR-Out (from SFT) 70.67 53.00 76.83 70.00 85.53 58.86
RL-Plain-CGMap-FFR-Out (from SFT) 70.67 48.00 79.17 68.40 85.79 71.52
Complete original main5 graphic.
Figure 5: SFT training dynamics under fixed spatial object order. Compared to randomized order (Figure 4), the overall learning trends are consistent: Plain-CGMap-FFR-Out achieves the highest QA accuracy despite not producing the most structurally perfect maps.

5 Related Works

The work connects spatial-cognition benchmarks with structured maps, reasoning supervision and multiview representations. Prior spatial-memory studies, coordinate-aware prompting, geometry-aware pretraining and reconstruction methods offer complementary approaches.

MindCube specifically stresses hidden relations and viewpoint consistency. Success on its canonical, template-based layouts remains narrower than general three-dimensional reconstruction, navigation or embodied deployment.

6 Conclusion

Constructing a simple map and then reasoning about it gives the strongest main training result. Supervised initialization is particularly useful before reinforcement learning. Extra views or passive map input alone are much less reliable for the tested model.

The strongest evidence concerns this benchmark, model and training setup. Generated explanations, aggregate accuracy and imperfect map similarity cannot alone establish human-equivalent mental representations or faithful reasoning.

Ethics Statement

The authors report using anonymized public data and self-collected scenes without personal identifiers, and compensating annotators above local minimum wages. They acknowledge environmental/geographic coverage limits, computational cost and potential dual use.

Reproducibility Statement

The paper supplies project, code, dataset and checkpoint links and states that processing/training/evaluation code accompanies the supplementary material. Reproduction should track the exact version and experimental condition because several appendix counts and settings differ from the main presentation.

Acknowledgments

Support is credited to Stanford HAI, ONR grants N00014-23-1-2355, N00014-22-1-2740 and N00014-21-1-2801, and DSO National Laboratories agreement DSOCO25017.

Appendix

The complete appendix covers dataset construction, evaluation, representations and metrics, all prompts/responses, and supervised and reinforcement-learning details. Original figures and technical examples remain available below.

Appendix A The Use of Large Language Models

The authors disclose using Gemini 2.5 Pro and GPT-5 for editing and literature-search assistance, with human review and responsibility for the final content.

Appendix B MindCube Benchmark

The benchmark appendix specifies image selection, annotation, question taxonomy and examples. These constraints define what its accuracy measures can support.

B.1 Details for Data Collection and Annotation

ARKitScenes camera trajectories are filtered for linear motion or near-orthogonal rotation, followed by manual checks for sharp images and recognizable objects. Among groups from DL3DV-10K and WildRGB-D require distinct surroundings and recoverable background objects. Around selects a reference front view with sufficient scene visibility.

Annotations cover front/back and left/right relations, optional grouping of objects into semantic entities, intrinsic object orientation and four visibility levels. Questions combine those labels through templates, retaining their provenance in IDs and metadata.

The image-count breakdown is inconsistent: 2,302 public-source plus 400 self-collected images totals 2,702, not the stated 3,268. The text also mentions four dynamics while the primary benchmark reports three settings. These source discrepancies are not silently reconciled.

Refer to caption
(a) camera pose 1
Refer to caption
(b) camera pose 2
Figure 1: Examples of camera poses in ArkitScenes
Complete original app2 graphic.
Figure 2: MindCube Bench construction pipeline.
Complete original app3 graphic.
Figure 3: Example of different question-related label combinations to generate QA pairs.

B.2 Details of our MindCube Benchmark

The three view settings and four label dimensions organize the benchmark’s intended reasoning demands. Their canonical orientations simplify interpretation and constrain generalization.

B.2.1 Three kinds of invisibility settings

Rotation captures two to four orthogonal directions from a nearly fixed point. Around asks models to maintain object permanence and convert left/right into front/back across occlusion. Among relates surrounding objects through a shared central anchor across four views.

These are controlled visibility problems. The stated requirement for “true 3D reasoning” is an interpretation of task design, not a proof that no alternative strategy can solve the questions.

B.2.2 Label taxonomy

Visual patterns are linear or rectilinear. Hypothetical dynamics include translation, rotation, simultaneous changes and sequential changes described in text. Relation queries distinguish agent–agent, agent–object and object–object relations; perspective labels distinguish the current view, another observer’s visibility and another observer’s relational frame.

The detailed tables show particularly weak object–object performance and substantial model/category differences. They retain older model variants such as GPT-4o and InternVL2, whereas the main table includes GPT-5 and InternVL3; the rows should not be treated as one identical evaluation roster.

Table 1: Performance of VLMs on MindCube across categories.(Part 1)
Model Overall Object Arrangement Perspective Taking
Linear Perp. Self Level1 Level2
LLaVA-Video-7B-Qwen2 41.9641.96 30.1230.12 43.1143.11 42.1942.19 60.7660.76 33.8033.80
Mantis(SigLip) 41.0441.04 50.99 40.0840.08 41.2041.20 54.4354.43 35.4135.41
GPT-4o 38.8138.81 29.1629.16 39.7539.75 39.0739.07 46.2046.20 31.8631.86
Qwen2.5-VL-3B-Instruct 33.2133.21 30.3430.34 33.4933.49 32.9632.96 46.8446.84 36.2836.28
LongVA-7B 29.4629.46 24.8824.88 29.9129.91 28.8128.81 51.9051.90 39.83
Qwen2.5-VL-7B-Instruct 29.2629.26 21.3521.35 30.0230.02 28.7728.77 46.8446.84 36.8136.81
deepseek-vl2-small 47.62 26.9126.91 49.63 48.32 56.3356.33 31.1131.11
Robobrain 37.3837.38 29.5329.53 38.1438.14 37.5637.56 55.0655.06 30.5730.57
Claude-sonnet-4 44.7544.75 47.6247.62 44.4844.48 45.32 49.3849.38 31.7431.74
Space-Mantis 22.8222.82 29.3229.32 22.1922.19 22.1522.15 45.5745.57 33.4833.48
InternVL2-8B 18.6818.68 13.1113.11 19.2219.22 17.8917.89 64.56 27.9927.99
Space-Qwen 33.2833.28 26.3226.32 33.9533.95 33.0633.06 46.8446.84 35.6335.63
LLaVA-Onevision-7B 47.43 44.0944.09 47.7547.75 48.0448.04 51.2751.27 33.4833.48
Spatial-MLLM 32.0632.06 20.9220.92 33.1333.13 31.7931.79 46.8446.84 35.2035.20
mPLUG-Owl3-7B 44.85 26.9126.91 46.5946.59 45.1545.15 60.1360.13 35.7435.74
Table 2: Performance of VLMs on MindCube across categories.(Part 2)
Model Relation Pattern Viewpoint Dynamics
A-A A-O O-O Rotation Meanwhile Sequence
LLaVA-Video-7B-Qwen2 36.2236.22 57.6157.61 26.6726.67 35.7135.71 30.1230.12 73.4573.45
Mantis(SigLip) 23.7823.78 64.1664.16 25.2425.24 37.6537.65 24.9924.99 82.7482.74
GPT-4o 49.30 48.3848.38 16.7016.70 32.6532.65 31.0931.09 59.7359.73
Qwen2.5-VL-3B-Instruct 37.8537.85 37.5137.51 20.6520.65 37.3737.37 27.8827.88 46.0546.05
LongVA-7B 19.7219.72 35.4935.49 25.5825.58 35.8935.89 24.6724.67 40.5040.50
Qwen2.5-VL-7B-Instruct 31.4131.41 34.6734.67 15.6315.63 38.7638.76 22.8722.87 43.7643.76
deepseek-vl2-small 43.9843.98 68.27 25.3325.33 37.0037.00 32.9732.97 87.13
Robobrain 30.9430.94 49.1849.18 27.3727.37 35.8035.80 28.7928.79 59.6659.66
Claude-sonnet-4 41.7841.78 67.2567.25 15.8515.85 48.42 34.76 69.5369.53
Space-Mantis 28.1828.18 17.0317.03 20.8920.89 37.6537.65 24.9824.98 14.4614.46
InternVL2-8B 15.6715.67 12.4712.47 24.5824.58 36.4536.45 21.7821.78 7.367.36
Space-Qwen 31.5931.59 38.1438.14 26.1326.13 38.0238.02 28.5128.51 44.5844.58
LLaVA-Onevision-7B 42.2842.28 65.8765.87 29.79 36.4536.45 33.8033.80 84.3884.38
Spatial-MLLM 27.7227.72 37.7537.75 25.8025.80 38.3938.39 26.8426.84 44.1944.19
mPLUG-Owl3-7B 47.8047.80 62.2962.29 18.8318.83 37.8437.84 31.0231.02 81.5581.55

B.3 Examples

The three complete examples illustrate Among and two Around configurations, including original answer options, visibility assumptions and marked correct answers.

Complete original app4 graphic.
Figure 4: Example of among setting.
Complete original app5 graphic.
Figure 5: Example-1 of around setting.
Complete original app6 graphic.
Figure 6: Example-2 of around setting.

Appendix C Evaluation on MindCube

This appendix retains the exact evaluation prompts, caption baselines, human comparison, controlled perturbations and failure examples.

C.1 Prompt Templates for Evaluation

The baseline prompt requests one option only, with no reasoning explanation. Its exact wording is preserved for reproducibility.

Evaluation Prompt Prefix
Based on these images, answer the question based on this rule: You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ‘A. above’ from ‘ A. above B. under C. front D. behind.’, your response should only be ‘A. above’.

The Question is:

C.2 Details in text only evaluation

Replacing images with brief or spatially detailed captions reduces accuracy for three tested models. In this table, LLaVA-OneVision drops from 47.08% with images to 34.17% with brief captions or 35.92% with dense captions.

The captioning and answer prompts are preserved. This demonstrates information loss for the tested caption pipeline; it does not show that every possible textual scene representation is inferior. These baseline numbers differ from the main full-benchmark table.

Prompt for Brief Captioning
Describe this image briefly.

Prompt for Dense Captioning
Describe this image in detail, specifially focusing on the spatial relationship between objects.

Text-only evaluation Prompt Prefix
You need to gather information about each image based on the descriptions I provide below, and answer the given questions using those textual descriptions, without directly viewing the images.

Image 1: <Caption 1>

…

Image N: <Caption N>

Table 3: Text-only (T) evaluation vs. original evaluation with image inputs (I). The results highlight a significant performance drop when the original image input is replaced with the corresponding text-based caption, particularly with the brief captions. In all cases, model performance decreases notably, underscoring that our benchmark is vision-centric.
Model Brief (T) Dense (T) Original (I)
RoboBrain-8B 33.92% ↓\downarrow7.83% 35.58% ↓\downarrow6.17% 41.75%
LLaVA-OneVision-7B 34.17% ↓\downarrow12.91% 35.92% ↓\downarrow11.16% 47.08%
Qwen2.5-VL-7B-Instruct 27.00% ↓\downarrow5.33% 28.75% ↓\downarrow3.58% 32.33%

C.3 Human Evaluation

Five annotators independently answer the Tiny benchmark, averaging 94.55% with a 94.20–94.77% range, compared with the reported GPT-4o score of 36.54%. This is a large gap on the sampled tasks, not a claim about all human or model spatial abilities.

Table 4: Comparison of Human and GPT-4 Performance (%)
Model/Annotator GPT4-o Human-max Human-min Human-avg
Accuracy 36.54 94.77 94.20 94.55

C.4 Evaluation Setup

The evaluation roster spans native multi-image models, interleaved image/text models, proprietary APIs and spatially finetuned variants. Model versions differ between the main and some appendix tables, so comparisons must retain their specific row labels.

C.5 Analysis in settings

Controlled Around and Among tests probe sensitivity to visibility, camera height, appearance and reference-object count.

C.5.1 Around

The source reports a 50.7% decrease between fully visible and fully occluded conditions, a 28.7% drop across the mostly-visible/mostly-occluded boundary, and 24.8% higher accuracy from elevated views. It gives p < 0.01 for the first comparison but does not provide enough protocol/statistical detail here to reconstruct those estimates.

Illustrated failures suggest reliance on visible correspondences and local proximity. That diagnosis is plausible, but selected explanations do not uniquely identify the model’s internal mechanism.

Complete original app7 graphic.
Figure 7: Example of complex spatial arrangement in around setting.
Complete original app8 graphic.
Figure 8: Example of different camera heights’ impact in around setting.

C.5.2 Among

Attribute invariance changes the central object’s appearance without moving the surrounding layout. Quantity sensitivity adds central reference objects. Paired answers are grouped as both correct, both wrong, or one correct and one wrong.

SpaceQwen is reported to have roughly 40% inconsistency, and extra reference objects do not reliably help. The plot legend uses “Consistent Changes/Wrong Way/Inconsistent Changes,” while the prose defines correctness-based categories; the original plot and definitions are retained without conflating those labels.

Refer to caption
Figure 9: Paired question-answers inconsistency in two tests. We report the proportions of IC, CC and WW. Notably, SpaceQwen has a highest inconsistency(around 40%). GPT-4o and LLaVA-Onevision exhibit more balanced performance.
Refer to caption
Figure 10: Examples in Attribute Invariance Test.
Refer to caption
Figure 11: Examples in Quantity Sensitivity Test.

C.6 Failure case analysis

Selected GPT-4o responses misidentify which views correspond or reverse the subject of a “behind” relation. The examples motivate global scene integration, but do not measure the prevalence of those failure mechanisms.

Complete original app12 graphic.
Figure 12: Failure case analysis. We show GPT4-o’s reasoning process. In case 1, the model is unable to establish the spatial location corresponding to each view; In case 2, the model confuses the subject of the “behind” relationship.

Appendix D Data Structures as Cognitive Scaffolds, Evaluation Metrics, and Input-Output Configurations

This section defines the representations and graph metrics, then provides complete prompts for each input/output condition.

D.1 Data Structures as Cognitive Scaffolds

The cognitive-science motivation treats spatial models as useful, sometimes incomplete representations that support transformations and inference. The computational study tests view interpolation, structured maps and verbal reasoning as observable approximations.

D.1.1 View Interpolation for Dynamic Updating

One or two intermediate frames are inserted between consecutive input views. Frames are selected heuristically from source videos when available; otherwise Stable Virtual Camera synthesizes them. Thus the condition can mix retrieved and generated imagery, with potential synthesis artifacts.

D.1.2 Cognitive Maps for Integrated Spatial Layouts

Maps use a 10×10 top-down grid, from [0,0] at the upper left to [9,9] at the lower right. Object records contain positions and optional facing labels; augmented maps additionally encode camera positions and directions. “Inner/outer” denote orientation perpendicular to the grid.

Plain maps omit camera records. JSON is an interface for text-generating models and a measurable output representation, rather than a claim to reproduce neural cognitive maps or metric-accurate 3D geometry.

D.1.3 Free Form Reasoning

Reasoning templates proceed through individual-view observation, recognition of shared objects across views, and question-directed spatial inference or imagined movement. These are supervised/generated explanations; their presence alone does not establish faithfulness to the model’s computation.

D.2 Evaluation Metrics

Answer accuracy measures task completion; graph metrics separately assess whether the intermediate map matches the annotated scene.

D.2.1 QA Accuracy

QA accuracy is the fraction of questions with the correct final answer, expressed as a percentage. It does not score whether a supporting explanation is correct.

Equation
QA Accuracy=NcorrectNtotal×100%\text{QA Accuracy}=\frac{N_{\text{correct}}}{N_{\text{total}}}\times 100\%

D.2.2 Graph Metrics for Cognitive Maps

Graph evaluation checks format, aligns permitted rotations, compares object relations and orientations, and tests exact relational agreement. Syntax validity and geometric accuracy are separate outcomes.

Notation.

Each map contains named objects with two-dimensional positions and optional facing labels. Shared objects form the intersection used by several similarity terms.

Extended directional relation.

The directional relation uses the dominant coordinate difference to assign left/right/up/down, with vertical direction winning equal-magnitude ties. Near-coincident objects can receive inner/outer labels using facing annotations and a 0.5 distance threshold. The original piecewise definition and relation-matrix equation are retained.

Equations
dir⁡(oi,oj)={right|xj−xi|>|yj−yi|​ and ​xj>xi,left|xj−xi|>|yj−yi|​ and ​xj<xi,down|yj−yi|≥|xj−xi|​ and ​yj>yi,up|yj−yi|≥|xj−xi|​ and ​yj<yi,inner‖pj−pi‖2<δ​ and ​(fi=inner∨fj=outer),outer‖pj−pi‖2<δ​ and ​(fi=outer∨fj=inner),∅otherwise,\mathrm{dir}(o_{i},o_{j})=\begin{cases}\text{right}&|x_{j}-x_{i}|\textgreater|y_{j}-y_{i}|\text{ and }x_{j}\textgreater x_{i},\\ \text{left}&|x_{j}-x_{i}|\textgreater|y_{j}-y_{i}|\text{ and }x_{j}<x_{i},\\ \text{down}&|y_{j}-y_{i}|\geq|x_{j}-x_{i}|\text{ and }y_{j}\textgreater y_{i},\\ \text{up}&|y_{j}-y_{i}|\geq|x_{j}-x_{i}|\text{ and }y_{j}<y_{i},\\ \text{inner}&\|p_{j}-p_{i}\|_{2}<\delta\text{ and }(f_{i}=\text{inner}\ \lor\ f_{j}=\text{outer}),\\ \text{outer}&\|p_{j}-p_{i}\|_{2}<\delta\text{ and }(f_{i}=\text{outer}\ \lor\ f_{j}=\text{inner}),\\ \varnothing&\text{otherwise},\end{cases}
R⁡(oi,oj)=dir⁡(oi,oj).R(o_{i},o_{j})=\mathrm{dir}(o_{i},o_{j}).

Coverage.

Coverage is the fraction of ground-truth objects retrieved. Extra generated objects are not penalized by this term alone.

Equation
Cov=|𝒪c||𝒪⋆|∈[0,1].\mathrm{Cov}=\frac{|\mathcal{O}^{c}|}{|\mathcal{O}^{\star}|}\in[0,1].

Directional similarity.

Directional similarity measures agreement over defined ground-truth relations between shared objects. Missing-object coverage should therefore be read alongside this relation score.

Equations
𝒫⋆={(oi,oj)∈𝒪c×𝒪c∣i≠j,R⋆(oi,oj)≠∅}.\mathcal{P}^{\star}=\{(o_{i},o_{j})\in\mathcal{O}^{c}\times\mathcal{O}^{c}\mid i\neq j,\;R^{\star}(o_{i},o_{j})\neq\varnothing\}.
Sdir=|{(oi,oj)∈𝒫⋆∣Rg​(oi,oj)=R⋆​(oi,oj)}||𝒫⋆|∈[0,1],S_{\text{dir}}=\frac{\bigl|\{(o_{i},o_{j})\in\mathcal{P}^{\star}\mid R^{g}(o_{i},o_{j})=R^{\star}(o_{i},o_{j})\}\bigr|}{|\mathcal{P}^{\star}|}\in[0,1],

Facing similarity.

Facing similarity compares orientations for shared objects whose ground-truth facing is defined. The printed formula does not specify the empty-denominator convention.

Equations
ℱ⋆={oi∈𝒪c∣fi⋆≠∅}.\mathcal{F}^{\star}=\{o_{i}\in\mathcal{O}^{c}\mid f^{\star}_{i}\neq\varnothing\}.
Sface=|{oi∈ℱ⋆∣fig=fi⋆}||ℱ⋆|∈[0,1].S_{\text{face}}=\frac{\bigl|\{o_{i}\in\mathcal{F}^{\star}\mid f^{g}_{i}=f^{\star}_{i}\}\bigr|}{|\mathcal{F}^{\star}|}\in[0,1].

Overall similarity.

Overall similarity weights directional agreement by 0.7 and facing agreement by 0.3. Coverage is reported separately; it does not appear in this printed weighted formula.

Equation
Soverall=α⋅Sdir+(1−α)⋅Sface∈[0,1],S_{\text{overall}}=\alpha\cdot S_{\text{dir}}+(1-\alpha)\cdot S_{\text{face}}\in[0,1],

Rotation-invariant isomorphism.

Isomorphism requires complete relation-matrix agreement after one of the permitted rotations: right-angle turns about z and a right-angle turn about x or y. This is stricter than high average similarity, but remains a test of the chosen relational representation.

Equation
∃k∈{1,…,m}​∀oi,oj∈𝒪⋆:R⋆​(oi,oj)=R(k)g​(oi,oj),\exists\,k\in\{1,\dots,m\}\;\forall\,o_{i},o_{j}\in\mathcal{O}^{\star}:\quad R^{\star}(o_{i},o_{j})=R^{g}_{(k)}(o_{i},o_{j}),

Graph validity.

A valid map must parse as JSON, meet field constraints and contain at least one valid object position. Such minimal validity does not imply complete object coverage or a correct layout.

D.3 Prompts for All Input-Output Configurations

All original prompt instructions, image examples, JSON templates and answer-format requirements follow. The specified object categories and viewpoint descriptors are part of the experimental input.

D.3.1 Example for Raw-QA

Raw-QA requests only a tagged multiple-choice answer from the four views.

Example Prompt for Raw-QA

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.2 Example for FFR

FFR adds a step-by-step explanation inside think tags before the final tagged answer.

Example Prompt for FFR: Free-Form Reasoning

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
Please do step by step reasoning first, then give your final answer. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should be this format: ’<think>(replace with your reasoning here)</think><answer>A. Above</answer>’.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.3 Example for VI-1 and VI-2

VI-1 and VI-2 expand four original viewpoints into eight or twelve images, updating the question’s image indices while retaining direct answering.

Prompt for VI-1: View Interpolation with 1 Frame

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these 8 images showing the white jar from different viewpoints (from front (image 1) to left (image 3), from left (image 3) to back (image 5), from back (image 5) to right (image 7), from right (image 7) back to front (image 1)), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 7, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

Prompt for VI-2: View Interpolation with 2 Frames

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Question]
Based on these 12 images showing the white jar from different viewpoints (from front (image 1) to left (image 4), from left (image 4) to back (image 7), from back (image 7) to right (image 10), from right (image 10) back to front (image 1)), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 10, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.4 Example for Aug-CGMap-In

Aug-CGMap-In supplies the annotated object and camera map as additional context. The prompt still describes examining the images; the supplied map is privileged structured information.

Prompt for Aug-CGMap-In: Grounded Augmented Cognitive Map as Input

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. Also, we provide you a cognitive map that shows the general layout for the scene. Please use the cognitive map to reason and answer the question.
[Answer Instruction]
You only need to provide *ONE* correct answer selecting from the options listed below. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should **only** be ’<answer>A. Above</answer>’.
[Cognitive Map Format]
We provide you a 2D grid map of the scene that is related to the question you should answer. Below is the description of the map:
- The map uses a 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- The map is shown in the bird’s view
- Directions are defined as:
* up = towards the top of the grid (decreasing y-value)
* right = towards the right of the grid (increasing x-value)
* down = towards the bottom of the grid (increasing y-value)
* left = towards the left of the grid (decreasing x-value)
* inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
* outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- "objects" lists all important items in the scene with their positions
- "facing" indicates which direction an object is oriented towards (when applicable)
- "views" represents the different camera viewpoints in the scene
Below is the cognitive map of the scene related to the question. Please use it to reason and answer the question.

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [5, 5]},
    {"name": "bed sheet with a floral pattern",
    "position": [5, 8]},
    {"name": "white headboard", "position": [2, 5]},
    {"name": "clothes rack", "position": [5, 2]},
    {"name": "table with cups on it", "position": [8, 5]}
  ],
  "views": [
    {"name": "Image 1", "position": [5, 6], "facing": "up"},
    {"name": "Image 2", "position": [4, 5], "facing": "right"},
    {"name": "Image 3", "position": [5, 4], "facing": "down"},
    {"name": "Image 4", "position": [6, 5], "facing": "left"}
  ]
}
‘‘‘

[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.5 Example for Aug-CGMap-Out

Aug-CGMap-Out gives object categories and asks for a generated object/camera map followed by the answer, without explicitly requesting a reasoning chain.

Prompt for Aug-CGMap-Out: Ask VLM to Output Augmented Cognitive Map and Direct Answer

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
- outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations accordingly
- Include camera positions and directions for each view
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your answer**:

‘‘‘json
{
  "objects": [
    {"name": "object_name", "position": [x, y],
      "facing": "direction"},
    {"name": "object_without_orientation", "position": [x, y]}
  ],
  "views": [
    {"name": "View/Image 1", "position": [x, y],
      "facing": "direction"},
    {"name": "View/Image 2", "position": [x, y],
      "facing": "direction"}
  ]
}
‘‘‘

2. Next, provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above".
3. In general, your response’s format should be like "Based on my observation, the answer is: <cogmap>(Replace with your cogmap here)</cogmap><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.6 Example for Plain-CGMap-Out

Plain-CGMap-Out requests only the object map and final answer. The original schema/example includes placeholders and comments and is preserved as a prompt template, not executable JSON.

Prompt for Plain-CGMap-Out: Ask VLM to Output Plain Cognitive Map and Direct Answer

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Object positions must maintain accurate relative spatial relationships
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:

‘‘‘json
{
    "object_category_1": {"position": [x, y]},
    "object_category_2": {"position": [x, y],
        "facing": "direction"},
    # if the object is asked for orientation
    ...
}
‘‘‘

2. Next, provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is: <cogmap>(Replace with your cogmap here)</cogmap><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.7 Example for Plain-CGMap-FFR-Out

Plain-CGMap-FFR-Out requests an object map, then detailed reasoning, then the selected answer. This is the main successful training configuration.

Prompt for Plain-CGMap-FFR-Out: Ask VLM to Output Plain Cognitive Map and Free-Form Reasoning

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Object positions must maintain accurate relative spatial relationships
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:

‘‘‘json
{
    "object_category_1": {"position": [x, y]},
    "object_category_2": {"position": [x, y],
    "facing": "direction"},
    # if the object is asked for orientation
    ...
}
‘‘‘

2. Next, please also provide your reasons step by step in details, then provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is: <cogmap>(Replace with your cogmap here)</cogmap><think>(Replace with your reasoning here)</think><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.8 Example for Aug-CGMap-FFR-Out

The augmented joint prompt additionally requires camera positions and directions. Its extra fields increase the mapping burden.

Prompt for Aut-CGMap-FFR-Out: Ask VLM to Output Augmented Cognitive Map and Free-Form Reasoning

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a 10x10 grid coordinate system.
[Rules]
1. Focus ONLY on these categories of objects in the scene: {white jar, bed sheet with a floral pattern, white headboard, clothes rack, table with cups on it}
2. Create a cognitive map with the following structure in the bird’s view:
- A 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- up = towards the top of the grid (decreasing y)
- right = towards the right of the grid (increasing x)
- down = towards the bottom of the grid (increasing y)
- left = towards the left of the grid (decreasing x)
- inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
- outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- Include positions of all objects from the specified categories
- Estimate the center location (coordinates [x, y]) of each instance within provided categories
- If a category contains multiple instances, include all of them
- Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects
- Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations accordingly
- Include camera positions and directions for each view
3. Carefully integrate information from all views to create a single coherent spatial representation.
[Answer Instruction]
1. Given the provided views and main objects mentioned in the above rules, you **MUST** present your cognitive map in the following JSON format **before your reasoning**:

‘‘‘json
{
  "objects": [
    {"name": "object_name", "position": [x, y],
    "facing": "direction"},
    {"name": "object_without_orientation", "position": [x, y]}
  ],
  "views": [
    {"name": "View/Image 1", "position": [x, y],
      "facing": "direction"},
    {"name": "View/Image 2", "position": [x, y],
      "facing": "direction"}
  ]
}
‘‘‘

2. Next, please also provide your reasons step by step in details, then provide *ONE* correct answer selecting from the options. Your answer field must be in the format like "A. Above"
3. In general, your response’s format should be like "Based on my observation, the answer is: <cogmap>(Replace with your cogmap here)</cogmap><think>(Replace with your reasoning here)</think><answer>(Replace with your answer here)</answer>". Your option must be from the available options.
[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

D.3.9 Example for CGMap-In-FFR-Out

CGMap-In-FFR-Out supplies a ground-truth augmented map and requests reasoning before the answer, separating map use from map generation.

Prompt for CGMap-In-FFR-Out: Input VLM with Grounded Cognitive Map and Output with Free-Form Reasoning

[Uncaptioned image]

[Task]
Your task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. Also, we provide you a cognitive map that shows the general layout for the scene. Please use the cognitive map to reason and answer the question.
[Answer Instruction]
Please do step by step reasoning first, then give your final answer. For example, if you think the correct answer is ’A. Above’ from ’A. Above B. Under C. Front D. Behind’, your response should be this format: ’<think>(replace with your reasoning here)</think><answer>A. Above</answer>’. [Cognitive Map Format]
We provide you a 2D grid map of the scene that is related to the question you should answer. Below is the description of the map:
- The map uses a 10x10 grid where [0,0] is at the top-left corner and [9,9] is at the bottom-right corner
- The map is shown in the bird’s view
- Directions are defined as:
* up = towards the top of the grid (decreasing y-value)
* right = towards the right of the grid (increasing x-value)
* down = towards the bottom of the grid (increasing y-value)
* left = towards the left of the grid (decreasing x-value)
* inner = straight into the 2D map (perpendicular to the grid, pointing away from you)
* outer = straight out of the 2D map (perpendicular to the grid, pointing towards you)
- "objects" lists all important items in the scene with their positions
- "facing" indicates which direction an object is oriented towards (when applicable)
- "views" represents the different camera viewpoints in the scene
Below is the cognitive map of the scene related to the question. Please use it to reason and answer the question.

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [5, 5]},
    {"name": "bed sheet with a floral pattern",
        "position": [5, 8]},
    {"name": "white headboard", "position": [2, 5]},
    {"name": "clothes rack", "position": [5, 2]},
    {"name": "table with cups on it", "position": [8, 5]}
  ],
  "views": [
    {"name": "Image 1", "position": [5, 6], "facing": "up"},
    {"name": "Image 2", "position": [4, 5], "facing": "right"},
    {"name": "Image 3", "position": [5, 4], "facing": "down"},
    {"name": "Image 4", "position": [6, 5], "facing": "left"}
  ]
}
‘‘‘

[Question]
Based on these four images (image 1, 2, 3, and 4) showing the white jar from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 4, what is to the left of the white jar?
A. Table with cups on it B. Clothes rack C. Bed sheet with a floral pattern D. White headboard

Appendix E Which Scaffolds Best Guide Spatial Thinking in Unchanged VLMs?

Frozen-model examples expose the distinction between formatting a map and constructing useful geometry. The complete outputs, including errors and source color markings, are retained.

E.1 VLM Response Examples for Configurations in Section D.3

On the illustrated jar question, several direct-answer conditions select the correct option, while generated maps contain out-of-range coordinates, collapsed camera positions or missing objects. Some accompanying explanations contradict their final answers. These cases illustrate failures rather than representative success rates.

Output for Raw-QA: Input Prompt Refer to D.3.1

C. Bed sheet with a floral pattern

Output for VI: Input Prompt Refer to D.3.3

B. Clothes rack

Output for Aug-CGMap-In: Input Prompt Refer to D.3.4

C. Bed sheet with a floral pattern

Output for FFR: Input Prompt Refer to D.3.2

<think>To determine what is to the left of the white jar from the viewpoint presented in image 4, let’s analyze the scene:
- In image 4, the white jar is placed on a table covered with a pink tablecloth featuring heart patterns.
- The table has a glass top and is positioned against a wall.
- To the left of the jar, there is a piece of furniture that appears to be a bed or a bed frame.
Given this information, the object to the left of the white jar is most likely a bed or a bed frame. </think>
<answer>C. Bed sheet with a floral pattern</answer>

Output for Aug-CGMap-FFR-Out: Input Prompt Refer to D.3.8

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [150, 446],
        "facing": "left"},
    {"name": "bed sheet with a floral pattern",
        "position": [0, 317],
    "facing": "left"},
    {"name": "clothes rack", "position": [0, 108],
    "facing": "left"},
    {"name": "table with cups on it", "position": [188, 200],
    "facing": "left"},
    {"name": "white headboard", "position": [188, 317],
    "facing": "left"}
  ],
  "views": [
    {"name": "Image 1", "position": [0, 0], "facing": "left"},
    {"name": "Image 2", "position": [0, 0], "facing": "left"},
    {"name": "Image 3", "position": [0, 0], "facing": "left"},
    {"name": "Image 4", "position": [0, 0], "facing": "left"}
  ]
}
‘‘‘

<CogMap>
<Your cognitive map>
<Reasoning>
From the viewpoint presented in image 4, we can see that the white jar is positioned in the foreground, facing left. The bed sheet with a floral pattern is located directly behind the white jar, covering the lower part of the image. The clothes rack is visible on the left side of the image, slightly above the bed sheet. The table with cups on it is situated further back, to the right of the white jar. The white headboard is also visible, but it is not as prominent as the other objects due to its position behind the bed sheet.
Therefore, my answer is A. Table with cups on it.
</Answer>
Therefore, my answer is A. Table with cups on it.

Output for Plain-CGMap-FFR-Out: Input Prompt Refer to D.3.7

‘‘‘json
{
    "white_jar": {"position": [150, 440], "facing": "left"},
    "table_with_cups_on_it": {"position": [187, 260],
    "facing": "right"}
}
‘‘‘

<CogMap>
<Your cognitive map>
<Reasoning>
From the viewpoint presented in image 4, we can see that the white jar is positioned on the bed sheet with a floral pattern. The table with cups on it is located to the right of the white jar, as indicated by its position in the image. The clothes rack is not visible in this particular view, and the white headboard is not directly related to the white jar or the table with cups on it. Therefore, the correct answer is A. Table with cups on it.
<Answer>
Therefore, my answer is A. Table with cups on it.

Output for CGMap-In-FFR-Out: Input Prompt Refer to D.3.9

C. Bed sheet with a floral pattern

E.2 Additional Graph Metrics for Generated Graphs

Appendix Table 5 reports validity of 92.57% for augmented and 94.95% for plain maps, but exact isomorphism of only 0.10% and 7.43%. Augmented maps have higher weighted similarity, 51.12% versus 37.44%, despite lower exact agreement.

These values differ from the main Table 3 chart, which shows similarities 41.14/36.86 and isomorphism 11.45/16.64. The source does not explain the discrepancy; all values remain visible in their original contexts.

Table 5: Comparison of graph metrics for cognitive maps generated by different methods. The metrics evaluate the quality of the generated maps against the ground truth. Valid Rate: percentage of syntactically and semantically valid maps. Isomorphism Rate: percentage of maps that are structurally identical (isomorphic) to the ground truth, accounting for rotation. Overall Sim. (Similarity): a weighted score combining directional and facing similarity (Soverall=α⋅Sdir+(1−α)⋅SfaceS_{\text{overall}}=\alpha\cdot S_{\text{dir}}+(1-\alpha)\cdot S_{\text{face}}). Avg. Dir. Sim. (Average Directional Similarity): correctness of relative spatial relations between objects. Avg. Facing Sim. (Average Facing Similarity): correctness of object orientations. All values are percentages (%).
Method Valid Rate Isomorphism Rate Overall Sim. Avg. Dir. Sim. Avg. Facing Sim.
Aug-CGMap-FFR-Out 92.57 0.10 51.12 43.57 68.75
Plain-CGMap-FFR-Out 94.95 7.43 37.44 28.29 58.78

E.3 Further Analysis on View Interpolation

The denser-view analysis excludes Rotation because the source data do not support more than five interpolation steps. On the remaining settings, plain map/reasoning with raw views scores 47.41%, versus the direct-answer interpolation peak of 46.47%. These figures are not the full Tiny scores.

View density effects vary by model: Qwen2.5-VL-7B improves substantially at four inserted views, while Qwen3-VL-8B is inconsistent. GPT-5 and Qwen3-VL-235B generally decline as more views are added. The study demonstrates no universal benefit from additional frames, rather than universal harm.

Method (Qwen2.5VL-3B) VI-0 (Raw) VI-1 VI-2 VI-3 VI-4 VI-5 VI-6 VI-7
Plain-CGMap-FFR-Out 47.41 44.94 44.59 43.18 44.35 43.41 42.82 45.28
RawQA 43.76 46.47 45.53 44.94 44.35 44.24 45.18 44.59
FFR 45.53 43.71 43.83 41.65 41.00 40.36 40.89 40.77
Table 6: Comparison with Optimal View Interpolation (Qwen2.5-VL-3B).
Config VI-0 VI-1 VI-2 VI-3 VI-4 VI-5 VI-6 VI-7
Qwen2.5-VL-7B 37.80 34.90 35.60 45.30 47.40 46.50 46.80 46.80
Qwen3-VL-8B 33.80 36.60 37.60 35.20 35.30 33.90 35.80 35.80
Table 7: Scaling Analysis on Qwen2.5-VL-7B and Qwen3-VL-8B.
Model VI-1 VI-2 VI-3 VI-4 VI-5 VI-6 VI-7
GPT-5 46.59 45.18 44.24 44.59 42.59 43.53 42.35
Qwen3-VL-235B 38.94 38.59 37.29 37.18 35.29 36.12 36.00
Table 8: Performance of Large-Scale Models (GPT-5, Qwen3-VL-235B) across view densities.

E.4 Explicit Reasoning with Visual-of-Thought

A separate tool-use experiment renders explicit spatial sketches and returns them to the model for interpretation.

Explicit Visual Sketching via External Tools.

The model either receives a rendered ground-truth grid or generates drawing coordinates for Matplotlib, then reasons from the resulting image. Overall accuracy rises from 41.43% to 42.10% for supplied maps, and from 41.33% to 43.13% for generated-map reasoning.

The benefit is not uniform by setting: some Rotation or Around comparisons decline. This tool-mediated result extends the representation study; it is distinct from the main supervised map-then-reason policy.

Refer to caption
(a) Explicit Visual Map in Among Setting.
Refer to caption
(b) Explicit Visual Map in Around Setting.
Figure 13: Comparison of Map Representations. The visual map, generated via external tools, renders objects and viewpoints onto a 10×1010\times 10 grid, providing explicit geometric grounding.
Configuration Map Modality Overall Rotation Among Around
Raw-QA - 37.81 34.00 36.00 45.20
CGMap-In (Original) Text (JSON) 41.43 37.00 41.67 44.40
Img-CGMap-In (New) Visual (Grid) 42.10 31.50 44.17 45.60
Plain-CGMap (Original) Text (JSON) 41.33 25.00 39.67 58.40
Img-CGMap (New) Visual (Grid) 43.13 32.75 41.13 55.06
Table 9: Performance comparison between implicit text-based maps (JSON) and explicit visual-based maps (Image/Grid). Visual configurations consistently outperform their textual counterparts.

Appendix F Can We Teach VLMs to Build and Leverage Spatial Representations?

Supervised experiments use annotated layouts and templated reasoning chains to teach structured outputs. The appendix supplies data-generation rules, training settings and ablations.

F.1 Supervised Fine-Tuning Data Curation

Maps and reasoning chains provide complementary supervision: a schematic scene representation and a procedure for answering a question about it.

F.1.1 Cognitive Map Generation

A script reads setting labels and scene metadata, places objects on a canonical 10×10 grid, adds optional facing and camera records, and serializes the map as JSON. Around uses linear object layouts, Among a central cross-like arrangement, and Rotation a central camera with changing orientation. The code also handles a translation setting.

These targets encode relational templates rather than measured metric reconstructions. Algorithm 1 preserves the full Require/Ensure specification and 25 numbered steps.

Algorithm 1 Cognitive Map Generation
Require: Dataset DD containing items with spatial arrangement annotations
Ensure: Updated dataset with cognitive maps in JSON format
1: for all i​t​e​m∈Ditem\in D do
2:     s​e​t​t​i​n​g←setting\leftarrow Extract setting type from i​t​e​m.i​ditem.id
3:     Initialize empty cognitive map c​o​g​m​a​pcogmap ⊳\triangleright Position objects and views based on setting type
4:     if s​e​t​t​i​n​g=“around”setting=\text{``around''} then
5:         Position 2-4 objects in a line with coordinates like [4,5], [5,5], etc.
6:         Place views at cardinal positions based on camera angles
7:     else if s​e​t​t​i​n​g=“among”setting=\text{``among''} then
8:         Place center object at [5,5] and surrounding objects at [5,8], [2,5], [5,2], [8,5]
9:         Position views based on specified camera angles
10:     else if s​e​t​t​i​n​g=“translation”setting=\text{``translation''} then
11:         Position objects according to their spatial relationships (e.g., “on”, “down”)
12:         Place views to highlight these spatial relationships
13:     else if s​e​t​t​i​n​g=“rotation”setting=\text{``rotation''} then
14:         Arrange objects based on rotation type (clockwise, counterclockwise, etc.)
15:         Fix camera at [5,5] with varying facing directions
16:     end if⊳\triangleright Add orientation information where applicable
17:     for all o​b​j​e​c​t∈c​o​g​m​a​p.o​b​j​e​c​t​sobject\in cogmap.objects do
18:         if o​b​j​e​c​tobject has orientation then
19:             Add facing direction (“up”, “down”, “left”, “right”)
20:         end if
21:     end for
22:     Format c​o​g​m​a​pcogmap as structured JSON
23:     Add formatted cognitive map to i​t​e​mitem
24: end for
25: return Updated dataset DD

F.1.2 Free-Form Reasoning Generation

Manually designed templates describe visible objects, establish shared scene identity and apply the query’s perspective change or hypothetical motion. Figures 14–16 retain both template placeholders and filled examples for all three principal settings.

F.2 Detailed Experimental Setup

Table 10 specifies 10,000 QA pairs, three epochs, full-parameter tuning, learning rate 10⁻⁵, cosine scheduling and effective batch size 256 on two H100 GPUs. Images use 784–90,000 pixels and sequences up to 8,192 tokens, with BF16, AdamW and ZeRO Stage 3.

The text reconciles a four-GPU script default with the claimed two-GPU runs. However, the later hyperparameter table calls batch 512 the original configuration. These details are inconsistent, so the edition preserves both instead of presenting one as fully resolved.

Table 10: Training hyperparameters for SFT experiments with Qwen2.5-VL-3B-Instruct.
Parameter Value
Dataset size 10,000 QA pairs
Epochs 3
Learning rate 1e-5
Scheduler Cosine
Fine-tuning type Full-parameter
Batch Size 256
GPUs used 2 × NVIDIA H100
Max image resolution 90,000 pixels
Min image resolution 784 pixels
Model Max Length 8192 tokens
Weight Decay 0
Warmup Ratio 0.03
Max Grad Norm 1
Precision BF16
Optimizer AdamW
Complete original app14 graphic.
Figure 14: Example reasoning chain template for Rotation
Complete original app15 graphic.
Figure 15: Example reasoning chain template for Among
Complete original app16 graphic.
Figure 16: Example reasoning chain template for Around

F.3 VLM Response Examples After SFT for Configurations in Section D.3

The complete supervised outputs show that an apparently correct coordinate map can coexist with a wrong final answer or contradictory verbal reasoning. In the jar example, augmented map-only answers correctly, whereas several reasoning variants answer incorrectly. These selected examples qualify broad claims that more explanation necessarily improves an individual prediction.

Output for Raw-QA: Input Prompt Refer to D.3.1

<answer>D. White headboard</answer>

Output for Aug-CGMap-Out: Input Prompt Refer to D.3.5

Based on my observation, the answer is:
<cogmap>

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [5, 5]},
    {"name": "bed sheet with a floral pattern",
        "position": [5, 8]},
    {"name": "white headboard", "position": [2, 5]},
    {"name": "clothes rack", "position": [5, 2]},
    {"name": "table with cups on it", "position": [8, 5]}
  ],
  "views": [
    {"name": "Image 1", "position": [5, 6], "facing": "up"},
    {"name": "Image 2", "position": [4, 5], "facing": "right"},
    {"name": "Image 3", "position": [5, 4], "facing": "down"},
    {"name": "Image 4", "position": [6, 5], "facing": "left"}
  ]
}
’’’

</cogmap><answer>C. Bed sheet with a floral pattern</answer>

Output for Plain-CGMap-Out: Input Prompt Refer to D.3.6

Based on my observation, the answer is:
<cogmap>

‘‘‘json
{
    "white jar": {"position": [5, 5]},
    "bed sheet with a floral pattern": {"position": [5, 8]},
    "white headboard": {"position": [2, 5]},
    "clothes rack": {"position": [5, 2]},
    "table with cups on it": {"position": [8, 5]}
}
’’’

</cogmap><answer>B. Clothes rack </answer>

Output for FFR: Input Prompt Refer to D.3.2

<think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the bed sheet with a floral pattern. In image 3, I can see white jar in front of the clothes rack. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view bed sheet with a floral pattern behind white jar in the second view, it implies that in the first view, bed sheet with a floral pattern is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As clothes rack is observed behind white jar in the third view, it means that in the first view, clothes rack is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: table with cups on it is to the right of white jar, bed sheet with a floral pattern is to my behind, clothes rack is to the left of white jar. So the answer is B. Clothes rack</think><answer>B. Clothes rack</answer>

Output for Aug-CGMap-FFR-Out: Input Prompt Refer to D.3.8

Based on my observation, the answer is:
<cogmap>

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [5, 5]},
    {"name": "bed sheet with a floral pattern",
        "position": [5, 8]},
    {"name": "white headboard", "position": [2, 5]},
    {"name": "clothes rack", "position": [5, 2]},
    {"name": "table with cups on it", "position": [8, 5]}
  ],
  "views": [
    {"name": "Image 1", "position": [5, 6], "facing": "up"},
    {"name": "Image 2", "position": [4, 5], "facing": "right"},
    {"name": "Image 3", "position": [5, 4], "facing": "down"},
    {"name": "Image 4", "position": [6, 5], "facing": "left"}
  ]
}
’’’

</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the clothes rack. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view clothes rack behind white jar in the second view, it implies that in the first view, clothes rack is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, bed sheet with a floral pattern is to my behind, table with cups on it is to the left of white jar. So the answer is A. Table with cups on it.</think><answer>A. Table with cups on it</answer>

Output for Plain-CGMap-FFR-Out: Input Prompt Refer to D.3.7

Based on my observation, the answer is:
<cogmap>

‘‘‘json
{
    "white jar": {"position": [5, 5]},
    "bed sheet with a floral pattern": {"position": [5, 8]},
    "white headboard": {"position": [2, 5]},
    "clothes rack": {"position": [5, 2]},
    "table with cups on it": {"position": [8, 5]}
}
’’’

</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the table with cups on it. In image 2, I can see white jar in front of the clothes rack. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view clothes rack behind white jar in the second view, it implies that in the first view, clothes rack is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, bed sheet with a floral pattern is to my behind, table with cups on it is to the left of white jar. So the answer is A. Table with cups on it.</think><answer>A. Table with cups on it</answer>

F.4 Detailed Graph Metric Results for SFT Graph-Related Experiments

The four training curves separate graph convergence from answer accuracy. Map-only training learns structure quickly, while joint plain-map/reasoning training reaches the highest final QA score despite lower exact isomorphism. The curves do not include run-to-run uncertainty estimates.

Complete original app17 graphic.
Figure 17: Training dynamics for the Aug-CGMap-Out setting.
Complete original app18 graphic.
Figure 18: Training dynamics for the Plain-CGMap-Out setting.
Complete original app19 graphic.
Figure 19: Training dynamics for the Aug-CGMap-FFR-Out setting.
Complete original app20 graphic.
Figure 20: Training dynamics for the Plain-CGMap-FFR-Out setting, showing superior final performance.

F.5 Which Part of VLM is the Bottleneck for Spatial Understanding?

At step 57 in Raw-QA training, updating only the language model gives 51.43%, versus 52.67% for all modules and 37.81% for vision-only tuning or no tuning. This supports language-side adaptation in this experiment.

It does not establish that perception is never a bottleneck, nor does the table quantify the claimed computational saving. The conclusion is limited to the tested model, checkpoint and task.

Table 11: VLM Training Bottleneck Analysis (Step=57, in %). Performance is measured on the MindCube-Tiny benchmark under the Raw-QA setting.
Training Method Overall Rotation Among Around
Raw-QA (no fine-tuning) 37.81 34.00 36.00 45.20
Freeze LLM (Vision Encoder Only) 37.81 30.50 37.00 45.60
Freeze Vision Encoder (LLM Only) 51.43 34.00 50.00 68.80
Tune All Parts 52.67 34.50 52.50 67.60

F.6 Branching from Raw-QA SFT Checkpoint

A separate two-stage experiment starts from a direct-answer checkpoint and continues with scaffold inputs while still requesting only the final option. Its own baseline is 46.36%; map-input and augmented-map/reasoning conditions reach 49.00%, and reasoning reaches 46.82%.

These numbers use a different baseline from the main supervised table and do not outperform its 52.67% Raw-QA result. The stated curriculum benefit should be interpreted within this separate protocol.

Table 12: Performance of various methods after being fine-tuned from a Raw-QA SFT checkpoint. This two-stage training approach led to performance gains across all methods. All accuracies are reported as percentages (%).
Method Overall Rotation Among Around
Raw-QA 46.36 33.50 51.20 46.75
Aug-CGMap-In 49.00 35.50 53.20 50.50
FF Rsn 46.82 37.00 50.60 47.00
Aug-CGMap-FFR-Out 49.00 37.00 53.20 49.75

F.7 Hyperparameter Tuning Results

The tuning sweep prefers learning rate 10⁻⁵, batch 512 and warmup 0.03 among seven tested choices, reaching 52.67%. The PDF’s learning-rate multipliers 1, 2 and 4 are restored where HTML omitted them. Batch 512 here conflicts with Table 10’s 256; no broader optimum is established.

Table 13: SFT Hyperparameter Tuning Results
SFT (learning rate, batch size, warmup ratio) Acc (%)
(1×10−51\!\times\!10^{-5}, 512, 0.03) – Ours 52.67
(2×10−52\!\times\!10^{-5}, 512, 0.03) 51.71
(4×10−54\!\times\!10^{-5}, 512, 0.03) 51.52
(1×10−51\!\times\!10^{-5}, 256, 0.03) 50.86
(1×10−51\!\times\!10^{-5}, 1024, 0.03) 51.90
(1×10−51\!\times\!10^{-5}, 512, 0.01) 51.81
(1×10−51\!\times\!10^{-5}, 512, 0.10) 50.67

F.8 Effect of Removing Viewpoint Descriptors

Removing textual viewpoint descriptors tests possible language shortcuts. Frozen plain map/reasoning reaches 42.67%, versus Raw-QA’s 38.57%. Under supervised training, the best observed scores are 54.0% and 52.8%, a smaller 1.2-point gap.

The supervised table selects each configuration’s best checkpoint at different steps, rather than one common final checkpoint. The advantage persists in this test but is substantially narrower than the primary supervised result.

Table 14: QA accuracy (%) of Qwen2.5-VL-3B-Instruct on MindCube-Tiny without viewpoint descriptors under different configurations for frozen VLMs.
Config. Overall Rotation Among Around
Raw-QA 38.57 34.00 36.83 46.40
Aug-CGMap-In 32.00↓\downarrow 34.50 30.50 33.60
FFR 39.14↑\uparrow 32.50 33.83 57.20
Aug-CGMap-FFR-Out 38.95↑\uparrow 19.50 41.00 49.60
Plain-CGMap-FFR-Out 42.67↑\uparrow 28.50 39.83 60.80
CGMap-In-FFR-Out 40.67↑\uparrow 34.50 42.67 40.80
Table 15: Best QA accuracy (%) across SFT training steps without viewpoint descriptors. Each value reports the peak accuracy observed during training (up to step 57).
Raw-QA Aug-CGMap-Out Plain-CGMap-Out FFR Aug-CGMap-FFR-Out Plain-CGMap-FFR-Out
Best Acc. 52.8 51.6 52.8 52.2 53.8 54.0
@ Step 45 50 45 55 20 25

Appendix G Can Reinforcement Learning Further Refine Spatial Thought Processes?

The reinforcement-learning appendix details initialization, rollout limits and sparse reward components, plus complete illustrative outputs.

G.1 Detailed Experimental Setup

Each RL configuration trains for half an epoch with batch 32, prompt/response limits of 1,024/512 tokens, eight trajectories per rollout and one turn. Reported actor and critic learning rates are 10⁻⁶ and 10⁻⁵. Valid format earns +1 and a correct answer +5.

The appendix enumerates only three configurations, while the main table reports six, including plain-map variants and reasoning initialized from SFT. “From scratch” here means from the pretrained base VLM without task SFT, not random weights.

G.2 RL Reward Design Ablation

The combined reward gives 49.52%, answer-only reward 50.76%, and format-only reward 45.05%, compared with a frozen 40.57% baseline in this ablation. Answer correctness matters more than format reward; the combined choice is not the highest-scoring tested reward.

Table 16: RL Reward Ablation Study
RL Reward Setting (format score, answer score) Acc (%)
Frozen VLM (baseline) 40.57
Our Setting (+1, +5) 49.52
Answer Only (+0, +5) 50.76
Format Only (+5, +0) 45.05

G.3 VLM Response Examples After RL for Configurations in Section D.3

The retained examples contrast wrong free-form and base-initialized augmented-map answers with a correct answer after supervised initialization. The uninitialized map uses out-of-grid coordinates and repeated camera positions; the supervised map uses the intended schema. One selected successful output does not demonstrate general map faithfulness.

Output for FFR: Input Prompt Refer to D.3.2

<think>From image 4, the white jar is positioned on a surface with a floral-patterned bed sheet. To the left of the jar, there is a table with a pink tablecloth featuring heart shapes and a Hello Kitty design. There are also cups on the table. Therefore, the object to the left of the white jar is the table with the cups.</think><answer>A. Table with cups on it</answer>

Output for RL-Aug-CGMap-FFR-Out (from scratch): Input Prompt Refer to D.3.8

<cogmap>

{
  "objects": [
    {"name": "white jar", "position": [265, 436],
        "facing": "left"}
  ],
  "views": [
    {"name": "View/Image 1", "position": [265, 436],
      "facing": "left"},
    {"name": "View/Image 2", "position": [265, 436],
      "facing": "left"},
    {"name": "View/Image 3", "position": [265, 436],
      "facing": "left"},
    {"name": "View/Image 4", "position": [265, 436],
      "facing": "left"}
  ]
}

<Your cognitive map></cogmap>
<think>The white jar is located in the center of the 10x10 grid, facing to the left. The clothes rack is to the left of the white jar, as seen in the image 2 view. The bed sheet with a floral pattern is located to the right of the white jar, as seen in the image 1 view. The white headboard is not visible in any of the provided images.
Based on the provided images, the white jar is located in the center of the 10x10 grid, facing to the left. The clothes rack is to the left of the white jar, as seen in the image 2 view. The bed sheet with a floral pattern is located to the right of the white jar, as seen in the image 1 view. The white headboard is not visible in any of the provided images.</think><answer>A. Table with cups on it.</answer>

Output for RL-Aug-CGMap-FFR-Out (from SFT): Input Prompt Refer to D.3.8

Based on my observation, the answer is:
<cogmap>

‘‘‘json
{
  "objects": [
    {"name": "white jar", "position": [5, 5]},
    {"name": "bed sheet with a floral pattern",
        "position": [5, 8]},
    {"name": "white headboard", "position": [2, 5]},
    {"name": "clothes rack", "position": [5, 2]},
    {"name": "table with cups on it", "position": [8, 5]}
  ],
  "views": [
    {"name": "Image 1", "position": [5, 6], "facing": "up"},
    {"name": "Image 2", "position": [4, 5], "facing": "right"},
    {"name": "Image 3", "position": [5, 4], "facing": "down"},
    {"name": "Image 4", "position": [6, 5], "facing": "left"}
  ]
}
’’’

</cogmap><think>In this scene, I observe four images showing different perspectives. All images feature the white jar as the main object. In image 1, I can see white jar in front of the clothes rack. In image 2, I can see white jar in front of the table with cups on it. In image 3, I can see white jar in front of the bed sheet with a floral pattern. In image 4, I can see white jar in front of the white headboard. To identify the position change across views, I focus on the main object’s angle variation. Then, I analyze the angles and relative positions of other objects on the platform to back up this observation. I understand that: Image 1 is the initial view. Image 2 is captured after a 90-degree clockwise rotation from image 1. Image 3 is after another 90-degree clockwise rotation (180 degrees from image 1). Image 4 is after a further 90-degree clockwise rotation (270 degrees from image 1). Through analyzing these perspective changes, I can construct a complete spatial understanding: when I view table with cups on it behind white jar in the second view, it implies that in the first view, table with cups on it is on the right side of white jar. Similarly, when I see white headboard behind white jar in the fourth view, it indicates that in the first view, white headboard is on the left side of white jar. However, I am still uncertain about what lies behind me in the first view. Then, I recognize that I can examine the opposite view to find out. The opposite view of the fist view is the third view. As bed sheet with a floral pattern is observed behind white jar in the third view, it means that in the first view, bed sheet with a floral pattern is positioned behind me. This way, I can fully comprehend the spatial relationships of all objects in the entire scene. So, from the perspective of image 4: clothes rack is to the right of white jar, table with cups on it is to my behind, bed sheet with a floral pattern is to the left of white jar. So the answer is C. Bed sheet with a floral pattern.</think><answer>C. Bed sheet with a floral pattern</answer>

Concept demonstration

Open image ↗