Literature
Display
← All papers

Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?

Pingyue Zhang · Zihan Huang · Yue Wang · Jieyu Zhang · Letian Xue · Zihan Wang · Qineng Wang · Keshigeyan Chandrasegaran · Ruohan Zhang · Yejin Choi · Ranjay Krishna · Jiajun Wu · Li Fei-Fei · Manling Li

· ICLR 2026 ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

Theory of Space separates spatial competence into constructing a map through exploration, using it to answer questions, and revising it after changes. In matched procedural layouts, six foundation models generally reason better from scripted observation histories than from their own exploration; visual observations introduce especially severe orientation and map-maintenance errors. Explicit map probes help locate failures, while remaining prompted reports rather than direct measurements of hidden model state.

Website https://theory-of-space.github.io/
Code https://github.com/mll-lab-nu/Theory-of-Space
Data https://huggingface.co/datasets/MLL-Lab/tos-data

At a glance

The benchmark combines task-agnostic exploration, nine downstream spatial tasks, structured cognitive-map probes, unseen-region classification, and a changed-environment test. It contrasts text and rendered-image observations. The main result is a gap between passive reasoning and effective information collection: models take redundant steps, overlook regions, and retain obsolete estimates, especially in vision. The controlled simulator makes these behaviors inspectable but uses high-level actions and does not establish physical navigation competence.

1 Introduction

A correct answer after a supplied video does not show that an agent could choose the views needed to produce it. Motivated by work on active and passive spatial learning, the paper tests how models gather evidence before receiving specific questions. It also asks them to externalize maps throughout exploration and after the environment changes. These probes distinguish observation, self-tracking, map consistency, coverage, and updating errors; associations between them and task scores should not be read as direct access to internal representations. Figures 1–2 introduce the workflow and its accuracy–cost trade-off.

Original PDF art retaining vector labels and native embedded image detail.
Figure 1: Theory of Space: active exploration, probed belief, and evaluation. Left: a top-down view of agent trajectory under partial observability in multiple-room scenes. Middle: the agent’s action loop of moving, rotating, and observing in text- or vision-based environments, receiving egocentric observations and updating an internal belief. Right: evaluation through exploitation of the belief in spatial tasks and direct probing via probed cognitive maps.
Original PDF art retaining vector labels and native embedded image detail.
Figure 2: Evaluation accuracy vs. exploration cost for active exploration in vision-world. Faded icons mark the passive setting, where the agent gets a pre-generated exploration history and only reasons.

2 Theory of Space

Theory of Space is the ability to construct, revise, and exploit an internal spatial belief under partial observability. Formally, observations and actions build a history that supports a belief over spatial structure; new observations should revise that belief after a state change, and a belief-conditioned policy should support downstream tasks. The probabilistic formulation is conceptual: the implemented probes request deterministic coordinate-and-heading maps, not a full posterior distribution.

Original PDF callout preserving its full heading and body.

2.1 A Paradigm for assessing Theory of Space of Large Foundation Models

Agents first explore without a downstream question, aiming to build a useful map with minimal observation cost. Evaluation then tests route knowledge—landmark relations and egocentric transformations—and survey knowledge, including global coordinates and novel views. This distinction follows Siegel and White and Taylor and Tversky. Structured map reports and unseen-region predictions complement task accuracy. A further intervention changes object states to test revision; neither behavioral success nor a prompted map alone completely identifies the agent’s latent knowledge.

3 Benchmarking Theory of Space Ability for Foundation Models

The benchmark has two phases: an agent gathers a history of local observations by selecting actions, then answers spatial questions using that history. Separate probes measure the quality and evolution of its reported map, and a re-exploration phase tests whether it detects and incorporates changed objects.

3.1 Spatial Environment Construction

Scenes are integer-grid, connected multi-room layouts with named objects and cardinal headings. The default has three 6×6 rooms with four objects each. Agents know the room count and object names. High-level actions move directly to objects, rotate, observe, query visible-object coordinates, or terminate; movement feasibility and motor control are abstracted away. Text and vision share layouts, while observations have different informational and perceptual demands.

The field of view spans 90°. Text reports binned egocentric direction, distance, identity, and facing. The main specification uses six distance categories from coincident through very far, and eight allocentric directions; Figure 7(a) instead illustrates different distance boundaries. The vision condition includes calibration and canonical object-view images. Query costs two units and Observe one, but queries are rare and analyses mainly count turns. The full prompt permits multiple movement commands before one observation, so a reported step is not a primitive move or distance traveled.

3.2 Downstream Spatial Tasks

Nine open-ended tasks assess five route abilities—pairwise relations, perspective taking, perspective identification, action-to-view, and view-to-action—and four survey abilities: allocentric mapping, mental rotation, location-to-view, and view-to-location. Answers follow fixed formats rather than multiple-choice options. Figure 3 and Table 1 preserve the full taxonomy. Mental-rotation prompts explicitly imagine removing the walls, so this reasoning task differs from the room-bounded visibility used in exploration.

Original PDF art retaining vector labels and native embedded image detail.
Figure 3: Theory of Space exploitation task suite: it covers route-level egocentric reasoning and survey-level allocentric mapping. Route tasks evaluate path-based inference and egocentric observations. Survey tasks test global mapping, geometric transformation, and perspective conversion. Together they cover both local navigation reasoning and global spatial abstraction.
Original PDF art retaining vector labels and native embedded image detail.
Table 1: Task suite comparison: Route belief emphasizes egocentric, step-by-step path reasoning; Survey belief emphasizes allocentric mapping and novel view inference.

3.3 Assessment Dimensions

Construction efficiency measures how much observations reduce candidate locations: each object starts with M possible cells, and the score normalizes the sum of logarithms of its remaining candidate count. AC-3 propagates constraints. This is a marginal-domain reduction proxy, not the model’s measured entropy or a full joint posterior; vision instead uses object coverage.

Cognitive-map probes assess reported positions, pairwise directions, headings, self-tracking, temporal stability, and agreement between local and global maps. Uncertainty probes classify candidate points as previously unseen. Revision metrics measure detected changes, redundant exploration, revised correctness, and persistence of old estimates. Exploitation averages the nine downstream tasks, with direction/distance components weighted equally where applicable and normalized coordinate similarity for map answers. These measures expose different failure modes and are not interchangeable.

3.4 Exploration Strategies

Active agents choose actions using their growing histories. Passive agents receive histories generated by scripted proxies: Scout sweeps cardinal views and visits rooms, while Strategist maintains feasible object positions and selects views to resolve the largest remaining uncertainty. The standard vision baseline uses Scout and text uses Strategist. This supplies useful reference histories, but active and passive agents receive different evidence, and the two modalities do not share an identical baseline policy.

4 Evaluation and Analysis

The study evaluates GPT-5.2, Gemini-3 Pro, Claude-4.5 Sonnet, GLM-4.6V, Qwen3-VL, and InternVL3.5; InternVL is omitted from vision because of its ten-image limit. There are 100 scenes and three questions for each of nine tasks, totaling 2,700 questions per setting. Reasoning models and other models use different decoding settings, so this is not a compute-matched model comparison. The nominal active budget is 20 turns.

Gemini leads active average accuracy with 81.5% in text and 57.3% in vision; GPT scores 72.0% and 46.0%. On passive histories, GPT reaches 90.4% in text, while Gemini reaches 60.5% in vision. GPT and Gemini take 17.2 and 13.6 turns in active vision, versus roughly nine for the scripted baseline. Even alternate Scout text histories produce 83.9% and 86.7%, above these models’ own active histories. Three humans tested on only five scenes per modality score 86.7% in text and 96.4% in vision, rising to about 99% with instruments; this is a small reference sample.

More rooms generally raise cost and lower accuracy, although not every individual comparison is monotonic. GPT’s four-room vision result reports 23.2 turns, exceeding the nominal 20-turn budget, without a clear reconciliation. Information-gain curves and selected trajectories illustrate redundant exploration, omissions, and varied sweeping behaviors. These examples help explain the aggregate pattern but do not establish a universal strategy for each model.

Original PDF art retaining vector labels and native embedded image detail.
Table 2: Exploitation Performance (%\%) of Belief Construction via Active Exploration. Models autonomously plan actions and are evaluated on exploration cost, route-level reasoning, and survey-level reasoning across text- and vision-based environments. Gemini-3 Pro leads every task and all reasoning metrics, while GPT-5.2 achieves the lowest exploration cost in text-world. Humans outperform in both settings, especially in vision. ⋆Humans can use instruments such as protractors and compasses to infer object positions precisely.
Original PDF art retaining vector labels and native embedded image detail.
Table 3: Exploitation Performance (%\%) of Belief Construction via Passive Observations. Models are evaluated as passive comprehension agents on Route- and Survey-level reasoning using standardized observation logs from scripted proxy explorers, decoupling exploration from belief construction across text- and vision-based environments. Gemini-3 Pro leads most tasks in the vision-based world and achieves the best overall average, while GPT-5.2 leads the text-based world and attains the best overall average.
Original PDF callout preserving its full heading and body.
Text-based World
Methods 2-room 4-room
pass. act. steps pass. act. steps
GPT-5.2 92.3 77.8 6.2 86.5 66.0 16.4
Gemini-3 Pro 86.7 80.6 6.2 81.2 77.7 19.7
Vision-based World
Methods 2-room 4-room
pass. act. steps pass. act. steps
GPT-5.2 59.3 51.5 10.8 52.6 40.3 23.2
Gemini-3 Pro 58.3 57.8 6.6 56.2 51.5 19.7
Original PDF art retaining vector labels and native embedded image detail.
Table 4: Exploitation Performance (%\%) for Multi-Room Settings (2-room and 4-room). pass. for passive avg acc, act. for active avg acc, steps for average steps.
Original figure
Figure 4: Accumulated information gain over exploration steps in the text world.
Original PDF callout preserving its full heading and body.

5 How do Foundation Models Manage Internal Spatial Belief?

The analysis probes reported spatial representations during exploration, then tests awareness of unseen regions and revision after object changes. Local observations, a consolidated global map, and the agent’s estimated pose permit more specific diagnoses than downstream accuracy alone.

Original PDF art retaining vector labels and native embedded image detail.
Figure 5: Internal Spatial Belief Probing. At each step, the agent executes an action, receives an observation, and updates its spatial belief. We probe this belief by prompting the agent to (i) output a JSON-structured cognitive map of all observed objects and (ii) select the next unexplored position from a top-down view given a set of labeled candidate points. For clarity, the figure shows the probing process for a single step.

5.1 Cognitive Map Probing

Global-map prompts request observed objects, doors, and the agent in a coordinate frame anchored at the initial position, with north as the initial heading. Local prompts describe only the current field of view in agent-centered coordinates. Position scores multiply coverage by an exponential penalty on normalized root-mean-square error; final correctness averages position, pairwise direction, and heading scores. Stability measures whether error worsens from one turn to the next: an unchanged wrong estimate can therefore count as stable. Consistency likewise does not guarantee truth.

Final visual-map correctness is 32.2% for GPT and 52.1% for Gemini, versus 80.0% and 81.4% in text. Visual orientation scores fall to 20.2% and 32.2%, despite better first-observation orientation scores of 33.5% and 43.8%. Self-heading tracking is much stronger than object-heading estimation. The evolving maps suggest errors beyond first-view perception, but prompted readout error cannot be excluded.

Providing an oracle map raises downstream performance to about 95% in both modalities. Asking models to first produce their own explicit map slightly reduces performance, which could reflect representation limits or additional prompting burden. Across five independent map-probe runs per sample, map correctness correlates with task accuracy: GPT/Gemini correlations are 0.418/0.466 in text and 0.570/0.645 in vision, all reported p<.001. This supports diagnostic association, not a causal account of internal storage.

Original PDF art retaining vector labels and native embedded image detail.
Table 5: Spatial Belief Quality via Cognitive Map Probing. We measure final map correctness and turn-level perception, local global consistency, stability, self-tracking, and uncertainty in text- vs. vision-worlds. ori. for orientation and pos. for position. Across models, vision lags text on all metrics, with the largest drop on orientation and stability.
Original PDF callout preserving its full heading and body.
Original PDF callout preserving its full heading and body.

5.2 Uncertainty Map Probing

The probe supplies an object-free top-down map with labeled candidate points and asks which remain unobserved. Text uses a symbolic grid; vision uses an image. GPT’s F1 is 64.5% in text and 53.7% in vision; Gemini reaches 79.2% and 70.2%. These are coverage-classification scores rather than calibrated uncertainty estimates. They align with the reported pattern of GPT gathering evidence quickly then plateauing, while Gemini continues improving, without isolating that strategy as the sole cause.

Original PDF art retaining vector labels and native embedded image detail.
Table 6: Pearson correlation (rr) between spatial-belief correctness and downstream evaluation performance. All correlations are significant (p<.001p<.001).
Original figure
Figure 6: Accumulated Information Gain and Cognitive Map Correctness over steps.

5.3 Belief Revision Task

After initial exploration, four objects are each moved or rotated. The prompt explicitly announces the number of changes, returns the agent to the starting position facing north, and preserves its history; object identities and change types remain to be discovered. The design draws on spatial belief revision. It measures change-identification F1, total turns, turns after the last changed object was observed, revised map quality, and attraction toward old estimates.

Positional inertia combines alignment of the new error with the old-location offset and a Gaussian weight for staying near the old prediction. Its noise scale is the root-mean-square error on first re-observed unchanged objects. Orientation inertia simply checks whether the reported heading remains unchanged. These operational indicators are useful, but persistence alone does not prove a particular cognitive mechanism.

Text change-identification F1 is about 98% for both models and both change types. Vision is substantially weaker for rotations: GPT/Gemini orientation F1 is 14.3%/23.9%, versus position F1 of 68.0%/82.5%. They spend 6.20/3.23 redundant visual turns, compared with 0.55/0.18 in text. Visual orientation inertia is 68.9%/51.1%, and revised orientation correctness only 16.7%/30.3%, making facing estimation and revision major bottlenecks.

Equation
sip​o​s=𝐞i⊤​𝐯i‖𝐞i‖​‖𝐯i‖+ϵ⏟Directional alignment ​(cos⁡θi)⋅exp⁡(−‖𝐛in​e​w−𝐛io​l​d‖22​σ2)⏟Proximity weight ​(wi).s_{i}^{pos}=\underbrace{\frac{\mathbf{e}_{i}^{\top}\mathbf{v}_{i}}{\|\mathbf{e}_{i}\|\,\|\mathbf{v}_{i}\|+\epsilon}}_{\text{Directional alignment }(\cos\theta_{i})}\cdot\underbrace{\exp\!\left(-\frac{\|\mathbf{b}_{i}^{new}-\mathbf{b}_{i}^{old}\|^{2}}{2\sigma^{2}}\right)}_{\text{Proximity weight }(w_{i})}.
Original PDF art retaining vector labels and native embedded image detail.
Table 7: Belief updating under environmental shifts. After relocating/reorienting k=4k{=}4 objects, we evaluate change identification, re-exploration cost (including redundancy (red.)), and belief correctness/update in text- vs. vision-worlds. Vision agents require more redundant steps and show severe orientation inertia, failing to overwrite obsolete facing beliefs despite new evidence.
Original PDF callout preserving its full heading and body.

6 Related Work

Earlier textual, image, multi-view, and video benchmarks assess spatial reasoning from supplied evidence; task-driven embodied benchmarks require exploration for questions, navigation, or manipulation. Theory of Space instead evaluates zero-shot, task-agnostic exploration and explicit map reports. EXCALIBUR also studies task-agnostic exploration, using reinforcement learning; the distinction here is the training-free foundation-model setting and the process-level probes. Claims about possible goal leakage from training are methodological concerns, not demonstrated failures of every earlier agent.

7 Conclusions

The benchmark identifies information gathering, visual orientation, map stability, and updating after changes as weaknesses even when passive spatial reasoning is relatively strong. Its value is the linked examination of trajectories, reported maps, and downstream answers. Limits include synthetic tree-shaped layouts, high-level teleport-like movement, a known object inventory, different text/vision representations and proxies, a small human sample, and map probes that may themselves affect responses. The results do not establish real-robot navigation performance or directly reveal hidden beliefs. Multi-agent exploration and more robust uncertainty-guided mapping remain future directions.

Appendix

The appendix preserves implementation details, complete prompts, additional room-count experiments, and qualitative trajectories and analysis interfaces.

Appendix A Technical Details

Technical details specify the simulator, observation formats, constraint-based scoring, scripted explorers, and full task and probing prompts.

A.1 Benchmark Construction

A Gym-like interface exposes discrete interaction with a partially observed grid world and 384×384 rendered views. Objects occupy integer coordinates with north/east/south/west headings. Room adjacency is a connected tree, excluding loop closure through cyclic layouts. Randomized spawn poses and seeded scene generation provide controlled instances; the implementation abstracts away continuous motion.

Text-based World

Observe returns visible object identities, binned direction and distance, and facing when available. Visibility is confined to the current room; standing in a doorway opens views into both adjoining rooms. Optional observation-noise modules are described, without a corresponding main-result noise sweep. Figure 7(a)’s illustrative near/mid/far boundaries differ from the six-bin main specification and prompt, so they should not be silently treated as one consistent discretization.

Vision-based World

The visual scene library contains 293 models in six categories and 37 subtypes, with each object type used at most once per scene. Placement enforces spacing and collision constraints. Prompts include distance-calibration images and front/side object references to help infer appearance and heading. Shared random seeds control layouts, while images and symbolic observations remain different inputs.

Refer to caption
(a) Field of view (FOV) specification for the agent in our tasks. The FOV spans 90° in front of the agent and is divided into angular bins (e.g., front, front-slight left, front-left) and distance ranges (near [0,2], mid [2,5], far [5,10]). This egocentric perception defines how spatial relations are observed and reported.
Original figure
(b) Distribution of all 3D models used in our vision tasks.
Figure 7: Demonstration figures for FOV and 3D model distribution
Refer to caption
Figure 8: Example of distance cues in the vision prompt. The colored cylinders illustrate objects placed at different distances from the agent: yellow at 2 m, blue at 1 m, red at 2 m, and green at 3 m, providing calibration for mapping visual observations to discretized distance bins.
Refer to caption
Figure 9: Object appearance and orientation cues in the vision prompt. Objects with facing direction are shown from both the front and side views, while objects without inherent orientation are displayed only from the front view. This provides the agent with consistent visual references for recognizing shape and facing.

Information Gain Calculation

Each object initially has the whole 20×20 map as its candidate domain. Observations impose directional, distance, room-visibility, and collision constraints. AC-3 repeatedly removes candidates lacking support in related domains until reaching arc consistency. It does not guarantee a globally consistent joint assignment; in these valid, exact trajectories the authors argue that the true assignment remains supported, preventing empty domains. Domain reduction consequently measures information present in observations, rather than what the language model has actually inferred.

Proxy agents

Scout performs four cardinal observations at a location, then visits adjacent rooms in a fixed order and repeats. Strategist starts with a sweep, selects the object with the largest feasible-position domain, moves to a constraining viewpoint, and observes directions with unresolved pairwise relations. It continues until current-room domains become singletons before advancing. Both proxies have explicit procedural bookkeeping that the evaluated models must approximate through their histories.

Prompts

Figures 10–13 retain all exploration, task, global/local-map, changed-environment, and unseen-region prompts. Action syntax permits zero or more JumpTo/Rotate commands followed by exactly one Observe, Query, or Term; Term must stand alone, and the first turn forbids JumpTo. The revision prompt tells the agent how many objects changed and requests an object-and-change-type report. Global coordinates are anchored at the starting pose; local coordinates use +y forward and +x right. These details matter when interpreting cost, revision difficulty, and coordinate errors.

Original figure
Figure 10: Exploration prompts
Original figure
Figure 11: Evaluation prompt design. We show the prompt for each evaluation task.
Original figure
Figure 12: Belief probing prompt design. We use these prompts to ask the model to output a cognitive map or select unobserved points.
Original PDF art retaining vector labels and native embedded image detail.
Figure 13: The symbol map and the image map provide parallel representations of the same environment for text and vision settings in uncertainty probing prompts.

Appendix B Evaluation Setups

Evaluation instances use IDs 0–99 as random seeds for paired text and vision layouts and task sampling. The reproducibility claim concerns shared generated scenes; the actual observation streams differ by modality.

Additional Results

Two- and four-room tests keep the same room dimensions and objects per room; the four-room layout connects the main room to all others. Tables 8–9 report two rooms and Tables 10–11 report four rooms, despite a prose sentence incorrectly calling the latter three-room results. Higher resolution is said not to improve performance, but no detailed numerical resolution comparison is supplied here. The additional tables support increased exploration cost with room count and persistent active/passive gaps, with individual nonmonotonic accuracy changes.

direction persp.take perc.dec act2view view2act alloc.map ment.rot loc2view view2loc
Static (S) Dynamic (D) Static (S) Dynamic (D)
Methods Route Survey Avg.
Vision-based World
Proprietary Models
GPT-5.2 39.2 37.3 63.3 53.8 58.3 68.2 92.7 52.3 68.6     59.3
Gemini-3 Pro 57.8 33.9 53.8 48.5 58.7 64.6 83.3 54.7 69.8     58.3
Text-based World
Proprietary Models
GPT-5.2 85.3 92.0 99.0 90.0 83.0 97.2 99.7 89.5 95.2     92.3
Gemini-3 Pro 88.2 86.7 91.7 87.3 79.3 90.1 92.7 81.5 82.9     86.7
Original PDF art retaining vector labels and native embedded image detail.
Table 8: Exploitation Performance (%\%) via Passive Observations under two rooms settings.
direction persp.take perc.dec. act2view view2act alloc.map ment.rot loc2view view2loc
Static (S) Dynamic (D) Static (S) Dynamic (D)
Methods Avg.cost Route Survey Avg.
Vision-based World
Proprietary Models
GPT-5.2 10.8 41.3 36.2 48.2 49.0 54.7 56.9 72.0 45.2 59.7     51.5
Gemini-3 Pro 6.6 51.7 36.3 63.0 47.2 56.0 63.4 85.0 50.3 67.5     57.8
Text-based World
Proprietary Models
GPT-5.2 6.2 68.7 67.3 90.0 76.8 64.0 83.4 92.7 73.7 83.7     77.8
Gemini-3 Pro 6.2 76.0 68.3 89.0 77.2 72.7 83.1 96.0 77.5 86.2     80.6
Original PDF art retaining vector labels and native embedded image detail.
Table 9: Exploitation Performance (%\%) via Active Exploration under two rooms settings.
direction persp.take perc.dec act2view view2act alloc.map ment.rot loc2view view2loc
Static (S) Dynamic (D) Static (S) Dynamic (D)
Methods Route Survey Avg.
Vision-based World
Proprietary Models
GPT-5.2 47.0 37.7 59.7 38.3 40.3 60.1 73.7 50.5 65.9     52.6
Gemini-3 Pro 63.5 35.5 58.7 42.8 43.0 64.4 81.7 48.8 67.4     56.2
Text-based World
Proprietary Models
GPT-5.2 83.8 88.2 94.3 86.8 62.7 94.8 93.7 82.0 92.5     86.5
Gemini-3 Pro 81.2 91.3 96.7 82.2 68.3 76.8 81.3 74.2 79.0     81.2
Original PDF art retaining vector labels and native embedded image detail.
Table 10: Exploitation Performance (%\%) via Passive Observations under four rooms settings.
direction persp.take perc.dec. act2view view2act alloc.map ment.rot loc2view view2loc
Static (S) Dynamic (D) Static (S) Dynamic (D)
Methods Avg.cost Route Survey Avg.
Vision-based World
Proprietary Models
GPT-5.2 23.2 41.2 33.2 49.0 30.8 30.7 32.5 49.7 40.5 55.4     40.3
Gemini-3 Pro 19.7 59.8 34.2 60.3 34.7 46.0 56.8 62.7 44.0 64.8     51.5
Text-based World
Proprietary Models
GPT-5.2 16.4 65.3 69.0 74.3 62.8 44.3 66.6 76.3 57.5 77.8     66.0
Gemini-3 Pro 19.7 76.3 77.2 91.7 73.3 64.3 77.0 83.7 74.0 81.9     77.7
Original PDF art retaining vector labels and native embedded image detail.
Table 11: Exploitation Performance (%\%) via Active Exploration under four rooms settings.

Appendix C Additional Visualization Examples

Figure 14 gives concrete examples and strict answer formats for every route and survey task. The visual setting substitutes rendered views for textual scene descriptions; task outputs remain structured spatial answers.

Original PDF art retaining vector labels and native embedded image detail.
Figure 14: Examples of task formats and answer styles used. Each block illustrates a spatial reasoning task type in our suite (Route-level and Survey-level), including the corresponding input context and an example open-ended answer that must follow a strict output format. In the vision setting, textual scene descriptions in the questions are replaced by rendered observation images.

Cognitive map output by models

Figures 15–16 compare GPT’s predicted and ground-truth maps turn by turn in one text and one vision example. They make changing coordinate errors and retained or omitted objects visible, complementing aggregate correctness scores without constituting an additional systematic evaluation.

Exploration pattern examples by models

Figures 17–21 show successful room sweeping, premature door-driven omissions, and less organized exploration. Both GPT and Gemini have examples of door finding and systematic sweeping, while another Gemini trace pivots around objects. These are selected trajectory patterns, so the labels should not be interpreted as fixed model-wide policies.

Analysis Platform

Figures 22–26 preserve the authors’ analysis interface: aggregate charts, text and vision exploration metrics, and individual turns with actions and cognitive maps. The screenshots illustrate how trajectories and reported state were inspected; they introduce no separate benchmark results.

Original figure
Figure 15: GPT-5.2’s turn-by-turn cognitive map in text world during exploration.
Original figure
Figure 16: GPT-5.2’s turn-by-turn cognitive map in vision world during exploration.
Refer to caption
Figure 17: Example trajectory illustrating GPT-5.2’s door-finding strategy and systematic sweeping pattern: Upon detecting a door, the agent navigates toward it and executes a strategic rotation to maximize environmental coverage. The process terminates once all target objects have been successfully identified.
Refer to caption
Figure 18: Example trajectory illustrating GPT-5.2’s omission pattern: Observing the door too early may lead the agent to skip the rest of the exploration, causing incomplete environmental discovery.
Refer to caption
Figure 19: Example trajectory illustrating GEMINI-3 Pro’s door-finding strategy and systematic sweeping pattern in vision world: Upon detecting a door, the agent navigates toward it and executes a strategic rotation to maximize environmental coverage. The process terminates once all target objects have been successfully identified.
Refer to caption
Figure 20: Example trajectory illustrating GEMINI-3 Pro’s object sweeping pattern mostly found in text world: Orbit the starting object using it as the pivot point. Randomly select an observed door to jump to a new object, then resume pivoting around the new target in a continuous loop.
Refer to caption
Figure 21: Example trajectory illustrating CLAUDE-4.5 Sonnet’s exploration pattern: There is no clear exploration pattern.
Refer to caption
Figure 22: Platform designed by us for analysis (chart)
Refer to caption
Figure 23: Visualization Platform for analysis: Metrics for active exploration in text world
Refer to caption
Figure 24: Visualization Platform for analysis: Metrics for active exploration in vision world
Refer to caption
Figure 25: Visualization Platform for analysis: one turn of active exploration in text-world, including agent’s action and cognitive map.
Refer to caption
Figure 26: Visualization Platform for analysis: one turn of active exploration in vision-world
Concept demonstration

Open image ↗