Overview
ESI-Bench evaluates whether an agent can obtain the observations needed to answer a spatial question. The original overview illustrates all 29 subcategories, including hidden objects, reflective surfaces, physical tests, counting, navigation and temporal change.
At a glance
The benchmark contains 3,081 simulated instances across ten categories. Agents choose perception, movement and manipulation actions, then answer within 30 steps. Active exploration generally helps over fixed or randomly collected views, but substantial gaps remain to humans and informative oracle trajectories. Accurate 3D scene graphs can help; reconstructed graphs can introduce damaging geometric errors. The results diagnose evidence acquisition and belief updating under a high-level simulator interface, rather than validating low-level robot control or real-world reliability.
1 Introduction
The central question is which observation to acquire, not merely how to interpret an already useful image. Questions deliberately hide information behind viewpoint ambiguity, occlusion, containers or physical interaction. The taxonomy draws on Spelke and Kinzler’s core-knowledge framework. By varying access to views and actions, the benchmark studies how poor evidence selection can compound perceptual errors. Claims about “action blindness” describe the evaluated models and task conditions; oracle gains do not prove that perception is irrelevant or universally solvable.
2 Related Works
Existing spatial benchmarks emphasize fixed images, videos, multiview inputs or partial observation; embodied benchmarks add navigation and interaction. ESI-Bench combines broad spatial categories with hidden variables that require active inspection. Geometry-augmented reasoning and explicit reasoning traces address interpretation, while active-perception methods address observation selection. Table 1 retains the authors’ detailed scope comparison, including action access, hidden states and cognitive categories.
| Benchmark | Type | Active? | Action | Hidden | Core |
|---|---|---|---|---|---|
| VSR (Liu et al., 2023) | Spatial | ✗ | P | ✗ | Geom |
| BLINK (Fu et al., 2024) | Spatial | ✗ | P | ✗ | Obj, Geom, Num |
| 3DSRBench (Ma et al., 2025) | Spatial | ✗ | P | ✗ | Obj, Geom |
| VSI-Bench (Yang et al., 2025a) | Spatial | ✗ | P | ✗ | Obj, Geom, Num |
| MMSI-Bench (Yang et al., 2025d) | Spatial | ✗ | P | ✗ | Obj, Geom |
| MindCube (Wang et al., 2026) | Spatial | ✗ | P | ✓ | Geom, Phys |
| PhysBench (Chow et al., 2025) | Spatial | ✗ | P | ✓ | Obj, Geom, Phys |
| CausalSpatial (Ma et al., 2026) | Spatial | ✗ | P | ✓ | Obj, Phys |
| EmbSpatial-Bench (Du et al., 2024) | Spatial | ✗ | P | ✗ | Obj, Geom |
| OpenEQA (Majumdar et al., 2024) | Embodied | ✓ | L, P | ✗ | Obj, Geom |
| EmbodiedBench (Yang et al., 2025b) | Embodied | ✓ | L, P, M | ✗ | Obj, Geom, Agt |
| EmbodiedEval (Cheng et al., 2025) | Embodied | ✓ | L, P, M | ✗ | Obj, Geom, Agt |
| EXPRESS-Bench (Jiang et al., 2025) | Embodied | ✓ | L, P | ✗ | Obj, Geom |
| ESPIRE (Zhao et al., 2026) | Spatial+Embodied | ✗ | P, M | ✗ | Obj, Geom |
| CHAIN (Wu et al., 2026b) | Embodied | ✓ | P, M | ✓ | Obj, Geom, Phys |
| ESI-Bench (ours) | Spatial+Embodied | ✓ | L, P, M | ✓ | Obj, Geom, Num, Phys, Agt |
3 ESI-Bench
The benchmark couples simulator-derived labels with verified interaction trajectories. Its construction process seeks physically valid, answerable questions whose initial view is insufficient.
3.1 Benchmark Setup
The setup fixes an initial scene and pose, a question, a labelled answer and a shared action budget. Agents control evidence acquisition through discrete, high-level commands.
Task Definition
Each instance specifies a scene, initial pose, question and ground truth. The agent accumulates observations through actions and terminates with an answer plus confidence in at most 30 steps. Questions constrain the expected answer type, such as a count, category, yes/no relation or ordering. Correctness is defined by matching the labelled answer; detailed free-text normalization is not specified here.
Simulation Environment.
OmniGibson/BEHAVIOR-1K provides 51 scenes, more than 300 rooms and about 9,000 object instances across 1,829 categories. Physics, transparency, reflections, fluids and articulated states support the hidden-variable tasks. Simulator scene graphs supply poses, bounding boxes, contacts, containment and room assignments for construction and labels. These controlled observations differ from noisy real sensor data.
| Action | Description |
|---|---|
| Locomotion | |
| move_forward / move_backward | Translate agent along viewing axis |
| move_left / move_right | Translate agent laterally |
| move_up / move_down | Translate agent vertically |
| Perception | |
| turn_left / turn_right | Rotate agent horizontally |
| turn_up / turn_down | Rotate agent vertically |
| Manipulation | |
| pick_up obj | Pick up object |
| put obj inside obj | Put object inside object |
| put obj on obj | Put object on object |
| fill obj with water | Fill object with water |
| pour obj from obj to obj | Pour object from object to object |
| Terminal | |
| answer(, ) | Commit to final answer with confidence |
3.2 Task Construction
GPT-4o proposes candidates; simulator checks and independent human review determine which instances survive. Proposal generation is not treated as a source of ground-truth correctness.
Task Proposal.
Given scene metadata, task requirements and a random 200-category candidate set, GPT-4o proposes objects, placements, initial viewpoints and an informative action trajectory. Labels are derived from the instantiated scene state.
Scene Instantiation.
Candidate placements undergo collision checks, physics settling, refreshed bounding-box checks, visibility verification and relevant contact tests. Invalid configurations are rejected.
Agent Trajectory Collection.
The proposed trajectory is executed step by step with rendered observations and repeated state checks. A failure at any step rejects that candidate trajectory.
Metadata Saving.
Saved metadata includes scene/room/floor identity, object models and poses, agent pose, visibility flags, question, answer and action sequence, enabling deterministic reconstruction of the task description.
Human Verification and Generator Bias Audit.
Three annotators review correctness, answerability and non-triviality, with majority decisions resolving disagreement. Separate audits test wording/category shortcuts, diversity and matched human-authored tasks. These checks reduce identifiable artifacts; they do not establish absence of every generator bias.
| Category | Definition |
|---|---|
| Physical Structure | Reveal hidden containment capacity through manipulation |
| Physical Dynamics | Predict stability and motion under physical forces |
| Specular Reflection | Infer object relations from mirror reflections |
| Perceptual Grounding | Resolve viewpoint-dependent perceptual ambiguities |
| Metric Comparison | Overcome forced-perspective distortions in size and distance |
| Spatial Relations | Navigate to vantage points that break projective symmetry |
| Cognitive Mapping | Construct topological representations through locomotion |
| Enumerative Perception | Count objects despite occlusion, merging, or enclosure |
| Temporal Understanding | Infer state changes through interaction and observation |
| Action Sequencing | Reason over action sequences and causal dependencies |
3.3 Task Categories and Statistics
Ten categories cover physical structure/dynamics, reflections, perceptual grounding, metric comparison, spatial relations, cognitive mapping, counting, temporal understanding and action sequencing. The original diagrams preserve their definitions, action requirements and distribution across 29 subcategories.
4 Experiments
The experiments vary both observation access and the agent’s control over acquiring evidence. Per-category results are more informative than a single aggregate because failure modes differ substantially.
| 2D + VLM | 3D + LLM | Human Performance | ||||||||||||||||||
| GPT-5 | Gemini 3.1 | VGGT + Gemini | GT 3D + Gemini | |||||||||||||||||
| Category / Subcategory |
Passive
|
Passive
|
Active
|
GT
|
Passive
|
Passive
|
Active
|
GT
|
Passive
|
Passive
|
Active
|
GT
|
Passive
|
Passive
|
Active
|
GT
|
Passive
|
Passive
|
Active
|
GT
|
| Perceptual Grounding | ||||||||||||||||||||
| View Hallucination | 11.7 | 20.2 | 60.1 | 87.8 | 39.9 | 32.9 | 68.1 | 91.1 | 56.3 | 59.0 | 76.1 | 87.6 | 61.6 | 60.6 | 74.1 | 89.0 | 52.2 | 45.8 | 80.8 | 83.8 |
| Partial Occlusion | 32.6 | 26.3 | 47.4 | 86.3 | 30.5 | 32.6 | 70.5 | 88.4 | 51.6 | 48.4 | 53.7 | 86.3 | 56.8 | 55.8 | 72.6 | 91.6 | 37.9 | 34.7 | 80.0 | 87.4 |
| Material Transparency | 30.3 | 36.7 | 66.1 | 96.3 | 44.0 | 45.0 | 52.3 | 88.0 | 37.4 | 29.4 | 31.8 | 90.9 | 27.8 | 31.2 | 60.4 | 100 | 41.3 | 44.0 | 93.6 | 97.2 |
| Physical Structure | ||||||||||||||||||||
| Rigid Containment | 45.0 | 42.5 | 42.5 | 95.0 | 47.5 | 40.0 | 67.5 | 97.5 | 27.5 | 37.5 | 57.5 | 72.5 | 45.0 | 42.5 | 65.0 | 95.0 | 47.5 | 42.5 | 92.5 | 100 |
| Liquid Volume | 66.2 | 66.2 | 81.6 | 86.0 | 69.9 | 67.6 | 80.9 | 86.8 | 65.4 | 56.7 | 71.3 | 77.2 | 74.5 | 74.5 | 83.1 | 86.8 | 79.4 | 66.9 | 89.7 | 86.8 |
| Deformable | 42.9 | 41.8 | 55.1 | 75.5 | 34.7 | 41.8 | 43.9 | 78.6 | 43.9 | 42.9 | 49.0 | 81.6 | 46.9 | 54.1 | 99.0 | 99.0 | 57.1 | 60.2 | 76.5 | 82.7 |
| Physical Dynamics | ||||||||||||||||||||
| Inclined Plane | 57.4 | 60.7 | 77.0 | 86.9 | 65.6 | 62.3 | 83.6 | 88.5 | 67.2 | 62.3 | 80.3 | 86.9 | 63.9 | 63.9 | 83.6 | 91.8 | 60.7 | 62.3 | 83.6 | 83.6 |
| Stacking & Stability | 34.8 | 37.1 | 62.9 | 86.5 | 38.2 | 36.0 | 78.7 | 84.3 | 34.8 | 39.3 | 55.1 | 62.9 | 27.2 | 33.7 | 68.5 | 86.5 | 36.0 | 39.3 | 84.3 | 86.5 |
| Specular Reflection | ||||||||||||||||||||
| Reflection Authoring | 68.7 | 70.7 | 70.3 | 73.7 | 60.6 | 60.6 | 64.9 | 67.0 | 52.5 | 54.6 | 53.5 | 55.6 | 55.6 | 58.6 | 55.6 | 60.6 | 94.9 | 88.9 | 96.0 | 96.0 |
| Spatial Relations | 50.4 | 38.9 | 54.8 | 58.4 | 43.4 | 42.5 | 44.2 | 46.9 | 41.6 | 39.8 | 41.6 | 43.4 | 38.9 | 35.4 | 54.2 | 41.6 | 78.8 | 78.8 | 84.1 | 87.6 |
| Correspondence | 39.8 | 51.1 | 52.3 | 56.8 | 37.5 | 40.9 | 42.0 | 42.0 | 37.5 | 43.2 | 48.9 | 48.9 | 31.8 | 39.8 | 42.3 | 48.9 | 85.2 | 80.7 | 89.8 | 92.0 |
| Spatial Relations | ||||||||||||||||||||
| Linear Alignment | 27.7 | 31.9 | 42.6 | 60.6 | 47.9 | 44.6 | 67.0 | 77.7 | 45.7 | 38.3 | 53.2 | 59.6 | 73.4 | 73.4 | 84.0 | 89.4 | 51.1 | 39.4 | 73.4 | 79.8 |
| Geometric Configuration | 25.3 | 20.4 | 26.0 | 26.0 | 27.5 | 22.3 | 27.5 | 44.6 | 9.9 | 11.3 | 18.6 | 32.6 | 70.8 | 73.6 | 88.0 | 100 | 32.4 | 33.5 | 74.3 | 86.3 |
| Physical Contact | 40.0 | 41.7 | 64.2 | 90.0 | 60.8 | 55.8 | 70.0 | 74.2 | 59.2 | 41.7 | 59.2 | 78.2 | 65.8 | 67.5 | 70.8 | 72.5 | 35.8 | 40.8 | 88.3 | 90.8 |
| Metric Comparison | ||||||||||||||||||||
| Dimensional Size | 42.5 | 44.9 | 67.7 | 80.3 | 44.3 | 41.3 | 68.3 | 82.6 | 50.4 | 40.1 | 59.3 | 80.8 | 58.6 | 60.5 | 69.9 | 85.9 | 48.5 | 56.9 | 82.6 | 91.9 |
| Spatial Distance | 53.9 | 49.1 | 58.6 | 73.7 | 52.6 | 50.5 | 59.9 | 80.3 | 50.7 | 47.3 | 55.0 | 71.4 | 59.6 | 61.2 | 64.5 | 96.7 | 57.9 | 59.9 | 69.1 | 78.9 |
| Cognitive Mapping | ||||||||||||||||||||
| Connectivity | 68.3 | 70.0 | 68.3 | 78.3 | 51.7 | 48.3 | 60.0 | 85.0 | 66.7 | 66.7 | 73.3 | 83.3 | 65.0 | 65.0 | 71.7 | 86.7 | 68.3 | 68.3 | 81.7 | 91.7 |
| Traversable Passage | 68.3 | 66.7 | 71.7 | 73.3 | 66.7 | 61.7 | 73.3 | 78.3 | 68.3 | 63.3 | 66.7 | 78.3 | 71.7 | 73.3 | 71.7 | 80.0 | 70.0 | 71.7 | 78.3 | 85.0 |
| Regional Boundary | 65.0 | 63.8 | 65.0 | 67.5 | 63.8 | 62.5 | 65.0 | 65.0 | 65.0 | 65.0 | 62.5 | 67.5 | 60.0 | 68.8 | 62.5 | 68.8 | 61.3 | 62.5 | 67.5 | 70.0 |
| Long-Term Navigation | 40.0 | 38.3 | 41.7 | 50.0 | 33.3 | 35.0 | 33.3 | 41.7 | 36.7 | 40.0 | 43.3 | 51.6 | 36.7 | 41.7 | 43.3 | 40.0 | 35.0 | 36.7 | 51.7 | 61.7 |
| Enumerative Perception | ||||||||||||||||||||
| Counting w Occlusion | 3.3 | 3.3 | 13.3 | 56.7 | 3.3 | 3.3 | 10.0 | 63.3 | 0.0 | 0.0 | 13.3 | 53.3 | 33.3 | 80.0 | 83.3 | 100.0 | 6.7 | 10.0 | 53.3 | 76.7 |
| Spatial Segmentation | 3.3 | 6.7 | 26.7 | 63.3 | 3.3 | 3.3 | 16.7 | 70.0 | 3.3 | 3.3 | 23.3 | 56.7 | 43.3 | 80.0 | 66.7 | 86.7 | 13.3 | 23.3 | 63.3 | 70.0 |
| Merged Observation | 38.3 | 35.0 | 51.7 | 50.0 | 43.3 | 51.7 | 61.7 | 75.0 | 63.3 | 51.7 | 55.0 | 65.0 | 51.7 | 63.3 | 93.3 | 100.0 | 60.0 | 60.0 | 65.0 | 73.3 |
| Category Ambiguity | 8.3 | 10.0 | 8.3 | 41.7 | 13.3 | 13.3 | 15.0 | 48.3 | 21.7 | 26.7 | 26.7 | 46.7 | 18.3 | 76.7 | 40.0 | 93.3 | 23.3 | 25.0 | 51.7 | 61.7 |
| Structural Enclosure | 5.0 | 10.0 | 22.5 | 67.5 | 2.5 | 0.0 | 10.0 | 52.5 | 0.0 | 0.0 | 12.5 | 52.5 | 40.0 | 60.0 | 50.0 | 100.0 | 10.0 | 22.5 | 42.5 | 77.5 |
| Illumination Variability | 6.0 | 22.0 | 22.0 | 46.0 | 12.0 | 16.0 | 22.0 | 58.0 | 10.0 | 28.0 | 30.0 | 40.0 | 20.0 | 84.0 | 42.0 | 96.0 | 20.0 | 34.0 | 62.0 | 66.0 |
| Temporal Understanding | ||||||||||||||||||||
| Unobserved Change | 40.5 | 41.2 | 51.4 | 77.0 | 37.2 | 37.8 | 47.3 | 74.5 | 37.8 | 41.9 | 45.8 | 76.6 | 39.9 | 42.6 | 46.5 | 74.5 | 39.4 | 39.4 | 70.8 | 81.0 |
| Agent Observation | 40.6 | 30.1 | 51.0 | 72.7 | 37.6 | 36.1 | 58.6 | 65.4 | 32.7 | 27.9 | 34.2 | 56.5 | 36.3 | 33.3 | 76.7 | 87.8 | 38.3 | 42.1 | 83.5 | 90.2 |
| Action Sequencing | ||||||||||||||||||||
| Action Order Inference | 36.4 | 37.7 | 41.6 | 67.5 | 44.2 | 46.8 | 54.5 | 72.7 | 35.1 | 31.1 | 50.6 | 58.4 | 44.2 | 46.8 | 51.9 | 75.3 | 40.3 | 41.6 | 74.0 | 81.8 |
4.1 Models and Evaluation Setup
Four conditions provide the initial view, 30 random passive views, up to 30 actively selected steps, or views from a verified ground-truth trajectory. GPT-5 and Gemini 3.1 use images; VGGT+Gemini uses reconstructed 3D scene graphs, while ground-truth 3D+Gemini uses simulator geometry. Models receive no task-specific fine-tuning, and matched subsets support comparisons. Random multiview is an intentionally unselected baseline; oracle trajectories and perfect geometry offer different information advantages, so the gaps are diagnostic rather than a perfectly isolated causal decomposition.
4.2 What Really Holds Spatial Intelligence Back: Seeing or Acting?
Active agents discover several ways to resolve the same question, such as moving behind a glass, inspecting from above, lifting it or pouring its contents. Yet oracle trajectories often improve performance much further: GPT-5 rises from 42.5% active to 95.0% oracle-view accuracy on rigid containment, and from 64.2% to 90.0% on contact. Random views can hurt, as spatial-distance accuracy falls from 53.9% single-view to 49.1% random-multiview. Geometry classification and reflections remain difficult even with good views. These results support both action-selection failures and residual interpretation failures; no universal “perceptual ceiling” is established.
4.3 When Does 3D Help, and When Does It Hurt?
For Gemini, perfect 3D raises geometric-configuration accuracy from 27.5% to 70.8%, while VGGT reconstruction lowers it to 9.9%. On occluded counting, the corresponding scores are 3.3%, 33.3% and 0%. Thus explicit geometry helps when reliable, but fragmented objects, false instances and incorrect relations can mislead downstream symbolic reasoning. Perfect simulator geometry is an oracle condition, not a demonstrated deployable reconstruction pipeline.
4.4 How Far Are Models From Human-Level Spatial Reasoning?
Humans are often closer to models when both receive fixed views, but gain much more from controlling exploration. For example, human active accuracy reaches 88.3% on contact and 93.6% on transparent containment, compared with 64.2% for GPT-5 and 52.3% for Gemini respectively. Qualitative traces show premature confident answers and repeated confirmatory motions. Appendix K supplies behavioral proxies for viewpoint diversity, contrastive evidence and answer revision; these support the observed gap without establishing a general impossibility of improving it through better models or prompting.
5 Conclusion
ESI-Bench makes information acquisition part of spatial evaluation. Its main contribution is a controlled diagnostic setting that exposes failures in selecting actions, integrating evidence and deciding when to answer. Success remains constrained by the simulator, task construction and high-level action interface.
Acknowledgement
The work acknowledges support from ONR MURI, the Korean IITP/MSIT program, TRI’s university program and Google’s Gemini Academic Program compute resources.
Appendix A Contribution Statement
Yining Hong led the idea, integration, writing and most task pipelines; Jiageng Liu developed deformable-object, reflection, mapping, counting and change-detection pipelines and related assets; Han Yin contributed related work and figures. The remaining authors are credited with advising.
Appendix B Broader Impacts & Limitations
The intended benefit is diagnostic: identifying premature commitment and poor evidence acquisition before deploying embodied systems. Scenes are synthetic and avoid collecting real personal data. The explicit limitation is rendering realism, including textures, lighting, blur and sensor effects. The high-level action abstraction and simulation setting further limit conclusions about real robot execution and safety.
Appendix C Per-Category Task Description
The full taxonomy table retains every subcategory’s definition and example. Material transparency belongs to perceptual grounding in this taxonomy, although Appendix Q.1 labels it under physical structure; the source naming inconsistency is preserved.
| Category | Subcategory | Description | Example |
| Physical | Rigid Containment | Plan the placement of multiple objects across multiple containers | How can all the toys be fit into the boxes? |
| Structure | Liquid Volume | Compare liquid-holding capacity of containers | Which container has larger volume? |
| Deformable | Infer structure from deformable object | What’s under the sweater, apple or mug? | |
| Physical | Inclined Plane | Predict object motion and stability on slopes | Can the apple sit stable on the slope? |
| Dynamics | Stacking & Stability | Whether objects stack or balance given shape, mass and geometry | How to stack and stabilize these three objects? |
| Specular | Reflection Authoring | Distinguish real objects from mirror reflections | Is the object real or reflected? |
| Reflection | Spatial Relations | Infer relations among objects across mirror and real-world views | What relations does the mirror reveal? |
| Correspondence | Identify which objects appear in mirror given the real-world scene | Which of the 3 snacks appears in the mirror? | |
| Perceptual | Partial Occlusion | Reason about objects hidden behind other scene elements | Golf Stick or Umbrella behind the Wall? |
| Grounding | View Hallucination | Detect objects whose visibility changes critically with viewing angle | Cabinet or piano from this view? |
| Material Transparency | Reason about objects seen through transparent surfaces | Is the object inside the glass or not? | |
| Metric | Dimensional Size | Compare relative sizes of objects | Which vase is larger? |
| Comparison | Spatial Distance | Compare relative distances with respect to a reference object | Which flower is closer to the table? |
| Spatial | Linear Alignment | Whether objects are arranged along a common axis | Do the glasses form a line? |
| Relations | Geometric Configuration | Identify the shape formed by a set of objects | Do the cups form an equilateral triangle? |
| Physical Contact | Detect whether two or more objects are in direct contact | Are the teddy bears touching each other? | |
| Cognitive | Connectivity | Whether two locations or regions are mutually reachable | Is region A connected to C? |
| Mapping | Traversable Passage | Identify navigable corridors or passageways between regions | Passage between the rooms? |
| Regional Boundary | Identify and delineate distinct functional spatial regions | Boundaries of this region? | |
| Long-Term Navigation | Plan multi-step navigation trajectories toward a distant goal | What room is behind the wall? | |
| Enumerative | Counting w Occlusion | Count objects partially obscured by other scene elements | How many balls are under the blanket? |
| Perception | Spatial Segmentation | Count objects separated across distinct spatial regions | One or two cylinders separated by the post? |
| Category Ambiguity | Count visually similar objects requiring fine-grained distinction | How many apples among the balls? | |
| Merged Observation | Count groups that appear visually merged from a single viewpoint | How many separate stacks of books? | |
| Illumination Variability | Count objects under challenging or non-uniform lighting | Objects in the dim scene? | |
| Structural Enclosure | Count objects hidden within enclosed or covered spaces | Objects in the microwave? | |
| Temporal | Unobserved Change | Infer scene changes that occurred during an unobserved interval | What changed when I looked away? |
| Understanding | Agent Observation | Reason about scene dynamics from other agents | Other robot’s world model? |
| Action Sequencing | Action Order Inference | Determine the correct procedural ordering of a sequence of actions | What is the action sequence for assembly? |
Appendix D Per-Category Task Distribution
The original distribution charts show how instances are divided among each category’s subcategories. These counts contextualize aggregate accuracy and do not imply balanced difficulty.
Appendix E Human Study Protocol
The human study uses the same discrete interface and step budget as the active models, with independent task judgments and recorded exploration traces.
Participants and compensation.
Twenty-eight previously unexposed volunteers with normal or corrected vision were recruited and paid above local minimum wage. The paper reports institutional compliance and no required IRB review.
Interface.
The web interface shows the same egocentric pixels and offers the same locomotion, rotation, manipulation and answer/confidence commands, with a 30-step cap.
Task assignment.
Participants each cover at least six categories; each instance receives at least five independent attempts. The paper says answer disagreements are resolved by a third annotator but gives limited detail on reconciling that wording with the five-person protocol.
Instructions.
A brief tutorial and two worked examples explain exploration. Participants receive no task-specific solution hints and are told to answer when sufficiently confident.
Data collected.
Recorded data include all actions and frames, final answers/confidence, stopping step and any intermediate confidence ratings.
Appendix F More Benchmark Examples
Additional examples retain the visual diversity of geometry, observation, counting, transparency, contact, size, slope, distance, deformation, occlusion and reflection tasks.
Appendix G More Qualitative Examples
The supplementary trajectories show alternative useful strategies as well as persistent reflection mistakes and merged-object undercounting. They illustrate mechanisms and failures rather than estimating their prevalence.
Appendix H Human Verification and Generator Bias Analysis
The generator audit checks candidate validity, potential shortcuts and task-source difficulty. Simulator and human filtering are central: these statistics concern reviewed proposals rather than unfiltered GPT-4o outputs.
H.1 Human Verification Protocol
Reviewers see the question/answer, initial image, verified trajectory, final simulator state and relevant metadata. Each instance must pass three distinct criteria.
Correctness.
Correctness requires physically plausible placements, executable transitions and an answer consistent with simulator state; floating, disappearing or intersecting objects are rejected.
Answerability.
Answerability requires a unique answer that can be revealed through the available actions within budget. Invisible evidence, subjective interpretation and multiple plausible labels trigger rejection.
Non-triviality.
Non-triviality rejects obvious initial-view solutions and category shortcuts. Three independent binary reviews use majority resolution, and an instance must pass every axis.
H.2 Verification Scores for GPT-4o-Generated Tasks
After automatic filtering, human verification averages 92.0% correctness, 91.0% answerability, 88.4% non-triviality and 85.1% passing all three. Enumerative perception has the lowest overall pass rate at 78.6%, while cognitive mapping reaches 91.2%. Final evaluation excludes rejected candidates.
| Category | Correctness | Answerability | Non-triviality | Overall Pass |
|---|---|---|---|---|
| Perceptual Grounding | 94.7 | 93.8 | 91.6 | 88.9 |
| Physical Structure | 91.3 | 90.5 | 88.2 | 84.6 |
| Physical Dynamics | 89.8 | 88.7 | 86.9 | 82.1 |
| Specular Reflection | 93.4 | 92.1 | 89.5 | 86.7 |
| Spatial Relations | 92.6 | 91.4 | 87.8 | 85.2 |
| Metric Comparison | 95.1 | 94.3 | 90.7 | 89.4 |
| Cognitive Mapping | 96.4 | 95.8 | 92.6 | 91.2 |
| Enumerative Perception | 87.9 | 86.5 | 83.9 | 78.6 |
| Temporal Understanding | 90.7 | 89.9 | 87.1 | 83.6 |
| Action Sequencing | 88.5 | 87.3 | 85.4 | 80.9 |
| Average | 92.0 | 91.0 | 88.4 | 85.1 |
H.3 Bias Audit for GPT-4o-Generated Tasks
Shortcut tests remove images and interaction histories, then expose either the question alone or the question plus an unordered object-category list. Diversity statistics complement these tests but do not exhaust possible biases.
Linguistic bias.
Question-only accuracy averages 36.6% in the audit versus 42.5% passive single-view. The full table is retained because chance rates vary by answer type and the paper does not provide a uniform chance baseline.
Object-category bias.
Metadata-only accuracy averages 39.3%, while active exploration averages 56.9%. This ordering is not universal: in counting, active accuracy is 22.6%, below question-only 24.3% and metadata-only 26.1%; some other category priors also match or exceed passive images. The table therefore supports an aggregate advantage, not the prose’s claim that active exploration beats every shortcut in every category. Average normalized answer entropy is 0.92 and object-diversity score 0.85.
| Category | Question-Only | Metadata-Only | Passive Single | Active |
|---|---|---|---|---|
| Perceptual Grounding | 26.1 | 28.3 | 32.8 | 66.3 |
| Physical Structure | 31.5 | 35.0 | 50.7 | 64.1 |
| Physical Dynamics | 49.2 | 51.0 | 51.9 | 81.2 |
| Specular Reflection | 47.2 | 49.5 | 47.2 | 50.4 |
| Spatial Relations | 30.8 | 35.1 | 45.4 | 54.8 |
| Metric Comparison | 49.2 | 50.8 | 48.5 | 64.1 |
| Cognitive Mapping | 47.6 | 50.1 | 53.9 | 57.9 |
| Enumerative Perception | 24.3 | 26.1 | 13.0 | 22.6 |
| Temporal Understanding | 35.7 | 38.1 | 37.4 | 53.0 |
| Action Sequencing | 24.8 | 29.3 | 44.2 | 54.5 |
| Average | 36.6 | 39.3 | 42.5 | 56.9 |
| Category | Answer Balance | Object Diversity |
|---|---|---|
| Perceptual Grounding | 0.92 | 0.87 |
| Physical Structure | 0.89 | 0.84 |
| Physical Dynamics | 0.95 | 0.82 |
| Specular Reflection | 0.93 | 0.85 |
| Spatial Relations | 0.90 | 0.88 |
| Metric Comparison | 0.96 | 0.86 |
| Cognitive Mapping | 0.94 | 0.80 |
| Enumerative Perception | 0.88 | 0.89 |
| Temporal Understanding | 0.91 | 0.83 |
| Action Sequencing | 0.87 | 0.81 |
| Average | 0.92 | 0.85 |
H.4 Comparison with Human-Generated Tasks
Matched GPT-generated tasks are consistently somewhat easier than human-authored ones in Table 7: gaps range from 2.7 to 4.4 points, averaging 3.5. The model/human ordering is similar across sources, which argues against a large reversal of difficulty but does not demonstrate statistical equivalence or zero generator bias.
| Model | GPT-4o-Generated | Human-Generated | Absolute Gap |
|---|---|---|---|
| GPT-5 Active | 51.6 | 48.9 | 2.7 |
| Gemini 3.1 Active | 56.9 | 53.4 | 3.5 |
| VGGT + Gemini Active | 51.6 | 48.1 | 3.5 |
| GT 3D + Gemini Active | 68.0 | 63.6 | 4.4 |
| Human Active | 75.3 | 72.1 | 3.2 |
| Average | 60.7 | 57.2 | 3.5 |
Appendix I Diverse Emergent Behaviors in Active Exploration
Human annotation of successful trajectories finds three to six strategy clusters in seven representative subcategories; transparent containment has six, while partial occlusion, contact and size have four each. These are post-hoc clusters of successes, not a frequency distribution over all attempts. Diversity alone does not imply accurate confidence or reliable stopping.
| Subcategory | # Emergent Strategy Clusters |
|---|---|
| Partial Occlusion | 4 |
| Material Transparency | 6 |
| Rigid Containment | 3 |
| Liquid Volume | 3 |
| Physical Contact | 4 |
| Spatial Distance | 3 |
| Dimensional Size | 4 |
Appendix J Failure Analysis of VGGT-Based 3D Augmentation
Manual analysis identifies duplicated object fragments, hallucinated instances and corrupted spatial relations in VGGT-derived scene graphs. These errors explain why adding a seemingly authoritative graph can degrade counting and fine-grained geometry. The proposed direction is uncertainty-aware reconstruction and graph construction; no quantitative error-frequency breakdown is supplied.
| Failure Mode | Main Effect | Affected Tasks |
|---|---|---|
| Object duplication | One partially observed object is reconstructed as multiple fragments or repeated instances, causing over-counting. | Counting w Occlusion; Merged Observation |
| Object hallucination | Noisy geometry or background fragments are converted into spurious object proposals. | Enumerative Perception; Perceptual Grounding |
| Spatial-relation corruption | Depth errors distort relative position, contact, containment, and near/far relations. | Spatial Relations; Metric Comparison; Physical Structure |
Appendix K Operationalizing the Metacognitive Gap
Humans show 71.8% view diversity, 62.7% contrastive-view rate and 41.3% answer revision; Gemini scores 43.5/31.5/18.9 and GPT-5 39.2/28.7/16.4. These measures operationalize exploration behavior. Answer changes are not necessarily correct updates, and viewpoint diversity is not itself calibration; the paper treats these as complementary proxies, not a complete metacognition benchmark.
| Axis | Definition | Signal |
|---|---|---|
| Evidence sufficiency | Whether the agent gathers sufficiently diverse observations before answering. | View diversity |
| Falsification seeking | Whether actions seek evidence that could disconfirm the current hypothesis. | Contrastive views |
| Belief revision | Whether the agent changes its answer after contradictory evidence. | Answer updates |
| Agent | View Diversity | Contrastive View Rate | Belief Revision Rate |
|---|---|---|---|
| Human Active | 71.8 | 62.7 | 41.3 |
| Gemini 3.1 Active | 43.5 | 31.5 | 18.9 |
| GPT-5 Active | 39.2 | 28.7 | 16.4 |
Appendix L Analysis of Passive Multi-View Evaluation
Thirty randomly selected, unordered views can miss useful evidence, overload the model, lack action-linked alignment or introduce apparent contradictions. The paper explicitly cautions that this is not an optimal multiview baseline. Better selection, ranking or cross-view alignment could improve it, so the result concerns unstructured view accumulation rather than multiview reasoning in general.
| Source | Possible Effect |
|---|---|
| Random view selection | Views may cover the scene broadly but miss the diagnostic evidence needed for the question. |
| Image overload | Many irrelevant images can dilute the few useful observations and increase reasoning burden. |
| No action grounding | Views are not connected to self-selected actions, making spatial integration harder. |
| Conflicting evidence | Occluded or partial views can introduce misleading cues across images. |
Appendix M Why a Synthetic Simulator Benchmark?
Simulation gives exact hidden-state labels and repeatable control over occlusion, contact, containment and connectivity. It deliberately reduces sensor and actuation confounds to diagnose evidence selection. The authors make no direct sim-to-real transfer claim; real deployment adds calibration, control, noise and environment variation.
Appendix N Active Agent Prompting and Action Selection
At every step the model receives the question, current image, prior action-observation history and action vocabulary, then selects one action. Execution returns a new frame until the model answers or exhausts 30 steps.
Action-selection prompt.
The complete original prompt is retained below, including placeholders, action signatures and the required Reason/Action response format. It asks the agent to gather evidence and delay answering until enough is available; it does not prescribe task-specific strategies.
Action parsing and execution.
Invalid or multiple actions receive one repair prompt. A second invalid output consumes a step without moving the agent. Object references are matched to observed identifiers where possible; physically failed manipulations remain part of the trajectory. Thus high-level control reduces motor confounds but does not remove every execution failure.
Appendix O Why High-Level Actions?
High-level actions isolate decisions about where to look, what to manipulate and when to stop. The benchmark intentionally omits continuous motor commands and robot-specific controller learning. Low-level competence is a complementary evaluation dimension, not measured by these results.
Appendix P Step Budget Ablation
Gemini’s budget ablation improves rapidly through roughly 15 steps, levels off around 20–30, and slightly declines beyond 40. The 30-step cap is justified for this tested setup; the curve does not prove that longer, better-designed exploration could never help.
Appendix Q Per-Category Task Construction Details
The following construction notes preserve each pipeline’s object choice, physical setup, initial ambiguity and label rule. These engineering definitions matter because labels often use explicit geometric thresholds rather than unrestricted real-world concepts. Several descriptions omit parameters or have inconsistent wording; those gaps are identified rather than filled in.
Initial Agent Viewpoint Setup.
Initial views deliberately hide diagnostic structure through collinearity, grazing angles, occlusion or forced perspective. Instances with no task-relevant visibility in any verified view are rejected. The construction tests targeted ambiguity rather than a random sample of everyday camera poses.
Q.1 Physical Structure: Material Transparency
Transparent containment distinguishes an object inside a clear vessel from one aligned behind it.
Object selection.
Containers and small objects come from separate human-curated category pools, with models drawn from BEHAVIOR-1K.
Scale.
The object’s largest dimension is capped at half the container’s narrowest dimension, with an additional sampled size limit. The source leaves that sampling interval symbolic.
Placement.
Positive placements use Inside-state sampling or a settled interior fallback plus bounding-box containment. Negative placements try lateral offsets and correct the final gap to 1 cm; unsuccessful placements are rejected.
Agent intitial spotion and view.
The initial camera aligns the object with the vessel, maximizing ambiguity between inside and behind.
Ground-truth label.
The label is binary inside versus outside.
Q.2 Physical Structure: Liquid Volume
Liquid capacity is defined operationally by the simulator’s water-particle measurement.
Object selection.
Container pairs differ by at least 200 particles of premeasured capacity. Offline dipping counts the largest connected water-particle component using a spatial-neighbor graph.
Pouring Simulation.
A filled container’s particles are transferred to the other vessel, yielding complete transfer or overflow. This is the reported simulation procedure.
Ground-truth label.
The answer identifies the vessel with greater offline particle capacity.
Q.3 Physical Structure: Deformable
The deformable task asks which object is hidden under a cloth.
Object selection.
A GPT-proposed pool is filtered to portable objects of roughly 5–15 cm average radius. Each question uses one target, three category distractors and a cloth no larger than 50 cm per side.
Object placement.
The target settles at a free floor location; a downward-moving cloth is dropped over it.
Agent initial view.
The camera uses a fixed azimuth, a distance/height based on cloth footprint and a 70° field of view.
Ground-truth label.
The answer is the hidden category among four shuffled choices.
Q.4 Physical Dynamics: Inclined Plane
Slope stability depends on randomized physical parameters and finite simulation thresholds.
Object selection.
A compatible nonstructural object is selected and its mass perturbed.
Slope construction.
Slope angle is 10–45°, base half-length 20–30 cm, static friction 0.3–3.0 and dynamic friction no greater than static. Restitution is 0.1; physics and action timesteps are 1/240 s and 1/60 s.
Simulation and label derivation.
After 30 simulation steps, more than 3 cm horizontal motion indicates sliding; a drop more than 5 cm below the slope bottom indicates falling.
Agent initial viewpoint.
The initial view shows the object near the slope before the test.
Ground-truth label.
The final task label is stable versus not stable, although the construction also distinguishes sliding from falling.
Q.5 Physical Dynamics: Stacking and Stability
Stacking tests which object orders satisfy the simulator’s stability checks.
Object selection.
GPT-4o selects two or three solid, plausibly stackable objects from the sampled category set.
Scale normalization.
All objects receive a common horizontal footprint based on the largest original extent, with proportional vertical scaling.
Placement.
Each permutation settles for 60 steps. A layer must be elevated by at least 2 cm, overlap horizontally with AABB IoU at least 0.1, and retain an up-axis dot product above 0.9. These are operational stability criteria.
Ground-truth label.
Valid stacking orders are derived from the settled placement checks; the full text describes both support overlap and tilt checks.
Q.6 Metric Comparison: Spatial Distance
Distance comparison creates a forced-perspective ambiguity around a reference object.
Object selection.
Reference objects need at least two bounding-box corners with more than 0.5 m clearance; GPT selects two similar-category comparison objects.
Placement.
Objects are placed diagonally outward from different reference corners at near/far distances. The source omits both numeric distances, so they cannot be reconstructed from this description alone.
Agent initial position and view.
The initial pose makes apparent size and occlusion unreliable indicators of distance.
Ground-truth label.
The answer selects the designated nearer or farther object, according to the question.
Q.7 Metric Comparison: Dimensional Size
Size comparison separates physical scale from apparent image size.
Object selection.
Two instances of one household model use scales 1.0 and a sampled 1.2–1.3.
Reference objects.
Distinct reference objects sit within 5 cm of each target’s boundary and at least 30 cm from the other target, providing names for the alternatives.
Agent initial position and view.
The initial camera creates unequal distances and partial occlusion.
Ground-truth label.
The answer uses the associated reference object to identify the larger or smaller target.
Q.8 Spatial Relations: Physical Contact
Contact questions distinguish physical proximity from misleading projection.
Object selection.
Two small-to-medium objects are selected, excluding tiny items and large furniture.
Placement.
Positive cases align faces with small positional variation and use segmentation-mask adjacency; negatives are moved until gaps exceed 5 cm. Live bounding boxes verify the final gaps, alongside the broader pipeline’s contact checks.
Initial agent position and view.
The starting view makes touching and separation visually confusable.
Ground-truth label.
The answer is touching or not touching under the construction’s geometric/contact criteria.
Q.9 Spatial Relations: Linear Alignment and Geometric Configuration
One construction family covers both collinearity and triangle shape.
Object selection.
Three compact objects with nonzero height are selected.
Linear Alignment placement.
Positive line cases use collinear centers; negative cases introduce lateral offsets.
Geometric Configuration placement.
Equilateral examples use side lengths of 0.4–0.65 m and reject object sets with more than a 2× footprint ratio. The paragraph does not fully describe generation of the other triangle classes.
Agent initial position and view.
A grazing view near the centroid makes distinct 3D arrangements project similarly.
Ground-truth labels.
Line alignment is binary; triangle labels distinguish equilateral, isosceles and random configurations.
Q.10 Physical Structure: Rigid Containment
Rigid containment asks for a complete object-to-container assignment.
Object selection.
The pipeline selects containers of several sizes and three or four objects, with scaling intended to give each object a specific valid destination.
Pick-place simulation.
Placement uses physics settling, a complete AABB containment check and a fallback attempt; grasp/release and eye-camera frames represent each manipulation stage. The source omits the fallback primitive’s name.
Ground-truth label.
The answer specifies each object’s correct target container.
Q.11 Action Sequencing: Action Order Inference
Action sequencing combines fit constraints with a placement hierarchy.
Object selection.
GPT proposes a two- or three-object dependency chain, with fitting candidates, near-miss candidates and confusers.
Layout.
The fixed target is central, with alternatives arranged to either side on a support surface.
Ground-truth label.
The answer specifies the valid candidate and relation at each step of the required sequence.
Q.12 Temporal Understanding: Agent Observation
Agent observation counts what a navigating robot passes while another agent watches.
Object selection.
One to three target categories are chosen for placement along a path.
Agent setup.
A navigating robot follows a validated path of at least ten waypoints. A fixed observer watches from 5 m height with the ceiling removed; invalid paths are resampled.
Object placement.
Targets are placed on suitable surfaces within 1 m of the planned route.
Observer agent position and view.
The observer remains at the starting overhead location and tracks the moving robot.
Ground-truth label.
Labels count the categories/instances passed within the proximity threshold; this is the simulator’s operational encounter criterion.
Q.13 Temporal Understanding: Unobserved Changes
Unobserved changes compare box contents across two generated phases.
Object selection.
Questions ask whether a change occurred, which change occurred, or which colored box changed. Contents may be replaced, removed, added or left unchanged.
Object placement.
Collision-aware box placements remain fixed while phase-one contents are removed and phase-two contents are spawned.
Agent initial view.
A clear boundary viewpoint is chosen to show as many boxes as possible.
Ground-truth label.
Labels come from sampled states: yes/no, change type or changed box. Oracle close-ups open boxes in each phase. The displayed action vocabulary does not explicitly list an open command, leaving that interaction detail underspecified.
Q.14 Perceptual Grounding: Partial Occlusion
Partial occlusion uses a target and a plausible confusable occluder.
Object selection.
A curated target pool and compatibility map determine the pair; the occluder is scaled to 1.1× the target’s dimensions.
Placement.
Both objects sit on a table, with the occluder near a target bounding-box corner.
Agent initial viewpoint.
The camera is 15° off their aligned axis at target-center height, showing only ambiguous partial evidence.
Ground-truth label.
The label is the target’s actual category.
Q.15 Perceptual Grounding: View Hallucination
View hallucination tests recognition from a misleading object angle.
Object selection.
The target and confusable alternatives come from a curated category/view-condition map.
Placement.
The target is placed on a valid table or floor support.
Agent initial viewpoint.
The initial azimuth is selected for high category ambiguity.
Ground-truth label.
The correct category is contrasted with the stored visual confusers.
Q.16 Specular Reflection : Reflection Authorization
Reflection authoring distinguishes reflected visibility from a directly seen real object.
Object selection.
Nonstructural, nonreflective floor-placeable objects are filtered to an equivalent radius of 2–20 cm.
Mirror and initial-view setup.
A free-standing mirror approximately faces the room center with up to 30° yaw variation. The initial camera is in front, roughly 2 m away, with an unobstructed mirror view.
Object placement.
Positive examples are mirror-visible but not directly visible; negatives are directly visible without a valid mirror ray. Geometry and visibility checks enforce these conditions.
Ground-truth label.
The answer is yes when the queried object is seen through the mirror, otherwise no.
Q.17 Specular Reflection : Spatial Relationship
Reflection distance asks about real object positions seen through a mirror.
Object selection.
Two categories are sampled from the reflection-compatible pool.
Mirror and initial-view setup.
Mirror and camera setup follows the reflection-authoring pipeline.
Object placement.
Both objects must have valid reflected rays within mirror width and field of view, with floor and occlusion checks.
Ground-truth label.
The closer real 3D position determines A or B; ties select A. Reflected image distance is not the label rule.
Q.18 Specular Reflection : Scene Correspondence
Reflection correspondence identifies which real instance produces a reflection.
Object selection.
The source describes sampling from the same reflection pool, while the placement stage uses three identical instances.
Mirror and initial-view setup.
Mirror/camera setup follows the prior reflection pipeline.
Object placement.
One instance lies along the reflected viewing ray about 1–2 m from the mirror; two alternatives lie approximately perpendicular to it. Separation, collision and small yaw variations preserve distinguishable candidates.
Ground-truth label.
The answer is the target real object’s rendered left/middle/right position.
Q.19 Enumerative Perception : Counting w Occlusion
Occluded counting labels all successfully placed targets, including initially hidden ones.
Object selection.
Common objects have no dimension above 25 cm and are not smaller than 5 cm in every dimension.
Agent initial viewpoint.
The initial camera uses the supplied/default floor pose, about 1.25 m height and a 30° downward pitch.
Occluder selection and Object placement.
Furniture occluders hide targets placed just beyond them along the viewing direction; a ray–bounding-box test verifies initial occlusion.
Ground-truth label.
The target count determines the answer among four numeric options.
Q.20 Enumerative Perception : Spatial Segmentation
Spatial segmentation places countable objects within visually dividing furniture.
Object selection.
Targets use the same size-filtered counting pool.
Divider selection and Object placement.
Chair-like dividers surround floor targets at their bounding-box centers, creating fragmented visual appearances.
Agent initial viewpoint.
The camera follows the shared counting setup.
Ground-truth label.
The answer is the number of successfully placed target instances, not the number of visible fragments.
Q.21 Enumerative Perception : Merged Observation
Merged observation makes adjacent instances appear to be one object.
Object selection.
Targets use the shared counting pool.
Object placement.
Objects form a tight, collision-checked cluster with small offsets and occasional stacking.
Agent initial viewpoint.
The initial pose follows the shared counting setup.
Ground-truth label.
The label counts distinct placed instances despite their merged appearance.
Q.22 Enumerative Perception : Category Ambiguity
Category ambiguity adds visually similar distractors to counting scenes.
Object selection.
For each target category, the three most similar CLIP-scored categories supply confusers; one confuser is added for each target.
Object placement.
Target/confuser pairs occupy nearby valid floor positions without collision.
Agent initial viewpoint.
The initial pose follows the shared counting setup.
Ground-truth label.
Only target-category instances contribute to the count.
Q.23 Enumerative Perception : Structural Enclosure
Structural enclosure hides counting targets in boxes.
Object selection.
The counting pool is paired with box models intended to open and close.
Object placement.
A random subset of supported or floor-placed boxes contains targets; the others remain empty, and boxes begin closed when possible.
Agent initial viewpoint.
The camera follows the common counting setup.
Ground-truth label.
The label counts targets in occupied boxes. As in the change task, the paper’s displayed action list does not spell out how box opening is invoked.
Q.24 Enumerative Perception : Illumination Variability
Illumination variability changes visibility while holding object count fixed.
Object selection.
The shared counting pool supplies the targets.
Object placement.
Objects occupy valid floor locations; ambient intensity and directional lighting are varied without moving them.
Agent initial viewpoint.
The initial pose follows the common counting setup.
Ground-truth label.
The label is the unchanged number of placed target instances.
Q.25 Cognitive Mapping : Connectivity
Connectivity is labelled by navigability between room regions.
Agent initial viewpoint.
Each room gets a high-clearance reference point and eight evenly spaced views; the first relevant view supplies the main context.
Question Generation.
Room segmentation and sampled navigable points define pairs; the simulator’s shortest-path query tests whether a route exists.
Ground-truth label.
A valid route gives yes; no route gives no.
Q.26 Cognitive Mapping : Traversable Passage
Traversable passage asks whether a particular shortest route passes through another room.
Agent initial viewpoint.
The reference views follow the connectivity setup.
Question Generation.
The returned shortest path is projected onto room regions and checked for the queried intermediate region.
Ground-truth label.
The answer is yes if that region occurs between source and destination. This is a property of the selected shortest path, not a claim that every possible route must pass through it.
Q.27 Cognitive Mapping : Regional Boundary
Regional boundary questions relate objects to segmented room regions.
Object Selection.
Structural, oversized and robot objects are excluded; referenced objects receive four object-centered views.
Agent initial viewpoint.
The room context follows the connectivity setup.
Question Generation.
Questions test object membership, shared region or which of two regions is nearer.
Ground-truth label.
Membership uses assigned regions. Nearness uses distance from object center to the region bounding box, with region-center distance breaking ties.
Q.28 Cognitive Mapping : Long-Term Navigation
Long-term navigation uses distant connected room pairs and their planned routes.
Agent initial viewpoint.
Reference observations follow the connectivity setup.
Question Generation.
Reachable pairs are ranked by route length; selected long routes generate next-region and turn/action questions.
Ground-truth label.
Labels follow the chosen shortest-path region sequence; heading changes produce left, right or back turns.