Literature
Display
← All papers

ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Yining Hong · Jiageng Liu · Han Yin · Manling Li · Leonidas Guibas · Li Fei-Fei · Jiajun Wu · Yejin Choi

Preprint posted · arXiv (preprint) ↗

Robotics20 min text · CondensedOriginal paper ↗
Contents

Overview

ESI-Bench evaluates whether an agent can obtain the observations needed to answer a spatial question. The original overview illustrates all 29 subcategories, including hidden objects, reflective surfaces, physical tests, counting, navigation and temporal change.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 1: ESI-Bench is a comprehensive benchmark for embodied spatial intelligence, spanning 10 task categories and 29 subcategories organized around Spelke’s four core knowledge systems (Spelke and Kinzler, 2007): object representation, layout and geometry, number representation, and agents and goal-directed actions.

At a glance

The benchmark contains 3,081 simulated instances across ten categories. Agents choose perception, movement and manipulation actions, then answer within 30 steps. Active exploration generally helps over fixed or randomly collected views, but substantial gaps remain to humans and informative oracle trajectories. Accurate 3D scene graphs can help; reconstructed graphs can introduce damaging geometric errors. The results diagnose evidence acquisition and belief updating under a high-level simulator interface, rather than validating low-level robot control or real-world reliability.

1 Introduction

The central question is which observation to acquire, not merely how to interpret an already useful image. Questions deliberately hide information behind viewpoint ambiguity, occlusion, containers or physical interaction. The taxonomy draws on Spelke and Kinzler’s core-knowledge framework. By varying access to views and actions, the benchmark studies how poor evidence selection can compound perceptual errors. Claims about “action blindness” describe the evaluated models and task conditions; oracle gains do not prove that perception is irrelevant or universally solvable.

2 Related Works

Existing spatial benchmarks emphasize fixed images, videos, multiview inputs or partial observation; embodied benchmarks add navigation and interaction. ESI-Bench combines broad spatial categories with hidden variables that require active inspection. Geometry-augmented reasoning and explicit reasoning traces address interpretation, while active-perception methods address observation selection. Table 1 retains the authors’ detailed scope comparison, including action access, hidden states and cognitive categories.

Table 1: Comparison with spatial reasoning and embodied benchmarks. Here, embodied means that an agent is situated in an embodied environment and acts through an explicit action space in the environment. Active?: whether the benchmark supports active perception. Action Space: L = locomotion, P = perception, M = manipulation. Core Knowledge: Obj = object, Geom = geometry, Num = number, Phys = physics, Agt = agent. Hidden: whether the benchmark targets latent properties such as occlusion, containment, dynamics that are inaccessible from passive observation.
Benchmark Type Active? Action Hidden Core
VSR (Liu et al., 2023) Spatial ✗ P ✗ Geom
BLINK (Fu et al., 2024) Spatial ✗ P ✗ Obj, Geom, Num
3DSRBench (Ma et al., 2025) Spatial ✗ P ✗ Obj, Geom
VSI-Bench (Yang et al., 2025a) Spatial ✗ P ✗ Obj, Geom, Num
MMSI-Bench (Yang et al., 2025d) Spatial ✗ P ✗ Obj, Geom
MindCube (Wang et al., 2026) Spatial ✗ P ✓ Geom, Phys
PhysBench (Chow et al., 2025) Spatial ✗ P ✓ Obj, Geom, Phys
CausalSpatial (Ma et al., 2026) Spatial ✗ P ✓ Obj, Phys
EmbSpatial-Bench (Du et al., 2024) Spatial ✗ P ✗ Obj, Geom
OpenEQA (Majumdar et al., 2024) Embodied ✓ L, P ✗ Obj, Geom
EmbodiedBench (Yang et al., 2025b) Embodied ✓ L, P, M ✗ Obj, Geom, Agt
EmbodiedEval (Cheng et al., 2025) Embodied ✓ L, P, M ✗ Obj, Geom, Agt
EXPRESS-Bench (Jiang et al., 2025) Embodied ✓ L, P ✗ Obj, Geom
ESPIRE (Zhao et al., 2026) Spatial+Embodied ✗ P, M ✗ Obj, Geom
CHAIN (Wu et al., 2026b) Embodied ✓ P, M ✓ Obj, Geom, Phys
ESI-Bench (ours) Spatial+Embodied ✓ L, P, M ✓ Obj, Geom, Num, Phys, Agt

3 ESI-Bench

The benchmark couples simulator-derived labels with verified interaction trajectories. Its construction process seeks physically valid, answerable questions whose initial view is insufficient.

3.1 Benchmark Setup

The setup fixes an initial scene and pose, a question, a labelled answer and a shared action budget. Agents control evidence acquisition through discrete, high-level commands.

Task Definition

Each instance specifies a scene, initial pose, question and ground truth. The agent accumulates observations through actions and terminates with an answer plus confidence in at most 30 steps. Questions constrain the expected answer type, such as a count, category, yes/no relation or ordering. Correctness is defined by matching the labelled answer; detailed free-text normalization is not specified here.

Simulation Environment.

OmniGibson/BEHAVIOR-1K provides 51 scenes, more than 300 rooms and about 9,000 object instances across 1,829 categories. Physics, transparency, reflections, fluids and articulated states support the hidden-variable tasks. Simulator scene graphs supply poses, bounding boxes, contacts, containment and room assignments for construction and labels. These controlled observations differ from noisy real sensor data.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
(a) A dataset example including question, answer and trajectory, from the Rigid Containment sub-category.
Action Description
Locomotion
move_forward / move_backward Translate agent along viewing axis
move_left / move_right Translate agent laterally
move_up / move_down Translate agent vertically
Perception
turn_left / turn_right Rotate agent horizontally
turn_up / turn_down Rotate agent vertically
Manipulation
pick_up obj Pick up object
put obj inside obj Put object inside object
put obj on obj Put object on object
fill obj with water Fill object with water
pour obj from obj to obj Pour object from object to object
Terminal
answer(y^\hat{y}, cc) Commit to final answer with confidence cc
(b) ESI-Bench agent action space.
Refer to caption
(c) Distribution of task categories.
Figure 2: Overview of ESI-Bench: dataset example, agent action space, and task distribution.

3.2 Task Construction

GPT-4o proposes candidates; simulator checks and independent human review determine which instances survive. Proposal generation is not treated as a source of ground-truth correctness.

Task Proposal.

Given scene metadata, task requirements and a random 200-category candidate set, GPT-4o proposes objects, placements, initial viewpoints and an informative action trajectory. Labels are derived from the instantiated scene state.

Scene Instantiation.

Candidate placements undergo collision checks, physics settling, refreshed bounding-box checks, visibility verification and relevant contact tests. Invalid configurations are rejected.

Agent Trajectory Collection.

The proposed trajectory is executed step by step with rendered observations and repeated state checks. A failure at any step rejects that candidate trajectory.

Metadata Saving.

Saved metadata includes scene/room/floor identity, object models and poses, agent pose, visibility flags, question, answer and action sequence, enabling deterministic reconstruction of the task description.

Human Verification and Generator Bias Audit.

Three annotators review correctness, answerability and non-triviality, with majority decisions resolving disagreement. Separate audits test wording/category shortcuts, diversity and matched human-authored tasks. These checks reduce identifiable artifacts; they do not establish absence of every generator bias.

Category Definition
Physical Structure Reveal hidden containment capacity through manipulation
Physical Dynamics Predict stability and motion under physical forces
Specular Reflection Infer object relations from mirror reflections
Perceptual Grounding Resolve viewpoint-dependent perceptual ambiguities
Metric Comparison Overcome forced-perspective distortions in size and distance
Spatial Relations Navigate to vantage points that break projective symmetry
Cognitive Mapping Construct topological representations through locomotion
Enumerative Perception Count objects despite occlusion, merging, or enclosure
Temporal Understanding Infer state changes through interaction and observation
Action Sequencing Reason over action sequences and causal dependencies
(a) ESI-Bench task categories and definitions.
Original figure
(b) Required action types per category.
Figure 3: ESI-Bench task categories (L). Combination and level of embodied action types (R).

3.3 Task Categories and Statistics

Ten categories cover physical structure/dynamics, reflections, perceptual grounding, metric comparison, spatial relations, cognitive mapping, counting, temporal understanding and action sequencing. The original diagrams preserve their definitions, action requirements and distribution across 29 subcategories.

4 Experiments

The experiments vary both observation access and the agent’s control over acquiring evidence. Per-category results are more informative than a single aggregate because failure modes differ substantially.

Table 2: ESI-Bench results (%) across passive, active, and oracle paradigms for 2D+VLMs, 3D+LLMs, and humans, showing active exploration benefits and the gap to oracle action selection.
2D + VLM 3D + LLM Human Performance
GPT-5 Gemini 3.1 VGGT + Gemini GT 3D + Gemini
Category / Subcategory

Passive
Single

Passive
Multiple

Active
Multiple

GT
Passive

Passive
Single

Passive
Multiple

Active
Multiple

GT
Passive

Passive
Single

Passive
Multiple

Active
Multiple

GT
Passive

Passive
Single

Passive
Multiple

Active
Multiple

GT
Passive

Passive
Single

Passive
Multiple

Active
Multiple

GT
Passive

Perceptual Grounding
View Hallucination 11.7 20.2 60.1 87.8 39.9 32.9 68.1 91.1 56.3 59.0 76.1 87.6 61.6 60.6 74.1 89.0 52.2 45.8 80.8 83.8
Partial Occlusion 32.6 26.3 47.4 86.3 30.5 32.6 70.5 88.4 51.6 48.4 53.7 86.3 56.8 55.8 72.6 91.6 37.9 34.7 80.0 87.4
Material Transparency 30.3 36.7 66.1 96.3 44.0 45.0 52.3 88.0 37.4 29.4 31.8 90.9 27.8 31.2 60.4 100 41.3 44.0 93.6 97.2
Physical Structure
Rigid Containment 45.0 42.5 42.5 95.0 47.5 40.0 67.5 97.5 27.5 37.5 57.5 72.5 45.0 42.5 65.0 95.0 47.5 42.5 92.5 100
Liquid Volume 66.2 66.2 81.6 86.0 69.9 67.6 80.9 86.8 65.4 56.7 71.3 77.2 74.5 74.5 83.1 86.8 79.4 66.9 89.7 86.8
Deformable 42.9 41.8 55.1 75.5 34.7 41.8 43.9 78.6 43.9 42.9 49.0 81.6 46.9 54.1 99.0 99.0 57.1 60.2 76.5 82.7
Physical Dynamics
Inclined Plane 57.4 60.7 77.0 86.9 65.6 62.3 83.6 88.5 67.2 62.3 80.3 86.9 63.9 63.9 83.6 91.8 60.7 62.3 83.6 83.6
Stacking & Stability 34.8 37.1 62.9 86.5 38.2 36.0 78.7 84.3 34.8 39.3 55.1 62.9 27.2 33.7 68.5 86.5 36.0 39.3 84.3 86.5
Specular Reflection
Reflection Authoring 68.7 70.7 70.3 73.7 60.6 60.6 64.9 67.0 52.5 54.6 53.5 55.6 55.6 58.6 55.6 60.6 94.9 88.9 96.0 96.0
Spatial Relations 50.4 38.9 54.8 58.4 43.4 42.5 44.2 46.9 41.6 39.8 41.6 43.4 38.9 35.4 54.2 41.6 78.8 78.8 84.1 87.6
Correspondence 39.8 51.1 52.3 56.8 37.5 40.9 42.0 42.0 37.5 43.2 48.9 48.9 31.8 39.8 42.3 48.9 85.2 80.7 89.8 92.0
Spatial Relations
Linear Alignment 27.7 31.9 42.6 60.6 47.9 44.6 67.0 77.7 45.7 38.3 53.2 59.6 73.4 73.4 84.0 89.4 51.1 39.4 73.4 79.8
Geometric Configuration 25.3 20.4 26.0 26.0 27.5 22.3 27.5 44.6 9.9 11.3 18.6 32.6 70.8 73.6 88.0 100 32.4 33.5 74.3 86.3
Physical Contact 40.0 41.7 64.2 90.0 60.8 55.8 70.0 74.2 59.2 41.7 59.2 78.2 65.8 67.5 70.8 72.5 35.8 40.8 88.3 90.8
Metric Comparison
Dimensional Size 42.5 44.9 67.7 80.3 44.3 41.3 68.3 82.6 50.4 40.1 59.3 80.8 58.6 60.5 69.9 85.9 48.5 56.9 82.6 91.9
Spatial Distance 53.9 49.1 58.6 73.7 52.6 50.5 59.9 80.3 50.7 47.3 55.0 71.4 59.6 61.2 64.5 96.7 57.9 59.9 69.1 78.9
Cognitive Mapping
Connectivity 68.3 70.0 68.3 78.3 51.7 48.3 60.0 85.0 66.7 66.7 73.3 83.3 65.0 65.0 71.7 86.7 68.3 68.3 81.7 91.7
Traversable Passage 68.3 66.7 71.7 73.3 66.7 61.7 73.3 78.3 68.3 63.3 66.7 78.3 71.7 73.3 71.7 80.0 70.0 71.7 78.3 85.0
Regional Boundary 65.0 63.8 65.0 67.5 63.8 62.5 65.0 65.0 65.0 65.0 62.5 67.5 60.0 68.8 62.5 68.8 61.3 62.5 67.5 70.0
Long-Term Navigation 40.0 38.3 41.7 50.0 33.3 35.0 33.3 41.7 36.7 40.0 43.3 51.6 36.7 41.7 43.3 40.0 35.0 36.7 51.7 61.7
Enumerative Perception
Counting w Occlusion 3.3 3.3 13.3 56.7 3.3 3.3 10.0 63.3 0.0 0.0 13.3 53.3 33.3 80.0 83.3 100.0 6.7 10.0 53.3 76.7
Spatial Segmentation 3.3 6.7 26.7 63.3 3.3 3.3 16.7 70.0 3.3 3.3 23.3 56.7 43.3 80.0 66.7 86.7 13.3 23.3 63.3 70.0
Merged Observation 38.3 35.0 51.7 50.0 43.3 51.7 61.7 75.0 63.3 51.7 55.0 65.0 51.7 63.3 93.3 100.0 60.0 60.0 65.0 73.3
Category Ambiguity 8.3 10.0 8.3 41.7 13.3 13.3 15.0 48.3 21.7 26.7 26.7 46.7 18.3 76.7 40.0 93.3 23.3 25.0 51.7 61.7
Structural Enclosure 5.0 10.0 22.5 67.5 2.5 0.0 10.0 52.5 0.0 0.0 12.5 52.5 40.0 60.0 50.0 100.0 10.0 22.5 42.5 77.5
Illumination Variability 6.0 22.0 22.0 46.0 12.0 16.0 22.0 58.0 10.0 28.0 30.0 40.0 20.0 84.0 42.0 96.0 20.0 34.0 62.0 66.0
Temporal Understanding
Unobserved Change 40.5 41.2 51.4 77.0 37.2 37.8 47.3 74.5 37.8 41.9 45.8 76.6 39.9 42.6 46.5 74.5 39.4 39.4 70.8 81.0
Agent Observation 40.6 30.1 51.0 72.7 37.6 36.1 58.6 65.4 32.7 27.9 34.2 56.5 36.3 33.3 76.7 87.8 38.3 42.1 83.5 90.2
Action Sequencing
Action Order Inference 36.4 37.7 41.6 67.5 44.2 46.8 54.5 72.7 35.1 31.1 50.6 58.4 44.2 46.8 51.9 75.3 40.3 41.6 74.0 81.8

4.1 Models and Evaluation Setup

Four conditions provide the initial view, 30 random passive views, up to 30 actively selected steps, or views from a verified ground-truth trajectory. GPT-5 and Gemini 3.1 use images; VGGT+Gemini uses reconstructed 3D scene graphs, while ground-truth 3D+Gemini uses simulator geometry. Models receive no task-specific fine-tuning, and matched subsets support comparisons. Random multiview is an intentionally unselected baseline; oracle trajectories and perfect geometry offer different information advantages, so the gaps are diagnostic rather than a perfectly isolated causal decomposition.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 4: Qualitative Study, showing success & failure modes and reasons behind model behavior.

4.2 What Really Holds Spatial Intelligence Back: Seeing or Acting?

Active agents discover several ways to resolve the same question, such as moving behind a glass, inspecting from above, lifting it or pouring its contents. Yet oracle trajectories often improve performance much further: GPT-5 rises from 42.5% active to 95.0% oracle-view accuracy on rigid containment, and from 64.2% to 90.0% on contact. Random views can hurt, as spatial-distance accuracy falls from 53.9% single-view to 49.1% random-multiview. Geometry classification and reflections remain difficult even with good views. These results support both action-selection failures and residual interpretation failures; no universal “perceptual ceiling” is established.

Action Blindness Dominates Perceptual Blindness, while Their Coupling Drives Failure Cascades • Without explicit instruction, agents spontaneously acquire spatial strategies, yielding substantial gains over passive multi-view, which often adds noise rather than signal. • For most tasks, perception is not the bottleneck: with the right viewpoint, the agent succeeds; yet some tasks hit a hard perceptual ceiling that no action can overcome. • Suboptimal actions produce bad views, which in turn produce worse actions, cascading into reasoning failures unrecoverable within the step budget.

4.3 When Does 3D Help, and When Does It Hurt?

For Gemini, perfect 3D raises geometric-configuration accuracy from 27.5% to 70.8%, while VGGT reconstruction lowers it to 9.9%. On occluded counting, the corresponding scores are 3.3%, 33.3% and 0%. Thus explicit geometry helps when reliable, but fragmented objects, false instances and incorrect relations can mislead downstream symbolic reasoning. Perfect simulator geometry is an oracle condition, not a demonstrated deployable reconstruction pipeline.

Original figure
Figure 5: Average number of active exploration steps to reach a correct answer for GPT-5 (solid) and Gemini 3.1 (outlined), grouped by Spelke’s four core knowledge systems.
3D Helps When Geometry Is Perfect, But Imperfect Reconstruction Actively Misleads • Explicit 3D grounding improves performance on tasks where depth and occlusion render 2D projections fundamentally ambiguous. • Imperfect 3D reconstruction degrades performance, proving more harmful than 2D baselines.

4.4 How Far Are Models From Human-Level Spatial Reasoning?

Humans are often closer to models when both receive fixed views, but gain much more from controlling exploration. For example, human active accuracy reaches 88.3% on contact and 93.6% on transparent containment, compared with 64.2% for GPT-5 and 52.3% for Gemini respectively. Qualitative traces show premature confident answers and repeated confirmatory motions. Appendix K supplies behavioral proxies for viewpoint diversity, contrastive evidence and answer revision; these support the observed gap without establishing a general impossibility of improving it through better models or prompting.

Lack of Epistemic Calibration: Models Can See But Do Not Know When They Have Seen Enough • Models commit prematurely with high confidence; humans treat uncertainty as a signal to keep looking, not a reason to answer. • Humans seek viewpoints that falsify their hypothesis; models seek confirmation and move in the same direction repeatedly. • Humans revise beliefs when contradicted; models anchor to their first impression, revealing a fundamental absence of belief updating.

5 Conclusion

ESI-Bench makes information acquisition part of spatial evaluation. Its main contribution is a controlled diagnostic setting that exposes failures in selecting actions, integrating evidence and deciding when to answer. Success remains constrained by the simulator, task construction and high-level action interface.

Acknowledgement

The work acknowledges support from ONR MURI, the Korean IITP/MSIT program, TRI’s university program and Google’s Gemini Academic Program compute resources.

Appendix A Contribution Statement

Yining Hong led the idea, integration, writing and most task pipelines; Jiageng Liu developed deformable-object, reflection, mapping, counting and change-detection pipelines and related assets; Han Yin contributed related work and figures. The remaining authors are credited with advising.

Appendix B Broader Impacts & Limitations

The intended benefit is diagnostic: identifying premature commitment and poor evidence acquisition before deploying embodied systems. Scenes are synthetic and avoid collecting real personal data. The explicit limitation is rendering realism, including textures, lighting, blur and sensor effects. The high-level action abstraction and simulation setting further limit conclusions about real robot execution and safety.

Appendix C Per-Category Task Description

The full taxonomy table retains every subcategory’s definition and example. Material transparency belongs to perceptual grounding in this taxonomy, although Appendix Q.1 labels it under physical structure; the source naming inconsistency is preserved.

Table 3: ESI-Bench task taxonomy and definitions of 10 categories spanning 29 subcategories. Colors correspond to Spelke’s four core knowledge systems: object representation, layout and geometry, number representation, and agents and goal-directed actions.
Category Subcategory Description Example
Physical Rigid Containment Plan the placement of multiple objects across multiple containers How can all the toys be fit into the boxes?
Structure Liquid Volume Compare liquid-holding capacity of containers Which container has larger volume?
Deformable Infer structure from deformable object What’s under the sweater, apple or mug?
Physical Inclined Plane Predict object motion and stability on slopes Can the apple sit stable on the slope?
Dynamics Stacking & Stability Whether objects stack or balance given shape, mass and geometry How to stack and stabilize these three objects?
Specular Reflection Authoring Distinguish real objects from mirror reflections Is the object real or reflected?
Reflection Spatial Relations Infer relations among objects across mirror and real-world views What relations does the mirror reveal?
Correspondence Identify which objects appear in mirror given the real-world scene Which of the 3 snacks appears in the mirror?
Perceptual Partial Occlusion Reason about objects hidden behind other scene elements Golf Stick or Umbrella behind the Wall?
Grounding View Hallucination Detect objects whose visibility changes critically with viewing angle Cabinet or piano from this view?
Material Transparency Reason about objects seen through transparent surfaces Is the object inside the glass or not?
Metric Dimensional Size Compare relative sizes of objects Which vase is larger?
Comparison Spatial Distance Compare relative distances with respect to a reference object Which flower is closer to the table?
Spatial Linear Alignment Whether objects are arranged along a common axis Do the glasses form a line?
Relations Geometric Configuration Identify the shape formed by a set of objects Do the cups form an equilateral triangle?
Physical Contact Detect whether two or more objects are in direct contact Are the teddy bears touching each other?
Cognitive Connectivity Whether two locations or regions are mutually reachable Is region A connected to C?
Mapping Traversable Passage Identify navigable corridors or passageways between regions Passage between the rooms?
Regional Boundary Identify and delineate distinct functional spatial regions Boundaries of this region?
Long-Term Navigation Plan multi-step navigation trajectories toward a distant goal What room is behind the wall?
Enumerative Counting w Occlusion Count objects partially obscured by other scene elements How many balls are under the blanket?
Perception Spatial Segmentation Count objects separated across distinct spatial regions One or two cylinders separated by the post?
Category Ambiguity Count visually similar objects requiring fine-grained distinction How many apples among the balls?
Merged Observation Count groups that appear visually merged from a single viewpoint How many separate stacks of books?
Illumination Variability Count objects under challenging or non-uniform lighting Objects in the dim scene?
Structural Enclosure Count objects hidden within enclosed or covered spaces Objects in the microwave?
Temporal Unobserved Change Infer scene changes that occurred during an unobserved interval What changed when I looked away?
Understanding Agent Observation Reason about scene dynamics from other agents Other robot’s world model?
Action Sequencing Action Order Inference Determine the correct procedural ordering of a sequence of actions What is the action sequence for assembly?

Appendix D Per-Category Task Distribution

The original distribution charts show how instances are divided among each category’s subcategories. These counts contextualize aggregate accuracy and do not imply balanced difficulty.

Original figure
Figure 6: Subcategory distribution within each of the 10 ESI-Bench task categories.

Appendix E Human Study Protocol

The human study uses the same discrete interface and step budget as the active models, with independent task judgments and recorded exploration traces.

Participants and compensation.

Twenty-eight previously unexposed volunteers with normal or corrected vision were recruited and paid above local minimum wage. The paper reports institutional compliance and no required IRB review.

Interface.

The web interface shows the same egocentric pixels and offers the same locomotion, rotation, manipulation and answer/confidence commands, with a 30-step cap.

Task assignment.

Participants each cover at least six categories; each instance receives at least five independent attempts. The paper says answer disagreements are resolved by a third annotator but gives limited detail on reconciling that wording with the five-person protocol.

Instructions.

A brief tutorial and two worked examples explain exploration. Participants receive no task-specific solution hints and are told to answer when sufficiently confident.

Data collected.

Recorded data include all actions and frames, final answers/confidence, stopping step and any intermediate confidence ratings.

Appendix F More Benchmark Examples

Additional examples retain the visual diversity of geometry, observation, counting, transparency, contact, size, slope, distance, deformation, occlusion and reflection tasks.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 7: Additional benchmark examples from ESI-Bench, organized by core knowledge systems: object representation, layout and geometry, number representation, and agents and goal-directed actions.

Appendix G More Qualitative Examples

The supplementary trajectories show alternative useful strategies as well as persistent reflection mistakes and merged-object undercounting. They illustrate mechanisms and failures rather than estimating their prevalence.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 8: Additional qualitative examples illustrating emergent agent behaviors and failure modes: (1) active agents employ emergent spatial strategies such as top-down viewing and object manipulation to compare sizes; (2) active agents explore alternative viewpoints or interact with the environment to identify hidden objects; (3) tricky mirror angle designs cause persistent misjudgment; (4) densely packed objects lead to merged perception across viewpoints, resulting in undercounting.

Appendix H Human Verification and Generator Bias Analysis

The generator audit checks candidate validity, potential shortcuts and task-source difficulty. Simulator and human filtering are central: these statistics concern reviewed proposals rather than unfiltered GPT-4o outputs.

H.1 Human Verification Protocol

Reviewers see the question/answer, initial image, verified trajectory, final simulator state and relevant metadata. Each instance must pass three distinct criteria.

Correctness.

Correctness requires physically plausible placements, executable transitions and an answer consistent with simulator state; floating, disappearing or intersecting objects are rejected.

Answerability.

Answerability requires a unique answer that can be revealed through the available actions within budget. Invisible evidence, subjective interpretation and multiple plausible labels trigger rejection.

Non-triviality.

Non-triviality rejects obvious initial-view solutions and category shortcuts. Three independent binary reviews use majority resolution, and an instance must pass every axis.

H.2 Verification Scores for GPT-4o-Generated Tasks

After automatic filtering, human verification averages 92.0% correctness, 91.0% answerability, 88.4% non-triviality and 85.1% passing all three. Enumerative perception has the lowest overall pass rate at 78.6%, while cognitive mapping reaches 91.2%. Final evaluation excludes rejected candidates.

Category Correctness Answerability Non-triviality Overall Pass
Perceptual Grounding 94.7 93.8 91.6 88.9
Physical Structure 91.3 90.5 88.2 84.6
Physical Dynamics 89.8 88.7 86.9 82.1
Specular Reflection 93.4 92.1 89.5 86.7
Spatial Relations 92.6 91.4 87.8 85.2
Metric Comparison 95.1 94.3 90.7 89.4
Cognitive Mapping 96.4 95.8 92.6 91.2
Enumerative Perception 87.9 86.5 83.9 78.6
Temporal Understanding 90.7 89.9 87.1 83.6
Action Sequencing 88.5 87.3 85.4 80.9
Average 92.0 91.0 88.4 85.1
Table 4: Human verification scores for GPT-4o-generated task proposals after simulator filtering. Correctness measures physical and semantic consistency with the simulator state; answerability measures whether the question has a unique answer obtainable through the available action space; non-triviality measures whether the task requires embodied evidence acquisition rather than being solvable from the initial observation or priors. Overall pass denotes the percentage of instances that pass all three checks.

H.3 Bias Audit for GPT-4o-Generated Tasks

Shortcut tests remove images and interaction histories, then expose either the question alone or the question plus an unordered object-category list. Diversity statistics complement these tests but do not exhaust possible biases.

Linguistic bias.

Question-only accuracy averages 36.6% in the audit versus 42.5% passive single-view. The full table is retained because chance rates vary by answer type and the paper does not provide a uniform chance baseline.

Object-category bias.

Metadata-only accuracy averages 39.3%, while active exploration averages 56.9%. This ordering is not universal: in counting, active accuracy is 22.6%, below question-only 24.3% and metadata-only 26.1%; some other category priors also match or exceed passive images. The table therefore supports an aggregate advantage, not the prose’s claim that active exploration beats every shortcut in every category. Average normalized answer entropy is 0.92 and object-diversity score 0.85.

Category Question-Only Metadata-Only Passive Single Active
Perceptual Grounding 26.1 28.3 32.8 66.3
Physical Structure 31.5 35.0 50.7 64.1
Physical Dynamics 49.2 51.0 51.9 81.2
Specular Reflection 47.2 49.5 47.2 50.4
Spatial Relations 30.8 35.1 45.4 54.8
Metric Comparison 49.2 50.8 48.5 64.1
Cognitive Mapping 47.6 50.1 53.9 57.9
Enumerative Perception 24.3 26.1 13.0 22.6
Temporal Understanding 35.7 38.1 37.4 53.0
Action Sequencing 24.8 29.3 44.2 54.5
Average 36.6 39.3 42.5 56.9
Table 5: Shortcut baseline analysis for GPT-4o-generated tasks. Question-only uses only the natural-language question. Metadata-only uses the question and unordered task-relevant object categories, but no observations, positions, action history, or simulator state. Passive Single and Active report Gemini 3.1 category-level averages from the main evaluation. Shortcut baselines remain below active exploration, suggesting that benchmark performance is not explained by linguistic or object-category artifacts.
Category Answer Balance Object Diversity
Perceptual Grounding 0.92 0.87
Physical Structure 0.89 0.84
Physical Dynamics 0.95 0.82
Specular Reflection 0.93 0.85
Spatial Relations 0.90 0.88
Metric Comparison 0.96 0.86
Cognitive Mapping 0.94 0.80
Enumerative Perception 0.88 0.89
Temporal Understanding 0.91 0.83
Action Sequencing 0.87 0.81
Average 0.92 0.85
Table 6: Bias and diversity statistics for GPT-4o-generated tasks. Answer balance is normalized answer entropy; object diversity is normalized entropy over task-relevant object categories. Higher values indicate less concentration around a single answer or object set.

H.4 Comparison with Human-Generated Tasks

Matched GPT-generated tasks are consistently somewhat easier than human-authored ones in Table 7: gaps range from 2.7 to 4.4 points, averaging 3.5. The model/human ordering is similar across sources, which argues against a large reversal of difficulty but does not demonstrate statistical equivalence or zero generator bias.

Model GPT-4o-Generated Human-Generated Absolute Gap
GPT-5 Active 51.6 48.9 2.7
Gemini 3.1 Active 56.9 53.4 3.5
VGGT + Gemini Active 51.6 48.1 3.5
GT 3D + Gemini Active 68.0 63.6 4.4
Human Active 75.3 72.1 3.2
Average 60.7 57.2 3.5
Table 7: Accuracy comparison between GPT-4o-generated and human-generated task subsets under active exploration. The gaps remain within a few percentage points, suggesting that GPT-4o-generated tasks have similar difficulty to human-generated tasks.

Appendix I Diverse Emergent Behaviors in Active Exploration

Human annotation of successful trajectories finds three to six strategy clusters in seven representative subcategories; transparent containment has six, while partial occlusion, contact and size have four each. These are post-hoc clusters of successes, not a frequency distribution over all attempts. Diversity alone does not imply accurate confidence or reliable stopping.

Subcategory # Emergent Strategy Clusters
Partial Occlusion 4
Material Transparency 6
Rigid Containment 3
Liquid Volume 3
Physical Contact 4
Spatial Distance 3
Dimensional Size 4
Table 8: Human-annotated diversity of emergent behaviors in successful active trajectories. For each representative subcategory, annotators group trajectories into distinct behavioral clusters. The table reports the number of clusters discovered, without imposing predefined strategy labels.

Appendix J Failure Analysis of VGGT-Based 3D Augmentation

Manual analysis identifies duplicated object fragments, hallucinated instances and corrupted spatial relations in VGGT-derived scene graphs. These errors explain why adding a seemingly authoritative graph can degrade counting and fine-grained geometry. The proposed direction is uncertainty-aware reconstruction and graph construction; no quantitative error-frequency breakdown is supplied.

Failure Mode Main Effect Affected Tasks
Object duplication One partially observed object is reconstructed as multiple fragments or repeated instances, causing over-counting. Counting w Occlusion; Merged Observation
Object hallucination Noisy geometry or background fragments are converted into spurious object proposals. Enumerative Perception; Perceptual Grounding
Spatial-relation corruption Depth errors distort relative position, contact, containment, and near/far relations. Spatial Relations; Metric Comparison; Physical Structure
Table 9: Common failure modes of VGGT-based 3D augmentation. The dominant issue is object duplication from noisy depth and partial reconstruction, followed by object hallucination and corrupted spatial relations.

Appendix K Operationalizing the Metacognitive Gap

Humans show 71.8% view diversity, 62.7% contrastive-view rate and 41.3% answer revision; Gemini scores 43.5/31.5/18.9 and GPT-5 39.2/28.7/16.4. These measures operationalize exploration behavior. Answer changes are not necessarily correct updates, and viewpoint diversity is not itself calibration; the paper treats these as complementary proxies, not a complete metacognition benchmark.

Axis Definition Signal
Evidence sufficiency Whether the agent gathers sufficiently diverse observations before answering. View diversity
Falsification seeking Whether actions seek evidence that could disconfirm the current hypothesis. Contrastive views
Belief revision Whether the agent changes its answer after contradictory evidence. Answer updates
Table 10: Operational measures for the metacognitive gap observed in ESI-Bench.
Agent View Diversity Contrastive View Rate Belief Revision Rate
Human Active 71.8 62.7 41.3
Gemini 3.1 Active 43.5 31.5 18.9
GPT-5 Active 39.2 28.7 16.4
Table 11: Trajectory-level measures of the metacognitive gap. Humans acquire more diverse and contrastive evidence, and revise beliefs more often than current MLLMs. All values are percentages.

Appendix L Analysis of Passive Multi-View Evaluation

Thirty randomly selected, unordered views can miss useful evidence, overload the model, lack action-linked alignment or introduce apparent contradictions. The paper explicitly cautions that this is not an optimal multiview baseline. Better selection, ranking or cross-view alignment could improve it, so the result concerns unstructured view accumulation rather than multiview reasoning in general.

Source Possible Effect
Random view selection Views may cover the scene broadly but miss the diagnostic evidence needed for the question.
Image overload Many irrelevant images can dilute the few useful observations and increase reasoning burden.
No action grounding Views are not connected to self-selected actions, making spatial integration harder.
Conflicting evidence Occluded or partial views can introduce misleading cues across images.
Table 12: Possible reasons why passive multi-view input may fail to improve over passive single-view input.

Appendix M Why a Synthetic Simulator Benchmark?

Simulation gives exact hidden-state labels and repeatable control over occlusion, contact, containment and connectivity. It deliberately reduces sensor and actuation confounds to diagnose evidence selection. The authors make no direct sim-to-real transfer claim; real deployment adds calibration, control, noise and environment variation.

Appendix N Active Agent Prompting and Action Selection

At every step the model receives the question, current image, prior action-observation history and action vocabulary, then selects one action. Execution returns a new frame until the model answers or exhausts 30 steps.

Action-selection prompt.

The complete original prompt is retained below, including placeholders, action signatures and the required Reason/Action response format. It asks the agent to gather evidence and delay answering until enough is available; it does not prescribe task-specific strategies.

Active Exploration Prompt You are an embodied agent in a 3D environment. Your goal is to answer the question by actively exploring the scene. You may take one action at each step. Use actions only when they help you gather evidence. Do not answer until you have enough evidence. Question: {question} Current observation: You are given the current egocentric image. Action history: {previous actions and brief observation summaries} Available actions: • move_forward , move_backward , move_left , move_right , move_up , move_down • turn_left , turn_right , turn_up , turn_down • pick_up(obj) • put(obj, inside, obj) • put(obj, on, obj) • fill(obj, with_water) • pour(obj, from, obj, to, obj) • answer(answer, confidence) Choose exactly one next action from the available action space. Return your response in the following format: Reason: briefly explain what evidence you need next. Action: one valid action.

Action parsing and execution.

Invalid or multiple actions receive one repair prompt. A second invalid output consumes a step without moving the agent. Object references are matched to observed identifiers where possible; physically failed manipulations remain part of the trajectory. Thus high-level control reduces motor confounds but does not remove every execution failure.

Appendix O Why High-Level Actions?

High-level actions isolate decisions about where to look, what to manipulate and when to stop. The benchmark intentionally omits continuous motor commands and robot-specific controller learning. Low-level competence is a complementary evaluation dimension, not measured by these results.

Appendix P Step Budget Ablation

Gemini’s budget ablation improves rapidly through roughly 15 steps, levels off around 20–30, and slightly declines beyond 40. The 30-step cap is justified for this tested setup; the curve does not prove that longer, better-designed exploration could never help.

Original figure
Figure 9: Step budget ablation for Gemini 3.1 Active. Performance rises quickly up to 15–20 steps, saturates around 30 steps, and slightly decreases after 40 steps.

Appendix Q Per-Category Task Construction Details

The following construction notes preserve each pipeline’s object choice, physical setup, initial ambiguity and label rule. These engineering definitions matter because labels often use explicit geometric thresholds rather than unrestricted real-world concepts. Several descriptions omit parameters or have inconsistent wording; those gaps are identified rather than filled in.

Initial Agent Viewpoint Setup.

Initial views deliberately hide diagnostic structure through collinearity, grazing angles, occlusion or forced perspective. Instances with no task-relevant visibility in any verified view are rejected. The construction tests targeted ambiguity rather than a random sample of everyday camera poses.

Q.1 Physical Structure: Material Transparency

Transparent containment distinguishes an object inside a clear vessel from one aligned behind it.

Object selection.

Containers and small objects come from separate human-curated category pools, with models drawn from BEHAVIOR-1K.

Scale.

The object’s largest dimension is capped at half the container’s narrowest dimension, with an additional sampled size limit. The source leaves that sampling interval symbolic.

Placement.

Positive placements use Inside-state sampling or a settled interior fallback plus bounding-box containment. Negative placements try lateral offsets and correct the final gap to 1 cm; unsuccessful placements are rejected.

Agent intitial spotion and view.

The initial camera aligns the object with the vessel, maximizing ambiguity between inside and behind.

Ground-truth label.

The label is binary inside versus outside.

Q.2 Physical Structure: Liquid Volume

Liquid capacity is defined operationally by the simulator’s water-particle measurement.

Object selection.

Container pairs differ by at least 200 particles of premeasured capacity. Offline dipping counts the largest connected water-particle component using a spatial-neighbor graph.

Pouring Simulation.

A filled container’s particles are transferred to the other vessel, yielding complete transfer or overflow. This is the reported simulation procedure.

Ground-truth label.

The answer identifies the vessel with greater offline particle capacity.

Q.3 Physical Structure: Deformable

The deformable task asks which object is hidden under a cloth.

Object selection.

A GPT-proposed pool is filtered to portable objects of roughly 5–15 cm average radius. Each question uses one target, three category distractors and a cloth no larger than 50 cm per side.

Object placement.

The target settles at a free floor location; a downward-moving cloth is dropped over it.

Agent initial view.

The camera uses a fixed azimuth, a distance/height based on cloth footprint and a 70° field of view.

Ground-truth label.

The answer is the hidden category among four shuffled choices.

Q.4 Physical Dynamics: Inclined Plane

Slope stability depends on randomized physical parameters and finite simulation thresholds.

Object selection.

A compatible nonstructural object is selected and its mass perturbed.

Slope construction.

Slope angle is 10–45°, base half-length 20–30 cm, static friction 0.3–3.0 and dynamic friction no greater than static. Restitution is 0.1; physics and action timesteps are 1/240 s and 1/60 s.

Simulation and label derivation.

After 30 simulation steps, more than 3 cm horizontal motion indicates sliding; a drop more than 5 cm below the slope bottom indicates falling.

Agent initial viewpoint.

The initial view shows the object near the slope before the test.

Ground-truth label.

The final task label is stable versus not stable, although the construction also distinguishes sliding from falling.

Q.5 Physical Dynamics: Stacking and Stability

Stacking tests which object orders satisfy the simulator’s stability checks.

Object selection.

GPT-4o selects two or three solid, plausibly stackable objects from the sampled category set.

Scale normalization.

All objects receive a common horizontal footprint based on the largest original extent, with proportional vertical scaling.

Placement.

Each permutation settles for 60 steps. A layer must be elevated by at least 2 cm, overlap horizontally with AABB IoU at least 0.1, and retain an up-axis dot product above 0.9. These are operational stability criteria.

Ground-truth label.

Valid stacking orders are derived from the settled placement checks; the full text describes both support overlap and tilt checks.

Q.6 Metric Comparison: Spatial Distance

Distance comparison creates a forced-perspective ambiguity around a reference object.

Object selection.

Reference objects need at least two bounding-box corners with more than 0.5 m clearance; GPT selects two similar-category comparison objects.

Placement.

Objects are placed diagonally outward from different reference corners at near/far distances. The source omits both numeric distances, so they cannot be reconstructed from this description alone.

Agent initial position and view.

The initial pose makes apparent size and occlusion unreliable indicators of distance.

Ground-truth label.

The answer selects the designated nearer or farther object, according to the question.

Q.7 Metric Comparison: Dimensional Size

Size comparison separates physical scale from apparent image size.

Object selection.

Two instances of one household model use scales 1.0 and a sampled 1.2–1.3.

Reference objects.

Distinct reference objects sit within 5 cm of each target’s boundary and at least 30 cm from the other target, providing names for the alternatives.

Agent initial position and view.

The initial camera creates unequal distances and partial occlusion.

Ground-truth label.

The answer uses the associated reference object to identify the larger or smaller target.

Q.8 Spatial Relations: Physical Contact

Contact questions distinguish physical proximity from misleading projection.

Object selection.

Two small-to-medium objects are selected, excluding tiny items and large furniture.

Placement.

Positive cases align faces with small positional variation and use segmentation-mask adjacency; negatives are moved until gaps exceed 5 cm. Live bounding boxes verify the final gaps, alongside the broader pipeline’s contact checks.

Initial agent position and view.

The starting view makes touching and separation visually confusable.

Ground-truth label.

The answer is touching or not touching under the construction’s geometric/contact criteria.

Q.9 Spatial Relations: Linear Alignment and Geometric Configuration

One construction family covers both collinearity and triangle shape.

Object selection.

Three compact objects with nonzero height are selected.

Linear Alignment placement.

Positive line cases use collinear centers; negative cases introduce lateral offsets.

Geometric Configuration placement.

Equilateral examples use side lengths of 0.4–0.65 m and reject object sets with more than a 2× footprint ratio. The paragraph does not fully describe generation of the other triangle classes.

Agent initial position and view.

A grazing view near the centroid makes distinct 3D arrangements project similarly.

Ground-truth labels.

Line alignment is binary; triangle labels distinguish equilateral, isosceles and random configurations.

Q.10 Physical Structure: Rigid Containment

Rigid containment asks for a complete object-to-container assignment.

Object selection.

The pipeline selects containers of several sizes and three or four objects, with scaling intended to give each object a specific valid destination.

Pick-place simulation.

Placement uses physics settling, a complete AABB containment check and a fallback attempt; grasp/release and eye-camera frames represent each manipulation stage. The source omits the fallback primitive’s name.

Ground-truth label.

The answer specifies each object’s correct target container.

Q.11 Action Sequencing: Action Order Inference

Action sequencing combines fit constraints with a placement hierarchy.

Object selection.

GPT proposes a two- or three-object dependency chain, with fitting candidates, near-miss candidates and confusers.

Layout.

The fixed target is central, with alternatives arranged to either side on a support surface.

Ground-truth label.

The answer specifies the valid candidate and relation at each step of the required sequence.

Q.12 Temporal Understanding: Agent Observation

Agent observation counts what a navigating robot passes while another agent watches.

Object selection.

One to three target categories are chosen for placement along a path.

Agent setup.

A navigating robot follows a validated path of at least ten waypoints. A fixed observer watches from 5 m height with the ceiling removed; invalid paths are resampled.

Object placement.

Targets are placed on suitable surfaces within 1 m of the planned route.

Observer agent position and view.

The observer remains at the starting overhead location and tracks the moving robot.

Ground-truth label.

Labels count the categories/instances passed within the proximity threshold; this is the simulator’s operational encounter criterion.

Q.13 Temporal Understanding: Unobserved Changes

Unobserved changes compare box contents across two generated phases.

Object selection.

Questions ask whether a change occurred, which change occurred, or which colored box changed. Contents may be replaced, removed, added or left unchanged.

Object placement.

Collision-aware box placements remain fixed while phase-one contents are removed and phase-two contents are spawned.

Agent initial view.

A clear boundary viewpoint is chosen to show as many boxes as possible.

Ground-truth label.

Labels come from sampled states: yes/no, change type or changed box. Oracle close-ups open boxes in each phase. The displayed action vocabulary does not explicitly list an open command, leaving that interaction detail underspecified.

Q.14 Perceptual Grounding: Partial Occlusion

Partial occlusion uses a target and a plausible confusable occluder.

Object selection.

A curated target pool and compatibility map determine the pair; the occluder is scaled to 1.1× the target’s dimensions.

Placement.

Both objects sit on a table, with the occluder near a target bounding-box corner.

Agent initial viewpoint.

The camera is 15° off their aligned axis at target-center height, showing only ambiguous partial evidence.

Ground-truth label.

The label is the target’s actual category.

Q.15 Perceptual Grounding: View Hallucination

View hallucination tests recognition from a misleading object angle.

Object selection.

The target and confusable alternatives come from a curated category/view-condition map.

Placement.

The target is placed on a valid table or floor support.

Agent initial viewpoint.

The initial azimuth is selected for high category ambiguity.

Ground-truth label.

The correct category is contrasted with the stored visual confusers.

Q.16 Specular Reflection : Reflection Authorization

Reflection authoring distinguishes reflected visibility from a directly seen real object.

Object selection.

Nonstructural, nonreflective floor-placeable objects are filtered to an equivalent radius of 2–20 cm.

Mirror and initial-view setup.

A free-standing mirror approximately faces the room center with up to 30° yaw variation. The initial camera is in front, roughly 2 m away, with an unobstructed mirror view.

Object placement.

Positive examples are mirror-visible but not directly visible; negatives are directly visible without a valid mirror ray. Geometry and visibility checks enforce these conditions.

Ground-truth label.

The answer is yes when the queried object is seen through the mirror, otherwise no.

Q.17 Specular Reflection : Spatial Relationship

Reflection distance asks about real object positions seen through a mirror.

Object selection.

Two categories are sampled from the reflection-compatible pool.

Mirror and initial-view setup.

Mirror and camera setup follows the reflection-authoring pipeline.

Object placement.

Both objects must have valid reflected rays within mirror width and field of view, with floor and occlusion checks.

Ground-truth label.

The closer real 3D position determines A or B; ties select A. Reflected image distance is not the label rule.

Q.18 Specular Reflection : Scene Correspondence

Reflection correspondence identifies which real instance produces a reflection.

Object selection.

The source describes sampling from the same reflection pool, while the placement stage uses three identical instances.

Mirror and initial-view setup.

Mirror/camera setup follows the prior reflection pipeline.

Object placement.

One instance lies along the reflected viewing ray about 1–2 m from the mirror; two alternatives lie approximately perpendicular to it. Separation, collision and small yaw variations preserve distinguishable candidates.

Ground-truth label.

The answer is the target real object’s rendered left/middle/right position.

Q.19 Enumerative Perception : Counting w Occlusion

Occluded counting labels all successfully placed targets, including initially hidden ones.

Object selection.

Common objects have no dimension above 25 cm and are not smaller than 5 cm in every dimension.

Agent initial viewpoint.

The initial camera uses the supplied/default floor pose, about 1.25 m height and a 30° downward pitch.

Occluder selection and Object placement.

Furniture occluders hide targets placed just beyond them along the viewing direction; a ray–bounding-box test verifies initial occlusion.

Ground-truth label.

The target count determines the answer among four numeric options.

Q.20 Enumerative Perception : Spatial Segmentation

Spatial segmentation places countable objects within visually dividing furniture.

Object selection.

Targets use the same size-filtered counting pool.

Divider selection and Object placement.

Chair-like dividers surround floor targets at their bounding-box centers, creating fragmented visual appearances.

Agent initial viewpoint.

The camera follows the shared counting setup.

Ground-truth label.

The answer is the number of successfully placed target instances, not the number of visible fragments.

Q.21 Enumerative Perception : Merged Observation

Merged observation makes adjacent instances appear to be one object.

Object selection.

Targets use the shared counting pool.

Object placement.

Objects form a tight, collision-checked cluster with small offsets and occasional stacking.

Agent initial viewpoint.

The initial pose follows the shared counting setup.

Ground-truth label.

The label counts distinct placed instances despite their merged appearance.

Q.22 Enumerative Perception : Category Ambiguity

Category ambiguity adds visually similar distractors to counting scenes.

Object selection.

For each target category, the three most similar CLIP-scored categories supply confusers; one confuser is added for each target.

Object placement.

Target/confuser pairs occupy nearby valid floor positions without collision.

Agent initial viewpoint.

The initial pose follows the shared counting setup.

Ground-truth label.

Only target-category instances contribute to the count.

Q.23 Enumerative Perception : Structural Enclosure

Structural enclosure hides counting targets in boxes.

Object selection.

The counting pool is paired with box models intended to open and close.

Object placement.

A random subset of supported or floor-placed boxes contains targets; the others remain empty, and boxes begin closed when possible.

Agent initial viewpoint.

The camera follows the common counting setup.

Ground-truth label.

The label counts targets in occupied boxes. As in the change task, the paper’s displayed action list does not spell out how box opening is invoked.

Q.24 Enumerative Perception : Illumination Variability

Illumination variability changes visibility while holding object count fixed.

Object selection.

The shared counting pool supplies the targets.

Object placement.

Objects occupy valid floor locations; ambient intensity and directional lighting are varied without moving them.

Agent initial viewpoint.

The initial pose follows the common counting setup.

Ground-truth label.

The label is the unchanged number of placed target instances.

Q.25 Cognitive Mapping : Connectivity

Connectivity is labelled by navigability between room regions.

Agent initial viewpoint.

Each room gets a high-clearance reference point and eight evenly spaced views; the first relevant view supplies the main context.

Question Generation.

Room segmentation and sampled navigable points define pairs; the simulator’s shortest-path query tests whether a route exists.

Ground-truth label.

A valid route gives yes; no route gives no.

Q.26 Cognitive Mapping : Traversable Passage

Traversable passage asks whether a particular shortest route passes through another room.

Agent initial viewpoint.

The reference views follow the connectivity setup.

Question Generation.

The returned shortest path is projected onto room regions and checked for the queried intermediate region.

Ground-truth label.

The answer is yes if that region occurs between source and destination. This is a property of the selected shortest path, not a claim that every possible route must pass through it.

Q.27 Cognitive Mapping : Regional Boundary

Regional boundary questions relate objects to segmented room regions.

Object Selection.

Structural, oversized and robot objects are excluded; referenced objects receive four object-centered views.

Agent initial viewpoint.

The room context follows the connectivity setup.

Question Generation.

Questions test object membership, shared region or which of two regions is nearer.

Ground-truth label.

Membership uses assigned regions. Nearness uses distance from object center to the region bounding box, with region-center distance breaking ties.

Q.28 Cognitive Mapping : Long-Term Navigation

Long-term navigation uses distant connected room pairs and their planned routes.

Agent initial viewpoint.

Reference observations follow the connectivity setup.

Question Generation.

Reachable pairs are ranked by route length; selected long routes generate next-region and turn/action questions.

Ground-truth label.

Labels follow the chosen shortest-path region sequence; heading changes produce left, right or back turns.

Concept demonstration

Open image ↗