Overview
ENACT measures whether a model can align egocentric observations with symbolic changes during household interactions. Its central result is a large human–model gap as sequences lengthen. This is an offline ordering benchmark and a proxy for embodied reasoning; successful ordering does not establish a controllable world model or successful robot behavior.
At a glance
The benchmark supplies 8,972 simulated questions from 29 household activities. Forward questions ask for the order of future images given actions; inverse questions ask for the action order given images. Thirty vision-language models generally struggle with longer sequences, whereas trained human annotators remain strong. Camera changes and left/right-hand confusion expose additional weaknesses. The contribution is an evaluation and data-generation pipeline, with no model training experiment.
1 Introduction
Motivated by embodied cognition (Smith and Gasser, 2005) and action-conditioned world models (Ha and Schmidhuber, 2018), ENACT replaces pixel generation with sequence ordering. This removes image-generation fidelity from the score while retaining observation–action consistency. Simulator state supplies an explicit symbolic action vocabulary. Affordances, memory, and embodiment are proposed interpretations of the resulting task demands, rather than independently isolated cognitive measurements.
2 ENACT : Egocentric Interactive Embodied Cognition Test
Each question combines egocentric images with state changes extracted from a simulator. The pipeline first identifies meaningful changes, then samples visible subsequences, shuffles one modality, and evaluates an ordering with a semantic verifier.
2.1 Problem Formulation
States are scene graphs, observations are RGB images, and actions are differences between graph states—not motor commands. A sampled trajectory contains L frames and L−1 transitions. Its action descriptions retain changes visible in both endpoint images. Forward modeling fixes the initial image and ordered actions while shuffling later images; inverse modeling fixes the images while shuffling actions. The POMDP framing describes partial observability, but the evaluation itself uses a supplied finite storyboard rather than online interaction.
2.2 Key-Frame Trajectories Synthesis for Scalable Data Generation
Predicate changes select candidate timestamps. Similarity filtering suppresses repetitive changes, and valid subsequences require nonempty visible transitions; object transformations allow transient occlusion. Skipping intermediate frames produces up to a binomial number of candidate questions from one episode. This increases question count without increasing the number of independent environments or demonstrations. A dynamic-programming sampler avoids enumerating every combination.
2.3 Dataset Overview and Evaluation Design
The data come from one trajectory for each of 29 BEHAVIOR activities (Li et al., 2024), with lengths 3–10 and roughly balanced forward/inverse questions. Task accuracy measures verifier acceptance of the entire answer, including semantically equivalent permutations. Pairwise accuracy measures local transition consistency and gives partial credit. Consequently, task accuracy should not be read as exact index-match accuracy; the appendix defines direction-dependent subset rules.
3 Experiments and Analysis
Experiments compare models with trained annotators, probe rendering and camera changes, and classify errors through state predicates. Most detailed perturbation studies cover two models and a small subset, so they do not establish universal properties of all vision-language models.
3.1 World Modeling as a Proxy for Evaluating Embodied Cognition
The study evaluates seven proprietary and 23 open-weight models with 512×512 images and shared prompts. At length 10, GPT-5 pairwise accuracy is 46.93% forward and 55.33% inverse, versus human 94.75% and 96.29%. Its task accuracy falls to 5.00% and 13.00%, versus human 84.00% and 85.00%. GPT-5 and Gemini 2.5 Pro are strong overall; some open models are competitive, and Cosmos-Reason1 is comparatively stable at longer horizons among similar-sized models. The inverse advantage is a broad trend, with ties and individual reversals in the tables; performance is not strictly monotonic for every model.
3.2 Sensitivity to Image Realism
A separate 960-question set uses manually annotated real videos from a kitchen, dinner table, and workspace, evaluated only with InternVL3.5-241B-A28B. Its length-10 task accuracy is 0% forward and 3.33% inverse; the same qualitative horizon difficulty appears. This limited, unmatched comparison supports transfer of the trend, not a general proof of zero sim-to-real gap. On 300 simulated questions, image-to-image restyling, path tracing, and simplified ray tracing produce small, statistically nonsignificant changes for the two tested models.
| Metric | Forward (Real-World) | Inverse (Real-World) | |||||||||||||||
| 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ||
| Task Accuracy | 73.33 | 50.00 | 33.33 | 13.33 | 8.33 | 3.33 | 0.00 | 0.00 | 90.00 | 85.00 | 55.00 | 38.33 | 21.67 | 13.33 | 6.67 | 3.33 | |
| Pairwise Accuracy | 80.00 | 67.78 | 61.25 | 49.00 | 38.89 | 37.14 | 26.88 | 25.74 | 90.00 | 88.33 | 72.92 | 57.67 | 44.17 | 43.57 | 31.04 | 26.85 | |
3.3 Do VLMs Exhibit Anthropocentric Bias on Human Vision?
GPT-5 mini performs worse under larger aperture changes and fisheye rendering. Figure 5 reports forward aperture-60 p=0.02, so the prose claim that every large change has p≤0.01 is too strong. Raising the camera from 1.75 to 2.25 m produces a reported 13-point forward drop; the inverse and 1.5 m comparisons are nonsignificant. InternVL shows weaker significance. These are sensitivities consistent with viewpoint bias; they do not separate training priors from visibility and geometric difficulty. The setup and figures use opposite sign conventions for Δ, so changes here are described in words.
3.4 Do VLMs Exhibit Anthropocentric Bias on Embodiment?
Changing the robot to white, randomly recolored, or skin-toned produces small nonsignificant differences in the selected models; this does not establish self-awareness. Hand-related predicate errors favor the right hand. For GPT-5 mini forward questions, 9.38% of left-hand changes are confused with right-hand changes, versus 4.67% in the reverse direction. The pattern also appears in humans. These observational error rates do not isolate a causal training-data explanation or measure physical hand preference.
3.5 Error Analysis
Error analysis converts output orderings back into symbolic changes, enabling more informative comparisons than index disagreement alone. These are errors induced by selected candidate permutations, not freely generated action descriptions.
3.5.1 Experimental Setup
For forward answers, reordered images imply a sequence of visible graph differences; inverse answers directly reorder the supplied actions. Component sets are compared for entity substitution, predicate substitution, polarity reversal, omitted changes, and extra changes. Counts are aggregated over questions and transitions, treating state differences as independent events. Reused trajectories and overlapping questions limit that independence assumption.
3.5.2 Error Distribution Analysis
For GPT-5, omissions plus hallucinated changes comprise 81.0% of forward structural errors and 83.6% of inverse errors. Forward errors are 43.9% hallucination and 37.1% omission; inverse errors split 41.8% each. These percentages describe the composition of classified component errors, not the fraction of all questions that fail. They support difficulty tracking which changes occurred, but do not by themselves identify the model’s internal reasoning process.
4 Related Work
ENACT connects embodied-cognition theory, action-conditioned dynamics, and vision-language robot reasoning. It differs from generative world-model evaluation by scoring ordering instead of predicted pixels. Relative to GVL’s observation-order value estimation (Ma et al., 2024), it interleaves observations with an explicit scene-graph action space and need not assume that a trajectory continually improves toward a goal.
5 Conclusions and Limitations
The benchmark exposes difficult interaction-conditioned ordering problems and offers a scalable simulator-based construction. Its interpretation depends on the symbolic abstraction, verifier, sampled activities, and evaluation protocol.
Conclusions.
The principal findings are a widening human–model gap with sequence length, generally stronger inverse ordering, sensitivity to some camera changes, and asymmetric hand-related errors. State-change omissions and additions dominate the structural analysis. These results motivate better embodied reasoning but do not demonstrate that benchmark gains improve actual planning or control.
Limitations.
The diagnostics cover a limited set of embodiments and settings; expensive ablations use a small model/data subset. No finetuning benefit is measured, and video-generation models are excluded because of physical inconsistency and evaluation difficulty. Symbolic changes omit continuous dynamics. Many questions are derived from only 29 underlying demonstrations, so combinatorial scale should not be confused with environmental diversity.
Ethics Statement
The authors report simulator-based construction, above-local-minimum-wage compensation for human evaluation, and no sensitive annotation content. They acknowledge simulator bias, computational cost, and potential dual use. The main benchmark is simulated; the separate real-video experiment should not be conflated with that data description.
Reproducibility Statement
The paper links a public repository and dataset and states that generation, verification, prompts, evaluation, and analyses are provided. This edition preserves those source links and the reported procedure; it does not independently reproduce the code or API-model runs.
Appendix
The appendix specifies the sampler, semantic verifier, annotation process, model configurations, additional results, prompts, and error-analysis algorithms. All original examples and displays are retained.
Appendix A ENACT : Egocentric Interactive Embodied Cognition Test
The construction appendix makes explicit how raw simulator replay becomes a symbolic transition graph, sampled questions, and semantically graded answers.
A.1 Notations
The notation tables distinguish raw frames, segmented frames, sampled paths, graph differences, visible differences, and action signatures. The HTML numbers appendix tables 3–12, whereas the PDF labels the same tables 5–14; no additional table bodies occupy the skipped PDF numbers. This edition keeps the HTML’s internally linked numbering.
| Notation | Short description | Notation | Short description |
| # frames in a raw replay | image height and width | ||
| index set | # segmented key frames | ||
| segmented timestamps | RGB & scene graph at timestamp | ||
| RGB image at time | scene graph at time | ||
| space of scene graphs | target trajectory length (steps) | ||
| # sampled trajectories | adjacency matrix on (DAG) | ||
| successors of node | # edges in the DAG | ||
| scene-graph difference (see long) | visibility predicate (see long) | ||
| visible-change extractor (see long) | action space | ||
| local action | action from to | ||
| key-frame trajectory (see long) | state sequence along | ||
| action sequence along | set of sampled trajectories | ||
| # paths of length ending at | end-node weight | ||
| predecessor set in backtracking | weighted discrete distribution | ||
| Iverson bracket (true=1, false=0) | datasets | ||
| component in signature | The operation key in component | ||
| The entity involved in component | The predicate in component |
| Notation | Longer description |
| Raw replay trajectory with RGB observations and scene graphs . | |
| A difference operator over scene graphs summarizing semantic changes (objects, relations, attributes) between frames and . | |
| Predicate returning iff the semantic difference is visually verifiable; induces an edge when and the predicate is true (frame skipping allowed). | |
| Action-level representation extracted from ; may be atomic or composite and can be empty when no visible semantic change exists. | |
| Key-frame trajectory: strictly increasing indices with valid edges for all . | |
| Sequences induced by : , . | |
| recurrence | Base: . Recurrence: for . |
| Deterministic finite-horizon fragment induced by with transition . | |
| Signature corresponding to an action , transformed from natural language to predicate-based structural format. |
A.2 Key-Frame Trajectories Synthesis for Scalable Data Generation
Temporal and semantic filtering controls which frames enter the candidate pool. Valid subsequences are paths through a graph whose edges encode visible changes.
A.2.1 Segmented Frames with Abstract State Changes
A new state must persist for 40 simulator frames, about 1.3 seconds at 30 Hz, to reduce premature symbolic events. Additional filtering suppresses small oscillations. A one-hot change signature is accepted only when cosine similarity with the previous accepted change is below 0.97. These are chosen hyperparameters for the studied household interactions, not general perceptual constants. The original scene-graph examples show the node, edge, and change formats.
A.2.2 Key-Frame Trajectories Synthesis
Frames form a directed acyclic graph with an edge for each valid visible transition. Dynamic programming counts length-ℓ paths ending at every node. Sampling an endpoint proportionally to its count and recursively sampling predecessors by their counts gives a uniform distribution over valid length-L paths. Repeated samples need not be unique. The abstraction makes semantic evaluation tractable but deliberately discards detailed motor dynamics; the original recurrence and complete algorithm remain below.
Equation
A.3 Dataset Statistics and Evaluation Design
Question lengths are approximately balanced, while the predicate vocabulary fixes which aspects of an interaction can affect the score. The verifier evaluates transition semantics rather than simply requiring the recorded permutation.
| Predicate Classes | ||
| RightGrasping | LeftGrasping | OnTop |
| Inside | Under | Contains |
| Covered | Open | ToggledOn |
| Cooked | Transition | |
A.3.1 Dataset Statistics
Eleven predicate classes represent spatial relations, open/on/cooked/covered states, transformations, and left/right grasping. Near-uniform counts across lengths 3–10 prevent short questions from dominating the benchmark. Balance across length and direction does not imply balanced hand usage, objects, or independent activities.
A.3.2 Evaluation Design
Each adjacent pair is encoded as a set of operation/entity/predicate components. The verifier separates the reference-visible subset from the full reference transition and computes the transition implied by a predicted ordering.
Online verifier.
An exact permutation is accepted. For same-length semantic alternatives, forward acceptance requires every reference-visible component to be contained in the predicted full difference; inverse acceptance requires each predicted action to be a subset of the full reference difference. The forward rule can therefore include additional components, while the inverse rule can omit reference components. This direction-dependent relaxation matters when interpreting the forward/inverse gap; neither rule is equivalent to full equality of transitions.
Equations
Metrics.
Task accuracy is the mean whole-answer acceptance rate. Pairwise accuracy averages subset-satisfying local matches; accepted answers receive one. Length-mismatched predictions get partial credit through a monotone alignment maximizing valid pairs, but fail task acceptance. The displayed formulas use L as the normalizer although the initial formulation has L frames and L−1 transitions; readers reproducing the metric should check the released implementation’s indexing convention.
Equations
A.3.3 ENACT Examples
The three-, six-, and nine-frame examples expose increasing visual and symbolic bookkeeping. They include the initial image, candidate future images or actions, the supplied conditioning sequence, and the reference ordering. Every original panel and action description is retained.
Appendix B Experiments and Analysis
The experimental appendix provides annotation details, model/API identifiers, full outcome tables, ablations, and visual examples for each intervention.
B.1 Human Annotation
Three trained annotators use the same instructions as the models. Their scores are an empirical comparison group, rather than a ceiling on human capability.
B.1.1 Annotation Interface & Human Performance Evaluation
Gradio interfaces support arranging images for forward questions and arranging actions between images for inverse questions. The preserved screenshots show candidate selection, placeholders, and task instructions. Interface assistance and training are part of the human protocol.
B.1.2 Inter-Annotator Agreement Analysis
Five items per annotator, direction, and length create 240 questions, each subsequently labeled by all three annotators. Ordinal Krippendorff α (Krippendorff, 2011) is 0.8320, with a 1,000-bootstrap 95% interval of [0.7879, 0.8682]. Exact three-way agreement occurs on 184/240 questions (76.67%); pairwise α ranges from 0.8180 to 0.8518. Agreement supports consistency of this annotation protocol, not exhaustive validity of the cognitive interpretation.
Equation
B.2 World Modeling as a Proxy for Evaluating Embodied Cognition
Full tables separate all-or-nothing acceptance from partial transition consistency and include contact, action-encoding, and verifier-relaxation analyses.
| Task | Action–Effect Reasoning | Causal Inference | Affordance Recognition | Embodied Awareness | Temporal Abstraction |
| Forward | |||||
| Inverse |
B.2.1 Experimental Setup
The protocol uses one template per task, 512×512 images, temperature zero, and a Python-list permutation response without explanation. The original model-version table and complete forward/inverse prompts are retained. A fixed prompt limits tuning differences, but temperature zero alone does not guarantee reproducibility across API versions or serving systems. The cognitive-capability table states the authors’ proposed interpretation of each task.
| Organization | Model Name | Release Date | Full Name |
| |
| Proprietary Models | |||||
| OpenAI | GPT-5 | 2025-08 | gpt-5-2025-08-07 | OpenAI API | |
| GPT-5-mini | 2025-08 | gpt-5-mini-2025-08-07 | OpenAI API | ||
| GPT-5-nano | 2025-08 | gpt-5-nano-2025-08-07 | OpenAI API | ||
| Gemini 2.5 Pro | 2025-06 | gemini-2.5-pro | Gemini API | ||
| Gemini 2.5 Flash | 2025-06 | gemini-2.5-flash | Gemini API | ||
| Gemini 2.5 Flash-Lite | 2025-06 | gemini-2.5-flash-lite | Gemini API | ||
| Anthropic | Claude Sonnet 4 | 2025-05 | claude-sonnet-4-20250514 | Anthropic API | |
| Open-Weight Models | |||||
| Zhipu AI | GLM-4.5V | 2025-08 | GLM-4.5V | Zhipu Foundation Model Open Platform API | |
| GLM-4.1V-Thinking | 2025-07 | GLM-4.1V-Thinking-FlashX | Zhipu Foundation Model Open Platform API | ||
| Meta | Llama-4-Scout-17B-16E-Ins | 2025-04 | meta-llama/Llama-4-Scout-17B-16E-Instruct | ModelScope API | |
| Llama-4-Mav-17B-128E-Ins | 2025-04 | meta-llama/Llama-4-Mav-17B-128E-Instruct | ModelScope API | ||
| Shanghai AI Lab | InternVL3.5-241B-A28B | 2025-08 | OpenGVLab/InternVL3.5-241B-A28B | Intern API | |
| InternVL3.5-14B | 2025-08 | OpenGVLab/InternVL3.5-14B | Hugging Face Transformers | ||
| InternVL3.5-8B | 2025-08 | OpenGVLab/InternVL3.5-8B | Hugging Face Transformers | ||
| InternVL3.5-4B | 2025-08 | OpenGVLab/InternVL3.5-4B | Hugging Face Transformers | ||
| Gemma-3-27b-it | 2025-03 | google/gemma-3-27b-it | Gemini API | ||
| Gemma-3-12b-it | 2025-03 | google/gemma-3-12b-it | Gemini API | ||
| Gemma-3-4b-it | 2025-03 | google/gemma-3-4b-it | Gemini API | ||
| Alibaba | QVQ-72B-Preview | 2024-12 | Qwen/QVQ-72B-Preview | ModelScope API | |
| Qwen2.5-VL-72B-Ins | 2025-01 | Qwen/Qwen2.5-VL-72B-Instruct | ModelScope API | ||
| Qwen2.5-VL-32B-Ins | 2025-01 | Qwen/Qwen2.5-VL-32B-Instruct | ModelScope API | ||
| Qwen2.5-VL-7B-Ins | 2025-01 | Qwen/Qwen2.5-VL-7B-Instruct | Hugging Face Transformers | ||
| Qwen2.5-VL-3B-Ins | 2025-01 | Qwen/Qwen2.5-VL-3B-Instruct | Hugging Face Transformers | ||
| AIDC | Ovis2.5-9B | 2025-08 | AIDC-AI/Ovis2.5-9B | Hugging Face Transformers | |
| Ovis2.5-2B | 2025-08 | AIDC-AI/Ovis2.5-2B | Hugging Face Transformers | ||
| OpenBMB | MiniCPM-V-4.5 | 2025-08 | openbmb/MiniCPM-V-4.5 | Hugging Face Transformers | |
| MiniCPM-o-2.6 | 2025-01 | openbmb/MiniCPM-o-2.6 | Hugging Face Transformers | ||
| Hugging Face | Idefics3-8B-Llama3 | 2024-08 | HuggingFaceM4/Idefics3-8B-Llama3 | Hugging Face Transformers | |
| Nvidia | Cosmos-Reason1 | 2025-05 | nvidia/Cosmos-Reason1 | Hugging Face Transformers | |
B.2.2 Detailed Results
The complete task and pairwise tables show how partial local consistency can survive when complete ordering collapses. Human length-10 task accuracy remains 84–85%, while the strongest model task scores are much lower. Preserve both metrics when comparing models; a single average would conceal horizon dependence and verifier behavior.
B.2.3 Contact Experiment
Adding binary contact changes to key-frame selection preserves the broad horizon degradation and inverse advantage for InternVL3.5-241B. At three frames, task accuracy is 86.67% forward and 90.00% inverse; at seven, 3.45% and 16.67%. This checks one alternative sampling criterion, without establishing robustness to arbitrary physical predicates or all models.
| Metric | Forward (with contact changes) | Inverse (with contact changes) | |||||||||||||||
| 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ||
| Task Accuracy | 86.67 | 43.33 | 36.67 | 20.00 | 3.45 | 3.33 | 0.00 | 0.00 | 90.00 | 73.33 | 30.00 | 26.67 | 16.67 | 3.33 | 6.67 | 0.00 | |
| Pairwise Accuracy | 90.00 | 72.22 | 60.00 | 53.33 | 48.28 | 42.38 | 31.67 | 34.44 | 90.00 | 82.22 | 55.83 | 57.33 | 46.11 | 32.86 | 38.75 | 27.78 | |
B.2.4 How Does Action Representation Affect VLMs’ Performance?
On 2,304 questions, natural-language, structured-symbolic, and emoji-style action encodings retain broadly similar direction and horizon trends for InternVL. The inverse advantage therefore is not solely dependent on familiar natural-language verbs. Absolute scores vary and some individual cells tie or reverse; changing representation does not isolate every linguistic or verifier-related explanation.
| Metric | Forward | Inverse | |||||||||||||||
| 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ||
| Vanilla (Natural Language) | |||||||||||||||||
| Task Accuracy | 68.97 | 35.17 | 27.59 | 8.97 | 6.90 | 2.07 | 0.69 | 0.00 | 83.45 | 60.69 | 44.14 | 24.83 | 13.79 | 7.59 | 4.14 | 0.00 | |
| Pairwise Accuracy | 76.21 | 57.70 | 57.41 | 41.24 | 38.85 | 30.34 | 29.05 | 26.52 | 83.45 | 69.66 | 61.72 | 53.10 | 47.01 | 40.39 | 34.40 | 27.25 | |
| Symbolic Predicates | |||||||||||||||||
| Task Accuracy | 67.59 | 40.00 | 20.00 | 9.66 | 4.14 | 0.69 | 1.38 | 0.00 | 79.86 | 51.03 | 40.00 | 23.45 | 8.97 | 5.52 | 2.76 | 0.73 | |
| Pairwise Accuracy | 74.48 | 61.61 | 48.97 | 41.52 | 35.75 | 32.12 | 30.34 | 25.68 | 79.86 | 62.99 | 58.62 | 51.59 | 39.43 | 36.85 | 32.07 | 24.33 | |
| Emoji-Style Encodings | |||||||||||||||||
| Task Accuracy | 65.52 | 44.14 | 18.62 | 11.03 | 8.28 | 0.69 | 0.69 | 0.00 | 77.24 | 48.28 | 35.17 | 23.45 | 12.41 | 7.59 | 2.76 | 0.00 | |
| Pairwise Accuracy | 73.10 | 64.60 | 48.62 | 39.45 | 38.05 | 27.49 | 26.72 | 23.93 | 77.93 | 61.38 | 59.14 | 45.38 | 40.80 | 36.06 | 27.16 | 25.30 | |
B.2.5 How Often Do Accepted Predictions Omit Parts of Transitions?
Among semantically accepted answers, reported data-level strict-subset mismatch is 1.65% for GPT-5, 1.49% for InternVL, and 2.26% for humans. Pair-level mismatch is larger for models: 15.31% and 13.96%, versus 3.62%. The section describes subset acceptance generically, but the formal forward rule uses the opposite inclusion direction. These omission statistics should therefore be read with the precise task-specific verifier in mind.
| Model | Data-Level Mismatch (%) | Pair-Level Mismatch (%) |
| GPT-5 | 1.65 | 15.31 |
| Human | 2.26 | 3.62 |
| InternVL3.5-241B | 1.49 | 13.96 |
B.3 Additional Ablation Experiments Common Setup
The common ablation set contains 50 questions per direction at lengths 3, 6, and 9: 300 total. GPT-5 mini and InternVL3.5-241B-A28B are evaluated with two-tailed unpaired Welch tests and a p<0.05 threshold. Questions reuse matched trajectories across variants; independent API sessions do not alone establish statistical independence of their difficulty. No multiple-comparison correction is described. Nonsignificance is not an equivalence test.
B.4 Sensitivity to Image Realism
Rendering studies compare the baseline with generated photographic styling and two simulator-renderer variants. They probe image appearance while attempting to hold the interaction fixed, rather than comparing matched real and simulated executions.
B.4.1 Realistic: Generated Images as Real World Proxy
GPT-image-1 restyles simulator frames with instructions to preserve objects, geometry, camera, and actions. The complete prompt specifies lighting, materials, and fidelity targets. Those requested thresholds are prompt text, not evidence that every generated image was measured to satisfy them. Generated photographs remain synthetic, and imperfect action or gripper preservation can confound a realism comparison.
B.4.2 Path Tracing Setup
The higher-fidelity variant uses Isaac Sim’s path-tracing renderer. Its retained example shows the same action sequence under different lighting and rendering.
B.4.3 Ray Tracing Only Setup
The simplified ray-tracing variant disables reflections, DLSS, ambient occlusion, sampled lighting, ambient light, and flow. The example retains the interaction sequence while changing visual appearance. It is a particular rendering intervention, not a general ordering of all possible image realism.
| Metric | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| Forward (with random perturbations) | |||||||||
| Task Accuracy | |||||||||
| Pairwise Accuracy | |||||||||
| Inverse (with random perturbations) | |||||||||
| Task Accuracy | |||||||||
| Pairwise Accuracy | |||||||||
B.4.4 Robustness to Noise in Predicate Deltas
Random perturbations of predicate deltas on a 2,304-question InternVL subset yield small reported standard deviations over three seeds and preserve broad trends. The section does not specify a complete perturbation distribution or rate, limiting the breadth of the robustness claim. Small seed variation does not establish robustness to systematic annotation errors.
B.5 Sensitivity to Camera Configurations
Camera experiments change aperture, projection, or height. Original example frames and result plots are retained so visibility changes can be inspected alongside accuracy.
B.5.1 Camera Aperture Setup
The baseline aperture setting is 40, compared with 30, 60, and 80. These are simulator camera parameters rather than explicitly reported field-of-view angles.
B.5.2 Fisheye Lens Setup
The fisheye variant uses Isaac Sim’s fisheyePolynomial setting. Its distorted projection changes the mapping between image position and scene geometry; the source compares it with everyday wide-angle cameras.
B.5.3 Camera Height Setup
Camera height varies from the 1.75 m baseline to 2.25 m and 1.5 m. The lower offset is limited because stronger lowering hides relevant objects. This selection itself constrains the range of embodiments tested.
B.6 Do VLMs Have Embodied Biases?
Appearance interventions and predicate-level handedness analysis address different questions: visual robustness versus asymmetric interpretation of left/right interactions.
B.6.1 Robot Appearance
The robot is rendered white, with a new random color at every frame, or skin-toned. These changes test robustness to surface appearance while retaining body geometry and the underlying action sequence.
B.6.2 Handedness
Across the displayed models and humans, right-hand precision and recall are higher, and left-to-right confusion exceeds right-to-left confusion. These component-level measurements are derived from inferred transitions, with standard-error bars. Shared dataset and annotation asymmetries remain possible contributors.
Appendix C Error Analysis
The error appendix defines structural and semantic categories, provides original pseudocode, and illustrates each category with selected examples. These examples explain the taxonomy rather than estimate prevalence on their own.
C.1 Methodology for Error Calculations
Action signatures turn natural-language descriptions into finite sets of atomic changes. Comparing those sets supports both structural error classification and semantic labeling.
Signature Modeling from Scene Graph-level Differences
An atomic component records an edge operation between two entities, a node-property operation, or a transition from one property to another. The original display preserves the tuple definitions. This representation ties evaluation to the chosen symbolic ontology.
Equation
Error Modeling from Signatures
The pipeline separates matched, missing, and extra components, then detects polarity, predicate, and entity substitutions before assigning residual omissions and hallucinations. A predefined predicate mapping supplies spatial, functional, material, or agent-interaction labels. Algorithm 2 explicitly filters unequal-length signature sequences before this analysis, so its error population need not equal every failed benchmark answer. The complete algorithms remain available for inspecting these choices.
Handedness Asymmetry Error Modeling
Precision divides matched hand components by predicted components; recall divides them by reference components. A confusion event requires missing components for one hand and hallucinated components for the opposite hand, without a same-hand hallucinated component. The procedure counts these symbolic substitutions, not physical hand movements or a demonstrated motor preference.
C.2 Structural Error Analysis
The plots distinguish total error counts from within-error proportions. Humans produce far fewer errors, even when their relative category mix resembles a model’s. The examples show wrong objects, wrong predicates, reversed changes, omitted changes, and extra changes. They should not be interpreted as unconstrained text hallucinations because predictions select from supplied candidates.
C.3 Semantic Error Analysis
Spatial relations and agent interactions dominate the semantic error distributions. Forward tasks contain relatively more spatial-relation errors, while inverse tasks contain more agent-interaction errors. Original examples cover spatial, functional, material, and grasp-related mistakes for both task directions.
Appendix D The Use of Large Language Models
The authors disclose using Gemini 2.5 Pro and GPT-5 for writing assistance and related-work retrieval, with human revision, fact-checking, and responsibility for the manuscript.