Literature
Display
← All papers

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction

Qineng Wang · Wenlong Huang · Yu Zhou · Hang Yin · Tianwei Bao · Jianwen Lyu · Weiyu Liu · Ruohan Zhang · Jiajun Wu · Li Fei-Fei · Manling Li

Preprint posted · arXiv (preprint) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

ENACT measures whether a model can align egocentric observations with symbolic changes during household interactions. Its central result is a large human–model gap as sequences lengthen. This is an offline ordering benchmark and a proxy for embodied reasoning; successful ordering does not establish a controllable world model or successful robot behavior.

Original figure 1, retaining native vector text and original embedded pixels.
Figure 1: ENACT casts embodied cognition evaluation as world modeling through egocentric interaction. Grounded in POMDP framework, ENACT considers two tasks from diverse activities and scenes: forward world modeling (ordering observations given actions) and inverse world modeling (ordering actions given observations). Evaluation shows that performance of VLMs drops as interaction horizons lengthen, performs better on inverse task, and lags behind humans.

At a glance

The benchmark supplies 8,972 simulated questions from 29 household activities. Forward questions ask for the order of future images given actions; inverse questions ask for the action order given images. Thirty vision-language models generally struggle with longer sequences, whereas trained human annotators remain strong. Camera changes and left/right-hand confusion expose additional weaknesses. The contribution is an evaluation and data-generation pipeline, with no model training experiment.

1 Introduction

Motivated by embodied cognition (Smith and Gasser, 2005) and action-conditioned world models (Ha and Schmidhuber, 2018), ENACT replaces pixel generation with sequence ordering. This removes image-generation fidelity from the score while retaining observation–action consistency. Simulator state supplies an explicit symbolic action vocabulary. Affordances, memory, and embodiment are proposed interpretations of the resulting task demands, rather than independently isolated cognitive measurements.

Original figure 2, retaining native vector text and original embedded pixels.
Figure 2: Overview of ENACT data curation pipeline. We first obtain aligned scene graphs (states) and RGB observations from a mobile manipulation dataset in a robotics simulation (BEHAVIOR). The trajectory is then segmented by identifying key-frames where an abstract state change occurs (i.e., the scene graph difference is non-empty). From this set of key-frames, we sample multiple key-frame trajectories, which are used to construct the forward and inverse world modeling VQA questions. Here NN refers to the number of all sampled trajectories across all step lengths.

2 ENACT : Egocentric Interactive Embodied Cognition Test

Each question combines egocentric images with state changes extracted from a simulator. The pipeline first identifies meaningful changes, then samples visible subsequences, shuffles one modality, and evaluates an ordering with a semantic verifier.

2.1 Problem Formulation

States are scene graphs, observations are RGB images, and actions are differences between graph states—not motor commands. A sampled trajectory contains L frames and L−1 transitions. Its action descriptions retain changes visible in both endpoint images. Forward modeling fixes the initial image and ordered actions while shuffling later images; inverse modeling fixes the images while shuffling actions. The POMDP framing describes partial observability, but the evaluation itself uses a supplied finite storyboard rather than online interaction.

2.2 Key-Frame Trajectories Synthesis for Scalable Data Generation

Predicate changes select candidate timestamps. Similarity filtering suppresses repetitive changes, and valid subsequences require nonempty visible transitions; object transformations allow transient occlusion. Skipping intermediate frames produces up to a binomial number of candidate questions from one episode. This increases question count without increasing the number of independent environments or demonstrations. A dynamic-programming sampler avoids enumerating every combination.

Original figure 3, retaining native vector text and original embedded pixels.
Figure 3: Data sources and QA examples. ENACT is built from diverse, long-horizon activities performed by real robots (left). We provide examples for (mid) forward world modeling and (right) inverse world modeling. More QA examples and prompts are available in the Appendix A.3.3.

2.3 Dataset Overview and Evaluation Design

The data come from one trajectory for each of 29 BEHAVIOR activities (Li et al., 2024), with lengths 3–10 and roughly balanced forward/inverse questions. Task accuracy measures verifier acceptance of the entire answer, including semantically equivalent permutations. Pairwise accuracy measures local transition consistency and gives partial credit. Consequently, task accuracy should not be read as exact index-match accuracy; the appendix defines direction-dependent subset rules.

Original table 1, preserving every value, label, and highlight.
Table 1: Evaluation on ENACT (Pairwise Accuracy). Dark gray indicates the best result within each category (Proprietary or Open-Weight Models), and Light gray denotes the second-best result within the category. Complete results are in Table 7 (Task Accuracy) and Table 8 (Pairwise Accuracy).

3 Experiments and Analysis

Experiments compare models with trained annotators, probe rendering and camera changes, and classify errors through state predicates. Most detailed perturbation studies cover two models and a small subset, so they do not establish universal properties of all vision-language models.

3.1 World Modeling as a Proxy for Evaluating Embodied Cognition

The study evaluates seven proprietary and 23 open-weight models with 512×512 images and shared prompts. At length 10, GPT-5 pairwise accuracy is 46.93% forward and 55.33% inverse, versus human 94.75% and 96.29%. Its task accuracy falls to 5.00% and 13.00%, versus human 84.00% and 85.00%. GPT-5 and Gemini 2.5 Pro are strong overall; some open models are competitive, and Cosmos-Reason1 is comparatively stable at longer horizons among similar-sized models. The inverse advantage is a broad trend, with ties and individual reversals in the tables; performance is not strictly monotonic for every model.

Key Takeaways: World Modeling as a Proxy for Evaluating Embodied Cognition • Higher performance on the inverse task than the forward one highlights stronger language-based retrospection than action-conditioned visual reasoning. • Long-horizon degradation reveals limited interactive, spatial memory under partial observability. • The human–model gap shows that current VLMs are still far from robust embodied world models in mobile manipulation settings.

3.2 Sensitivity to Image Realism

A separate 960-question set uses manually annotated real videos from a kitchen, dinner table, and workspace, evaluated only with InternVL3.5-241B-A28B. Its length-10 task accuracy is 0% forward and 3.33% inverse; the same qualitative horizon difficulty appears. This limited, unmatched comparison supports transfer of the trend, not a general proof of zero sim-to-real gap. On 300 simulated questions, image-to-image restyling, path tracing, and simplified ray tracing produce small, statistically nonsignificant changes for the two tested models.

Original figure 4, retaining native vector text and original embedded pixels.
Figure 4: Real-World Evaluations. Key frames from the three real-world scenes used in our evaluation: kitchen, dinner table, and workspace. Together, these scenes contain diverse rigid, deformable, and articulated objects in diverse environments with varying lighting conditions.
Metric Forward (Real-World) Inverse (Real-World)
3 4 5 6 7 8 9 10 3 4 5 6 7 8 9 10
Task Accuracy 73.33 50.00 33.33 13.33 8.33 3.33 0.00 0.00 90.00 85.00 55.00 38.33 21.67 13.33 6.67 3.33
Pairwise Accuracy 80.00 67.78 61.25 49.00 38.89 37.14 26.88 25.74 90.00 88.33 72.92 57.67 44.17 43.57 31.04 26.85
Table 2: Real-world evaluation of InternVL3.5-241B-A28B. Task and pairwise accuracies (%) on 960 QA pairs generated from real-world manipulation videos. We report performance across different interaction horizons for both forward and inverse world modeling.
Key Takeaways: Image Realism • Real-world evaluation mirrors our simulation findings with minimal sim-to-real gap, while simulation offers a more controlled and reproducible testbed. • Insensitivity to rendering variations indicates bottlenecks in multi-step interaction reasoning rather than low-level image realism.

3.3 Do VLMs Exhibit Anthropocentric Bias on Human Vision?

GPT-5 mini performs worse under larger aperture changes and fisheye rendering. Figure 5 reports forward aperture-60 p=0.02, so the prose claim that every large change has p≤0.01 is too strong. Raising the camera from 1.75 to 2.25 m produces a reported 13-point forward drop; the inverse and 1.5 m comparisons are nonsignificant. InternVL shows weaker significance. These are sensitivities consistent with viewpoint bias; they do not separate training priors from visibility and geometric difficulty. The setup and figures use opposite sign conventions for Δ, so changes here are described in words.

Key Takeaways: Anthropocentric Bias on Human Vision • Performance degradation on non-standard views implies VLMs are biased towards human-like egocentric viewpoints and intrinsics. • Reliance on human-centric visual priors limits generalization to diverse robotic embodiments with non-standard optics.
Original figure 5, retaining native vector text and original embedded pixels.
Figure 5: Evaluations on image realism and anthropocentric bias on human vision through ENACT. Heatmaps show two-tailed unpaired t-test results against the baseline, using Pairwise Accuracy. p<0.05p<0.05 is considered significant. Darker red means more significant. Δ\Delta is the performance change from the baseline. If significant and Δ<0\Delta<0, the setting is worse than the baseline. C.2 reports the robot’s performance on the left- and right-hand predicates, where Mixing is the proportion of ground truth left or right cases that are predicted as the other hand (i.e., mixing one hand into the other). ±\pm means standard error.

3.4 Do VLMs Exhibit Anthropocentric Bias on Embodiment?

Changing the robot to white, randomly recolored, or skin-toned produces small nonsignificant differences in the selected models; this does not establish self-awareness. Hand-related predicate errors favor the right hand. For GPT-5 mini forward questions, 9.38% of left-hand changes are confused with right-hand changes, versus 4.67% in the reverse direction. The pattern also appears in humans. These observational error rates do not isolate a causal training-data explanation or measure physical hand preference.

Key Takeaways: Anthropocentric bias on Embodiment • VLMs are robust to the embodiment appearance variations. • VLMs exhibit strong right-handed bias, which is consistent with human handedness distribution.

3.5 Error Analysis

Error analysis converts output orderings back into symbolic changes, enabling more informative comparisons than index disagreement alone. These are errors induced by selected candidate permutations, not freely generated action descriptions.

3.5.1 Experimental Setup

For forward answers, reordered images imply a sequence of visible graph differences; inverse answers directly reorder the supplied actions. Component sets are compared for entity substitution, predicate substitution, polarity reversal, omitted changes, and extra changes. Counts are aggregated over questions and transitions, treating state differences as independent events. Reused trajectories and overlapping questions limit that independence assumption.

3.5.2 Error Distribution Analysis

For GPT-5, omissions plus hallucinated changes comprise 81.0% of forward structural errors and 83.6% of inverse errors. Forward errors are 43.9% hallucination and 37.1% omission; inverse errors split 41.8% each. These percentages describe the composition of classified component errors, not the fraction of all questions that fail. They support difficulty tracking which changes occurred, but do not by themselves identify the model’s internal reasoning process.

Original figure
Figure 6: Error Distribution, broken down by forward and inverse tasks, evaluated on GPT-5.
Key Takeaways: Structural Error Analysis • The dominance of hallucinations suggests models rely on learned textual priors rather than faithful visual grounding. • High omission rates reveal a fundamental deficit in tracking object persistence across complex state transitions under egocentric partial observation.

4 Related Work

ENACT connects embodied-cognition theory, action-conditioned dynamics, and vision-language robot reasoning. It differs from generative world-model evaluation by scoring ordering instead of predicted pixels. Relative to GVL’s observation-order value estimation (Ma et al., 2024), it interleaves observations with an explicit scene-graph action space and need not assume that a trajectory continually improves toward a goal.

5 Conclusions and Limitations

The benchmark exposes difficult interaction-conditioned ordering problems and offers a scalable simulator-based construction. Its interpretation depends on the symbolic abstraction, verifier, sampled activities, and evaluation protocol.

Conclusions.

The principal findings are a widening human–model gap with sequence length, generally stronger inverse ordering, sensitivity to some camera changes, and asymmetric hand-related errors. State-change omissions and additions dominate the structural analysis. These results motivate better embodied reasoning but do not demonstrate that benchmark gains improve actual planning or control.

Limitations.

The diagnostics cover a limited set of embodiments and settings; expensive ablations use a small model/data subset. No finetuning benefit is measured, and video-generation models are excluded because of physical inconsistency and evaluation difficulty. Symbolic changes omit continuous dynamics. Many questions are derived from only 29 underlying demonstrations, so combinatorial scale should not be confused with environmental diversity.

Ethics Statement

The authors report simulator-based construction, above-local-minimum-wage compensation for human evaluation, and no sensitive annotation content. They acknowledge simulator bias, computational cost, and potential dual use. The main benchmark is simulated; the separate real-video experiment should not be conflated with that data description.

Reproducibility Statement

The paper links a public repository and dataset and states that generation, verification, prompts, evaluation, and analyses are provided. This edition preserves those source links and the reported procedure; it does not independently reproduce the code or API-model runs.

Appendix

The appendix specifies the sampler, semantic verifier, annotation process, model configurations, additional results, prompts, and error-analysis algorithms. All original examples and displays are retained.

Appendix A ENACT : Egocentric Interactive Embodied Cognition Test

The construction appendix makes explicit how raw simulator replay becomes a symbolic transition graph, sampled questions, and semantically graded answers.

A.1 Notations

The notation tables distinguish raw frames, segmented frames, sampled paths, graph differences, visible differences, and action signatures. The HTML numbers appendix tables 3–12, whereas the PDF labels the same tables 5–14; no additional table bodies occupy the skipped PDF numbers. This edition keeps the HTML’s internally linked numbering.

Notation Short description Notation Short description
TT # frames in a raw replay H,WH,W image height and width
[M][M] index set {1,2,…,M}\{1,2,\dots,M\} MM # segmented key frames
𝒦\mathcal{K} segmented timestamps {t1<⋯<tM}\{t_{1}<\cdots<t_{M}\} (oi,si)(o_{i},s_{i}) RGB & scene graph at timestamp tit_{i}
oto_{t} RGB image at time tt sts_{t} scene graph at time tt
𝒢\mathcal{G} space of scene graphs LL target trajectory length (steps)
RR # sampled trajectories EE adjacency matrix on [M][M] (DAG)
Adj⁡(i)\mathrm{Adj}(i) successors of node ii |E||E| # edges in the DAG
δ⁡(⋅,⋅)\delta(\cdot,\cdot) scene-graph difference (see long) Vis⁡(⋅)\mathrm{Vis}(\cdot) visibility predicate (see long)
Δvis​(⋅,⋅)\Delta_{\mathrm{vis}}(\cdot,\cdot) visible-change extractor (see long) 𝒜\mathcal{A} action space
aia_{i} local action si+1−sis_{i+1}-s_{i} ai→ja_{i\to j} action from ii to jj
π\pi key-frame trajectory (see long) SπS_{\pi} state sequence along π\pi
AπA_{\pi} action sequence along π\pi Π\Pi set of sampled trajectories
D​P​[ℓ,i]DP[\ell,i] # paths of length ℓ\ell ending at ii wiw_{i} end-node weight D​P​[L,i]DP[L,i]
𝒫\mathcal{P} predecessor set in backtracking Categorical⁡(w)\mathrm{Categorical}(w) weighted discrete distribution
𝟏​{⋅}\mathbf{1}\{\cdot\} Iverson bracket (true=1, false=0) 𝒟\mathcal{D} datasets
cc component in signature ais​i​ga_{i}^{sig} γ,transition\gamma,\mathrm{transition} The operation key in component
ee The entity involved in component ρ\rho The predicate in component
Notation Longer description
𝒯={(ot,st)}t=1T\mathcal{T}=\{(o_{t},s_{t})\}_{t=1}^{T} Raw replay trajectory with RGB observations ot∈ℝH×W×3o_{t}\in\mathbb{R}^{H\times W\times 3} and scene graphs st∈𝒢s_{t}\in\mathcal{G}.
δ⁡(si,sj)\delta(s_{i},s_{j}) A difference operator over scene graphs summarizing semantic changes (objects, relations, attributes) between frames ii and jj.
Vis⁡(δ⁡(si,sj))\mathrm{Vis}(\delta(s_{i},s_{j})) Predicate returning 11 iff the semantic difference is visually verifiable; induces an edge i→ji\!\to\!j when i<ji<j and the predicate is true (frame skipping allowed).
Δvis​(si,sj)∈𝒜∪{∅}\Delta_{\mathrm{vis}}(s_{i},s_{j})\in\mathcal{A}\cup\{\varnothing\} Action-level representation extracted from δ⁡(si,sj)\delta(s_{i},s_{j}); may be atomic or composite and can be empty when no visible semantic change exists.
π=(i1,…,iL)\pi=(i_{1},\ldots,i_{L}) Key-frame trajectory: strictly increasing indices with valid edges Eiℓ,iℓ+1=1E_{i_{\ell},i_{\ell+1}}=1 for all ℓ=1,…,L−1\ell=1,\dots,L-1.
Sπ,AπS_{\pi},\,A_{\pi} Sequences induced by π\pi: Sπ=(si1,…,siL)S_{\pi}=(s_{i_{1}},\ldots,s_{i_{L}}),  Aπ=(ai1→i2,…,aiL−1→iL)A_{\pi}=(a_{i_{1}\to i_{2}},\ldots,a_{i_{L-1}\to i_{L}}).
D​P​[ℓ,i]DP[\ell,i] recurrence Base: D​P​[1,i]=1DP[1,i]=1. Recurrence: D​P​[ℓ,i]=∑j<iD​P​[ℓ−1,j]⋅Ej​iDP[\ell,i]=\sum_{j<i}DP[\ell-1,j]\cdot E_{ji} for ℓ=2,…,L\ell=2,\ldots,L.
ℳπ=⟨{siℓ},{aiℓ→iℓ+1},P⟩\mathcal{M}_{\pi}=\langle\{s_{i_{\ell}}\},\{a_{i_{\ell}\to i_{\ell+1}}\},P\rangle Deterministic finite-horizon fragment induced by π\pi with transition P⁡(siℓ,aiℓ→iℓ+1)=siℓ+1P(s_{i_{\ell}},a_{i_{\ell}\to i_{\ell+1}})=s_{i_{\ell+1}}.
ais​i​ga_{i}^{sig} Signature corresponding to an action aia_{i}, transformed from natural language to predicate-based structural format.
Table 3: Notation used throughout the paper.

A.2 Key-Frame Trajectories Synthesis for Scalable Data Generation

Temporal and semantic filtering controls which frames enter the candidate pool. Valid subsequences are paths through a graph whose edges encode visible changes.

A.2.1 Segmented Frames with Abstract State Changes

A new state must persist for 40 simulator frames, about 1.3 seconds at 30 Hz, to reduce premature symbolic events. Additional filtering suppresses small oscillations. A one-hot change signature is accepted only when cosine similarity with the previous accepted change is below 0.97. These are chosen hyperparameters for the studied household interactions, not general perceptual constants. The original scene-graph examples show the node, edge, and change formats.

Original figure 7, retaining native vector text and original embedded pixels.
Figure 7: A scene graph representation detailing the entities (nodes) and their semantic or physical connections (edges) within the BEHAVIOR (Li et al., 2024a) environment.
Original figure 8, retaining native vector text and original embedded pixels.
Figure 8: An example of a scene graph difference, representing a state change by specifying the added and removed edges between objects.

A.2.2 Key-Frame Trajectories Synthesis

Frames form a directed acyclic graph with an edge for each valid visible transition. Dynamic programming counts length-ℓ paths ending at every node. Sampling an endpoint proportionally to its count and recursively sampling predecessors by their counts gives a uniform distribution over valid length-L paths. Repeated samples need not be unique. The abstraction makes semantic evaluation tractable but deliberately discards detailed motor dynamics; the original recurrence and complete algorithm remain below.

Algorithm 1 KFTS: Key-Frame Trajectory Sampling
Original algorithm 1, retaining native vector text and original embedded pixels.
Equation
D​P​[ℓ,i]=∑j<iD​P​[ℓ−1,j]⋅Ej​iDP[\ell,i]=\sum_{j<i}DP[\ell-1,j]\cdot E_{ji}

A.3 Dataset Statistics and Evaluation Design

Question lengths are approximately balanced, while the predicate vocabulary fixes which aspects of an interaction can affect the score. The verifier evaluates transition semantics rather than simply requiring the recorded permutation.

Table 4: The 11 predicate classes used to define abstract state changes in our benchmark.
Predicate Classes
RightGrasping LeftGrasping OnTop
Inside Under Contains
Covered Open ToggledOn
Cooked Transition
Original figure 9, retaining native vector text and original embedded pixels.
Figure 9: The distribution of problems by the number of steps in our ENACT benchmark dataset is shown for both forward (left) and inverse (right) world modeling tasks. The dataset is balanced, with a nearly uniform distribution of problems ranging from 3 to 10 steps.

A.3.1 Dataset Statistics

Eleven predicate classes represent spatial relations, open/on/cooked/covered states, transformations, and left/right grasping. Near-uniform counts across lengths 3–10 prevent short questions from dominating the benchmark. Balance across length and direction does not imply balanced hand usage, objects, or independent activities.

A.3.2 Evaluation Design

Each adjacent pair is encoded as a set of operation/entity/predicate components. The verifier separates the reference-visible subset from the full reference transition and computes the transition implied by a predicted ordering.

Online verifier.

An exact permutation is accepted. For same-length semantic alternatives, forward acceptance requires every reference-visible component to be contained in the predicted full difference; inverse acceptance requires each predicted action to be a subset of the full reference difference. The forward rule can therefore include additional components, while the inverse rule can omit reference components. This direction-dependent relaxation matters when interpreting the forward/inverse gap; neither rule is equivalent to full equality of transitions.

Equations
Ci⊆C~i.C_{i}\subseteq\tilde{C}_{i}.
C~i⊆Fi,\tilde{C}_{i}\subseteq F_{i},

Metrics.

Task accuracy is the mean whole-answer acceptance rate. Pairwise accuracy averages subset-satisfying local matches; accepted answers receive one. Length-mismatched predictions get partial credit through a monotone alignment maximizing valid pairs, but fail task acceptance. The displayed formulas use L as the normalizer although the initial formulation has L frames and L−1 transitions; readers reproducing the metric should check the released implementation’s indexing convention.

Equations
TA=1|𝒟|​∑x∈𝒟𝟏​{accepted​(x)}.\mathrm{TA}=\frac{1}{|\mathcal{D}|}\sum_{x\in\mathcal{D}}\mathbf{1}\{\text{accepted}(x)\}.
PA(x)=1L∑i=1L𝟏{Ci⊆C~i(forward)orC~i⊆Fi(inverse)}.\mathrm{PA}(x)=\frac{1}{L}\sum_{i=1}^{L}\mathbf{1}\{\,C_{i}\subseteq\tilde{C}_{i}\ \text{(forward)}\ \text{or}\ \tilde{C}_{i}\subseteq F_{i}\ \text{(inverse)}\,\}.
PA=∑x#​correct pairs in ​x∑xLx.\mathrm{PA}=\frac{\sum_{x}\#\text{correct pairs in }x}{\sum_{x}L_{x}}.

A.3.3 ENACT Examples

The three-, six-, and nine-frame examples expose increasing visual and symbolic bookkeeping. They include the initial image, candidate future images or actions, the supplied conditioning sequence, and the reference ordering. Every original panel and action description is retained.

Original figure 10, retaining native vector text and original embedded pixels.
Figure 10: 3-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.
Original figure 11, retaining native vector text and original embedded pixels.
Figure 11: 6-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.
Original figure 12, retaining native vector text and original embedded pixels.
Figure 12: 9-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.

Appendix B Experiments and Analysis

The experimental appendix provides annotation details, model/API identifiers, full outcome tables, ablations, and visual examples for each intervention.

B.1 Human Annotation

Three trained annotators use the same instructions as the models. Their scores are an empirical comparison group, rather than a ceiling on human capability.

B.1.1 Annotation Interface & Human Performance Evaluation

Gradio interfaces support arranging images for forward questions and arranging actions between images for inverse questions. The preserved screenshots show candidate selection, placeholders, and task instructions. Interface assistance and training are part of the human protocol.

Refer to caption
Figure 13: The annotation interface used for evaluating human performance on Forward World Modeling problems. Annotators are presented with a “Current State” image (top left) and an ordered list of textual actions. The main task is to fill the “Next State” slots by selecting the correct image from the shuffled Candidate Image Library on the right. The annotator must follow the sequence of actions, using the result of the previous action as the starting point for the next, to determine the correct chronological order of all future states.
Refer to caption
Figure 14: The annotation interface used for evaluating human performance on Inverse World Modeling problems. Annotators are shown an ordered sequence of state transitions, displayed as pairs of “Current State” and “Next State” images. For each transition, their task is to select the correct action description from a shuffled Candidate Action Library that caused the visual change between the two states.

B.1.2 Inter-Annotator Agreement Analysis

Five items per annotator, direction, and length create 240 questions, each subsequently labeled by all three annotators. Ordinal Krippendorff α (Krippendorff, 2011) is 0.8320, with a 1,000-bootstrap 95% interval of [0.7879, 0.8682]. Exact three-way agreement occurs on 184/240 questions (76.67%); pairwise α ranges from 0.8180 to 0.8518. Agreement supports consistency of this annotation protocol, not exhaustive validity of the cognitive interpretation.

Equation
α=1−DoDe\alpha=1-\frac{D_{o}}{D_{e}}

B.2 World Modeling as a Proxy for Evaluating Embodied Cognition

Full tables separate all-or-nothing acceptance from partial transition consistency and include contact, action-encoding, and verifier-relaxation analyses.

Task Action–Effect Reasoning Causal Inference Affordance Recognition Embodied Awareness Temporal Abstraction
Forward ✓\checkmark ×\times ✓\checkmark ✓\checkmark ✓\checkmark
Inverse ×\times ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark
Table 5: We posit that ENACT reflects embodied world modeling reasoning rather than simple temporal correlation. Unlike passive video prediction, our formulation explicitly conditions state transitions on actions (Forward World Modeling) and infers actions from state changes (Inverse World Modeling), thereby evaluating VLMs as transition models. Based on your suggestion, we include a table that relates ENACT’s evaluation tasks to specific cognitive constructs.

B.2.1 Experimental Setup

The protocol uses one template per task, 512×512 images, temperature zero, and a Python-list permutation response without explanation. The original model-version table and complete forward/inverse prompts are retained. A fixed prompt limits tuning differences, but temperature zero alone does not guarantee reproducibility across API versions or serving systems. The cognitive-capability table states the authors’ proposed interpretation of each task.

Organization Model Name Release Date Full Name
Evaluation Pipeline
Proprietary Models
OpenAI GPT-5 2025-08 gpt-5-2025-08-07 OpenAI API
GPT-5-mini 2025-08 gpt-5-mini-2025-08-07 OpenAI API
GPT-5-nano 2025-08 gpt-5-nano-2025-08-07 OpenAI API
Google Gemini 2.5 Pro 2025-06 gemini-2.5-pro Gemini API
Gemini 2.5 Flash 2025-06 gemini-2.5-flash Gemini API
Gemini 2.5 Flash-Lite 2025-06 gemini-2.5-flash-lite Gemini API
Anthropic Claude Sonnet 4 2025-05 claude-sonnet-4-20250514 Anthropic API
Open-Weight Models
Zhipu AI GLM-4.5V 2025-08 GLM-4.5V Zhipu Foundation Model Open Platform API
GLM-4.1V-Thinking 2025-07 GLM-4.1V-Thinking-FlashX Zhipu Foundation Model Open Platform API
Meta Llama-4-Scout-17B-16E-Ins 2025-04 meta-llama/Llama-4-Scout-17B-16E-Instruct ModelScope API
Llama-4-Mav-17B-128E-Ins 2025-04 meta-llama/Llama-4-Mav-17B-128E-Instruct ModelScope API
Shanghai AI Lab InternVL3.5-241B-A28B 2025-08 OpenGVLab/InternVL3.5-241B-A28B Intern API
InternVL3.5-14B 2025-08 OpenGVLab/InternVL3.5-14B Hugging Face Transformers
InternVL3.5-8B 2025-08 OpenGVLab/InternVL3.5-8B Hugging Face Transformers
InternVL3.5-4B 2025-08 OpenGVLab/InternVL3.5-4B Hugging Face Transformers
Google Gemma-3-27b-it 2025-03 google/gemma-3-27b-it Gemini API
Gemma-3-12b-it 2025-03 google/gemma-3-12b-it Gemini API
Gemma-3-4b-it 2025-03 google/gemma-3-4b-it Gemini API
Alibaba QVQ-72B-Preview 2024-12 Qwen/QVQ-72B-Preview ModelScope API
Qwen2.5-VL-72B-Ins 2025-01 Qwen/Qwen2.5-VL-72B-Instruct ModelScope API
Qwen2.5-VL-32B-Ins 2025-01 Qwen/Qwen2.5-VL-32B-Instruct ModelScope API
Qwen2.5-VL-7B-Ins 2025-01 Qwen/Qwen2.5-VL-7B-Instruct Hugging Face Transformers
Qwen2.5-VL-3B-Ins 2025-01 Qwen/Qwen2.5-VL-3B-Instruct Hugging Face Transformers
AIDC Ovis2.5-9B 2025-08 AIDC-AI/Ovis2.5-9B Hugging Face Transformers
Ovis2.5-2B 2025-08 AIDC-AI/Ovis2.5-2B Hugging Face Transformers
OpenBMB MiniCPM-V-4.5 2025-08 openbmb/MiniCPM-V-4.5 Hugging Face Transformers
MiniCPM-o-2.6 2025-01 openbmb/MiniCPM-o-2.6 Hugging Face Transformers
Hugging Face Idefics3-8B-Llama3 2024-08 HuggingFaceM4/Idefics3-8B-Llama3 Hugging Face Transformers
Nvidia Cosmos-Reason1 2025-05 nvidia/Cosmos-Reason1 Hugging Face Transformers
Table 6: Details of Vision Language Models (VLMs) assessed in this study.
Original figure 15, retaining native vector text and original embedded pixels.
Original figure 15, retaining native vector text and original embedded pixels.
Figure 15: The prompt used to evaluate VLMs on the multi-step Forward World Modeling task. The model must order shuffled future state images by reasoning over a given action sequence.
Original figure 16, retaining native vector text and original embedded pixels.
Figure 16: The prompt used to evaluate VLMs on the multi-step Inverse World Modeling task. The model must order a set of shuffled actions by reasoning over an ordered sequence of state images.

B.2.2 Detailed Results

The complete task and pairwise tables show how partial local consistency can survive when complete ordering collapses. Human length-10 task accuracy remains 84–85%, while the strongest model task scores are much lower. Preserve both metrics when comparing models; a single average would conceal horizon dependence and verifier behavior.

Original table 7, preserving every value, label, and highlight.
Table 7: Evaluation on ENACT (Task Accuracy). Dark gray indicates the best result within each category (Proprietary or Open-Weight Models), and Light gray denotes the second-best result within the category.
Original table 8, preserving every value, label, and highlight.
Table 8: Evaluation on ENACT (Pairwise Accuracy). Dark gray indicates the best result within each category (Proprietary or Open-Weight Models), and Light gray denotes the second-best result within the category.

B.2.3 Contact Experiment

Adding binary contact changes to key-frame selection preserves the broad horizon degradation and inverse advantage for InternVL3.5-241B. At three frames, task accuracy is 86.67% forward and 90.00% inverse; at seven, 3.45% and 16.67%. This checks one alternative sampling criterion, without establishing robustness to arbitrary physical predicates or all models.

Metric Forward (with contact changes) Inverse (with contact changes)
3 4 5 6 7 8 9 10 3 4 5 6 7 8 9 10
Task Accuracy 86.67 43.33 36.67 20.00 3.45 3.33 0.00 0.00 90.00 73.33 30.00 26.67 16.67 3.33 6.67 0.00
Pairwise Accuracy 90.00 72.22 60.00 53.33 48.28 42.38 31.67 34.44 90.00 82.22 55.83 57.33 46.11 32.86 38.75 27.78
Table 9: Effect of including contact changes in key-frame selection. Task and pairwise accuracies (%) of InternVL3.5–241B when key frames are triggered by both state changes and contact changes.

B.2.4 How Does Action Representation Affect VLMs’ Performance?

On 2,304 questions, natural-language, structured-symbolic, and emoji-style action encodings retain broadly similar direction and horizon trends for InternVL. The inverse advantage therefore is not solely dependent on familiar natural-language verbs. Absolute scores vary and some individual cells tie or reverse; changing representation does not isolate every linguistic or verifier-related explanation.

Metric Forward Inverse
3 4 5 6 7 8 9 10 3 4 5 6 7 8 9 10
Vanilla (Natural Language)
Task Accuracy 68.97 35.17 27.59 8.97 6.90 2.07 0.69 0.00 83.45 60.69 44.14 24.83 13.79 7.59 4.14 0.00
Pairwise Accuracy 76.21 57.70 57.41 41.24 38.85 30.34 29.05 26.52 83.45 69.66 61.72 53.10 47.01 40.39 34.40 27.25
Symbolic Predicates
Task Accuracy 67.59 40.00 20.00 9.66 4.14 0.69 1.38 0.00 79.86 51.03 40.00 23.45 8.97 5.52 2.76 0.73
Pairwise Accuracy 74.48 61.61 48.97 41.52 35.75 32.12 30.34 25.68 79.86 62.99 58.62 51.59 39.43 36.85 32.07 24.33
Emoji-Style Encodings
Task Accuracy 65.52 44.14 18.62 11.03 8.28 0.69 0.69 0.00 77.24 48.28 35.17 23.45 12.41 7.59 2.76 0.00
Pairwise Accuracy 73.10 64.60 48.62 39.45 38.05 27.49 26.72 23.93 77.93 61.38 59.14 45.38 40.80 36.06 27.16 25.30
Table 10: Effect of action representation on InternVL3.5–241B. Task and pairwise accuracies (%) on a 2,304-QA subset of ENACT under three action encodings.

B.2.5 How Often Do Accepted Predictions Omit Parts of Transitions?

Among semantically accepted answers, reported data-level strict-subset mismatch is 1.65% for GPT-5, 1.49% for InternVL, and 2.26% for humans. Pair-level mismatch is larger for models: 15.31% and 13.96%, versus 3.62%. The section describes subset acceptance generically, but the formal forward rule uses the opposite inclusion direction. These omission statistics should therefore be read with the precise task-specific verifier in mind.

Model Data-Level Mismatch (%) Pair-Level Mismatch (%)
GPT-5 1.65 15.31
Human 2.26 3.62
InternVL3.5-241B 1.49 13.96
Table 11: Mismatch rates among semantically accepted predictions. Data-level and pair-level mismatch ratios (%) for cases where the semantic verifier accepts a prediction but its predicate set is a strict subset of the ground-truth transition.

B.3 Additional Ablation Experiments Common Setup

The common ablation set contains 50 questions per direction at lengths 3, 6, and 9: 300 total. GPT-5 mini and InternVL3.5-241B-A28B are evaluated with two-tailed unpaired Welch tests and a p<0.05 threshold. Questions reuse matched trajectories across variants; independent API sessions do not alone establish statistical independence of their difficulty. No multiple-comparison correction is described. Nonsignificance is not an equivalence test.

Original figure 17, retaining native vector text and original embedded pixels.
Figure 17: Illustrative trajectories of Forward World Modeling and Inverse World Modeling for a representative baseline question.
Original figure 18, retaining native vector text and original embedded pixels.
Figure 18: Ablation experiment results with InternVL3.5-241B-A28B on ENACT. Heatmaps show two-tailed unpaired p-values against the baseline, using Pairwise Accuracy. p<0.05p<0.05 is considered significant. Darker red means more significant. Δ\Delta is the performance change from the baseline. If significant and Δ<0\Delta<0, the setting is worse than the baseline. C.2 reports the robot’s performance on the left- and right-hand predicates, where Mixing is the proportion of ground truth left or right cases that are predicted as the other hand (i.e., mixing one hand into the other hand). Note that, although InternVL3.5-241B-A28B performance is less significant than GPT-5 mini, the |Δ||\Delta| across unnatural camera configurations still remains high (>0.05>0.05) when the same settings are significant for GPT-5 mini.

B.4 Sensitivity to Image Realism

Rendering studies compare the baseline with generated photographic styling and two simulator-renderer variants. They probe image appearance while attempting to hold the interaction fixed, rather than comparing matched real and simulated executions.

Original figure
Figure 19: Comparison between Cosmos-Reason1 and other similar-sized models.

B.4.1 Realistic: Generated Images as Real World Proxy

GPT-image-1 restyles simulator frames with instructions to preserve objects, geometry, camera, and actions. The complete prompt specifies lighting, materials, and fidelity targets. Those requested thresholds are prompt text, not evidence that every generated image was measured to satisfy them. Generated photographs remain synthetic, and imperfect action or gripper preservation can confound a realism comparison.

Original figure
(a)
Original figure
(b)
Figure 20: Ablating image realism with GPT-5 and InternVL3.5-241B-A28B. (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across four rendering settings—Realistic, Path Tracing, Ray Tracing (Baseline), and Ray Tracing Only. Error bars denote ±SEM. The baseline x-tick is bolded.
Original figure 21, retaining native vector text and original embedded pixels.
Figure 21: Examples of simulator frames converted into realistic styles for both Forward World Modeling (left) and Inverse World Modeling (right) trajectories.
Original figure 22, retaining native vector text and original embedded pixels.
Figure 22: The prompt used to generate realistic photographic style images from segmented frames (of a replayed robot trajectory).

B.4.2 Path Tracing Setup

The higher-fidelity variant uses Isaac Sim’s path-tracing renderer. Its retained example shows the same action sequence under different lighting and rendering.

Original figure 23, retaining native vector text and original embedded pixels.
Figure 23: The figure illustrates Forward World Modeling (left) and Inverse World Modeling (right) trajectories rendered using the path tracing engine in NVIDIA Isaac Sim.

B.4.3 Ray Tracing Only Setup

The simplified ray-tracing variant disables reflections, DLSS, ambient occlusion, sampled lighting, ambient light, and flow. The example retains the interaction sequence while changing visual appearance. It is a particular rendering intervention, not a general ordering of all possible image realism.

Original figure 24, retaining native vector text and original embedded pixels.
Figure 24: Examples of an intermediate rendering style created with a simplified ray tracing pipeline for Forward World Modeling (left) and Inverse World Modeling (right) trajectories.
Metric 3 4 5 6 7 8 9 10
Forward (with random perturbations)
Task Accuracy 68.51±1.4468.51\pm 1.44 37.47±2.1137.47\pm 2.11 25.75±1.7425.75\pm 1.74 10.80±1.5910.80\pm 1.59 4.60±2.114.60\pm 2.11 1.61±0.801.61\pm 0.80 0.92±0.400.92\pm 0.40 0.00±0.000.00\pm 0.00
Pairwise Accuracy 75.52±1.1975.52\pm 1.19 59.77±1.8259.77\pm 1.82 54.48±2.5454.48\pm 2.54 42.25±1.3042.25\pm 1.30 37.36±1.3037.36\pm 1.30 30.90±0.5930.90\pm 0.59 28.76±0.4328.76\pm 0.43 25.98±1.0125.98\pm 1.01
Inverse (with random perturbations)
Task Accuracy 83.45±0.6983.45\pm 0.69 58.39±3.4058.39\pm 3.40 43.91±1.0543.91\pm 1.05 22.76±2.0722.76\pm 2.07 13.79±0.0013.79\pm 0.00 6.67±2.226.67\pm 2.22 4.14±2.074.14\pm 2.07 0.73±1.260.73\pm 1.26
Pairwise Accuracy 83.45±0.6983.45\pm 0.69 68.74±1.4068.74\pm 1.40 61.49±0.7261.49\pm 0.72 52.46±0.6952.46\pm 0.69 45.59±1.2345.59\pm 1.23 39.70±1.6439.70\pm 1.64 35.09±2.5735.09\pm 2.57 29.01±1.6429.01\pm 1.64
Table 12: Robustness to random perturbations in predicate deltas. Mean ±\pm standard deviation (%) over three random seeds on a 2,304-QA subset of ENACT for InternVL3.5–241B.

B.4.4 Robustness to Noise in Predicate Deltas

Random perturbations of predicate deltas on a 2,304-question InternVL subset yield small reported standard deviations over three seeds and preserve broad trends. The section does not specify a complete perturbation distribution or rate, limiting the breadth of the robustness claim. Small seed variation does not establish robustness to systematic annotation errors.

B.5 Sensitivity to Camera Configurations

Camera experiments change aperture, projection, or height. Original example frames and result plots are retained so visibility changes can be inspected alongside accuracy.

B.5.1 Camera Aperture Setup

The baseline aperture setting is 40, compared with 30, 60, and 80. These are simulator camera parameters rather than explicitly reported field-of-view angles.

Original figure 25, retaining native vector text and original embedded pixels.
Figure 25: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with a camera aperture of 30.
Original figure 26, retaining native vector text and original embedded pixels.
Figure 26: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with a camera aperture of 60.
Original figure 27, retaining native vector text and original embedded pixels.
Figure 27: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with a camera aperture of 80.
Original figure
(a)
Original figure
(b)
Figure 28: Ablating camera field-of-view (FOV) with GPT-5 and InternVL-3.5-241B-A28B. (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across five lens settings—Aperture 30, Aperture 40 (Baseline), Aperture 60, Aperture 80, and Fisheye. Error bars denote ±SEM; the baseline tick is bolded.

B.5.2 Fisheye Lens Setup

The fisheye variant uses Isaac Sim’s fisheyePolynomial setting. Its distorted projection changes the mapping between image position and scene geometry; the source compares it with everyday wide-angle cameras.

Original figure 29, retaining native vector text and original embedded pixels.
Figure 29: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with a fisheye-style camera.

B.5.3 Camera Height Setup

Camera height varies from the 1.75 m baseline to 2.25 m and 1.5 m. The lower offset is limited because stronger lowering hides relevant objects. This selection itself constrains the range of embodiments tested.

Original figure 30, retaining native vector text and original embedded pixels.
Figure 30: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured from a camera height of 2.25 m.
Original figure 31, retaining native vector text and original embedded pixels.
Figure 31: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured from a camera height of 1.5 m.
Original figure
(a)
Original figure
(b)
Figure 32: Ablating camera height with GPT-5 and InternVL-3.5-241B-A28B. (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across three viewpoints (High, Mid baseline, and Low). Error bars denote ±\pmSEM; the baseline tick is bolded.

B.6 Do VLMs Have Embodied Biases?

Appearance interventions and predicate-level handedness analysis address different questions: visual robustness versus asymmetric interpretation of left/right interactions.

B.6.1 Robot Appearance

The robot is rendered white, with a new random color at every frame, or skin-toned. These changes test robustness to surface appearance while retaining body geometry and the underlying action sequence.

Original figure 33, retaining native vector text and original embedded pixels.
Figure 33: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in white.
Original figure 34, retaining native vector text and original embedded pixels.
Figure 34: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in a random color at each frame.
Original figure 35, retaining native vector text and original embedded pixels.
Figure 35: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in a human skin–like color.
Original figure 36, preserving every value, label, and highlight.
Figure 36: Ablating robot appearance with GPT-5 and InternVL-3.5-241B-A28B.(a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across four styles—White Color, Fancy (Baseline), Random Color, and Skin Color. Error bars denote ±SEM; the baseline tick is bolded.

B.6.2 Handedness

Across the displayed models and humans, right-hand precision and recall are higher, and left-to-right confusion exceeds right-to-left confusion. These component-level measurements are derived from inferred transitions, with standard-error bars. Shared dataset and annotation asymmetries remain possible contributors.

Original figure
(a)
Original figure
(b)
Figure 37: Illustration of overlaps between ground truth and GPT-5 predictions sets for left and right hands related signatures in forward and inverse tasks. The size of ellipses project the total counts of signatures, and overlaps denote matched signatures (center regions) or mixing errors (cross-hand overlaps).
Original figure
(a)
Original figure
(b)
Figure 38: Precision of left/right hand related components prediction in (a) forward and (b) inverse tasks, with models Gemini2.5Pro, GPT-5, GPT-5 mini, InternVL3.5-241B-A28B, and Human. Error bars indicate the standard error (SE).
Original figure
(a)
Original figure
(b)
Figure 39: Recall of left/right hand related components prediction in forward task, with models Gemini2.5Pro, GPT-5, GPT-5 mini, InternVL3.5-241B-A28B and Human. Error bars indicate the standard error (SE).
Original figure
(a)
Original figure
(b)
Figure 40: Hand-mixing rate, i.e.the ratio of left/right hand-mixing to all ground truth left/right and components in (a) forward and (b) inverse task, with models Gemini2.5Pro, GPT-5, GPT-5 mini, InternVL3.5-241B-A28B and Human. Error bars indicate the standard error (SE).

Appendix C Error Analysis

The error appendix defines structural and semantic categories, provides original pseudocode, and illustrates each category with selected examples. These examples explain the taxonomy rather than estimate prevalence on their own.

C.1 Methodology for Error Calculations

Action signatures turn natural-language descriptions into finite sets of atomic changes. Comparing those sets supports both structural error classification and semantic labeling.

Signature Modeling from Scene Graph-level Differences

An atomic component records an edge operation between two entities, a node-property operation, or a transition from one property to another. The original display preserves the tuple definitions. This representation ties evaluation to the chosen symbolic ontology.

Equation
ci::=(γ,e1,ρ,e2)∣(γ,e,ρ)∣(transition,e,ρfrom→ρto),γ∈{add,remove}c_{i}::=(\gamma,e_{1},\rho,e_{2})\;\mid\;(\gamma,e,\rho)\;\mid\;(\mathrm{transition},e,\rho_{\mathrm{from}}\rightarrow\rho_{\mathrm{to}}),\quad\gamma\in\{\mathrm{add},\mathrm{remove}\}

Error Modeling from Signatures

The pipeline separates matched, missing, and extra components, then detects polarity, predicate, and entity substitutions before assigning residual omissions and hallucinations. A predefined predicate mapping supplies spatial, functional, material, or agent-interaction labels. Algorithm 2 explicitly filters unequal-length signature sequences before this analysis, so its error population need not equal every failed benchmark answer. The complete algorithms remain available for inspecting these choices.

Handedness Asymmetry Error Modeling

Precision divides matched hand components by predicted components; recall divides them by reference components. A confusion event requires missing components for one hand and hallucinated components for the opposite hand, without a same-hand hallucinated component. The procedure counts these symbolic substitutions, not physical hand movements or a demonstrated motor preference.

C.2 Structural Error Analysis

The plots distinguish total error counts from within-error proportions. Humans produce far fewer errors, even when their relative category mix resembles a model’s. The examples show wrong objects, wrong predicates, reversed changes, omitted changes, and extra changes. They should not be interpreted as unconstrained text hallucinations because predictions select from supplied candidates.

Original figure
(a)
Original figure
(b)
Figure 41: The amount of total errors made by Gemini2.5Pro, GPT-5, GPT-5 mini, InternVL3.5-241B-A28B, and Human, under (a) forward tasks and (b) inverse tasks.
Original figure
Figure 42: The structural error distributions of typical LLMs (GPT-5, GPT-5 mini, Gemini2.5Pro and InternVL3.5-241B-A28B (referred as InternVL3.5… in figure)) and Human-level prediction in both forward and inverse tasks.
Original figure 43, retaining native vector text and original embedded pixels.
Figure 43: Example of structural error Entity Substitution by GPT-5 under forward and inverse tasks.
Original figure 44, retaining native vector text and original embedded pixels.
Figure 44: Example of structural error Predicate Substitution by GPT-5 under forward and inverse tasks.
Original figure 45, retaining native vector text and original embedded pixels.
Figure 45: Example of structural error Polarity Inversion by GPT-5 under forward and inverse tasks.
Original figure 46, retaining native vector text and original embedded pixels.
Figure 46: Example of structural error Omission by GPT-5 under forward and inverse tasks.
Original figure 47, retaining native vector text and original embedded pixels.
Figure 47: Example of structural error Hallucination by GPT-5 under forward and inverse tasks.

C.3 Semantic Error Analysis

Spatial relations and agent interactions dominate the semantic error distributions. Forward tasks contain relatively more spatial-relation errors, while inverse tasks contain more agent-interaction errors. Original examples cover spatial, functional, material, and grasp-related mistakes for both task directions.

Original figure
Figure 48: The semantic error distributions of typical LLMs (GPT-5, GPT-5 mini, Gemini2.5Pro and InternVL3.5-241B-A28B (referred as InternVL3.5… in figure)) and Human-level prediction in both forward and inverse tasks.
Original figure 49, retaining native vector text and original embedded pixels.
Figure 49: Example of semantic error Spatial Relations by GPT-5 under forward and inverse tasks.
Original figure 50, retaining native vector text and original embedded pixels.
Figure 50: Example of semantic error Functional States by GPT-5 under forward and inverse tasks.
Original figure 51, retaining native vector text and original embedded pixels.
Figure 51: Example of semantic error Material States by GPT-5 under forward and inverse tasks.
Original figure 52, retaining native vector text and original embedded pixels.
Figure 52: Example of semantic error Agent Interactions by GPT-5 under forward and inverse tasks.
Algorithm 2 Action-level Parsing of Signatures Data
Original algorithm 2, retaining native vector text and original embedded pixels.
Algorithm 3 Action-level Structural and Semantic Error Categorization
Original algorithm 3, retaining native vector text and original embedded pixels.
Algorithm 4 Dataset-Level Detection of Left–Right Hand Confusion
Original algorithm 4, retaining native vector text and original embedded pixels.

Appendix D The Use of Large Language Models

The authors disclose using Gemini 2.5 Pro and GPT-5 for writing assistance and related-work retrieval, with human revision, fact-checking, and responsibility for the manuscript.

Concept demonstration

Open image ↗