Literature
Display
← All papers

Embodied First Exam

Kangrui Wang · Qineng Wang · Changhe Chen · Pingyue Zhang · Chi Gui · Feishi Wang · Canyu Chen · Wenli Xiao · Xiaoyu Huang · Tonghe Zhang · Linjie Li · Zihan Wang · Minyeong Kim · Yiyou Sun · Mingkai Deng · Jinyu Hou · Qi Zhu · Han Liu · Eric P. Xing · Huaxiu Yao · Zhiwen Fan · Yuejiang Liu · Dawn Song · Pieter Abbeel · Li Fei-Fei · Jiajun Wu · Yuke Zhu · Guanya Shi · Ruohan Zhang · Manling Li

Preprint posted · arXiv (preprint) ↗

AI agents35 min text · CondensedOriginal paper ↗
Contents

Original overview

A large, agent-native robotics exam spanning direct actions, generated controllers and policy training. The original overview, omitted from the source HTML, is restored from the PDF; figure labels follow the PDF throughout this edition. Every original figure and table is retained alongside edited section notes.

Figure 1: Overview of the Embodied First Exam, a comprehensive evaluation framework for coding agents on robotics tasks. It is large and diverse, spanning 1,156 tasks, 35 suites and 47 robots, and easy to run and evaluate: one command runs any coding agent, and each task can be graded in up to three evaluation settings.
Figure 1: Overview of the Embodied First Exam, a comprehensive evaluation framework for coding agents on robotics tasks. It is large and diverse, spanning 1,156 tasks, 35 suites and 47 robots, and easy to run and evaluate: one command runs any coding agent, and each task can be graded in up to three evaluation settings.

At a glance

Embodied First Exam evaluates general-purpose coding agents on 1,156 simulated tasks in 35 suites using 47 robots. Agents may act directly, write controllers or train policies. Standard access provides robot sensors and usually no reset; Realtime also keeps the world running; Privileged supplies simulator state and unlimited resets. On the selected 456-task hard core, the best tested agent solves 33.6% of Standard tests. One attempt per agent/test and agent-generated failure labels limit statistical and causal conclusions.

1 Introduction

Robot competence is evaluated as the whole engineering loop: read an interface, select a control approach, execute it and revise from feedback. Paired settings separate success with simulator access and retries from success through sensors under limited recovery. On 391 tasks with both settings, the leading agent solves about 77% Privileged versus 30% Standard. Stage graphs localize failures; controlled runs suggest resets help more than exact object state, particularly at precise-contact stages. These findings concern the reported agents and selected simulated tasks, not physical deployment readiness.

Original research question.
Original research question.
Figure 2: Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs mark the approaches used (model is direct control), and the framed line is a policy.
Figure 2 : Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs mark the approaches used ( model is direct control), and the framed line is a policy.

2 The Embodied First Exam: A Unified Evaluation Framework for Coding Agents in Robotics

The framework specifies task interfaces, isolated verification, resource budgets and suite coverage while leaving control strategy to the agent. A test is one task in one evaluation setting.

Figure 3: The EFE framework and evaluation protocol. (a) Our unified framework supports different models, harnesses, control approaches, and simulators. (b) In Privileged, the agent and simulator share a container, and the verifier replays the submitted answer separately. In Standard and Realtime, the simulator and verifier are separated from the agent and grade the live episode.
Figure 3 : The EFE framework and evaluation protocol. (a) Our unified framework supports different models, harnesses, control approaches, and simulators. (b) In Privileged , the agent and simulator share a container, and the verifier replays the submitted answer separately. In Standard and Realtime , the simulator and verifier are separated from the agent and grade the live episode.

2.1 Framework and Evaluation Protocol

Tasks use Harbor containers: the agent can install packages, write programs and train models, while an inaccessible verifier grades the outcome. Privileged shares the simulator with the agent and replays its submitted trajectory/controller separately. Standard exposes sensor/proprioceptive queries and actions through a separate service, pauses between requests and usually forbids resets. Realtime keeps simulation advancing, sometimes at a fixed fraction of real time. Paired tests share scene, success rule and time limit. Privileged is an approximate aggregate upper benchmark, not a per-task mathematical bound.

2.2 Resource Constraints for Coding Agents

Each task caps wall time, tokens, compute, memory and disk, with optional robot-step limits. No attempt exhausts its token budget; reaching wall time ends the run and grades the current result. Privileged resets are unlimited; nine suites allow 3–50 explicit resets in Standard/Realtime. Resources can change which control strategies are feasible, so benchmark results describe the complete model–harness–resource setting.

Original figure
Figure 4: Resource constraints across the exam. The large number and the tick give the median over tests (for GPU memory, over the tests that reserve a GPU), the line spans the range, and the dark box spans the middle half of the tests.

2.3 Task Suites

Nine new suites contribute 411 tasks; 26 adapted suites contribute 745. New tasks emphasize contact-rich dexterity, dynamics/opponents, image goals, safety and hidden-dynamics engineering. Adapted tasks preserve original scenes/robots and success rules, sometimes strengthening criteria, while removing demonstrations and expert solutions. The 456-task core has 901 tests and excludes judged-easy or GPT-6 Luna-solved tasks except author-requested inclusions; this screening may bias against the related Astra model family. Defaults are 403 Standard and 53 Realtime tasks; 427 core tasks have stage graphs. The other 700 tasks remain available but are outside the main evaluation.

Figure 5: The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks, in the three groups of Section 2.3, and a frame from each of the 26 adapted suites, ordered by their main robot embodiment, from single arms to aerial robots.
Figure 5 : The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks,
in the three groups of Section 2.3 , and a frame from each of the 26 adapted suites, ordered by their
main robot embodiment, from single arms to aerial robots.
Figure 6: The core tasks by the stage categories of their task graphs, with an example task of each category. Each seat is one task in one category, and the angle of a category is proportional to its number of tasks. A task counts once in each category its task graph contains. The centre counts all 456 core tasks, and the seats cover the 427 with a task graph; no task graph has a Safety or Information stage, since the graphs of Embodied Safety Exam give its safety rules as conditions of their stages (Section H.1).
Figure 6 : The core tasks by the stage categories of their task graphs, with an example task of each category. Each seat is one task in one category, and the angle of a category is proportional to its number of tasks. A task counts once in each category its task graph contains. The centre counts all 456 core tasks, and the seats cover the 427 with a task graph; no task graph has a Safety or Information stage, since the graphs of Embodied Safety Exam give its safety rules as conditions of their stages ( Section H.1 ).

3 Analysis

Eight model–harness combinations run at high reasoning effort, once per core test. Verifier pass/fail is distinct from the subsequent interpretation of why an attempt failed. Harness configuration is part of the measured agent.

Stage-level failure labels.

Task graphs specify required world-state changes and precedence. Each failed attempt receives at most one failure stage: an irreversible event or the earliest uncorrected mistake, which can occur before prerequisites. Up to two observable failure phenomena supplement the label. A stage is reached when its condition is met or failure occurs there. GPT-6.1 Sol labels most attempts, Claude Sonnet 5.5 the remainder and reviews every file; these are model-assisted annotations, not independent human ground truth.

Original figure
(a)
Original figure
(b)
Figure 7: Task graph and progress maps of carrying in groceries (BEHAVIOR-1K). Panel (a) gives the stages, the order in which they must be met, and the type and contact of each stage. Panel (b) gives the share of the stages whose conditions hold at each moment of the graded episode, for three attempts. Red lines mark failure stages and their failure phenomena (Section H.6).

3.1 Overall Results

The strongest three agents solve 24.1–33.6% of 456 Standard tests and collectively cover 48%. On the same 54 tasks, Astra solves 23 Standard versus 17 Realtime tests, Opus 10 versus 4, and Argon 15 versus 5. No agent solves 63% of Realtime tests, versus 50% of their Standard counterparts. At least half the Standard tests remain unsolved by the strongest three in 14 of 22 suites with five or more tests. Single-run outcomes do not establish repeatability or confidence intervals.

Original figure
Figure 8: Success rate of each agent on the core set, with solved tests in parentheses. The harness dsh is DeepSeek Harness, DeepSeek’s own harness.
Original figure
Figure 9: Every attempt on the core Privileged and Standard tests, with the agents in the order of Figure 8.

3.2 What Causes the Drop from the Privileged to the Standard Setting?

On 391 paired tasks, Astra solves 303 Privileged tests and 116 Standard tests; 191 Privileged successes fail Standard. Separate controlled runs raise Standard success from 28% to 45% with exact state, 53% with resets/saved states and 68% with both, versus 77% Privileged. The source controlled comparison uses task defaults (including Realtime) and reports 28%, versus the paired Standard leaderboard’s roughly 30%; these are retained as distinct reported settings. Precise stages show larger setting gaps than coarse stages. Relative to a second-run control hiding success feedback, state alone resolves about two additional precise-stage failures among 91, while resets resolve about 16; exact state helps more at coarse stages. These observations support retries/feedback as important, without isolating every remaining Privileged difference.

Original Key Takeaways: Privileged versus Standard.
Original Key Takeaways: Privileged versus Standard.
Original figure
Figure 10: Success rates by suite, for suites with at least five core tests in the setting. Dots give the success rate of each of the three strongest agents, and bars the share of tests that none of them solves. The IKEA assembly tasks of RoboARC count as their own suite.
Original figure
Figure 11: The 391 core tasks that have both a Privileged and a Standard test, split for each agent by the settings in which it solves them.
Original figure
(a)
Original figure
(b)
Original figure
(c)
Figure 12: (a) Success rate of GPT-6 Astra in Codex on the 391 core tasks that have both settings, as features of the Privileged Setting are added to the Standard interface. (b) Failure rate of each stage type compared in at least two suites, eight agents. (c) Share of GPT-6 Astra’s failures at precise and coarse stages that each run solves.

3.3 At Which Stage of a Task Do Agents Fail?

Different agents’ located failures concentrate at the same stage: 61% pair agreement in Standard versus a 37% position-matched random baseline, or 56% versus 31% for the strongest three. The modal stage is first in 65% of tests with a unique mode. After excluding tasks whose mode is first in both settings, 47/83 share the same modal stage versus about 9% under the null. Pooled reached-stage failure is 66% for material changes, 45% for body motion and 20% for mechanisms; robot/suite composition confounds causal interpretations of these category rates.

Original Key Takeaways: Where Agents Fail.
Original Key Takeaways: Where Agents Fail.
Figure 13: Claude Opus 5.5 installing the front guard of a fan (DexHandExam) in the Privileged and the Standard Setting. Red marks where the Standard attempt goes wrong. Times are seconds on the replay of the Standard episode.
Figure 13 : Claude Opus 5.5 installing the front guard of a fan (DexHandExam) in the Privileged and the Standard Setting. Red marks where the Standard attempt goes wrong. Times are seconds on the replay of the Standard episode.
Figure 14: (a) Concentration of failure stages, mean over tests, and share of tasks with the same most common failure stage in both settings, without the tasks where it is the first stage in both. Gray bars: random failure stages; black lines: 95% intervals. (b) Failure rate of a stage once reached, by category, both settings pooled (Table 18).
Figure 14 : (a) Concentration of failure stages, mean over tests, and share of tasks with the same most common failure stage in both settings, without the tasks where it is the first stage in both. Gray bars: random failure stages; black lines: 95% intervals. (b) Failure rate of a stage once reached, by category, both settings pooled ( Table 18 ).
Figure 15: Catching a ball twice (RoboDynamicExam): the stages that the attempt of each agent reached and the stage at which it failed, in each setting, and Standard replay frames at three failures at stage 2.
Figure 15 : Catching a ball twice (RoboDynamicExam): the stages that the attempt of each agent reached and the stage at which it failed, in each setting, and Standard replay frames at three failures at stage 2.

3.4 How Do Agents Differ?

Weaker agents more often fail before required contact or before completing any stage. Strong agents’ Standard successes overlap incompletely: Opus and Argon collectively solve 64 tests Astra fails, whereas Privileged overlap is much greater. One run per test means some complementarity may reflect stochastic variation. Harness changes matter: Astra scores 34% in Codex versus 22% in Claude Code; Argon scores 29%, 25% and 24% in Codex, Claude Code and Antigravity CLI. Cross-provider adapters and session termination after content filtering complicate attributing these differences solely to the harness.

Original Key Takeaways: Differences between Agents.
Original Key Takeaways: Differences between Agents.
Figure 16: (a) Failure profile of each agent; zero progress means that the attempt reached no stage. (b) Share of the row agent’s solved Standard tests (first column) that the column agent also solves; the last column uses any agent above it in Figure 8. (c) Success rate by model and harness on the 456 core tests of the Standard Setting; hatched bars mark other harnesses.
Figure 16 : (a) Failure profile of each agent; zero progress means that the attempt reached no stage. (b) Share of the row agent’s solved Standard tests (first column) that the column agent also solves; the last column uses any agent above it in Figure 8 . (c) Success rate by model and harness on the 456 core tests of the Standard Setting; hatched bars mark other harnesses.
Figure 17: Preparing a lunch box (BEHAVIOR-1K, Standard Setting): a head-camera frame of four failed attempts, with the failure circled.
Figure 17 : Preparing a lunch box (BEHAVIOR-1K, Standard Setting): a head-camera frame of four failed attempts, with the failure circled.

3.5 Which Control Approach Do Agents Use?

In a paired soccer example, the GPU version also adds training libraries, parallel simulator copies and a training example. Ten of twelve Standard and seven of eight Realtime attempts train policies; all three Standard wins and the one Realtime win use trained policies. No CPU-version attempt wins those settings. This bundled, single-task comparison cannot isolate GPU availability as the cause. Across 1,489 labeled sampled attempts, 61% of Standard attempts act directly, 82% of Privileged attempts use code, and 97% of Realtime attempts use code; only six sampled attempts train policies. The soccer-specific counts include extra harness runs.

Original Key Takeaways: How Agents Control the Robot.
Original Key Takeaways: How Agents Control the Robot.
Original figure
(a)
Original figure
(b)
Figure 18: (a) Share of each approach by setting in 1,489 sampled attempts. Mixed attempts switched approach midway or used two approaches equally. (b) Soccer attempts in the Standard and Realtime settings that trained a policy, scored or won.
Figure 19: GPT-6 Astra in Codex on the two soccer versions in the Standard Setting. Each row gives the approach and result, two replay frames and the goals over match time.
Figure 19 : GPT-6 Astra in Codex on the two soccer versions in the Standard Setting. Each row gives the approach and result, two replay frames and the goals over match time.

4 Related Work

Related work spans digital coding-agent evaluation, language-model robot systems, agent harnesses and conventional robotics benchmarks. The distinguishing scope is free choice of control approach under explicit physical-interface constraints.

Benchmarks for coding agents.

Software and terminal benchmarks measure program completion, while computer-use benchmarks test interaction with digital environments. EFE extends this pattern to sensor interpretation and potentially irreversible robot actions, with a verifier the agent cannot modify. Interface design and resistance to exploiting the grader remain central evaluation concerns.

Language models for robots.

Earlier systems compose skills, generate robot code, impose value/keypoint constraints or write rewards, generally through purpose-built interfaces. EFE asks an unchanged general-purpose coding agent to read the supplied control interface and choose its own approach. Code as Policies illustrates the code-generation lineage rather than a baseline reproduced across this exam.

Agentic robots and their harnesses.

Agentic robot studies use direct control, program synthesis and policy training, often fixing the approach or adding skill libraries. EFE permits all three. Native provider harnesses run unmodified; cross-provider comparisons use an adapter, so those results include integration choices as well as model behavior.

Benchmarks for robots.

Visuomotor benchmarks provide fixed observations/actions to learned policies. Coding agents can inspect and execute arbitrary programs, requiring explicit access restrictions. EFE adapts 26 benchmarks and broadens embodiment coverage while separating full-simulator access from sensor-only control. This scope complements existing agentic benchmarks rather than proving their tasks easier.

5 Conclusion

The exam exposes a large gap between privileged simulator engineering and constrained robot execution. Resets, precision requirements, shared difficult stages and model–harness interactions explain parts of that gap. The reported leading agent still fails most selected Standard tasks; high Privileged success is not evidence of reliable physical autonomy.

Future work.

Future evaluation should repeat each test to estimate variability, broaden Realtime coverage and transfer the same sensor/action interfaces to physical robots with reliable grading. The real buzz-wire demonstration is a first example, not a systematic physical benchmark.

Limitations.

Main comparative results are simulated and single-attempt. Stage and control-approach annotations are agent-generated. Appendix J further discusses selection, annotation, reproducibility and deployment limits; the physical demonstration is separate from the benchmark results.

Acknowledgements

The authors disclose AI assistance in coding and writing. They also disclose Pieter Abbeel’s concurrent Berkeley/Amazon appointments and state that the work was performed at Berkeley without Amazon association.

Appendix

The appendices specify suite construction, protocols, interfaces, additional analyses, example attempts, controlled runs, failure-label taxonomy, a physical demonstration, limitations and extended related work. All original tables and figures are retained.

Appendix A Suites and Tasks

The suite appendix defines task packaging, coverage and the distinction between a task, a setting-specific test and the selected hard core.

What a task contains.

Each task includes a goal/instruction, coordinate/action conventions, resource configuration, pinned container runtime and hashed frozen scene. The isolated tests directory contains grading; reference solutions remain inaccessible to the agent. The inventory counts 47 distinct robots across suites without double-counting shared embodiments. Time limits span 30 minutes–4 hours: 590 tasks allow one hour and 351 allow at least two hours.

Table 1: Size of the exam. Rows list the number of suites (with the number built for the exam in parentheses), robots, tasks, tests, tasks with a full or partial reference solution, core tasks and core tests.
Item Count
Suites (new) 35 (9)
Robots 47
Tasks 1,156
Tests 2,288
Tasks with a full / partial reference solution 686 / 78
Core tasks 456
Core tests 901
Table 2: The suites of the exam. The 9 suites that we built come first, followed by the 26 suites adapted from existing benchmarks, which Section A.2 cites. The Kind column gives the kinds of robot body in the suite, nine over the exam (mobile manip. = mobile manipulator; car = wheeled car). The Robots column counts distinct robot models, the Tests column counts tests over all tasks, and the Core column counts tasks in the core set.
Suite Simulation backend Kind Robots Tasks Tests Core
DexHandExam MuJoCo hand 3 30 60 23
Embodied Olympics - Fun MuJoCo humanoid 2 79 179 56
Embodied Olympics - Match MuJoCo arm, humanoid, mobile manip. 5 14 41 13
Embodied Safety Exam MuJoCo arm, two arms, mobile manip. 4 72 76 50
iSpatialGym OmniGibson (Isaac Sim) mobile manip. 1 12 24 8
RoboARC MuJoCo arm, two arms 3 106 212 58
RoboDynamicExam ManiSkill / SAPIEN arm, two arms 2 25 69 10
RoboEngineer MuJoCo + learned dynamics aerial, car, humanoid, quadruped 5 30 30 15
RoboPaint ManiSkill / SAPIEN arm 1 43 86 22
ABC Sim MuJoCo two arms 1 29 60 0
AM-Bench Isaac Lab / Isaac Sim aerial 1 9 17 2
AutoBio MuJoCo arm, two arms 4 9 18 0
BEHAVIOR-1K OmniGibson (Isaac Sim) mobile manip. 1 100 200 71
BiGym 2.0 MuJoCo humanoid 1 13 39 10
DexJoCo MuJoCo hand 2 11 22 2
DexToolBench MuJoCo hand 1 21 42 20
EmbodiedSWE Isaac Lab / Isaac Sim arm, two arms, humanoid 3 24 48 12
HumanoidBench MuJoCo humanoid 2 10 30 8
KinDER MuJoCo mobile manip. 1 5 10 4
LabUtopia Isaac Sim arm 1 9 18 0
LIBERO-MAX robosuite / MuJoCo arm 1 64 128 2
ManiGuard OmniGibson (Isaac Sim) arm 1 30 60 7
MetaWorld+ MuJoCo arm 1 50 100 2
MolmoSpaces MuJoCo arm, mobile manip. 2 8 16 2
MuJoCo Playground MuJoCo arm, two arms, quadruped 5 13 27 7
MyoSuite MuJoCo musculoskeletal 6 22 45 13
RoboCasa robosuite / MuJoCo mobile manip. 1 29 58 2
RoboCasa-GR1 robosuite / MuJoCo humanoid 1 24 48 1
RoboCasa365 robosuite / MuJoCo mobile manip. 1 50 101 5
RoboDojo Isaac Lab / Isaac Sim two arms 1 54 102 12
RoboLab Isaac Lab / Isaac Sim arm 1 53 106 6
RoboTwin 2.0 SAPIEN two arms 1 47 94 3
RoboWits Genesis two arms 1 15 30 6
VLABench MuJoCo (dm_control) arm 1 36 72 2
YAMLab Isaac Lab / Isaac Sim two arms 1 10 20 2
Total 47 1,156 2,288 456

A.1 New Suites

RoboARC, DexHandExam and RoboPaint test ropes/puzzles/assembly, contact-driven dexterity and tool strokes. RoboDynamicExam plus Olympics Fun/Match test moving objects, athletic skills and opponents. iSpatialGym supplies image-only goals; Embodied Safety Exam pairs hazardous and benign requests and grades physical behavior; RoboEngineer tunes controllers on accessible nominal dynamics before deployment to hidden dynamics. These nine suites broaden task types, but their success still depends on simulator fidelity and specified grading.

A.2 Adapted Suites

The 26 adapted suites cover single arms and laboratories; bimanual and mobile household manipulation; humanoid control; dexterous tools; musculoskeletal control; quadrupeds and aerial manipulation. The original full suite/robot/task-count table is retained, including all names and settings. Preserving an original benchmark’s scenes does not make its learned-policy protocol identical to EFE’s coding-agent evaluation.

Appendix B Protocol Details

Protocol details constrain resources, hidden reference solutions, leaderboard retries and the interpretation of control-approach labels.

Budgets.

Attempts receive 2–16 CPU cores, 4–39 GiB memory and task-specific GPU/time allocations. Network access is normally restricted to the model API and package registries, with harness web search disabled. Olympics Fun is an explicit open-network exception; its scores remain included in aggregates.

Reference solutions and integrity checks.

Every task must fail a no-action check. Of 1,156 tasks, 686 have full and 78 partial reference solutions; the remainder rely on original rules plus human review. Solutions are removed from agent containers. Before inclusion, an adversarial agent tries to pass without solving the task; discovered verifier exploits must be repaired. These checks improve integrity but do not prove absence of undiscovered exploits.

Leaderboard rules.

One completed attempt counts per test; infrastructure/API errors are rerun and the latest finished attempt is retained. Aggregates include open-network and special-runtime suites. The harness comparison uses all 456 Standard core tests. Pooling the strongest agents counts each agent/test attempt, rather than a single best-of-three trial, unless reporting union coverage explicitly.

Approach labels.

A coding agent labels 1,489 sampled attempt logs by the approach producing the graded answer. Code includes open-loop trajectories, feedback controllers and parameter search; direct control chooses each next action chunk after observation; policy training learns a controller first. Mixed attempts combine approaches, and exploratory/nonexecuting attempts can have no control plan. These labels characterize behavior, not controlled assignments of strategy.

Appendix C Interface, Grading and Physics

Interface, grading and physics details explain what each setting permits and how replay and live-episode results are checked.

C.1 Interface

The interface distinguishes access to simulator code/state from a separate sensor/action service. Common controls do not imply identical internal affordances across suites.

Privileged Setting: a simulator library in the agent’s container.

Privileged tooling rebuilds the frozen scene, exposes object states and records every applied low-level action. Agents may use original planning primitives or their own controllers; the submitted trajectory still contains the actual control sequence. Some suites instead submit a program executed in a fresh verifier simulator. This setting deliberately permits extensive simulator knowledge and search.

Standard Setting: the simulator as a service.

Standard/Realtime use a separate simulator service reached through an HTTP client. Calls expose robot/action specifications, calibrated images, joint readings, tool pose, timing and optional kinematics; actions return execution video. Object poses, segmentation, scene files, seeds and success-rule code stay hidden. Documented safety telemetry can include contact information without object pose. Images are binary data usable by local programs, not necessarily model-context inputs.

From a chunk to motion.

Action chunks specify valid control-period spacing up to 0.5 seconds; invalid timing is rejected, while joint-limit violations are clipped and reported. Suites retain their original controller abstractions and 5–500 Hz rates. Most stop at first success. RoboDojo instead hides success-triggered termination and grades whether success occurred within its episode, reducing an implicit success-query channel.

Realtime Setting: a simulation that keeps running.

After the first physics-advancing request, Realtime advances one control step every S control periods of wall time. New chunks replace queued actions from a chosen step; an empty queue holds the last target. Slowdown S is task-specific, and missed deadlines/stalls are logged. Thus Realtime imposes continuous evolution, but not always literal real-time simulation speed.

Suites with their own runtime.

RoboEngineer and Olympics Match use custom runtimes. RoboEngineer exposes a nominal simulator for controller tuning, then evaluates a selected controller on hidden-dynamics episodes. These interfaces are documented exceptions to the shared simulator service.

Step limits and time budgets.

There is no universal extra Standard/Realtime step cap, though suites retain their own limits or episode horizons. Privileged trajectories have a stated maximum row count. Wall time is the primary shared task limit; calls also cap returned images and usually last at most 15 minutes.

Limits.

Time, tokens, resets, slowdown, per-call allowances and container resources belong to each result’s configuration. A monitor enforces or records limits not handled by the runtime. Comparisons require matched configurations; unrestricted resets are specific to Privileged unless explicitly supplied by a task or controlled run.

C.2 Grading

Success is a simulator-state rule evaluated by isolated code. Replay grades Privileged submissions, while Standard/Realtime grade the actual service episode.

Success rules.

Adapted tasks retain original success predicates, occasionally with stricter conditions. New tasks check ordered phases, contacts, forces, scoring and fouls. Instructions disclose goal conditions and thresholds even where implementation and privileged state remain hidden.

Settle and hold windows.

Hold windows require sustained success and settle windows advance physics after the last action. Examples include one second of released/resting BEHAVIOR goals, five seconds for Baoding balls, and ten safe control steps for ManiGuard. The same rule applies across settings; a transient visually plausible state need not pass.

Privileged Setting: a fresh replay.

The Privileged verifier rebuilds and hashes the frozen scene, then replays low-level actions or executes a submitted program. Direct state edits in the agent’s simulator do not carry over. BEHAVIOR-1K, AM-Bench, RoboDojo and RoboLab receive up to three fresh replays and pass if any succeeds because their replay is nondeterministic; most suites replay once. This best-of-up-to-three grading is a specific exception.

Standard and Realtime settings: the live episode.

Standard/Realtime grade the live simulator record, not a later replay or the agent’s claimed outcome. At timeout, agent processes stop, advancing requests are refused and any settle window completes before grading. Agent-caused simulator errors fail; infrastructure failures produce no result and can be rerun. With resets, the last episode is graded, while first-episode success is separately recorded.

Isolation.

The verifier has no network and receives only declared regular output files or the service record. Agent and verifier/service containers share no writable volume. Submitted programs run without administrator privileges/network, and their processes end before results are written. Video rendering happens after grading and cannot change the result.

C.3 Physics

Suites retain heterogeneous simulators and physics models. Frozen starts and replay checks improve consistency, while acknowledged nondeterminism and grasp aids constrain realism.

Simulators.

Most suites use MuJoCo or wrappers; others use Isaac/PhysX, OmniGibson, SAPIEN/ManiSkill or Genesis. RoboEngineer’s drone tasks add learned ground-effect and wind dynamics. The exact simulator/version inventory is retained in the original suite table.

What is simulated.

All suites model rigid bodies, contacts and actuation limits; selected tasks add cloth, ropes, liquids, muscles or brush/paper interactions. Physics steps range from 1 ms to 1/60 s. These are heterogeneous modeled effects, not validated equivalence to every physical material or robot.

Determinism.

Fresh replays are tested on evaluation hardware, with up to three attempts where determinism is insufficient. Tasks with unreliable replay, including some GPU cloth/liquid tasks, are Standard-only and grade the live episode. Every reset checks the frozen starting-state hash.

Realism rules.

Robots must act through declared actuators/controllers, and direct simulator state edits do not affect grades. New sports/dexterity/assembly suites restrict scripted attachments and motion. Some manufacturer parameters are increased by up to 1.5–2×, with a disclosed high-bar grip exception. Rope tasks and adapted suites can retain documented grasp aids, including force-limited attachments and assisted grasping; realism is therefore suite-dependent.

Appendix D Additional Results

Additional results retain complete agent definitions, setting-specific denominators, cost estimates and suite/task breakdowns.

Agents.

The eight tested agents are Astra/Codex, Opus/Claude Code, Argon/Antigravity CLI, Kimi and Qwen/Claude Code, Gemini Flash/Antigravity CLI, GLM Flash/Claude Code and DeepSeek Flash/DeepSeek Harness. Each uses high reasoning effort. These named versions/configurations define this experiment; model names alone do not specify the whole agent.

Counting rules.

Setting totals are 456 Standard, 391 Privileged and 54 Realtime tests; every core task has a Standard test. Pooled rates count attempts, and suite breakdowns treat IKEA assembly separately from the rest of RoboARC. Different setting totals should not be compared as if they were identical task samples.

Data.

Main success rates come from the public leaderboard, whereas feature and cross-harness comparisons are additional controlled runs. The controlled comparison uses default task settings and reports a 28% baseline on its paired-task subset; the Standard leaderboard reports about 30%. These settings should remain explicit.

Original figure
Figure 20: Success rate of each agent on the core set. Each bar gives the share of the core tests of one setting that one agent solves.

Success rate of each agent.

Astra leads all three setting aggregates. The strongest-three order changes in Realtime, where Argon exceeds Opus. Scores outside the deliberately difficult core are higher, so core rates are not estimates over all 1,156 tasks or a representative deployment distribution.

Paired outcomes.

Four tasks are solved by Astra only in Standard, and 18 by Argon only in Standard, despite paired Privileged access. The aggregate Privileged advantage is therefore an empirical tendency, not monotonicity guaranteed for each task or attempt.

Results by suite.

At least half of Standard tests remain unsolved by the strongest three in 14 of 22 eligible suites. Only ManiGuard exceeds 60% pooled Standard success among these agents. Privileged performance is much higher in many suites, while HumanoidBench remains difficult. Suite coverage and a five-test inclusion threshold qualify these comparisons.

Original figure
Figure 21: Success rate of each agent against the mean list-price cost of its attempts.

Cost of an attempt.

Token logs are priced at November 2026 list rates, not actual bills. Astra’s mean Standard attempt is estimated at $27.1, about four times Opus/Argon. Failed attempts have higher median cost than solved attempts for every agent in Standard and Privileged. These source-time estimates omit a claim about current prices or total compute/labor expense.

Original figure
Figure 22: Share of the core tasks of each type whose Standard test each agent solves, with a gray line joining the three strongest agents. The number of core tasks of the type is in parentheses, and a task can have several types.

Results by task type.

Whole-task categories differ from individual stage types. Astra’s largest leads are in tool use and precision/contact, while Argon leads object transport at 39%. Pooling the strongest three yields 38% object-transport success versus 11% tool-use success. Task mixtures confound attributing these differences to a category alone.

Appendix E Further Analysis

Further analysis expands the main findings with definitions, paired comparisons, failure-stage controls and case studies. The detailed statistics and labeling protocol are retained below.

E.1 Causes of the Drop from the Privileged to the Standard Setting

Exact state and unlimited resets/saved states are independently added for Astra. Reset successes often start from saved intermediate states. Precise-stage failure increases more than coarse-stage failure under Standard, including in matched reached-stage comparisons. The no-success-signal run is only a rough repeat-attempt control: it also changes the interface. State plus resets still trails Privileged at precise failures, possibly because it lacks single-step physics and planning primitives. The analysis uses one agent and labels cannot separate perception error from control error.

E.2 Where Agents Fail

Failure-stage concentration exceeds both a position-matched baseline (61% versus 37%) and a stage-type baseline (48%). Shared bottlenecks often occur early but are not limited to the first stage. Category difficulty transfers weakly: held-out-suite stage-type rankings agree on only 55% of 313 pairs. Passing a common bottleneck correlates with eventual success, partly because it advances the attempt toward completion. Suite/robot composition prevents a causal interpretation of pooled category rates.

E.3 Differences between Agents

Agent differences are analyzed through solved-set overlap, failure progress and cross-harness comparisons. Denominators and single-run uncertainty matter when interpreting small solved sets.

Tests solved by one agent and by another.

Overlap matrix row A asks what fraction of A’s solved tests another agent also solves. Weaker agents solve only 11–46 Standard tests, so one test shifts a row by 2–9 points. Rank-breaking cases total 68/221 solved Standard tests versus 24/327 Privileged tests. Realtime matrices are omitted because most agents solve at most five tests.

Original figure
Figure 23: Share of the Privileged tests solved by the row agent that the column agent also solves, as in Figure 16, with the agents in the order of Figure 8.
Table 3: Tests that break the order of Figure 8, by suite and setting. “Tests” gives the number of tests of the suite in the setting. Each test that breaks the order is counted in the column of the highest agent in Figure 8 that solves it, and “Weaker” sums the five weaker agents. The six suites with the most such Standard tests and BEHAVIOR-1K, which has the most such Privileged tests, are listed, and “Other suites” sums the rest. RoboARC excludes its IKEA assembly tasks, which are counted under Other suites.
Standard Privileged
Suite Tests
Claude
Opus 5.5
Gemini 4
Argon
Weaker Tests
Claude
Opus 5.5
Gemini 4
Argon
Weaker
Embodied Safety Exam 50 8 5 0 – – – –
RoboARC 50 8 2 1 50 5 1 0
Embodied Olympics - Fun 56 5 1 0 56 1 0 0
ManiGuard 7 4 0 0 7 0 0 0
RoboDojo 12 0 2 1 12 0 0 0
Embodied Olympics - Match 13 1 1 1 13 2 0 0
BEHAVIOR-1K 71 1 1 0 71 6 1 1
Other suites 197 20 5 1 182 4 1 2
All 456 47 17 4 391 18 3 3

Where the order breaks.

Rank-breaking examples include a Flash agent folding a blanket with camera/depth-derived corners, another placing food from an image goal, and Argon opening a jam jar that all others fail. These checked successes show genuine diversity in this run; they do not establish that unsuccessful agents would fail again.

Figure 24: First and last frames of the graded episodes of two Standard tests that break the order of Figure 8. Gemini 3.8 Flash folds a blanket in RoboARC, and Gemini 4 Argon opens a jam jar in DexHandExam.
Figure 24 : First and last frames of the graded episodes of two Standard tests that break the order of Figure 8 . Gemini 3.8 Flash folds a blanket in RoboARC, and Gemini 4 Argon opens a jam jar in DexHandExam.

How the agents fail.

Astra usually completes more stages when it and another agent both fail. Weaker agents’ no-contact failure rates are higher in all 19 sufficiently represented suites. Astra’s larger raw share of lost-hold failures reflects more exposure to grasp stages: per reached grasp stage its rate is 5.6% versus 7.5% for others, with an interval including no difference. Exposure-normalized rates are preferable to interpreting raw failure shares alone.

Harness.

Matched-task harness runs show significant Astra Codex-versus-Claude-Code and Argon Codex-versus-Antigravity differences; Opus changes little. Content-filter handling is one plausible contributor to Argon’s gap: Antigravity terminates blocked sessions while other harnesses retry/continue. This is an observed integration difference, not an isolated causal experiment on that behavior.

Overlap among the three strongest agents.

Among the strongest three, 43 Standard tests are solved by all and 239 by none; Astra, Opus and Argon uniquely solve 56, 27 and 17. Privileged unique successes are 69, 13 and 3. These union/intersection counts describe complementarity within the tested attempts, not an evaluated agent ensemble or repeatable portfolio guarantee.

Figure 25: Core tests of each setting, grouped by which of the three strongest agents solve them.
Figure 25 : Core tests of each setting, grouped by which of the three strongest agents solve them.

E.4 Control Approaches

The sampled approach labels distinguish open-loop, closed-loop and search programs. GPU availability for simulation does not generally imply a training task. The soccer experiment bundles a GPU, learning libraries, parallel training copies and an example, so their effects cannot be separated. Standard/Realtime GPU runs train in 17/20 cases and win 4/20; CPU runs win none, although one distills a handcrafted controller by supervised learning. Privileged wins reuse the opponent’s walking network, a different mechanism. Repeated models across settings/harnesses make these counts dependent, and no significance test is claimed.

Table 4: Attempts of the agents on the two versions of the soccer task, in their own harness and in the harness comparison. “Played with it” counts the attempts that played the match with the policy they trained. The Privileged Setting does not record goals.
Setting Version Attempts Trained with RL Played with it Scored a goal Won
Standard GPU 12 10 9 7 3
CPUs only 12 0 0 3 0
Realtime GPU 8 7 5 2 1
CPUs only 8 0 0 0 0
Privileged GPU 8 5 0 – 2
CPUs only 8 0 0 – 3

Appendix F Example Attempts

Selected public attempts illustrate successful adaptation and distinct failure mechanisms. They are examples rather than a balanced or statistically representative sample of trajectories.

RoboLab, the white mug in the center of the table ( Standard , solved by Gemini 4 Argon).

Argon uses calibrated depth to recognize that a mug is wider than the gripper, grasps its rim with one finger inside, and carefully withdraws after placement. Its sole Standard episode succeeds in about half an hour, illustrating geometry-aware grasp selection.

DexToolBench, red brush ( Standard , solved by none of the three strongest agents).

Opus grasps a brush and reaches the first five of 38 goal poses, but its palm-down grasp cannot make a required 90-degree axial turn. Reconfiguring the arm throws the brush from the hand; initial grasp success does not ensure later manipulability.

RoboARC, IKEA Stegesund ( Privileged , failed).

Astra assembles parts and starts bolting an IKEA unit with privileged state, but limited key rotation makes one bolt take 40 minutes. The second remains 3.7 mm short at the three-hour limit. Access and planning succeed partly while execution efficiency remains limiting.

BEHAVIOR-1K, setting a table ( Privileged , failed).

Astra spends 62 minutes placing two plates before its mobile robot tips against an island while approaching the fridge. Privileged access does not eliminate balance and navigation failures.

RoboARC, threading a rope through six holes ( Standard , failed).

Opus maps six rope-threading holes from depth but loses its grasp whenever the wrist turns, threading none. Perception of the target geometry does not resolve the contact-control bottleneck.

RoboDojo, inserting a key ( Standard , failed).

Opus localizes a key and lock but drops the key at least nine times during transfers and large moves. Its best insertion remains about 9 mm off, showing coupled hold stability and precision failures.

RoboDynamicExam, toss, recatch and deliver ( Standard , failed).

Astra catches and tosses a ball after using three allowed resets, but the final recatch triggers a limit violation at the tilted cup. Intermediate task progress is insufficient when contact constraints invalidate the episode.

RoboPaint, a character in running script ( Standard , failed).

Astra calibrates a brush and retouches a character without resets. Overlap/edge scores pass, but the inked path exceeds 1,450 mm and one region misses its 80% coverage target. Optimizing part of a composite criterion can violate another constraint.

HumanoidBench, opening a door ( Standard , failed).

Opus maintains humanoid stance and locates a door handle, then tries over ten manipulation variants using 45 resets in 27 minutes. The door never opens despite substantial handle displacement; retries alone do not ensure a correct mechanism interaction.

Embodied Olympics - Fun, dribble slalom ( Privileged solved, Standard failed).

Argon passes all 13 slalom checkpoints using parallel privileged search and saved states. With sensor-only control, its lateral position estimate is about 9 cm off and the best episode passes four poles before timeout. The example combines state-estimation and retry-access changes.

BiGym 2.0, closing the dishwasher ( Privileged , solved).

Argon reads dishwasher geometry, pushes racks inward, crouches and hooks the door edge to swing it shut. Its 693-step Privileged trajectory passes replay, illustrating code-based use of simulator geometry.

RoboCasa365, defrosting meat in the microwave ( Privileged , solved).

Opus combines Jacobian inverse kinematics and feedback base control to move frozen meat into a microwave, close it and start defrosting. The submitted Privileged trajectory passes after 46 minutes of agent work.

ManiGuard, dusty transfer (Gemini 4 Argon, Privileged , solved).

Argon wipes 18 dust particles from a box and tips a tray so a potato rolls into it without direct contact. Its Privileged answer passes after 26 minutes, satisfying both goal and manipulation constraint.

RoboPaint, cursive character ( Privileged , solved).

Astra reimplements the disclosed brush model in C++ and fits three strokes before execution. The first executed strokes pass all checks and the answer is submitted after four minutes. This demonstrates the powerful model-access affordance of Privileged evaluation.

BEHAVIOR-1K, bringing in plywood ( Privileged solved, Standard failed).

Opus develops and tests separate grasping, door-opening and carrying scripts in a resettable simulator, passing the plywood task after 72 minutes. In Standard it carries two sheets but moves the base without image checks, hits a curb and falls.

ManiGuard, dusty transfer (GPT-6 Astra, Privileged solved, Standard failed).

Astra repairs earlier potato-contact mistakes through privileged resets and a changed wrist angle. In Standard, one prohibited finger contact during regrasping fails the task even though the potato later reaches the box. Final goal appearance does not erase safety-rule history.

DexToolBench, screwdriver ( Standard , failed).

Opus estimates a screwdriver’s axis but not its rotation around that axis. All 20 allowed resets are used without reaching any of 76 goal poses, illustrating unresolved orientation ambiguity.

ManiGuard, closing a jar ( Standard , solved by Claude Opus 5.5).

Opus infers a jar hinge from depth, slides closed fingers under the lid and rotates it shut, then tests the grasp with a 1 cm lift before transport. Astra instead knocks the jar over in its attempt; the case illustrates one observed success difference.

RoboDynamicExam, toss and recatch ( Privileged ).

Opus uses the arm Jacobian to bound downward acceleration at the toss point and plans a toss/recatch within that limit. This Privileged example illustrates dynamics-aware program design; the paragraph does not supply a separate success-rate comparison.

RoboWits, tipping a cube out of a container ( Privileged ).

Opus tests six pouring positions and retains the steadiest one when tipping a cube from a container. The example shows simulator-assisted parameter search rather than a learned policy.

RoboTwin 2.0, three bottles into a dustbin ( Privileged and Standard ).

Astra scripts bottle handovers using task source and planning primitives in Privileged, submitting after five minutes. In Standard it controls one end-effector chunk at a time from images, recovers a knocked-over bottle and finishes just before the deadline. Control approach changes with access.

Embodied Olympics - Match, badminton ( Realtime , won).

Astra writes a background badminton controller that tracks a shuttle from two cameras, fits drag and swings the racket. It revises the program between rallies from logs and wins 11–4 in Realtime. Fast feedback runs in code rather than waiting for model reasoning at every step.

Appendix G Controlled Runs

Controlled runs add exact object state, reset/saved-state access, both, or remove success feedback. Cross-harness runs use adapters. The source describes these runs on core tasks’ default settings and reports a 28% baseline on the 391-task subset, distinct from its approximately 30% paired Standard leaderboard result; the reported settings/denominators are retained.

Baseline.

Feature comparisons keep Astra/Codex, reasoning effort, time limits and task versions fixed and use the default test of each core task. The source calls the leaderboard run the baseline. Because defaults include Realtime tasks, the controlled subset should not silently be equated with all paired Standard tests.

Object-state query.

Read-only state calls expose object poses, sizes, bounds, velocities and articulated joint states without advancing simulation. They omit success rules, thresholds, goal zones, future objects, hidden hints and hazard labels; RoboPaint coverage is an explicit progress exception. The intervention supplies geometry without revealing every task answer.

Unlimited resets with saved states.

The nominally unlimited-reset intervention raises the budget to 1,000 and adds save/restore handles. The live service still grades the last episode, and restoring state does not erase recorded rule violations. This is extensive retry access with retained constraint history.

Both.

The combined run adds both the filtered object-state queries and the enlarged reset/save/restore interface. Its effect need not equal the sum of the two separate interventions.

No success signal.

Removing success feedback also disables automatic success termination and grades the end state after the agent finishes, timeout or failure. A success later undone becomes failure. This run is therefore an imperfect repeat-attempt control, not an unchanged second sample.

Harness comparison.

Cross-provider adapters translate API requests/responses and also adapt system prompts/model metadata. They approximate native integrations rather than holding every instruction detail identical. All 456 Standard tests are scored, with paired exact McNemar tests for outcome differences.

Further results of the controlled runs.

State queries help especially in visually difficult iSpatialGym, RoboDojo and BEHAVIOR settings; they change little in some sports/dexterity suites. Among reset successes, 14% occur in the first episode and 66% begin from saved states. This indicates targeted retry use rather than merely restarting entire tasks.

Table 5: Success rate (%) of GPT-6 Astra in Codex in the controlled runs, on the 391 core tasks that have both a Privileged and a Standard test, and in the Realtime column on the 50 Realtime tests of these tasks. The Standard column includes the Realtime tests. The Privileged row uses the Privileged tests of the same tasks.
Run Standard Realtime
Standard (baseline) 28 30
+ object-state query 45 38
+ unlimited resets and save/load states 53 64
+ both 68 80
Privileged 77 84
No success signal 31 26
Table 6: Success rate (%) of each model in each harness, on the 456 core tests of the Standard Setting. The first number counts the final result, and the second counts first-episode success. The model’s own harness is marked with an asterisk.
Model (tests) Codex Claude Code Antigravity CLI
GPT-6 Astra (456) 34 / 26∗ 22 / 18 –
Claude Opus 5.5 (456) 26 / 21 25 / 21∗ –
Gemini 4 Argon (456) 29 / 25 25 / 24 24 / 21∗

Further results of the harness comparison.

Argon improves significantly in Codex versus Antigravity under final and first-episode grading, with much gain from BEHAVIOR. Its Claude Code difference is not significant. Astra’s Codex advantage over Claude Code is significant under both counts; Opus’s difference is not. These adapter-mediated results are reported separately from the native-harness leaderboard.

Appendix H Failure Labels

Failure analysis releases graphs and labels for stages, progress, visible phenomena and evidence-supported root causes. Exclusions and annotation limitations determine which comparisons those labels can support.

H.1 Labeling Procedure

A writer agent builds graphs and labels attempts, and a second agent reviews files using frozen prompts. Human input defines the taxonomy and corrects earlier examples, rather than independently annotating every final attempt.

Task graph.

Graphs come from instructions, verifier code and a successful attempt where available. Each stage describes a meaningful world-state change; quality thresholds and prohibitions belong inside its condition. Conditions true initially can be stages and later become false. Contact annotations list alternative robot-part/contact-mode combinations. The 427 graphs contain 2,411 stages, averaging 5.6 per task.

Progress map.

Progress maps record intervals when each condition holds in replay video or logs, so progress can decrease. Reaching a stage means meeting it at some time or failing there. Missing submissions often leave no readable progress; the labels do not reconstruct unseen execution.

Failure stage, phenomena and root causes.

An irreversible event determines the failure stage unless an earlier stage already failed permanently; otherwise use the earliest uncorrected mistake. Up to two phenomena and root causes require supporting video/log evidence. Timeout itself is not a cause. Root causes can stay broad: 47% of analyzed failures are simply labeled Control, limiting mechanistic interpretation.

Writer and reviewer.

GPT-6.1 Sol writes every task graph and most attempt labels; Claude Sonnet 5.5 writes the remainder and reviews all files, with up to two repair rounds and final confirmation. Prompts are frozen before final labeling. A separate agent-labeling pass classifies control approaches.

Human oversight.

Humans settle the taxonomy and review earlier labels in meetings. Discovered errors become worked examples in the prompts, followed by relabeling. This oversight supplies explicit rules but does not yield independent human agreement scores for the complete final dataset.

Core set covered.

Coverage is 6,779 attempts on 857 tests of 427 tasks, with at most one labeled attempt per agent/test. It omits 29 core tasks from stage-graph analysis and does not constitute repeated trials.

H.2 Taxonomy

The taxonomy separates stage action/type, required contact (robot part × interaction), visible failure phenomenon and proposed root cause. Nine stage categories organize comparisons. Root-cause labels can remain at a broad category, and multiple phenomena/causes may attach to one failure; these layers should not be conflated.

Table 7: Stage types, by category. Competition stages (score, defend, win) are counted by category only.
Category Stage types
Relation place, put in, take out, gather, stack, insert, assemble, disassemble, screw/unscrew, thread/hang, connect, hand over
Pose pick up, regrasp, push, pull, flip over, adjust pose, follow path, stabilize
Dynamic throw, strike, catch, dribble
Mechanism open, close, rotate, press, switch on/off
Material fold, flatten, untangle, pour, scoop, cut, clean, stir, draw/write, change temperature, ignite
Body navigate, traverse, jump, perform acrobatics, hang and swing, maintain balance, maintain pose
Safety refuse, warn, mitigate
Information search, inspect, communicate
Competition score, defend, win
Table 8: Contact of a stage: who touches and how.
Who
Fingers/gripper one hand or gripper touches the object directly, also when the pose of a held tool is graded
Bimanual two hands or arms act on the stage together
Tool the robot acts through a held or mounted tool
Legs/torso feet, legs, head, torso or landing gear on a support surface
Mobile base wheels or a wheeled base, while moving
No contact no contact to label, as when flying, observing or reporting
How
Grasp the hand closes on the object
Push/sweep push, nudge or sweep without closing the hand
Press/wipe press or wipe with pressure on the contact surface
Cradle/scoop support, catch or scoop the object from below
Hook hook with a finger or a tool
In-hand adjustment reorient or reposition the object with the fingers without letting go
Support the body or base bears weight, as in standing or landing
Avoid contact pass without touching a named object
Table 9: Failure phenomena: what visibly went wrong at the failure stage, up to two per attempt.
Category Phenomenon Meaning
Contact Failed to make contact the contact the stage needs never existed, as in a missed grasp or fingers that never closed on the object
Lost hold held and then lost: slipped or dropped while held or carried
Abnormal contact force visible or sensed wrong force, such as a part not fully inserted or a latch not engaged
Failed to release the object stays in the hand or tool when the stage needs it to leave
Unstable after release after a planned release the object falls over, rolls away or bounces
Drags the base manipulating a part moves the body that should stay still
Motion Inaccurate pose the hand, tool or object misses the required position or orientation
Obstruction/collision blocked by contact, or knocking over or displacing objects
Out of reach evidence of a reachability or joint limit
Dynamic timing error contact with or release of a moving object too early or too late
Whole body Loss of balance any robot loses support or tips over
Gait failure walking, running or jumping falls short without a fall
Constraint Constraint violation a task rule broken, with no more specific phenomenon
Reporting Missing report, incorrect report a graded information act, such as a warning or a required answer, missing or wrong
Not attempted Not attempted a stage the graded attempt needed was never started
Table 10: Root causes: why the failure happened, up to two per attempt, each with its evidence.
Category Root cause Meaning
Cognition Task understanding the behavior shows the task was misread
Reasoning/planning wrong goal, stage order or overall strategy
Perception the robot acted on a wrong estimate of where or what something is
Safety judgment records show a hazard was not recognized or misjudged
False completion belief a false belief that the task was done affected stopping or decisions
Control Grasp and contact design the contact geometry or load layout does not suit the stage
Trajectory or IK goal and contact plan are feasible, but no trajectory or IK solution is found or followed
Feedback control tracking or stabilization under feedback fails
Parameter tuning parameters shown to be wrong
Delivery Delivery operation error recording, saving or checking the submitted answer went wrong, such as recording without a reset
Task or environment Task or environment every legal solution is ruled out, or an event the agent cannot react to

H.3 Excluded Attempts

Of 5,272 failures among labeled attempts, 500 are excluded from main label-based failure analyses: 161 filter cutoffs and 422 missing submissions overlap in 83 cases. They remain benchmark failures. Stage-progress analyses make an additional exclusion described below.

Table 11: Failed attempts excluded from the failure analysis, by agent and setting. P, S and R stand for the Privileged, Standard and Realtime settings. The column “with failure stage” counts filter cutoffs whose labels also give a failure stage with a physical phenomenon. The column “Excluded” counts filter cutoffs and missing trajectories, and an attempt in both groups counts once. GPT-6 Astra, Claude Opus 5.5 and DeepSeek V4.1 Flash are not listed: they have no filter cutoffs and 1, 2 and 4 missing trajectories in the Privileged Setting.
Agent Setting Failed Filter cutoffs with failure stage Missing trajectory Excluded
Gemini 4 Argon P 221 108 9 134 163
S 326 36 22 0 36
R 47 9 3 0 9
Kimi 3.1 Pro P 238 0 0 49 49
Gemini 3.8 Flash P 291 4 1 97 98
S 400 3 2 0 3
Qwen3.8 Max P 311 1 0 94 94
GLM 5.3 Flash P 340 0 0 41 41
All eight agents 5,272 161 37 422 500

Filter cutoffs, 161 attempts.

Filter cutoffs are concentrated in Argon (153/161), whose harness ends blocked sessions. Thirty-seven cutoff attempts also have a located physical failure, including 34 Argon cases. Excluding the whole group therefore especially affects Argon’s failure profile; it does not remove them from success-rate denominators.

Missing trajectories, 422 attempts.

The 422 missing Privileged submissions have no graded trajectory/replay to annotate. Most are the agent’s own failure rather than infrastructure failure. Missing answers comprise about a third of Gemini Flash and Qwen Privileged failures, so label-based statistics underrepresent this failure mode.

Failures without readable progress, 101 attempts.

Another 101 failed attempts lack readable stage progress. Stage-rate/prediction calculations therefore use 6,178 passed/failed attempts. These 101 remain in statistics not requiring progress; their 72 located failure stages remain eligible for concentration/modal-stage analysis. Denominators differ intentionally by statistic.

H.4 Failures by Suite, Robot Body, Agent and Setting

The heatmaps split 4,772 analyzed failures by progress, stage category, phenomenon and root cause. Cells are shares of failures, not exposure-normalized failure rates. Multiple phenomena/causes can make blocks exceed 100%; small groups and unreadable progress are marked. Pooled suite/body rows mix settings.

Original figure
Figure 26: Failures by suite. Each cell gives the share (%) of the suite’s analyzed failures in the column, with the three settings pooled, and nn is the number of failures. A failure can have up to two phenomena and up to two root causes, so these two blocks can add up to more than 100%.
Original figure
Figure 27: Failures by robot body, agent and setting, in the same format as Figure 26. Here too, the phenomena and the root causes can add up to more than 100%.

By suite.

Some suite profiles follow their construction: painting failures are material stages, matches competition stages, and tool waypoints pose stages. Progress and phenomena still vary within categories; relation-stage failures can involve lost holds in one suite and absent contact in another. These descriptive distributions do not identify universal category difficulty.

By robot body.

Body profiles are strongly confounded by suites and settings: three quarters of mobile-manipulator failures come from BEHAVIOR, and all quadruped failures from MuJoCo Playground. Balance dominates several legged/mobile groups; contact acquisition is common for dexterous/musculoskeletal systems. The rows do not isolate embodiment effects.

By agent and setting.

Relation stages comprise the largest failure share for every agent. Weaker agents more often make no progress or fail to establish contact. Realtime has more dynamic/competition timing failures, while Standard has more collisions and balance loss than Privileged. Different task mixes prevent treating these pooled setting differences as causal.

H.5 Further Statistics

The statistics define reached-stage denominators, precise/coarse grouping, position-matched null models and exposure-normalized agent profiles. Their assumptions and exclusions remain essential to the reported comparisons.

Failure rate of a stage.

A stage failure rate divides failures there by attempts reaching it; repeated same-type stages count separately. A stage can count as met even if true initially or later undone. Setting comparisons compute rates within suites, requiring at least eight reaches in each setting, then weight suites by the smaller reach count.

Table 12: Failure rate of each stage type in percent, compared within suites: the types compared in at least two suites, as in Figure 12(b), and the types of one suite with a weight of at least 25. P is the Privileged and S the Standard Setting. The Suites column gives the number of suites compared, and the Weight column the sum of the smaller counts. Types compared in only one suite are less reliable. Close includes stages that already hold at the start, such as a closed fridge, which count as reached even when the attempt never acts on them.
Stage type Suites Weight P S S −- P
Clean 2 68 33 84 +51
Fold 1 35 38 89 +51
Assemble 2 56 34 80 +46
Strike 2 221 11 55 +44
Perform acrobatics 1 29 28 71 +43
Flatten 1 36 42 84 +42
Adjust pose 5 71 23 64 +41
Dribble 1 42 14 55 +41
Catch 2 100 18 57 +39
Throw 1 56 61 100 +39
Insert 2 46 25 62 +36
Draw/write 1 165 59 94 +35
Jump 2 134 36 66 +30
Traverse 2 95 27 54 +27
Stack 2 23 28 54 +26
Disassemble 2 47 44 70 +26
Thread/hang 3 208 24 49 +25
Follow path 4 497 23 47 +24
Put in 12 797 28 48 +20
Open 4 265 29 47 +18
Pick up 5 84 29 47 +18
Switch on/off 2 24 10 25 +15
Place 10 484 23 35 +12
Close 7 264 5 9 +4
Press 2 96 1 2 +1
Table 13: Failure rate by contact in percent, compared within suites as in Table 12. Bimanual stages are the hardest in both settings, and among the values of how, the failure rates of in-hand adjustment and of cradling or scooping rise the most.
Value Suites Weight P S S −- P
Who Legs/torso 6 844 27 64 +37
Bimanual 10 741 39 71 +32
Tool 12 910 24 55 +31
Fingers/gripper 26 3,420 24 46 +22
Mobile base 6 1,169 26 43 +18
No contact 3 173 11 24 +13
How In-hand adjustment 4 96 29 70 +41
Cradle/scoop 8 268 23 57 +34
Support 11 1,956 27 52 +25
Avoid contact 3 230 24 49 +25
Press/wipe 6 389 32 55 +23
Grasp 25 3,391 24 46 +22
Hook 4 186 31 50 +19
Push/sweep 16 640 17 31 +14

The Privileged and Standard settings.

Thirteen precise types require fits, paths/poses or dynamic contact; twelve coarse types require regions, containers, grasps or mechanism states. Others belong to neither group. Within-suite rates rise 22%→56% for precise and 22%→38% for coarse stages. Matching agent/task/stage reaches reduces the difference in increases to ten points (95% interval 5–16). Follow-path waypoints make up 38% of matched precise stages, and excluding them changes the estimate. Labels cannot separate perception from control or identify each access feature’s contribution on their own.

State and resets at precise stages.

Among Astra’s 256 located Standard failures, 91 are precise and 97 coarse. Resets outperform state alone on discordant precise failures (18 of 22, p=0.004), and adding state to resets helps further (13 of 16 discordant cases, p=0.021). Both features recover about 53% of Privileged’s excess recovery over the imperfect repeat control (95% interval 35–70%). Remaining gaps concentrate in paths, threading and assembly. Tests treat failed tasks as independent and represent uneven suite mixtures.

Table 14: Share (%) of GPT-6 Astra’s labeled failures in the Standard Setting that each controlled run solves, by type of failure stage, with 95% intervals over failures. Material stages include drawing, pouring and cleaning. The Privileged row uses the Privileged test of the same task.
Run Precise (91) Coarse (97) Material (36) Other (32)
No success signal 19 (11–26) 14 (8–22) 28 (14–42) 16 (3–28)
+ object-state query 21 (13–30) 34 (25–43) 42 (25–58) 28 (13–44)
+ unlimited resets and save/load states 36 (26–46) 44 (35–54) 47 (31–64) 31 (16–47)
+ both 47 (37–58) 67 (58–76) 69 (56–83) 41 (22–56)
Privileged 73 (64–81) 60 (51–69) 83 (69–94) 72 (56–88)

Concentration of failure stages.

Concentration is the Simpson pair-agreement index D=Σₖ nₖ(nₖ−1)/[n(n−1)]. Average it over eligible tests and bootstrap tests 2,000 times. The null uses a leave-task-out ten-bin distribution of relative failure position, giving expected agreement Σₖqₖ² rather than uniform stages. Standard agreement exceeds this baseline even for tasks with at least three stages and for the strongest three alone. Each pair compares different agents because only one attempt per agent/test exists; independence in the null is an assumption.

Shared failure stages.

Of 190 Standard tests with at least three stages and four failed agents, 64 place at least 80% of failures at one stage. The original table retains one example per suite. The repeated-catch task has a shared recatch bottleneck in Realtime as well as Standard.

Table 15: Examples of shared failure stages in the Standard Setting, one per suite. The Stages column gives the number of stages of the task. The column “Failed here” gives the failed attempts that failed at this stage over all failed attempts with a located failure stage, without the attempts excluded in Section H.3.
Suite Task Stages Shared stage Type Failed here
BEHAVIOR-1K clean up branches and twigs 5 first branch into the bin put in 8/8
Embodied Safety Exam three spice jars onto a tray 3 second jar onto the tray place 8/8
RoboARC thread a fiber bundle through rings around a post 18 first fiber through its ring thread/hang 7/8
RoboCasa365 defrost meat in the microwave 4 container into the microwave put in 7/7
DexHandExam install the front guard of a fan 5 seat the front guard assemble 6/7
RoboDynamicExam catch, return through a slot, deliver to a moving cart 3 recatch after the rebound catch 7/8
RoboDojo put two slices of bread into a toaster and press the lever 3 first slice into a toaster slot put in 6/7

The common failure stage.

A unique modal failure stage is compared with a sequential hazard model: stage k has probability proportional to hₖ ∏_{j<k}(1−hⱼ), with type rates shrunk toward pooled rates by ten attempts. Type predictions find the Standard mode in 73% of tests versus 65% for always choosing the first stage, yet underpredict concentration. Only 40% of same-stage Standard failure pairs share a phenomenon. Getting past another agents’ modal bottleneck predicts success partly because more of the task is complete.

Table 16: Stage types and the common failure stage, in percent: how often a rule names the most common failure stage of a Standard test, over the 264 Standard tests that have one and over those among them whose tasks have at least two types of stages, and the concentration of failure stages, observed and expected if each agent failed independently at the given rates.
Naming rule All Two or more stage types
Stage-type rates, other tasks 73 67
Stage-type rates, same suite 75 71
Always the first stage 65 56
Concentration Privileged Standard
Observed 50 61
Stage-type rates, other tasks 36 48
Stage-type rates, same suite 40 51
Random failure stages 32 37
Table 17: Getting past the common failure stage, in percent. For each attempt, the common stage is the most common failure stage of the other agents on the same test. The table covers the 1,235 Privileged and 1,874 Standard attempts that have such a stage and a readable status at it.
Privileged Standard
Passed the test, all attempts covered 30 10
Passed the test, attempts past the common stage 64 33
Passed the test, past a common stage that is not the first 76 41
Failed exactly at the common stage, attempts that never met it 77 83

The common failure stage across settings.

The modal failure stage agrees across settings in 152/188 tasks (81%), versus 49% under independent position-matched draws. Removing the 105 first-stage-in-both cases leaves 47/83 (57%, interval 46–67%) versus 9% under the corresponding null. This describes shared failure locations, not identical plans or mechanisms.

Pooled failure rates.

Pooled category rates count every reached stage across suites and settings. Type rows require occurrence in at least three suites and at least 30 total reaches. These exposure counts differ from shares of failed attempts in the heatmaps.

Table 18: Failure rate (%) of each stage category and of the stage types found in at least three suites, in the Privileged Setting, in the Standard Setting and in both together, with the stages at which attempts failed over the stages that attempts reached. The Suites column gives the number of suites with the stage type.
Category Stage type Suites Privileged Standard Both
Mechanism all 18 14 (124/856) 27 (220/825) 20 (344/1,681)
Press 6 2 (3/179) 6 (7/114) 3 (10/293)
Close 9 5 (15/287) 8 (25/305) 7 (40/592)
Switch on/off 4 11 (5/44) 24 (8/33) 17 (13/77)
Open 12 29 (92/321) 47 (164/350) 38 (256/671)
Pose all 16 16 (253/1,592) 51 (389/763) 27 (642/2,355)
Follow path 6 14 (191/1,334) 50 (267/534) 25 (458/1,868)
Pick up 10 24 (28/116) 42 (51/121) 33 (79/237)
Adjust pose 9 24 (29/123) 67 (65/97) 43 (94/220)
Dynamic all 3 17 (121/699) 62 (266/430) 34 (387/1,129)
Strike 3 12 (50/418) 56 (128/229) 28 (178/647)
Relation all 26 26 (615/2,356) 45 (1,051/2,350) 35 (1,666/4,706)
Place 19 23 (150/651) 32 (289/895) 28 (439/1,546)
Thread/hang 6 22 (69/311) 49 (113/229) 34 (182/540)
Put in 20 28 (274/994) 47 (426/905) 37 (700/1,899)
Screw/unscrew 3 22 (14/64) 81 (26/32) 42 (40/96)
Insert 5 23 (16/69) 64 (47/74) 44 (63/143)
Stack 4 25 (10/40) 66 (25/38) 45 (35/78)
Competition all 2 18 (26/141) 50 (67/133) 34 (93/274)
Body all 7 28 (143/504) 67 (267/401) 45 (410/905)
Traverse 4 22 (43/194) 58 (62/106) 35 (105/300)
Navigate 5 33 (8/24) 68 (52/77) 59 (60/101)
Material all 7 44 (189/433) 84 (447/535) 66 (636/968)
Clean 3 33 (28/84) 82 (62/76) 56 (90/160)
Pour 3 25 (3/12) 72 (113/156) 69 (116/168)
Original figure
Figure 28: Failure rate of a stage once reached, by stage category and by the robot part that makes contact, with the Privileged and Standard settings pooled (Table 18). Each cell gives the rate, the stages failed over the stages reached, and a bar on a scale from 0 to 100%. The rows are those of Figure 14, at the same heights.

Interpreting the stage rates.

Stage rates confound suite/robot identity, stage size and contact demands. Relation stages often include grasping, carrying and release, while other stages are much smaller. Closing failure is 19% when initially satisfied stages are excluded but 7% when they count. One-hand, bimanual and body-supported relation rates differ, but do not establish contact modality as the cause. The ordering of middle categories changes between settings.

What predicts the failure stage.

Five-fold task-held-out prediction reduces log loss over a setting-only model by 7.4% with stage type and 6.9% with contact, versus 13% with agent identity. Held-out-suite difficulty rankings agree on only 55% of stage-type pairs, 53% of contact-mode pairs and 66% of category pairs. These predictive associations transfer imperfectly and are not causal explanations.

Failure profiles of the agents.

Failure profiles include many progress ties and overlapping phenomenon labels. Per reached contact stage, weaker agents’ excess no-contact failure is about 11 points (interval 9–13), consistently higher across 19 represented suites. Astra’s raw lost-hold share is high because it reaches more grasp stages; its per-grasp rate is lower than the pooled others but the difference interval includes zero. Constraint-violation shares likewise change interpretation when divided by all attempts.

Table 19: Failure profile of each agent, given as the share in percent of its failed attempts in the Standard and Realtime settings with each label. Zero progress, an attempt that met no stage of the task, counts only Standard attempts on tasks with at least two stages whose progress shows whether they met a stage. Of each agent’s failed Standard attempts on such tasks, 3 to 33% have no readable progress and are left out, the most for Qwen3.8 Max. The highest value of each row is in bold. The rows from “Failed to make contact” to “Not attempted” are phenomena, that row gives the position of the failure stage, and the two rows after it are root causes. The two rows “per stage reached” divide by stages instead of failed attempts: among the reached stages that need contact (or a grasp), the share at which the attempt failed without contact (or with a lost hold). The last two rows compare each agent with GPT-6 Astra on the tests with at least three stages that both failed, excluding ties: the share of these tests on which GPT-6 Astra reached more stages, and their number.
Astra Opus Argon Kimi G-3.8 Flash Qwen DeepSeek GLM
Failed attempts 316 372 328 432 449 433 467 472
Zero progress, Standard 27 36 32 44 47 64 59 59
Failed to make contact 9.8 7.3 10.1 16.7 18.7 20.3 21.2 24.2
Lost hold 17.4 12.6 10.1 12.5 10.5 7.2 9.4 10.8
Inaccurate pose 20.9 20.4 23.5 25.2 21.2 11.1 12.8 14.2
Obstruction/collision 17.1 15.1 8.2 6.5 10.0 11.5 18.8 12.9
Constraint violation 6.6 4.3 4.3 3.0 1.8 4.6 3.6 3.4
Loss of balance 12.7 17.2 9.8 13.0 12.7 19.9 20.1 18.9
Not attempted 0.3 2.7 10.7 0.5 3.1 8.8 4.5 3.8
Failed while exploring 3.2 7.3 4.3 1.6 3.8 8.1 13.7 7.6
False completion belief 0.6 0.3 0.9 0.9 7.6 1.6 1.3 3.2
Feedback control or trajectory/IK 4.7 14.0 3.7 10.2 4.7 9.7 12.0 8.1
No contact, per contact stage reached 2.8 2.7 3.7 9.1 11.9 17.2 16.0 18.5
Lost hold, per grasp stage reached 5.6 6.5 4.4 8.1 8.1 8.4 8.9 9.8
GPT-6 Astra reached more stages – 78 70 92 95 100 99 98
Tests compared – 87 61 95 92 79 120 111

Limitations of the labels.

Labels use video/logs without privileged simulator state, so perception and motion errors are often indistinguishable. Shared model family between the main writer and Astra may bias comparisons. Writers leave root causes empty at different rates, making cross-agent root-cause shares unsuitable comparisons. Analyses therefore rely mainly on stages and visible phenomena; single attempts and small groups leave wide uncertainty.

H.6 Case Studies

Detailed case studies retain full graphs, stage progress and source frames. Times refer to the graded replay rather than the agent’s wall-clock reasoning time.

Carrying in groceries ( Figure 7 ).

For groceries, Astra passes all six stages with Privileged access. Opus opens doors and places the tomato but falls carrying milk; DeepSeek drops the tomato. The fridge-closed stage starts satisfied and becomes false when opened, illustrating why progress can decrease and prerequisites do not fully order every observed condition.

Original figure
Figure 29: Task graph of a DexHandExam task, installing the front guard of a fan, and the stages that three attempts met over time. A bar shows when a stage’s condition held, and a cross marks the failure stage.
Original figure
Figure 30: Progress maps of the eight agents on preparing a lunch box (BEHAVIOR-1K, Standard Setting). Crosses mark the failure stages.

Installing a fan guard ( Figure 29 ).

Opus seats a fan guard and closes clips Privileged, but Standard holds it 36–39 degrees off axis against a 5-degree tolerance and moves the fan while pushing. Astra briefly seats it in Standard but pulls it off while withdrawing fingers. Shared assembly difficulty has different local failure mechanisms.

Catching a ball twice ( Figure 15 ).

Seven of eight Standard agents catch the incoming ball but fail the recatch after pouring through a slot. Four violate forbidden cup–ramp/slot contact, another misses timing, and others fail to release the ball. Kimi fails the first catch. Privileged succeeds for three agents while three others submit no answer, illustrating both shared bottlenecks and annotation exclusions.

Preparing a lunch box ( Figure 30 ).

All agents fail lunch-box preparation. Astra packs four foods then tips over; Argon packs four but opens the wrong cabinet; Opus loses the sandwich before a later fall. Weaker agents typically pack fewer or none. The earliest uncorrected mistake can precede the final visible accident, and initially closed-fridge progress can later be lost.

Figure 31: Standard Setting. (a) RoboARC, threading a rope. (b) RoboCasa365, fetching cereal and a bowl; the column of stage 4 shows the end of the episode.
Figure 31 : Standard Setting. (a) RoboARC, threading a rope . (b) RoboCasa365, fetching cereal and a bowl ; the column of stage 4 shows the end of the episode.

Threading a rope through five passages ( Section 3.4 , Figure 31(a) ).

Opus completes five ordered rope passages. Astra and Argon reach three before the second ring blocks progress; most weaker agents fail at the first passage, often without a useful hold. The figure retains the original stage graph and frames so later-stage failures are not confused with no initial contact.

Fetching cereal and a bowl ( Section 3.4 , Figure 31(b) ).

In cereal-and-bowl retrieval, stronger agents place the cereal but struggle to retain/place the bowl or close the cabinet. Four weaker agents fail without the contact their failure stage requires. Similar final failure status therefore hides substantially different progress and manipulation competence.

Figure 32: BEHAVIOR-1K, disposing of batteries, Standard Setting: the task graph and the stages of each agent, where the wastebasket on the floor holds from the start, and head-camera frames as the robot tips over.
Figure 32 : BEHAVIOR-1K, disposing of batteries , Standard Setting: the task graph and the stages of each agent, where the wastebasket on the floor holds from the start, and head-camera frames as the robot tips over.
Figure 33: Privileged and Standard settings. (a) RoboTwin 2.0, hanging a mug: progress maps of GPT-6 Astra. (b) Embodied Olympics - Fun, heading a ball ten times: the headers of each agent in the Privileged (P) and Standard (S) settings.
Figure 33 : Privileged and Standard settings. (a) RoboTwin 2.0, hanging a mug : progress maps of GPT-6 Astra. (b) Embodied Olympics - Fun, heading a ball ten times : the headers of each agent in the Privileged (P) and Standard (S) settings.

Disposing of batteries ( Section 3.3 , Figure 32 ).

The strongest three each dispose of the first battery before tipping during travel for the second. Astra later rights the robot and grasps another battery, but the recorded fall remains the failure. Four weaker agents fail at the first battery, while Kimi completes it but cannot locate another. The shared later bottleneck is mobile-body control, not identical motion.

Hanging a mug ( Section 3.2 , Figure 33(a) ).

Astra hangs a mug in Privileged but in Standard releases before the handle is securely on the peg; three tries fail. Opus passes both settings. The example shows a precise relation error and reminds that Privileged/Standard gaps are agent-specific.

Heading a ball ten times ( Section 3.2 , Figure 33(b) ).

Four agents complete ten consecutive headers Privileged, but their final Standard episodes reach five, four, two and one. Five resets allow flight fitting and timing adjustments, yet the last episode alone counts. Camera estimation, timing, aiming and retained progress interact; selected traces do not isolate one cause.

Appendix I Real-Robot Demonstration

Two coding agents each run eight consecutive physical buzz-wire trials through a sensor/action interface. This demonstration is outside the exam and excluded from every benchmark success rate.

Figure 34: Buzz wire game on a physical robot arm. (a) Four frames in time order of one trial of GPT-6 Astra and one of Claude Opus 5.5, with a close-up of the light and the buzzer and the command being executed. In the last frame of Claude Opus 5.5’s trial, the loop has touched the wire and the light is on. (b) The plot gives the progress score of each agent in each of its eight trials.
Figure 34 : Buzz wire game on a physical robot arm. (a) Four frames in time order of one trial of GPT-6 Astra and one of Claude Opus 5.5, with a close-up of the light and the buzzer and the command being executed. In the last frame of Claude Opus 5.5’s trial, the loop has touched the wire and the light is on. (b) The plot gives the progress score of each agent in each of its eight trials.

Task.

A robot guides a metal loop along a bent wire without contact. Touching closes an electrical circuit that lights an indicator and sounds a buzzer, supplying an observable failure signal analogous to the simulated task.

Commands and video.

Discrete commands move or rotate the gripper. Video overlays identify the active command and show the contact indicator. The control interface differs from unrestricted joint-level policy evaluation.

Agents and trials.

Astra and Opus each perform one eight-trial sequence on the same wire. There are two agents but no repeated independent sequences per agent or broader robot/task sample.

Progress score.

Progress is the percentage of wire traversed before termination. Only 100% denotes completion without contact; partial progress should not be reported as task success.

Results.

Astra’s first-four/last-four average progress is 65.0%/93.8%, and Opus’s is 35.0%/62.5%. Astra completes trials seven and eight; Opus completes none. Both improve on this sequence with reversals in individual trials. The ranking matches the two models’ simulation aggregate but does not validate simulation-to-hardware predictiveness.

Scope.

One robot, one wire task and one eight-trial sequence per agent demonstrate interface feasibility only. Increasing progress cannot be separated from run variation or familiarity, and no systematic transfer correlation is measured. Repeated physical tests with independent grading remain future work.

Appendix J Limitations

Results are specific to model–harness configurations and one attempt per test. Hard-core screening may favor or disadvantage model families, small suite samples give unstable rates, and model-generated labels can be biased. Of 1,156 tasks, 392 lack any reference solution, so solvability in every setting is unproven. Unequal suite budgets, open-network exceptions and possible public-solution memorization further constrain comparison. Except for the small separate demonstration, results are simulated and their hardware predictiveness is unknown.

Appendix K Extended Related Work

The extended review situates EFE among coding-agent evaluation, robot learning, model-driven planning, general-purpose robot agents and their harnesses. These cited systems establish context; their published results are not matched EFE baselines.

Benchmarks for coding agents.

Digital-agent benchmarks cover code, terminals, research, GPU engineering, security and computer use. Tool interfaces, iterative feedback and verifier integrity affect performance. EFE carries these concerns into sensor-based and sometimes irreversible control while preserving each native harness and isolating the grader.

Benchmarks for robot learning.

Robot-learning benchmarks span manipulation, locomotion, dexterity, aerial and laboratory tasks, generally exposing fixed observations to a learned policy. Coding agents need additional file/state-access restrictions because they can inspect their runtime. EFE adapts 24 such benchmarks plus EmbodiedSWE and ABC Sim; direct comparisons with learned policies under matched observations are proposed future work.

Language models for robot planning and control.

Language-model robotics includes skill planning, code generation, value/keypoint constraints, reward design, task generation and reusable skill libraries. Most pair a model with engineered primitives or training pipelines. EFE evaluates the broader process of selecting and using such techniques through a general coding interface, without claiming to remove the contribution of classical control engineering.

Embodied benchmarks for language models.

Embodied language-model benchmarks commonly define a fixed skill or discretized action vocabulary. EFE instead exposes robot controls and allows the agent to determine how actions are produced. This broadens engineering freedom while increasing the importance of access rules and runtime budgets.

Agentic robots.

Recent agents directly emit actions, write executable controllers/skill libraries, or train policies through simulation or real-world loops. Many studies fix one approach or provide specialized tools. EFE supports all three and records the agent’s choice, so strategy selection is itself an observed outcome rather than a controlled independent variable.

Harnesses for agentic robots.

Robot harnesses differ in action abstraction, learned-policy tools, state representations, motion previews, memory and verification/recovery support. Those features can materially change competence. EFE’s native-harness and adapter-mediated comparisons make this dependence visible, with the adapter qualifications stated in Appendix G.

Benchmarks for agentic robots.

Agentic robot benchmarks vary interface abstraction, embodiments, exploration and safety checks. EFE combines 47 robots of nine types with paired privileged/sensor-only settings. Its safety suite grades physical simulated behavior rather than refusal text alone; this broad coverage still does not establish real-world safety or exhaustive capability measurement.

Concept demonstration

Open image ↗