Original overview
A large, agent-native robotics exam spanning direct actions, generated controllers and policy training. The original overview, omitted from the source HTML, is restored from the PDF; figure labels follow the PDF throughout this edition. Every original figure and table is retained alongside edited section notes.
At a glance
Embodied First Exam evaluates general-purpose coding agents on 1,156 simulated tasks in 35 suites using 47 robots. Agents may act directly, write controllers or train policies. Standard access provides robot sensors and usually no reset; Realtime also keeps the world running; Privileged supplies simulator state and unlimited resets. On the selected 456-task hard core, the best tested agent solves 33.6% of Standard tests. One attempt per agent/test and agent-generated failure labels limit statistical and causal conclusions.
1 Introduction
Robot competence is evaluated as the whole engineering loop: read an interface, select a control approach, execute it and revise from feedback. Paired settings separate success with simulator access and retries from success through sensors under limited recovery. On 391 tasks with both settings, the leading agent solves about 77% Privileged versus 30% Standard. Stage graphs localize failures; controlled runs suggest resets help more than exact object state, particularly at precise-contact stages. These findings concern the reported agents and selected simulated tasks, not physical deployment readiness.
2 The Embodied First Exam: A Unified Evaluation Framework for Coding Agents in Robotics
The framework specifies task interfaces, isolated verification, resource budgets and suite coverage while leaving control strategy to the agent. A test is one task in one evaluation setting.
2.1 Framework and Evaluation Protocol
Tasks use Harbor containers: the agent can install packages, write programs and train models, while an inaccessible verifier grades the outcome. Privileged shares the simulator with the agent and replays its submitted trajectory/controller separately. Standard exposes sensor/proprioceptive queries and actions through a separate service, pauses between requests and usually forbids resets. Realtime keeps simulation advancing, sometimes at a fixed fraction of real time. Paired tests share scene, success rule and time limit. Privileged is an approximate aggregate upper benchmark, not a per-task mathematical bound.
2.2 Resource Constraints for Coding Agents
Each task caps wall time, tokens, compute, memory and disk, with optional robot-step limits. No attempt exhausts its token budget; reaching wall time ends the run and grades the current result. Privileged resets are unlimited; nine suites allow 3–50 explicit resets in Standard/Realtime. Resources can change which control strategies are feasible, so benchmark results describe the complete model–harness–resource setting.
2.3 Task Suites
Nine new suites contribute 411 tasks; 26 adapted suites contribute 745. New tasks emphasize contact-rich dexterity, dynamics/opponents, image goals, safety and hidden-dynamics engineering. Adapted tasks preserve original scenes/robots and success rules, sometimes strengthening criteria, while removing demonstrations and expert solutions. The 456-task core has 901 tests and excludes judged-easy or GPT-6 Luna-solved tasks except author-requested inclusions; this screening may bias against the related Astra model family. Defaults are 403 Standard and 53 Realtime tasks; 427 core tasks have stage graphs. The other 700 tasks remain available but are outside the main evaluation.
3 Analysis
Eight model–harness combinations run at high reasoning effort, once per core test. Verifier pass/fail is distinct from the subsequent interpretation of why an attempt failed. Harness configuration is part of the measured agent.
Stage-level failure labels.
Task graphs specify required world-state changes and precedence. Each failed attempt receives at most one failure stage: an irreversible event or the earliest uncorrected mistake, which can occur before prerequisites. Up to two observable failure phenomena supplement the label. A stage is reached when its condition is met or failure occurs there. GPT-6.1 Sol labels most attempts, Claude Sonnet 5.5 the remainder and reviews every file; these are model-assisted annotations, not independent human ground truth.
3.1 Overall Results
The strongest three agents solve 24.1–33.6% of 456 Standard tests and collectively cover 48%. On the same 54 tasks, Astra solves 23 Standard versus 17 Realtime tests, Opus 10 versus 4, and Argon 15 versus 5. No agent solves 63% of Realtime tests, versus 50% of their Standard counterparts. At least half the Standard tests remain unsolved by the strongest three in 14 of 22 suites with five or more tests. Single-run outcomes do not establish repeatability or confidence intervals.
3.2 What Causes the Drop from the Privileged to the Standard Setting?
On 391 paired tasks, Astra solves 303 Privileged tests and 116 Standard tests; 191 Privileged successes fail Standard. Separate controlled runs raise Standard success from 28% to 45% with exact state, 53% with resets/saved states and 68% with both, versus 77% Privileged. The source controlled comparison uses task defaults (including Realtime) and reports 28%, versus the paired Standard leaderboard’s roughly 30%; these are retained as distinct reported settings. Precise stages show larger setting gaps than coarse stages. Relative to a second-run control hiding success feedback, state alone resolves about two additional precise-stage failures among 91, while resets resolve about 16; exact state helps more at coarse stages. These observations support retries/feedback as important, without isolating every remaining Privileged difference.
3.3 At Which Stage of a Task Do Agents Fail?
Different agents’ located failures concentrate at the same stage: 61% pair agreement in Standard versus a 37% position-matched random baseline, or 56% versus 31% for the strongest three. The modal stage is first in 65% of tests with a unique mode. After excluding tasks whose mode is first in both settings, 47/83 share the same modal stage versus about 9% under the null. Pooled reached-stage failure is 66% for material changes, 45% for body motion and 20% for mechanisms; robot/suite composition confounds causal interpretations of these category rates.
3.4 How Do Agents Differ?
Weaker agents more often fail before required contact or before completing any stage. Strong agents’ Standard successes overlap incompletely: Opus and Argon collectively solve 64 tests Astra fails, whereas Privileged overlap is much greater. One run per test means some complementarity may reflect stochastic variation. Harness changes matter: Astra scores 34% in Codex versus 22% in Claude Code; Argon scores 29%, 25% and 24% in Codex, Claude Code and Antigravity CLI. Cross-provider adapters and session termination after content filtering complicate attributing these differences solely to the harness.
3.5 Which Control Approach Do Agents Use?
In a paired soccer example, the GPU version also adds training libraries, parallel simulator copies and a training example. Ten of twelve Standard and seven of eight Realtime attempts train policies; all three Standard wins and the one Realtime win use trained policies. No CPU-version attempt wins those settings. This bundled, single-task comparison cannot isolate GPU availability as the cause. Across 1,489 labeled sampled attempts, 61% of Standard attempts act directly, 82% of Privileged attempts use code, and 97% of Realtime attempts use code; only six sampled attempts train policies. The soccer-specific counts include extra harness runs.
4 Related Work
Related work spans digital coding-agent evaluation, language-model robot systems, agent harnesses and conventional robotics benchmarks. The distinguishing scope is free choice of control approach under explicit physical-interface constraints.
Benchmarks for coding agents.
Software and terminal benchmarks measure program completion, while computer-use benchmarks test interaction with digital environments. EFE extends this pattern to sensor interpretation and potentially irreversible robot actions, with a verifier the agent cannot modify. Interface design and resistance to exploiting the grader remain central evaluation concerns.
Language models for robots.
Earlier systems compose skills, generate robot code, impose value/keypoint constraints or write rewards, generally through purpose-built interfaces. EFE asks an unchanged general-purpose coding agent to read the supplied control interface and choose its own approach. Code as Policies illustrates the code-generation lineage rather than a baseline reproduced across this exam.
Agentic robots and their harnesses.
Agentic robot studies use direct control, program synthesis and policy training, often fixing the approach or adding skill libraries. EFE permits all three. Native provider harnesses run unmodified; cross-provider comparisons use an adapter, so those results include integration choices as well as model behavior.
Benchmarks for robots.
Visuomotor benchmarks provide fixed observations/actions to learned policies. Coding agents can inspect and execute arbitrary programs, requiring explicit access restrictions. EFE adapts 26 benchmarks and broadens embodiment coverage while separating full-simulator access from sensor-only control. This scope complements existing agentic benchmarks rather than proving their tasks easier.
5 Conclusion
The exam exposes a large gap between privileged simulator engineering and constrained robot execution. Resets, precision requirements, shared difficult stages and model–harness interactions explain parts of that gap. The reported leading agent still fails most selected Standard tasks; high Privileged success is not evidence of reliable physical autonomy.
Future work.
Future evaluation should repeat each test to estimate variability, broaden Realtime coverage and transfer the same sensor/action interfaces to physical robots with reliable grading. The real buzz-wire demonstration is a first example, not a systematic physical benchmark.
Limitations.
Main comparative results are simulated and single-attempt. Stage and control-approach annotations are agent-generated. Appendix J further discusses selection, annotation, reproducibility and deployment limits; the physical demonstration is separate from the benchmark results.
Acknowledgements
The authors disclose AI assistance in coding and writing. They also disclose Pieter Abbeel’s concurrent Berkeley/Amazon appointments and state that the work was performed at Berkeley without Amazon association.
Appendix
The appendices specify suite construction, protocols, interfaces, additional analyses, example attempts, controlled runs, failure-label taxonomy, a physical demonstration, limitations and extended related work. All original tables and figures are retained.
Appendix A Suites and Tasks
The suite appendix defines task packaging, coverage and the distinction between a task, a setting-specific test and the selected hard core.
What a task contains.
Each task includes a goal/instruction, coordinate/action conventions, resource configuration, pinned container runtime and hashed frozen scene. The isolated tests directory contains grading; reference solutions remain inaccessible to the agent. The inventory counts 47 distinct robots across suites without double-counting shared embodiments. Time limits span 30 minutes–4 hours: 590 tasks allow one hour and 351 allow at least two hours.
| Item | Count |
| Suites (new) | 35 (9) |
| Robots | 47 |
| Tasks | 1,156 |
| Tests | 2,288 |
| Tasks with a full / partial reference solution | 686 / 78 |
| Core tasks | 456 |
| Core tests | 901 |
| Suite | Simulation backend | Kind | Robots | Tasks | Tests | Core |
| DexHandExam | MuJoCo | hand | 3 | 30 | 60 | 23 |
| Embodied Olympics - Fun | MuJoCo | humanoid | 2 | 79 | 179 | 56 |
| Embodied Olympics - Match | MuJoCo | arm, humanoid, mobile manip. | 5 | 14 | 41 | 13 |
| Embodied Safety Exam | MuJoCo | arm, two arms, mobile manip. | 4 | 72 | 76 | 50 |
| iSpatialGym | OmniGibson (Isaac Sim) | mobile manip. | 1 | 12 | 24 | 8 |
| RoboARC | MuJoCo | arm, two arms | 3 | 106 | 212 | 58 |
| RoboDynamicExam | ManiSkill / SAPIEN | arm, two arms | 2 | 25 | 69 | 10 |
| RoboEngineer | MuJoCo + learned dynamics | aerial, car, humanoid, quadruped | 5 | 30 | 30 | 15 |
| RoboPaint | ManiSkill / SAPIEN | arm | 1 | 43 | 86 | 22 |
| ABC Sim | MuJoCo | two arms | 1 | 29 | 60 | 0 |
| AM-Bench | Isaac Lab / Isaac Sim | aerial | 1 | 9 | 17 | 2 |
| AutoBio | MuJoCo | arm, two arms | 4 | 9 | 18 | 0 |
| BEHAVIOR-1K | OmniGibson (Isaac Sim) | mobile manip. | 1 | 100 | 200 | 71 |
| BiGym 2.0 | MuJoCo | humanoid | 1 | 13 | 39 | 10 |
| DexJoCo | MuJoCo | hand | 2 | 11 | 22 | 2 |
| DexToolBench | MuJoCo | hand | 1 | 21 | 42 | 20 |
| EmbodiedSWE | Isaac Lab / Isaac Sim | arm, two arms, humanoid | 3 | 24 | 48 | 12 |
| HumanoidBench | MuJoCo | humanoid | 2 | 10 | 30 | 8 |
| KinDER | MuJoCo | mobile manip. | 1 | 5 | 10 | 4 |
| LabUtopia | Isaac Sim | arm | 1 | 9 | 18 | 0 |
| LIBERO-MAX | robosuite / MuJoCo | arm | 1 | 64 | 128 | 2 |
| ManiGuard | OmniGibson (Isaac Sim) | arm | 1 | 30 | 60 | 7 |
| MetaWorld+ | MuJoCo | arm | 1 | 50 | 100 | 2 |
| MolmoSpaces | MuJoCo | arm, mobile manip. | 2 | 8 | 16 | 2 |
| MuJoCo Playground | MuJoCo | arm, two arms, quadruped | 5 | 13 | 27 | 7 |
| MyoSuite | MuJoCo | musculoskeletal | 6 | 22 | 45 | 13 |
| RoboCasa | robosuite / MuJoCo | mobile manip. | 1 | 29 | 58 | 2 |
| RoboCasa-GR1 | robosuite / MuJoCo | humanoid | 1 | 24 | 48 | 1 |
| RoboCasa365 | robosuite / MuJoCo | mobile manip. | 1 | 50 | 101 | 5 |
| RoboDojo | Isaac Lab / Isaac Sim | two arms | 1 | 54 | 102 | 12 |
| RoboLab | Isaac Lab / Isaac Sim | arm | 1 | 53 | 106 | 6 |
| RoboTwin 2.0 | SAPIEN | two arms | 1 | 47 | 94 | 3 |
| RoboWits | Genesis | two arms | 1 | 15 | 30 | 6 |
| VLABench | MuJoCo (dm_control) | arm | 1 | 36 | 72 | 2 |
| YAMLab | Isaac Lab / Isaac Sim | two arms | 1 | 10 | 20 | 2 |
| Total | 47 | 1,156 | 2,288 | 456 |
A.1 New Suites
RoboARC, DexHandExam and RoboPaint test ropes/puzzles/assembly, contact-driven dexterity and tool strokes. RoboDynamicExam plus Olympics Fun/Match test moving objects, athletic skills and opponents. iSpatialGym supplies image-only goals; Embodied Safety Exam pairs hazardous and benign requests and grades physical behavior; RoboEngineer tunes controllers on accessible nominal dynamics before deployment to hidden dynamics. These nine suites broaden task types, but their success still depends on simulator fidelity and specified grading.
A.2 Adapted Suites
The 26 adapted suites cover single arms and laboratories; bimanual and mobile household manipulation; humanoid control; dexterous tools; musculoskeletal control; quadrupeds and aerial manipulation. The original full suite/robot/task-count table is retained, including all names and settings. Preserving an original benchmark’s scenes does not make its learned-policy protocol identical to EFE’s coding-agent evaluation.
Appendix B Protocol Details
Protocol details constrain resources, hidden reference solutions, leaderboard retries and the interpretation of control-approach labels.
Budgets.
Attempts receive 2–16 CPU cores, 4–39 GiB memory and task-specific GPU/time allocations. Network access is normally restricted to the model API and package registries, with harness web search disabled. Olympics Fun is an explicit open-network exception; its scores remain included in aggregates.
Reference solutions and integrity checks.
Every task must fail a no-action check. Of 1,156 tasks, 686 have full and 78 partial reference solutions; the remainder rely on original rules plus human review. Solutions are removed from agent containers. Before inclusion, an adversarial agent tries to pass without solving the task; discovered verifier exploits must be repaired. These checks improve integrity but do not prove absence of undiscovered exploits.
Leaderboard rules.
One completed attempt counts per test; infrastructure/API errors are rerun and the latest finished attempt is retained. Aggregates include open-network and special-runtime suites. The harness comparison uses all 456 Standard core tests. Pooling the strongest agents counts each agent/test attempt, rather than a single best-of-three trial, unless reporting union coverage explicitly.
Approach labels.
A coding agent labels 1,489 sampled attempt logs by the approach producing the graded answer. Code includes open-loop trajectories, feedback controllers and parameter search; direct control chooses each next action chunk after observation; policy training learns a controller first. Mixed attempts combine approaches, and exploratory/nonexecuting attempts can have no control plan. These labels characterize behavior, not controlled assignments of strategy.
Appendix C Interface, Grading and Physics
Interface, grading and physics details explain what each setting permits and how replay and live-episode results are checked.
C.1 Interface
The interface distinguishes access to simulator code/state from a separate sensor/action service. Common controls do not imply identical internal affordances across suites.
Privileged Setting: a simulator library in the agent’s container.
Privileged tooling rebuilds the frozen scene, exposes object states and records every applied low-level action. Agents may use original planning primitives or their own controllers; the submitted trajectory still contains the actual control sequence. Some suites instead submit a program executed in a fresh verifier simulator. This setting deliberately permits extensive simulator knowledge and search.
Standard Setting: the simulator as a service.
Standard/Realtime use a separate simulator service reached through an HTTP client. Calls expose robot/action specifications, calibrated images, joint readings, tool pose, timing and optional kinematics; actions return execution video. Object poses, segmentation, scene files, seeds and success-rule code stay hidden. Documented safety telemetry can include contact information without object pose. Images are binary data usable by local programs, not necessarily model-context inputs.
From a chunk to motion.
Action chunks specify valid control-period spacing up to 0.5 seconds; invalid timing is rejected, while joint-limit violations are clipped and reported. Suites retain their original controller abstractions and 5–500 Hz rates. Most stop at first success. RoboDojo instead hides success-triggered termination and grades whether success occurred within its episode, reducing an implicit success-query channel.
Realtime Setting: a simulation that keeps running.
After the first physics-advancing request, Realtime advances one control step every S control periods of wall time. New chunks replace queued actions from a chosen step; an empty queue holds the last target. Slowdown S is task-specific, and missed deadlines/stalls are logged. Thus Realtime imposes continuous evolution, but not always literal real-time simulation speed.
Suites with their own runtime.
RoboEngineer and Olympics Match use custom runtimes. RoboEngineer exposes a nominal simulator for controller tuning, then evaluates a selected controller on hidden-dynamics episodes. These interfaces are documented exceptions to the shared simulator service.
Step limits and time budgets.
There is no universal extra Standard/Realtime step cap, though suites retain their own limits or episode horizons. Privileged trajectories have a stated maximum row count. Wall time is the primary shared task limit; calls also cap returned images and usually last at most 15 minutes.
Limits.
Time, tokens, resets, slowdown, per-call allowances and container resources belong to each result’s configuration. A monitor enforces or records limits not handled by the runtime. Comparisons require matched configurations; unrestricted resets are specific to Privileged unless explicitly supplied by a task or controlled run.
C.2 Grading
Success is a simulator-state rule evaluated by isolated code. Replay grades Privileged submissions, while Standard/Realtime grade the actual service episode.
Success rules.
Adapted tasks retain original success predicates, occasionally with stricter conditions. New tasks check ordered phases, contacts, forces, scoring and fouls. Instructions disclose goal conditions and thresholds even where implementation and privileged state remain hidden.
Settle and hold windows.
Hold windows require sustained success and settle windows advance physics after the last action. Examples include one second of released/resting BEHAVIOR goals, five seconds for Baoding balls, and ten safe control steps for ManiGuard. The same rule applies across settings; a transient visually plausible state need not pass.
Privileged Setting: a fresh replay.
The Privileged verifier rebuilds and hashes the frozen scene, then replays low-level actions or executes a submitted program. Direct state edits in the agent’s simulator do not carry over. BEHAVIOR-1K, AM-Bench, RoboDojo and RoboLab receive up to three fresh replays and pass if any succeeds because their replay is nondeterministic; most suites replay once. This best-of-up-to-three grading is a specific exception.
Standard and Realtime settings: the live episode.
Standard/Realtime grade the live simulator record, not a later replay or the agent’s claimed outcome. At timeout, agent processes stop, advancing requests are refused and any settle window completes before grading. Agent-caused simulator errors fail; infrastructure failures produce no result and can be rerun. With resets, the last episode is graded, while first-episode success is separately recorded.
Isolation.
The verifier has no network and receives only declared regular output files or the service record. Agent and verifier/service containers share no writable volume. Submitted programs run without administrator privileges/network, and their processes end before results are written. Video rendering happens after grading and cannot change the result.
C.3 Physics
Suites retain heterogeneous simulators and physics models. Frozen starts and replay checks improve consistency, while acknowledged nondeterminism and grasp aids constrain realism.
Simulators.
Most suites use MuJoCo or wrappers; others use Isaac/PhysX, OmniGibson, SAPIEN/ManiSkill or Genesis. RoboEngineer’s drone tasks add learned ground-effect and wind dynamics. The exact simulator/version inventory is retained in the original suite table.
What is simulated.
All suites model rigid bodies, contacts and actuation limits; selected tasks add cloth, ropes, liquids, muscles or brush/paper interactions. Physics steps range from 1 ms to 1/60 s. These are heterogeneous modeled effects, not validated equivalence to every physical material or robot.
Determinism.
Fresh replays are tested on evaluation hardware, with up to three attempts where determinism is insufficient. Tasks with unreliable replay, including some GPU cloth/liquid tasks, are Standard-only and grade the live episode. Every reset checks the frozen starting-state hash.
Realism rules.
Robots must act through declared actuators/controllers, and direct simulator state edits do not affect grades. New sports/dexterity/assembly suites restrict scripted attachments and motion. Some manufacturer parameters are increased by up to 1.5–2×, with a disclosed high-bar grip exception. Rope tasks and adapted suites can retain documented grasp aids, including force-limited attachments and assisted grasping; realism is therefore suite-dependent.
Appendix D Additional Results
Additional results retain complete agent definitions, setting-specific denominators, cost estimates and suite/task breakdowns.
Agents.
The eight tested agents are Astra/Codex, Opus/Claude Code, Argon/Antigravity CLI, Kimi and Qwen/Claude Code, Gemini Flash/Antigravity CLI, GLM Flash/Claude Code and DeepSeek Flash/DeepSeek Harness. Each uses high reasoning effort. These named versions/configurations define this experiment; model names alone do not specify the whole agent.
Counting rules.
Setting totals are 456 Standard, 391 Privileged and 54 Realtime tests; every core task has a Standard test. Pooled rates count attempts, and suite breakdowns treat IKEA assembly separately from the rest of RoboARC. Different setting totals should not be compared as if they were identical task samples.
Data.
Main success rates come from the public leaderboard, whereas feature and cross-harness comparisons are additional controlled runs. The controlled comparison uses default task settings and reports a 28% baseline on its paired-task subset; the Standard leaderboard reports about 30%. These settings should remain explicit.
Success rate of each agent.
Astra leads all three setting aggregates. The strongest-three order changes in Realtime, where Argon exceeds Opus. Scores outside the deliberately difficult core are higher, so core rates are not estimates over all 1,156 tasks or a representative deployment distribution.
Paired outcomes.
Four tasks are solved by Astra only in Standard, and 18 by Argon only in Standard, despite paired Privileged access. The aggregate Privileged advantage is therefore an empirical tendency, not monotonicity guaranteed for each task or attempt.
Results by suite.
At least half of Standard tests remain unsolved by the strongest three in 14 of 22 eligible suites. Only ManiGuard exceeds 60% pooled Standard success among these agents. Privileged performance is much higher in many suites, while HumanoidBench remains difficult. Suite coverage and a five-test inclusion threshold qualify these comparisons.
Cost of an attempt.
Token logs are priced at November 2026 list rates, not actual bills. Astra’s mean Standard attempt is estimated at $27.1, about four times Opus/Argon. Failed attempts have higher median cost than solved attempts for every agent in Standard and Privileged. These source-time estimates omit a claim about current prices or total compute/labor expense.
Results by task type.
Whole-task categories differ from individual stage types. Astra’s largest leads are in tool use and precision/contact, while Argon leads object transport at 39%. Pooling the strongest three yields 38% object-transport success versus 11% tool-use success. Task mixtures confound attributing these differences to a category alone.
Appendix E Further Analysis
Further analysis expands the main findings with definitions, paired comparisons, failure-stage controls and case studies. The detailed statistics and labeling protocol are retained below.
E.1 Causes of the Drop from the Privileged to the Standard Setting
Exact state and unlimited resets/saved states are independently added for Astra. Reset successes often start from saved intermediate states. Precise-stage failure increases more than coarse-stage failure under Standard, including in matched reached-stage comparisons. The no-success-signal run is only a rough repeat-attempt control: it also changes the interface. State plus resets still trails Privileged at precise failures, possibly because it lacks single-step physics and planning primitives. The analysis uses one agent and labels cannot separate perception error from control error.
E.2 Where Agents Fail
Failure-stage concentration exceeds both a position-matched baseline (61% versus 37%) and a stage-type baseline (48%). Shared bottlenecks often occur early but are not limited to the first stage. Category difficulty transfers weakly: held-out-suite stage-type rankings agree on only 55% of 313 pairs. Passing a common bottleneck correlates with eventual success, partly because it advances the attempt toward completion. Suite/robot composition prevents a causal interpretation of pooled category rates.
E.3 Differences between Agents
Agent differences are analyzed through solved-set overlap, failure progress and cross-harness comparisons. Denominators and single-run uncertainty matter when interpreting small solved sets.
Tests solved by one agent and by another.
Overlap matrix row A asks what fraction of A’s solved tests another agent also solves. Weaker agents solve only 11–46 Standard tests, so one test shifts a row by 2–9 points. Rank-breaking cases total 68/221 solved Standard tests versus 24/327 Privileged tests. Realtime matrices are omitted because most agents solve at most five tests.
| Standard | Privileged | |||||||||||||||
| Suite | Tests |
|
|
Weaker | Tests |
|
|
Weaker | ||||||||
| Embodied Safety Exam | 50 | 8 | 5 | 0 | – | – | – | – | ||||||||
| RoboARC | 50 | 8 | 2 | 1 | 50 | 5 | 1 | 0 | ||||||||
| Embodied Olympics - Fun | 56 | 5 | 1 | 0 | 56 | 1 | 0 | 0 | ||||||||
| ManiGuard | 7 | 4 | 0 | 0 | 7 | 0 | 0 | 0 | ||||||||
| RoboDojo | 12 | 0 | 2 | 1 | 12 | 0 | 0 | 0 | ||||||||
| Embodied Olympics - Match | 13 | 1 | 1 | 1 | 13 | 2 | 0 | 0 | ||||||||
| BEHAVIOR-1K | 71 | 1 | 1 | 0 | 71 | 6 | 1 | 1 | ||||||||
| Other suites | 197 | 20 | 5 | 1 | 182 | 4 | 1 | 2 | ||||||||
| All | 456 | 47 | 17 | 4 | 391 | 18 | 3 | 3 | ||||||||
Where the order breaks.
Rank-breaking examples include a Flash agent folding a blanket with camera/depth-derived corners, another placing food from an image goal, and Argon opening a jam jar that all others fail. These checked successes show genuine diversity in this run; they do not establish that unsuccessful agents would fail again.
How the agents fail.
Astra usually completes more stages when it and another agent both fail. Weaker agents’ no-contact failure rates are higher in all 19 sufficiently represented suites. Astra’s larger raw share of lost-hold failures reflects more exposure to grasp stages: per reached grasp stage its rate is 5.6% versus 7.5% for others, with an interval including no difference. Exposure-normalized rates are preferable to interpreting raw failure shares alone.
Harness.
Matched-task harness runs show significant Astra Codex-versus-Claude-Code and Argon Codex-versus-Antigravity differences; Opus changes little. Content-filter handling is one plausible contributor to Argon’s gap: Antigravity terminates blocked sessions while other harnesses retry/continue. This is an observed integration difference, not an isolated causal experiment on that behavior.
Overlap among the three strongest agents.
Among the strongest three, 43 Standard tests are solved by all and 239 by none; Astra, Opus and Argon uniquely solve 56, 27 and 17. Privileged unique successes are 69, 13 and 3. These union/intersection counts describe complementarity within the tested attempts, not an evaluated agent ensemble or repeatable portfolio guarantee.
E.4 Control Approaches
The sampled approach labels distinguish open-loop, closed-loop and search programs. GPU availability for simulation does not generally imply a training task. The soccer experiment bundles a GPU, learning libraries, parallel training copies and an example, so their effects cannot be separated. Standard/Realtime GPU runs train in 17/20 cases and win 4/20; CPU runs win none, although one distills a handcrafted controller by supervised learning. Privileged wins reuse the opponent’s walking network, a different mechanism. Repeated models across settings/harnesses make these counts dependent, and no significance test is claimed.
| Setting | Version | Attempts | Trained with RL | Played with it | Scored a goal | Won |
| Standard | GPU | 12 | 10 | 9 | 7 | 3 |
| CPUs only | 12 | 0 | 0 | 3 | 0 | |
| Realtime | GPU | 8 | 7 | 5 | 2 | 1 |
| CPUs only | 8 | 0 | 0 | 0 | 0 | |
| Privileged | GPU | 8 | 5 | 0 | – | 2 |
| CPUs only | 8 | 0 | 0 | – | 3 |
Appendix F Example Attempts
Selected public attempts illustrate successful adaptation and distinct failure mechanisms. They are examples rather than a balanced or statistically representative sample of trajectories.
RoboLab, the white mug in the center of the table ( Standard , solved by Gemini 4 Argon).
Argon uses calibrated depth to recognize that a mug is wider than the gripper, grasps its rim with one finger inside, and carefully withdraws after placement. Its sole Standard episode succeeds in about half an hour, illustrating geometry-aware grasp selection.
DexToolBench, red brush ( Standard , solved by none of the three strongest agents).
Opus grasps a brush and reaches the first five of 38 goal poses, but its palm-down grasp cannot make a required 90-degree axial turn. Reconfiguring the arm throws the brush from the hand; initial grasp success does not ensure later manipulability.
RoboARC, IKEA Stegesund ( Privileged , failed).
Astra assembles parts and starts bolting an IKEA unit with privileged state, but limited key rotation makes one bolt take 40 minutes. The second remains 3.7 mm short at the three-hour limit. Access and planning succeed partly while execution efficiency remains limiting.
BEHAVIOR-1K, setting a table ( Privileged , failed).
Astra spends 62 minutes placing two plates before its mobile robot tips against an island while approaching the fridge. Privileged access does not eliminate balance and navigation failures.
RoboARC, threading a rope through six holes ( Standard , failed).
Opus maps six rope-threading holes from depth but loses its grasp whenever the wrist turns, threading none. Perception of the target geometry does not resolve the contact-control bottleneck.
RoboDojo, inserting a key ( Standard , failed).
Opus localizes a key and lock but drops the key at least nine times during transfers and large moves. Its best insertion remains about 9 mm off, showing coupled hold stability and precision failures.
RoboDynamicExam, toss, recatch and deliver ( Standard , failed).
Astra catches and tosses a ball after using three allowed resets, but the final recatch triggers a limit violation at the tilted cup. Intermediate task progress is insufficient when contact constraints invalidate the episode.
RoboPaint, a character in running script ( Standard , failed).
Astra calibrates a brush and retouches a character without resets. Overlap/edge scores pass, but the inked path exceeds 1,450 mm and one region misses its 80% coverage target. Optimizing part of a composite criterion can violate another constraint.
HumanoidBench, opening a door ( Standard , failed).
Opus maintains humanoid stance and locates a door handle, then tries over ten manipulation variants using 45 resets in 27 minutes. The door never opens despite substantial handle displacement; retries alone do not ensure a correct mechanism interaction.
Embodied Olympics - Fun, dribble slalom ( Privileged solved, Standard failed).
Argon passes all 13 slalom checkpoints using parallel privileged search and saved states. With sensor-only control, its lateral position estimate is about 9 cm off and the best episode passes four poles before timeout. The example combines state-estimation and retry-access changes.
BiGym 2.0, closing the dishwasher ( Privileged , solved).
Argon reads dishwasher geometry, pushes racks inward, crouches and hooks the door edge to swing it shut. Its 693-step Privileged trajectory passes replay, illustrating code-based use of simulator geometry.
RoboCasa365, defrosting meat in the microwave ( Privileged , solved).
Opus combines Jacobian inverse kinematics and feedback base control to move frozen meat into a microwave, close it and start defrosting. The submitted Privileged trajectory passes after 46 minutes of agent work.
ManiGuard, dusty transfer (Gemini 4 Argon, Privileged , solved).
Argon wipes 18 dust particles from a box and tips a tray so a potato rolls into it without direct contact. Its Privileged answer passes after 26 minutes, satisfying both goal and manipulation constraint.
RoboPaint, cursive character ( Privileged , solved).
Astra reimplements the disclosed brush model in C++ and fits three strokes before execution. The first executed strokes pass all checks and the answer is submitted after four minutes. This demonstrates the powerful model-access affordance of Privileged evaluation.
BEHAVIOR-1K, bringing in plywood ( Privileged solved, Standard failed).
Opus develops and tests separate grasping, door-opening and carrying scripts in a resettable simulator, passing the plywood task after 72 minutes. In Standard it carries two sheets but moves the base without image checks, hits a curb and falls.
ManiGuard, dusty transfer (GPT-6 Astra, Privileged solved, Standard failed).
Astra repairs earlier potato-contact mistakes through privileged resets and a changed wrist angle. In Standard, one prohibited finger contact during regrasping fails the task even though the potato later reaches the box. Final goal appearance does not erase safety-rule history.
DexToolBench, screwdriver ( Standard , failed).
Opus estimates a screwdriver’s axis but not its rotation around that axis. All 20 allowed resets are used without reaching any of 76 goal poses, illustrating unresolved orientation ambiguity.
ManiGuard, closing a jar ( Standard , solved by Claude Opus 5.5).
Opus infers a jar hinge from depth, slides closed fingers under the lid and rotates it shut, then tests the grasp with a 1 cm lift before transport. Astra instead knocks the jar over in its attempt; the case illustrates one observed success difference.
RoboDynamicExam, toss and recatch ( Privileged ).
Opus uses the arm Jacobian to bound downward acceleration at the toss point and plans a toss/recatch within that limit. This Privileged example illustrates dynamics-aware program design; the paragraph does not supply a separate success-rate comparison.
RoboWits, tipping a cube out of a container ( Privileged ).
Opus tests six pouring positions and retains the steadiest one when tipping a cube from a container. The example shows simulator-assisted parameter search rather than a learned policy.
RoboTwin 2.0, three bottles into a dustbin ( Privileged and Standard ).
Astra scripts bottle handovers using task source and planning primitives in Privileged, submitting after five minutes. In Standard it controls one end-effector chunk at a time from images, recovers a knocked-over bottle and finishes just before the deadline. Control approach changes with access.
Embodied Olympics - Match, badminton ( Realtime , won).
Astra writes a background badminton controller that tracks a shuttle from two cameras, fits drag and swings the racket. It revises the program between rallies from logs and wins 11–4 in Realtime. Fast feedback runs in code rather than waiting for model reasoning at every step.
Appendix G Controlled Runs
Controlled runs add exact object state, reset/saved-state access, both, or remove success feedback. Cross-harness runs use adapters. The source describes these runs on core tasks’ default settings and reports a 28% baseline on the 391-task subset, distinct from its approximately 30% paired Standard leaderboard result; the reported settings/denominators are retained.
Baseline.
Feature comparisons keep Astra/Codex, reasoning effort, time limits and task versions fixed and use the default test of each core task. The source calls the leaderboard run the baseline. Because defaults include Realtime tasks, the controlled subset should not silently be equated with all paired Standard tests.
Object-state query.
Read-only state calls expose object poses, sizes, bounds, velocities and articulated joint states without advancing simulation. They omit success rules, thresholds, goal zones, future objects, hidden hints and hazard labels; RoboPaint coverage is an explicit progress exception. The intervention supplies geometry without revealing every task answer.
Unlimited resets with saved states.
The nominally unlimited-reset intervention raises the budget to 1,000 and adds save/restore handles. The live service still grades the last episode, and restoring state does not erase recorded rule violations. This is extensive retry access with retained constraint history.
Both.
The combined run adds both the filtered object-state queries and the enlarged reset/save/restore interface. Its effect need not equal the sum of the two separate interventions.
No success signal.
Removing success feedback also disables automatic success termination and grades the end state after the agent finishes, timeout or failure. A success later undone becomes failure. This run is therefore an imperfect repeat-attempt control, not an unchanged second sample.
Harness comparison.
Cross-provider adapters translate API requests/responses and also adapt system prompts/model metadata. They approximate native integrations rather than holding every instruction detail identical. All 456 Standard tests are scored, with paired exact McNemar tests for outcome differences.
Further results of the controlled runs.
State queries help especially in visually difficult iSpatialGym, RoboDojo and BEHAVIOR settings; they change little in some sports/dexterity suites. Among reset successes, 14% occur in the first episode and 66% begin from saved states. This indicates targeted retry use rather than merely restarting entire tasks.
| Run | Standard | Realtime |
| Standard (baseline) | 28 | 30 |
| + object-state query | 45 | 38 |
| + unlimited resets and save/load states | 53 | 64 |
| + both | 68 | 80 |
| Privileged | 77 | 84 |
| No success signal | 31 | 26 |
| Model (tests) | Codex | Claude Code | Antigravity CLI |
| GPT-6 Astra (456) | 34 / 26∗ | 22 / 18 | – |
| Claude Opus 5.5 (456) | 26 / 21 | 25 / 21∗ | – |
| Gemini 4 Argon (456) | 29 / 25 | 25 / 24 | 24 / 21∗ |
Further results of the harness comparison.
Argon improves significantly in Codex versus Antigravity under final and first-episode grading, with much gain from BEHAVIOR. Its Claude Code difference is not significant. Astra’s Codex advantage over Claude Code is significant under both counts; Opus’s difference is not. These adapter-mediated results are reported separately from the native-harness leaderboard.
Appendix H Failure Labels
Failure analysis releases graphs and labels for stages, progress, visible phenomena and evidence-supported root causes. Exclusions and annotation limitations determine which comparisons those labels can support.
H.1 Labeling Procedure
A writer agent builds graphs and labels attempts, and a second agent reviews files using frozen prompts. Human input defines the taxonomy and corrects earlier examples, rather than independently annotating every final attempt.
Task graph.
Graphs come from instructions, verifier code and a successful attempt where available. Each stage describes a meaningful world-state change; quality thresholds and prohibitions belong inside its condition. Conditions true initially can be stages and later become false. Contact annotations list alternative robot-part/contact-mode combinations. The 427 graphs contain 2,411 stages, averaging 5.6 per task.
Progress map.
Progress maps record intervals when each condition holds in replay video or logs, so progress can decrease. Reaching a stage means meeting it at some time or failing there. Missing submissions often leave no readable progress; the labels do not reconstruct unseen execution.
Failure stage, phenomena and root causes.
An irreversible event determines the failure stage unless an earlier stage already failed permanently; otherwise use the earliest uncorrected mistake. Up to two phenomena and root causes require supporting video/log evidence. Timeout itself is not a cause. Root causes can stay broad: 47% of analyzed failures are simply labeled Control, limiting mechanistic interpretation.
Writer and reviewer.
GPT-6.1 Sol writes every task graph and most attempt labels; Claude Sonnet 5.5 writes the remainder and reviews all files, with up to two repair rounds and final confirmation. Prompts are frozen before final labeling. A separate agent-labeling pass classifies control approaches.
Human oversight.
Humans settle the taxonomy and review earlier labels in meetings. Discovered errors become worked examples in the prompts, followed by relabeling. This oversight supplies explicit rules but does not yield independent human agreement scores for the complete final dataset.
Core set covered.
Coverage is 6,779 attempts on 857 tests of 427 tasks, with at most one labeled attempt per agent/test. It omits 29 core tasks from stage-graph analysis and does not constitute repeated trials.
H.2 Taxonomy
The taxonomy separates stage action/type, required contact (robot part × interaction), visible failure phenomenon and proposed root cause. Nine stage categories organize comparisons. Root-cause labels can remain at a broad category, and multiple phenomena/causes may attach to one failure; these layers should not be conflated.
| Category | Stage types |
| Relation | place, put in, take out, gather, stack, insert, assemble, disassemble, screw/unscrew, thread/hang, connect, hand over |
| Pose | pick up, regrasp, push, pull, flip over, adjust pose, follow path, stabilize |
| Dynamic | throw, strike, catch, dribble |
| Mechanism | open, close, rotate, press, switch on/off |
| Material | fold, flatten, untangle, pour, scoop, cut, clean, stir, draw/write, change temperature, ignite |
| Body | navigate, traverse, jump, perform acrobatics, hang and swing, maintain balance, maintain pose |
| Safety | refuse, warn, mitigate |
| Information | search, inspect, communicate |
| Competition | score, defend, win |
| Who | |
| Fingers/gripper | one hand or gripper touches the object directly, also when the pose of a held tool is graded |
| Bimanual | two hands or arms act on the stage together |
| Tool | the robot acts through a held or mounted tool |
| Legs/torso | feet, legs, head, torso or landing gear on a support surface |
| Mobile base | wheels or a wheeled base, while moving |
| No contact | no contact to label, as when flying, observing or reporting |
| How | |
| Grasp | the hand closes on the object |
| Push/sweep | push, nudge or sweep without closing the hand |
| Press/wipe | press or wipe with pressure on the contact surface |
| Cradle/scoop | support, catch or scoop the object from below |
| Hook | hook with a finger or a tool |
| In-hand adjustment | reorient or reposition the object with the fingers without letting go |
| Support | the body or base bears weight, as in standing or landing |
| Avoid contact | pass without touching a named object |
| Category | Phenomenon | Meaning |
| Contact | Failed to make contact | the contact the stage needs never existed, as in a missed grasp or fingers that never closed on the object |
| Lost hold | held and then lost: slipped or dropped while held or carried | |
| Abnormal contact force | visible or sensed wrong force, such as a part not fully inserted or a latch not engaged | |
| Failed to release | the object stays in the hand or tool when the stage needs it to leave | |
| Unstable after release | after a planned release the object falls over, rolls away or bounces | |
| Drags the base | manipulating a part moves the body that should stay still | |
| Motion | Inaccurate pose | the hand, tool or object misses the required position or orientation |
| Obstruction/collision | blocked by contact, or knocking over or displacing objects | |
| Out of reach | evidence of a reachability or joint limit | |
| Dynamic timing error | contact with or release of a moving object too early or too late | |
| Whole body | Loss of balance | any robot loses support or tips over |
| Gait failure | walking, running or jumping falls short without a fall | |
| Constraint | Constraint violation | a task rule broken, with no more specific phenomenon |
| Reporting | Missing report, incorrect report | a graded information act, such as a warning or a required answer, missing or wrong |
| Not attempted | Not attempted | a stage the graded attempt needed was never started |
| Category | Root cause | Meaning |
| Cognition | Task understanding | the behavior shows the task was misread |
| Reasoning/planning | wrong goal, stage order or overall strategy | |
| Perception | the robot acted on a wrong estimate of where or what something is | |
| Safety judgment | records show a hazard was not recognized or misjudged | |
| False completion belief | a false belief that the task was done affected stopping or decisions | |
| Control | Grasp and contact design | the contact geometry or load layout does not suit the stage |
| Trajectory or IK | goal and contact plan are feasible, but no trajectory or IK solution is found or followed | |
| Feedback control | tracking or stabilization under feedback fails | |
| Parameter tuning | parameters shown to be wrong | |
| Delivery | Delivery operation error | recording, saving or checking the submitted answer went wrong, such as recording without a reset |
| Task or environment | Task or environment | every legal solution is ruled out, or an event the agent cannot react to |
H.3 Excluded Attempts
Of 5,272 failures among labeled attempts, 500 are excluded from main label-based failure analyses: 161 filter cutoffs and 422 missing submissions overlap in 83 cases. They remain benchmark failures. Stage-progress analyses make an additional exclusion described below.
| Agent | Setting | Failed | Filter cutoffs | with failure stage | Missing trajectory | Excluded |
| Gemini 4 Argon | P | 221 | 108 | 9 | 134 | 163 |
| S | 326 | 36 | 22 | 0 | 36 | |
| R | 47 | 9 | 3 | 0 | 9 | |
| Kimi 3.1 Pro | P | 238 | 0 | 0 | 49 | 49 |
| Gemini 3.8 Flash | P | 291 | 4 | 1 | 97 | 98 |
| S | 400 | 3 | 2 | 0 | 3 | |
| Qwen3.8 Max | P | 311 | 1 | 0 | 94 | 94 |
| GLM 5.3 Flash | P | 340 | 0 | 0 | 41 | 41 |
| All eight agents | 5,272 | 161 | 37 | 422 | 500 |
Filter cutoffs, 161 attempts.
Filter cutoffs are concentrated in Argon (153/161), whose harness ends blocked sessions. Thirty-seven cutoff attempts also have a located physical failure, including 34 Argon cases. Excluding the whole group therefore especially affects Argon’s failure profile; it does not remove them from success-rate denominators.
Missing trajectories, 422 attempts.
The 422 missing Privileged submissions have no graded trajectory/replay to annotate. Most are the agent’s own failure rather than infrastructure failure. Missing answers comprise about a third of Gemini Flash and Qwen Privileged failures, so label-based statistics underrepresent this failure mode.
Failures without readable progress, 101 attempts.
Another 101 failed attempts lack readable stage progress. Stage-rate/prediction calculations therefore use 6,178 passed/failed attempts. These 101 remain in statistics not requiring progress; their 72 located failure stages remain eligible for concentration/modal-stage analysis. Denominators differ intentionally by statistic.
H.4 Failures by Suite, Robot Body, Agent and Setting
The heatmaps split 4,772 analyzed failures by progress, stage category, phenomenon and root cause. Cells are shares of failures, not exposure-normalized failure rates. Multiple phenomena/causes can make blocks exceed 100%; small groups and unreadable progress are marked. Pooled suite/body rows mix settings.
By suite.
Some suite profiles follow their construction: painting failures are material stages, matches competition stages, and tool waypoints pose stages. Progress and phenomena still vary within categories; relation-stage failures can involve lost holds in one suite and absent contact in another. These descriptive distributions do not identify universal category difficulty.
By robot body.
Body profiles are strongly confounded by suites and settings: three quarters of mobile-manipulator failures come from BEHAVIOR, and all quadruped failures from MuJoCo Playground. Balance dominates several legged/mobile groups; contact acquisition is common for dexterous/musculoskeletal systems. The rows do not isolate embodiment effects.
By agent and setting.
Relation stages comprise the largest failure share for every agent. Weaker agents more often make no progress or fail to establish contact. Realtime has more dynamic/competition timing failures, while Standard has more collisions and balance loss than Privileged. Different task mixes prevent treating these pooled setting differences as causal.
H.5 Further Statistics
The statistics define reached-stage denominators, precise/coarse grouping, position-matched null models and exposure-normalized agent profiles. Their assumptions and exclusions remain essential to the reported comparisons.
Failure rate of a stage.
A stage failure rate divides failures there by attempts reaching it; repeated same-type stages count separately. A stage can count as met even if true initially or later undone. Setting comparisons compute rates within suites, requiring at least eight reaches in each setting, then weight suites by the smaller reach count.
| Stage type | Suites | Weight | P | S | S P |
| Clean | 2 | 68 | 33 | 84 | +51 |
| Fold | 1 | 35 | 38 | 89 | +51 |
| Assemble | 2 | 56 | 34 | 80 | +46 |
| Strike | 2 | 221 | 11 | 55 | +44 |
| Perform acrobatics | 1 | 29 | 28 | 71 | +43 |
| Flatten | 1 | 36 | 42 | 84 | +42 |
| Adjust pose | 5 | 71 | 23 | 64 | +41 |
| Dribble | 1 | 42 | 14 | 55 | +41 |
| Catch | 2 | 100 | 18 | 57 | +39 |
| Throw | 1 | 56 | 61 | 100 | +39 |
| Insert | 2 | 46 | 25 | 62 | +36 |
| Draw/write | 1 | 165 | 59 | 94 | +35 |
| Jump | 2 | 134 | 36 | 66 | +30 |
| Traverse | 2 | 95 | 27 | 54 | +27 |
| Stack | 2 | 23 | 28 | 54 | +26 |
| Disassemble | 2 | 47 | 44 | 70 | +26 |
| Thread/hang | 3 | 208 | 24 | 49 | +25 |
| Follow path | 4 | 497 | 23 | 47 | +24 |
| Put in | 12 | 797 | 28 | 48 | +20 |
| Open | 4 | 265 | 29 | 47 | +18 |
| Pick up | 5 | 84 | 29 | 47 | +18 |
| Switch on/off | 2 | 24 | 10 | 25 | +15 |
| Place | 10 | 484 | 23 | 35 | +12 |
| Close | 7 | 264 | 5 | 9 | +4 |
| Press | 2 | 96 | 1 | 2 | +1 |
| Value | Suites | Weight | P | S | S P | |
| Who | Legs/torso | 6 | 844 | 27 | 64 | +37 |
| Bimanual | 10 | 741 | 39 | 71 | +32 | |
| Tool | 12 | 910 | 24 | 55 | +31 | |
| Fingers/gripper | 26 | 3,420 | 24 | 46 | +22 | |
| Mobile base | 6 | 1,169 | 26 | 43 | +18 | |
| No contact | 3 | 173 | 11 | 24 | +13 | |
| How | In-hand adjustment | 4 | 96 | 29 | 70 | +41 |
| Cradle/scoop | 8 | 268 | 23 | 57 | +34 | |
| Support | 11 | 1,956 | 27 | 52 | +25 | |
| Avoid contact | 3 | 230 | 24 | 49 | +25 | |
| Press/wipe | 6 | 389 | 32 | 55 | +23 | |
| Grasp | 25 | 3,391 | 24 | 46 | +22 | |
| Hook | 4 | 186 | 31 | 50 | +19 | |
| Push/sweep | 16 | 640 | 17 | 31 | +14 |
The Privileged and Standard settings.
Thirteen precise types require fits, paths/poses or dynamic contact; twelve coarse types require regions, containers, grasps or mechanism states. Others belong to neither group. Within-suite rates rise 22%→56% for precise and 22%→38% for coarse stages. Matching agent/task/stage reaches reduces the difference in increases to ten points (95% interval 5–16). Follow-path waypoints make up 38% of matched precise stages, and excluding them changes the estimate. Labels cannot separate perception from control or identify each access feature’s contribution on their own.
State and resets at precise stages.
Among Astra’s 256 located Standard failures, 91 are precise and 97 coarse. Resets outperform state alone on discordant precise failures (18 of 22, p=0.004), and adding state to resets helps further (13 of 16 discordant cases, p=0.021). Both features recover about 53% of Privileged’s excess recovery over the imperfect repeat control (95% interval 35–70%). Remaining gaps concentrate in paths, threading and assembly. Tests treat failed tasks as independent and represent uneven suite mixtures.
| Run | Precise (91) | Coarse (97) | Material (36) | Other (32) |
| No success signal | 19 (11–26) | 14 (8–22) | 28 (14–42) | 16 (3–28) |
| + object-state query | 21 (13–30) | 34 (25–43) | 42 (25–58) | 28 (13–44) |
| + unlimited resets and save/load states | 36 (26–46) | 44 (35–54) | 47 (31–64) | 31 (16–47) |
| + both | 47 (37–58) | 67 (58–76) | 69 (56–83) | 41 (22–56) |
| Privileged | 73 (64–81) | 60 (51–69) | 83 (69–94) | 72 (56–88) |
Concentration of failure stages.
Concentration is the Simpson pair-agreement index D=Σₖ nₖ(nₖ−1)/[n(n−1)]. Average it over eligible tests and bootstrap tests 2,000 times. The null uses a leave-task-out ten-bin distribution of relative failure position, giving expected agreement Σₖqₖ² rather than uniform stages. Standard agreement exceeds this baseline even for tasks with at least three stages and for the strongest three alone. Each pair compares different agents because only one attempt per agent/test exists; independence in the null is an assumption.
Shared failure stages.
Of 190 Standard tests with at least three stages and four failed agents, 64 place at least 80% of failures at one stage. The original table retains one example per suite. The repeated-catch task has a shared recatch bottleneck in Realtime as well as Standard.
| Suite | Task | Stages | Shared stage | Type | Failed here |
| BEHAVIOR-1K | clean up branches and twigs | 5 | first branch into the bin | put in | 8/8 |
| Embodied Safety Exam | three spice jars onto a tray | 3 | second jar onto the tray | place | 8/8 |
| RoboARC | thread a fiber bundle through rings around a post | 18 | first fiber through its ring | thread/hang | 7/8 |
| RoboCasa365 | defrost meat in the microwave | 4 | container into the microwave | put in | 7/7 |
| DexHandExam | install the front guard of a fan | 5 | seat the front guard | assemble | 6/7 |
| RoboDynamicExam | catch, return through a slot, deliver to a moving cart | 3 | recatch after the rebound | catch | 7/8 |
| RoboDojo | put two slices of bread into a toaster and press the lever | 3 | first slice into a toaster slot | put in | 6/7 |
The common failure stage.
A unique modal failure stage is compared with a sequential hazard model: stage k has probability proportional to hₖ ∏_{j<k}(1−hⱼ), with type rates shrunk toward pooled rates by ten attempts. Type predictions find the Standard mode in 73% of tests versus 65% for always choosing the first stage, yet underpredict concentration. Only 40% of same-stage Standard failure pairs share a phenomenon. Getting past another agents’ modal bottleneck predicts success partly because more of the task is complete.
| Naming rule | All | Two or more stage types |
| Stage-type rates, other tasks | 73 | 67 |
| Stage-type rates, same suite | 75 | 71 |
| Always the first stage | 65 | 56 |
| Concentration | Privileged | Standard |
| Observed | 50 | 61 |
| Stage-type rates, other tasks | 36 | 48 |
| Stage-type rates, same suite | 40 | 51 |
| Random failure stages | 32 | 37 |
| Privileged | Standard | |
| Passed the test, all attempts covered | 30 | 10 |
| Passed the test, attempts past the common stage | 64 | 33 |
| Passed the test, past a common stage that is not the first | 76 | 41 |
| Failed exactly at the common stage, attempts that never met it | 77 | 83 |
The common failure stage across settings.
The modal failure stage agrees across settings in 152/188 tasks (81%), versus 49% under independent position-matched draws. Removing the 105 first-stage-in-both cases leaves 47/83 (57%, interval 46–67%) versus 9% under the corresponding null. This describes shared failure locations, not identical plans or mechanisms.
Pooled failure rates.
Pooled category rates count every reached stage across suites and settings. Type rows require occurrence in at least three suites and at least 30 total reaches. These exposure counts differ from shares of failed attempts in the heatmaps.
| Category | Stage type | Suites | Privileged | Standard | Both |
| Mechanism | all | 18 | 14 (124/856) | 27 (220/825) | 20 (344/1,681) |
| Press | 6 | 2 (3/179) | 6 (7/114) | 3 (10/293) | |
| Close | 9 | 5 (15/287) | 8 (25/305) | 7 (40/592) | |
| Switch on/off | 4 | 11 (5/44) | 24 (8/33) | 17 (13/77) | |
| Open | 12 | 29 (92/321) | 47 (164/350) | 38 (256/671) | |
| Pose | all | 16 | 16 (253/1,592) | 51 (389/763) | 27 (642/2,355) |
| Follow path | 6 | 14 (191/1,334) | 50 (267/534) | 25 (458/1,868) | |
| Pick up | 10 | 24 (28/116) | 42 (51/121) | 33 (79/237) | |
| Adjust pose | 9 | 24 (29/123) | 67 (65/97) | 43 (94/220) | |
| Dynamic | all | 3 | 17 (121/699) | 62 (266/430) | 34 (387/1,129) |
| Strike | 3 | 12 (50/418) | 56 (128/229) | 28 (178/647) | |
| Relation | all | 26 | 26 (615/2,356) | 45 (1,051/2,350) | 35 (1,666/4,706) |
| Place | 19 | 23 (150/651) | 32 (289/895) | 28 (439/1,546) | |
| Thread/hang | 6 | 22 (69/311) | 49 (113/229) | 34 (182/540) | |
| Put in | 20 | 28 (274/994) | 47 (426/905) | 37 (700/1,899) | |
| Screw/unscrew | 3 | 22 (14/64) | 81 (26/32) | 42 (40/96) | |
| Insert | 5 | 23 (16/69) | 64 (47/74) | 44 (63/143) | |
| Stack | 4 | 25 (10/40) | 66 (25/38) | 45 (35/78) | |
| Competition | all | 2 | 18 (26/141) | 50 (67/133) | 34 (93/274) |
| Body | all | 7 | 28 (143/504) | 67 (267/401) | 45 (410/905) |
| Traverse | 4 | 22 (43/194) | 58 (62/106) | 35 (105/300) | |
| Navigate | 5 | 33 (8/24) | 68 (52/77) | 59 (60/101) | |
| Material | all | 7 | 44 (189/433) | 84 (447/535) | 66 (636/968) |
| Clean | 3 | 33 (28/84) | 82 (62/76) | 56 (90/160) | |
| Pour | 3 | 25 (3/12) | 72 (113/156) | 69 (116/168) |
Interpreting the stage rates.
Stage rates confound suite/robot identity, stage size and contact demands. Relation stages often include grasping, carrying and release, while other stages are much smaller. Closing failure is 19% when initially satisfied stages are excluded but 7% when they count. One-hand, bimanual and body-supported relation rates differ, but do not establish contact modality as the cause. The ordering of middle categories changes between settings.
What predicts the failure stage.
Five-fold task-held-out prediction reduces log loss over a setting-only model by 7.4% with stage type and 6.9% with contact, versus 13% with agent identity. Held-out-suite difficulty rankings agree on only 55% of stage-type pairs, 53% of contact-mode pairs and 66% of category pairs. These predictive associations transfer imperfectly and are not causal explanations.
Failure profiles of the agents.
Failure profiles include many progress ties and overlapping phenomenon labels. Per reached contact stage, weaker agents’ excess no-contact failure is about 11 points (interval 9–13), consistently higher across 19 represented suites. Astra’s raw lost-hold share is high because it reaches more grasp stages; its per-grasp rate is lower than the pooled others but the difference interval includes zero. Constraint-violation shares likewise change interpretation when divided by all attempts.
| Astra | Opus | Argon | Kimi | G-3.8 Flash | Qwen | DeepSeek | GLM | |
| Failed attempts | 316 | 372 | 328 | 432 | 449 | 433 | 467 | 472 |
| Zero progress, Standard | 27 | 36 | 32 | 44 | 47 | 64 | 59 | 59 |
| Failed to make contact | 9.8 | 7.3 | 10.1 | 16.7 | 18.7 | 20.3 | 21.2 | 24.2 |
| Lost hold | 17.4 | 12.6 | 10.1 | 12.5 | 10.5 | 7.2 | 9.4 | 10.8 |
| Inaccurate pose | 20.9 | 20.4 | 23.5 | 25.2 | 21.2 | 11.1 | 12.8 | 14.2 |
| Obstruction/collision | 17.1 | 15.1 | 8.2 | 6.5 | 10.0 | 11.5 | 18.8 | 12.9 |
| Constraint violation | 6.6 | 4.3 | 4.3 | 3.0 | 1.8 | 4.6 | 3.6 | 3.4 |
| Loss of balance | 12.7 | 17.2 | 9.8 | 13.0 | 12.7 | 19.9 | 20.1 | 18.9 |
| Not attempted | 0.3 | 2.7 | 10.7 | 0.5 | 3.1 | 8.8 | 4.5 | 3.8 |
| Failed while exploring | 3.2 | 7.3 | 4.3 | 1.6 | 3.8 | 8.1 | 13.7 | 7.6 |
| False completion belief | 0.6 | 0.3 | 0.9 | 0.9 | 7.6 | 1.6 | 1.3 | 3.2 |
| Feedback control or trajectory/IK | 4.7 | 14.0 | 3.7 | 10.2 | 4.7 | 9.7 | 12.0 | 8.1 |
| No contact, per contact stage reached | 2.8 | 2.7 | 3.7 | 9.1 | 11.9 | 17.2 | 16.0 | 18.5 |
| Lost hold, per grasp stage reached | 5.6 | 6.5 | 4.4 | 8.1 | 8.1 | 8.4 | 8.9 | 9.8 |
| GPT-6 Astra reached more stages | – | 78 | 70 | 92 | 95 | 100 | 99 | 98 |
| Tests compared | – | 87 | 61 | 95 | 92 | 79 | 120 | 111 |
Limitations of the labels.
Labels use video/logs without privileged simulator state, so perception and motion errors are often indistinguishable. Shared model family between the main writer and Astra may bias comparisons. Writers leave root causes empty at different rates, making cross-agent root-cause shares unsuitable comparisons. Analyses therefore rely mainly on stages and visible phenomena; single attempts and small groups leave wide uncertainty.
H.6 Case Studies
Detailed case studies retain full graphs, stage progress and source frames. Times refer to the graded replay rather than the agent’s wall-clock reasoning time.
Carrying in groceries ( Figure 7 ).
For groceries, Astra passes all six stages with Privileged access. Opus opens doors and places the tomato but falls carrying milk; DeepSeek drops the tomato. The fridge-closed stage starts satisfied and becomes false when opened, illustrating why progress can decrease and prerequisites do not fully order every observed condition.
Installing a fan guard ( Figure 29 ).
Opus seats a fan guard and closes clips Privileged, but Standard holds it 36–39 degrees off axis against a 5-degree tolerance and moves the fan while pushing. Astra briefly seats it in Standard but pulls it off while withdrawing fingers. Shared assembly difficulty has different local failure mechanisms.
Catching a ball twice ( Figure 15 ).
Seven of eight Standard agents catch the incoming ball but fail the recatch after pouring through a slot. Four violate forbidden cup–ramp/slot contact, another misses timing, and others fail to release the ball. Kimi fails the first catch. Privileged succeeds for three agents while three others submit no answer, illustrating both shared bottlenecks and annotation exclusions.
Preparing a lunch box ( Figure 30 ).
All agents fail lunch-box preparation. Astra packs four foods then tips over; Argon packs four but opens the wrong cabinet; Opus loses the sandwich before a later fall. Weaker agents typically pack fewer or none. The earliest uncorrected mistake can precede the final visible accident, and initially closed-fridge progress can later be lost.
Threading a rope through five passages ( Section 3.4 , Figure 31(a) ).
Opus completes five ordered rope passages. Astra and Argon reach three before the second ring blocks progress; most weaker agents fail at the first passage, often without a useful hold. The figure retains the original stage graph and frames so later-stage failures are not confused with no initial contact.
Fetching cereal and a bowl ( Section 3.4 , Figure 31(b) ).
In cereal-and-bowl retrieval, stronger agents place the cereal but struggle to retain/place the bowl or close the cabinet. Four weaker agents fail without the contact their failure stage requires. Similar final failure status therefore hides substantially different progress and manipulation competence.
Disposing of batteries ( Section 3.3 , Figure 32 ).
The strongest three each dispose of the first battery before tipping during travel for the second. Astra later rights the robot and grasps another battery, but the recorded fall remains the failure. Four weaker agents fail at the first battery, while Kimi completes it but cannot locate another. The shared later bottleneck is mobile-body control, not identical motion.
Hanging a mug ( Section 3.2 , Figure 33(a) ).
Astra hangs a mug in Privileged but in Standard releases before the handle is securely on the peg; three tries fail. Opus passes both settings. The example shows a precise relation error and reminds that Privileged/Standard gaps are agent-specific.
Heading a ball ten times ( Section 3.2 , Figure 33(b) ).
Four agents complete ten consecutive headers Privileged, but their final Standard episodes reach five, four, two and one. Five resets allow flight fitting and timing adjustments, yet the last episode alone counts. Camera estimation, timing, aiming and retained progress interact; selected traces do not isolate one cause.
Appendix I Real-Robot Demonstration
Two coding agents each run eight consecutive physical buzz-wire trials through a sensor/action interface. This demonstration is outside the exam and excluded from every benchmark success rate.
Task.
A robot guides a metal loop along a bent wire without contact. Touching closes an electrical circuit that lights an indicator and sounds a buzzer, supplying an observable failure signal analogous to the simulated task.
Commands and video.
Discrete commands move or rotate the gripper. Video overlays identify the active command and show the contact indicator. The control interface differs from unrestricted joint-level policy evaluation.
Agents and trials.
Astra and Opus each perform one eight-trial sequence on the same wire. There are two agents but no repeated independent sequences per agent or broader robot/task sample.
Progress score.
Progress is the percentage of wire traversed before termination. Only 100% denotes completion without contact; partial progress should not be reported as task success.
Results.
Astra’s first-four/last-four average progress is 65.0%/93.8%, and Opus’s is 35.0%/62.5%. Astra completes trials seven and eight; Opus completes none. Both improve on this sequence with reversals in individual trials. The ranking matches the two models’ simulation aggregate but does not validate simulation-to-hardware predictiveness.
Scope.
One robot, one wire task and one eight-trial sequence per agent demonstrate interface feasibility only. Increasing progress cannot be separated from run variation or familiarity, and no systematic transfer correlation is measured. Repeated physical tests with independent grading remain future work.
Appendix J Limitations
Results are specific to model–harness configurations and one attempt per test. Hard-core screening may favor or disadvantage model families, small suite samples give unstable rates, and model-generated labels can be biased. Of 1,156 tasks, 392 lack any reference solution, so solvability in every setting is unproven. Unequal suite budgets, open-network exceptions and possible public-solution memorization further constrain comparison. Except for the small separate demonstration, results are simulated and their hardware predictiveness is unknown.
Appendix K Extended Related Work
The extended review situates EFE among coding-agent evaluation, robot learning, model-driven planning, general-purpose robot agents and their harnesses. These cited systems establish context; their published results are not matched EFE baselines.
Benchmarks for coding agents.
Digital-agent benchmarks cover code, terminals, research, GPU engineering, security and computer use. Tool interfaces, iterative feedback and verifier integrity affect performance. EFE carries these concerns into sensor-based and sometimes irreversible control while preserving each native harness and isolating the grader.
Benchmarks for robot learning.
Robot-learning benchmarks span manipulation, locomotion, dexterity, aerial and laboratory tasks, generally exposing fixed observations to a learned policy. Coding agents need additional file/state-access restrictions because they can inspect their runtime. EFE adapts 24 such benchmarks plus EmbodiedSWE and ABC Sim; direct comparisons with learned policies under matched observations are proposed future work.
Language models for robot planning and control.
Language-model robotics includes skill planning, code generation, value/keypoint constraints, reward design, task generation and reusable skill libraries. Most pair a model with engineered primitives or training pipelines. EFE evaluates the broader process of selecting and using such techniques through a general coding interface, without claiming to remove the contribution of classical control engineering.
Embodied benchmarks for language models.
Embodied language-model benchmarks commonly define a fixed skill or discretized action vocabulary. EFE instead exposes robot controls and allows the agent to determine how actions are produced. This broadens engineering freedom while increasing the importance of access rules and runtime budgets.
Agentic robots.
Recent agents directly emit actions, write executable controllers/skill libraries, or train policies through simulation or real-world loops. Many studies fix one approach or provide specialized tools. EFE supports all three and records the agent’s choice, so strategy selection is itself an observed outcome rather than a controlled independent variable.
Harnesses for agentic robots.
Robot harnesses differ in action abstraction, learned-policy tools, state representations, motion previews, memory and verification/recovery support. Those features can materially change competence. EFE’s native-harness and adapter-mediated comparisons make this dependence visible, with the adapter qualifications stated in Appendix G.
Benchmarks for agentic robots.
Agentic robot benchmarks vary interface abstraction, embodiments, exploration and safety checks. EFE combines 47 robots of nine types with paired privileged/sensor-only settings. Its safety suite grades physical simulated behavior rather than refusal text alone; this broad coverage still does not establish real-world safety or exhaustive capability measurement.