Literature
Display
← All papers

SimEX: Simulation-Integrated Robotics AutoResearch

Jiaheng Hu · Roberto Martín-Martín · Peter Stone · Rocky Duan · Zhenyu Jiang · Guanya Shi

Preprint posted · arXiv (preprint) ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

SimEX develops reusable robot code in an approximate simulator, then uses a few deployment trials to correct the simulator and screen repairs before trying the robot again.

At a glance

Simulation-Integrated Robotics AutoResearch optimizes a toolbox of perception/control functions and usage notes, rather than directly training a neural policy. Open-ended simulated probing builds initial capabilities; a second stage alternates limited physical evidence with simulator alignment and candidate-repair tests. Controlled experiments span three manipulation families and four coding-agent configurations. On one dual-arm platform, the final system succeeds in twenty-six of thirty physical evaluation trials after five adaptation trials per task. The reported ten minutes measures robot interaction, not total agent reasoning, simulation, reset, or optimization time.

1 Introduction

Directly generated robot code can fail when physical details are unknown, while repeated real-world debugging is expensive. SimEX uses simulation as an experimental workspace in which a coding agent measures limitations, develops procedures, and tests revisions. Physical feedback then corrects relevant simulation mismatches. The approach aims to transfer reusable code and documented knowledge, but does not establish that arbitrary coarse simulators suffice for every robot or task.

Refer to caption
Figure 1: SimEX brings coding agents to physical robots through simulation. (1) The agent develops diverse robot capabilities through open-ended autoresearch in simulation. (2) The agent then efficiently adapts to the real robot within a few trials, turning each real rollout into many simulated tests of candidate repairs. Bottom: physical-robot rollouts using the optimized toolboxes.

2 Method

Preparation builds an approximate scene and initial interface. Stage one expands the toolbox through simulated tasks; stage two concentrates physical trials into a short deployment phase with simulated diagnosis between trials. Human operators still initialize, monitor, and reset the physical scene.

Refer to caption
Figure 2: Overview of SimEX. Preparation reconstructs the physical workspace in simulation and initializes the toolbox. Stage 1 runs independent probe–focus–optimize cycles, retaining improvements on focused tasks and capability-bank replay. Stage 2 executes one robot rollout, aligns the simulator through fixed-policy replay, creates candidate repairs, and returns simulated screening evidence to the coding agent as supportive evidence for repair selection.

2.1 Problem Setup

The persistent artifact consists of toolbox.py, containing perception/control primitives, and skill.md, describing their use and limitations. Each task receives a fresh one-shot Code as Policies call that writes an executable program; the task program is discarded after execution while shared toolbox updates persist.

The fixed low-level interface supplies raw color/depth images, proprioception, inverse kinematics, reachability queries, Cartesian/joint targets, and gripper commands. It provides no task-level manipulation skills or privileged scene state to the execution program. Agent-generated simulation success predicates can inspect simulator state for scoring, which is a separate privileged evaluation role. The coding agent controls optimization decisions; the harness executes and reports outcomes.

2.2 Preparation: Constructing the simulation sandbox

A coding agent reconstructs the workspace from an image and object description using an existing empty YAM station template and predefined mesh library. It chooses/configures assets and initializes perception code and interface documentation. This is an automated approximate reconstruction within supplied infrastructure, not calibration or scene understanding from an entirely unstructured starting point.

2.3 Stage 1: Open-ended capability discovery in simulation

Each cycle broadly probes self-generated tasks, placements, object properties, and robot starts, with agent-written success predicates. The agent chooses one systematic weakness and fixes a focused task set while hill-climbing toolbox revisions. Replay from a bank of previously solved tasks discourages regressions, after which probing seeks another weakness. The absence of a single prescribed training metric does not mean there is no scoring: each generated test has an executable predicate whose correctness remains part of the agent-created setup.

Four independent optimization members each run twenty iterations with their own toolbox and capability bank. At probe boundaries they can read short experiment descriptions and reimplement ideas, but cannot copy another implementation. This diversifies discovery while preserving distinct trajectories; it is not unconstrained file sharing or a proven guarantee of broad capability coverage.

2.4 Stage 2: Simulation-assisted few-trial adaptation on the robot

Each deployment rollout records camera evidence and the executed script/transcript. The agent replays that exact script in simulation and adjusts the simulator until the behavior relevant to the failure is reproduced according to its qualitative judgment. It then proposes distinct hypotheses and separate toolbox repairs, screens the current toolbox and candidates on the fixed-script replay plus additional test tasks, and chooses a repair or retains the old toolbox.

Candidate ranking is not an automatic highest-score rule: the agent judges the trustworthiness of simulation given remaining mismatch. One physical trial can therefore support many offline experiments, but neither simulator alignment nor repair selection is certified to match the real system. The updated simulator persists into later screens.

3 Experiments

The experiments combine repeated transfer between independently implemented simulators, a smaller hardware study, component ablations, and a comparison to online agent control. Their protocols and evidential scales differ.

3.1 Experimental setup

The common platform is a dual-arm YAM with one overhead and two wrist cameras. Seven evaluation tasks per family cover plate transport, towel manipulation, and barcode scanning. In controlled experiments, adaptive methods have five deployment trials per run, then freeze the toolbox and generate fresh programs for five evaluation rollouts. Five runs yield twenty-five scheduled evaluations per method/task. Family means weight tasks equally, and confidence intervals bootstrap runs within tasks. The default source configuration uses Fable 5.1 with maximum reasoning in simulation discovery and medium reasoning during adaptation.

Original figure
Figure 3: Aggregate performance across the three evaluation task families.

3.2 Controlled sim-to-sim evaluation

Rigid-object experiments transfer from MuJoCo to Isaac Sim; deformable experiments transfer from FLASH to MJWarp Flex. The target simulator’s state and source are hidden from the agent, while camera evidence and task outcomes remain available. This gives a controlled transfer gap, not a replacement for all hardware testing.

The raw tables give SimEX 165/175 plate evaluations, 118/175 towel evaluations, and 149/175 barcode evaluations, corresponding to about 94%, 67%, and 85%. It matches or exceeds every external baseline on each of the twenty-one tasks, but does not exceed every internal ablation on every task. For example, direct repair scores one additional success on the hidden-barcode block and two-bottle cases. Both stage-level ablations are worse in family averages.

Actual adaptation-rollout curves show that the initial simulated toolbox can start near zero transfer success, yet one deployment trial enables a substantial repair. Simulation-guided repair extracts more value from that feedback in the reported curves. These are deployment-iteration outcomes, distinct from the final frozen-toolbox evaluation counts.

Original figure
Figure 4: Online adaptation (Stage 2). Each point is the average success rate of the deployment rollouts at that iteration.
Original figure
Figure 5: Fine-grained ablations. Points show mean task success, with the full SimEX result marked by the red line. Every ablation reduces performance.

3.3 Real-robot evaluation

Hardware testing uses one representative task per family: stow four plates, half-fold a towel, and scan/stow two objects. After five adaptation trials, roughly ten minutes of robot interaction per task, SimEX succeeds in 10/10, 8/10, and 8/10 held-out physical trials. The strongest baseline totals three successes across the thirty evaluations; every baseline fails the two-object barcode task. A human resets the scene between trials. These are promising results on three tasks and one platform, without the five-independent-run hardware replication used in the controlled simulator study.

Table 1: Physical-robot successful evaluation trials out of 10.
Method Plate Towel Barcode
Direct CaP 0/100/10 0/100/10 0/100/10
ENPIRE∗ 2/102/10 1/101/10 0/100/10
ZS Sim2Real 0/100/10 2/102/10 0/100/10
ASPIRE∗ 2/102/10 0/100/10 0/100/10
SimEX (ours) 𝟏𝟎/𝟏𝟎10/10 𝟖/𝟏𝟎8/10 𝟖/𝟏𝟎8/10

3.4 Ablations and further analysis

Removing population search, replay, multiple repair alternatives, or explicit hypotheses lowers aggregate barcode performance. Individual tasks can improve or tie under an ablation. The no-population condition uses one process rather than four, so reduced total exploration budget accompanies reduced diversity and prevents attributing the entire drop to diversity alone.

Replacing both optimizer and program-writing model preserves SimEX’s aggregate advantage across the four reported agent configurations. Barcode success varies from about 67% with GPT-5.5 to 82% with GPT-6 Astra; this is an empirical comparison under the paper’s settings, not a timeless model ranking. A separate three-task test finds one pre-execution program-synthesis call more practical than serial online inference: all three generated validation programs finish, while none of fifteen online-control rollouts finishes within budget.

Refer to caption
Figure 6: Cross-agent comparison. Our main conclusion holds with different coding agents.

4 Related Work

SimEX combines robot program generation, iterative coding-agent improvement, generated simulation tasks, and sim-to-real adaptation. Its distinctive use of simulation is to develop a persistent code toolbox and diagnose candidate repairs between scarce deployment trials, rather than solely produce demonstrations or optimize a fixed neural controller.

5 Conclusion

Approximate simulation plus limited deployment grounding can support useful robot capability development in the tested setting. Limits include a single hardware platform, manual scene resets, substantial optimization compute, dependence on supplied low-level interfaces/templates, and agent judgment in self-generated scoring, simulator alignment, and repair choice. The results support the two-stage workflow, without demonstrating fully autonomous physical operation or universal transfer from arbitrary simulations.

Appendix A Implementation details

Implementation details retain budgets, the exact one-shot program-writing prompt, and measured coding-agent overhead.

A.1 Hyperparameters

Each of four discovery members runs twenty optimization iterations, with twenty-four to thirty-two broad probes, about ten focused configurations, and four to six replay tasks per candidate. Adaptation uses five robot rollouts, four candidate repairs per iteration, and five agent-written test scenes. Evaluation uses five runs and five rollouts per run. These are experiment budgets, not the number of physical evaluation trials in the separate hardware table.

Table 2: Default hyperparameters used in the main experiments. Population size, per-process pretraining budget, deployment-trial budget, and evaluation protocol are matched across methods whenever the corresponding phase applies.
Phase Hyperparameter Value
Stage 1 & 2 Coding-agent orchestrator Fable 5.1
Stage 1 & 2 Policy-writing model Fable 5.1
Stage 1 Reasoning effort max
Stage 2 Reasoning effort medium
Stage 1 Independent population members 4
Stage 1 Optimization iterations per member 20
Stage 1 Broad probe configurations per iteration 24–32
Stage 1 Focused configurations per selected sub-goal ∼\sim10
Stage 1 Capability-bank replay configurations per candidate 4–6
Stage 2 Robot rollouts per iteration 1
Stage 2 Repair candidates per iteration 4
Stage 2 Adaptation trials per run 5
Stage 2 Agent-written test scenes per iteration 5
Evaluation Independent runs per method–task pair 5
Evaluation I.i.d. evaluation trials per run 5

A.2 Policy-writing prompt

The shared text-only prompt concatenates skill documentation, toolbox source, task name/description, and exact plan/script markers. It allows programs to bypass toolbox helpers while forbidding privileged object-state access. The model must produce a plan of at most two hundred words and complete Python code in one turn; tool use or multiple turns voids a trial rather than triggering a retry. Both original PDF prompt portions are retained as native vectors, preserving slots, markers, colors, and line structure that the HTML conversion otherwise leaves as unresolved macros.

Original policy-writing prompt, first part, preserving exact slot and marker formatting.Original policy-writing prompt, continuation, preserving exact slot and marker formatting.
Original policy-writing prompt (Appendix A.2), including its continuation on the following PDF page.

A.3 Agent thinking time and token usage

For barcode optimization, the reported average discovery cost per population member is 840 orchestrator calls, 5.4 hours of agent thinking, and 1.98 million output tokens including 1.16 million reasoning tokens. Adaptation averages 124 calls, thirty-one minutes of thinking, and 161,000 output tokens per run. Cache-read inputs are separately large. These costs exclude interpreting the ten-minute robot-interaction figure as total training time; after optimization, one task’s program synthesis is around thirty seconds.

Table 3: Coding agent orchestration cost.
Stage 1 (per member) Stage 2 (per run)
Orchestrator calls 840 124
Agent thinking time 5.4 h 31 min
Output tokens (reasoning) 1.98M (1.16M) 161k (64k)
Cache-read input tokens 155M 19.5M

Appendix B Evaluation task details

The evaluation families test different contact, reach, deformation, and logical-order requirements. All seven task definitions in each family are reflected in the retained initial-state figures and detailed notes below.

Refer to caption
Figure 7: Physical setups for the three evaluation task families: plate to tote, towel folding, and barcode scanning. All use the same dual-arm YAM platform and workspace enclosure, while task-specific objects and fixtures produce distinct manipulation challenges.

B.1 Plate to tote

Plates must be lifted at least five centimetres and end inside the tote, while unrequested plates remain outside. The seven variants include a same-side pair, a diagonal pair, a far outboard plate, a long far-plate-to-near-tote carry, any three of four plates, a perturbed single-plate scene, and a perturbed diagonal pair. The shifted tote is translated 3.5 centimetres, shifted laterally three centimetres, and rotated six degrees, with plate placements also perturbed. Flat plates require a geometry-aware rim grasp rather than a simple top-down pinch.

Refer to caption
Figure 8: Initial states of the seven plate to tote tasks. (Isaac Sim)

B.2 Towel folding

Seven towel variants cover rotated or left-shifted translation, right-shifted or rotated compaction, left-shifted or rotated half-folds, and a rotated double fold. Translation requires at least five centimetres of center motion; compaction permits at most eighty percent of flat footprint; half-fold variants use a sixty-two-percent threshold; double folding requires at most thirty-eight percent. The rotated half-fold additionally requires one side at most sixty-two percent and the other at least seventy-eight percent of the original length, lying flat rather than piled. Thus success is more specific than merely reducing projected area for every variant.

Refer to caption
Figure 9: Initial states of the seven towel folding tasks. (mjlab)

B.3 Barcode scanning

Barcode tasks require exposing and aligning the correct label before stowing, under scanner-window and tote-wall constraints. The seven cases are a wall-side bottle with inward label, an across-tote bottle with label facing away, a far upright bottle, a floor-facing hidden-label block, a wall-side floor-facing prism, two bottles, and a bottle-plus-block scene. Multi-object cases require both scans and both deliveries; grasp selection, reorientation, and action order all matter.

Refer to caption
Figure 10: Initial states of the seven barcode scanning tasks. (Isaac Sim)

Appendix C Baseline details

Baselines use a shared interface, initial toolbox, task seeds, model settings, and evaluation protocol. Asterisks indicate methods reimplemented to fit this setting rather than original released systems run unchanged.

Controlled comparison.

All methods receive the same raw observations, calibration, kinematic queries, and generic motion wrappers without task-level skills. Where simulated pretraining applies, the budget is four members with twenty iterations each; where deployment adaptation applies, the budget is five trials. Initial sandboxes are shared. Matching these budgets controls major setup differences but does not imply identical model-call counts, search behavior, or realized total compute across methods.

Setting-matched baselines.

Setting-matched implementations preserve the central mechanism of the cited baseline under the common code-policy interface and limited trial budget. Results therefore compare those controlled adaptations, rather than claiming to reproduce every capability of the original systems.

Methods.

Direct CaP writes a fresh task program from the initial toolbox. ENPIRE adaptation uses deployment feedback alone. Zero-Shot Sim2Real uses the full simulated discovery stage without deployment adaptation, following the transfer principle of DrEureka. ASPIRE builds a textual skill library in simulation and uses it for repair. No Discovery replaces open-ended probing with fixed-target optimization; Direct Repair retains the discovery toolbox but removes simulator replay, hypothesis generation, and candidate screening from adaptation.

Appendix D Additional results

The complete appendix tables retain per-task success counts, component ablations, all four model configurations, and the separate online-control latency test.

D.1 Per-task sim-to-sim results

Every cell counts successes among twenty-five scheduled evaluations. Plate performance is generally high after adaptation, while towel folds remain difficult: the rotated half-fold succeeds only eight times and the left-shifted half-fold eleven times. SimEX’s aggregate gains coexist with these failures and ties or slight losses to internal variants on selected barcode tasks.

Table 4: Per-task controlled sim-to-sim results, reported as successful evaluation trials out of 25.
Scene Task Direct CaP ENPIRE∗ Zero-Shot Sim2Real ASPIRE∗ SimEX (no discovery) SimEX (direct repair) SimEX (ours)
Plate to tote Same-side pair 1/251/25 4/254/25 1/251/25 0/250/25 16/2516/25 𝟐𝟏/𝟐𝟓21/25 𝟐𝟏/𝟐𝟓21/25
Diagonal pair 3/253/25 8/258/25 0/250/25 0/250/25 12/2512/25 14/2514/25 𝟐𝟒/𝟐𝟓24/25
Far outboard plate 17/2517/25 14/2514/25 12/2512/25 17/2517/25 15/2515/25 19/2519/25 𝟐𝟐/𝟐𝟓22/25
Far plate to near tote 9/259/25 18/2518/25 10/2510/25 19/2519/25 20/2520/25 𝟐𝟓/𝟐𝟓25/25 𝟐𝟓/𝟐𝟓25/25
Any three 0/250/25 6/256/25 0/250/25 0/250/25 𝟐𝟑/𝟐𝟓23/25 16/2516/25 𝟐𝟑/𝟐𝟓23/25
Shifted single 11/2511/25 5/255/25 𝟐𝟓/𝟐𝟓25/25 16/2516/25 6/256/25 23/2523/25 𝟐𝟓/𝟐𝟓25/25
Shifted diagonal pair 3/253/25 7/257/25 0/250/25 0/250/25 13/2513/25 20/2520/25 𝟐𝟓/𝟐𝟓25/25
Towel folding Rotated shift 0/250/25 8/258/25 5/255/25 0/250/25 16/2516/25 5/255/25 𝟐𝟏/𝟐𝟓21/25
Left-shifted shift 4/254/25 4/254/25 5/255/25 11/2511/25 14/2514/25 10/2510/25 𝟏𝟗/𝟐𝟓19/25
Right-shifted shrink 5/255/25 14/2514/25 17/2517/25 20/2520/25 19/2519/25 15/2515/25 𝟐𝟒/𝟐𝟓24/25
Rotated shrink 10/2510/25 16/2516/25 12/2512/25 18/2518/25 20/2520/25 19/2519/25 𝟐𝟏/𝟐𝟓21/25
Left-shifted half fold 7/257/25 7/257/25 8/258/25 𝟏𝟏/𝟐𝟓11/25 9/259/25 10/2510/25 𝟏𝟏/𝟐𝟓11/25
Rotated half fold 1/251/25 3/253/25 5/255/25 3/253/25 2/252/25 5/255/25 𝟖/𝟐𝟓8/25
Rotated double fold 3/253/25 8/258/25 11/2511/25 4/254/25 2/252/25 4/254/25 𝟏𝟒/𝟐𝟓14/25
Barcode scanning Wall-side bottle 0/250/25 0/250/25 0/250/25 0/250/25 2/252/25 17/2517/25 𝟐𝟓/𝟐𝟓25/25
Across-tote bottle 0/250/25 0/250/25 0/250/25 1/251/25 7/257/25 17/2517/25 𝟐𝟒/𝟐𝟓24/25
Far upright bottle 0/250/25 0/250/25 0/250/25 15/2515/25 17/2517/25 16/2516/25 𝟐𝟒/𝟐𝟓24/25
Hidden-barcode block 1/251/25 0/250/25 0/250/25 10/2510/25 8/258/25 𝟐𝟓/𝟐𝟓25/25 24/2524/25
Wall-side prism 0/250/25 0/250/25 0/250/25 6/256/25 6/256/25 3/253/25 𝟏𝟗/𝟐𝟓19/25
Two bottles 0/250/25 0/250/25 2/252/25 0/250/25 2/252/25 𝟐𝟎/𝟐𝟓20/25 19/2519/25
Bottle + block 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 4/254/25 𝟏𝟒/𝟐𝟓14/25

D.2 Per-task ablation results

The barcode component ablations remove population search, capability-bank replay, alternative repairs, or hypothesis generation. All reduce aggregate success, but no single component improves every row: several simplified variants reach 25/25 on far-upright or hidden-block cases, and removing the capability bank reaches 21/25 on the wall-side prism versus full SimEX’s 19/25. The single-process population ablation also uses fewer total discovery iterations.

Table 5: Per-task barcode scanning ablations, reported as successful evaluation trials out of 25.
Task SimEX (ours) No population (Stage 1) No capability bank (Stage 1) No repair alternatives (Stage 2) No hypothesis generation (Stage 2)
Wall-side bottle 25/2525/25 17/2517/25 15/2515/25 24/2524/25 23/2523/25
Across-tote bottle 24/2524/25 20/2520/25 14/2514/25 25/2525/25 20/2520/25
Far upright bottle 24/2524/25 25/2525/25 25/2525/25 25/2525/25 20/2520/25
Hidden-barcode block 24/2524/25 25/2525/25 24/2524/25 25/2525/25 25/2525/25
Wall-side prism 19/2519/25 15/2515/25 21/2521/25 5/255/25 13/2513/25
Two bottles 19/2519/25 12/2512/25 15/2515/25 18/2518/25 16/2516/25
Bottle + block 14/2514/25 1/251/25 7/257/25 14/2514/25 9/259/25

D.3 Per-task results across coding agents

Across the seven barcode tasks, the full tables yield 149/175 successes for Fable 5.1, 134/175 for Opus 5, 117/175 for GPT-5.5, and 144/175 for GPT-6 Astra. SimEX is strongest in aggregate for each configuration, with more than fifty percentage points over the best external baseline. Individual tasks still show ties, internal-ablation advantages, and substantial failures—for example, the wall-side prism and bottle-plus-block tasks remain difficult for some models.

Original figure
Figure 11: Across all coding agents, SimEX consistently outperforms the strongest baseline methods (marked with darker diamond) by a large margin.
Table 6: Per-task barcode scanning results for the all-Fable 5.1 configuration used in the main experiments, reported as successful evaluation trials out of 25.
Task Direct CaP ENPIRE∗ Zero-Shot Sim2Real ASPIRE∗ SimEX (no discovery) SimEX (direct repair) SimEX (ours)
Wall-side bottle 0/250/25 0/250/25 0/250/25 0/250/25 2/252/25 17/2517/25 𝟐𝟓/𝟐𝟓25/25
Across-tote bottle 0/250/25 0/250/25 0/250/25 1/251/25 7/257/25 17/2517/25 𝟐𝟒/𝟐𝟓24/25
Far upright bottle 0/250/25 0/250/25 0/250/25 15/2515/25 17/2517/25 16/2516/25 𝟐𝟒/𝟐𝟓24/25
Hidden-barcode block 1/251/25 0/250/25 0/250/25 10/2510/25 8/258/25 𝟐𝟓/𝟐𝟓25/25 24/2524/25
Wall-side prism 0/250/25 0/250/25 0/250/25 6/256/25 6/256/25 3/253/25 𝟏𝟗/𝟐𝟓19/25
Two bottles 0/250/25 0/250/25 2/252/25 0/250/25 2/252/25 𝟐𝟎/𝟐𝟓20/25 19/2519/25
Bottle + block 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 4/254/25 𝟏𝟒/𝟐𝟓14/25
Table 7: Per-task barcode scanning results for the end-to-end Opus 5 configuration, reported as successful evaluation trials out of 25.
Task Direct CaP ENPIRE∗ Zero-Shot Sim2Real ASPIRE∗ SimEX (no discovery) SimEX (direct repair) SimEX (ours)
Wall-side bottle 0/250/25 4/254/25 0/250/25 0/250/25 0/250/25 16/2516/25 𝟐𝟓/𝟐𝟓25/25
Across-tote bottle 0/250/25 1/251/25 0/250/25 0/250/25 3/253/25 20/2520/25 𝟐𝟓/𝟐𝟓25/25
Far upright bottle 0/250/25 8/258/25 0/250/25 0/250/25 7/257/25 20/2520/25 𝟐𝟐/𝟐𝟓22/25
Hidden-barcode block 0/250/25 2/252/25 0/250/25 0/250/25 17/2517/25 𝟐𝟒/𝟐𝟓24/25 23/2523/25
Wall-side prism 0/250/25 0/250/25 0/250/25 0/250/25 4/254/25 2/252/25 𝟏𝟎/𝟐𝟓10/25
Two bottles 0/250/25 0/250/25 3/253/25 0/250/25 1/251/25 21/2521/25 𝟐𝟑/𝟐𝟓23/25
Bottle + block 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 5/255/25 𝟔/𝟐𝟓6/25
Table 8: Per-task barcode scanning results for the end-to-end GPT-5.5 configuration, reported as successful evaluation trials out of 25.
Task Direct CaP ENPIRE∗ Zero-Shot Sim2Real ASPIRE∗ SimEX (no discovery) SimEX (direct repair) SimEX (ours)
Wall-side bottle 0/250/25 0/250/25 0/250/25 0/250/25 2/252/25 19/2519/25 𝟐𝟏/𝟐𝟓21/25
Across-tote bottle 0/250/25 0/250/25 0/250/25 9/259/25 1/251/25 13/2513/25 𝟐𝟏/𝟐𝟓21/25
Far upright bottle 0/250/25 0/250/25 0/250/25 1/251/25 0/250/25 10/2510/25 𝟏𝟕/𝟐𝟓17/25
Hidden-barcode block 0/250/25 0/250/25 0/250/25 5/255/25 15/2515/25 𝟐𝟓/𝟐𝟓25/25 24/2524/25
Wall-side prism 0/250/25 0/250/25 0/250/25 𝟓/𝟐𝟓5/25 4/254/25 0/250/25 𝟓/𝟐𝟓5/25
Two bottles 0/250/25 0/250/25 7/257/25 0/250/25 0/250/25 𝟐𝟓/𝟐𝟓25/25 24/2524/25
Bottle + block 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 𝟓/𝟐𝟓5/25 𝟓/𝟐𝟓5/25
Table 9: Per-task barcode scanning results for the end-to-end GPT-6 Astra configuration, reported as successful evaluation trials out of 25.
Task Direct CaP ENPIRE∗ Zero-Shot Sim2Real ASPIRE∗ SimEX (no discovery) SimEX (direct repair) SimEX (ours)
Wall-side bottle 0/250/25 0/250/25 0/250/25 0/250/25 4/254/25 17/2517/25 𝟐𝟓/𝟐𝟓25/25
Across-tote bottle 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 17/2517/25 𝟐𝟒/𝟐𝟓24/25
Far upright bottle 0/250/25 0/250/25 0/250/25 0/250/25 6/256/25 20/2520/25 𝟐𝟓/𝟐𝟓25/25
Hidden-barcode block 0/250/25 0/250/25 0/250/25 10/2510/25 9/259/25 𝟐𝟓/𝟐𝟓25/25 𝟐𝟓/𝟐𝟓25/25
Wall-side prism 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 𝟖/𝟐𝟓8/25
Two bottles 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 17/2517/25 𝟐𝟎/𝟐𝟓20/25
Bottle + block 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 0/250/25 𝟏𝟕/𝟐𝟓17/25

D.4 Comparison with online agent control

The online-control baseline repeatedly emits low-level actions from camera-derived locations and proprioception, making about thirty model calls per rollout and waiting roughly forty minutes for responses. None of fifteen rollouts across three tasks completes within the allotted wall-clock budget. SimEX instead makes one roughly thirty-second synthesis call and runs code without model inference on the control path; all three validation programs finish. This comparison establishes a practical latency limitation in this setup, not that all future online models or all synthesized programs must behave similarly.

Appendix E Extended related work

Extended related work separates language-based robot interfaces, iterative agent optimization, environment generation, and transfer between simulators and physical systems.

LLM-based Robot Control.

Language models can generate plans/code or orchestrate supplied skills, but these approaches often assume a useful physical interface already exists. SimEX focuses on experimentally constructing and calibrating that reusable capability layer above a fixed low-level robot API.

Agentic Optimization.

Autoresearch iteratively edits and evaluates executable artifacts. SimEX applies this to persistent robot code and documentation, with open-ended self-generated simulation tests followed by deployment grounding. No fixed global discovery metric is prescribed, although local predicates and focused objectives still drive each optimization cycle.

Generative Simulation and Environment Design.

Environment-design systems generate scenes, curricula, and supervision for policy learning. SimEX uses related generation capabilities to stress-test and improve a symbolic code toolbox through tasks and executable predicates in an approximate target workspace.

Sim-to-Real Transfer.

Domain randomization and simulation pretraining followed by real adaptation motivate the two-stage structure. SimEX changes the learned artifact to reusable code and returns to simulation after deployment feedback to distinguish hypotheses and screen repairs. This complements rather than eliminates the need to identify consequential simulation mismatches.

Concept demonstration

Open image ↗