Overview
SimEX develops reusable robot code in an approximate simulator, then uses a few deployment trials to correct the simulator and screen repairs before trying the robot again.
At a glance
Simulation-Integrated Robotics AutoResearch optimizes a toolbox of perception/control functions and usage notes, rather than directly training a neural policy. Open-ended simulated probing builds initial capabilities; a second stage alternates limited physical evidence with simulator alignment and candidate-repair tests. Controlled experiments span three manipulation families and four coding-agent configurations. On one dual-arm platform, the final system succeeds in twenty-six of thirty physical evaluation trials after five adaptation trials per task. The reported ten minutes measures robot interaction, not total agent reasoning, simulation, reset, or optimization time.
1 Introduction
Directly generated robot code can fail when physical details are unknown, while repeated real-world debugging is expensive. SimEX uses simulation as an experimental workspace in which a coding agent measures limitations, develops procedures, and tests revisions. Physical feedback then corrects relevant simulation mismatches. The approach aims to transfer reusable code and documented knowledge, but does not establish that arbitrary coarse simulators suffice for every robot or task.
2 Method
Preparation builds an approximate scene and initial interface. Stage one expands the toolbox through simulated tasks; stage two concentrates physical trials into a short deployment phase with simulated diagnosis between trials. Human operators still initialize, monitor, and reset the physical scene.
2.1 Problem Setup
The persistent artifact consists of toolbox.py, containing perception/control primitives, and skill.md, describing their use and limitations. Each task receives a fresh one-shot Code as Policies call that writes an executable program; the task program is discarded after execution while shared toolbox updates persist.
The fixed low-level interface supplies raw color/depth images, proprioception, inverse kinematics, reachability queries, Cartesian/joint targets, and gripper commands. It provides no task-level manipulation skills or privileged scene state to the execution program. Agent-generated simulation success predicates can inspect simulator state for scoring, which is a separate privileged evaluation role. The coding agent controls optimization decisions; the harness executes and reports outcomes.
2.2 Preparation: Constructing the simulation sandbox
A coding agent reconstructs the workspace from an image and object description using an existing empty YAM station template and predefined mesh library. It chooses/configures assets and initializes perception code and interface documentation. This is an automated approximate reconstruction within supplied infrastructure, not calibration or scene understanding from an entirely unstructured starting point.
2.3 Stage 1: Open-ended capability discovery in simulation
Each cycle broadly probes self-generated tasks, placements, object properties, and robot starts, with agent-written success predicates. The agent chooses one systematic weakness and fixes a focused task set while hill-climbing toolbox revisions. Replay from a bank of previously solved tasks discourages regressions, after which probing seeks another weakness. The absence of a single prescribed training metric does not mean there is no scoring: each generated test has an executable predicate whose correctness remains part of the agent-created setup.
Four independent optimization members each run twenty iterations with their own toolbox and capability bank. At probe boundaries they can read short experiment descriptions and reimplement ideas, but cannot copy another implementation. This diversifies discovery while preserving distinct trajectories; it is not unconstrained file sharing or a proven guarantee of broad capability coverage.
2.4 Stage 2: Simulation-assisted few-trial adaptation on the robot
Each deployment rollout records camera evidence and the executed script/transcript. The agent replays that exact script in simulation and adjusts the simulator until the behavior relevant to the failure is reproduced according to its qualitative judgment. It then proposes distinct hypotheses and separate toolbox repairs, screens the current toolbox and candidates on the fixed-script replay plus additional test tasks, and chooses a repair or retains the old toolbox.
Candidate ranking is not an automatic highest-score rule: the agent judges the trustworthiness of simulation given remaining mismatch. One physical trial can therefore support many offline experiments, but neither simulator alignment nor repair selection is certified to match the real system. The updated simulator persists into later screens.
3 Experiments
The experiments combine repeated transfer between independently implemented simulators, a smaller hardware study, component ablations, and a comparison to online agent control. Their protocols and evidential scales differ.
3.1 Experimental setup
The common platform is a dual-arm YAM with one overhead and two wrist cameras. Seven evaluation tasks per family cover plate transport, towel manipulation, and barcode scanning. In controlled experiments, adaptive methods have five deployment trials per run, then freeze the toolbox and generate fresh programs for five evaluation rollouts. Five runs yield twenty-five scheduled evaluations per method/task. Family means weight tasks equally, and confidence intervals bootstrap runs within tasks. The default source configuration uses Fable 5.1 with maximum reasoning in simulation discovery and medium reasoning during adaptation.
3.2 Controlled sim-to-sim evaluation
Rigid-object experiments transfer from MuJoCo to Isaac Sim; deformable experiments transfer from FLASH to MJWarp Flex. The target simulator’s state and source are hidden from the agent, while camera evidence and task outcomes remain available. This gives a controlled transfer gap, not a replacement for all hardware testing.
The raw tables give SimEX 165/175 plate evaluations, 118/175 towel evaluations, and 149/175 barcode evaluations, corresponding to about 94%, 67%, and 85%. It matches or exceeds every external baseline on each of the twenty-one tasks, but does not exceed every internal ablation on every task. For example, direct repair scores one additional success on the hidden-barcode block and two-bottle cases. Both stage-level ablations are worse in family averages.
Actual adaptation-rollout curves show that the initial simulated toolbox can start near zero transfer success, yet one deployment trial enables a substantial repair. Simulation-guided repair extracts more value from that feedback in the reported curves. These are deployment-iteration outcomes, distinct from the final frozen-toolbox evaluation counts.
3.3 Real-robot evaluation
Hardware testing uses one representative task per family: stow four plates, half-fold a towel, and scan/stow two objects. After five adaptation trials, roughly ten minutes of robot interaction per task, SimEX succeeds in 10/10, 8/10, and 8/10 held-out physical trials. The strongest baseline totals three successes across the thirty evaluations; every baseline fails the two-object barcode task. A human resets the scene between trials. These are promising results on three tasks and one platform, without the five-independent-run hardware replication used in the controlled simulator study.
3.4 Ablations and further analysis
Removing population search, replay, multiple repair alternatives, or explicit hypotheses lowers aggregate barcode performance. Individual tasks can improve or tie under an ablation. The no-population condition uses one process rather than four, so reduced total exploration budget accompanies reduced diversity and prevents attributing the entire drop to diversity alone.
Replacing both optimizer and program-writing model preserves SimEX’s aggregate advantage across the four reported agent configurations. Barcode success varies from about 67% with GPT-5.5 to 82% with GPT-6 Astra; this is an empirical comparison under the paper’s settings, not a timeless model ranking. A separate three-task test finds one pre-execution program-synthesis call more practical than serial online inference: all three generated validation programs finish, while none of fifteen online-control rollouts finishes within budget.
4 Related Work
SimEX combines robot program generation, iterative coding-agent improvement, generated simulation tasks, and sim-to-real adaptation. Its distinctive use of simulation is to develop a persistent code toolbox and diagnose candidate repairs between scarce deployment trials, rather than solely produce demonstrations or optimize a fixed neural controller.
5 Conclusion
Approximate simulation plus limited deployment grounding can support useful robot capability development in the tested setting. Limits include a single hardware platform, manual scene resets, substantial optimization compute, dependence on supplied low-level interfaces/templates, and agent judgment in self-generated scoring, simulator alignment, and repair choice. The results support the two-stage workflow, without demonstrating fully autonomous physical operation or universal transfer from arbitrary simulations.
Appendix A Implementation details
Implementation details retain budgets, the exact one-shot program-writing prompt, and measured coding-agent overhead.
A.1 Hyperparameters
Each of four discovery members runs twenty optimization iterations, with twenty-four to thirty-two broad probes, about ten focused configurations, and four to six replay tasks per candidate. Adaptation uses five robot rollouts, four candidate repairs per iteration, and five agent-written test scenes. Evaluation uses five runs and five rollouts per run. These are experiment budgets, not the number of physical evaluation trials in the separate hardware table.
| Phase | Hyperparameter | Value |
| Stage 1 & 2 | Coding-agent orchestrator | Fable 5.1 |
| Stage 1 & 2 | Policy-writing model | Fable 5.1 |
| Stage 1 | Reasoning effort | max |
| Stage 2 | Reasoning effort | medium |
| Stage 1 | Independent population members | 4 |
| Stage 1 | Optimization iterations per member | 20 |
| Stage 1 | Broad probe configurations per iteration | 24–32 |
| Stage 1 | Focused configurations per selected sub-goal | 10 |
| Stage 1 | Capability-bank replay configurations per candidate | 4–6 |
| Stage 2 | Robot rollouts per iteration | 1 |
| Stage 2 | Repair candidates per iteration | 4 |
| Stage 2 | Adaptation trials per run | 5 |
| Stage 2 | Agent-written test scenes per iteration | 5 |
| Evaluation | Independent runs per method–task pair | 5 |
| Evaluation | I.i.d. evaluation trials per run | 5 |
A.2 Policy-writing prompt
The shared text-only prompt concatenates skill documentation, toolbox source, task name/description, and exact plan/script markers. It allows programs to bypass toolbox helpers while forbidding privileged object-state access. The model must produce a plan of at most two hundred words and complete Python code in one turn; tool use or multiple turns voids a trial rather than triggering a retry. Both original PDF prompt portions are retained as native vectors, preserving slots, markers, colors, and line structure that the HTML conversion otherwise leaves as unresolved macros.
A.3 Agent thinking time and token usage
For barcode optimization, the reported average discovery cost per population member is 840 orchestrator calls, 5.4 hours of agent thinking, and 1.98 million output tokens including 1.16 million reasoning tokens. Adaptation averages 124 calls, thirty-one minutes of thinking, and 161,000 output tokens per run. Cache-read inputs are separately large. These costs exclude interpreting the ten-minute robot-interaction figure as total training time; after optimization, one task’s program synthesis is around thirty seconds.
| Stage 1 (per member) | Stage 2 (per run) | |
|---|---|---|
| Orchestrator calls | 840 | 124 |
| Agent thinking time | 5.4 h | 31 min |
| Output tokens (reasoning) | 1.98M (1.16M) | 161k (64k) |
| Cache-read input tokens | 155M | 19.5M |
Appendix B Evaluation task details
The evaluation families test different contact, reach, deformation, and logical-order requirements. All seven task definitions in each family are reflected in the retained initial-state figures and detailed notes below.
B.1 Plate to tote
Plates must be lifted at least five centimetres and end inside the tote, while unrequested plates remain outside. The seven variants include a same-side pair, a diagonal pair, a far outboard plate, a long far-plate-to-near-tote carry, any three of four plates, a perturbed single-plate scene, and a perturbed diagonal pair. The shifted tote is translated 3.5 centimetres, shifted laterally three centimetres, and rotated six degrees, with plate placements also perturbed. Flat plates require a geometry-aware rim grasp rather than a simple top-down pinch.
B.2 Towel folding
Seven towel variants cover rotated or left-shifted translation, right-shifted or rotated compaction, left-shifted or rotated half-folds, and a rotated double fold. Translation requires at least five centimetres of center motion; compaction permits at most eighty percent of flat footprint; half-fold variants use a sixty-two-percent threshold; double folding requires at most thirty-eight percent. The rotated half-fold additionally requires one side at most sixty-two percent and the other at least seventy-eight percent of the original length, lying flat rather than piled. Thus success is more specific than merely reducing projected area for every variant.
B.3 Barcode scanning
Barcode tasks require exposing and aligning the correct label before stowing, under scanner-window and tote-wall constraints. The seven cases are a wall-side bottle with inward label, an across-tote bottle with label facing away, a far upright bottle, a floor-facing hidden-label block, a wall-side floor-facing prism, two bottles, and a bottle-plus-block scene. Multi-object cases require both scans and both deliveries; grasp selection, reorientation, and action order all matter.
Appendix C Baseline details
Baselines use a shared interface, initial toolbox, task seeds, model settings, and evaluation protocol. Asterisks indicate methods reimplemented to fit this setting rather than original released systems run unchanged.
Controlled comparison.
All methods receive the same raw observations, calibration, kinematic queries, and generic motion wrappers without task-level skills. Where simulated pretraining applies, the budget is four members with twenty iterations each; where deployment adaptation applies, the budget is five trials. Initial sandboxes are shared. Matching these budgets controls major setup differences but does not imply identical model-call counts, search behavior, or realized total compute across methods.
Setting-matched baselines.
Setting-matched implementations preserve the central mechanism of the cited baseline under the common code-policy interface and limited trial budget. Results therefore compare those controlled adaptations, rather than claiming to reproduce every capability of the original systems.
Methods.
Direct CaP writes a fresh task program from the initial toolbox. ENPIRE adaptation uses deployment feedback alone. Zero-Shot Sim2Real uses the full simulated discovery stage without deployment adaptation, following the transfer principle of DrEureka. ASPIRE builds a textual skill library in simulation and uses it for repair. No Discovery replaces open-ended probing with fixed-target optimization; Direct Repair retains the discovery toolbox but removes simulator replay, hypothesis generation, and candidate screening from adaptation.
Appendix D Additional results
The complete appendix tables retain per-task success counts, component ablations, all four model configurations, and the separate online-control latency test.
D.1 Per-task sim-to-sim results
Every cell counts successes among twenty-five scheduled evaluations. Plate performance is generally high after adaptation, while towel folds remain difficult: the rotated half-fold succeeds only eight times and the left-shifted half-fold eleven times. SimEX’s aggregate gains coexist with these failures and ties or slight losses to internal variants on selected barcode tasks.
| Scene | Task | Direct CaP | ENPIRE∗ | Zero-Shot Sim2Real | ASPIRE∗ | SimEX (no discovery) | SimEX (direct repair) | SimEX (ours) |
|---|---|---|---|---|---|---|---|---|
| Plate to tote | Same-side pair | |||||||
| Diagonal pair | ||||||||
| Far outboard plate | ||||||||
| Far plate to near tote | ||||||||
| Any three | ||||||||
| Shifted single | ||||||||
| Shifted diagonal pair | ||||||||
| Towel folding | Rotated shift | |||||||
| Left-shifted shift | ||||||||
| Right-shifted shrink | ||||||||
| Rotated shrink | ||||||||
| Left-shifted half fold | ||||||||
| Rotated half fold | ||||||||
| Rotated double fold | ||||||||
| Barcode scanning | Wall-side bottle | |||||||
| Across-tote bottle | ||||||||
| Far upright bottle | ||||||||
| Hidden-barcode block | ||||||||
| Wall-side prism | ||||||||
| Two bottles | ||||||||
| Bottle + block |
D.2 Per-task ablation results
The barcode component ablations remove population search, capability-bank replay, alternative repairs, or hypothesis generation. All reduce aggregate success, but no single component improves every row: several simplified variants reach 25/25 on far-upright or hidden-block cases, and removing the capability bank reaches 21/25 on the wall-side prism versus full SimEX’s 19/25. The single-process population ablation also uses fewer total discovery iterations.
| Task | SimEX (ours) | No population (Stage 1) | No capability bank (Stage 1) | No repair alternatives (Stage 2) | No hypothesis generation (Stage 2) |
|---|---|---|---|---|---|
| Wall-side bottle | |||||
| Across-tote bottle | |||||
| Far upright bottle | |||||
| Hidden-barcode block | |||||
| Wall-side prism | |||||
| Two bottles | |||||
| Bottle + block |
D.3 Per-task results across coding agents
Across the seven barcode tasks, the full tables yield 149/175 successes for Fable 5.1, 134/175 for Opus 5, 117/175 for GPT-5.5, and 144/175 for GPT-6 Astra. SimEX is strongest in aggregate for each configuration, with more than fifty percentage points over the best external baseline. Individual tasks still show ties, internal-ablation advantages, and substantial failures—for example, the wall-side prism and bottle-plus-block tasks remain difficult for some models.
D.4 Comparison with online agent control
The online-control baseline repeatedly emits low-level actions from camera-derived locations and proprioception, making about thirty model calls per rollout and waiting roughly forty minutes for responses. None of fifteen rollouts across three tasks completes within the allotted wall-clock budget. SimEX instead makes one roughly thirty-second synthesis call and runs code without model inference on the control path; all three validation programs finish. This comparison establishes a practical latency limitation in this setup, not that all future online models or all synthesized programs must behave similarly.
Appendix E Extended related work
Extended related work separates language-based robot interfaces, iterative agent optimization, environment generation, and transfer between simulators and physical systems.
LLM-based Robot Control.
Language models can generate plans/code or orchestrate supplied skills, but these approaches often assume a useful physical interface already exists. SimEX focuses on experimentally constructing and calibrating that reusable capability layer above a fixed low-level robot API.
Agentic Optimization.
Autoresearch iteratively edits and evaluates executable artifacts. SimEX applies this to persistent robot code and documentation, with open-ended self-generated simulation tests followed by deployment grounding. No fixed global discovery metric is prescribed, although local predicates and focused objectives still drive each optimization cycle.
Generative Simulation and Environment Design.
Environment-design systems generate scenes, curricula, and supervision for policy learning. SimEX uses related generation capabilities to stress-test and improve a symbolic code toolbox through tasks and executable predicates in an approximate target workspace.
Sim-to-Real Transfer.
Domain randomization and simulation pretraining followed by real adaptation motivate the two-stage structure. SimEX changes the learned artifact to reusable code and returns to simulation after deployment feedback to distinguish hypotheses and screen repairs. This complements rather than eliminates the need to identify consequential simulation mismatches.