Overview
The opening overview retains the video-to-simulation-to-robot workflow and examples of the eleven evaluated tasks.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/ac/acc6601e530ae2e8a54baac9ba4550966ad8b35ea4c34273dffb017b6063fdb2.webp)
At a glance
DexAgent turns one egocentric human video and a task prompt into simulated robot trajectories and policy-training data. An agent selects or writes reusable reconstruction, motion, and verification functions, iterating until checks pass or its budget expires. Real policies trained on the resulting data average 63.6% success across eleven tasks, compared with 18.2% for the strongest listed baseline. Its persistent tool library reduces later processing cost, especially when object assets recur. Passing agent-designed simulation checks is not a guarantee of physical correctness.
I INTRODUCTION
Human demonstrations provide useful motion priors, but fixed retargeting pipelines struggle with different mechanisms, deformability, and long task sequences. DexAgent adapts four stages—semantic interpretation, reconstruction, trajectory generation, and data augmentation—to each task’s properties. It retains generated skills and verifiers for reuse. The main contribution is this feedback-driven workflow and library, rather than a new low-level controller or a proof that its generated checks fully capture task validity.
II RELATED WORK
Prior work learns from human videos, co-trains on human and robot data, or reconstructs interactions in simulation before retargeting. Examples include Do as I Do [20] and SPIDER [21]. Agentic robot programming instead composes or synthesizes executable skills. DexAgent applies that approach to reconstruction and human-motion conversion, allowing different strategies for individual subgoals.
III METHODOLOGY
Each stage alternates generation and verification. Only passing outputs advance; budget exhaustion, including a 100-agent-turn cap where applicable, is recorded as failure. Semantic properties guide later tools and checks. The framework can use different vision-language backbones, while simulation and the evolving library provide execution feedback.
III-A Skills and Verifiers
Skills generate assets, poses, or motion; verifiers accept or reject them using simulation behavior. Both are typed Python functions. Revisions fork existing functions, preserving earlier versions, and new functions are retained. Reusable parameterization is an intended design constraint, but generalization depends on the quality of generated code and tests.
III-B Semantic Understanding
The agent identifies objects and candidate properties from the prompt, checks them against extracted video frames, and can use depth, gravity, or human-pose tools. It decomposes the demonstration into small subgoals with acting hands and observable outcomes. A shared scene-semantics file records object categories, required behavior, and task order in natural language; numeric physical parameters are resolved later.
III-C Property-Based Simulation Reconstruction
Reconstruction begins with property-specific checks: rigid-object shape and stability, articulated joint axes and travel, or deformable topology. The agent edits parameterized object assets until checks pass, then jointly retests properties and three additional variants. Rendered-mask overlap guides placement. Robot configuration is supplied; camera parameters are estimated or may be manually provided. Matching silhouettes and passing these tests constrain the model but cannot uniquely identify hidden geometry, friction, compliance, or true mechanics.
| Rigid | Articulated | |||||
| Method | F-5 | F-10 | CD | F-5 | F-10 | CD |
| HO [4] | 0.28 | 0.51 | 3.86 | 0.29 | 0.47 | 1.30 |
| IHOI [35] | 0.42 | 0.70 | 2.70 | 0.32 | 0.47 | 1.47 |
| HORSE [23] | 0.26 | 0.45 | 6.69 | 0.19 | 0.34 | 1.91 |
| MCC-HO [31] | 0.52 | 0.78 | 1.36 | 0.35 | 0.55 | 1.21 |
| G-HOP [36] | 0.69 | 0.91 | 0.63 | 0.07 | 0.09 | 1.23 |
| FoundationPose [30] | 0.71 | 0.91 | 0.49 | 0.40 | 0.60 | 1.26 |
| Any6D [7] | 0.71 | 0.91 | 0.50 | 0.38 | 0.60 | 1.24 |
| Do as I Do [20] | 0.72 | 0.91 | 0.49 | 0.40 | 0.61 | 1.25 |
| DexAgent (ours) | 0.83 | 0.96 | 0.29 | 0.47 | 0.68 | 1.08 |
III-D Trajectory Optimization
For each subgoal, the agent defines success conditions and chooses retargeting, contact optimization, or generated grasp code. Candidate poses undergo task-specific tests such as lifting, shaking, sliding, or rotation. Inverse kinematics and generated code connect valid poses. Earlier segments can be revised if they make later ones infeasible, and whole-trajectory checks assess limits and consistency before accepting the episode.
III-E Data Generation and Policy Training
Object and robot initial states are varied in simulation; each resulting episode must pass the same checks. VOID [14] removes demonstrated hands and objects from the source video, and rendered robot/object regions are composited and retextured in Blender [1]. Additional wrist views supply policy inputs. This creates diverse synthetic episodes from one human demonstration, but their realism remains dependent on reconstruction, rendering, and verifier coverage.
IV Data Quality Experiments
Data-quality tests separate human–object reconstruction from simulated robot-motion fidelity. MuJoCo [28] provides the physics environment for these experiments.
IV-A Human–Object Interaction Reconstruction
On HOI4D [9], DexAgent has the best listed scores for rigid and articulated reconstruction. Rigid-object F-5 improves from the strongest baseline’s 0.72 to 0.83 and Chamfer distance from 0.49 to 0.29; articulated F-5 improves from 0.40 to 0.47 and Chamfer distance from 1.21 to 1.08. These support reconstruction quality in the evaluated dataset; geometric metrics alone do not certify mechanism fidelity.
IV-B Robot Trajectory Generation Quality
OakInk [33] trajectories count as successful when mean object-position error is below 0.1 m and rotation error below 0.5 rad. DexAgent reaches 85.7% versus 81.0% for the strongest baseline, a 4.7-point absolute or 5.8% relative gain. Position error remains 0.03 m while rotation error drops from 0.15 to 0.12 rad. This simulation criterion is distinct from completing a real manipulation task.
| Data | Cup | Giftbox | Drawer | Rope Knot | Scissors | Drawing | Computer | Bottle | Biology | Battery | Multi-object | Total |
| Dex-retargeting [24] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 1/11 |
| Do as I Do [20] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 1/11 |
| Spider [21] | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | 6/11 |
| TopoRetarget [32] | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 3/11 |
| Egoinfinity [29] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 1/11 |
| V2D [16] | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 5/11 |
| GPT-6: Astra [19] | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | 5/11 |
| DexAgent (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 11/11 |
| Data | Cup | Giftbox | Drawer | Rope Knot | Scissors | Drawing | Computer | Bottle | Biology | Battery | Multi-object | Average |
| Dex-retargeting [24] | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0.0% |
| Do as I Do [20] | 50% | 50% | 10% | 0% | 0% | 0% | 10% | 0% | 0% | 0% | 0% | 10.9% |
| Spider [21] | 70% | 40% | 10% | 0% | 30% | 20% | 20% | 0% | 0% | 10% | 0% | 18.2% |
| TopoRetarget [32] | 60% | 10% | 0% | 0% | 10% | 10% | 0% | 0% | 0% | 0% | 0% | 8.2% |
| Egoinfinity [29] | 60% | 20% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 7.3% |
| V2D [16] | 70% | 40% | 30% | 0% | 0% | 10% | 10% | 0% | 0% | 0% | 30% | 17.3% |
| GPT-6: Astra [19] | 60% | 40% | 50% | 0% | 0% | 30% | 0% | 0% | 0% | 0% | 0% | 16.4% |
| DexAgent (ours) | 90% | 70% | 80% | 50% | 80% | 40% | 50% | 70% | 40% | 60% | 70% | 63.6% |
V Physical Experiments
Each real task starts from one complete 640×480, 30-frame-per-second egocentric human video and a task description. Reconstruction and augmentation use MuJoCo or Isaac Sim according to task requirements. Replay tests and learned-policy tests answer different questions.
V-A Evaluation Setup
The bimanual station uses two Sharpa hands, an ego camera, and two wrist cameras. Replay is attempted ten times, but a task receives a success mark if even one attempt works. For policy evaluation, 500 simulated episodes fine-tune π0.5 [22], followed by ten trials with randomized initial states. Baselines keep supported components and receive common implementations for missing stages, with shared augmentation and policy training. This controls parts of the pipeline, but the adapted systems are not identical to every baseline’s original deployment setup.
V-B Results Analysis
DexAgent obtains at least one successful replay on all eleven tasks, versus six for SPIDER. That 11/11 is task coverage under an any-of-ten criterion, not a 100% rollout success rate. Learned policies average 63.6% success, compared with SPIDER’s 18.2% and V2D’s 17.3%. Gains include 80% drawer and 50% rope-knot success; individual DexAgent task rates range from 40% to 90%. Ten trials per task give coarse estimates and do not establish reliability across arbitrary conditions.
In a 100-video run, the library reaches 85 skills and 168 verifiers by sample 78. Processing takes 4.3 hours for the first video, about 2.2 hours with skill reuse, and 13.1 minutes when assets can also be reused. Total time is 210 hours, or 2.1 hours per episode. The approximately 17× and 15.1× speedups compare the favorable asset-reuse regime with baseline per-episode times; they are not average speedups across the entire fresh run. Recurring-object examples are deliberately placed last, so ordering and reuse opportunities matter.
VI CONCLUSION
Adaptive tool choice, simulation feedback, and reuse improve the tested data-generation pipeline. Semantic errors, incomplete verifiers, and simulation-to-reality mismatch remain. Incorrect tools or assumptions can persist in the library, and force-sensitive interactions may fail despite passing checks. Stronger verification and reinforcement-learning fallbacks are proposed future work.
ACKNOWLEDGMENT
The authors acknowledge Sharpa’s equipment support and discussions with collaborators and the Stanford Vision and Learning community.
Appendix
The appendix retains augmented data examples, three scene-semantics listings, additional reconstruction and motion examples, and the complete library breakdown.
VI-A Additional Robot Data Generation Examples
For each task, 1,500 initial-state seeds are sampled and tested for reachability and feasible execution. Five hundred passing episodes are selected; more seeds are sampled if needed. Each is rendered from ego and both wrist views. The restored Figure 6 includes every example row and augmented-view column, showing how one human sequence becomes multiple synthetic training episodes.
VI-B Example Outputs from the Semantic Understanding Stage
The restored listings show how task semantics become object-property requirements, ordered subgoals, hand roles, and success conditions. They are the paper’s abbreviated examples with original ellipses, not complete executable scene specifications.
VI-B 1 Drawing and Wiping
The drawing example distinguishes persistent marker traces from eraser removal. The right hand writes digits while the left holds or uses the eraser; the final wiping goal removes only digit four and preserves the others. These properties must be represented and checked beyond generic rigid-body geometry.
## scene_semantics.json for drawing and wiping sample.
{
...,
"task_summary": "Write 1, 2, 3, and 4 with the right hand; erase only 4 with the left hand; return both tools
.",
"objects": [
{
"id": "board", "name": "Framed whiteboard",
"type": "rigid", ...,
"properties": [
...,
"Marker contact produces visible dark strokes that persist after the tip lifts.",
"Sliding eraser contact removes contacted strokes while leaving unwiped strokes visible."
], ...
},
{
"id": "marker", ...,
"type": "rigid", ...,
"properties": [
...,
"Moving the tip against the board leaves a visible dark trace."
], ...
},
{
"id": "eraser", ...,
"type": "rigid", ...,
"properties": [
...,
"Its broad wiping face removes marker strokes through sliding contact with the board."
], ...
}
],
...,
"subgoals": [
...,
{
...,
"acting_hand": "right",
"action": "Write digit 1.",
...,
"hand_roles": {
"left": "Holds the eraser at the left side.",
"right": "Moves the marker tip along the stroke."
},
"goal": "A dark 1 is visible on the board."
},
...,
{
...,
"acting_hand": "right",
"action": "Write digit 4 to the right of 3.",
...,
"goal": "The board displays 1, 2, 3, and 4 in order."
},
...,
{
...,
"acting_hand": "left",
"action": "Wipe repeatedly over digit 4.",
...,
"hand_roles": {
"left": "Slides the eraser back and forth over 4.",
"right": "Holds the marker away from the board."
},
"goal": "Digit 4 is no longer visibly readable; 1, 2, and 3 remain."
},
...
],
...
}VI-B 2 Open the Bottle
The bottle example requires a screw cap to detach only after turning and its attached tube to clear the mouth. It assigns stabilization and rotation to different hands. The listing explicitly distinguishes intended liquid transfer from what is observed: the pouring posture is visible, but liquid flow or level change is unresolved. That uncertainty should not be converted into a verified pouring outcome.
## scene_semantics.json for open the bottle sample.
{
...,
"task_summary": "Remove the screw cap, direct a pouring motion into the mug, and return the uncapped bottle
upright.",
"objects": [
{
"id": "bottle", ...,
"type": "articulated",
"parts": ["body", "mouth", "screw_cap", "attached_tube"],
"properties": [
...,
"The cap turns relative to the body before detaching from the threaded mouth.",
...,
"A long tube is carried with the cap; it must clear the mouth before the cap is moved aside.",
...
], ...
},
{
"id": "mug", ...,
"type": "rigid", ...,
"properties": [
"Open-topped receptacle with a rigid body and a stable resting base.",
...,
"Remains on the tabletop; neither hand grasps or stabilizes it."
], ...
}
],
...,
"subgoals": [
...,
{
...,
"acting_hand": "both",
"action": "Loosen the cap with repeated turns.",
...,
"hand_roles": {
"left": "Resists rotation of the body.",
"right": "Turns the cap and readjusts its grip as needed."
},
"goal": "Cap is loosened sufficiently for removal."
},
{
...,
"acting_hand": "both",
"action": "Lift off the cap assembly.",
...,
"hand_roles": {
"left": "Keeps the body upright and steady.",
"right": "Lifts the cap and its attached tube clear."
},
"goal": "Cap is detached and the tube clears the bottle mouth."
},
...,
{
...,
"acting_hand": "left",
"action": "Tilt and hold the bottle in a pouring posture.",
...,
"hand_roles": {
"left": "Tilts the bottle while aiming its mouth into the mug.",
"right": "Remains by the cap without manipulating the mug."
},
"goal": "Intended task outcome: water transfers from the bottle into the mug.",
"observed_outcome": "Pouring posture is visible; a liquid stream or level change is not clearly resolved."
},
...
],
...
}VI-B 3 Place Toy in Drawer
The drawer example specifies a prismatic joint and constrained off-axis motion, then sequences opening, toy placement, release, and closing. Its travel estimate is identified as visual rather than a measured limit. The original listing preserves these distinctions and hand assignments.
## scene_semantics.json for place toy in drawer sample.
{
...,
"objects": [
{
"id": "drawer_unit",
"type": "articulated",
"parts": ["housing", "sliding_tray", "front_handle"],
"appearance": {
"housing": "Gray",
"front_panel": "Gray",
"tray_interior": "Dark gray",
"handle": "Black, front-mounted pull handle"
},
"properties": [
"A single-DOF prismatic joint connects the tray to the housing.",
"Bidirectional translation along the housing’s longitudinal axis: outward to open and inward to close.",
"Relative rotation and translation perpendicular to the sliding axis are constrained.",
"Observed opening travel is approximately one housing depth; a rough visual estimate, not a measured
mechanical travel limit."
],
...
},
{"id": "toy", "name": "Small yellow plush toy", ...},
...
],
"subgoals": [
...,
{
"acting_hand": "left",
"action": "Pull the drawer outward.",
"object_refs": ["drawer_unit"],
"goal": "Drawer interior and the free front placement region are accessible."
},
{
"acting_hand": "right",
"action": "Pick up the toy.",
"object_refs": ["toy"],
"goal": "Toy is lifted clear of the tabletop."
},
{
"acting_hand": "right",
"action": "Move the toy over the open drawer.",
"object_refs": ["toy", "drawer_unit"],
"goal": "Toy is above the drawer interior, clear of the existing contents."
},
{
"acting_hand": "right",
"action": "Release the toy into the drawer.",
"object_refs": ["toy", "drawer_unit"],
"goal": "Toy rests in the front region of the tray without hand support."
},
...,
{
"acting_hand": "left",
"action": "Push the drawer closed using its front handle.",
"object_refs": ["drawer_unit"],
"goal": "Front panel returns to the closed position with the toy enclosed."
},
...
],
...
}VI-C Additional Simulation Reconstruction and Robot Trajectory Optimization Examples
Additional galleries follow rigid, articulated, and deformable tasks from human footage through reconstructed scenes to optimized robot motion. They illustrate the range of generated behaviors rather than adding independent success-rate measurements.
VI-D Current Library Tools Breakdown
The later library snapshot contains 103 skills and 188 verifiers, exceeding the earlier 100-video-run snapshot because task samples are also included. Six skill groups cover motion, reconstruction, perception, scene execution, grasping, and measurement. Six verifier groups cover task outcomes, grasp stress, mechanisms, episode consistency, scene conditions, and compilation/validity. The restored Figure 8 preserves the full counts and categories. Reusability is intended and demonstrated on selected sequences, not guaranteed by library size.