Literature
Display
← All papers

DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library

Youhui Wang · Yunzhu Li · Li Fei-Fei · Jiajun Wu · Huang Huang

Preprint posted · arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

The opening overview retains the video-to-simulation-to-robot workflow and examples of the eleven evaluated tasks.

[Uncaptioned image]
Fig. 1: DexAgent. An agentic Human2Sim2Robot framework that converts a single egocentric human video and a task prompt (Left) into robot training data. DexAgent uses property specific skills and verifiers from a self-evolving tool library (Middle Bottom) for each task. We evaluate DexAgent on 11 dexterous manipulation tasks spanning diverse object types and long-horizon interactions (Right), achieving the highest success rates among the evaluated methods.

At a glance

DexAgent turns one egocentric human video and a task prompt into simulated robot trajectories and policy-training data. An agent selects or writes reusable reconstruction, motion, and verification functions, iterating until checks pass or its budget expires. Real policies trained on the resulting data average 63.6% success across eleven tasks, compared with 18.2% for the strongest listed baseline. Its persistent tool library reduces later processing cost, especially when object assets recur. Passing agent-designed simulation checks is not a guarantee of physical correctness.

I INTRODUCTION

Human demonstrations provide useful motion priors, but fixed retargeting pipelines struggle with different mechanisms, deformability, and long task sequences. DexAgent adapts four stages—semantic interpretation, reconstruction, trajectory generation, and data augmentation—to each task’s properties. It retains generated skills and verifiers for reuse. The main contribution is this feedback-driven workflow and library, rather than a new low-level controller or a proof that its generated checks fully capture task validity.

II RELATED WORK

Prior work learns from human videos, co-trains on human and robot data, or reconstructs interactions in simulation before retargeting. Examples include Do as I Do [20] and SPIDER [21]. Agentic robot programming instead composes or synthesizes executable skills. DexAgent applies that approach to reconstruction and human-motion conversion, allowing different strategies for individual subgoals.

III METHODOLOGY

Each stage alternates generation and verification. Only passing outputs advance; budget exhaustion, including a 100-agent-turn cap where applicable, is recorded as failure. Semantic properties guide later tools and checks. The framework can use different vision-language backbones, while simulation and the evolving library provide execution feedback.

Refer to caption
Fig. 2: DexAgent converts a single egocentric human video and a task prompt into robot training data through four stages. Stage 1 decomposes the task into subgoals and identifies the manipulated objects and their key properties. Stage 2 reconstructs the objects in simulation using property-specific skills (blue) from the tool library, creating new skills when needed. Stage 3 generates robot-hand trajectories by combining optimization guided by human motion priors with code-generated motion. Stage 4 varies viewpoints and object states to produce diverse simulation data, inpaints the original human video, and retextures the augmented simulation videos for policy training.

III-A Skills and Verifiers

Skills generate assets, poses, or motion; verifiers accept or reject them using simulation behavior. Both are typed Python functions. Revisions fork existing functions, preserving earlier versions, and new functions are retained. Reusable parameterization is an intended design constraint, but generalization depends on the quality of generated code and tests.

III-B Semantic Understanding

The agent identifies objects and candidate properties from the prompt, checks them against extracted video frames, and can use depth, gravity, or human-pose tools. It decomposes the demonstration into small subgoals with acting hands and observable outcomes. A shared scene-semantics file records object categories, required behavior, and task order in natural language; numeric physical parameters are resolved later.

III-C Property-Based Simulation Reconstruction

Reconstruction begins with property-specific checks: rigid-object shape and stability, articulated joint axes and travel, or deformable topology. The agent edits parameterized object assets until checks pass, then jointly retests properties and three additional variants. Rendered-mask overlap guides placement. Robot configuration is supplied; camera parameters are estimated or may be manually provided. Matching silhouettes and passing these tests constrain the model but cannot uniquely identify hidden geometry, friction, compliance, or true mechanics.

TABLE I: Reconstruction accuracy on HOI4D, scored separately for rigid and articulated objects. Evaluation metrics are F-5 and F-10, which are F-scores at two distance thresholds, and CD, which is Chamfer distance. Best entry in each column in bold.
Rigid Articulated
Method F-5↑\uparrow F-10↑\uparrow CD↓\downarrow F-5↑\uparrow F-10↑\uparrow CD↓\downarrow
HO [4] 0.28 0.51 3.86 0.29 0.47 1.30
IHOI [35] 0.42 0.70 2.70 0.32 0.47 1.47
HORSE [23] 0.26 0.45 6.69 0.19 0.34 1.91
MCC-HO [31] 0.52 0.78 1.36 0.35 0.55 1.21
G-HOP [36] 0.69 0.91 0.63 0.07 0.09 1.23
FoundationPose [30] 0.71 0.91 0.49 0.40 0.60 1.26
Any6D [7] 0.71 0.91 0.50 0.38 0.60 1.24
Do as I Do [20] 0.72 0.91 0.49 0.40 0.61 1.25
DexAgent (ours) 0.83 0.96 0.29 0.47 0.68 1.08

III-D Trajectory Optimization

For each subgoal, the agent defines success conditions and chooses retargeting, contact optimization, or generated grasp code. Candidate poses undergo task-specific tests such as lifting, shaking, sliding, or rotation. Inverse kinematics and generated code connect valid poses. Earlier segments can be revised if they make later ones infeasible, and whole-trajectory checks assess limits and consistency before accepting the episode.

III-E Data Generation and Policy Training

Object and robot initial states are varied in simulation; each resulting episode must pass the same checks. VOID [14] removes demonstrated hands and objects from the source video, and rendered robot/object regions are composited and retextured in Blender [1]. Additional wrist views supply policy inputs. This creates diverse synthetic episodes from one human demonstration, but their realism remains dependent on reconstruction, rendering, and verifier coverage.

TABLE II: Human-to-robot retargeting quality on OakInk. Success is the share of clips whose converted trajectory passes the physics check. Ep​o​sE_{pos} and Er​o​tE_{rot} are the residual position and rotation gap from the demonstration, lower is better. The two indented rows ablate the strongest baseline.
Method Success %↑\uparrow Ep​o​sE_{pos} (m)↓\downarrow Er​o​tE_{rot} (rad)↓\downarrow
Dex-retargeting [24] 28.6 0.08 0.62
SPIDER [21] (mjwp) 71.4 0.04 0.57
SPIDER [21] (mjwp_act) 77.1 0.04 0.42
Do-as-I-Do [20] (Sharpa hand) 81.0 0.03 0.15
without transition reward 79.0 0.03 0.14
annealed sampling only 72.0 0.08 0.32
DexAgent (ours) 85.7 0.03 0.12

IV Data Quality Experiments

Data-quality tests separate human–object reconstruction from simulated robot-motion fidelity. MuJoCo [28] provides the physics environment for these experiments.

IV-A Human–Object Interaction Reconstruction

On HOI4D [9], DexAgent has the best listed scores for rigid and articulated reconstruction. Rigid-object F-5 improves from the strongest baseline’s 0.72 to 0.83 and Chamfer distance from 0.49 to 0.29; articulated F-5 improves from 0.40 to 0.47 and Chamfer distance from 1.21 to 1.08. These support reconstruction quality in the evaluated dataset; geometric metrics alone do not certify mechanism fidelity.

IV-B Robot Trajectory Generation Quality

OakInk [33] trajectories count as successful when mean object-position error is below 0.1 m and rotation error below 0.5 rad. DexAgent reaches 85.7% versus 81.0% for the strongest baseline, a 4.7-point absolute or 5.8% relative gain. Position error remains 0.03 m while rotation error drops from 0.15 to 0.12 rad. This simulation criterion is distinct from completing a real manipulation task.

Refer to caption
Fig. 3: Initial object-state distributions for policy evaluation. Object placements are randomized across trials for each task.
Refer to caption
Fig. 4: Processing cost falls as the library grows. Over 100 EgoDex samples the library reaches 85 skills and 168 verifiers. In terms of cost, V2D converting one sample takes 3.7 hours on average and GPT 6: Astra takes 3.3 hours on average at a flat rate, whereas DexAgent takes 2.1 hours on average in a fresh run, a 36.4% reduction comparing to GPT 6: Astra and a 43.2% reduction comparing to V2D.
Refer to caption
Fig. 5: Real experiments results on four selected tasks. Each row shows the original human demonstration (Column 1), its reconstructed simulation scene(Column 2), and representative frames from the resulting real-robot rollout (Columns 3–6). The examples span diverse manipulation behaviors, including deformable-object manipulation, multi-object interaction, precise insertion, and articulated object manipulation.
TABLE III: Real-world replay success across eleven tasks. Each converted trajectory is executed open loop, which scores the conversion rather than a learned policy. A task is marked successful if at least one of ten replay trials succeeds. GPT-6 Astra joined the benchmark after the first evaluation round.
Data Cup Giftbox Drawer Rope Knot Scissors Drawing Computer Bottle Biology Battery Multi-object Total
Dex-retargeting [24] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 1/11
Do as I Do [20] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 1/11
Spider [21] ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✗ 6/11
TopoRetarget [32] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 3/11
Egoinfinity [29] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 1/11
V2D [16] ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ 5/11
GPT-6: Astra [19] ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ 5/11
DexAgent (ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 11/11
TABLE IV: Real-world success rate on the same task set, measured as the percentage of ten closed-loop trials per task. Each policy is trained on the data converted and augmented by the same protocol. The average is over all eleven tasks.
Data Cup Giftbox Drawer Rope Knot Scissors Drawing Computer Bottle Biology Battery Multi-object Average
Dex-retargeting [24] 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.0%
Do as I Do [20] 50% 50% 10% 0% 0% 0% 10% 0% 0% 0% 0% 10.9%
Spider [21] 70% 40% 10% 0% 30% 20% 20% 0% 0% 10% 0% 18.2%
TopoRetarget [32] 60% 10% 0% 0% 10% 10% 0% 0% 0% 0% 0% 8.2%
Egoinfinity [29] 60% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% 7.3%
V2D [16] 70% 40% 30% 0% 0% 10% 10% 0% 0% 0% 30% 17.3%
GPT-6: Astra [19] 60% 40% 50% 0% 0% 30% 0% 0% 0% 0% 0% 16.4%
DexAgent (ours) 90% 70% 80% 50% 80% 40% 50% 70% 40% 60% 70% 63.6%

V Physical Experiments

Each real task starts from one complete 640×480, 30-frame-per-second egocentric human video and a task description. Reconstruction and augmentation use MuJoCo or Isaac Sim according to task requirements. Replay tests and learned-policy tests answer different questions.

V-A Evaluation Setup

The bimanual station uses two Sharpa hands, an ego camera, and two wrist cameras. Replay is attempted ten times, but a task receives a success mark if even one attempt works. For policy evaluation, 500 simulated episodes fine-tune π0.5 [22], followed by ten trials with randomized initial states. Baselines keep supported components and receive common implementations for missing stages, with shared augmentation and policy training. This controls parts of the pipeline, but the adapted systems are not identical to every baseline’s original deployment setup.

V-B Results Analysis

DexAgent obtains at least one successful replay on all eleven tasks, versus six for SPIDER. That 11/11 is task coverage under an any-of-ten criterion, not a 100% rollout success rate. Learned policies average 63.6% success, compared with SPIDER’s 18.2% and V2D’s 17.3%. Gains include 80% drawer and 50% rope-knot success; individual DexAgent task rates range from 40% to 90%. Ten trials per task give coarse estimates and do not establish reliability across arbitrary conditions.

In a 100-video run, the library reaches 85 skills and 168 verifiers by sample 78. Processing takes 4.3 hours for the first video, about 2.2 hours with skill reuse, and 13.1 minutes when assets can also be reused. Total time is 210 hours, or 2.1 hours per episode. The approximately 17× and 15.1× speedups compare the favorable asset-reuse regime with baseline per-episode times; they are not average speedups across the entire fresh run. Recurring-object examples are deliberately placed last, so ordering and reuse opportunities matter.

VI CONCLUSION

Adaptive tool choice, simulation feedback, and reuse improve the tested data-generation pipeline. Semantic errors, incomplete verifiers, and simulation-to-reality mismatch remain. Incorrect tools or assumptions can persist in the library, and force-sensitive interactions may fail despite passing checks. Stronger verification and reinforcement-learning fallbacks are proposed future work.

ACKNOWLEDGMENT

The authors acknowledge Sharpa’s equipment support and discussions with collaborators and the Stanford Vision and Learning community.

Appendix

The appendix retains augmented data examples, three scene-semantics listings, additional reconstruction and motion examples, and the complete library breakdown.

VI-A Additional Robot Data Generation Examples

For each task, 1,500 initial-state seeds are sampled and tested for reachability and feasible execution. Five hundred passing episodes are selected; more seeds are sampled if needed. Each is rendered from ego and both wrist views. The restored Figure 6 includes every example row and augmented-view column, showing how one human sequence becomes multiple synthetic training episodes.

Original Figure 6, restored from PDF with all panels and labels.
Fig. 6: Examples of robot data generation in Stage 4. Column 1 shows the original human demonstration. Column 2 shows the egocentric view after removing the demonstrator and rendering the robot and objects into the scene, after being retextured in Blender. Column 3-4 shows the corresponding generated left- and right-wrist camera views used for policy training. Column 5-6 shows examples generated from the same demonstration with different random seeds, illustrating the data diversification happened during data generation.

VI-B Example Outputs from the Semantic Understanding Stage

The restored listings show how task semantics become object-property requirements, ordered subgoals, hand roles, and success conditions. They are the paper’s abbreviated examples with original ellipses, not complete executable scene specifications.

VI-B 1 Drawing and Wiping

The drawing example distinguishes persistent marker traces from eraser removal. The right hand writes digits while the left holds or uses the eraser; the final wiping goal removes only digit four and preserves the others. These properties must be represented and checked beyond generic rigid-body geometry.

## scene_semantics.json for drawing and wiping sample.

{
    ...,
    "task_summary": "Write 1, 2, 3, and 4 with the right hand; erase only 4 with the left hand; return both tools
         .",
    "objects": [
      {
        "id": "board", "name": "Framed whiteboard",
        "type": "rigid", ...,
        "properties": [
          ...,
          "Marker contact produces visible dark strokes that persist after the tip lifts.",
          "Sliding eraser contact removes contacted strokes while leaving unwiped strokes visible."
        ], ...
      },
      {
        "id": "marker", ...,
        "type": "rigid", ...,
        "properties": [
          ...,
          "Moving the tip against the board leaves a visible dark trace."
        ], ...
      },
      {
        "id": "eraser", ...,
        "type": "rigid", ...,
        "properties": [
          ...,
          "Its broad wiping face removes marker strokes through sliding contact with the board."
        ], ...
      }
    ],
    ...,
    "subgoals": [
      ...,
      {
        ...,
        "acting_hand": "right",
        "action": "Write digit 1.",
        ...,
        "hand_roles": {
          "left": "Holds the eraser at the left side.",
          "right": "Moves the marker tip along the stroke."
        },
        "goal": "A dark 1 is visible on the board."
      },
      ...,
      {
        ...,
        "acting_hand": "right",
        "action": "Write digit 4 to the right of 3.",
        ...,
        "goal": "The board displays 1, 2, 3, and 4 in order."
      },
      ...,
      {
        ...,
        "acting_hand": "left",
        "action": "Wipe repeatedly over digit 4.",
        ...,
        "hand_roles": {
          "left": "Slides the eraser back and forth over 4.",
          "right": "Holds the marker away from the board."
        },
        "goal": "Digit 4 is no longer visibly readable; 1, 2, and 3 remain."
      },
      ...
    ],
    ...
}

VI-B 2 Open the Bottle

The bottle example requires a screw cap to detach only after turning and its attached tube to clear the mouth. It assigns stabilization and rotation to different hands. The listing explicitly distinguishes intended liquid transfer from what is observed: the pouring posture is visible, but liquid flow or level change is unresolved. That uncertainty should not be converted into a verified pouring outcome.

## scene_semantics.json for open the bottle sample.

{
    ...,
    "task_summary": "Remove the screw cap, direct a pouring motion into the mug, and return the uncapped bottle
         upright.",
    "objects": [
      {
        "id": "bottle", ...,
        "type": "articulated",
        "parts": ["body", "mouth", "screw_cap", "attached_tube"],
        "properties": [
          ...,
          "The cap turns relative to the body before detaching from the threaded mouth.",
          ...,
          "A long tube is carried with the cap; it must clear the mouth before the cap is moved aside.",
          ...
        ], ...
      },
      {
        "id": "mug", ...,
        "type": "rigid", ...,
        "properties": [
          "Open-topped receptacle with a rigid body and a stable resting base.",
          ...,
          "Remains on the tabletop; neither hand grasps or stabilizes it."
        ], ...
      }
    ],
    ...,
    "subgoals": [
      ...,
      {
        ...,
        "acting_hand": "both",
        "action": "Loosen the cap with repeated turns.",
        ...,
        "hand_roles": {
          "left": "Resists rotation of the body.",
          "right": "Turns the cap and readjusts its grip as needed."
        },
        "goal": "Cap is loosened sufficiently for removal."
      },
      {
        ...,
        "acting_hand": "both",
        "action": "Lift off the cap assembly.",
        ...,
        "hand_roles": {
          "left": "Keeps the body upright and steady.",
          "right": "Lifts the cap and its attached tube clear."
        },
        "goal": "Cap is detached and the tube clears the bottle mouth."
      },
      ...,
      {
        ...,
        "acting_hand": "left",
        "action": "Tilt and hold the bottle in a pouring posture.",
        ...,
        "hand_roles": {
          "left": "Tilts the bottle while aiming its mouth into the mug.",
          "right": "Remains by the cap without manipulating the mug."
        },
        "goal": "Intended task outcome: water transfers from the bottle into the mug.",
        "observed_outcome": "Pouring posture is visible; a liquid stream or level change is not clearly resolved."
      },
      ...
    ],
    ...
}

VI-B 3 Place Toy in Drawer

The drawer example specifies a prismatic joint and constrained off-axis motion, then sequences opening, toy placement, release, and closing. Its travel estimate is identified as visual rather than a measured limit. The original listing preserves these distinctions and hand assignments.

## scene_semantics.json for place toy in drawer sample.

{
    ...,
    "objects": [
      {
        "id": "drawer_unit",
        "type": "articulated",
        "parts": ["housing", "sliding_tray", "front_handle"],
        "appearance": {
          "housing": "Gray",
          "front_panel": "Gray",
          "tray_interior": "Dark gray",
          "handle": "Black, front-mounted pull handle"
        },
        "properties": [
          "A single-DOF prismatic joint connects the tray to the housing.",
          "Bidirectional translation along the housing’s longitudinal axis: outward to open and inward to close.",
          "Relative rotation and translation perpendicular to the sliding axis are constrained.",
          "Observed opening travel is approximately one housing depth; a rough visual estimate, not a measured
              mechanical travel limit."
        ],
        ...
      },
      {"id": "toy", "name": "Small yellow plush toy", ...},
      ...
    ],
    "subgoals": [
      ...,
      {
        "acting_hand": "left",
        "action": "Pull the drawer outward.",
        "object_refs": ["drawer_unit"],
        "goal": "Drawer interior and the free front placement region are accessible."
      },
      {
        "acting_hand": "right",
        "action": "Pick up the toy.",
        "object_refs": ["toy"],
        "goal": "Toy is lifted clear of the tabletop."
      },
      {
        "acting_hand": "right",
        "action": "Move the toy over the open drawer.",
        "object_refs": ["toy", "drawer_unit"],
        "goal": "Toy is above the drawer interior, clear of the existing contents."
      },
      {
        "acting_hand": "right",
        "action": "Release the toy into the drawer.",
        "object_refs": ["toy", "drawer_unit"],
        "goal": "Toy rests in the front region of the tray without hand support."
      },
      ...,
      {
        "acting_hand": "left",
        "action": "Push the drawer closed using its front handle.",
        "object_refs": ["drawer_unit"],
        "goal": "Front panel returns to the closed position with the toy enclosed."
      },
      ...
    ],
    ...
}

VI-C Additional Simulation Reconstruction and Robot Trajectory Optimization Examples

Additional galleries follow rigid, articulated, and deformable tasks from human footage through reconstructed scenes to optimized robot motion. They illustrate the range of generated behaviors rather than adding independent success-rate measurements.

Refer to caption
Fig. 7: Additional examples of property-based simulation reconstruction in Stage 2 and robot trajectory optimization in simulation in Stage 3. Column 1 shows the recorded human demonstration, and Column 2 shows the corresponding simulation scene reconstructed in Stage 2. Column 3-5 show representative frames t0t_{0}–t3t_{3} from the robot trajectory optimized in simulation from Stage 3.

VI-D Current Library Tools Breakdown

The later library snapshot contains 103 skills and 188 verifiers, exceeding the earlier 100-video-run snapshot because task samples are also included. Six skill groups cover motion, reconstruction, perception, scene execution, grasping, and measurement. Six verifier groups cover task outcomes, grasp stress, mechanisms, episode consistency, scene conditions, and compilation/validity. The restored Figure 8 preserves the full counts and categories. Reusability is intended and demonstrated on selected sequences, not guaranteed by library size.

Original Figure 8, restored from PDF with all panels and labels.
Fig. 8: Breakdown of the current DexAgent tool library. The library contains 103 skills and 188 verifiers grouped by functionality. Skills cover perception, reconstruction, grasp and pose generation, scene execution, measurement, and trajectory generation, while verifiers check task completion, grasp stability, reconstructed mechanisms, full-episode consistency, task-specific scene conditions, and scene validity. These skills and verifiers are expected to generalize to new and unseen samples, and the library is expected to continue to grow as DexAgent processes more new samples.
Concept demonstration

Open image ↗