Overview
Project page. This edition follows the complete twenty-one-page, 2 November 2026 v2, including all eight figures, six numbered tables, four numbered equations, task recipes and 49 references.
At a glance
DuoBench evaluates dual-arm manipulation on eleven simulated tasks and four corresponding physical tasks. Its main contribution is diagnostic: monotonic task stages, conditional transition pass rates and confidence intervals expose where policies fail even when final success is rare. Fifty demonstrations per task/domain support comparisons of ACT, π₀.₅ and X-VLA. Early interaction, coordinated object motion and independent use of both arms remain difficult under this fixed setup and training protocol.
1 Introduction
Binary success obscures the difference between failing to grasp, transfer an object or finish a coordinated motion. DuoBench combines a functional taxonomy, reproducible simulation/physical task recipes, teleoperated datasets and stage annotations. Reproducibility means other labs can recreate tasks and collect comparable data; it does not claim a trained policy will transfer unchanged between laboratories.
2 Related Work
Manipulation benchmarks vary in simulator, task scope, sensing and evaluation. DuoBench builds on this infrastructure while emphasizing different coordination roles for two arms and repeatable task definitions across domains.
Single-Arm Manipulation in Simulation
RLBench, Meta-World, ManiSkill and LIBERO provide diverse manipulation environments. Household, long-horizon and multimodal benchmarks broaden the challenge; unified interfaces ease model comparisons. Their breadth motivates a complementary analysis of which bimanual interactions fail.
Dual-Arm Manipulation in Simulation
TWIN/RLBench2, RoboTwin, BiCoord and dexterous/humanoid benchmarks cover two-arm tasks with different emphases. RoboEval already uses stagewise progress plus coordination, efficiency and safety metrics. DuoBench’s contribution is the particular combination of coordination categories, stage-conditional diagnostics and reproducible physical counterparts, not the invention of all stage-based evaluation.
Real-World and Hybrid Benchmarks
FMB and FurnitureBench use reproducible objects, while RoboMind supplies multi-embodiment data and RoboArena distributes physical evaluation. DuoBench prioritizes dual-arm task recipes and a common collection/inference stack, spanning coordination patterns rather than only object substitutions.
3 Methodology
The Robot Control Stack represents environments as wrapped Markov decision processes. Wrappers modify observations/actions, record demonstrations and expose task progress. The formal dataset pairs actions with delayed camera/proprioceptive observations, including an instruction and head/left-wrist/right-wrist images. Joint-space actions have sixteen coordinates: seven joints and one gripper command per arm.
The learning objective is behavior cloning. Raw teleoperation supplies Cartesian commands, later converted to joint targets by observation-initialized inverse kinematics; the abstract sixteen-dimensional interface should be read together with this conversion pipeline.
Equations 1–2
| (1) |
| (2) |
3.1 Task Stages
Task stages are milestones that can only increase. Conditions on the simulator state advance the recorded stage; independent-arm subtasks are merged with “at least one” and “all” conditions. Mean stage progress and the distribution of each episode’s highest stage remain informative when no complete task succeeds. Because progress is monotonic, losing an earlier physical configuration does not reduce its historical stage.
Conditional pass rate ρₖ asks how often an episode that reached stage k also reached k+1. It avoids hiding difficult late transitions behind early failures. Final success factors into these conditional probabilities without an independence assumption; the smallest defined pass rate names the bottleneck. If no episode reaches a stage, its later rate is undefined rather than evidence of zero competence.
Wilson 95% intervals quantify finite-rollout uncertainty. They do not capture variation across training seeds, task designs or laboratories. The stage count is a hand-designed progress measure, so equal normalized increments need not represent equal physical difficulty.
Equations 3–4
| (3) |
| (4) |
3.2 Franka Duo Setup
Two Franka FR3 arms carry Robotiq 2F-85 grippers. A Zed Mini main camera and two RealSense D405 wrist cameras provide three views. Mounts follow the manufacturer’s arrangement and specifications; the matching MuJoCo scene uses Menagerie and manufacturer assets. This is one fixed embodiment/sensing configuration.

Original selectable table
| Asymmetric Support | Bimanual Manipulation | Sequential Handoff | Parallel Execution |
|---|---|---|---|
| Hinge-Chest (3) | Ball-Maze (4) | Transfer-Cube (4) | Bin-Sort (5) |
| Spring-Door (3) | Carry-Pot (3) | Transfer-Reorient (4) | |
| Pour-Marbles (6) | Block-Balance (5) | Transfer-Gate (4) | |
| Join-Blocks (3) |
3.3 Bimanual Task Taxonomy
The taxonomy adapts Krebs and Asfour to four functional categories: asymmetric support, where one arm creates conditions for the other; joint manipulation of the same object through a closed kinematic chain; sequential handoff; and independent parallel execution. The last category permits simultaneous or sequential solutions and does not require direct arm interdependence.
Eleven tasks instantiate these roles. Hinge-Chest, Ball-Maze, Transfer-Cube and Bin-Sort are also reproduced physically, one per category. Each task randomizes at least object placement; the appendix retains exact stages, horizon statistics, printability and randomization. The taxonomy is a useful selected test set, not exhaustive coverage of bimanual behavior.
3.4 Task Composition, Data Collection and Evaluation
A MuJoCo MJSpec composer creates scenes; reset and stage wrappers access simulator state to randomize tasks and score milestones. Environments register with Gymnasium. Physical assets follow the same recipes, but a human operator judges stages on hardware.
The IRIS VR interface supports simulated and physical teleoperation. Recording simulator states enables replay under new visuals without new demonstrations. VLAgents runs policies and records trials with seeded initialization. Such replay is an available tool, not an augmentation used in the reported experiments.
4 Results
The comparison uses ACT, π₀.₅ and X-VLA. All execute thirty absolute joint-target actions before the next observation/prompt: one second open loop at 30 Hz. Trials start from a common home pose, with object positions drawn from the demonstration distribution. Simulation resets share a seed; physical placements are matched manually. Conclusions therefore depend on this observation cadence and in-distribution setup.
Original selectable table
| Task | ACT | X-VLA | |||||||
|---|---|---|---|---|---|---|---|---|---|
| SR [CI] | bottleneck stage () | SR [CI] | bottleneck stage () | SR [CI] | bottleneck stage () | ||||
| Asymmetric Support | |||||||||
| Hinge-Chest | 26.7 [22,32] | 0.55 | cube + lid (0.39) | 26.7 [22,32] | 0.39 | lid or cube (0.56) | 12.0 [9,16] | 0.30 | cube + lid (0.37) |
| Spring-Door | 22.3 [18,27] | 0.50 | box + door (0.33) | 19.7 [16,25] | 0.44 | box + door (0.39) | 13.7 [10,18] | 0.38 | box + door (0.37) |
| Pour-Marbles | 0.3 [0,2] | 0.12 | pour all (0.23) | 0.0 [0,1] | 0.08 | pour all (0.00) | 0.0 [0,1] | 0.07 | place cups (0.00) |
| Bimanual Manipulation | |||||||||
| Carry-Pot | 54.0 [48,60] | 0.70 | grasp + lift (0.55) | 26.0 [21,31] | 0.51 | grasp + lift (0.30) | 12.3 [9,17] | 0.40 | grasp + lift (0.14) |
| Ball-Maze | 8.0 [5,12] | 0.71 | ball to goal (0.09) | 5.0 [3,8] | 0.67 | ball to goal (0.06) | 2.0 [1,4] | 0.71 | ball to goal (0.02) |
| Block-Balance | 0.0 [0,1] | 0.12 | grasp rects (0.00) | 0.0 [0,1] | 0.06 | grasp rects (0.00) | 0.0 [0,1] | 0.15 | release (0.00) |
| Join-Blocks | 2.3 [1,5] | 0.17 | connect (0.06) | 0.0 [0,1] | 0.16 | to wall (0.00) | 0.0 [0,1] | 0.20 | to wall (0.00) |
| Sequential Handoff | |||||||||
| Transfer-Cube | 10.0 [7,14] | 0.14 | pick cube (0.20) | 13.0 [10,17] | 0.28 | pick cube (0.47) | 9.3 [7,13] | 0.17 | pick cube (0.33) |
| Transfer-Gate | 42.7 [37,48] | 0.49 | pick box (0.59) | 23.3 [19,28] | 0.35 | pick box (0.49) | 26.3 [22,32] | 0.36 | pick box (0.51) |
| Transfer-Reorient | 0.3 [0,2] | 0.35 | insert (0.01) | 1.3 [1,3] | 0.36 | insert (0.04) | 0.0 [0,1] | 0.30 | insert (0.00) |
| Parallel Execution | |||||||||
| Bin-Sort | 0.7 [0,2] | 0.11 | pick 2nd (0.09) | 1.7 [1,4] | 0.14 | pick 2nd (0.16) | 1.0 [0,3] | 0.13 | pick 2nd (0.21) |
4.1 Datasets
Fifty demonstrations for each of eleven simulated and four physical tasks yield 750 episodes. Raw recordings contain three 1,280×720 camera streams, joint/Cartesian states, instructions and domain-specific robot/simulator state. Conversion to LeRobot resizes images to 224×224, flattens state/action vectors and resolves Cartesian commands into joint targets.
The source reports 442,907 frames and an average of about 350 frames/12 seconds per episode. Dividing the stated total by 750 instead gives about 591 frames/19.7 seconds at 30 Hz; no distinction between these denominators is explained. The original figures are preserved rather than silently reconciled.
4.2 Evaluation in Simulation
ACT trains a separate policy on fifty demonstrations per task; the VLAs share one instruction-conditioned policy trained on all 550 simulated episodes. Each task/policy receives 300 rollouts, with per-task cutoffs calibrated from demonstrations and no image/replay augmentation. This compares the stated specialist and multitask training regimes, not equal total data or training compute.
ACT reaches 54.0% on Carry-Pot and 42.7% on Transfer-Gate, versus π₀.₅’s 26.0%/23.3% and X-VLA’s 12.3%/26.3%. Transfer-Cube is much lower at 10.0%/13.0%/9.3%; the authors suggest the gate constrains the handover configuration, but the cross-task contrast alone does not isolate that cause.
All three score zero on Block-Balance; Bin-Sort success is 0.7%/1.7%/1.0%. Seventeen of 33 task-policy combinations have a grasp-establishment bottleneck. Carry-Pot and Ball-Maze instead bottleneck after establishing contact, when both arms must move the shared object. Ball-Maze shows why progress matters: normalized progress 0.67–0.71 coexists with only 2–8% final success.
Unconditional failure-stage bars and conditional pass-rate profiles answer different questions. X-VLA can acquire the Transfer-Reorient peg more often yet struggle with handoff; Table 2 identifies final insertion as the minimum conditional pass rate. Every profile is retained in Appendix A.
4.3 Evaluation in Real-World
Physical evaluation retrains ACT per task and the VLAs jointly on the 200 physical demonstrations, then tests fifteen rollouts per task/policy. All three fail Hinge-Chest. π₀.₅ completes Transfer-Cube in 6/15 trials (40%) and Bin-Sort in 3/15 (20%); X-VLA completes Ball-Maze in 3/15, while ACT completes Ball-Maze and Bin-Sort once each.
Grasping and shared-object motion recur as difficulties across domains, but exact rankings and rates differ. Ball-Maze policies often establish the two-arm grasp without guiding the ball successfully. Transfer-Cube acquisition is poor for ACT/X-VLA on hardware, whereas π₀.₅ picks reliably enough that handover becomes its bottleneck. Four tasks with fifteen trials each support qualitative diagnostic comparisons, not a general simulation-to-reality calibration guarantee.
4.4 Mixed Training and Sim-to-Real Gap
Mixed ACT training uses one hundred demonstrations per replicated task; mixed VLA training uses all 750 episodes. Table 3 compares domain-specific policies on their own domain with mixed policies on both domains. It does not report a simulation-only policy evaluated zero-shot on hardware.
Real-world average success changes from about 3% to 7% for ACT, while π₀.₅ remains 15% and X-VLA remains 5%, with task-level trade-offs. π₀.₅ improves Transfer-Cube from 40% to 53% but falls on Bin-Sort from 20% to 7%; X-VLA falls on Ball-Maze from 20% to 7%. These changes involve only a few successes in fifteen trials.
Across the four replicated simulation tasks, mixed training reduces mean success from 11.3% to 9.8% for ACT, 11.6% to 4.8% for π₀.₅ and 6.1% to 3.6% for X-VLA. X-VLA shares one domain embedding across simulation and reality, a proposed explanation rather than an ablated cause. Naive mixing does not consistently improve both domains, and physical mean progress can decline even when average success is unchanged.
Original selectable table
| Model | ACT | X-VLA | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Train dataset | Sim | Real | Mixed | Sim | Real | Mixed | Sim | Real | Mixed | |||
| Eval domain | Sim | Real | Sim | Real | Sim | Real | Sim | Real | Sim | Real | Sim | Real |
| Hinge-Chest | 26.7 (0.55) | 0 (0.04) | 26.7 (0.56) | 0 (0.18) | 26.7 (0.39) | 0 (0.00) | 4.7 (0.08) | 0 (0.07) | 12.0 (0.30) | 0 (0.13) | 8.7 (0.19) | 0 (0.04) |
| Ball-Maze | 8.0 (0.71) | 7 (0.52) | 3.7 (0.69) | 13 (0.43) | 5.0 (0.67) | 0 (0.50) | 6.0 (0.69) | 0 (0.30) | 2.0 (0.71) | 20 (0.60) | 1.3 (0.66) | 7 (0.12) |
| Transfer-Cube | 10.0 (0.14) | 0 (0.00) | 8.3 (0.13) | 0 (0.00) | 13.0 (0.28) | 40 (0.53) | 8.0 (0.23) | 53 (0.60) | 9.3 (0.17) | 0 (0.02) | 4.0 (0.15) | 7 (0.13) |
| Bin-Sort | 0.7 (0.11) | 7 (0.28) | 0.7 (0.15) | 13 (0.37) | 1.7 (0.14) | 20 (0.47) | 0.7 (0.14) | 7 (0.29) | 1.0 (0.13) | 0 (0.21) | 0.3 (0.10) | 7 (0.20) |
| Overall | 11.3 (0.38) | 3 (0.21) | 9.8 (0.38) | 7 (0.25) | 11.6 (0.37) | 15 (0.37) | 4.8 (0.28) | 15 (0.31) | 6.1 (0.33) | 5 (0.24) | 3.6 (0.28) | 5 (0.12) |
5 Limitations
Physical stage labels require human judgment; taxonomy and milestone definitions reflect designer choices. Evaluation uses one hardware/camera setup, a small demonstration regime, one default training seed and only fifteen physical trials per cell. Neither inter-lab transfer, visual robustness nor broader embodiment coverage is established. The common one-second open-loop action interval also limits how directly these results generalize to more reactive controllers.
6 Conclusion
DuoBench supplies reproducible dual-arm tasks and diagnostic measurements that distinguish grasp acquisition, coordinated object motion, transfer and parallel action failures. Its data show substantial headroom under the tested training and control protocol. More task varieties, tactile-relevant challenges, richer sensing and improved cross-domain training remain future directions.
Acknowledgments
The authors acknowledge assistance with physical evaluation and support from GeniusRobot, the German research ministries, Robotics Institute Germany and NHR@FAU/BayernKI computing resources.
Appendix A Extended Results
A.1 Task Progress over Time
Figure 5 plots average reached stage over normalized rollout time for all eleven simulated tasks. Early rises indicate earlier milestone acquisition and plateaus indicate stalled progress. The recorded stage never decreases, so these curves cannot directly measure the loss of an already achieved physical configuration. Time normalization also prevents reading the horizontal axis as equal wall-clock duration across tasks.
A.2 Failure-Stage Distributions
The physical failure-stage bars retain the distribution of highest reached stage over fifteen trials, with green indicating complete success. They expose how early attrition differs across policies, while remaining distinct from conditional pass probabilities among episodes that reached a particular milestone.
A.3 Pass-Rate Profiles
All eleven simulated pass-rate profiles include Wilson intervals and named outgoing subinstructions. Curves stop when no rollout reaches the next stage. Late-stage estimates may be based on far fewer than 300 examples and can have wide intervals even though the initial evaluation budget is the same.
A.4 Real-World Results
Table 4 adds confidence intervals, progress and bottlenecks for real-only training. Zero successes out of fifteen has a Wilson interval rounded to 0–20%; six successes gives 40% with interval 20–64%. These broad intervals matter when interpreting apparently large differences between physical policies.
Original selectable table
| Task | ACT | X-VLA | |||||||
|---|---|---|---|---|---|---|---|---|---|
| SR [CI] | bottleneck stage () | SR [CI] | bottleneck stage () | SR [CI] | bottleneck stage () | ||||
| Hinge-Chest | 0 [0,20] | 0.04 | cube + lid (0.00) | 0 [0,20] | 0.00 | lid or cube (0.00) | 0 [0,20] | 0.13 | cube + lid (0.00) |
| Ball-Maze | 7 [1,30] | 0.52 | ball to goal (0.10) | 0 [0,20] | 0.50 | ball to goal (0.00) | 20 [7,45] | 0.60 | ball to goal (0.27) |
| Transfer-Cube | 0 [0,20] | 0.00 | pick cube (0.00) | 40 [20,64] | 0.53 | hand over (0.67) | 0 [0,20] | 0.02 | both grippers (0.00) |
| Bin-Sort | 7 [1,30] | 0.28 | pick 2nd (0.17) | 20 [7,45] | 0.47 | pick 2nd (0.44) | 0 [0,20] | 0.21 | place 2nd (0.00) |
Appendix B Training Details
LeRobot v0.5.1 and Accelerate train with bfloat16, one observation step, a default seed and no image augmentation. ACT uses an ImageNet-pretrained ResNet-18, 150,000 steps, effective batch 64 and one H100, without temporal ensembling. π₀.₅ and X-VLA fully fine-tune their public checkpoints for 40,000 steps: effective batches 64/192 on four H200/four H100 GPUs. Vision/language encoders remain trainable.
Table 5 preserves optimizer, schedule, clipping and normalization details. Chunk lengths are 100/50/30, but all execute only thirty actions before replanning. Joint targets are clipped to arm limits and gripper commands binarized. X-VLA zero-pads the sixteen-dimensional action to twenty dimensions, uses squared-error loss on actual action dimensions including grippers, and shares its domain embedding across datasets. Different pretraining, update counts and compute complicate model-only causal comparisons.
Original selectable table
| ACT | X-VLA | ||
| Initialization | from scratch (ImageNet ResNet-18) | lerobot/pi05_base | lerobot/xvla-base |
| Trainable parameters | all | all (VLM incl. vision encoder, action expert) | all (encoders, transformer, soft prompts) |
| Training regime | one policy per task | one policy for all tasks | one policy for all tasks |
| Training steps | 150 000 | 40 000 | 40 000 |
| Batch size | 64 (64) | 16 (64) | 48 (192) |
| Optimizer | AdamW, | AdamW, | AdamW, |
| Learning rate | , constant | , cosine decay | , cosine decay |
| Warmup / decay steps | – | 1 000 / 30 000 (to ) | 1 000 / 30 000 (to ) |
| Weight decay | 0.01 | 0 | |
| Gradient clipping | 10.0 | 1.0 | 10.0 |
| Action chunk size | 100 | 50 | 30 |
| Normalization (state / action) | mean-std / mean-std | quantiles / quantiles | none / mean-std |
| Inference flow / denoising steps | – | 10 | 10 |
| Precision | bf16 | bf16 | bf16 |
| GPUs | 1 H100 | 4 H200 | 4 H100 |
Appendix C Visual Randomization
The replayer restores recorded simulator states and renders new backgrounds, textures and lighting. Figure 8 retains twenty example views, including MolmoSpaces scenes and combined changes. Instruction-referenced objects retain their appearance. These are capability demonstrations; no reported model is trained or evaluated under these visual shifts.

Appendix D 3D Printing and Assembly
Physical task assets are available for 3D printing. The reported setup uses a Bambulab X1C; Hinge-Chest attaches its lid with two M3×2 cm screws and nuts. The complete recipes support task recreation, while other setups still need their own calibration and comparable demonstrations.
Appendix E Tasks
The full three-page task catalog below preserves every scene image, description, stage sequence and randomization rule. “Steps” is the 99th-percentile demonstration length, not a success rate or universal execution budget.
Asymmetric support covers Hinge-Chest (hold a lid while inserting a cube; three stages), Spring-Door (hold a sprung microwave door; three) and Pour-Marbles (grasp/lift two cups, pour one/all marbles, replace cups; six). Randomization changes object positions and, for pouring, which cup contains marbles.
Shared-object tasks are Ball-Maze (contact, grasp, lift, guide the ball; four stages, ten board designs), Carry-Pot (contact handles, lift jointly, place on stove; three), Block-Balance (place a beam, load both ends and release without tipping; five) and Join-Blocks (grasp, connect, attach to a wall peg; three). Object positions vary; manipulation and contact constraints differ substantially across them.
Sequential tasks transfer a cube to a bowl, pass a box through a gate onto a mat, or hand over/reorient a peg for insertion; each has four stages with placements constrained by the two arm workspaces. Bin-Sort uses five milestones for acquiring and placing two cubes, allowing alternative execution orders. Only Hinge-Chest, Ball-Maze, Transfer-Cube and Bin-Sort have physical counterparts in this evaluation.
Original selectable table
| Scene | Task | Description | Category | 3D Printable | Real-World | Steps | Key Subtasks | Randomization |
|---|---|---|---|---|---|---|---|---|
|
Hinge-Chest | Holding the lid of a small chest open while inserting a cube. One arm must hold the lid while the other inserts the cube. | Asymmetric Support | ✓ | ✓ | open the chest OR pick up the cube, pick up the cube AND open the chest, place the cube inside the chest | cube position, chest position | |
|
Spring-Door | A spring-loaded microwave door requires one arm to hold it open while the other inserts a box. | Asymmetric Support | open the microwave OR pick up the box, pick up the box AND open the microwave, place the box inside the microwave | box position, microwave position | |||
|
Pour-Marbles | Two cups, one containing marbles. Both cups must be picked up, and the marbles must be poured into the other cup before both cups are placed back. | Asymmetric Support | grasp at least one cup, grasp both cups, lift both cups, pour at least one marble into the target cup, pour all marbles into the target cup, place both cups | cup position, which cup contains marbles | |||
|
Ball-Maze | Pick up a maze board with both arms and tilt it so a ball rolls into a target region. | Bimanual Manipulation | ✓ | ✓ | make contact with the maze, grasp the maze with both arms, lift the maze and move the ball out of the start area, guide the ball from the start area toward the goal | board position, one out of 10 different boards selected | |
|
Carry-Pot | Carry a pot using both side handles and place it on a stove. Both arms are needed to lift the pot. | Bimanual Manipulation | make contact with a pot handle, grasp the pot by both handles and lift it, place the pot onto the stove | stove and pot positions within one half of the table each | |||
|
Block-Balance | Consists of a red support cube, a beam, and two additional rectangular blocks. The beam needs to be placed on the cube, and then both blocks need to be placed on the beam simultaneously. | Bimanual Manipulation | ✓ | grasp the beam with one arm, place the beam on the small cube, grasp both rectangles, place both rectangles onto the beam, release and retract while keeping the beam balanced | position of all blocks | ||
|
Join-Blocks | Connect two movable blocks together and then attach them to a peg on a third stationary block. | Bimanual Manipulation | ✓ | approach and grasp both blocks, connect the two blocks, connect the assembled blocks to the wall | position of both movable blocks, each on one half of the table | ||
|
Transfer-Cube | Hand over a cube between arms before placing it into a bowl. | Sequential Handoff | ✓ | ✓ | pick up the cube, bring both grippers into contact with the cube, transfer the cube to the other gripper, place the cube correctly and release it | cube and bowl positions, each on one half of the table | |
|
Transfer-Gate | Hand over a box between arms before placing it onto a mat. The box has to be passed through a gate. | Sequential Handoff | ✓ | pick up the box, pass the box through the ring, grab the box with the other hand, place the box on the mat | box, mat and goal positions, box and mat on each half of the table while the gate is in between | ||
|
Transfer-Reorient | The right arm picks up a peg and hands it over to the left arm in such a way that the left arm is then able to insert it into a socket. | Sequential Handoff | ✓ | pick up the peg, bring both grippers into contact with the peg, transfer the peg to the other gripper, insert the peg into the matching socket | peg and socket positions, each on one half of the table | ||
|
Bin-Sort | Sort two cubes into matching bowls, testing simultaneous execution rather than direct cooperation. | Parallel Execution | ✓ | ✓ | pick up at least one cube, pick up the other cube or place the picked cube in the correct bowl, place at least one cube in the correct bowl, pick up the second cube, place the second cube in the correct bowl | cube and bowl positions |