Overview
The illustrated compositions retain a previously manipulated object while the same hand opens a door, presses a button, opens a microwave, or operates a switch.

At a glance
DexCompose combines two frozen dexterous policies by assigning different fingers to object retention and downstream interaction. Release tests identify candidate allocations; two learned residual controllers stabilize the held object and adapt the downstream policy.
Across sixteen simulated task pairs, composite success averages 77.43%. The base policies stay frozen, but composition is not training-free: the method learns a retention stabilizer and a task-pair-specific downstream residual.
1 Introduction
Simply switching between full-hand policies can destroy a grasp needed by the next task. DexCompose treats redundant fingers as resources: identify which fingers can be released, assign action ownership, and learn corrections within those assignments.
The framework reuses base policies across four retention skills and four interaction skills. It reduces interference in simulation; physical deployment and longer skill sequences are not demonstrated.
2 Related Work
Earlier work learns reusable dexterous controllers, sequences skills, or reserves fingers for future interactions. Residual learning adds corrections to an existing controller. DexCompose combines these ideas through explicit finger ownership and asymmetric residuals while preserving the original base-policy weights. Its diffusion base policies build on prior diffusion-policy learning. [8]
3 Method
The pipeline first discovers a finger allocation, then learns controllers that respect it.
3.1 Problem Formulation
Task A establishes a physical outcome, such as a grasp. Starting from a successful Task-A state, the controller must complete Task B while retaining the object throughout. Both base policies use the same wrist-and-finger action coordinates; the composition objective requires both preservation and interaction success.
Equations 1–3
| (1) |
| (2) |
| (3) |
3.2 Finger Attribution
Candidate masks assign each of five fingers either to preservation or to the next task. Stored successful grasps are replayed while released fingers gradually open. Retention and clean-release rates reveal whether the object stays held without hidden support from supposedly free fingers. An LLM selects a task-aware mask from those diagnostics, considering downstream finger utility.
Equations 4–6
| (4) |
| (5) |
| (6) |
3.3 Dual Residual Stabilizer
The preserved fingers follow the stored grasp reference plus a bounded residual. The wrist and released fingers follow the frozen downstream policy plus a separate bounded residual. Hard masks prevent either command from writing into the other’s assigned subspace. Near-zero residual initialization starts near the nominal behavior; base policies and masks remain fixed during residual learning.
Equations 7–12
| (7) |
| (8) |
| (9) |
| (10) |
| (11) |
| (12) |
4 Experiments
The main experiment evaluates all sixteen combinations of four retention and four interaction tasks, followed by preservation diagnostics and component ablations.
| Composite Success Rate (%) | ||||||
|---|---|---|---|---|---|---|
| First Task | Second Task | Frozen | Decomp. | FullRes. | Ours-ZS | Ours |
| GraspBall | OpenDoor | 82.69±8.33 | ||||
| PushButton | 85.58±7.50 | |||||
| OpenMicrowave | 73.08±9.25 | |||||
| TurnOnSwitch | 71.88±8.93 | |||||
| PourMug | OpenDoor | 75.72±8.08 | ||||
| PushButton | 85.58±7.28 | |||||
| OpenMicrowave | 62.74±8.21 | |||||
| TurnOnSwitch | 64.90±9.89 | |||||
| PickCan | OpenDoor | 79.09±6.04 | ||||
| PushButton | 85.34±6.92 | |||||
| OpenMicrowave | 72.36±11.82 | |||||
| TurnOnSwitch | 65.38±6.81 | |||||
| PickStick | OpenDoor | 86.30±5.78 | ||||
| PushButton | 86.54±4.84 | |||||
| OpenMicrowave | 87.26±7.81 | |||||
| TurnOnSwitch | 74.52±6.10 | |||||
| Mean | 77.43±2.64 | |||||
4.1 Experimental Setup
Tests use a floating Shadow Hand in Isaac Lab. Each of eight base skills is learned from 50 human demonstrations; each retention skill supplies 4,096 successful held states for composition training. Each task pair has 32 rollouts per seed across eight seeds. Success requires retaining Task A’s object throughout and completing Task B.
4.2 Comparison with Baselines
Mean success is 3.09% for frozen-grasp chaining, 45.61% for decomposed action spaces, 61.60% for a full-action residual, 69.23% for Ours-ZS, and 77.43% for full DexCompose. The improvement over the strongest external baseline is 15.83 percentage points. Ours-ZS already includes a learned retention stabilizer, so its name does not imply an entirely untrained composition system.
4.3 Ablation and Diagnostic Analysis
Removing the retention residual reduces average success to 9.13%; removing finger allocation yields 54.52%, and removing action masking 59.13%. Removing the downstream residual or transition stage gives 69.23% and 69.02%. These comparisons identify retention stabilization as particularly important.
Failures depend on the downstream interaction: transitions threaten stability, switch contact can disturb the grasp, and microwave actuation can fail even when retention succeeds. A four-pair mask-selection comparison favors the LLM selector, 73.0% versus 66.5%, by considering useful free fingers rather than retention alone.
| Composite Success Rate (%) | |||||||
|---|---|---|---|---|---|---|---|
| First Task | Second Task | Ours | Ours-ZS | Finger | Task-A | Mask | Trans. |
| GraspBall | OpenDoor | 82.69±8.33 | |||||
| PushButton | 85.58±7.50 | ||||||
| OpenMicrowave | 73.08±9.25 | ||||||
| TurnOnSwitch | 71.88±8.93 | ||||||
| PourMug | OpenDoor | 75.72±8.08 | |||||
| PushButton | 85.58±7.28 | ||||||
| OpenMicrowave | 62.74±8.21 | ||||||
| TurnOnSwitch | 64.90±9.89 | ||||||
| PickCan | OpenDoor | 79.09±6.04 | |||||
| PushButton | 85.34±6.92 | ||||||
| OpenMicrowave | 72.36±11.82 | ||||||
| TurnOnSwitch | 65.38±6.81 | ||||||
| PickStick | OpenDoor | 86.30±5.78 | |||||
| PushButton | 86.54±4.84 | ||||||
| OpenMicrowave | 87.26±7.81 | ||||||
| TurnOnSwitch | 74.52±6.10 | ||||||
| Average | – | 77.43±2.64 | |||||
5 Conclusion
Explicit action ownership and learned stabilization enable two-skill compositions while reusing frozen base policies. The evidence covers these simulated task pairs. Extending to physical hands or longer sequences with several interacting skills remains untested here.
Appendix
The appendices specify base-policy training, residual learning, release tests, comparison implementations, and the mask-selection prompt.
Appendix A Base Policy Training and Evaluation Protocol
Base policies use 32–37 low-dimensional proprioceptive and geometric state features, rather than raw vision. Actions contain six wrist coordinates and 22 finger-joint targets. A FiLM-conditioned temporal UNet predicts 32 actions from two observations; execution horizons vary by task.
Training uses successful demonstration windows, a 100-step diffusion objective, AdamW at 10⁻⁴, batch size 256, a 2,000-step warmup, cosine decay, EMA, and gradient clipping. Table 4 gives standalone success from 82.21% to 100%. Composite evaluation additionally requires continuous retention while the interaction reaches its geometric success threshold.
Equations
| Task | Execute Horizon | Success Rate (%) |
|---|---|---|
| GraspBall | 8 | 100.0 |
| PickStick | 16 | 100.0 |
| PickCan | 16 | 100.0 |
| PourMug | 16 | 100.0 |
| OpenDoor | 24 | 97.59 |
| OpenMicrowave | 24 | 89.90 |
| PushButton | 16 | 100.0 |
| TurnOnSwitch | 30 | 82.21 |
Equation
Appendix B Learning Procedure and Implementation Details
The implementation uses one RTX 4090. Both residuals are learned with PPO; the allocation stage uses release-test statistics and an LLM rather than gradient training.
B.1 PPO Training: Task A Residual Stabilizer
The retention residual observes preserved-finger motion, object state in the palm frame, contacts, torque features, executed base action, and rollout phase. Its bounded corrections minimize object–palm distance plus a 0.001 residual penalty. A 256–256–128 ELU backbone trains for 1,000 PPO iterations in 1,024 environments, totaling 24,576,000 steps and about 20 minutes.
Equation 13
| (13) |
B.2 PPO Training: Task B Residual
The downstream residual additionally receives the frozen Task-B action and selected mask. Reward combines task progress, retention distance with weight one, and residual penalty 0.001. Optional joint updates adapt the Task-A residual at a reduced learning rate while base policies remain frozen.
Training uses eight environments, 200-step nominal rollouts, and 1,000 iterations: 1.6 million nominal step slots, with filtering reducing actual samples per update to roughly 408–750. Reported runtime is about four hours.
Equation 14
| (14) |
B.3 Training Curriculum
First collect held states; then train one retention stabilizer per Task-A skill under randomized disturbances. For each task pair, compute release diagnostics, choose a fixed mask, and train a downstream residual. Thus reuse avoids retraining full base policies, but additional pair-specific learning remains.
B.4 Release-Test Protocol
Each candidate mask receives 100 tests from randomly restored held states. Released fingers open over 100 simulation steps. Appendix retention checks use a five-centimeter palm-distance threshold and three-centimeter drop-height threshold at the end; clean release requires contact forces below 0.1 newton during the final twenty steps. Composite evaluation separately requires retention throughout execution.
Equation 15
| (15) |
B.5 Baseline Implementation Details
Frozen grasp holds finger targets fixed without residual correction. The decomposed baseline retrains base policies with fixed family-specific masks and cross-subspace action penalties. The residual baseline uses a single unrestricted correction trained with SAC; it differs in learning algorithm as well as ownership structure. Ours-ZS uses the learned retention controller and selected mask but no learned downstream correction.
Equation 16
| (16) |

B.6 Hyperparameters and Compute Resources
PPO uses learning rate 3×10⁻⁴, clipping 0.2, discount 0.99, GAE parameter 0.95, five epochs, four minibatches, and gradient norm limit one. Optional Task-A joint tuning uses 10⁻⁴. Both actors start with zero-mean corrections and log standard deviation −1.5. The separate training budgets and release thresholds are retained in their preceding sections.
B.7 LLM-Based Finger Mask Selection
The authors use GPT-5.4 with temperature zero to choose a mask from task descriptions, candidate masks, retention/clean-release rates, and free-finger counts. The original prompt is retained below. No model fine-tuning is used at this stage; low-temperature decoding is a configuration choice rather than a general reproducibility guarantee.
The heuristic maximizes retention subject to releasing at least two fingers. In the illustrated ball-and-door example, freeing the index finger helps downstream contact even though its selected grasp has lower immediate retention. This explains the measured benefit on selected pairs without establishing that an LLM is necessary for every allocation.
“You are selecting a finger-retention mask for a dual-task robotic manipulation problem. The robot hand has five fingers and must continue stabilizing the object from Task A while freeing sufficient fingers to execute Task B. Each candidate mask specifies which fingers remain in contact with the object and which fingers are released for downstream interaction.
A good mask should satisfy three objectives simultaneously: (1) maintain stable object retention, (2) allow clean finger release without disturbing the grasp, and (3) preserve dexterous and anatomically appropriate fingers for Task B.
When evaluating candidates, consider not only the numerical metrics but also the semantic suitability of the finger assignment. For example, thumb-index opposition is generally preferable for stabilizing slender cylindrical objects, while releasing the thumb may significantly reduce grasp stability even if the clean-release score is high. Avoid overly conservative masks that release too few fingers and overly aggressive masks that sacrifice object stability.
Select the single mask that provides the best overall trade-off between grasp stability and downstream manipulation capability.”Equation 17
| (17) |