Literature
Display
← All papers

Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?

Tyler Ga Wei Lum · Kushal Kedia · Karen Liu · Jeannette Bohg

Preprint posted · CoRL 2026 (accepted) ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

Play2Perfect first learns general grasping and in-hand pose control in simulation, then specializes that prior to contact-rich assembly with sparse CAD-derived goal rewards.

[Uncaptioned image]
Figure 1: Play2Perfect Overview. Before a robot can perfect precise assembly, it first learns to play. We pretrain a single goal-conditioned RL policy on task-agnostic dexterous object manipulation, producing a reusable prior for grasping, in-hand reorientation, and 6D pose control. This pretrained play policy is then finetuned in sparse-reward RL environments derived from CAD designs to solve diverse contact-rich assembly tasks, including tight insertions, screwing, and multi-part assembly.

At a glance

A goal-conditioned dexterous policy is pretrained by manipulating diverse primitive objects through random poses, then finetuned for insertion, beam assembly, or screwing. Ablations favor object diversity, coupled translation/rotation, varied trajectories, and precise play goals. Hardware insertion succeeds in six of ten trials at 0.5-millimetre nominal clearance; complete beam assembly succeeds in twelve of twenty and complete insertion-plus-screwing in five of ten. The reported thirty-three-fold learning speedup concerns downstream adaptation on a simplified comparison task and excludes the seven-day shared pretraining cost.

1 Introduction

Sparse assembly success is hard to discover from a random arm-and-hand policy because grasping, reorientation, transport, alignment, and contact must all work first. Play pretraining supplies reusable manipulation behaviors before assembly-specific optimization. The central claim is that useful play teaches finger-driven in-hand pose control, not merely moving a rigidly grasped object with the arm. The experiments study this prior and its specialization rather than a complete autonomous assembly planner.

2 Related Work

Dexterous imitation learning faces teleoperation and contact-feedback difficulties, while reinforcement learning often relies on dense rewards or specialized demonstrations. Previous play controllers mostly target free-space behavior or direct transfer. Play2Perfect uses play as an initialization for further task-specific learning, with CAD-defined sparse contact goals. Assembly environments and CAD structure are still supplied, so the approach reduces demonstration and dense-reward engineering without removing all task specification.

Refer to caption
Figure 2: What matters in dexterous play pretraining? We study the key factors that shape the learned manipulation prior. Our design emphasizes in-hand manipulation with fingers across diverse objects and trajectories, with 6D pose-reaching objectives and precise goal tolerances.

3 Play2Perfect

The same arm/hand policy is first trained on procedural objects and random free-space goals, then adapted in simulation to assembly environments and goal sequences derived from CAD. The final controller consumes object state and a goal pose rather than raw visual or tactile observations.

3.1 Dexterous Play Pretraining

The policy observes robot proprioception, current/target object poses, and bounding-box geometry. Procedural primitive combinations vary dimensions, density, centre of mass, and inertia. A play episode begins with grasp/lift and continues through nearby positions with substantial orientation changes, encouraging repeated in-hand reorientation.

Pretraining is shaped reinforcement learning: smoothness, grasp/lift, and pose-progress rewards accompany a large sparse goal-success bonus. The default one-centimetre keypoint-distance tolerance encourages fine pose control, while online random goal sequences broaden transitions. The later claim of sparse assembly rewards should not be read as sparse-only training throughout the entire pipeline.

Refer to caption
Figure 3: Assembly-by-Disassembly. Given a completed CAD assembly, we generate assembly steps by sequentially removing parts and reversing the disassembly sequence. Each step defines a sparse goal sequence: the final assembled pose and intermediate contact goals, e.g., pre-insert pose.

3.2 RL Finetuning on Assembly Tasks

An assembly-by-disassembly procedure reverses feasible part removals to define assembly steps. Each manipulated part has a CAD-derived relative transform to the assembled fixture. Sparse intermediate goals include a pre-contact insertion pose or successive ninety-degree screw-thread poses, followed by the final assembled pose. Finetuning retains smoothness regularization and sparse goal/retraction bonuses but removes play’s dense grasp/lift/progress terms. Thus supervision includes intermediate CAD-derived goals, not only one terminal reward at the end of the complete multi-part task.

3.3 Training Details and Sim-to-Real Transfer

Both stages use Split and Aggregate Policy Gradients with domain randomization for delays, pose errors, and dynamics. Deployment uses FoundationPose and the known CAD geometry. The controller runs at sixty hertz and part tracking at thirty; the detailed appendix specifies that fixture pose is estimated once at rollout start, while the part is tracked online.

4 Experiments

A twenty-two-degree-of-freedom Sharpa hand on a seven-joint KUKA arm is evaluated on tight insertion, two-step beam assembly, and screw-leg manipulation. Beam parts from Fabrica are printed at triple scale, and the FurnitureBench leg is made three times longer to improve grasping/tracking visibility. Four learned skills count beam’s two steps separately. Simulation evaluates five hundred randomized rollouts; hardware uses ten or twenty trials with fixed fixtures and varied starting parts. Completion time is reported over successful trials only.

4.1 Can Dense Rewards Overcome the Need for Play ?

On the main tasks, the pretrained policy reaches high success after roughly two to five hours of downstream learning, while sparse- and dense-reward scratch policies produce no successes after twenty-four hours. A simplified propped-up insertion initialization lets dense scratch learning succeed after over one hundred hours, versus about four hours for Play2Perfect; the authors report a thirty-three-fold downstream speedup.

That comparison does not include the common play prior’s seven days of pretraining. The scratch policy also learns a brittle balancing strategy, dropping to roughly twenty percent under a ten-newton perturbation and zero under stronger forces, whereas the pretrained strategy remains above seventy-five percent at the largest tested perturbations. These results isolate adaptation and strategy robustness within the tested initialization and force range.

Refer to caption
Figure 4: Dexterous Play Pretraining Enables Efficient Downstream Assembly Learning. Across four contact-rich assembly tasks, Play2Perfect rapidly learns successful policies from the shared dexterous prior, reaching high success within 2-5 hours. In contrast, training from scratch fails to make progress with either sparse task rewards or hand-engineered dense rewards.
Refer to caption
Figure 5: Dexterous Play Pretraining Induces Robust Assembly Strategies. After simplifying the initialization with an easy-grasp fixture, Scratch (dense reward) can learn the task, but it relies on a brittle strategy that balances the object rather than robustly grasping it. This shortcut fails sharply under external force perturbations. In contrast, Play2Perfect learns a more stable grasping and recovery strategy, maintaining high success across perturbation magnitudes.

4.2 Which Design Choices in Play Pretraining Matter Most for Downstream Assembly?

Ablations average four assembly skills over three seeds. Increasing procedural objects from ten through one hundred to one thousand improves stability and final success. Translation-only play lacks the orientation skills needed for assembly; rotation-only transfers substantially better but plateaus about ten points below coupled six-dimensional control. Online random trajectories outperform fixed banks of ten or one hundred. A ten-centimetre success tolerance transfers poorly, and five centimetres learns more slowly and plateaus near sixty percent. These are controlled changes within the paper’s pretraining recipe, not proof that object count alone captures all relevant diversity.

4.3 Is RL Finetuning Necessary for Precise, Contact-Rich Assembly?

Clearance is defined as the full difference between hole and peg cross-sectional dimensions: a thirty-by-twenty-millimetre peg in a thirty-one-by-twenty-one hole has one-millimetre nominal clearance. Training spans five to 0.5 millimetres; simulation tests extend down to 0.2, where finetuned success is eighty percent. Play-only reaches about seventy-five percent at forty millimetres but nearly zero by four. Hardware finetuning improves ten-millimetre, two-millimetre, and 0.5-millimetre results from 60%, 20%, and 0% to 100%, 90%, and 60%. The one-centimetre pose-goal tolerance and submillimetre geometric clearance are different quantities; contacts constrain the actual insertion.

Refer to caption
Figure 6: What Matters in Pretraining for Downstream Assembly Finetuning? We vary key play pretraining choices and evaluate downstream RL finetuning success averaged across four assembly tasks and three seeds. Pretraining transfers best when it encourages in-hand manipulation via 6D in-hand object control across diverse objects and trajectories with precise goal tolerances.
Refer to caption
Figure 7: Assembly Finetuning Enables Tight Insertion. We compare Play2Perfect against a frozen Play-only policy on insertion tasks with varying contact clearance. (Left) Both policies succeed at loose clearance, but only Play2Perfect succeeds at tight clearance. (Top right) Simulation sweeps show that Play2Perfect remains robust as clearance decreases, while Play-only rapidly degrades. (Bottom right) Real-world success rates show the same trend across different clearances.

4.4 Can Play2Perfect Transfer to the Real World?

Hardware insertion succeeds in 10/10, 9/10, and 6/10 trials as clearance tightens. Successful-run mean time increases from 6.8±1.5 seconds to 11.1±5.1 seconds. Beam step one succeeds 16/20 times; step two succeeds 12/16 conditional on step one, giving 12/20 end-to-end. Screw insertion succeeds 7/10, and subsequent screwing succeeds 5/7, giving 5/10 complete runs. These conditional rates must not be multiplied or averaged as if they came from independent fresh-start trials.

Reported successful full-task times include approach, grasp, transport, and contact, with beam at 21.0±8.9 seconds and insertion-plus-screwing at 15.6±2.9. There is no matched hardware timing baseline establishing a general speed advantage over grippers. Among sixty recorded real trials, eighteen failures include six timeouts, five insertion-phase drops, three reorientation drops, and four catastrophic tracking losses; subtler pose errors may also contribute.

Table 1: Real-World Assembly Results. Play2Perfect transfers zero-shot to real-world insertion, multi-part assembly, and screwing. Completion times are mean ±\pm std over successful trials, including grasping, transport, and assembly (* indicates success conditional on the preceding sub-task).
Tight-Insertion Assemble-Beam Screw-Leg
10mm 2mm 0.5mm Step 1 Step 2* Insert Screw*
Success Rate 10/10 9/10 6/10 16/20 12/16 7/10 5/7
Completion Time 6.8 ±\pm 1.5 s 9.4 ±\pm 1.9 s 11.1 ±\pm 5.1 s 21.0 ±\pm 8.9 s 15.6 ±\pm 2.9 s

4.5 Does Play2Perfect Transfer to Everyday Insertion Tasks?

The same pretrained checkpoint and finetuning recipe reaches ninety percent simulated success for iPhone-plug insertion after 8.4 hours and fork-to-rack insertion after 2.8 hours. These results extend the tested object/task geometries in simulation; they are not additional physical demonstrations of small-object perception or everyday household reliability.

Refer to caption
Figure 8: Transfer to Everyday Insertion Tasks. Play2Perfect achieves 90% success in simulation on iPhone plug and YCB fork insertion after just 8.4 and 2.8 hours of finetuning, respectively. We use the same pretrained checkpoint and training recipe as the main experiments.

5 Discussion and Limitations

Play pretraining enables efficient specialization to precise contact behaviors, but the system comprises short assembly skills with externally specified task sequencing, active parts, and goals. Policies are finetuned per task or benchmark family. Pose tracking can fail under hand occlusion or rapid motion, and the policy lacks direct observation of fixture/surrounding geometry beyond its goal representation. Broader sequencing, memory, multi-task learning, and visual/tactile input remain future directions.

Acknowledgements

Support includes Stanford Human-Centered Artificial Intelligence, an ONR Young Investigator Award, NSF grants, NSERC funding, and research/technical collaboration with Sharpa. The original acknowledgment retains named contributors and grant identifiers.

Appendix A Robustness to Noisy Pose Tracking and Failure Analysis

Simulation noise sweeps show tolerance beyond the training regime, with Screw-Leg the most sensitive skill. The appendix prose summarizes one-centimetre/five-degree pose noise, while the finetuning table distinguishes current-part noise from smaller goal-pose noise of two millimetres/one degree. The retained failure chart separates easily identified catastrophic tracking loss from other failures whose possible small pose errors cannot be isolated without ground truth.

Original figure
Figure 9: Pose Noise Sensitivity (Simulation). Success rates under increasing translational and rotational pose noise.
Original figure
Figure 10: Real-World Failure Analysis. Failure modes and relative frequencies across real-world rollouts.

Appendix B End-to-End Two-Step Beam Assembly

Twenty physical beam trials execute step two directly from successful step-one outcomes without resetting the intermediate assembly. Sixteen reach the second step and twelve finish both, yielding sixty percent overall. The four failed first steps and four failed second steps are part of that denominator.

Refer to caption
Figure 11: End-to-End Beam Assembly. Twenty real-world trials of both assembly steps in sequence, with four frames from a rollout. Step 2 success is conditional on Step 1 success.

Appendix C Additional Ablation Results

Per-task curves repeat object-count, objective, trajectory-bank, and goal-tolerance ablations on insertion, both beam steps, and screwing. They broadly support the averaged finding that precise orientation-capable play transfers better than translation-only manipulation. All curves remain available to assess differences in learning speed, saturation, and task-specific variation rather than relying solely on the aggregate trend.

Tight-Insertion

Original figure

Assemble-Beam Step 1

Original figure

Assemble-Beam Step 2

Original figure

Screw-Leg

Original figure
Figure 12: Per-task pretraining ablation results. We report the four play pretraining ablations from the main paper on each downstream assembly task. Across tasks, the results support the same conclusion as the averaged curves: pretraining transfers best when it teaches precise 6D in-hand object control across diverse objects and goal trajectories.

Appendix D Pose Tracking Failures with Smaller Objects

Original-size parts are heavily occluded by the hand, making real pose tracking unreliable. The figures show this limitation and motivate larger beam parts and a longer leg. Small-object simulation success does not remove the corresponding hardware perception bottleneck.

Refer to caption
Figure 13: Pose Tracking under Hand Occlusion. At the original 1×1\times object scale, hand occlusion makes precise object pose tracking difficult. Enlarging the benchmark objects improves their visibility during real-world manipulation.

Appendix E Real-World Experiment Additional Analysis

Qualitative rollouts show tilted approaches, local contact search, corrective in-hand motion, and regrasping after recoverable drops without a separate scripted recovery controller. Recovery depends on the object remaining in the workspace and continued usable pose estimates; it is not guaranteed for arbitrary failures.

Fast motion and occlusion can break tracking. The nearly ninety-degree rotational symmetry of the rectangular leg causes orientation ambiguity, so colored tape differentiates its faces. Fixtures are immovable in the main training simulation but taped to a foam tabletop on hardware, where contact can move or deform them. That mismatch matters particularly because the fixture goal estimate is fixed at rollout start. Most control failures occur during final contact interactions rather than ordinary grasp acquisition.

Refer to caption
Figure 14: Tilted Insertion and Local Search. A representative real-world Tight-Insertion rollout. The policy approaches the hole with a tilted insertion strategy, makes contact near the fixture, and performs a local search with small corrective motions before committing to insertion.
Refer to caption
Figure 15: Recovery After Drops. Representative real-world Assemble-Beam rollouts for both assembly steps. After dropping the part, the policy continues acting closed-loop, regrasps the object, retries the assembly motion, and completes the task without a scripted recovery controller.

Appendix F Unified vs. Task-Specific Policies

A unified policy co-trained across the four assembly skills achieves success comparable to separate policies in the reported experiment. This is a finite multi-task result within the studied family, not an unconditioned universal assembly controller.

Original figure
Figure 16: Unified vs. Task-Specific Policies. A unified policy trained on all tasks achieves comparable success rates to task-specific policies.

Appendix G Training with Movable Fixtures

Additional policies trained and evaluated with movable fixtures remain close to fixed-fixture performance for tight insertion and beam step one. Beam step two and screwing are more vulnerable to fixture displacement. This study changes training as well as evaluation and should not be read as evidence that the main fixed-fixture policy is equally robust without adaptation.

Original figure
Figure 17: Training with Movable Fixtures. Policies can be successfully trained with free, movable fixtures. Performance is compared against policies trained with fixed fixtures.

Appendix H Simulation and Computational Resources

Training uses one RTX A6000 with physics at 120 hertz and control at sixty. Play runs seven days with 24,576 parallel environments; each assembly finetuning run is budgeted one day with 12,288 environments, though useful success emerges earlier in learning curves. Contact-rich geometry increases memory costs. Within each wall-clock comparison figure, methods share GPU type, environment count, hyperparameters, and training budget; the pretraining cost remains separate from downstream speedup.

Appendix I Policy Architecture and RL Algorithm

The actor uses a 1,024-unit recurrent layer followed by a multilayer network to output twenty-nine joint commands. An asymmetric critic additionally receives noise-free, undelayed state, velocities, rewards, and progress features unavailable to the deployed actor. SAPG supplies population-based exploration; architecture and optimization settings are shared between play and finetuning. The full hyperparameter table remains available.

Table 2: Play policy architecture and SAPG hyperparameters.
Parameter Value
Actor network LSTM[1024] + MLP[1024, 1024, 512, 512]
Critic network MLP[1024, 1024, 512, 512]
Learning rate 1×10−41\times 10^{-4}
Minibatch size 98,304
SAPG block size 4,096
Entropy bonus scale 0.002
Discount factor γ\gamma 0.99
GAE parameter λ\lambda 0.95
PPO clip range 0.1

Appendix J Play Pretraining Details

Procedural objects combine a graspable primitive with a smaller component near one end, varying dimensions and independent densities to change shape and inertia. Episodes randomize arm/hand initialization and object placement, then sample reachable initial goals and later goals within ten centimetres and ninety degrees. A ten-second horizon and termination conditions bound play, including table contact force above one hundred newtons.

Four object-frame keypoints jointly encode translation and rotation. Observations use dimensions of the graspable primary component, while reward distance uses fixed 14-by-3-by-3-centimetre keypoint geometry to keep its translation/rotation tradeoff consistent. The 140-dimensional observation contains proprioception, palm-relative geometry/current pose, goal displacement, and object dimensions. Arm outputs are incremental joint targets, hand outputs absolute targets; limits and exponential smoothing are applied.

Pretraining combines velocity regularization, fingertip approach/lift shaping, best-so-far pose-progress rewards, and a sparse goal bonus. A lift flag activates after ten centimetres; goal and lift bonuses are 1,000 and 300. Pose/action/proprioceptive delays, geometry/table variation, and force/torque perturbations encourage transfer. All original formulas and randomization values are retained.

Equations 1–9
𝒦⁡(𝐬)={[sx/2sy/2sz/2],[sx/2−sy/2−sz/2],[−sx/2sy/2−sz/2],[−sx/2−sy/2sz/2]}.\mathcal{K}(\mathbf{s})=\left\{\begin{bmatrix}s_{x}/2\\ s_{y}/2\\ s_{z}/2\end{bmatrix},\begin{bmatrix}s_{x}/2\\ -s_{y}/2\\ -s_{z}/2\end{bmatrix},\begin{bmatrix}-s_{x}/2\\ s_{y}/2\\ -s_{z}/2\end{bmatrix},\begin{bmatrix}-s_{x}/2\\ -s_{y}/2\\ s_{z}/2\end{bmatrix}\right\}. (1)
𝐨i=Ro​𝐤i+𝐭o.\mathbf{o}_{i}=R_{o}\mathbf{k}_{i}+\mathbf{t}_{o}. (2)
d⁡(o,g)=maxi⁡‖𝐨i−𝐠i‖2.d(o,g)=\max_{i}\left\|\mathbf{o}_{i}-\mathbf{g}_{i}\right\|_{2}. (3)
r=rsmooth+rgrasp+𝕀grasped​rgoal.r=r_{\mathrm{smooth}}+r_{\mathrm{grasp}}+\mathbb{I}_{\mathrm{grasped}}r_{\mathrm{goal}}. (4)
rsmooth=−λarm​‖𝐪˙arm‖1−λhand​‖𝐪˙hand‖1.r_{\mathrm{smooth}}=-\lambda_{\mathrm{arm}}\left\|\dot{\mathbf{q}}^{\mathrm{arm}}\right\|_{1}-\lambda_{\mathrm{hand}}\left\|\dot{\mathbf{q}}^{\mathrm{hand}}\right\|_{1}. (5)
rgrasp\displaystyle r_{\mathrm{grasp}} =rapproach+(1−𝕀grasped)​rlift,\displaystyle=r_{\mathrm{approach}}+(1-\mathbb{I}_{\mathrm{grasped}})r_{\mathrm{lift}}, (6)
rapproach\displaystyle r_{\mathrm{approach}} =λapproach​max⁡(d¯ft∗−d¯ft,0),\displaystyle=\lambda_{\mathrm{approach}}\max\!\left(\bar{d}^{*}_{\mathrm{ft}}-\bar{d}_{\mathrm{ft}},0\right), (7)
rlift\displaystyle r_{\mathrm{lift}} =λliftmax(z−zinit,0)+Blifted𝕀[z≥zlifted],\displaystyle=\lambda_{\mathrm{lift}}\max(z-z_{\mathrm{init}},0)+B_{\mathrm{lifted}}\mathbb{I}[z\geq z_{\mathrm{lifted}}], (8)
rgoal=λgoalmax(d∗−d(ot,gt),0)+Bsucc𝕀[d(ot,gt)<ϵ],r_{\mathrm{goal}}=\lambda_{\mathrm{goal}}\max\!\left(d^{*}-d(o_{t},g_{t}),0\right)+B_{\mathrm{succ}}\mathbb{I}[d(o_{t},g_{t})<\epsilon], (9)
Table 3: Domain randomization used during play pretraining.
Parameter Randomization
Current/goal pose noise (translation) 11 cm
Current/goal pose noise (rotation) 5∘5^{\circ}
Object-pose delay 00–1010 control steps
Action and proprioception delay 00–33 control steps
Joint-velocity observation noise σ=0.1\sigma=0.1 rad/s
Object-dimension scale 90%90\%–110%110\%
Table-height variation ±1\pm 1 cm
External force perturbation 20.020.0 N
External torque perturbation 2.02.0 N m

Appendix K Pretraining Ablation Implementations

Ablations vary one pretraining factor while keeping architecture, reward coefficients, randomization, and downstream finetuning fixed. Object sets contain ten, one hundred, or one thousand procedural samples. Translation-only ignores goal orientation; rotation-only first grasps/lifts to a pose and then changes orientation at fixed position. Fixed trajectory banks use the same sampling bounds as online trajectories. Goal thresholds are one, five, or ten centimetres, with the rest of the objective unchanged.

Appendix L Assembly Finetuning Details

Assembly simulation uses convex decomposition for most geometry and resolution-256 signed distance fields at narrow mating interfaces, preserving contact shape without applying the most expensive representation everywhere. Parts begin at randomized planar positions and orientations before settling; fixtures also have randomized placements. CAD relative transforms make goals fixture-relative. Pre-insertion poses or ninety-degree thread steps define the sparse goal sequence.

Finetuning removes dense reaching/grasping/lifting/progress rewards. It retains joint-velocity regularization, a 1,000-point bonus on reaching each one-centimetre-tolerance goal, and a 1,000-point terminal retraction bonus when the assembled part remains at its goal and the palm retreats more than twenty centimetres. This explicitly rewards release/retraction as well as assembly.

Current-part observations receive one-centimetre/five-degree noise; goal-pose observations receive two-millimetre/one-degree noise. Fixture yaw varies within ten degrees, with pose delay up to ten control steps, action/proprioception delay up to three, dimension/table variation, and external forces/torques. These structured randomizations approximate specific transfer errors rather than covering every possible contact or perception failure.

Equations 10–14
gM=ft​Tpf.g_{M}=f_{t}T^{f}_{p}. (10)
rt=rsmooth+rgoal.r_{t}=r_{\mathrm{smooth}}+r_{\mathrm{goal}}. (11)
rsmooth=−λarm​‖𝐪˙arm‖1−λhand​‖𝐪˙hand‖1.r_{\mathrm{smooth}}=-\lambda_{\mathrm{arm}}\left\|\dot{\mathbf{q}}^{\mathrm{arm}}\right\|_{1}-\lambda_{\mathrm{hand}}\left\|\dot{\mathbf{q}}^{\mathrm{hand}}\right\|_{1}. (12)
rgoal=Bsucc𝕀[d(ot,gm)<ϵ]+rretract,r_{\mathrm{goal}}=B_{\mathrm{succ}}\mathbb{I}\!\left[d(o_{t},g_{m})<\epsilon\right]+r_{\mathrm{retract}}, (13)
rretract=Bretract𝕀[d(ot,gM)<ϵ∧‖𝐩tpalm−𝐩tobj‖2>0.2m].r_{\mathrm{retract}}=B_{\mathrm{retract}}\mathbb{I}\!\left[d(o_{t},g_{M})<\epsilon\;\land\;\left\|\mathbf{p}^{\mathrm{palm}}_{t}-\mathbf{p}^{\mathrm{obj}}_{t}\right\|_{2}>0.2~\mathrm{m}\right]. (14)
Table 4: Domain randomization used during assembly finetuning.
Parameter Randomization
Current part-pose noise (translation) 11 cm
Current part-pose noise (rotation) 5∘5^{\circ}
Goal-pose noise (translation) 22 mm
Goal-pose noise (rotation) 1∘1^{\circ}
Fixture yaw [−10∘,10∘][-10^{\circ},10^{\circ}]
Object-pose delay 00–1010 control steps
Action and proprioception delay 00–33 control steps
Joint-velocity observation noise σ=0.1\sigma=0.1 rad/s
Part-dimension noise scale 90%90\%–110%110\%
Table-height variation ±1\pm 1 cm
External force perturbation 20.020.0 N
External torque perturbation 2.02.0 N m

Appendix M Inference Time Pipeline

At rollout start, CAD geometry supplies the part model, grasp bounding box, and fixture-relative goal sequence; the fixture pose is estimated once. FoundationPose updates the part estimate at thirty hertz, and the sixty-hertz controller reuses the newest pose between updates. A goal advances when the keypoint pose error falls within tolerance. Joint targets go to the existing low-level controllers. Learned contact search, in-hand rotation, and regrasping operate inside this externally supplied goal-sequencing interface, without an additional scripted insertion, screw, or recovery controller.

Refer to caption
Figure 18: Inference-Time Pipeline. At deployment, CAD models are reused to estimate the current part pose, compute CAD-derived goal poses relative to the fixture, and define the grasp bounding box. The finetuned policy takes robot proprioception, part pose, goal pose, and the grasp bounding box as input, and outputs joint position targets for the arm and dexterous hand.
Refer to caption
Figure 19: Policy Observations During Real-World Deployment. (Top) A representative real-world Screw-Leg rollout, with (Bottom) the corresponding policy observations. The translucent green object and translucent axes denote the active sparse goal pose, while the opaque object and full-opacity axes denote the estimated current part pose observed by the policy. As the rollout progresses, the active goal advances along the screw axis with successive rotational offsets, and the policy tracks these goals by reorienting the leg in-hand. During fast motions, the visualized current pose can lag the real object due to object-pose tracking latency.
Concept demonstration

Open image ↗