Overview
Appendix
At a glance
UniDex combines a human-video-derived dexterous-hand dataset, an action representation shared across hands, a 3D vision–language–action policy and a human-data capture system. After task-specific robot fine-tuning, it reaches 81% average task progress and 76% average final success across five tool-use tasks. A coffee policy also transfers between hands without target-task fine-tuning, and a separate coffee experiment studies replacing some robot demonstrations with retargeted human examples.
1 Introduction
Dexterous foundation policies face scarce demonstrations, incompatible hand kinematics and high-dimensional actions. UniDex addresses these jointly: convert egocentric human interaction into robot-centered pointcloud/action data, align joints by functional role, pretrain a policy and then fine-tune it on robot demonstrations. The capture system extends the same conversion pipeline to task-specific human data.
The contribution is a practical suite evaluated on selected hands and tasks; “universal” is an ambition rather than demonstrated control of arbitrary hardware. This edition covers all 15 pages of v1, dated 23 March 2026, including 16 complete figures, the dataset table, embedded results tables, seven displayed formula containers, all appendices and 78 references.
2 Related Work
2.1 Dexterous Manipulation
Dexterous manipulation has progressed from analytical contact/grasp models to learned grasping, rotation and reorientation. UniDex targets everyday tool use that requires coordinated finger motion beyond a parallel-jaw gripper. Its five physical tasks provide broader behavior than isolated grasping, while still leaving most real-world dexterity untested.
2.2 Robot Foundation Policies and Unified Action Space
Many VLA systems start from gripper-focused datasets and action layouts. UniDex instead uses a common, function-based joint-slot mapping across dexterous hands. Related RDT-1B preserves control semantics across embodiments, while UniDex emphasizes corresponding finger functions. A common slot does not erase differences in joint limits, linkage, force or reachability; each hand still requires a defined mapping.
2.3 Learning from Human Videos
Human video offers abundant interaction but differs from robot appearance and kinematics. UniDex performs retargeting and visual replacement before pretraining, so supervision is already robot-centered during later fine-tuning. This shifts alignment work into dataset preparation rather than eliminating it. Manual calibration and contact-frame correction remain part of the method.
3 UniDex-Dataset
3.1 Overview
The source datasets are H2O, HOI4D, HOT3D and TACO. Clips receive instructions where needed, are segmented by instruction and filtered, then retargeted to Inspire, Leap, Shadow, Allegro, Ability, Oymotion, Xhand and Wuji hands. The resulting dataset has approximately nine million paired image/pointcloud/action frames at 30 fps and 52k trajectories, spanning the reported 6–24 active-DoF range.
These are transformed training trajectories, not 52k independently collected robot executions. Table 1 compares counts, hand variety, scenes and modalities; it also marks ActionNet’s pointcloud as very low quality rather than simply equating that modality with high-quality 3D observations. Figure 3 preserves the complete example montage and word clouds. The paper does not provide per-source/per-hand balance, quantitative retargeting error or an audit of physically successful replay for every converted trajectory.
3.2 Human-Robot Transformation
Transforming human data into robot trajectories requires overcoming two core gaps: kinematic and visual. We outline our methods below.
3.2.1 Kinematic Retargeting
Retargeting aligns robot fingertips to human fingertip targets. A six-DoF dummy-base offset permits translation/rotation of the robot hand relative to the observed human-hand frame. The forward kinematics compose the fixed human/dummy pose, offset and robot fingertip transform; stacking fingertip residuals defines the inverse-kinematics error.
PyBullet solves the multi-fingertip IK with joint limits and damping. Dependent mimic joints are updated from their masters using the prescribed affine relationship, followed by repeated error evaluation. A user adjusts base offsets and solver settings through a GUI, first calibrating each source-dataset/robot-hand pair and then correcting selected contact-rich frames.
The goal is plausible fingertip contact with modest interaction, not a formal contact-feasibility guarantee. The stated objective contains no explicit force-closure, friction, collision or object-dynamics constraint; small fingertip error alone cannot certify successful execution. Manual effort, failure rate and convergence across the full dataset are not quantified.
Equations 1–3
| (1) |
| (2) |
| (3) |
Original selectable caption
Table 1: Comparison between UniDex-Dataset and other dexterous manipulation datasets. UniDex-Dataset advances in total trajectories, variety across hands/actions/scenes, and supports for all perception modalities.| ✗ |
| ✓ |
Equation 4
| (4) |
3.2.2 Visual Alignment
RGB-D frames become pointclouds. WiLoR and SAM2 identify human hands, whose points are removed. Retargeted robot-hand meshes are inserted and rendered into the scene, then reprojected with a pinhole camera to respect depth ordering and the single-view deployment setting. This reduces appearance mismatch but cannot recover occluded object geometry or verify contact dynamics.
4 UniDex-VLA
4.1 Unified Action Space: FAAS
FAAS has 82 dimensions: 18 for two wrist poses and 64 for two sets of hand-joint slots. Each wrist uses two three-vectors for continuous rotation plus three translations; each hand has 32 joint slots. Functionally similar controllable channels share indices, including mimic-joint channels where the source treats them as actuators.
The main text says 21 base actuator slots are shared, whereas Appendix C allocates five slots to each of five fingers, totaling 25 finger positions, followed by two extra wrist slots and five future-use slots. These are not the same count; the complete per-hand mapping in Figure 15 is retained. Active actuators, full kinematic joints and reserved representation dimensions should not be conflated. FAAS avoids predicting human coordinates and retargeting them after inference, but still requires embodiment-specific indexing and control execution.
4.2 VLA Policy
The policy conditions a generative action model on 3D scene geometry, language and robot state. FAAS supplies a shared output layout, while the pointcloud provides explicit geometric information. The experiments test the combined architecture and pretraining, without independently isolating every contribution of 3D sensing, action alignment and dataset design.
4.2.1 Observations and Action Outputs
Inputs are a cropped/downsampled single-view colored pointcloud, a natural-language instruction and proprioception. Outputs are an action chunk in FAAS. Proprioceptive wrists use absolute poses; commanded wrist poses are relative to the first frame of the chunk. Joint states and commands use the corresponding abstracted slots. The main model’s exact action horizon and control rate are not stated, although the baseline appendix specifies some of those settings.
4.2.2 Model Architecture
The architecture follows the π₀ family but replaces PaliGemma’s 2D vision encoder with Uni3D. Uni3D inherits a pretrained vision-transformer representation aligned with image/text features; the backbone combines it with text/proprioception, and an action expert performs conditional flow matching. Ten Euler steps with step size 0.1 generate a chunk at inference.
The “No Pretrain” control removes UniDex-Dataset pretraining, not necessarily the pretrained language/visual components of this architecture. Its comparison is evidence for the robot-centered dataset initialization, not training every component from scratch.
5 Experiments
5.1 Experimental Setup
A seven-DoF Franka arm uses Inspire (6 active/12 full DoF), Wuji (20 active DoF) or Oymotion (6 active/11 full DoF), viewed by a RealSense L515. Inspire performs coffee pouring and sweeping; Wuji performs spray-bottle watering, scissors cutting and mouse use. Oymotion is used in the transfer experiment.
Each main task receives 50 robot demonstrations collected with OpenTeleVision, dex-retargeting and Apple Vision Pro. Coffee/sweeping/watering/cutting have grasp and execution stages; mouse use requires dragging a file into a folder and clicking. Average task progress averages stage success, while final success records complete task completion. The measures answer different questions.
Comparators are Diffusion Policy, 3D Diffusion Policy, gripper-pretrained π₀ and UniDex without dataset pretraining. π₀ and both UniDex variants use FAAS, while DP/DP3 retain lower-dimensional outputs. Architecture, perception, pretraining and some optimization settings therefore differ across methods; the no-pretraining variant is the closest dataset ablation.
5.2 Performance
Each method/task is evaluated for 20 trials. UniDex’s average task progress is 87.5% for coffee, 82.5% for sweeping, 85% for watering, 60% for cutting and 90% for mouse use. Their mean is 81%, compared with DP 29%, DP3 35%, π₀ 38% and UniDex without dataset pretraining 32.5%.
Average final success is separately reported as 76%, versus 22/30/35/23% for those comparators. The 81% headline is therefore not the complete-task success rate. Figure 11’s ±12.1-point progress spread matches the sample standard deviation across the five task scores; the other reported progress spreads likewise closely track across-task variation. They are not confidence intervals over evaluation trials or repeated model training.
For cutting, 60% progress versus the best competing 32.5% is a 27.5-point absolute gain, or approximately 84.6% relative improvement. The full model’s large gain over its 32.5%-average no-pretraining control supports dataset pretraining in this setup. Trial-level confidence intervals, multiple training seeds and a separate FAAS-versus-alternative-action ablation are not reported.
5.3 Generalization
Spatial transfer. On coffee placements outside the original demonstration regions, Figure 8 reports progress of 0% DP, 20% DP3, 20% π₀, 50% no-pretraining UniDex and 75% UniDex; adding DemoGen raises it to 95%. DemoGen translates segmented object pointclouds and realigns robot states using task/motion planning. The augmented result includes synthetic training coverage of new placements, rather than testing an entirely unmodified training distribution.
Object transfer. Replacing one black kettle with a smaller purple kettle changes color, size, handle and spout. Figure 9 reports 80% progress for UniDex, versus 0/15/10/30% for DP/DP3/π₀/no-pretraining UniDex. This is a useful single-object transfer test, not an exhaustive test of general tool understanding.
Hand transfer. The coffee policy fine-tuned on Inspire gives 40% progress on Wuji and 60% on Oymotion without further task fine-tuning; the corresponding π₀ results are 0/10% and no-pretraining results 0/5%. Figure 10 explicitly labels these values average task progress, although the body calls them success. Both target hands already occur among the eight pretraining embodiments, so the result is task-skill transfer across known hand types, not control of wholly unseen hardware. Each generalization setting uses ten trials.
5.4 UniDex-Cap for Human-Robot Data Co-train
UniDex-Cap rigidly mounts a RealSense L515 to Apple Vision Pro, combines RGB-D with hand/head poses, applies the same retargeting pipeline, changes viewpoint and downsamples human motion to teleoperation speed. The coffee study varies 0–40 robot demonstrations and 0–90 transformed human demonstrations, with 20 evaluation trials per cell.
Human examples generally help when robot data are available, but the zero-robot row remains zero throughout. The effect is not strictly monotonic: at 40 robot demonstrations, increasing human demonstrations from 20 to 30 lowers displayed progress from 80 to 77; at 30 robot demonstrations, 50→60 human examples gives 87→80. The source’s “consistently improves” statement is too strong for every cell.
The proposed roughly two-human-for-one-robot exchange rate is an empirical boundary estimate for this task and range, not a universal conversion factor. The main coffee baseline is 87.5% progress, shown rounded to 87 in the heatmap legend. Human demonstrations are reported about 5.2× faster to collect, but total savings also depend on calibration, manual correction, processing, hardware and mixing requirements; these overheads are not quantified.
6 Conclusion and Limitation
The suite shows that retargeted human demonstrations can improve dexterous policy pretraining and reduce some task-specific robot-data demand. Its evaluated scope is five tasks on two main hands plus coffee transfer to two pretraining-known hand types. The authors identify unused action-free or weakly labeled egocentric data as a scaling opportunity. Broader hardware coverage, verified physical replay, retargeting labor and statistical robustness remain open.
7 Acknowledgment
We would like to give special thanks to Wuji Technology Inc. for providing the hardware support, and Hojin Bae, Haoxu Huang, Shaoting Zhu for their technical support. Tsinghua University Dushi Program supports this project.
Appendix A Training Details
A.1 UniDex-VLA Flow-Matching Loss
The target velocity for the linear path Aτ=τA+(1−τ)ε is A−ε, with standard Gaussian ε. Equation 5 uses an expected norm of velocity error; the original expression has norm bars but no explicit square. This edition preserves that stated objective instead of silently replacing it with a squared-error formula.
There is a notation inconsistency: the stated conditional Gaussian covariance is (1−τ)I, while the given sampling equation yields (1−τ)²I. The sampling rule and original formula are both retained; this identifies a discrepancy in the paper, not a verified implementation error. Starting from Gaussian actions, forward Euler with δ=0.1 advances the vector field to the denoised chunk.
Equations 5
| (5) |
A.2 UniDex-VLA Pretraining
Pretraining uses eight NVIDIA H800 GPUs, batch size 128, three epochs (about 30k steps) and roughly 24 hours. AdamW starts at 10⁻⁴ learning rate with a cosine schedule; the source additionally states a factor-0.95 decay at the second epoch. Weight decay is 10⁻¹⁰ and gradient norm is clipped at 1. Sampling/chunking details are needed to relate the reported step count to the nine-million-frame dataset.
A.3 UniDex-VLA Post-training
Task fine-tuning uses two H800 GPUs, batch size eight and AdamW at 2.5×10⁻⁵ without a scheduler. Ordinary data train for 50 epochs, about 3k steps/four hours; DemoGen-augmented data train for two epochs, about 1.8k steps/2.5 hours. Both use weight decay 10⁻¹⁰ and gradient clipping at 1. These are distinct data/optimization regimes.
A.4 Baselines
DP and DP3 use AdamW at 10⁻⁴, observation horizon four, action horizon 32, batch size 32 and 400 epochs. π₀ uses 2.5×10⁻⁵, batch size eight, 50 epochs and ten denoising steps. Baselines are described as trained to validation convergence. The UniDex no-pretraining variant uses the same post-training settings as the full model; broader comparisons do not use identical budgets or input/output representations.
Appendix B Human-in-the-loop Retargeting GUI
The web GUI exposes left/right dummy-base translations and rotations, IK iterations, mimic-joint settings, damping and residual thresholds. Figure 14 retains the complete interface and pointcloud visualization. It makes manual retargeting inspectable, but the paper’s “few tweaks” description is not accompanied by timing, inter-annotator consistency or error statistics.

Appendix C FAAS Details
For each hand, indices 0–4, 5–9, 10–14, 15–19 and 20–24 designate thumb, index, middle, ring and little fingers. Indices 25–26 hold extra Shadow wrist joints; 27–31 are reserved. Figure 15 preserves both views of all eight hand mappings at the original 7,560×2,755 resolution. This detailed 25-finger-slot allocation should be distinguished from the main text’s 21 shared actuator-slot count and from each hand’s active DoFs.

Appendix D UniDex-Cap Setup Calibration
A rigid printed mount stabilizes the Vision Pro–RealSense extrinsic relationship for a given user. A short clip and sliders align the estimated hand skeleton to the depth pointcloud; the resulting homogeneous transform converts Vision Pro poses into camera coordinates. Figure 16 retains before/after calibration views and Equation 6 the coordinate conversion.
A spatial rigid transform establishes coordinate alignment, not time synchronization by itself. The system is described as synchronized, but timestamp alignment, latency/error measurement and quantitative calibration accuracy are not detailed. Mechanical rigidity also does not remove the need to calibrate a changed mounting/user arrangement.
Equation 6
| (6) |
Appendix E Core Contribution List
Gu Zhang leads the project, dataset/model/action-space design, robot infrastructure and writing. Qicheng Xu leads VLA training and pipeline optimization; Haozhe Zhang leads dataset processing and improves robot/capture systems. Jianhan Ma implements retargeting and visualizations; Long He implements DemoGen; Yiming Bao collects robot/human data and contributes to DemoGen. The appendix identifies these six core contributors, while the paper’s complete 19-author list remains above.