Literature
Display
← All papers

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Gu Zhang · Qicheng Xu · Haozhe Zhang · Jianhan Ma · Long He · Yiming Bao · Zeyu Ping · Zhecheng Yuan · Chenhao Lu · Chengbo Yuan · Tianhai Liang · Xiaoyu Tian · Maanping Shao · Feihong Zhang · Mingyu Ding · Yang Gao · Hao Zhao · Hang Zhao · Huazhe Xu

· CVPR 2026 ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

Appendix

At a glance

UniDex combines a human-video-derived dexterous-hand dataset, an action representation shared across hands, a 3D vision–language–action policy and a human-data capture system. After task-specific robot fine-tuning, it reaches 81% average task progress and 76% average final success across five tool-use tasks. A coffee policy also transfers between hands without target-task fine-tuning, and a separate coffee experiment studies replacing some robot demonstrations with retargeted human examples.

1 Introduction

Dexterous foundation policies face scarce demonstrations, incompatible hand kinematics and high-dimensional actions. UniDex addresses these jointly: convert egocentric human interaction into robot-centered pointcloud/action data, align joints by functional role, pretrain a policy and then fine-tune it on robot demonstrations. The capture system extends the same conversion pipeline to task-specific human data.

The contribution is a practical suite evaluated on selected hands and tasks; “universal” is an ambition rather than demonstrated control of arbitrary hardware. This edition covers all 15 pages of v1, dated 23 March 2026, including 16 complete figures, the dataset table, embedded results tables, seven displayed formula containers, all appendices and 78 references.

Complete original PDF S0.F1
Figure 1: We introduce UniDex, a robot foundation suite for heterogeneous dexterous hand embodiments. We first curate UniDex-Dataset from egocentric human videos to obtain a diverse, robot-centric dataset for large-scale pretraining. Building on this, we train UniDex-VLA, a unified 3D VLA model that is finetuned with task demonstrations and evaluated on challenging real-world tool-use tasks. The policy exhibits strong dexterous capabilities, zero-shot object and cross-hand generalization and significantly outperforming existing VLA baselines. In addition, we design a practical setup, UniDex-Cap to support human–robot data co-training, further reducing the data cost.

2 Related Work

2.1 Dexterous Manipulation

Dexterous manipulation has progressed from analytical contact/grasp models to learned grasping, rotation and reorientation. UniDex targets everyday tool use that requires coordinated finger motion beyond a parallel-jaw gripper. Its five physical tasks provide broader behavior than isolated grasping, while still leaving most real-world dexterity untested.

2.2 Robot Foundation Policies and Unified Action Space

Many VLA systems start from gripper-focused datasets and action layouts. UniDex instead uses a common, function-based joint-slot mapping across dexterous hands. Related RDT-1B preserves control semantics across embodiments, while UniDex emphasizes corresponding finger functions. A common slot does not erase differences in joint limits, linkage, force or reachability; each hand still requires a defined mapping.

2.3 Learning from Human Videos

Human video offers abundant interaction but differs from robot appearance and kinematics. UniDex performs retargeting and visual replacement before pretraining, so supervision is already robot-centered during later fine-tuning. This shifts alignment work into dataset preparation rather than eliminating it. Manual calibration and contact-frame correction remain part of the method.

3 UniDex-Dataset

3.1 Overview

The source datasets are H2O, HOI4D, HOT3D and TACO. Clips receive instructions where needed, are segmented by instruction and filtered, then retargeted to Inspire, Leap, Shadow, Allegro, Ability, Oymotion, Xhand and Wuji hands. The resulting dataset has approximately nine million paired image/pointcloud/action frames at 30 fps and 52k trajectories, spanning the reported 6–24 active-DoF range.

These are transformed training trajectories, not 52k independently collected robot executions. Table 1 compares counts, hand variety, scenes and modalities; it also marks ActionNet’s pointcloud as very low quality rather than simply equating that modality with high-quality 3D observations. Figure 3 preserves the complete example montage and word clouds. The paper does not provide per-source/per-hand balance, quantitative retargeting error or an audit of physically successful replay for every converted trajectory.

Complete original PDF S3.F2
Figure 2: The figure illustrates the complete human–robot transformation pipeline. Starting from the raw scene pointcloud, we first mask out the human hands. We then perform human-in-the-loop retargeting through a user-friendly GUI in which the user only needs to adjust slider bars to modify the dummy base offset. ① shows the retargeted result without adjustment, whereas ③ shows the final configuration with improved, more plausible hand–object contact. Finally, after kinematic retargeting, we attach the retargeted robot dexterous hands to the scene.
Complete original PDF S3.F3
Figure 3: UniDex-Dataset visualization. We show a verb–object word cloud and a subset of UniDex-Dataset. Colors denote different hands (arbitrarily assigned; black corresponds to pretraining data). UniDex-Dataset spans diverse everyday tasks across a wide range of dexterous hand embodiments, including using a mobile phone, opening a milk carton, stir-frying with a spatula, lifting a chair, solving a Rubik’s cube, and more.

3.2 Human-Robot Transformation

Transforming human data into robot trajectories requires overcoming two core gaps: kinematic and visual. We outline our methods below.

3.2.1 Kinematic Retargeting

Retargeting aligns robot fingertips to human fingertip targets. A six-DoF dummy-base offset permits translation/rotation of the robot hand relative to the observed human-hand frame. The forward kinematics compose the fixed human/dummy pose, offset and robot fingertip transform; stacking fingertip residuals defines the inverse-kinematics error.

PyBullet solves the multi-fingertip IK with joint limits and damping. Dependent mimic joints are updated from their masters using the prescribed affine relationship, followed by repeated error evaluation. A user adjusts base offsets and solver settings through a GUI, first calibrating each source-dataset/robot-hand pair and then correcting selected contact-rich frames.

The goal is plausible fingertip contact with modest interaction, not a formal contact-feasibility guarantee. The stated objective contains no explicit force-closure, friction, collision or object-dynamics constraint; small fingertip error alone cannot certify successful execution. Manual effort, failure rate and convergence across the full dataset are not quantified.

Equations 1–3
X⋆=[x1⋆,…,xm⋆]∈ℝ3×m,X^{\star}=\big[x_{1}^{\star},\ldots,x_{m}^{\star}\big]\in\mathbb{R}^{3\times m}, (1)
xi​(q,Toffset)=Trans⁡(Tworlddummy​Toffset​Ti​(q))∈ℝ3,x_{i}(q;T_{\text{offset}})\;=\;\operatorname{Trans}\!\left(T_{\text{world}}^{\text{dummy}}\,T_{\text{offset}}\,T_{i}(q)\right)\in\mathbb{R}^{3}, (2)
e⁡(q,Toffset)=[x1​(q,Toffset)−x1⋆xm​(q,Toffset)−xm⋆]∈ℝ3​m.e(q,T_{\text{offset}})=\begin{bmatrix}x_{1}(q;T_{\text{offset}})-x_{1}^{\star}\\ \vdots\\ x_{m}(q;T_{\text{offset}})-x_{m}^{\star}\end{bmatrix}\in\mathbb{R}^{3m}. (3)
Table 1 original caption. The overlaid cross and check mark denotes very low-quality ActionNet pointclouds.
Original selectable captionTable 1: Comparison between UniDex-Dataset and other dexterous manipulation datasets. UniDex-Dataset advances in total trajectories, variety across hands/actions/scenes, and supports for all perception modalities.
✗
✓
denotes the pointcloud in ActionNet [20] is very low-quality.
Complete original PDF S3.T1
Original selectable table
Dataset # of Trajectories # of Hands Language Annotations Varied Scenes RGB Depth Pointcloud
UniDex-Dataset 52K 8 ✓ ✓ ✓ ✓ ✓
ActionNet [20] 30K 2 ✓ ✗ ✓ ✓
✗
✓
RoboMind [56] 19K 1 ✓ ✗ ✓ ✓ ✗
RealDex [35] 2K 2 ✓ ✗ ✓ ✓ ✓
Equation 4
qjs=k​qjm+cq_{j_{s}}=k\,q_{j_{m}}+c (4)

3.2.2 Visual Alignment

RGB-D frames become pointclouds. WiLoR and SAM2 identify human hands, whose points are removed. Retargeted robot-hand meshes are inserted and rendered into the scene, then reprojected with a pinhole camera to respect depth ordering and the single-view deployment setting. This reduces appearance mismatch but cannot recover occluded object geometry or verify contact dynamics.

4 UniDex-VLA

4.1 Unified Action Space: FAAS

FAAS has 82 dimensions: 18 for two wrist poses and 64 for two sets of hand-joint slots. Each wrist uses two three-vectors for continuous rotation plus three translations; each hand has 32 joint slots. Functionally similar controllable channels share indices, including mimic-joint channels where the source treats them as actuators.

The main text says 21 base actuator slots are shared, whereas Appendix C allocates five slots to each of five fingers, totaling 25 finger positions, followed by two extra wrist slots and five future-use slots. These are not the same count; the complete per-hand mapping in Figure 15 is retained. Active actuators, full kinematic joints and reserved representation dimensions should not be conflated. FAAS avoids predicting human coordinates and retargeting them after inference, but still requires embodiment-specific indexing and control execution.

Complete original PDF S4.F4
Figure 4: Function–Actuator–Aligned Space (FAAS). We show the thumb and ring fingers of Oymotion (11 actuators), Allegro (16), Inspire (12), and Wuji (20), with colors denoting individual joints, curves indicating rotation directions, and dotted lines indicating rotation axes. Indices {0,1,3,5,6} are aligned across all four hands because the corresponding joints share similar functional roles.
Complete original PDF S4.F5
Figure 5: Overview of UniDex-VLA. At time tt, the model consumes a single-view colored pointcloud PtP_{t}, a language instruction ℓt\ell_{t}, and proprioception qtq_{t}, and predicts an HH-step action chunk At=[at,…,at+H−1]A_{t}=[a_{t},\ldots,a_{t+H-1}] expressed in the unified action space FAAS. Uni3D [78] encodes the colored pointcloud; features are fused with text and proprioception in the backbone and decoded into FAAS actions. The policy is pretrained on UniDex-Dataset and optimized with a conditional flow-matching objective.

4.2 VLA Policy

The policy conditions a generative action model on 3D scene geometry, language and robot state. FAAS supplies a shared output layout, while the pointcloud provides explicit geometric information. The experiments test the combined architecture and pretraining, without independently isolating every contribution of 3D sensing, action alignment and dataset design.

4.2.1 Observations and Action Outputs

Inputs are a cropped/downsampled single-view colored pointcloud, a natural-language instruction and proprioception. Outputs are an action chunk in FAAS. Proprioceptive wrists use absolute poses; commanded wrist poses are relative to the first frame of the chunk. Joint states and commands use the corresponding abstracted slots. The main model’s exact action horizon and control rate are not stated, although the baseline appendix specifies some of those settings.

4.2.2 Model Architecture

The architecture follows the π₀ family but replaces PaliGemma’s 2D vision encoder with Uni3D. Uni3D inherits a pretrained vision-transformer representation aligned with image/text features; the backbone combines it with text/proprioception, and an action expert performs conditional flow matching. Ten Euler steps with step size 0.1 generate a chunk at inference.

The “No Pretrain” control removes UniDex-Dataset pretraining, not necessarily the pretrained language/visual components of this architecture. Its comparison is evidence for the robot-centered dataset initialization, not training every component from scratch.

5 Experiments

5.1 Experimental Setup

A seven-DoF Franka arm uses Inspire (6 active/12 full DoF), Wuji (20 active DoF) or Oymotion (6 active/11 full DoF), viewed by a RealSense L515. Inspire performs coffee pouring and sweeping; Wuji performs spray-bottle watering, scissors cutting and mouse use. Oymotion is used in the transfer experiment.

Each main task receives 50 robot demonstrations collected with OpenTeleVision, dex-retargeting and Apple Vision Pro. Coffee/sweeping/watering/cutting have grasp and execution stages; mouse use requires dragging a file into a folder and clicking. Average task progress averages stage success, while final success records complete task completion. The measures answer different questions.

Comparators are Diffusion Policy, 3D Diffusion Policy, gripper-pretrained π₀ and UniDex without dataset pretraining. π₀ and both UniDex variants use FAAS, while DP/DP3 retain lower-dimensional outputs. Architecture, perception, pretraining and some optimization settings therefore differ across methods; the no-pretraining variant is the closest dataset ablation.

Complete original PDF S5.F6
Figure 6: Real-world experiments setup overview
Complete original PDF S5.F7
Figure 7: Our real-robot benchmark comprises 5 challenging tool-use tasks. We visualize the key stages of each task, illustrating the precise dexterous control required to successfully complete them.
Complete original PDF S5.F8
Figure 8: Spatial generalization. Left: the kettle and dripper are placed at out-of-distribution (OOD) positions relative to training demonstrations. Red and green lines circling regions denote the training placement ranges for the kettle and dripper, respectively. Right: average task progress for different methods (10 trials each).
Complete original PDF S5.F9
Figure 9: Object generalization. Left: we replace the original black kettle with a smaller purple kettle that differs in color, size, and functional parts (handle & spout). Right: average task progress for different methods (10 trials each).
Complete original PDF S5.F10
Figure 10: Hand generalization (zero-shot skill transfer). We transfer a policy trained on the Inspire Hand to Wuji and Oymotion. Table reports average task progress (%) under zero-shot deployment (10 trials each).
Original selectable table
Hand Type π0\pi_{0} UniDex-VLA (No Pretrain) UniDex-VLA
Wuji 0% 0% 40%
Oymotion 10% 5% 60%

5.2 Performance

Each method/task is evaluated for 20 trials. UniDex’s average task progress is 87.5% for coffee, 82.5% for sweeping, 85% for watering, 60% for cutting and 90% for mouse use. Their mean is 81%, compared with DP 29%, DP3 35%, π₀ 38% and UniDex without dataset pretraining 32.5%.

Average final success is separately reported as 76%, versus 22/30/35/23% for those comparators. The 81% headline is therefore not the complete-task success rate. Figure 11’s ±12.1-point progress spread matches the sample standard deviation across the five task scores; the other reported progress spreads likewise closely track across-task variation. They are not confidence intervals over evaluation trials or repeated model training.

For cutting, 60% progress versus the best competing 32.5% is a 27.5-point absolute gain, or approximately 84.6% relative improvement. The full model’s large gain over its 32.5%-average no-pretraining control supports dataset pretraining in this setup. Trial-level confidence intervals, multiple training seeds and a separate FAAS-versus-alternative-action ablation are not reported.

Complete original PDF S5.F11
Figure 11: Average task progress across five real-world tasks (top), with aggregate averages of average task progress and final success rate (bottom) over 5 tasks. Each task/algorithm uses 20 trials.
Original selectable table
Model DP DP3 π0\pi_{0} UniDex-VLA (No Pretrain) UniDex-VLA
Average Task Progress (avg) 29.0±19.9%29.0\pm 19.9\% 35.0±17.1%35.0\pm 17.1\% 38.0±7.4%38.0\pm 7.4\% 32.5±18.5%32.5\pm 18.5\% 81.0±12.1%\mathbf{81.0\pm 12.1\%}
Final Success Rate (avg) 22.0±22.5%22.0\pm 22.5\% 30.0±18.7%30.0\pm 18.7\% 35.0±10.0%35.0\pm 10.0\% 23.0±12.0%23.0\pm 12.0\% 76.0±17.8%\mathbf{76.0\pm 17.8\%}

5.3 Generalization

Spatial transfer. On coffee placements outside the original demonstration regions, Figure 8 reports progress of 0% DP, 20% DP3, 20% π₀, 50% no-pretraining UniDex and 75% UniDex; adding DemoGen raises it to 95%. DemoGen translates segmented object pointclouds and realigns robot states using task/motion planning. The augmented result includes synthetic training coverage of new placements, rather than testing an entirely unmodified training distribution.

Object transfer. Replacing one black kettle with a smaller purple kettle changes color, size, handle and spout. Figure 9 reports 80% progress for UniDex, versus 0/15/10/30% for DP/DP3/π₀/no-pretraining UniDex. This is a useful single-object transfer test, not an exhaustive test of general tool understanding.

Hand transfer. The coffee policy fine-tuned on Inspire gives 40% progress on Wuji and 60% on Oymotion without further task fine-tuning; the corresponding π₀ results are 0/10% and no-pretraining results 0/5%. Figure 10 explicitly labels these values average task progress, although the body calls them success. Both target hands already occur among the eight pretraining embodiments, so the result is task-skill transfer across known hand types, not control of wholly unseen hardware. Each generalization setting uses ten trials.

5.4 UniDex-Cap for Human-Robot Data Co-train

UniDex-Cap rigidly mounts a RealSense L515 to Apple Vision Pro, combines RGB-D with hand/head poses, applies the same retargeting pipeline, changes viewpoint and downsamples human motion to teleoperation speed. The coffee study varies 0–40 robot demonstrations and 0–90 transformed human demonstrations, with 20 evaluation trials per cell.

Human examples generally help when robot data are available, but the zero-robot row remains zero throughout. The effect is not strictly monotonic: at 40 robot demonstrations, increasing human demonstrations from 20 to 30 lowers displayed progress from 80 to 77; at 30 robot demonstrations, 50→60 human examples gives 87→80. The source’s “consistently improves” statement is too strong for every cell.

The proposed roughly two-human-for-one-robot exchange rate is an empirical boundary estimate for this task and range, not a universal conversion factor. The main coffee baseline is 87.5% progress, shown rounded to 87 in the heatmap legend. Human demonstrations are reported about 5.2× faster to collect, but total savings also depend on calibration, manual correction, processing, hardware and mixing requirements; these overheads are not quantified.

Complete original PDF S5.F12
Figure 12: (a,b) show the components of UniDex-Cap. (c,d) shows the example captured data and converted robot-executable trajectories.
Complete original PDF S5.F13
Figure 13: Human-Robot co-training. Average task progress versus the numbers of transformed human demos (hh) and robot demos (rr). Colors indicate different performance bands (green: comparable to the r=50r{=}50 robot-only result). Each point averages over 20 trials.

6 Conclusion and Limitation

The suite shows that retargeted human demonstrations can improve dexterous policy pretraining and reduce some task-specific robot-data demand. Its evaluated scope is five tasks on two main hands plus coffee transfer to two pretraining-known hand types. The authors identify unused action-free or weakly labeled egocentric data as a scaling opportunity. Broader hardware coverage, verified physical replay, retargeting labor and statistical robustness remain open.

7 Acknowledgment

We would like to give special thanks to Wuji Technology Inc. for providing the hardware support, and Hojin Bae, Haoxu Huang, Shaoting Zhu for their technical support. Tsinghua University Dushi Program supports this project.

Appendix A Training Details

A.1 UniDex-VLA Flow-Matching Loss

The target velocity for the linear path Aτ=τA+(1−τ)ε is A−ε, with standard Gaussian ε. Equation 5 uses an expected norm of velocity error; the original expression has norm bars but no explicit square. This edition preserves that stated objective instead of silently replacing it with a squared-error formula.

There is a notation inconsistency: the stated conditional Gaussian covariance is (1−τ)I, while the given sampling equation yields (1−τ)²I. The sampling rule and original formula are both retained; this identifies a discrepancy in the paper, not a verified implementation error. Starting from Gaussian actions, forward Euler with δ=0.1 advances the vector field to the denoised chunk.

Equations 5
Lτ​(θ)=𝔼p⁡(At∣ot),q⁡(Atτ∣At)​[‖vθ​(Atτ,ot)−u⁡(Atτ∣At)‖],L^{\tau}(\theta)=\mathbb{E}_{p(A_{t}\mid o_{t}),\,q(A_{t}^{\tau}\mid A_{t})}\left[\bigl\|v_{\theta}(A_{t}^{\tau},o_{t})-u(A_{t}^{\tau}\mid A_{t})\bigr\|\right], (5)
Atτ+δ=Atτ+δ​vθ​(Atτ,ot),A_{t}^{\tau+\delta}\;=\;A_{t}^{\tau}+\delta\,v_{\theta}\!\left(A_{t}^{\tau},\,o_{t}\right),

A.2 UniDex-VLA Pretraining

Pretraining uses eight NVIDIA H800 GPUs, batch size 128, three epochs (about 30k steps) and roughly 24 hours. AdamW starts at 10⁻⁴ learning rate with a cosine schedule; the source additionally states a factor-0.95 decay at the second epoch. Weight decay is 10⁻¹⁰ and gradient norm is clipped at 1. Sampling/chunking details are needed to relate the reported step count to the nine-million-frame dataset.

A.3 UniDex-VLA Post-training

Task fine-tuning uses two H800 GPUs, batch size eight and AdamW at 2.5×10⁻⁵ without a scheduler. Ordinary data train for 50 epochs, about 3k steps/four hours; DemoGen-augmented data train for two epochs, about 1.8k steps/2.5 hours. Both use weight decay 10⁻¹⁰ and gradient clipping at 1. These are distinct data/optimization regimes.

A.4 Baselines

DP and DP3 use AdamW at 10⁻⁴, observation horizon four, action horizon 32, batch size 32 and 400 epochs. π₀ uses 2.5×10⁻⁵, batch size eight, 50 epochs and ten denoising steps. Baselines are described as trained to validation convergence. The UniDex no-pretraining variant uses the same post-training settings as the full model; broader comparisons do not use identical budgets or input/output representations.

Appendix B Human-in-the-loop Retargeting GUI

The web GUI exposes left/right dummy-base translations and rotations, IK iterations, mimic-joint settings, damping and residual thresholds. Figure 14 retains the complete interface and pointcloud visualization. It makes manual retargeting inspectable, but the paper’s “few tweaks” description is not accompanied by timing, inter-annotator consistency or error statistics.

Complete original PDF A2.F14
Figure 14: Human-friendly web-basedGUI for retargeting human demonstrations to robot executions. Users can adjust the IK parameters, dummy links, and other settings through the GUI to obtain satisfactory retargeted robot trajectories.

Appendix C FAAS Details

For each hand, indices 0–4, 5–9, 10–14, 15–19 and 20–24 designate thumb, index, middle, ring and little fingers. Indices 25–26 hold extra Shadow wrist joints; 27–31 are reserved. Figure 15 preserves both views of all eight hand mappings at the original 7,560×2,755 resolution. This detailed 25-finger-slot allocation should be distinguished from the main text’s 21 shared actuator-slot count and from each hand’s active DoFs.

Complete original PDF A3.F15
Figure 15: Joint mappings of different robotic hands used in FAAS. From left to right are Ability, Allegro, Inspire, Leap, Oymotion, Shadow, Wuji, and Xhand. The two rows show different views of the joint mappings on the right hand.

Appendix D UniDex-Cap Setup Calibration

A rigid printed mount stabilizes the Vision Pro–RealSense extrinsic relationship for a given user. A short clip and sliders align the estimated hand skeleton to the depth pointcloud; the resulting homogeneous transform converts Vision Pro poses into camera coordinates. Figure 16 retains before/after calibration views and Equation 6 the coordinate conversion.

A spatial rigid transform establishes coordinate alignment, not time synchronization by itself. The system is described as synchronized, but timestamp alignment, latency/error measurement and quantitative calibration accuracy are not detailed. Mechanical rigidity also does not remove the need to calibrate a changed mounting/user arrangement.

Equation 6
PRS=TRSVP​PVP,P_{\mathrm{RS}}\;=\;T^{\mathrm{VP}}_{\mathrm{RS}}\,P_{\mathrm{VP}}, (6)
Complete original PDF A4.F16
Figure 16: GUI for UniDex-Cap calibration. (a) shows the initial state before calibration; (b) shows the calibrated result where the hand poses captured by Vision Pro align with the 3D point cloud captured by the RealSense L515 camera.

Appendix E Core Contribution List

Gu Zhang leads the project, dataset/model/action-space design, robot infrastructure and writing. Qicheng Xu leads VLA training and pipeline optimization; Haozhe Zhang leads dataset processing and improves robot/capture systems. Jianhan Ma implements retargeting and visualizations; Long He implements DemoGen; Yiming Bao collects robot/human data and contributes to DemoGen. The appendix identifies these six core contributors, while the paper’s complete 19-author list remains above.

Concept demonstration

Open image ↗