Literature
Display
← All papers

EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

Ruijie Zheng · Dantong Niu · Yuqi Xie · Jing Wang · Mengda Xu · Yunfan Jiang · Fernando Castañeda · Fengyuan Hu · You Liang Tan · Letian Fu · Trevor Darrell · Furong Huang · Yuke Zhu · Danfei Xu · Linxi Fan

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

Complete original PDF S0.F1
Figure 1: EgoScale: Two-stage human-to-robot learning framework. A flow-based Vision-Language-Action (VLA) policy is first pretrained on 20,854 hours of egocentric human videos using wrist motion and retargeted dexterous hand actions. A lightweight mid-training stage with aligned human robot play data (pairs highlighted with green and gray boundaries) adapts the representation to robot sensing and control. The resulting policy is post-trained on downstream tasks, enabling efficient learning of dexterous manipulation and one-shot generalization to unseen skills.

At a glance

EgoScale learns wrist and finger actions from 20,854 hours of egocentric human video, then uses a small aligned human–robot dataset to adapt the policy to robot control. On five dexterous tasks, the complete recipe raises average success from 2% to 56%. Human-data scale improves the tested policies, but the fitted scaling formula, evaluation counts and some scoring rules contain inconsistencies that limit quantitative interpretation.

1 Introduction

The main idea is to separate broad human motion learning from precise embodiment alignment. Human pretraining supplies diverse action supervision; matched human–robot mid-training connects it to robot cameras and joints; task-specific post-training supplies the target behavior. Robot demonstrations remain necessary.

This edition covers all 22 pages of arXiv v1, posted 18 February 2026, with all 15 authors verified. All ten numbered figures, both Figure 2 panels, the displayed scaling equation, complete appendix evaluation instructions/rubrics and 46 references are retained.

2 Method

A shared flow-based policy predicts future action chunks from images and language. Human motion is converted to relative wrist poses and robot-hand joint targets. A learnable placeholder replaces missing human proprioception; embodiment-specific adapters handle robot state and hand-action interfaces.

2.1 Human Action Representation

Combine estimated camera-to-world motion with camera-frame wrist pose to obtain the wrist’s world pose. The printed relative transform is ΔWₜ=(W₀)⁻¹Wₜ: motion relative to the chunk’s reference wrist, although the prose calls it consecutive-frame motion. A common global coordinate transform cancels, but tracking errors and robot reachability constraints remain.

Retarget human hand keypoints into the Sharpa hand’s 22 joint coordinates under joint limits. This retains finger-level supervision rather than using wrist motion alone. The common action description does not make human and robot kinematics physically identical.

2.2 Human Data Sources and Processing

Stage I totals 20,854 hours, mostly unconstrained egocentric recordings, plus 829 hours of accurately tracked EgoDex data within the mixture. Reported coverage is 9,869 scenes, 6,015 tasks and 43,237 objects. SLAM and hand tracking provide noisy action labels for 30-fps video; data quantity does not make these labels exact.

Stage II uses 344 tabletop tasks, approximately 30 human and five robot trajectories per task: about 50 human hours and four robot hours. Matched viewpoints and calibrated cameras align observations; Vive wrist trackers and Manus gloves provide synchronized motion. This is comparable task/scene collection, not a claim that every human frame has a paired robot action.

Complete original PDF S2.F2.sf1
(a) Egocentric Human Data Collection.
Complete original PDF S2.F2.sf2
(b) EgoScale Model Architecture.
Figure 2: Human Data Collection and Model Architecture. (Left) Aligned human-robot mid-training data are collected using the same sensing setup as the robot. Vive trackers and Manus gloves capture arm and hand motion, while one head-mounted camera and two wrist-mounted cameras record egocentric and wrist views, enabling consistent perception–action alignment. (Right) A flow-based VLA policy with a VLM backbone and DiT action expert. Human and robot data are unified through a wrist-level action representation, with lightweight embodiment-specific adapters for proprioception and hand actions.

2.3 Model Architecture

The policy follows the GR00T N1 flow-based design. Vision-language features condition a diffusion-transformer action expert. Human samples use a learned state token; robot samples supply proprioception. Lightweight embodiment-specific encoders/decoders adapt state and hand commands while the visual/language representation, action expert and relative wrist representation are shared.

2.4 Training Recipe

Human pretraining updates every parameter for 100,000 steps with batch size 8,192, learning rate 5×10⁻⁵ and 256 GB200 GPUs. Aligned mid-training uses 50,000 steps, batch 2,048 and rate 3×10⁻⁵. The paper says its vision-language backbone is frozen while the separately identified vision encoder and action expert are updated.

Task post-training uses 10,000 steps, batch 512 and rate 3×10⁻⁵. The vision encoder is frozen after mid-training, otherwise updated. These differing adaptation rules accompany the checkpoint comparisons; the total training investment is much larger than the final few robot demonstrations alone.

2.5 Robot Systems and Control

The primary system is Galaxea R1 Pro with two seven-degree-of-freedom arms and two 22-DoF Sharpa Wave hands. Base and torso stay fixed. Arm commands use relative end-effector poses; hand commands are joint targets. One head camera and two palm-facing wrist cameras supply complementary views of hand–object contact.

3 Experiment

The five questions concern human pretraining, data scaling, aligned mid-training, transfer to another hand design and the choice of hand-action representation. Separate partial-progress scores from complete success throughout.

3.1 Experiment Setup

Main tasks are shirt rolling, single-card separation/sorting, tongs-based fruit transfer, bottle-cap unscrewing and syringe liquid transfer. Use 100 robot demonstrations per task except shirt rolling, which uses 20. Four bottle types contribute 25 demonstrations each. Image overlays standardize initial scenes.

The source reports two training seeds and generally ten trials per checkpoint. Its exception incorrectly names Task III while discussing bottles, and Appendix B says four bottles×four trials but also “12 trials total.” Those counts cannot all be correct; retain the published scores without inventing a resolved denominator or confidence interval.

Complete original PDF S3.F3
Figure 3: Post-Training Evaluation Tasks. Five dexterous manipulation tasks used to evaluate post-training performance

3.2 Large-Scale Human Pretraining Is Key to Strong Dexterous Manipulation Policy Performance

Figure 4 gives mean completion/success of 0.24/0.02 with no human pretraining, 0.53/0.28 with aligned mid-training only, 0.71/0.38 with human pretraining only, and 0.83/0.56 with both. The abstract’s “54%” gain is therefore 54 percentage points of success, not a 54% relative improvement.

The combined recipe’s success rates are 50% shirt, 65% card, 65% tongs, 59% bottle and 42% syringe. It has the highest average, but human pretraining alone does better on card success, 70% versus 65%. Improvements are not uniformly monotonic for every task and metric.

Complete original PDF S3.F4
Figure 4: Main Experimental Results. Comparison of Human Pre-train + Mid-Training, Human Pretraining, and No Pretraining across five dexterous manipulation tasks under two evaluation metrics.
Complete original PDF S3.F5
Figure 5: Scaling behavior of human pretraining. Left: Human validation loss versus training steps for models pretrained with increasing amounts of egocentric human data (1k–20k hours). Larger datasets yield stable, monotonic improvements, while smaller datasets exhibit early overfitting. Center: Optimal validation loss at convergence as a function of human data scale, revealing a near-perfect log-linear scaling law (R2=0.9983R^{2}=0.9983). Right: Downstream robot performance after post-training, measured by average task completion score, improves consistently with increased human data scale. Together, these results demonstrate predictable scaling of learned action representations and their direct translation to improved dexterous manipulation performance.

3.3 Policy Performance Scales with Pretraining Data Size

Scale human data through 1k, 2k, 4k, 10k and 20k hours, then directly post-train without aligned mid-training. Mean robot completion rises 0.30→0.45→0.48→0.57→0.71. On 2,000 held-out human episodes, evaluate 20 timesteps each, average 16 sampled action chunks per timestep, and measure wrist/hand mean squared error. This ensemble-averaged offline metric differs from a single sampled deployment action.

The five-point fit reports R²=0.9983; small datasets overfit while larger ones improve longer. This is evidence within the measured range, not validated out-of-range forecasting or a model-size scaling law. Equation 1’s units are inconsistent: with D literally in hours, D=20,000 gives a negative loss, whereas the plot is positive. A rescaled D or different intercept is needed; the printed coefficients cannot be used directly with raw hours.

Equation 1
𝐋=0.024−0.003⋅ln⁡(𝐃),\mathbf{L=0.024-0.003\cdot\ln(D)}, (1)

3.4 Aligned Human-Robot Mid-Training Enables One-Shot Transfer

“One-shot” means one robot trajectory plus 100 aligned human trajectories for the target task/object. Bottle adaptation uses three geometries, each with its own one robot and 100 human demonstrations. Figure 6 reports full-recipe completion/success of 0.92/0.88 for shirt folding and 0.64/0.556 for bottles; the prose instead reports the latter success as 0.55.

The claim that omitting either stage makes both tasks fail is too broad: shirt success is already 0.15 with mid-training alone and 0.35 with human pretraining alone, although both bottle baselines are zero. The introduction mentions folding in mid-training while this section says Fold Shirt itself is absent; shared motion primitives are not the same as wholly unseen motion or absence from the large human corpus.

Complete original PDF S3.F6
Figure 6: Aligned mid-training enables emergent one-shot transfer. During post-training, the policy is trained on only a single robot demonstration per task, together with aligned human demonstrations (100 trajectories per object).

3.5 Human Pretraining Enables Cross-embodiment Transfer

Transfer to Unitree G1’s shorter arms and seven-DoF tri-finger hands uses embodiment adapters and G1 play data during mid-training. A separate Homie policy handles lower-body balance/locomotion; EgoScale predicts upper-body commands. This is adapted transfer, not zero-shot deployment on an unseen body.

For pen-in-bin and dish handover, full completion is 0.83/0.88 versus mid-training-only 0.45/0.54 and human-pretraining-only 0.68/0.58. Full success is 0.67/0.50 versus 0.30/0.20 for either baseline: gains of 37 and 30 points. The source reports two seeds×ten trials, but some plotted rates, including 0.67, do not align exactly with that denominator; rounding or additional averaging is not explained.

Complete original PDF S3.F7
Figure 7: Mid-training enables cross-embodiment generalization to G1 Tasks. Integrating a small amount of G1 play data during mid-training enables strong transfer to a low-DoF tri-finger hand, outperforming mid-training or pre-training only.

3.6 Hand Action Space Design for Human Pretraining

On card/tongs/bottle tasks, completion is 0.56/0.24/0.26 for wrist-only supervision, 0.17/0.76/0.55 for fingertip supervision and 0.74/0.79/0.61 for retargeted joints. Full joints are best on all three here. Fingertips help tongs and bottles but hurt cards relative to wrist-only, so richer geometric labels are not automatically better. Reported contact/grasp failure descriptions are qualitative explanations.

Complete original PDF S3.F8
Figure 8: Task completion score across tasks for different human pretraining action representations.

4 Related Work

Human-video learning spans visual representation pretraining, high-level planning and dense action supervision. EgoMimic, DexWild and EgoVLA provide nearby alignment/retargeting approaches; EgoScale emphasizes much larger human data and articulated-hand targets.

Prior robot-data scaling studies motivate separating data diversity from raw demonstration count. Dexterous manipulation also builds on analytic grasping, learned dynamics, contact representations and transferable hand policies. The present experiments compare training variants of one framework rather than benchmarking every cited system.

5 Acknowledgement

The authors credit robot operators for data collection/evaluation and colleagues for technical feedback. The results depend on substantial collection and compute infrastructure as well as the compact downstream demonstration sets.

6 Conclusion

The measured result is that larger human-action datasets and aligned mid-training substantially improve these dexterous policies. Unlabeled-video learning, joint scaling of data/model capacity, stronger long-horizon composition and zero-shot novel-task execution are proposed future directions. No saturation over five data sizes does not guarantee unlimited gains, and noisy tracking, calibration requirements, narrow task coverage and inconsistent evaluation documentation remain practical limits.

Appendix A Details of Robot System

Figure 9 preserves both robot layouts and sensor placements: an OAK-D-Wide head camera and two OAK-1-Wide wrist cameras, with Sharpa hands on R1 Pro and tri-finger hands on G1. The robot-system appendix is A, despite the main text referring readers to Appendix B for the system figure.

Complete original PDF A1.F9
Figure 9: Visualization of the robot system setups on the Galaxea R1 Pro and Unitree G1 platforms. The Galaxea R1 Pro is equipped with 22-DoF Sharpa dexterous hands, while the Unitree G1 is mounted with 7-DoF tri-finger hands. Both robots have with two OAK-1-Wide cameras for the wrist view and an OAK-D-Wide head-mounted camera to provide egocentric visual observations.

Appendix B Task Descriptions and Evaluation Rubric

The appendix defines completion scores separately from binary end-to-end success. Shirt/card tasks use the furthest achieved milestone; other tasks sum subskill rewards. Exact instructions, all numeric weights and trial-count wording are preserved below because they define the reported metric and expose unresolved arithmetic inconsistencies.

R1 Pro Sharpa

The full source rubric covers five standard tasks and two one-shot tasks. Standard tongs rewards tool grasp, fruit pickup, placement and tool return; the syringe rubric separately scores pickup, targeting, plunger actions, release and disposal. Bottle trial counts conflict as noted above.

One-shot shirt weights sum to 1.1 before the optional messy-fold penalty, while one-shot bottle weights sum to 1.2 if all listed stages are additive. Treating the two rotation milestones as mutually exclusive would change the latter total, but the text does not specify that convention or clipping. The claimed [0,1] range is therefore not fully defined by the printed rules; originals are retained rather than normalized silently.

Original R1 Pro Sharpa evaluation instructions and scoring
  • Task I: Shirt Rolling. Text Instruction: “Roll the T-shirt and put it into the basket.” The robot must coordinate both hands to progressively fold, roll, and place a deformable T-shirt into a basket.

    Grading rubric (progress-based):

    • 0.00.0: No folding behavior.

    • 0.30.3: Executes an initial fold.

    • 0.50.5: Executes multiple folds or partial rolling.

    • 0.80.8: Completes a continuous rolling motion into a compact shape.

    • 1.01.0: Places the rolled shirt into the basket.

    Evaluation trials: 10 trials.

  • Task II: Card Sorting. Text Instruction: “Pick up the card and sort it into the correct card holder.” The robot must separate a single card from a tightly stacked deck and insert it into the correct holder.

    Grading rubric (progress-based):

    • 0.00.0: No successful pickup; entire deck disturbed or lifted.

    • 0.30.3: Poor grasp; multiple cards lifted unintentionally.

    • 0.50.5: Single card successfully picked up.

    • 0.70.7: Card placed with noticeable disturbance or incorrect insertion.

    • 0.90.9: Correct card placed correctly with minor disturbance.

    • 1.01.0: Single correct card cleanly placed into the correct holder.

  • Task III: Tong Fruit Transfer. Text Instruction: “Use the tong to pick up the fruit and place it into the basket.” The robot must first grasp the tong and then use the tool to pick up a target object and place it into a container. This task is evaluated with 2 random fruits, lemon and plum, each with 5 evaluation trials.

    Grading rubric (additive):

    • +0.4{+0.4}: Successfully grasps the tongs.

    • +0.2{+0.2}: Picks up the fruit using the tongs.

    • +0.2{+0.2}: Places the fruit into the basket.

    • +0.2{+0.2}: Returns the tongs to the table.

  • Task IV: Bottle Cap Unscrewing. Text Instruction: “Unscrew the cap from the bottle.” The robot must grasp a bottle, rotate the cap multiple times, remove it, and place it on the table. To test generalization across object variations, we evaluate 4 bottle configurations with different heights and cap sizes. Performance is measured over 12 trials in total, with 4 trials conducted for each bottle.

    Grading rubric (additive):

    • +0.1+0.1: Grasps the bottle.

    • +0.5+0.5: Unscrews the cap with at least three continuous rotations.

    • +0.2+0.2: Fully removes the cap.

    • +0.2+0.2: Places the cap on the table.

  • Task V: Syringe Liquid Transfer. Text Instruction: “Pick up the syringe, draw liquid from tube A, inject it into tube B, and throw the syringe into the trash can.” This task requires precise tool manipulation, spatial alignment, and long-horizon sequencing.

    Grading rubric (additive):

    • +0.1+0.1: Picks up the syringe.

    • +0.1+0.1: Correctly aims at tube A.

    • +0.2+0.2: Pulls the plunger to draw liquid.

    • +0.1+0.1: Re-aims at tube B.

    • +0.2+0.2: Pushes the plunger to inject liquid.

    • +0.2+0.2: Hands over or releases the syringe.

    • +0.1+0.1: Disposes of the syringe into the trash can.

    To evaluate one-shot adaptation, we consider two additional manipulation tasks on the R1 Pro Sharpa platform. For each task, the policy is post-trained using a single robot demonstration, and performance is evaluated using a task-specific completion score in [0,1][0,1].

  • Task VIII: One-Shot T-Shirt Folding. Text Instruction: “Fold the T-shirt.”

    Grading rubric (additive):

    • +0.4+0.4: Successfully folds at least one sleeve.

    • +0.4+0.4: Successfully folds both sleeves.

    • +0.3+0.3: Executes a bottom fold.

    During the evaluation, we also subtract a 0.1 if it does one of the fold but the folding is messy.

  • Task IX: One-Shot Bottle Cap Unscrewing. Text Instruction: “Unscrew the cap from the water bottle.” The task decomposes naturally into discrete, well-defined manipulation steps and is evaluated using additive scoring.

    Grading rubric (additive):

    • +0.1+0.1: Grasps the bottle.

    • +0.2+0.2: Performs one to two successful unscrewing rotations.

    • +0.5+0.5: Performs at least three continuous unscrewing rotations.

    • +0.2+0.2: Fully removes the cap.

    • +0.2+0.2: Places the cap on the table.

For all tasks, we additionally report a binary task success rate, where a trial is considered successful only if the task is completed end-to-end according to the instruction.

G1 Tasks

G1’s canister/marker task assigns four rewards of 0.25. Dish handover assigns 0.11 for pickup, handover and rack placement for each of three plates; those nine printed terms total 0.99 although the text says a maximum of 1.0. This may be rounding, but the exact convention is unstated.

Original G1 evaluation instructions and scoring

In addition to the R1 Pro Sharpa evaluations, we evaluate cross-embodiment generalization on the Unitree G1 platform using two manipulation tasks. Both tasks are evaluated using additive scoring, reflecting their decomposition into discrete, well-defined sub-skills.

  • Pen in Bin. Text Instruction: “Marker canister task.” The robot must open a canister, pick up a marker, and place the marker into the canister.

    Grading rubric (additive):

    • +0.25+0.25: Picks up or opens the canister.

    • +0.25+0.25: Places the canister down stably.

    • +0.25+0.25: Picks up the marker.

    • +0.25+0.25: Places the marker into the canister.

  • Dish Handover in Rack Text Instruction: “Put plates on dishrack.” The robot is presented with three plates on a table and must transfer them, potentially between hands, and place them upright into a dish rack.

    Grading rubric (additive, per plate): For each plate, we assign:

    • 0.110.11: Picks up the plate.

    • 0.110.11: Transfers the plate between hands.

    • 0.110.11: Places the plate upright into the dish rack.

    The final task completion score is computed by summing scores across all three plates, yielding a maximum score of 1.01.0 per trial.

Appendix C Details of In-the-wild Human Data Curation

The in-the-wild data analysis describes a long tail of categories, environments, tasks and objects. The complete Figure 10 combines the word cloud, charts and all seven rows of egocentric examples.

Complete original PDF A3.F10
Figure 10: Statistical distributions and qualitative examples of the egocentric human activity dataset, showing long-tailed coverage across categories, environments, tasks, and objects.

C.1 Dataset Statistics and Analysis

Dataset statistics combine category hours, environment/task frequency and an object word cloud. They characterize coverage but do not establish balanced sampling or independent generalization across all listed scene/object categories.

C.2 Category Distribution

Categories above 50 hours are shown. Retail/consumer goods account for 20.1%, fashion 11.8%, repair services 11.5%, food/beverage 11.5%, home 9.5%, construction/hardware 7.7%, food processing 6.7% and printing/design 4.4%; many smaller categories complete the mixture.

C.3 Environment Diversity

Homes, flower shops, electronics/furniture repair, woodworking and clothing stores are prominent environments. The plot omits environments below its 50-hour display threshold, so its visible bars are not an exhaustive list of the 9,869 scenes.

C.4 Task Coverage

Frequent tasks include folding clothes, cleaning shoes, potting plants, ironing, assembling boxes and food preparation, with many packing, sorting, repairing and decorating activities. Long procedures offer varied motion, but dataset duration alone does not measure a policy’s planning horizon.

C.5 Object Vocabulary

The vocabulary includes boxes, bags, bowls, trays, tape, clothes, dough, batteries and circuit boards. The word cloud illustrates repeated objects across diverse tasks; it is not a quantitative test of compositional generalization.

Appendix D Hand Retargeting Algorithm

Use URDF forward kinematics to map 22 Sharpa joint angles to 20 robot keypoint poses. Solve a nonlinear optimization with CasADi/IPOPT, joint-limit constraints, warm starts from the preceding frame and a first-order exponential smoothing filter. The source does not enumerate all objective weights or add explicit collision/contact constraints.

The paper mixes 21-keypoint human estimates with 25-transform glove data; this appendix even describes “21” as represented by “25” keypoints. The exact correspondence is not specified. Joint limits and geometric retargeting do not certify dynamic feasibility or contact force quality.

D.1 Additional Details on Cross-Embodiment Transfer to G1

G1 transfer retains the pretrained shared representation while adapting state/action interfaces and training on G1 play. The reduced finger count and workspace are handled through learned adapters and target-body experience.

Shared Wrist Action.

Relative wrist poses provide a common coordinate convention across humans and robots. This appendix calls them frame-to-frame, whereas the explicit earlier formula references W₀. Either convention needs a consistent execution interpretation; coordinate invariance alone does not guarantee the same trajectory is reachable by both robots.

Hand Action Adaptation.

Embodiment-conditioned multilayer perceptrons encode proprioception and noisy actions and decode the shared transformer’s output into the body’s joint space. These small interfaces change while the main vision-language and action networks remain shared across embodiments.

Embodiment-Specific Mid-Training.

Mid-training updates the vision encoder, action expert and state/action encoders/decoders on a mixture that includes G1 play; the component called the vision-language backbone remains frozen. The resulting target-body adaptation is an explicit part of the successful transfer recipe.

Discussion.

The G1 policy starts from learned human motion and is then aligned to G1 control. Better results than the matched G1-data baseline support reusable representations for this second embodiment. They do not establish a universal motor prior requiring no target-robot data.

Concept demonstration

Open image ↗