Literature
Display
← All papers

WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation

Chuan Qin · Shaoting Zhu · Siyuan Luo · Siqiao Huang · Hongyu Zhao · Shanaka Baduge · Hang Zhao

Preprint posted · arXiv (preprint) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

The complete overview figure retains the heterogeneous pre-training sources, three training stages and physical robot demonstrations. Predicted body/root references and articulated-hand commands share one physical action interface.

Complete original Fig. 1: Overview of WB-WAM. Left: Heterogeneous video and motion pre-training, PICO mid-training,
and task-specific robot adaptation share an explicit physical action
representation for body and dexterous hand prediction.
Right: Real-world deployments cover locomotion and manipulation, together with language-conditioned fruit selection
and placement.
Fig. 1: Overview of WB-WAM. Left: Heterogeneous video and motion pre-training, PICO mid-training, and task-specific robot adaptation share an explicit physical action representation for body and dexterous hand prediction. Right: Real-world deployments cover locomotion and manipulation, together with language-conditioned fruit selection and placement.

At a glance

WB-WAM learns humanoid body, root and dexterous-hand references jointly with future video. A shared physical action space lets 1,880.2 hours of heterogeneous data contribute complementary labels, followed by retargeted human demonstrations and robot adaptation.

The model reports 81.9% mean simulation success and 84% across five physical tasks. Task-aligned human mid-training raises four-task success with 30 robot demonstrations per task to 73.75%, versus 65% using 100 robot demonstrations without that stage. This reduces robot-data collection while adding human supervision of the same tasks.

I INTRODUCTION

Humanoid manipulation requires coordinated body motion, locomotion and articulated hands. Conventional arm-focused pre-training offers limited direct supervision of these combined physical references.

The proposed three-stage recipe combines partially annotated external data, egocentric human transfer and task-specific robot learning. Body/root outputs use a frozen SONIC tracker at deployment; hand references directly command finger joints. Main experiments use separate task policies rather than a single policy solving every demonstrated task.

II Related Work

The paper connects humanoid foundation models with world action models. Its central design choice is direct physical whole-body action supervision during video pre-training.

II-A Humanoid Foundation Models for Loco-manipulation

GR00T, Ψ0, WholeBodyVLA and OpenHLM transfer heterogeneous human/robot experience using different action interfaces. These include continuous actions, task-space tokens and latent visual actions.

WB-WAM instead aligns available body, root and hand annotations to explicit G1 physical references before robot adaptation. Its empirical comparisons do not isolate this representation from all other architectural and training differences [6] [8].

II-B World Action Models

World action models couple predictive visual learning with action generation through inverse dynamics, autoregressive representations or coupled denoising. Humanoid variants often predict compact plans or controller latents.

WB-WAM builds on Fast-WAM’s separate video/action experts and current-frame inference pathway, while extending pre-training supervision to explicit body and hand motion [9].

III Methods

The prediction interface remains a physical body-root-hand trajectory across all three stages. Video and action experts learn jointly; a frozen controller encoder is used only when executing predicted body/root references.

Complete original Fig. 2: WB-WAM architecture and three-stage training framework. Top: Progressive training integrates heterogeneous supervision,
retargeted PICO motion, and real-robot demonstrations.
Bottom: Video and action experts jointly model visual dynamics and
whole-body actions conditioned on vision, language, and proprioception.
Body and root references are executed through SONIC, while hand
references directly control the finger joints.
Fig. 2: WB-WAM architecture and three-stage training framework. Top: Progressive training integrates heterogeneous supervision, retargeted PICO motion, and real-robot demonstrations. Bottom: Video and action experts jointly model visual dynamics and whole-body actions conditioned on vision, language, and proprioception. Body and root references are executed through SONIC, while hand references directly control the finger joints.

III-A Preliminary: World Action Modeling with MoT

The Mixture-of-Transformers backbone has modality-specific video and action parameters, with attention allowing the action expert to use video features. A frozen video VAE encodes future RGB targets.

Flow matching interpolates between clean targets and Gaussian noise, predicting noise-minus-target velocity. Video and action flow times are sampled independently. The original equations remain selectable below.

Equation 1
𝐲τ=(1−τ)​𝐲0+τ​ϵ,𝐮=ϵ−𝐲0.\mathbf{y}_{\tau}=(1-\tau)\mathbf{y}_{0}+\tau\boldsymbol{\epsilon},\qquad\mathbf{u}=\boldsymbol{\epsilon}-\mathbf{y}_{0}. (1)

III-B Problem Formulation

Predictions condition on an egocentric RGB image, proprioception and language. The 72-dimensional physical reference contains 29 body joint positions, three root quantities—roll, pitch and yaw angular velocity—and 20 joint references per hand.

These are reference commands, rather than direct body motor torques or a learned controller latent. The trained output head has 96 coordinates, with at most 72 used.

Equations 2–3
p𝜽(𝐕t,𝐀t∣𝐈t,𝐬t,ℓ),p_{\boldsymbol{\theta}}\left(\mathbf{V}_{t},\mathbf{A}_{t}\mid\mathbf{I}_{t},\mathbf{s}_{t},\ell\right), (2)
𝐚t=[𝐪tB,𝐫t,𝐪tL,𝐪tR]∈ℝ72.\mathbf{a}_{t}=\left[\mathbf{q}_{t}^{B},\mathbf{r}_{t},\mathbf{q}_{t}^{L},\mathbf{q}_{t}^{R}\right]\in\mathbb{R}^{72}. (3)

III-C Stage I: Heterogeneous Body–Hand Pre-training

Stage I mixes video-body-hand, video-hand and text-body sources. A binary validity mask identifies observed action coordinates; it masks the noisy action input and sets unobserved/padded target flow velocities to zero.

The action loss averages across the full trajectory tensor, including those zero-target entries. This is distinct from simply excluding every missing coordinate from the loss denominator. Video loss uses future latent frames only for examples with video.

Action conditioning alternates between full video and current frame, while a separate noisy video branch learns prediction. Video-free samples disable visual conditioning. At deployment, the current-frame pathway can generate actions without synthesizing future video.

Equations 4–5
𝐮a=𝐌⊙(ϵa−𝐀0).\mathbf{u}_{a}=\mathbf{M}\odot\bigl(\boldsymbol{\epsilon}_{a}-\mathbf{A}_{0}\bigr). (4)
ℒ⁡(𝜽,𝒟)=λvid​ℒvid+λact​ℒact.\mathcal{L}(\boldsymbol{\theta};\mathcal{D})=\lambda_{\mathrm{vid}}\mathcal{L}_{\mathrm{vid}}+\lambda_{\mathrm{act}}\mathcal{L}_{\mathrm{act}}. (5)

III-D Stage II: Mid-training with Retargeted PICO Motion

Stage II continues joint video/action learning using only the PICO human collection, retargeted into the shared physical space. The evaluated transfer tasks are included in that human collection.

This is transfer from task-aligned human demonstrations to limited robot demonstrations. Retargeted kinematic feasibility alone does not establish dynamic balance or feasible contact on the physical robot.

III-E Stage III: Task-Specific Real-Robot Post-training

Stage III initializes an independent policy for each robot task and trains on its teleoperated robot demonstrations. An auxiliary forward-kinematics loss penalizes G1 link position/orientation errors in the pelvis frame, after reconstructing and denormalizing body references.

The full video/action and kinematic objectives are retained. This version does not specify all loss weights, selected links or the complete optimizer schedule [30].

Equation 6
ℒIII=ℒ⁡(𝜽,𝒟Rk)+λFK​(ℒpos+βrot​ℒrot),\mathcal{L}_{\mathrm{III}}=\mathcal{L}(\boldsymbol{\theta};\mathcal{D}_{R}^{k})+\lambda_{\mathrm{FK}}\left(\mathcal{L}_{\mathrm{pos}}+\beta_{\mathrm{rot}}\mathcal{L}_{\mathrm{rot}}\right), (6)

IV WB-Datasets

WB-Datasets adds 22 hours of PICO recordings and 1,011 robot episodes to the nine external pre-training sources. Human and robot collections serve different training stages.

The external 1,880.2-hour corpus includes motion-only data, so that total is not entirely paired video with complete 72-dimensional labels.

IV-A Heterogeneous Pre-training Data

The nine sources cover egocentric video, humanoid robot trajectories and text-conditioned body motion. Human body motion is retargeted to G1 using GMR where needed; human hands are mapped to Wuji joints. Already compatible robot data undergo schema/coordinate alignment.

Curation checks poses, discontinuities, self-collisions and unavailable video, with manual inspection of ambiguous detector scores. Missing modalities and action coordinates remain explicit in metadata and masks. The full composition diagram and all nine percentages are preserved [38].

Complete original Fig. 3: Composition of 9-source dataset used for pre-training.
Fig. 3: Composition of 9-source dataset used for pre-training.
Complete original Fig. 4: Pre-training data curation with pose validity,
motion continuity, collision, and video quality checks.
Fig. 4: Pre-training data curation with pose validity, motion continuity, collision, and video quality checks.

IV-B PICO Egocentric Mid-training Data

A PICO 4 Ultra and five trackers record synchronized egocentric RGB and SMPL body motion at 20 Hz. GMR initialization is refined by inverse kinematics using joint/velocity limits, support constraints and collision avoidance.

MINT reconstructs hand keypoints from video before Wuji retargeting. The resulting collection contains 13,396 episodes across 73 tasks; each of the four transfer tasks has about 150 aligned human demonstrations [39].

IV-C SONIC Real-Robot Post-training Data

Robot data use SONIC teleoperation, PICO body/root tracking and MANUS gloves for the Wuji hands. Eight task groups cover household manipulation and fruit selection.

The reported 242,668 frames at 20 Hz correspond to 3.37 hours of sampled coverage. The eight groups comprise 1,011 episodes; the fruit group contains apple, lemon and orange targets.

V Experiments

Experiments compare simulation and physical performance, then test human mid-training, language-conditioned fruit selection and a qualitative visual variation. All original task-level counts and milestone rates remain.

Complete original Fig. 5: Robot hardware and system.
Fig. 5: Robot hardware and system.

V-A Training Details

The video expert starts from Wan2.2; the narrower action transformer uses structural parameter transfer. The action projections and proprioception encoder are randomly initialized. Video/action experts and the state encoder train jointly; VAE and UMT5 remain frozen.

Pre-training runs one epoch, batch 16, with a 3:1:1 mix of video-body-hand, video-hand and text-body samples. Full-video/current-frame action conditioning is selected with equal probability. The paper does not give a complete compute, parameter-count or training-time budget.

The five-task hardware comparison gives each baseline the same task data and 50 epochs. Human transfer uses ten epochs on all 73 PICO tasks, followed by 50 robot-training epochs on 30 demonstrations per task. The main hardware comparison starts directly from Stage I, without PICO mid-training.

V-B Deployment and System Details

The physical system is a Unitree G1 with two Wuji hands and a RealSense D455. At each replanning step, 20 denoising iterations predict 32 action timesteps; the first 20 are played at 20 Hz.

Body/root references form a ten-frame, 0.1-second-spaced motion window for the frozen SONIC encoder, producing a 64-dimensional tracker latent. Tracking runs at 50 Hz and command publication at 500 Hz; hands bypass the encoder.

The system diagram reports about 0.7 Hz model inference. How that timing is reconciled with one-second chunk playback, including asynchronous scheduling or command buffering, is not specified [12].

V-C Evaluation Metrics

Hardware evaluation uses 20 trials per task. Final success is separate from approach, contact and, for cart pushing, locomotion milestones. Task progress averages the stated milestones, including final success; means weight tasks equally.

Milestone rates are measured over all trials, rather than conditioned on prior milestones. The tables therefore need not form a monotone success funnel. Hardware confidence intervals and variation over independent policy-training seeds are not reported.

Complete original TABLE I: Comparison on HumanoidArena [ 40 ] .
For each task, we report the strongest baseline from the original benchmark.
TABLE I: Comparison on HumanoidArena [40]. For each task, we report the strongest baseline from the original benchmark.
Accessible table and original citation links
Method Football DoubleDesk P&PBox OpenDoor SitSofa Boxing VisNavi Mean SR
Best baseline (per task) DP π0.5\pi_{0.5} DP DP DP DP FM FM 45.5±\pm25.2% ACT 47.6±\pm24.2%
45.0±\pm10.8% 43.3±\pm6.2% 75.0±\pm4.1% 85.0±\pm10.8% 78.3±\pm14.3% 76.7±\pm2.4% 38.3±\pm14.3% π0.5\pi_{0.5} 51.2±\pm24.8% DP 60.0±\pm24.2%
WB-WAM 70.0±\pm8.2% 65.0±\pm4.1% 86.7±\pm2.4% 98.3±\pm2.4% 95.0±\pm4.1% 81.7±\pm2.4% 76.7±\pm4.7% 81.9±\pm12.3%
Complete original TABLE II: Main comparison with 20 real-robot trials per task.
WB-WAM achieves the highest mean SR and mean TP.
TABLE II: Main comparison with 20 real-robot trials per task. WB-WAM achieves the highest mean SR and mean TP.
Accessible table and original citation links
w/ locomotion w/o locomotion
Method Wipe the table Close the curtain Make the bed Move the pillow Tidy the cloth Mean SR Mean TP
A C S A C S A C S C S C S
ACT [41] 0/20 0/20 0/20 0/20 0/20 0/20 0/20 0/20 0/20 16/20 8/20 19/20 12/20 20% 27.5%
π0.5\pi_{0.5} [42] 13/20 12/20 12/20 11/20 10/20 6/20 11/20 10/20 10/20 19/20 8/20 16/20 16/20 52% 61.17%
GR00T N1.6 [43] 16/20 15/20 15/20 18/20 13/20 12/20 0/20 0/20 0/20 6/20 0/20 17/20 15/20 42% 48.67%
Fast-WAM [9] 0/20 0/20 0/20 8/20 3/20 2/20 0/20 0/20 0/20 4/20 0/20 9/20 4/20 6% 12.83%
DiT4DiT [24] 12/20 6/20 3/20 16/20 15/20 7/20 16/20 9/20 9/20 5/20 0/20 16/20 12/20 31% 47.5%
OpenWAM [26] 18/20 18/20 18/20 11/20 10/20 8/20 18/20 17/20 17/20 20/20 20/20 17/20 17/20 80% 82%
WB-WAM 16/20 16/20 15/20 16/20 15/20 13/20 19/20 18/20 18/20 20/20 20/20 18/20 18/20 84% 86.67%

V-D Main Comparison

Simulation uses the HumanoidArena SONIC setting; physical comparisons use a common execution interface. These comparisons evaluate the complete systems rather than an isolated ablation of body-hand supervision or the auxiliary kinematic loss.

V-D 1 Simulation Benchmark Evaluation

WB-WAM reports 81.9% mean success across seven simulation tasks, exceeding the strongest reported baseline for each task. The “best baseline” row selects different methods across tasks; it is not one policy with that combined performance.

Per-task rates and reported deviations remain in Table I, alongside individual baseline model means. This version does not describe simulation rollout counts, seed counts or a statistical significance test, so “significantly” should be read as the reported numerical improvement [40].

V-D 2 Real-World Comparison

Each of five physical tasks uses roughly 100 demonstrations and 20 evaluation trials per method. WB-WAM reaches 84% mean final success versus OpenWAM’s 80%, an improvement of four percentage points or four additional successes in 100 trials.

It is strongest on the overall mean, rather than every task: OpenWAM achieves 90% on wiping versus WB-WAM’s 75%; pillow success ties at 100%. WB-WAM averages 76.7% on the three tasks requiring locomotion, versus 71.7% for OpenWAM. The small study does not establish broad platform or task generality [26].

Complete original Fig. 6: Real-world experiments. WB-WAM enables the robot to perform a variety of whole-body manipulation tasks.
Fig. 6: Real-world experiments. WB-WAM enables the robot to perform a variety of whole-body manipulation tasks.

V-E PICO Mid-training

Using the shared PICO-midtrained checkpoint and 30 robot demonstrations per task yields 73.75% mean success on four tasks. Direct adaptation scores 48.75% with 30 and 65% with 100 robot demonstrations.

The result uses 70% fewer robot demonstrations, supplemented by the full 22-hour human dataset, including about 150 same-task human demonstrations per target. It measures robot-data efficiency rather than a 70% reduction in total supervision or transfer to unseen tasks.

Checkout reaches only 9/20 successes even with human transfer. The reported gains therefore coexist with substantial task-specific failure, and no cost-matched human-versus-robot data collection analysis is provided.

Complete original TABLE III: Effect of PICO mid-training with 20 real-robot trials per condition.
TABLE III: Effect of PICO mid-training with 20 real-robot trials per condition.
Accessible table and original citation links
w/ locomotion w/o locomotion
Training route Robot [-1pt]Demos Close the curtain Push the cart Wipe the table Checkout Mean SR Mean TP
A C S C L S A C S C S
Direct post-training 30 13/20 12/20 12/20 13/20 11/20 11/20 12/20 11/20 11/20 6/20 5/20 48.75% 51.04%
Direct post-training 100 16/20 15/20 13/20 19/20 16/20 16/20 16/20 16/20 15/20 10/20 8/20 65% 70.42%
w/ PICO mid-training 30 19/20 17/20 16/20 16/20 17/20 16/20 19/20 18/20 18/20 12/20 9/20 73.75% 78.13%

V-F Language-Conditioned Manipulation

A single fruit policy uses about 300 demonstrations across three categories and selects targets through language. Final success is 15/20 for orange, 17/20 for lemon and 16/20 for apple.

This shows conditional selection among the trained fruits. There is no reported unseen fruit category, compositional instruction or general language-understanding evaluation.

Complete original TABLE IV: Language-conditioned fruit manipulation.
TABLE IV: Language-conditioned fruit manipulation.
Accessible table and original citation links
Orange Lemon Apple
Method C S C S C S
WB-WAM 17/20 15/20 18/20 17/20 19/20 16/20

V-G Visuomotor Generalization

Cart pushing is demonstrated with additional objects changing the cart’s appearance. Figure 7 distinguishes three predicted future frames from the physical execution examples.

The changed scene is absent from the dataset, but evidence here is qualitative: no trial count, success rate or comparison is provided. Optional future-video rollout illustrates the video expert; ordinary action execution can omit that synthesis.

Complete original Fig. 7: Video prediction and real-world execution
under visual variation. The three central frames show a video rollout generated by the video expert under a modified visual condition. Real-robot cart pushing is shown in a familiar scene (left) and a novel scene absent from the dataset (right).
Fig. 7: Video prediction and real-world execution under visual variation. The three central frames show a video rollout generated by the video expert under a modified visual condition. Real-robot cart pushing is shown in a familiar scene (left) and a novel scene absent from the dataset (right).

VI Conclusion

The paper demonstrates a workable physical action interface for heterogeneous whole-body learning and task-aligned human-to-robot transfer. Its strongest evidence is the tested G1 task suite.

Limits include one robot platform, mostly separate task policies, small hardware rollout sets and no isolated component ablation. Transfer to unseen tasks and reliable contact execution remain future work.

ACKNOWLEDGMENT

The authors acknowledge BONES-SEED motion data. They disclose AI-generated illustrative elements in Figures 1–4, including the circular graphic, purple human, proprioception illustration and joint-command-discontinuity schematic. These acknowledgments remain attached to the preserved original artwork.

Original project demonstrations

The authors’ project page supplies 26 original clips: both camera views for seven simulation tasks, ten physical task/fruit demonstrations and two five-consecutive-success galleries. All videos and original posters are preserved locally.

Simulation clips play at 2× speed on the project page; consecutive-success clips are encoded at 5×. Their captions retain those distinctions. Selected successful rollouts supplement the quantitative tables rather than define an additional success-rate estimate.

Simulation · Football · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · Football · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · DoubleDesk · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · DoubleDesk · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · P&PBox · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · P&PBox · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · OpenDoor · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · OpenDoor · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · SitSofa · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · SitSofa · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · Boxing · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · Boxing · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · VisNavi · Third-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Simulation · VisNavi · First-person. The project page plays this simulation clip at 2× speed; the original video file is unchanged.
Hardware · Wipe the table. Original project task demonstration.
Hardware · Close the curtain. Original project task demonstration.
Hardware · Make the bed. Original project task demonstration.
Hardware · Move the pillow. Original project task demonstration.
Hardware · Push the cart. Original project task demonstration.
Hardware · Checkout. Original project task demonstration.
Hardware · Tidy up cloth. Original project task demonstration.
Hardware · Apple. Original project task demonstration.
Hardware · Orange. Original project task demonstration.
Hardware · Lemon. Original project task demonstration.
Hardware · Five consecutive successes · Close the curtain. The original project clip is encoded at 5× speed.
Hardware · Five consecutive successes · Tidy up cloth. The original project clip is encoded at 5× speed.
Concept demonstration

Open image ↗