Overview
The complete overview figure retains the heterogeneous pre-training sources, three training stages and physical robot demonstrations. Predicted body/root references and articulated-hand commands share one physical action interface.
At a glance
WB-WAM learns humanoid body, root and dexterous-hand references jointly with future video. A shared physical action space lets 1,880.2 hours of heterogeneous data contribute complementary labels, followed by retargeted human demonstrations and robot adaptation.
The model reports 81.9% mean simulation success and 84% across five physical tasks. Task-aligned human mid-training raises four-task success with 30 robot demonstrations per task to 73.75%, versus 65% using 100 robot demonstrations without that stage. This reduces robot-data collection while adding human supervision of the same tasks.
I INTRODUCTION
Humanoid manipulation requires coordinated body motion, locomotion and articulated hands. Conventional arm-focused pre-training offers limited direct supervision of these combined physical references.
The proposed three-stage recipe combines partially annotated external data, egocentric human transfer and task-specific robot learning. Body/root outputs use a frozen SONIC tracker at deployment; hand references directly command finger joints. Main experiments use separate task policies rather than a single policy solving every demonstrated task.
II Related Work
The paper connects humanoid foundation models with world action models. Its central design choice is direct physical whole-body action supervision during video pre-training.
II-A Humanoid Foundation Models for Loco-manipulation
GR00T, Ψ0, WholeBodyVLA and OpenHLM transfer heterogeneous human/robot experience using different action interfaces. These include continuous actions, task-space tokens and latent visual actions.
WB-WAM instead aligns available body, root and hand annotations to explicit G1 physical references before robot adaptation. Its empirical comparisons do not isolate this representation from all other architectural and training differences [6] [8].
II-B World Action Models
World action models couple predictive visual learning with action generation through inverse dynamics, autoregressive representations or coupled denoising. Humanoid variants often predict compact plans or controller latents.
WB-WAM builds on Fast-WAM’s separate video/action experts and current-frame inference pathway, while extending pre-training supervision to explicit body and hand motion [9].
III Methods
The prediction interface remains a physical body-root-hand trajectory across all three stages. Video and action experts learn jointly; a frozen controller encoder is used only when executing predicted body/root references.
III-A Preliminary: World Action Modeling with MoT
The Mixture-of-Transformers backbone has modality-specific video and action parameters, with attention allowing the action expert to use video features. A frozen video VAE encodes future RGB targets.
Flow matching interpolates between clean targets and Gaussian noise, predicting noise-minus-target velocity. Video and action flow times are sampled independently. The original equations remain selectable below.
Equation 1
| (1) |
III-B Problem Formulation
Predictions condition on an egocentric RGB image, proprioception and language. The 72-dimensional physical reference contains 29 body joint positions, three root quantities—roll, pitch and yaw angular velocity—and 20 joint references per hand.
These are reference commands, rather than direct body motor torques or a learned controller latent. The trained output head has 96 coordinates, with at most 72 used.
Equations 2–3
| (2) |
| (3) |
III-C Stage I: Heterogeneous Body–Hand Pre-training
Stage I mixes video-body-hand, video-hand and text-body sources. A binary validity mask identifies observed action coordinates; it masks the noisy action input and sets unobserved/padded target flow velocities to zero.
The action loss averages across the full trajectory tensor, including those zero-target entries. This is distinct from simply excluding every missing coordinate from the loss denominator. Video loss uses future latent frames only for examples with video.
Action conditioning alternates between full video and current frame, while a separate noisy video branch learns prediction. Video-free samples disable visual conditioning. At deployment, the current-frame pathway can generate actions without synthesizing future video.
Equations 4–5
| (4) |
| (5) |
III-D Stage II: Mid-training with Retargeted PICO Motion
Stage II continues joint video/action learning using only the PICO human collection, retargeted into the shared physical space. The evaluated transfer tasks are included in that human collection.
This is transfer from task-aligned human demonstrations to limited robot demonstrations. Retargeted kinematic feasibility alone does not establish dynamic balance or feasible contact on the physical robot.
III-E Stage III: Task-Specific Real-Robot Post-training
Stage III initializes an independent policy for each robot task and trains on its teleoperated robot demonstrations. An auxiliary forward-kinematics loss penalizes G1 link position/orientation errors in the pelvis frame, after reconstructing and denormalizing body references.
The full video/action and kinematic objectives are retained. This version does not specify all loss weights, selected links or the complete optimizer schedule [30].
Equation 6
| (6) |
IV WB-Datasets
WB-Datasets adds 22 hours of PICO recordings and 1,011 robot episodes to the nine external pre-training sources. Human and robot collections serve different training stages.
The external 1,880.2-hour corpus includes motion-only data, so that total is not entirely paired video with complete 72-dimensional labels.
IV-A Heterogeneous Pre-training Data
The nine sources cover egocentric video, humanoid robot trajectories and text-conditioned body motion. Human body motion is retargeted to G1 using GMR where needed; human hands are mapped to Wuji joints. Already compatible robot data undergo schema/coordinate alignment.
Curation checks poses, discontinuities, self-collisions and unavailable video, with manual inspection of ambiguous detector scores. Missing modalities and action coordinates remain explicit in metadata and masks. The full composition diagram and all nine percentages are preserved [38].
IV-B PICO Egocentric Mid-training Data
A PICO 4 Ultra and five trackers record synchronized egocentric RGB and SMPL body motion at 20 Hz. GMR initialization is refined by inverse kinematics using joint/velocity limits, support constraints and collision avoidance.
MINT reconstructs hand keypoints from video before Wuji retargeting. The resulting collection contains 13,396 episodes across 73 tasks; each of the four transfer tasks has about 150 aligned human demonstrations [39].
IV-C SONIC Real-Robot Post-training Data
Robot data use SONIC teleoperation, PICO body/root tracking and MANUS gloves for the Wuji hands. Eight task groups cover household manipulation and fruit selection.
The reported 242,668 frames at 20 Hz correspond to 3.37 hours of sampled coverage. The eight groups comprise 1,011 episodes; the fruit group contains apple, lemon and orange targets.
V Experiments
Experiments compare simulation and physical performance, then test human mid-training, language-conditioned fruit selection and a qualitative visual variation. All original task-level counts and milestone rates remain.
V-A Training Details
The video expert starts from Wan2.2; the narrower action transformer uses structural parameter transfer. The action projections and proprioception encoder are randomly initialized. Video/action experts and the state encoder train jointly; VAE and UMT5 remain frozen.
Pre-training runs one epoch, batch 16, with a 3:1:1 mix of video-body-hand, video-hand and text-body samples. Full-video/current-frame action conditioning is selected with equal probability. The paper does not give a complete compute, parameter-count or training-time budget.
The five-task hardware comparison gives each baseline the same task data and 50 epochs. Human transfer uses ten epochs on all 73 PICO tasks, followed by 50 robot-training epochs on 30 demonstrations per task. The main hardware comparison starts directly from Stage I, without PICO mid-training.
V-B Deployment and System Details
The physical system is a Unitree G1 with two Wuji hands and a RealSense D455. At each replanning step, 20 denoising iterations predict 32 action timesteps; the first 20 are played at 20 Hz.
Body/root references form a ten-frame, 0.1-second-spaced motion window for the frozen SONIC encoder, producing a 64-dimensional tracker latent. Tracking runs at 50 Hz and command publication at 500 Hz; hands bypass the encoder.
The system diagram reports about 0.7 Hz model inference. How that timing is reconciled with one-second chunk playback, including asynchronous scheduling or command buffering, is not specified [12].
V-C Evaluation Metrics
Hardware evaluation uses 20 trials per task. Final success is separate from approach, contact and, for cart pushing, locomotion milestones. Task progress averages the stated milestones, including final success; means weight tasks equally.
Milestone rates are measured over all trials, rather than conditioned on prior milestones. The tables therefore need not form a monotone success funnel. Hardware confidence intervals and variation over independent policy-training seeds are not reported.
Accessible table and original citation links
| Method | Football | DoubleDesk | P&PBox | OpenDoor | SitSofa | Boxing | VisNavi | Mean SR | |
|---|---|---|---|---|---|---|---|---|---|
| Best baseline (per task) | DP | DP | DP | DP | DP | FM | FM 45.525.2% | ACT 47.624.2% | |
| 45.010.8% | 43.36.2% | 75.04.1% | 85.010.8% | 78.314.3% | 76.72.4% | 38.314.3% | 51.224.8% | DP 60.024.2% | |
| WB-WAM | 70.08.2% | 65.04.1% | 86.72.4% | 98.32.4% | 95.04.1% | 81.72.4% | 76.74.7% | 81.912.3% | |
Accessible table and original citation links
| w/ locomotion | w/o locomotion | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Wipe the table | Close the curtain | Make the bed | Move the pillow | Tidy the cloth | Mean SR | Mean TP | ||||||||
| A | C | S | A | C | S | A | C | S | C | S | C | S | |||
| ACT [41] | 0/20 | 0/20 | 0/20 | 0/20 | 0/20 | 0/20 | 0/20 | 0/20 | 0/20 | 16/20 | 8/20 | 19/20 | 12/20 | 20% | 27.5% |
| [42] | 13/20 | 12/20 | 12/20 | 11/20 | 10/20 | 6/20 | 11/20 | 10/20 | 10/20 | 19/20 | 8/20 | 16/20 | 16/20 | 52% | 61.17% |
| GR00T N1.6 [43] | 16/20 | 15/20 | 15/20 | 18/20 | 13/20 | 12/20 | 0/20 | 0/20 | 0/20 | 6/20 | 0/20 | 17/20 | 15/20 | 42% | 48.67% |
| Fast-WAM [9] | 0/20 | 0/20 | 0/20 | 8/20 | 3/20 | 2/20 | 0/20 | 0/20 | 0/20 | 4/20 | 0/20 | 9/20 | 4/20 | 6% | 12.83% |
| DiT4DiT [24] | 12/20 | 6/20 | 3/20 | 16/20 | 15/20 | 7/20 | 16/20 | 9/20 | 9/20 | 5/20 | 0/20 | 16/20 | 12/20 | 31% | 47.5% |
| OpenWAM [26] | 18/20 | 18/20 | 18/20 | 11/20 | 10/20 | 8/20 | 18/20 | 17/20 | 17/20 | 20/20 | 20/20 | 17/20 | 17/20 | 80% | 82% |
| WB-WAM | 16/20 | 16/20 | 15/20 | 16/20 | 15/20 | 13/20 | 19/20 | 18/20 | 18/20 | 20/20 | 20/20 | 18/20 | 18/20 | 84% | 86.67% |
V-D Main Comparison
Simulation uses the HumanoidArena SONIC setting; physical comparisons use a common execution interface. These comparisons evaluate the complete systems rather than an isolated ablation of body-hand supervision or the auxiliary kinematic loss.
V-D 1 Simulation Benchmark Evaluation
WB-WAM reports 81.9% mean success across seven simulation tasks, exceeding the strongest reported baseline for each task. The “best baseline” row selects different methods across tasks; it is not one policy with that combined performance.
Per-task rates and reported deviations remain in Table I, alongside individual baseline model means. This version does not describe simulation rollout counts, seed counts or a statistical significance test, so “significantly” should be read as the reported numerical improvement [40].
V-D 2 Real-World Comparison
Each of five physical tasks uses roughly 100 demonstrations and 20 evaluation trials per method. WB-WAM reaches 84% mean final success versus OpenWAM’s 80%, an improvement of four percentage points or four additional successes in 100 trials.
It is strongest on the overall mean, rather than every task: OpenWAM achieves 90% on wiping versus WB-WAM’s 75%; pillow success ties at 100%. WB-WAM averages 76.7% on the three tasks requiring locomotion, versus 71.7% for OpenWAM. The small study does not establish broad platform or task generality [26].
V-E PICO Mid-training
Using the shared PICO-midtrained checkpoint and 30 robot demonstrations per task yields 73.75% mean success on four tasks. Direct adaptation scores 48.75% with 30 and 65% with 100 robot demonstrations.
The result uses 70% fewer robot demonstrations, supplemented by the full 22-hour human dataset, including about 150 same-task human demonstrations per target. It measures robot-data efficiency rather than a 70% reduction in total supervision or transfer to unseen tasks.
Checkout reaches only 9/20 successes even with human transfer. The reported gains therefore coexist with substantial task-specific failure, and no cost-matched human-versus-robot data collection analysis is provided.
Accessible table and original citation links
| w/ locomotion | w/o locomotion | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Training route | Robot [-1pt]Demos | Close the curtain | Push the cart | Wipe the table | Checkout | Mean SR | Mean TP | |||||||
| A | C | S | C | L | S | A | C | S | C | S | ||||
| Direct post-training | 30 | 13/20 | 12/20 | 12/20 | 13/20 | 11/20 | 11/20 | 12/20 | 11/20 | 11/20 | 6/20 | 5/20 | 48.75% | 51.04% |
| Direct post-training | 100 | 16/20 | 15/20 | 13/20 | 19/20 | 16/20 | 16/20 | 16/20 | 16/20 | 15/20 | 10/20 | 8/20 | 65% | 70.42% |
| w/ PICO mid-training | 30 | 19/20 | 17/20 | 16/20 | 16/20 | 17/20 | 16/20 | 19/20 | 18/20 | 18/20 | 12/20 | 9/20 | 73.75% | 78.13% |
V-F Language-Conditioned Manipulation
A single fruit policy uses about 300 demonstrations across three categories and selects targets through language. Final success is 15/20 for orange, 17/20 for lemon and 16/20 for apple.
This shows conditional selection among the trained fruits. There is no reported unseen fruit category, compositional instruction or general language-understanding evaluation.
Accessible table and original citation links
| Orange | Lemon | Apple | ||||
|---|---|---|---|---|---|---|
| Method | C | S | C | S | C | S |
| WB-WAM | 17/20 | 15/20 | 18/20 | 17/20 | 19/20 | 16/20 |
V-G Visuomotor Generalization
Cart pushing is demonstrated with additional objects changing the cart’s appearance. Figure 7 distinguishes three predicted future frames from the physical execution examples.
The changed scene is absent from the dataset, but evidence here is qualitative: no trial count, success rate or comparison is provided. Optional future-video rollout illustrates the video expert; ordinary action execution can omit that synthesis.
VI Conclusion
The paper demonstrates a workable physical action interface for heterogeneous whole-body learning and task-aligned human-to-robot transfer. Its strongest evidence is the tested G1 task suite.
Limits include one robot platform, mostly separate task policies, small hardware rollout sets and no isolated component ablation. Transfer to unseen tasks and reliable contact execution remain future work.
ACKNOWLEDGMENT
The authors acknowledge BONES-SEED motion data. They disclose AI-generated illustrative elements in Figures 1–4, including the circular graphic, purple human, proprioception illustration and joint-command-discontinuity schematic. These acknowledgments remain attached to the preserved original artwork.
Original project demonstrations
The authors’ project page supplies 26 original clips: both camera views for seven simulation tasks, ten physical task/fruit demonstrations and two five-consecutive-success galleries. All videos and original posters are preserved locally.
Simulation clips play at 2× speed on the project page; consecutive-success clips are encoded at 5×. Their captions retain those distinctions. Selected successful rollouts supplement the quantitative tables rather than define an additional success-rate estimate.