Overview
OpenHLM is an empirical recipe for controlling a humanoid’s arms, torso and legs with one vision-language-action policy. It compares teleoperation interfaces, policy adaptations and cheaper demonstration sources, then releases a working system. The edition retains every figure, table and original task card, including its complete scoring rubric.
At a glance
A joint-based whole-body teleoperation interface, a robot-pretrained π₀.₅ policy adapted to the humanoid action space, and heterogeneous co-training form the proposed recipe. On Unitree G1, the policy reaches 89% average progress across eight training tasks. Stationary same-robot demonstrations raise progress on four held-out tasks from 33% to 87%; robot-free HuMI data transfer new semantics more effectively than new motion. In a longer fruit-arrangement task, HuMI co-training reaches 87.5% progress with about half the recorded demonstration duration of the tested full-teleoperation systems. Scores measure partial progress, not binary success.
1 Introduction
A humanoid’s legs and torso can manipulate objects and extend reach, whereas decoupled navigation and arm control restrict those capabilities. The study asks how to collect demonstrations, adapt a pretrained VLA, and expand its skill set cheaply. Its contribution is a controlled empirical roadmap and open system, not a new foundation-model architecture. SONIC supplies low-level tracking; π₀.₅ supplies robot pretraining; HuMI supplies robot-free manipulation demonstrations.
2 Design Goals and Task Suite
Three design goals are simultaneous control of the full body, language steering with modest task-specific data, and extensibility through cheaper demonstrations. HLM-12 comprises twelve language-conditioned tasks in four families: locomotion plus pick-and-place, workspace extension, feet/body parts as manipulators, and manipulation under environmental constraints. Examples require squatting between shelf levels, pressing a bin pedal, opening a drawer and closing it with a leg, extracting a sword, or pushing a cart. Each task has a detailed partial-credit rubric retained in Appendix A.
3 Building a Whole-Body Loco-Manipulation System: A Roadmap
The roadmap first chooses a controller/teleoperation interface, then adapts the high-level VLA, then tests additional demonstration sources. Most design ablations use the first four tasks; the resulting recipe is evaluated across eight training tasks and four held-out tasks before a separate long-horizon comparison.
3.1 Low-Level Controller & Teleoperation
A high-level policy supplies whole-body targets to a low-level tracking controller and PD actuator control. Joint-based teleoperation retargets PICO-captured human motion through GMR and tracks it with SONIC. With forty demonstrations per task and five rollouts per policy, it is the only tested interface capable of all three comparison tasks: cola placement, shelf transfer and pedal-operated disposal. It averages 88% across the four-task joint-versus-SMPL experiment, versus 75% for SMPL. It is not uniformly best in Table 1: decoupled control scores 93.3% on shelf transfer versus joint control’s 80%. Joint demonstrations use fewer steps and have lower measured roll/pitch vibration, while hardware-failure reduction is qualitative. A 0.2-second future-reference preview is selected from a separate single-task sweep. The main text’s typical 10 Hz/50 Hz hierarchy is contextual; actual inference timing is specified separately in Appendix C.4.
| Cola Placement | Shelf Cup Transfer | Bottle Disposal | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Teleop method | Prog. (%) | Time (s) | Steps | Prog. (%) | Time (s) | Steps | Prog. (%) | Time (s) | Steps |
| Decoupled control | 36.7 | 42.3 | 33.6 | 38.4 | ✗ | ||||
| VR 3-point teleoperation | 67.8 | 12.3 | ✗ | ✗ | |||||
| Joint-based whole-body teleop. | 40.2 | 12.0 | 41.8 | 10.5 | 29.7 | 10.3 | |||
| Task | Joint-based | VR three-point | Decoupled |
|---|---|---|---|
| Cola Placement | 0.187 | 0.235 | 0.439 |
| Shelf Cup Transfer | 0.178 | – | 0.424 |
3.2 Whole-Body VLA Policy Design
Adapt π₀.₅’s 32-channel projections to 34 channels by preserving existing weights and initializing the additions. Retain its bimanual action ordering, append legs/waist/root channels, predict absolute joint targets, include proprioception, and keep ten-step flow-matching generation. Changing one interface choice at a time causes small, potentially noisy differences; combining relative targets with no proprioception instead leads to catastrophic drift. Robot-pretrained π₀.₅ reaches about 91% progress, versus 60% with a PaliGemma-only visual-language initialization and 42% from scratch. Similar action MSE does not imply similar on-robot behavior. One-step flow matching and a separately trained drifting expert each lose roughly twenty progress points despite reducing generation latency from about 90 to 60 ms on a 5080 GPU and even lowering MSE. Forty demonstrations per task is a practical default, not a universal sample-complexity law.
3.3 Heterogeneous Co-Training
Co-train the eight-task whole-body set with forty manipulation-only demonstrations for each of four held-out tasks. Stationary same-robot teleoperation raises held-out progress from 33% to 87%, near the all-twelve-task oracle’s 94%, while maintaining training-task performance. It supplies both new object/instruction grounding and the new vessel-tilt motion needed for pouring. HuMI instead uses two handheld grippers plus pelvis/feet trackers and offline inverse kinematics. It matches stationary co-training on tasks 9–11, which reuse familiar motions, but fails to learn pouring at this data scale. The authors suggest camera, gripper and motion-retargeting gaps as explanations; these are not isolated causal tests. Recorded duration for forty demonstrations averages 21 minutes for full teleoperation, 13 for stationary teleoperation and 7 for HuMI, excluding reset and cooling overhead.
4 Long-Horizon System Comparison
Fruit arrangement requires two ordered fruit selections from low and medium tables, then placement on a high shelf. Five fruits give twenty ordered pairs, with six demonstrations each. GR00T N1.6 and Ψ₀ receive full decoupled-control demonstrations for all pairs. OpenHLM receives 36 whole-body demonstrations on six pairs using three fruits, plus 84 HuMI demonstrations for pairs containing lemon or tomato; its oracle receives 120 whole-body demonstrations. On ten evaluated pairs, progress is 48.8±4.4% for Ψ₀, 57.5±4.6% for GR00T, 87.5±3.7% for HuMI co-training and 97.5±1.7% for the oracle. Recorded demonstration times are 2.70, 2.70, 1.14 and 2.73 hours. This is a system comparison with different data/control pipelines; notably Ψ₀ is adapted to the GR00T controller, not evaluated with its original end-to-end stack.
5 Related Work
Whole-body tracking and teleoperation prior work establishes feasibility, while this study compares interfaces specifically as a source of language-policy training data. Existing VLAs contribute pretrained visual, language and action representations; the experiments show transfer from nonhumanoid robot data, not that humanoid pretraining is inherently inferior. Cheap human demonstrations connect to UMI, motion priors and egocentric video learning. Concurrent EgoHumanoid emphasizes paired human/robot demonstrations and environment generalization; OpenHLM focuses on new tasks outside the whole-body teleoperation set.
6 Discussion, Conclusion and Limitations
The tested recipe combines whole-body joint teleoperation, robot-pretrained policy adaptation and targeted cheap-data co-training. Evidence is restricted to Unitree G1, limited scenes and a limited exploration of model designs. HuMI does not close the new-motion gap at the tested scale. The mismatch between offline MSE and physical behavior, and the degradation from one-step generation, remain unexplained. Broader embodiments, architectures, visual alignment and larger robot-free datasets are future work; the paper does not establish robust general-purpose humanoid autonomy.
Core Contributors
The five core contributors share overlapping responsibility. Yingdong Hu led writing and high-level policy design; Haodong Zhu led teleoperation/evaluation software; Boyuan Zheng led co-training collection and baseline implementation; Yihang Hu led hardware, task scenes and collection; Tong Zhang led low-level control and testing. Each also contributed to adjacent software, data or integration work; the source explicitly cautions against treating these roles as a ranking.
Acknowledgments
The authors acknowledge Shanghai Qi Zhi Institute and Spirit AI, Tsinghua’s Keystone/JCEMAI program, Xiongan AI Institute, and Beijing Natural Science Foundation support JR25019.
Appendices
The appendices preserve task instructions and rubrics, hardware details, training and baseline adaptations, and supplementary scaling results.
Appendix A The HLM-12 Benchmark
HLM-12 task cards specify prompts, physical procedures, scored subtasks and capability families. The evaluation uses consistent initial scenes and reports normalized progress with standard errors.
A.1 The 12 Tasks
Tasks 1–4 are cola placement (three points), shelf cup transfer (three), pedal-operated bottle disposal (four), and bimanual jar opening (three). Tasks 5–8 are toy stowing with drawer opening and leg closure (five), sword extraction (four), cart loading/pushing (four), and shuttlecock extraction onto a racket (three). Held-out tasks 9–11 substitute a toy pig, gum can and shelf cube into familiar placement/transfer motions (three points each); task 12 pours dice into a plate and sets down the empty cup (four). Rubrics usually combine successful approach/return into one point and separately credit correct grasp and object placement. All twelve original cards, photographs, complete prompts and scoring details remain below; they are not replaced by these summaries.
A.2 Overall Evaluation Protocol
Each policy/task pair receives five physical rollouts, with target locations and distractors varied. The same five photographed initial scenes are reused across policies. Progress is earned rubric points divided by total possible points; for example, completing cola placement except for landing on the pad earns two of three points. Means and standard errors therefore quantify partial completion, not five independent binary successes. Full humanoid scene resets and repeated navigation make trials costly.
Appendix B Hardware Setup
The platform combines a G1 humanoid, wrist grippers, head/wrist cameras, PICO whole-body teleoperation, and a separate HuMI human demonstration rig.
B.1 Humanoid Robot
Unitree G1 carries two ChangingTek CTAG2F90-D parallel grippers, a head-mounted SV1-25 stereo fisheye camera and two wrist-mounted RealSense D405 cameras. Cables are secured along the back to reduce interference with whole-body motion.
B.2 Whole-Body Teleoperation Hardware
PICO 4 Ultra supplies one head-mounted display, two hand controllers and two leg trackers. Their relative poses reconstruct the operator’s motion for retargeting. Live head and wrist views provide visual feedback, while hand triggers control gripper opening.
B.3 Teleoperation-Free Data Collection Hardware
HuMI uses two GoPro-equipped UMI-style handheld grippers and five VIVE Ultimate trackers: one on each gripper, one at the pelvis and one at each foot. Gripper shape/opening and camera placement approximate the robot but do not match it exactly. A laptop records synchronized wrist images, gripper widths and tracked poses; an online IK preview helps the operator keep demonstrations feasible for the humanoid.
Appendix C Implementation Details
Implementation details define the action representations, initialization/ordering choices, co-training alignment and exact training/inference configuration.
C.1 Low-Level Controller & Teleoperation
All four teleoperation variants use the same visual inputs, while their recorded body command spaces differ. The robot has 29 joints; a whole-body target adds root roll, pitch and yaw rate for 32 body channels. Two independently recorded gripper scalars bring the high-level joint-based action and proprioceptive vectors to 34 dimensions.
Decoupled control teleoperation.
Decoupled control specifies three waist angles, seven joints per arm, one base-height command and three planar navigation velocities: 21 body channels, plus the two grippers. A learned locomotion controller handles balance and legs. The operator cannot directly place an individual foot, which prevents pedal manipulation even though base-height control can support some shelf reaching.
VR 3-point teleoperation.
The sparse VR interface combines three planar navigation commands with positions and quaternions for head and two wrists: 3+9+12=24 body channels, plus grippers. A motion planner infers the lower-body motion. This is expressive for arms but gives only indirect control of torso, knees and feet.
Joint-based whole-body teleoperation.
PICO whole-body motion is retargeted online through GMR into the robot’s joints, then tracked by SONIC. The 32 body targets comprise root roll/pitch/yaw-rate, fourteen arm joints, twelve leg joints and three waist joints. This supports direct foot use, squatting and coordinated locomotion/manipulation.
SMPL-based whole-body teleoperation.
SMPL records a human representation before robot retargeting: three root terms, a six-dimensional wrist/arm refinement and 72 human joint coordinates, or 81 body channels plus grippers. Its greater redundancy is a proposed explanation for inferior policy learning, not an independently isolated cause. Errors include lifting a heel without clearing the toes and passing a can during inaccurate approach.
C.2 Whole-Body VLA Policy Design
The VLA experiments change its action/proprioception interface, pretrained initialization and generation procedure. Most main ablations alter one choice from the default at a time.
Action and proprioception interface.
Projection surgery preserves pretrained entries and Xavier-initializes the added 32→34 dimensions. Default ordering is left arm/gripper, right arm/gripper, legs, waist, then root; humanoid-native ordering moves root and legs first. Absolute versus relative targets concerns the 29 actuated joints; the appendix defines the relative target as action minus the chunk’s initial state. Proprioception has the same 34-channel layout. Combining absent proprioception with relative commands can cause drift, even though the separate ablations show only small differences.
Pretraining ablations.
The full π₀.₅ initialization includes robot-pretrained action components and a PaliGemma visual-language backbone. The PaliGemma-only ablation retains the backbone but randomly initializes action-side layers; the scratch baseline initializes everything randomly. π₀.₅’s pretraining data here contain static or wheeled dual-arm robots, with no humanoid data. Its superior physical recovery behavior is not captured by the nearly indistinguishable validation MSE of the first two variants.
One-step action generation.
One-step flow matching changes inference integration from ten steps to one without retraining. The drifting alternative retrains a replacement action expert with a drifting objective while preserving the backbone. Both target faster generation but perform substantially worse on the four-task robot evaluation; the explanation involving temporal jitter/smoothness remains a hypothesis.
C.3 Heterogeneous Co-Training
Cheap demonstrations retain manipulation segments while omitting approach and departure walking. Their usefulness differs between same-embodiment stationary teleoperation and retargeted human data.
Stationary teleoperation demonstrations.
Stationary teleoperation uses the same robot and joint interface after it has already reached the workspace. It includes arms, grippers, torso height and in-place turns, then stops before departure. Thus it preserves relevant whole-body manipulation while saving locomotion, reset and scene-collection effort.
Stationary HuMI demonstrations.
HuMI records corresponding manipulation phases without the robot, then applies offline IK to produce the same state/action format. GoPro fisheye rectification reduces but does not eliminate camera mismatch. To align with the teleoperation tracker’s 0.2-second future-reference preview, HuMI actions are shifted forward by that interval. Retargeting and gripper differences remain plausible obstacles to new-motion transfer.
C.4 Hyperparameters
Training uses AdamW (β₁=0.9, β₂=0.95), peak learning rate 10⁻⁴, cosine decay, 1,000 warmup steps, 30,000 total steps, batch 128, horizon 50 and 224×224 observations. Four A800 GPUs take about 24 hours. Head images receive crop/rotation augmentation; all images receive brightness/contrast/saturation changes. At inference, generate fifty actions at 30 Hz, execute twenty-five, and replan every 5/6 second. This concrete configuration should not be confused with the typical hierarchy frequencies introduced earlier.
Appendix D System-Level Comparison on Long-Horizon Tasks
The longer experiment compares complete systems on language-directed fruit arrangement across surfaces of different heights. Its original task card and evaluation-pair table remain intact.
D.1 Task Details & Evaluation Protocol
The robot starts away from a 40 cm coffee table, takes the first named fruit with its right hand, takes the second from a 70 cm table with its left, places them into separate containers at 95 cm, and returns. Eight points separately credit navigation, each correct pick, shelf alignment, each placement and return. Twenty ordered pairs of five fruits yield 120 demonstrations per condition. Ten selected pairs are evaluated once each, with every fruit appearing twice in each role and photographed initial scenes shared across methods; these ten progress scores underlie the reported mean and standard error.
D.2 Baselines & Data Collection
GR00T N1.6 uses a Cosmos-2B VLM and diffusion-transformer action expert. The adaptation replaces its hand controls with two grippers, adds wrist cameras and retains decoupled arm/waist/height/navigation commands; it trains 50,000 steps with batch 32 and fifty-action chunks. Ψ₀ uses a Qwen3-VL backbone and flow-based MMDiT expert. To reuse the same demonstrations/controller, its 36-dimensional layout is retained with unused hand channels masked and state channels zero-padded, and its extra target-yaw term is unused. It trains 50,000 steps, batch 64, with thirty-action chunks and RTC. The authors expect limited degradation from this adaptation but do not validate the original Ψ₀ end-to-end controller in this comparison.
Appendix E Additional Experimental Results
Additional experiments separate preview latency, whole-body demonstration count, per-task co-training behavior and HuMI demonstration count.
E.1 Future-frame Preview Latency Analysis
On the single cola task, future previews of 0, 0.2, 0.4 and 0.6 seconds give about 60%, 67%, 60% and 13% progress. Zero preview feels responsive but produces vibration/stomping; 0.6 seconds creates difficult lag. At 0.2 seconds, average recorded demonstration duration is 35.2 seconds, close to zero preview’s 34.3. The main multitask policy’s 86.7% cola score is a different experiment: mixed related tasks provide additional learning signal. A different tracker could change the optimal preview.
E.2 Scaling of Teleoperation Demonstrations
Across the four-task subset, five, ten, twenty and forty demonstrations per task yield roughly 52%, 60%, 85% and 91% progress. The largest improvement is ten to twenty. Forty demonstrations are retained as the working default, estimated at about 1.5 hours of skilled-operator collection per medium task; that collection-effort estimate includes more than the recorded trajectory durations used in other tables.
E.3 Per-Task Results on the HLM-12 Benchmark
The complete twelve-task plot shows no clear training-task regression from co-training. Both cheap-data sources nearly reach the oracle on motion-reuse tasks 9–11. Only stationary same-robot demonstrations succeed on task 12’s new pouring motion, explaining the aggregate difference. The plot preserves individual tasks and all three aggregates rather than hiding them behind one average.
E.4 Scaling HuMI Demonstrations
With the eight-task whole-body set fixed, five, ten, twenty and forty HuMI demonstrations per held-out motion-reuse task give about 42%, 67%, 76% and 84% average progress. Gains diminish at higher counts. This sweep concerns tasks 9–11, not pouring; it supports economical semantic grounding but does not demonstrate new-motion transfer.