Literature
Display
← All papers

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

Yingdong Hu · Haodong Zhu · Boyuan Zheng · Yihang Hu · Tong Zhang · Zunhao Chen · Junming Zhao · Ruiqian Nai · Yang Gao

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

OpenHLM is an empirical recipe for controlling a humanoid’s arms, torso and legs with one vision-language-action policy. It compares teleoperation interfaces, policy adaptations and cheaper demonstration sources, then releases a working system. The edition retains every figure, table and original task card, including its complete scoring rubric.

Original PDF visual retaining every panel, native raster resolution and vector text.
Figure 1: Overview of OpenHLM. A roadmap of controlled experiments in three phases: (I) we compare teleop interfaces for a low-level whole-body controller and adopt a joint-based interface; (II) we adapt a manipulation VLA to the humanoid’s full action space along several design axes; (III) we extend the policy to new objects and instructions by co-training the full loco-manipulation data with stationary teleop or HuMI [37] demonstrations.

At a glance

A joint-based whole-body teleoperation interface, a robot-pretrained π₀.₅ policy adapted to the humanoid action space, and heterogeneous co-training form the proposed recipe. On Unitree G1, the policy reaches 89% average progress across eight training tasks. Stationary same-robot demonstrations raise progress on four held-out tasks from 33% to 87%; robot-free HuMI data transfer new semantics more effectively than new motion. In a longer fruit-arrangement task, HuMI co-training reaches 87.5% progress with about half the recorded demonstration duration of the tested full-teleoperation systems. Scores measure partial progress, not binary success.

1 Introduction

A humanoid’s legs and torso can manipulate objects and extend reach, whereas decoupled navigation and arm control restrict those capabilities. The study asks how to collect demonstrations, adapt a pretrained VLA, and expand its skill set cheaply. Its contribution is a controlled empirical roadmap and open system, not a new foundation-model architecture. SONIC supplies low-level tracking; π₀.₅ supplies robot pretraining; HuMI supplies robot-free manipulation demonstrations.

2 Design Goals and Task Suite

Three design goals are simultaneous control of the full body, language steering with modest task-specific data, and extensibility through cheaper demonstrations. HLM-12 comprises twelve language-conditioned tasks in four families: locomotion plus pick-and-place, workspace extension, feet/body parts as manipulators, and manipulation under environmental constraints. Examples require squatting between shelf levels, pressing a bin pedal, opening a drawer and closing it with a leg, extracting a sword, or pushing a cart. Each task has a detailed partial-credit rubric retained in Appendix A.

Original PDF visual retaining every panel, native raster resolution and vector text.
Figure 2: The HLM-12 Benchmark. Tasks fall into four capability families targeting different aspects of whole-body behavior, with one representative per family shown. Full task specifications are in Appendix A.

3 Building a Whole-Body Loco-Manipulation System: A Roadmap

The roadmap first chooses a controller/teleoperation interface, then adapts the high-level VLA, then tests additional demonstration sources. Most design ablations use the first four tasks; the resulting recipe is evaluated across eight training tasks and four held-out tasks before a separate long-horizon comparison.

3.1 Low-Level Controller & Teleoperation

A high-level policy supplies whole-body targets to a low-level tracking controller and PD actuator control. Joint-based teleoperation retargets PICO-captured human motion through GMR and tracks it with SONIC. With forty demonstrations per task and five rollouts per policy, it is the only tested interface capable of all three comparison tasks: cola placement, shelf transfer and pedal-operated disposal. It averages 88% across the four-task joint-versus-SMPL experiment, versus 75% for SMPL. It is not uniformly best in Table 1: decoupled control scores 93.3% on shelf transfer versus joint control’s 80%. Joint demonstrations use fewer steps and have lower measured roll/pitch vibration, while hardware-failure reduction is qualitative. A 0.2-second future-reference preview is selected from a separate single-task sweep. The main text’s typical 10 Hz/50 Hz hierarchy is contextual; actual inference timing is specified separately in Appendix C.4.

Cola Placement Shelf Cup Transfer Bottle Disposal
Teleop method Prog. (%) Time (s) Steps Prog. (%) Time (s) Steps Prog. (%) Time (s) Steps
Decoupled control 66.7± 14.966.7{\scriptstyle\,\pm\,14.9} 36.7 42.3 93.3± 6.793.3{\scriptstyle\,\pm\,6.7} 33.6 38.4 ✗
VR 3-point teleoperation 40.0± 6.740.0{\scriptstyle\,\pm\,6.7} 67.8 12.3 ✗ ✗
Joint-based whole-body teleop. 86.7± 8.286.7{\scriptstyle\,\pm\,8.2} 40.2 12.0 80.0± 13.380.0{\scriptstyle\,\pm\,13.3} 41.8 10.5 85.0± 6.185.0{\scriptstyle\,\pm\,6.1} 29.7 10.3
Table 1: Teleop method comparison. Prog. (%), mean task progress over 5 rollouts; Time (s), mean rollout duration; Steps, mean footsteps per rollout. A purple ✗ marks a task the method cannot perform by construction: Shelf Cup Transfer requires squatting; Bottle Disposal requires the foot to depress a pedal.
Task Joint-based VR three-point Decoupled
Cola Placement 0.187 0.235 0.439
Shelf Cup Transfer 0.178 – 0.424
Table 2: Average IMU vibration amplitudes (A=sqrt​(avg.​(ϕ˙2+θ˙2))A=\text{sqrt}(\text{avg.}(\dot{\phi}^{2}+\dot{\theta}^{2}))) in the collected demonstration dataset, evaluated above three methods across two tasks.
Original figure
Figure 3: Joint-based vs. SMPL whole-body teleop.

3.2 Whole-Body VLA Policy Design

Adapt π₀.₅’s 32-channel projections to 34 channels by preserving existing weights and initializing the additions. Retain its bimanual action ordering, append legs/waist/root channels, predict absolute joint targets, include proprioception, and keep ten-step flow-matching generation. Changing one interface choice at a time causes small, potentially noisy differences; combining relative targets with no proprioception instead leads to catastrophic drift. Robot-pretrained π₀.₅ reaches about 91% progress, versus 60% with a PaliGemma-only visual-language initialization and 42% from scratch. Similar action MSE does not imply similar on-robot behavior. One-step flow matching and a separately trained drifting expert each lose roughly twenty progress points despite reducing generation latency from about 90 to 60 ms on a 5080 GPU and even lowering MSE. Forty demonstrations per task is a practical default, not a universal sample-complexity law.

Original figure
Figure 4: VLA design ablations on the 4-task subset. Amber: interface ablations (one choice flipped per bar); performance gaps are minor. Rose: pretraining ablations; robot pretraining (π0.5\pi_{0.5}) dominates. Sage: one-step action generation; both underperform the 10-step baseline.

3.3 Heterogeneous Co-Training

Co-train the eight-task whole-body set with forty manipulation-only demonstrations for each of four held-out tasks. Stationary same-robot teleoperation raises held-out progress from 33% to 87%, near the all-twelve-task oracle’s 94%, while maintaining training-task performance. It supplies both new object/instruction grounding and the new vessel-tilt motion needed for pouring. HuMI instead uses two handheld grippers plus pelvis/feet trackers and offline inverse kinematics. It matches stationary co-training on tasks 9–11, which reuse familiar motions, but fails to learn pouring at this data scale. The authors suggest camera, gripper and motion-retargeting gaps as explanations; these are not isolated causal tests. Recorded duration for forty demonstrations averages 21 minutes for full teleoperation, 13 for stationary teleoperation and 7 for HuMI, excluding reset and cooling overhead.

Original figure
Figure 5: Heterogeneous co-training results. Progress on the 4 held-out tasks and averages over training tasks (1–8), motion-reuse held-out tasks (9–11), and all held-out tasks (9–12). Full breakdown in Appendix E.3.

4 Long-Horizon System Comparison

Fruit arrangement requires two ordered fruit selections from low and medium tables, then placement on a high shelf. Five fruits give twenty ordered pairs, with six demonstrations each. GR00T N1.6 and Ψ₀ receive full decoupled-control demonstrations for all pairs. OpenHLM receives 36 whole-body demonstrations on six pairs using three fruits, plus 84 HuMI demonstrations for pairs containing lemon or tomato; its oracle receives 120 whole-body demonstrations. On ten evaluated pairs, progress is 48.8±4.4% for Ψ₀, 57.5±4.6% for GR00T, 87.5±3.7% for HuMI co-training and 97.5±1.7% for the oracle. Recorded demonstration times are 2.70, 2.70, 1.14 and 2.73 hours. This is a system comparison with different data/control pipelines; notably Ψ₀ is adapted to the GR00T controller, not evaluated with its original end-to-end stack.

Refer to caption
Figure 6: Long-horizon task.
Method (long-horizon task) Progress (%) Demo Dur.
Ψ0\Psi_{0} 48.8± 4.448.8{\scriptstyle\,\pm\,4.4} 2.702.70 h
GR00T N1.6 57.5± 4.657.5{\scriptstyle\,\pm\,4.6} 2.702.70 h
OpenHLM  (HuMI co-training) 87.5± 3.787.5{\scriptstyle\,\pm\,3.7} 1.141.14 h
OpenHLM  (teleop oracle) 97.5± 1.797.5{\scriptstyle\,\pm\,1.7} 2.732.73 h
Table 3: Task progress & demo duration.

5 Related Work

Whole-body tracking and teleoperation prior work establishes feasibility, while this study compares interfaces specifically as a source of language-policy training data. Existing VLAs contribute pretrained visual, language and action representations; the experiments show transfer from nonhumanoid robot data, not that humanoid pretraining is inherently inferior. Cheap human demonstrations connect to UMI, motion priors and egocentric video learning. Concurrent EgoHumanoid emphasizes paired human/robot demonstrations and environment generalization; OpenHLM focuses on new tasks outside the whole-body teleoperation set.

6 Discussion, Conclusion and Limitations

The tested recipe combines whole-body joint teleoperation, robot-pretrained policy adaptation and targeted cheap-data co-training. Evidence is restricted to Unitree G1, limited scenes and a limited exploration of model designs. HuMI does not close the new-motion gap at the tested scale. The mismatch between offline MSE and physical behavior, and the degradation from one-step generation, remain unexplained. Broader embodiments, architectures, visual alignment and larger robot-free datasets are future work; the paper does not establish robust general-purpose humanoid autonomy.

Core Contributors

The five core contributors share overlapping responsibility. Yingdong Hu led writing and high-level policy design; Haodong Zhu led teleoperation/evaluation software; Boyuan Zheng led co-training collection and baseline implementation; Yihang Hu led hardware, task scenes and collection; Tong Zhang led low-level control and testing. Each also contributed to adjacent software, data or integration work; the source explicitly cautions against treating these roles as a ranking.

Acknowledgments

The authors acknowledge Shanghai Qi Zhi Institute and Spirit AI, Tsinghua’s Keystone/JCEMAI program, Xiongan AI Institute, and Beijing Natural Science Foundation support JR25019.

Appendices

The appendices preserve task instructions and rubrics, hardware details, training and baseline adaptations, and supplementary scaling results.

Appendix A The HLM-12 Benchmark

HLM-12 task cards specify prompts, physical procedures, scored subtasks and capability families. The evaluation uses consistent initial scenes and reports normalized progress with standard errors.

A.1 The 12 Tasks

Tasks 1–4 are cola placement (three points), shelf cup transfer (three), pedal-operated bottle disposal (four), and bimanual jar opening (three). Tasks 5–8 are toy stowing with drawer opening and leg closure (five), sword extraction (four), cart loading/pushing (four), and shuttlecock extraction onto a racket (three). Held-out tasks 9–11 substitute a toy pig, gum can and shelf cube into familiar placement/transfer motions (three points each); task 12 pours dice into a plate and sets down the empty cup (four). Rubrics usually combine successful approach/return into one point and separately credit correct grasp and object placement. All twelve original cards, photographs, complete prompts and scoring details remain below; they are not replaced by these summaries.

Task 1: Cola Placement • Language Prompt: Walk forward to the table, put the red can on the orange mouse pad with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white table, locate the cola can placed randomly among other distractor objects, and carefully grasp it without pushing it away or causing it to fall. It then turns about 90 degrees to the left using its feet and waist to face the mouse pad, and places the can onto it. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot picks up the correct object (the cola can). (1 pt) – The cola can is placed exactly on the target location (the orange mouse pad). (1 pt) • Capability Stressed: Pick-and-place with locomotion .
Task 1: Cola Placement
Task 2: Shelf Cup Transfer • Language Prompt: Walk forward to the shelf, put the white cup from the upper level to the lower level with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the shelf, locate and grasp the white cup placed randomly among other distractor objects on the upper level, then bend down and carefully place it onto the lower level of the shelf. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the shelf and later returns to its initial location. (1 pt) – The robot picks up the correct object (the white cup). (1 pt) – The white cup is successfully placed on the lower level of the shelf. (1 pt) • Capability Stressed: Whole-body workspace extension .
Task 2: Shelf Cup Transfer
Task 3: Bottle Disposal • Language Prompt: Walk forward to the dust bin, throw the green bottle into the dust bin, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the dust bin, locate and grasp the green bottle among other distractor objects on the table to the right, then press the pedal with the left foot to open the dust bin lid, and place the bottle completely inside the bin. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 4 points in total: – The robot successfully walks to the correct position in front of the dust bin and later returns to its initial location. (1 pt) – The robot picks up the correct object (the green bottle). (1 pt) – The robot successfully presses the pedal to keep the lid open. (1 pt) – The bottle is successfully placed inside the bin. (1 pt) • Capability Stressed: Using body parts as manipulators .
Task 3: Bottle Disposal
Task 4: Jar Opening • Language Prompt: Walk forward to the white table, open the red jar with both hands, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white table, locate the red jar placed randomly among other distractor objects, grasp the upper lid with the right hand and the lower body with the left hand, then separate the two parts by pulling them apart with both hands. After opening the jar, the robot places both parts back onto the table. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot uses the right hand to grasp the correct part of the correct object (the upper lid of the red jar). (1 pt) – The robot uses the left hand to grasp the correct part (the lower body of the red jar), and the jar is successfully opened. (1 pt) • Capability Stressed: Loco-manipulation under environmental constraint .
Task 4: Jar Opening
Task 5: Toy Stowing • Language Prompt: Walk forward to the drawer, pick up the blue toy bear with the left hand, open the drawer with the right hand and drop the blue toy bear inside, close the drawer with the right leg, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the drawer, locate and grasp the blue toy bear among other distractor objects with the left gripper, then close the right gripper, insert it into the small gap of the drawer, and pull the drawer open. After placing the toy bear inside the drawer, the robot uses the right leg to gently push the drawer closed. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 5 points in total: – The robot successfully walks to the correct position in front of the drawer and later returns to its initial location. (1 pt) – The robot picks up the correct object (the blue toy bear). (1 pt) – The robot successfully inserts the right gripper into the gap and pulls the drawer open. (1 pt) – The toy bear is placed exactly inside the drawer. (1 pt) – The drawer is gently pushed closed using the right leg. (1 pt) • Capability Stressed: Using body parts as manipulators .
Task 5: Toy Stowing
Task 6: Sword Extraction • Language Prompt: Walk forward to the white drawer, pull out the sword with both hands, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white cabinet, locate the sword on the rack, grasp the hilt with the right hand and lift the sword, then grasp the scabbard with the left hand and pull the hilt and scabbard apart with both hands to extract the sword. Afterward, the robot turns around and walks back to its initial location while holding the sword. • Evaluation Rubric: 4 points in total: – The robot successfully walks to the correct position in front of the cabinet and later returns to its initial location. (1 pt) – The robot uses the right hand to grasp the correct part of the correct object (the hilt of the sword). (1 pt) – The robot uses the left hand to grasp the correct part (the scabbard of the sword). (1 pt) – The robot successfully pulls the sword out of the scabbard. (1 pt) • Capability Stressed: Loco-manipulation under environmental constraint .
Task 6: Sword Extraction
Task 7: Cart Pushing • Language Prompt: Walk forward to the cart, throw the box onto the cart, then push the cart away. • Detailed Process: The robot needs to walk to an appropriate position in front of the cart, locate and grasp the box on the table to the right, and drop it into the large box on the cart. It then grasps the left and right sides of the cart handle sequentially with the corresponding hands. Afterward, the robot pushes the cart forward to carry the contents away. • Evaluation Rubric: 4 points in total: – The robot successfully walks to the correct position in front of the cart and then successfully pushes the cart forward. (1 pt) – The robot uses the right hand to grasp the correct object (the yellow iron box). (1 pt) – The iron box is placed exactly into the large box on the cart. (1 pt) – The robot successfully grasps both sides of the cart handle. (1 pt) • Capability Stressed: Loco-manipulation under environmental constraint .
Task 7: Cart Pushing
Task 8: Shuttlecock Setup • Language Prompt: Walk forward to the black table, pull out one shuttle and put it on the racket with the left hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the black table, locate the upside-down shuttlecock dispenser placed on the tabletop, grasp the bottom shuttlecock with the left gripper and pull it downward to remove it, then place it onto the racket face on the left side. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot uses the left hand to locate the shuttlecock dispenser and pull out one shuttlecock successfully. (1 pt) – The robot places the shuttlecock exactly onto the racket face. (1 pt) • Capability Stressed: Loco-manipulation under environmental constraint .
Task 8: Shuttlecock Setup
Task 9: Pig Placement • Language Prompt: Walk forward to the table, put the yellow toy pig on the blue mouse pad with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white table, locate the yellow toy pig placed randomly among other distractor objects, and carefully grasp it. It then turns about 90 degrees to the left using its feet and waist to face the mouse pad, and places the toy pig onto it. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot picks up the correct object (the yellow toy pig). (1 pt) – The toy pig is placed exactly on the target location (the blue mouse pad). (1 pt) • Capability Stressed: Pick-and-place with locomotion .
Task 9: Pig Placement
Task 10: Gum Can Placement • Language Prompt: Walk forward to the table, put the pink gum can on the maroon mouse pad with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white table, locate the pink gum can placed randomly among other distractor objects, and carefully grasp it. It then turns about 90 degrees to the left using its feet and waist to face the mouse pad, and places the gum can onto it. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot picks up the correct object (the pink gum can). (1 pt) – The gum can is placed exactly on the target location (the maroon mouse pad). (1 pt) • Capability Stressed: Pick-and-place with locomotion .
Task 10: Gum Can Placement
Task 11: Shelf Cube Transfer • Language Prompt: Walk forward to the shelf, put the cube from the upper level to the lower level with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the shelf, locate and grasp the cube placed randomly among other distractor objects on the upper level, then bend down and carefully place it onto the lower level of the shelf. Finally, the robot turns around and walks back to its initial location. • Evaluation Rubric: 3 points in total: – The robot successfully walks to the correct position in front of the shelf and later returns to its initial location. (1 pt) – The robot picks up the correct object (the cube). (1 pt) – The cube is successfully placed on the lower level of the shelf. (1 pt) • Capability Stressed: Whole-body workspace extension .
Task 11: Shelf Cube Transfer
Task 12: Pouring • Language Prompt: Walk forward to the white table, pour the contents from the blue cup into the white plate with the right hand, turn around and walk back. • Detailed Process: The robot needs to walk to an appropriate position in front of the white table, locate the blue cup placed randomly among other distractor objects, grasp it with the right hand, and pour the dice inside into the white plate placed on the towel at the center of the table. The robot then places the empty cup onto the left side of the table while turning around to the left, and finally walks back to its initial location. • Evaluation Rubric: 4 points in total: – The robot successfully walks to the correct position in front of the table and later returns to its initial location. (1 pt) – The robot uses the right hand to grasp the correct object (the blue cup). (1 pt) – The robot pours the dices successfully into the white plate. (1 pt) – The robot places the empty cup on the left side of the table. (1 pt) • Capability Stressed: Loco-manipulation under environmental constraint .
Task 12: Pouring

A.2 Overall Evaluation Protocol

Each policy/task pair receives five physical rollouts, with target locations and distractors varied. The same five photographed initial scenes are reused across policies. Progress is earned rubric points divided by total possible points; for example, completing cola placement except for landing on the pad earns two of three points. Means and standard errors therefore quantify partial completion, not five independent binary successes. Full humanoid scene resets and repeated navigation make trials costly.

Appendix B Hardware Setup

The platform combines a G1 humanoid, wrist grippers, head/wrist cameras, PICO whole-body teleoperation, and a separate HuMI human demonstration rig.

B.1 Humanoid Robot

Unitree G1 carries two ChangingTek CTAG2F90-D parallel grippers, a head-mounted SV1-25 stereo fisheye camera and two wrist-mounted RealSense D405 cameras. Cables are secured along the back to reduce interference with whole-body motion.

Original PDF visual retaining every panel, native raster resolution and vector text.
Figure 7: Humanoid robot hardware. The Unitree G1 is equipped with wrist-mounted grippers and onboard cameras.
Original PDF visual retaining every panel, native raster resolution and vector text.
Figure 8: HuMI hardware. Handheld grippers and body trackers are used to collect teleoperation-free demonstrations.

B.2 Whole-Body Teleoperation Hardware

PICO 4 Ultra supplies one head-mounted display, two hand controllers and two leg trackers. Their relative poses reconstruct the operator’s motion for retargeting. Live head and wrist views provide visual feedback, while hand triggers control gripper opening.

B.3 Teleoperation-Free Data Collection Hardware

HuMI uses two GoPro-equipped UMI-style handheld grippers and five VIVE Ultimate trackers: one on each gripper, one at the pelvis and one at each foot. Gripper shape/opening and camera placement approximate the robot but do not match it exactly. A laptop records synchronized wrist images, gripper widths and tracked poses; an online IK preview helps the operator keep demonstrations feasible for the humanoid.

Original PDF visual retaining every panel, native raster resolution and vector text.
Figure 9: Whole-body teleoperation scene and HMD snapshot, of the same frame. Left: the PICO4U kit provides the HMD, two handheld controllers, and two leg trackers for live teleoperation. Right: an in-headset egocentric camera view streamed to the operator during teleoperation. The fisheye head view is placed in the center, with the 2 wrist views placed on its top-left and top-right corners.

Appendix C Implementation Details

Implementation details define the action representations, initialization/ordering choices, co-training alignment and exact training/inference configuration.

C.1 Low-Level Controller & Teleoperation

All four teleoperation variants use the same visual inputs, while their recorded body command spaces differ. The robot has 29 joints; a whole-body target adds root roll, pitch and yaw rate for 32 body channels. Two independently recorded gripper scalars bring the high-level joint-based action and proprioceptive vectors to 34 dimensions.

Decoupled control teleoperation.

Decoupled control specifies three waist angles, seven joints per arm, one base-height command and three planar navigation velocities: 21 body channels, plus the two grippers. A learned locomotion controller handles balance and legs. The operator cannot directly place an individual foot, which prevents pedal manipulation even though base-height control can support some shelf reaching.

VR 3-point teleoperation.

The sparse VR interface combines three planar navigation commands with positions and quaternions for head and two wrists: 3+9+12=24 body channels, plus grippers. A motion planner infers the lower-body motion. This is expressive for arms but gives only indirect control of torso, knees and feet.

Joint-based whole-body teleoperation.

PICO whole-body motion is retargeted online through GMR into the robot’s joints, then tracked by SONIC. The 32 body targets comprise root roll/pitch/yaw-rate, fourteen arm joints, twelve leg joints and three waist joints. This supports direct foot use, squatting and coordinated locomotion/manipulation.

SMPL-based whole-body teleoperation.

SMPL records a human representation before robot retargeting: three root terms, a six-dimensional wrist/arm refinement and 72 human joint coordinates, or 81 body channels plus grippers. Its greater redundancy is a proposed explanation for inferior policy learning, not an independently isolated cause. Errors include lifting a heel without clearing the toes and passing a can during inaccurate approach.

C.2 Whole-Body VLA Policy Design

The VLA experiments change its action/proprioception interface, pretrained initialization and generation procedure. Most main ablations alter one choice from the default at a time.

Action and proprioception interface.

Projection surgery preserves pretrained entries and Xavier-initializes the added 32→34 dimensions. Default ordering is left arm/gripper, right arm/gripper, legs, waist, then root; humanoid-native ordering moves root and legs first. Absolute versus relative targets concerns the 29 actuated joints; the appendix defines the relative target as action minus the chunk’s initial state. Proprioception has the same 34-channel layout. Combining absent proprioception with relative commands can cause drift, even though the separate ablations show only small differences.

Pretraining ablations.

The full π₀.₅ initialization includes robot-pretrained action components and a PaliGemma visual-language backbone. The PaliGemma-only ablation retains the backbone but randomly initializes action-side layers; the scratch baseline initializes everything randomly. π₀.₅’s pretraining data here contain static or wheeled dual-arm robots, with no humanoid data. Its superior physical recovery behavior is not captured by the nearly indistinguishable validation MSE of the first two variants.

One-step action generation.

One-step flow matching changes inference integration from ten steps to one without retraining. The drifting alternative retrains a replacement action expert with a drifting objective while preserving the backbone. Both target faster generation but perform substantially worse on the four-task robot evaluation; the explanation involving temporal jitter/smoothness remains a hypothesis.

C.3 Heterogeneous Co-Training

Cheap demonstrations retain manipulation segments while omitting approach and departure walking. Their usefulness differs between same-embodiment stationary teleoperation and retargeted human data.

Stationary teleoperation demonstrations.

Stationary teleoperation uses the same robot and joint interface after it has already reached the workspace. It includes arms, grippers, torso height and in-place turns, then stops before departure. Thus it preserves relevant whole-body manipulation while saving locomotion, reset and scene-collection effort.

Stationary HuMI demonstrations.

HuMI records corresponding manipulation phases without the robot, then applies offline IK to produce the same state/action format. GoPro fisheye rectification reduces but does not eliminate camera mismatch. To align with the teleoperation tracker’s 0.2-second future-reference preview, HuMI actions are shifted forward by that interval. Retargeting and gripper differences remain plausible obstacles to new-motion transfer.

C.4 Hyperparameters

Training uses AdamW (β₁=0.9, β₂=0.95), peak learning rate 10⁻⁴, cosine decay, 1,000 warmup steps, 30,000 total steps, batch 128, horizon 50 and 224×224 observations. Four A800 GPUs take about 24 hours. Head images receive crop/rotation augmentation; all images receive brightness/contrast/saturation changes. At inference, generate fifty actions at 30 Hz, execute twenty-five, and replan every 5/6 second. This concrete configuration should not be confused with the typical hierarchy frequencies introduced earlier.

Hyperparameter Value
Optimizer AdamW
Optimizer momentum β1=0.9,β2=0.95\beta_{1}=0.9,\ \beta_{2}=0.95
Peak learning rate 1×10−41\times 10^{-4}
Learning rate schedule Cosine decay
Warmup steps 1​k1\mathrm{k}
Training steps 30​k30\mathrm{k}
Batch size 128128
Action horizon 5050
Observation resolution 224×224224\times 224
Table 4: Training hyperparameters. Default configuration for OpenHLM VLA training.

Appendix D System-Level Comparison on Long-Horizon Tasks

The longer experiment compares complete systems on language-directed fruit arrangement across surfaces of different heights. Its original task card and evaluation-pair table remain intact.

D.1 Task Details & Evaluation Protocol

The robot starts away from a 40 cm coffee table, takes the first named fruit with its right hand, takes the second from a 70 cm table with its left, places them into separate containers at 95 cm, and returns. Eight points separately credit navigation, each correct pick, shelf alignment, each placement and return. Twenty ordered pairs of five fruits yield 120 demonstrations per condition. Ten selected pairs are evaluated once each, with every fruit appearing twice in each role and photographed initial scenes shared across methods; these ten progress scores underlie the reported mean and standard error.

Long-horizon Task: Fruit Arrangement • Language Prompt: Pick the {fruit 1} with the right hand, pick the {fruit 2} with the left hand, and place them on the shelf. • Detailed Process: The robot is required to pick up two designated fruits from a set of distractor fruits using both hands, and then place them into two target containers on the top level of a shelf. Specifically, the robot first picks {fruit 1} from a low coffee table at a height of 40 cm using its right hand, and then picks {fruit 2} from a medium-height table at a height of 70 cm using its left hand. The target containers are placed on the top level of the shelf at a height of 95 cm. These surfaces span a wide range of heights within the Unitree G1 humanoid’s whole-body reachable workspace. The robot must turn and walk to appropriate positions beside each platform before performing the corresponding manipulations, so that the target objects are brought into its reachable range. The robot starts several steps away from the first table and is required to return to the starting location after completing all manipulation steps, forming a closed-loop task sequence for experimental convenience. • Evaluation Rule: The task is evaluated with a total of 8 points: – The robot successfully walks to an appropriate position beside the low coffee table at a height of 40 cm. (1 pt) – The robot kneels down and correctly picks up {fruit 1} among the distractor fruits on the low coffee table using its right hand. (1 pt) – The robot stands up and successfully walks to an appropriate position beside the medium-height table at a height of 70 cm. (1 pt) – The robot correctly picks up {fruit 2} among the distractor fruits on the medium-height table using its left hand. (1 pt) – The robot turns toward the shelf and adjusts to an appropriate pose for placing the fruits. (1 pt) – The robot raises its right hand above the top level of the shelf and releases {fruit 1} into the blue plate. (1 pt) – The robot raises its left hand above the top level of the shelf and releases {fruit 2} into the orange basket. (1 pt) – The robot turns around and returns to the starting location. (1 pt) • Capability Stressed: Long-horizon compositional loco-manipulation .
Long-horizon Task: Fruit Arrangement
rollout index 0 1 2 3 4 5 6 7 8 9 fruit 1 banana peach mangosteen lemon tomato banana peach mangosteen lemon tomato fruit 2 peach mangosteen banana peach banana lemon tomato lemon tomato mangosteen Table 5: Chosen pairs for long-horizon task evaluation. Each fruit appears as ‘fruit 1’ and ‘fruit 2’ twice. Held-out fruits ( i.e. , lemon and tomato) of OpenHLM (HuMI co-training) are highlighted.
Table 5: Chosen pairs for long-horizon task evaluation. Each fruit appears as ‘fruit 1’ and ‘fruit 2’ twice. Held-out fruits (i.e., lemon and tomato) of OpenHLM  (HuMI co-training) are highlighted.

D.2 Baselines & Data Collection

GR00T N1.6 uses a Cosmos-2B VLM and diffusion-transformer action expert. The adaptation replaces its hand controls with two grippers, adds wrist cameras and retains decoupled arm/waist/height/navigation commands; it trains 50,000 steps with batch 32 and fifty-action chunks. Ψ₀ uses a Qwen3-VL backbone and flow-based MMDiT expert. To reuse the same demonstrations/controller, its 36-dimensional layout is retained with unused hand channels masked and state channels zero-padded, and its extra target-yaw term is unused. It trains 50,000 steps, batch 64, with thirty-action chunks and RTC. The authors expect limited degradation from this adaptation but do not validate the original Ψ₀ end-to-end controller in this comparison.

Appendix E Additional Experimental Results

Additional experiments separate preview latency, whole-body demonstration count, per-task co-training behavior and HuMI demonstration count.

E.1 Future-frame Preview Latency Analysis

On the single cola task, future previews of 0, 0.2, 0.4 and 0.6 seconds give about 60%, 67%, 60% and 13% progress. Zero preview feels responsive but produces vibration/stomping; 0.6 seconds creates difficult lag. At 0.2 seconds, average recorded demonstration duration is 35.2 seconds, close to zero preview’s 34.3. The main multitask policy’s 86.7% cola score is a different experiment: mixed related tasks provide additional learning signal. A different tracker could change the optimal preview.

Original figure
Figure 10: Future-frame preview latency sweep on Cola Placement.

E.2 Scaling of Teleoperation Demonstrations

Across the four-task subset, five, ten, twenty and forty demonstrations per task yield roughly 52%, 60%, 85% and 91% progress. The largest improvement is ten to twenty. Forty demonstrations are retained as the working default, estimated at about 1.5 hours of skilled-operator collection per medium task; that collection-effort estimate includes more than the recorded trajectory durations used in other tables.

Original figure
Figure 11: Whole-body teleop data scaling.

E.3 Per-Task Results on the HLM-12 Benchmark

The complete twelve-task plot shows no clear training-task regression from co-training. Both cheap-data sources nearly reach the oracle on motion-reuse tasks 9–11. Only stationary same-robot demonstrations succeed on task 12’s new pouring motion, explaining the aggregate difference. The plot preserves individual tasks and all three aggregates rather than hiding them behind one average.

Original figure
Figure 12: Per-task breakdown of heterogeneous co-training results. Task progress for all 12 tasks and three aggregates (rightmost). Four conditions: 8-task baseline, stationary co-training, HuMI co-training, and 12-task oracle. Co-training does not regress training tasks. On held-out tasks, both methods reach near-oracle on motion-reuse tasks (9–11); stationary succeeds on the new-motion task (12), HuMI does not.

E.4 Scaling HuMI Demonstrations

With the eight-task whole-body set fixed, five, ten, twenty and forty HuMI demonstrations per held-out motion-reuse task give about 42%, 67%, 76% and 84% average progress. Gains diminish at higher counts. This sweep concerns tasks 9–11, not pouring; it supports economical semantic grounding but does not demonstrate new-motion transfer.

Original figure
Figure 13: Scaling HuMI demonstrations per task. Average task progress on the three motion-reuse held-out tasks (Tasks 9–11) as we vary the number of HuMI demonstrations per task, with the 8-task whole-body teleop set fixed. Performance climbs from 42% at 5 demos to 84% at 40 demos.
Concept demonstration

Open image ↗