Overview
This edition covers the complete nine-page arXiv v4, dated 21 July 2026. All seven figures and five tables are retained, including the opening figure hidden inside the HTML title-note markup and three appendix tables omitted from the HTML section hierarchy. Zichao Hu is the correspondence author.
At a glance
VOFA combines a depth-based high-level pushing policy with a frozen force-adaptive whole-body controller on the Booster T1 humanoid. Privileged-state reinforcement learning trains a teacher; DAgger distills it into a student using onboard proprioception, low-resolution depth histories and a supplied goal position.
The student reaches 91.38% reported overall simulation success. Real goal-position tests yield 25 successes in thirty trials; a separate 17-kg box condition succeeds three times in five attempts. Relative goals come from motion capture during deployment, while object feedback is visual.
I Introduction
Pushing a large object to a location requires locomotion, contact stability and feedback under unknown mass/friction. The hierarchy separates choosing robot–object approach geometry from maintaining whole-body stability under interaction forces.
The high-level policy learns to move around the box before contact, using an alignment reward rather than manually staged approach/push training. A force-perturbation-trained controller handles contact, while visual augmentation supports transfer from simulation.
Selected examples show recovery after lateral box displacement, off-center loading and goal changes. The study concerns one box-shaped object and supplied spatial goals; broader rearrangement with multiple objects remains future work.
II Related Work
Related work separates reusable humanoid whole-body controllers from autonomous perception-driven locomotion and manipulation. VOFA combines these lines for visually guided pushing under varied object properties.
II-A Humanoid Whole Body Control
Motion-imitation controllers provide coordinated locomotion and balance, while teleoperation systems extend them to object interaction. FALCON specifically trains against unknown end-effector forces. VOFA uses this controller as a frozen low-level component and learns a goal-directed visual command policy above it.
II-B Sim-to-Real Reinforcement Learning for Autonomous Humanoid Loco-Manipulation
Earlier dynamic object-goal pushing and other legged manipulation systems often use privileged object state. VisualMimic, VIRAL and pixel-to-action systems introduce visual control. This paper focuses on the combination of goal variation, onboard object sensing and mass/friction variation; it does not rerun all those systems as direct benchmarks.
III Method
At each control step, map proprioceptive/depth histories and a desired object-goal position to a high-level command, then map that command through the whole-body controller to joint targets. Both policy loops run at 50 Hz, but new depth observations are processed at 5 Hz.
The deployed student receives onboard object observations, while the goal is an external relative-position input. The exact policy/history notation remains below; the simplified low-level conditional notation does not enumerate all controller internals.
Equations
III-A High-Level Policy Design and Training
Train a teacher with privileged state, then imitate its command choices with a visual student. This separates discovering useful pushing behavior from learning a representation robust to deployment sensing.
Teacher Policy Design
Use PPO with an asymmetric actor–critic. Actor inputs include base linear/angular velocity, projected gravity, previous high-level command, upper-body joint state, both end-effector–object relative positions, object orientation and goal position, expressed in the robot base frame.
The critic additionally receives contact, object center of mass, mass, dimensions and linear/angular velocity. Commands stay within the low-level controller’s valid range. Table III retains the authors’ exact observation dimensions, including unusual entries that are not silently corrected.
Equations
Student Policy Design
DAgger trains the student on its visited states with teacher commands. Temporal 1D convolutions encode proprioceptive history; a CNN encodes each depth frame; an MLP fuses those features for command prediction.
The student omits privileged object state and base linear velocity, using angular velocity, gravity, previous command and joint positions/velocities. Maintain three recent depth frames, process depth at 5 Hz and reduce 640×360 input to 32×32. Table IV specifies a 67-dimensional proprioceptive input and 3×32×32 image history.
Printed history indices t−50:t and t−3:t are retained exactly; the prose explicitly specifies three buffered depth frames, so inclusive endpoint notation should not be used to invent a fourth frame.
Equations
III-B Low-Level Policy Design and Training
Pretrain FALCON for 10,000 PPO epochs with AMASS motions and a curriculum of increasing external forces at the end effectors, then freeze it during high-level training. Force adaptation is learned through perturbations, without a reported explicit object-mass estimator or new force-sensor input.
The high-level command controls locomotion and residual upper-body joint targets. Fix walking mode, root height and waist yaw to defaults 1, 0.62 and 0. The complete command definitions are preserved below; Table III gives thirteen previous-command dimensions.
Equations
III-C Reward Design
Equations 1–2 reward both hands reaching the box via the harmonic mean of two exponential distance scores, discouraging a single hand from dominating. Equation 3 rewards proximity of object and goal. Equation 4 rewards head alignment only while the object lies inside the specified field-of-view limit.
Equation 5 favors the robot being opposite the goal across the object, encouraging repositioning before pushing. Its normalized-angle convention is not fully specified. Table V retains all shaping scales, weights and penalties for command changes, object tilt and upper-joint actions.
Equations 1–5
| (1) |
Selectable original MathML
| (2) |
| (3) |
| (4) |
| (5) |
| Light | Medium | Heavy | Extra Heavy (Unseen) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | SR(%) | Pre-C.(%) | Post-C.(%) | SR(%) | Pre-C.(%) | Post-C.(%) | SR (%) | Pre-C.(%) | Post-C.(%) | SR(%) | Pre-C.(%) | Post-C.(%) |
| Teacher w/ FA (ours) | 97.40 | 1.12 | 1.34 | 98.22 | 0.92 | 0.74 | 97.46 | 1.06 | 1.08 | 87.92 | 0.88 | 4.16 |
| Teacher w/o FA | ||||||||||||
| Student w/ FA (ours) | 93.50 | 1.46 | 3.06 | 93.30 | 1.40 | 2.74 | 89.16 | 1.22 | 3.22 | 69.00 | 1.30 | 6.30 |
| Student w/o FA | ||||||||||||
III-D Environment Setup and Domain Randomization
Train in Isaac Gym with box mass 1–8 kg and friction 0.1–1.0. Sample goals over the full 360° at radius 2.5–3.5 m; start the box 1.8–2.2 m away within ±60° of the robot’s view.
Match ZED 2i intrinsics, then perturb camera translation by ±3 cm on each axis and orientation by ±5° yaw/pitch and ±2° roll. Add upper-image far-depth patches, spatially correlated Gaussian depth noise and pixel dropout to mimic background/sensor artifacts.
Figure 3’s labels identify (a) unaugmented simulation, (b) augmented simulation and (c) reality. Some prose cross-references reverse (b)/(c); the original panels and captions are retained with their printed labels.
Selectable original panel labels
(a) Sim Vanilla
(b) Sim w/ Aug.
(c) Real
Selectable original panel labels
(a) w/ Force-Adaptive Controller (ours)
(b) w/o Force-Adaptive Controller
IV Experimental Results
Simulation evaluates goal angle, object mass and force-adaptation ablations. Real tests evaluate goal configurations, mass changes and visual randomization. Separate qualitative demonstrations inspect response to disturbances, asymmetric loading and sequential goals.
Selectable original panel labels
(a) w/ Align Reward (ours)
(b) w/o Align Reward
IV-A Performance Across Initial Configurations and Masses
Success requires placing the box within 0.3 m of the goal and maintaining that position for two seconds. Pre-C. and Post-C. count robot falls before and after object contact. Remaining failures include failure to reach the goal; fall percentages are not the complete failure rate.
Across front/lateral/rear angle groups, the force-adaptive teacher exceeds 97% and the student exceeds 90% success. Reported overall rates are 97.84%/91.38%. Some overall entries do not equal the unweighted means of their three displayed groups; simulation sample counts and aggregation details are not supplied.
For light 1–3, medium 3–5 and heavy 5–8 kg boxes, student success is 93.50%, 93.30% and 89.16%. Unseen 8–12 kg boxes yield 69.00% versus 32.00% without force adaptation. The heavy condition therefore falls below 90%; the broad headline does not apply to every mass group.
Selectable original panel labels
(a) Different Goal Positions
(b) Different Object Masses
Selectable original panel labels
(a) External perturbation recovery.
(b) Sequential multi-goal pushing.
IV-B Effect of Force-Adaptive Whole-Body Control on VOFA
Remove random end-effector force perturbations during low-level pretraining, then train corresponding high-level teacher/student policies. This keeps the hierarchical task design while changing the controller’s force experience.
Without force adaptation, reported overall teacher/student success falls to 76.94%/59.32%, with poorer lateral/rear and heavier-box performance. Figure 4 illustrates unstable kicking rather than steady hand pushing. The comparison establishes the value of the trained controller within this system; it is not a direct measured-force feedback ablation.
IV-C Impact of Object–Goal Alignment Reward
Without alignment shaping, the reach reward encourages premature contact from the wrong side, blocking useful exploration of walking around the box. Figure 5 shows the resulting failure and the aligned policy’s repositioning behavior.
This ablation is qualitative in the paper: no separate numeric success table, repeated-run distribution or reward-weight sensitivity curve is supplied.
IV-D Sim-to-Real Transfer Performance
Deploy on Booster T1 with a 5-kg cardboard box approximately 1.0×1.0×0.7 m. Motion capture supplies the relative goal; the policy still locates/interacts with the object from depth. A separate localization system is suggested as a possible replacement, but is not evaluated.
Ten trials per goal condition give 9/10 front, 8/10 lateral and 8/10 rear successes. In a separate fixed 4-m front-goal experiment, add 0, 2.5, 4.5 or 12 kg payload to the 5-kg box, giving total masses 5, 7.5, 9.5 and 17 kg.
Five trials per mass produce 4/5, 5/5, 5/5 and 3/5 successes with force adaptation, versus 4/5, 4/5, 2/5 and 1/5 without it. Both 9.5 and 17 kg exceed the simulation training mass limit. The 17-kg case demonstrates capability at 60% observed success, not a reliable-success guarantee.
IV-E Effectiveness of Visual Randomization
Visual augmentation improves real front/lateral/rear success counts from 8/10, 5/10 and 3/10 to 9/10, 8/10 and 8/10. The largest gains occur during longer repositioning, when viewpoint/background changes accumulate.
The comparison removes the combined augmentation strategy; it does not isolate the effect of each noise type. Observed confusion with background structures supports the proposed explanation, while ten trials per condition limit precision.
IV-F Closed-Loop Object Pushing
Three qualitative tests inspect feedback use beyond nominal straight pushing. They demonstrate selected successful responses rather than estimate recovery probabilities over standardized disturbance distributions.
External Perturbation.
A lateral impulse displaces the box. The robot stops forward motion, reorients to find it, navigates around it to restore object–goal alignment and resumes pushing. Figure 7a preserves the complete illustrated sequence.
Off-Centered Mass Distribution.
Weights attached to one side shift the center of mass, causing rotation and lateral deviation during pushing. Figure 1d shows the robot detecting drift and repositioning to continue toward the goal. Offset magnitudes and repeated-trial rates are not reported.
Sequential Goal Switching.
After reaching a front goal, supply a new goal behind the robot. It navigates around the same box and pushes toward the updated goal without reset. Figure 7b retains every frame; external goal updates are provided rather than autonomously selected.
V Conclusion, Limitations, and Future Work
The combination of privileged teacher learning, visual distillation, alignment shaping and force-adaptive control enables closed-loop pushing under varied goals and object masses.
Current evidence is limited to a single box-shaped object and relatively simple scenes. The policy assumes a relative goal position from another system, with motion capture used here. Multi-object clutter, more diverse background ambiguity and joint onboard object/goal localization remain open extensions.
Appendix: Teacher observation dimensions
Table III retains every actor/critic observation dimension. Actor entries sum to 56; adding the listed privileged critic entries gives 84. The printed entries include EE-object relative position 9, object rotation matrix 3, object mass 9 and object dimension 9.
Several labels/dimensions are unusual for the described physical quantities and may reflect reporting errors or unstated encodings. They remain unchanged because the paper does not resolve them; no corrected tensor shape is inferred.
| Actor observations | Dimensions |
|---|---|
| EE-object relative position | 9 |
| Object rotation matrix | 3 |
| Arm dof position | 10 |
| Arm dof velocity | 10 |
| Base linear velocity | 3 |
| Base angular velocity | 3 |
| Projected gravity | 3 |
| Relative goal position | 2 |
| Last action | 13 |
| Total dim | 56 |
| Priv. critic observations | Dimensions |
| Object contact | 1 |
| Object CoM | 3 |
| Object mass | 9 |
| Object dimension | 9 |
| Object linear velocity | 3 |
| Object angular velocity | 3 |
| Total dim | 84 |
Appendix: Student observation dimensions
Table IV gives 67 proprioceptive dimensions: gravity 3, angular velocity 3, joint position/velocity 23 each, previous action 13 and goal position 2. Three depth frames at 32×32 form the image input. Temporal history encoding is additional to this per-step vector.
| Student proprioception observations | Dimensions |
|---|---|
| Projected gravity | 3 |
| Base angular velocity | 3 |
| Dof position | 23 |
| Dof velocity | 23 |
| Last action | 13 |
| Relative goal position | 2 |
| Total dim | 67 |
| Student image observations | Dimensions |
| Depth image |
Appendix: Reward specification
Table V specifies (σ, weight) as reach (2.0, 1.25), push (10.0, 2.5), head tracking (0.5, 1.0) and alignment (0.25, 2.0). Penalty weights are −1.0 for command smoothness and −0.5 each for object tilt and upper-joint action magnitude.
All original penalty expressions and values are preserved. The paper does not provide every PPO/DAgger optimizer setting, exact normalized-angle mapping or full episode termination specification.
| Main Reward | Param. | Weight |
|---|---|---|
| Reach object | ||
| Push object to goal | ||
| Head track object | ||
| Object-goal alignment | ||
| Auxilary Penalty | Expression | Weight |
| Command smoothness | ||
| Object balance | ||
| Upper joint action |