Overview
This edition covers the complete sixteen-page arXiv v1 dated 4 June 2026, including the reference-motion, reward, randomization, collision-geometry and deployment appendices. All ten figures and five numbered tables remain available.
At a glance
LadderMan trains geometry-specific climbing experts from one human reference, then combines imitation and reinforcement learning into one depth-conditioned climbing policy. Foundation-model stereo depth supports transfer to a Unitree G1; a separate dual-agent controller balances on the ladder while tracking teleoperated upper-body targets.
Experiments cover simulated geometry variation and three real ladders. Results demonstrate this system’s feasibility, with limited real-world trials and qualitative manipulation examples; they do not establish autonomous task-level manipulation or unrestricted ladder generalization.
1 Introduction
Sparse rungs require coordinated hand and foot contacts, so small perception or control errors can destabilize the entire humanoid. Conventional model-based climbing relies on accurate environment models, while many learned locomotion policies emphasize the legs rather than sustained whole-body support.
The pipeline combines asymmetric motion tracking, ladder-relative contact targets, expert distillation with reinforcement learning, and cleaner depth perception. A separate teleoperation controller decouples upper-body tracking from lower-body stabilization. The paper promises future code/models; that promise is not evidence of release status.
2 Related Works
Prior humanoid ladder control and quadruped ladder learning motivate adapting contacts beyond one geometry. The latter uses privileged scene information and specialized hardware rather than the depth-driven humanoid setting studied here.
Mapping-based locomotion can miss thin rungs and introduce calibration/latency errors. RPL and other direct-depth methods avoid explicit maps but primarily address lower-body locomotion; perceptive humanoid parkour emphasizes dynamic interactions. LadderMan instead targets sustained, precise multi-contact climbing.
3 Method
First learn state-based experts for inclination/rung-spacing pairs; then distill them into one direction-commanded depth policy. Deployment combines stereo depth with the climbing controller. On-ladder manipulation uses a separately trained two-agent policy after a control switch, rather than the same climbing network generating task-level actions.
3.1 Learning Multiple Expert Policies from Single Reference Motion
A single OptiTrack reference supplies joint/body targets and manually annotated hand/foot contact timing. Relax upper-body tracking by halving its weight and error sensitivity, while keeping stronger lower-body tracking. Geometry-specific rung targets then let hands and feet relocate without discarding the original climbing pattern.
Experts observe reference motion, proprioception, privileged body state and pelvis-to-ladder pose. Contact loss for more than thirty simulation frames triggers termination; local/global tracking errors and randomized dynamics, initial states and ladder geometry further shape training.
The printed task-reward formula is the positive root-to-target distance even though the text describes stabilization and maximizing return. Its sign/weight is not resolved in the paper; the exact equation is retained without silently converting it into a negative-distance reward.
Equations 1–2
| (1) |
| (2) |
3.2 Unified Climbing Policy via Hybrid Imitation and Reinforcement Learning
Pure DAgger distillation accumulates errors and cannot fully cover geometry variations. Add a PPO objective using expert rewards plus KL regularization toward expert actions, then anneal the imitation weight to favor task optimization.
The deployed policy receives proprioception, depth and an up/down command and outputs whole-body joint targets. Warp renders training depth. The source does not fully specify expert assignment over the geometry space, the total expert count or the imitation-weight schedule.
Equation 3
| (3) |
3.3 Bridging Sim-to-Real Perception Gap with Vision Foundation Model
Fast-FoundationStereo estimates depth from stereo images to reduce missing pixels and noisy rung geometry. Training still adds Gaussian noise written N(0,.005), .05 pixel dropout and clipping to .1–2 m; the notation does not explicitly distinguish variance from standard deviation.
Rung-focused masking removes non-rung regions with probability .1 during training, discouraging reliance on specific rails and supports. It is not described as a required test-time segmentation input. Figure 3 labels dropout p=.9 whereas the method and appendix state .05; both original sources remain unchanged.
3.4 On-Ladder Manipulation via Dual-Agent Learning
Separate upper- and lower-body actors/critics receive whole-body proprioception. Only the upper agent receives arm/wrist/waist targets: AMASS poses during training and inverse-kinematics targets from teleoperation at deployment. The lower agent rewards contacts and pelvic stabilization; concatenate their joint targets for control.
Upper-body collisions with the ladder are disabled during manipulation training to enlarge the workspace. This design does not establish collision avoidance in deployment. The approach follows FALCON’s decomposition, and remains human-guided manipulation.
Equations
4 Experimental Results
IsaacSim trains and evaluates policies for a 1.3-m, 29-degree-of-freedom G1 without hardware modifications. A single reference is collected on a 65.5° ladder with 24.8-cm rung spacing. Experiments test geometry variation, expert-learning ablations, transfer components and teleoperation stability.
4.1 Real-World Evaluation
Three real ladders have inclination/spacing/rung dimensions A=(65.5°,24.8 cm,60×8 cm), B=(69.2°,23.1 cm,28×8 cm), and C=(64.8°,25.5 cm,30×20 cm). A supplies the reference motion; B/C differ in geometry and material. Ten randomized-start trials per ladder yield 9/10, 6/10 and 7/10 successful runs.
Selected sequences show painting adjustment, bulb replacement/tightening and box handover after climbing. PICO 4 Ultra headset/controllers and ankle trackers provide motion targets. The TWIST2 comparison shows balance loss after switching controllers, but reports no task-wise manipulation success counts or aggregate failure rates.
Two human subjects across three ladders average about 3.2 s/rung versus the robot’s 3.4. This limited timing comparison does not establish population-level equivalence; the supplement retains one complete ascent/descent pair.
4.2 Simulation Evaluation
Evaluate 24 inclination/spacing pairs: 55/60/65/70° crossed with 20/22/24/26/28/30 cm. Randomize rung size and cuboid/cylindrical shape and run 100 episodes per pair. The depth policy reaches 95–100% in the central 55–65°,24–28-cm region; the prose reports about 57% near extreme boundaries.
The blind BeyondMimic-style baseline peaks at 49% near the reference and is near zero for many other pairs. The result supports perception plus this full training system; it is not a matched ablation of visual input alone. Figure 5 retains every heatmap cell and both geometry panels.
4.3 Ablation Studies
After 8,000 training iterations, judge whether a stable expert emerges on three geometries. DeepMimic and the no-contact variant succeed only at the reference geometry; removing climbing reward succeeds at the small/reference ladders but fails at the large ladder. Full hybrid tracking succeeds at all three. This is a binary learning outcome, not a measured rollout success rate.
Real-world full/no-RL/no-VFM/no-masking results are 9/2/3/0 out of ten on A, 6/0/0/0 on B and 7/0/2/0 on C. These small samples support the usefulness of each component; no independent-seed variability, confidence intervals or significance tests are supplied.
The HTML expert-ablation prose points to Table 2, but the expert-learning results are Table 1 in both captions and the PDF. Table 2 contains transfer ablations.
| Method | |||
| DeepMimic [17] | ✗ | ✓ | ✗ |
| Ours | ✓ | ✓ | ✓ |
| w/o contact tracking | ✗ | ✓ | ✗ |
| w/o climb reward | ✓ | ✓ | ✗ |
5 Conclusion and Limitation
The system combines geometry-adapted contacts, expert distillation, perceptual transfer and separately controlled on-ladder manipulation. It has not been extended to vertical ladders, which demand different force and stability margins.
The limitation section says evaluation is primarily on inclines up to 75°, while the explicitly enumerated simulation grid ends at 70° and the three real ladders at 69.2°. The lack of dexterous end effectors also limits manipulation. These bounded demonstrations do not establish general industrial deployment readiness.
Appendix A Method
The appendix retains the full decision-process formulation, observations, rewards, randomization ranges, networks and deployment details underlying the main pipeline.
A.1 Learning Multiple Expert Policies from a Single Reference Motion
An expert maps observations to joint-position targets executed by a proportional–derivative controller, maximizing discounted reward. The one complete climbing reference is captured at 65.5°/24.8 cm, with four end-effector contact annotations made once; geometry supplies the corresponding target rung positions.
Table 3 preserves reference body/joint poses and phase, root angular velocity/orientation, joint-position history, previous actions and privileged geometry/body errors. Table 4 gives separate upper/lower tracking weights/temperatures, .3 hand/foot contact rewards and all regularizers.
Randomize mass, center of mass, joint offsets, friction/restitution and reference initialization; perturb each target ladder by ±1° inclination and ±1 cm spacing. Terminate local/global pose deviations above .5 m or 1.2 rad after 25 steps, or required contact loss after thirty.
PPO uses 4,096 environments on one L40S for 8,000 epochs. Actor/critic hidden sizes are [512,256,256]/[512,256,128]. Tables preserve exact ranges, including broad friction values; the overall expert grid, total compute and several PPO settings are not fully reported.
Equations 4–6
| (4) |
| (5) |
| (6) |
| Observation Group | Description | Equation |
| Reference Motion | reference body local pose | |
| reference joint position | ||
| reference motion phase | ||
| Proprioception | root angle velocity | |
| root orientation | ||
| joint position | ||
| previous actions |
| Observation Group | Description | Equation |
| Privilege State | ladder relative pose | |
| body local pose | ||
| body velocity | ||
| root velocity | ||
| body pose error | ||
| body velocity error | ||
| joint position error |
Equations 7–8
| (7) |
| (8) |
| Reward | Weight | Temperature | |
| Motion Tracking | |||
| Body Local Pose | Upper Body | ||
| Lower Body | |||
| Body Velocity | Upper Body | ||
| Lower Body | |||
| Joint Tracking | Upper Body | ||
| Lower Body | |||
| Root Pose | - | ||
| Contact Tracking | |||
| Hands Contact Position | |||
| Feet Contact Position | |||
| Reward | Weight |
| Regularization Penalties | |
| Action Rate L2 | |
| Joint Position Limits L1 | |
| Joint Velocity Penalty | |
| Joint Torque Limits L1 | |
| Feet Impact Force L2 | |
| Feet Slip | |
| Feet Air Time | |
| Domain Randomization | Sampling Distribution | Domain Randomization | Sampling Distribution |
| Physical parameters | Reference initialization | ||
| Link mass | Reference root position [m] | ||
| Torso COM offset [m] | Reference root rotation [rad] | ||
| Joint position offset [rad] | Reference joint position [rad] | ||
| Static friction coefficient | Ladder Configuration | ||
| Dynamic friction coefficient | Ladder inclination [∘] | ||
| Restitution coefficient | Ladder rung spacing [m] |
A.2 Unified Climbing Policy via Hybrid Imitation and Reinforcement Learning
The unified actor receives climbing direction, proprioception and 60×80 depth. Its MLP is [2048,1024,512,256,128], its critic [512,256,128], and its depth encoder three convolutions with eight channels each and kernel size five.
Warp supplies simulation depth. Alongside .05 dropout, Gaussian noise and .1–2-m clipping, camera extrinsics vary ±1 cm and ±1.5° because the onboard camera is not rigidly mounted. The source does not give a complete depth-model latency or end-to-end control-delay budget.
A.3 Bridging the Sim-to-Real Gap
Cleaner stereo depth addresses perception; accurate collision geometry addresses contact dynamics. Simplified TWIST2/HDMI models poorly represent ladder contacts. LadderMan starts from OmniRetarget and emphasizes improved shin/hand/foot geometry beyond simple primitives.
The figures preserve the model comparisons, but the text does not specify a complete mesh-generation procedure or quantify a controlled collision-model ablation. Small contact discrepancies are presented as a practical transfer concern.
A.4 On-Ladder Manipulation via Dual-Agent Learning
The separate manipulation policy uses two actor/critic pairs with [512,256,256]/[512,256,128] hidden layers. Upper-body target tracking and lower-body stabilization are trained as distinct objectives; the appendix does not supply a separate quantitative manipulation benchmark.
Equation
Appendix B Experimental Results
Supplementary deployment and visual comparisons document how policies, stereo inference and teleoperation are combined, and retain reference adaptation, depth and human timing examples.
B.1 Experiment Setup
Climbing and manipulation policies run at 50 Hz on the onboard Jetson Orin. PICO 4 Ultra streams target motion without an external motion-capture system at deployment; motion capture was used to collect the reference.
Fast-FoundationStereo processes stereo grayscale from the D435i with infrared emitter and automatic exposure disabled. Its inference runs on an RTX 4090 during deployment. Thus onboard policy execution is distinct from fully onboard perception; the paper gives no complete communication/latency characterization.
B.2 Additional Results
At 55°/20 cm, the learned policy visibly adapts hand/foot contacts relative to the 65.5°/24.8-cm reference while retaining its overall pattern. This selected sequence complements, rather than replaces, the grid evaluation.
One three-rung ascent plus three-rung descent takes 19.3 s for the robot and 18.2 s for a human. Depth sequences show cleaner predicted rung structure than raw sensing, but no metric depth-error comparison or safety guarantee accompanies them.