Literature
Display
← All papers

LadderMan: Learning Humanoid Perceptive Ladder Climbing

Siheng Zhao · Yuanhang Zhang · Ziqi Lu · Pieter Abbeel · Rocky Duan · Koushil Sreenath · Yue Wang · C. Karen Liu · Guanya Shi

Preprint posted · CoRL 2026 (accepted) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete sixteen-page arXiv v1 dated 4 June 2026, including the reference-motion, reward, randomization, collision-geometry and deployment appendices. All ten figures and five numbered tables remain available.

Complete original figure with every panel, label and legend.
Figure 1: LadderMan enables a Unitree G1 humanoid to robustly climb diverse ladders using a single perceptive policy via zero-shot sim-to-real transfer. Building on the learned climbing capability, LadderMan further supports stable whole-body on-ladder manipulation via teleoperation, including adjusting paintings, replacing light bulbs, and box handover.

At a glance

LadderMan trains geometry-specific climbing experts from one human reference, then combines imitation and reinforcement learning into one depth-conditioned climbing policy. Foundation-model stereo depth supports transfer to a Unitree G1; a separate dual-agent controller balances on the ladder while tracking teleoperated upper-body targets.

Experiments cover simulated geometry variation and three real ladders. Results demonstrate this system’s feasibility, with limited real-world trials and qualitative manipulation examples; they do not establish autonomous task-level manipulation or unrestricted ladder generalization.

1 Introduction

Sparse rungs require coordinated hand and foot contacts, so small perception or control errors can destabilize the entire humanoid. Conventional model-based climbing relies on accurate environment models, while many learned locomotion policies emphasize the legs rather than sustained whole-body support.

The pipeline combines asymmetric motion tracking, ladder-relative contact targets, expert distillation with reinforcement learning, and cleaner depth perception. A separate teleoperation controller decouples upper-body tracking from lower-body stabilization. The paper promises future code/models; that promise is not evidence of release status.

2 Related Works

Prior humanoid ladder control and quadruped ladder learning motivate adapting contacts beyond one geometry. The latter uses privileged scene information and specialized hardware rather than the depth-driven humanoid setting studied here.

Mapping-based locomotion can miss thin rungs and introduce calibration/latency errors. RPL and other direct-depth methods avoid explicit maps but primarily address lower-body locomotion; perceptive humanoid parkour emphasizes dynamic interactions. LadderMan instead targets sustained, precise multi-contact climbing.

3 Method

First learn state-based experts for inclination/rung-spacing pairs; then distill them into one direction-commanded depth policy. Deployment combines stereo depth with the climbing controller. On-ladder manipulation uses a separately trained two-agent policy after a control switch, rather than the same climbing network generating task-level actions.

3.1 Learning Multiple Expert Policies from Single Reference Motion

A single OptiTrack reference supplies joint/body targets and manually annotated hand/foot contact timing. Relax upper-body tracking by halving its weight and error sensitivity, while keeping stronger lower-body tracking. Geometry-specific rung targets then let hands and feet relocate without discarding the original climbing pattern.

Experts observe reference motion, proprioception, privileged body state and pelvis-to-ladder pose. Contact loss for more than thirty simulation frames triggers termination; local/global tracking errors and randomized dynamics, initial states and ladder geometry further shape training.

The printed task-reward formula is the positive root-to-target distance even though the text describes stabilization and maximizing return. Its sign/weight is not resolved in the paper; the exact equation is retained without silently converting it into a negative-distance reward.

Complete original figure with every panel, label and legend.
Figure 2: Overview of LadderMan. Starting from a single reference motion, we first learn multiple expert climbing policies for different ladder geometries using hybrid motion tracking. These experts are then distilled into a unified visuomotor policy through hybrid DAgger and RL. Building on the climbing policy, we further train a dual-agent manipulation policy for stable on-ladder teleoperation. To bridge the sim-to-real perception gap, we use VFM for robust real-world deployment.
Equations 1–2
rXtracking=ω2​exp⁡(−σ2​‖X^​[Ku]−X⁡[Ku]‖2)+ω​exp⁡(−σ​‖X^​[Kl]−X⁡[Kl]‖2).r^{\text{tracking}}_{X}=\frac{\omega}{2}\exp\left(-\frac{\sigma}{2}\|\hat{X}[K_{u}]-X[K_{u}]\|_{2}\right)+\omega\exp\left(-\sigma\|\hat{X}[K_{l}]-X[K_{l}]\|_{2}\right). (1)
rcontact=ω​c^​exp⁡(−σ​‖P−P^‖2)+(1−c^),rtask=‖R−𝒢‖2,r^{\text{contact}}=\omega\hat{c}\exp\left(-\sigma\|P-\hat{P}\|_{2}\right)+(1-\hat{c}),~~r^{\text{task}}=\|R-\mathcal{G}\|_{2}, (2)
rXtracking=ω​exp⁡(−σ​‖X^​[K]−X⁡[K]‖2),r^{\text{tracking}}_{X}=\omega\exp(-\sigma\|\hat{X}[K]-X[K]\|_{2}),

3.2 Unified Climbing Policy via Hybrid Imitation and Reinforcement Learning

Pure DAgger distillation accumulates errors and cannot fully cover geometry variations. Add a PPO objective using expert rewards plus KL regularization toward expert actions, then anneal the imitation weight to favor task optimization.

The deployed policy receives proprioception, depth and an up/down command and outputs whole-body joint targets. Warp renders training depth. The source does not fully specify expert assignment over the geometry space, the total expert count or the imitation-weight schedule.

Equation 3
L(πvisual)=LRL(πvisual)+λDKL(πvisual∥πexpert),L(\pi^{\text{visual}})=L_{\text{RL}}(\pi^{\text{visual}})+\lambda D_{\text{KL}}\left(\pi^{\text{visual}}\;\|\;\pi^{\text{expert}}\right), (3)
Complete original figure with every panel, label and legend.
Figure 3: We bridge the visual gap between simulation and real world by applying rung-focused masking, vision foundation model [27], and minimalist noise augmentation.

3.3 Bridging Sim-to-Real Perception Gap with Vision Foundation Model

Fast-FoundationStereo estimates depth from stereo images to reduce missing pixels and noisy rung geometry. Training still adds Gaussian noise written N(0,.005), .05 pixel dropout and clipping to .1–2 m; the notation does not explicitly distinguish variance from standard deviation.

Rung-focused masking removes non-rung regions with probability .1 during training, discouraging reliance on specific rails and supports. It is not described as a required test-time segmentation input. Figure 3 labels dropout p=.9 whereas the method and appendix state .05; both original sources remain unchanged.

3.4 On-Ladder Manipulation via Dual-Agent Learning

Separate upper- and lower-body actors/critics receive whole-body proprioception. Only the upper agent receives arm/wrist/waist targets: AMASS poses during training and inverse-kinematics targets from teleoperation at deployment. The lower agent rewards contacts and pelvic stabilization; concatenate their joint targets for control.

Upper-body collisions with the ladder are disabled during manipulation training to enlarge the workspace. This design does not establish collision avoidance in deployment. The approach follows FALCON’s decomposition, and remains human-guided manipulation.

Complete original figure with every panel, label and legend.
Figure 4: Real-world results of LadderMan. The humanoid performs zero-shot sim-to-real ladder climbing across diverse real-world ladders (A, B, and C) and executes various on-ladder manipulation tasks while maintaining stable balance. Additional video results are available on our website.
Equations
πmanip​(st,gt)=[πu​(st,gt);πl​(st)]\pi^{\text{manip}}(s_{t},g_{t})=\left[\pi^{u}(s_{t},g_{t});\pi^{l}(s_{t})\right]
at=[atl;atu]a_{t}=[a_{t}^{{l}};a_{t}^{{u}}]

4 Experimental Results

IsaacSim trains and evaluates policies for a 1.3-m, 29-degree-of-freedom G1 without hardware modifications. A single reference is collected on a 65.5° ladder with 24.8-cm rung spacing. Experiments test geometry variation, expert-learning ablations, transfer components and teleoperation stability.

4.1 Real-World Evaluation

Three real ladders have inclination/spacing/rung dimensions A=(65.5°,24.8 cm,60×8 cm), B=(69.2°,23.1 cm,28×8 cm), and C=(64.8°,25.5 cm,30×20 cm). A supplies the reference motion; B/C differ in geometry and material. Ten randomized-start trials per ladder yield 9/10, 6/10 and 7/10 successful runs.

Selected sequences show painting adjustment, bulb replacement/tightening and box handover after climbing. PICO 4 Ultra headset/controllers and ankle trackers provide motion targets. The TWIST2 comparison shows balance loss after switching controllers, but reports no task-wise manipulation success counts or aggregate failure rates.

Two human subjects across three ladders average about 3.2 s/rung versus the robot’s 3.4. This limited timing comparison does not establish population-level equivalence; the supplement retains one complete ascent/descent pair.

Complete original figure with every panel, label and legend.
Figure 5: (a) Success rates of LadderMan and the blind motion tracking baseline across ladder configurations with varying inclination angles and rung spacings. (b) Representative ladder geometries used in simulation (top) and real-world evaluation (bottom).
Complete original figure with every panel, label and legend.
Figure 6: Failure case of the off-the-shelf teleoperation policy TWIST2 under ladder constraints.

4.2 Simulation Evaluation

Evaluate 24 inclination/spacing pairs: 55/60/65/70° crossed with 20/22/24/26/28/30 cm. Randomize rung size and cuboid/cylindrical shape and run 100 episodes per pair. The depth policy reaches 95–100% in the central 55–65°,24–28-cm region; the prose reports about 57% near extreme boundaries.

The blind BeyondMimic-style baseline peaks at 49% near the reference and is near zero for many other pairs. The result supports perception plus this full training system; it is not a matched ablation of visual input alone. Figure 5 retains every heatmap cell and both geometry panels.

4.3 Ablation Studies

After 8,000 training iterations, judge whether a stable expert emerges on three geometries. DeepMimic and the no-contact variant succeed only at the reference geometry; removing climbing reward succeeds at the small/reference ladders but fails at the large ladder. Full hybrid tracking succeeds at all three. This is a binary learning outcome, not a measured rollout success rate.

Real-world full/no-RL/no-VFM/no-masking results are 9/2/3/0 out of ten on A, 6/0/0/0 on B and 7/0/2/0 on C. These small samples support the usefulness of each component; no independent-seed variability, confidence intervals or significance tests are supplied.

The HTML expert-ablation prose points to Table 2, but the expert-learning results are Table 1 in both captions and the PDF. Table 2 contains transfer ablations.

Method (55∘,20​cm)(55^{\circ},20$\mathrm{cm}$) (65.5∘,24.8​cm)⋆~~(65.5^{\circ},24.8$\mathrm{cm}$)^{\star} (70∘,30​cm)(70^{\circ},30$\mathrm{cm}$)
DeepMimic [17] ✗ ✓ ✗
Ours ✓ ✓ ✓
w/o contact tracking ✗ ✓ ✗
w/o climb reward ✓ ✓ ✗
Table 1: Ablation study on expert policy learning methods across different ladder geometries. ⋆ denotes the ladder configuration where the reference motion was collected.
Method Ladder A Ladder B Ladder C
Ours 9/10 6/10 7/10
w/o RL 2/10 0/10 0/10
w/o VFM 3/10 0/10 2/10
w/o RFM 0/10 0/10 0/10
Table 2: Real-world ablation of key components for robust sim-to-real transfer across 33 different ladders.

5 Conclusion and Limitation

The system combines geometry-adapted contacts, expert distillation, perceptual transfer and separately controlled on-ladder manipulation. It has not been extended to vertical ladders, which demand different force and stability margins.

The limitation section says evaluation is primarily on inclines up to 75°, while the explicitly enumerated simulation grid ends at 70° and the three real ladders at 69.2°. The lack of dexterous end effectors also limits manipulation. These bounded demonstrations do not establish general industrial deployment readiness.

Appendix A Method

The appendix retains the full decision-process formulation, observations, rewards, randomization ranges, networks and deployment details underlying the main pipeline.

A.1 Learning Multiple Expert Policies from a Single Reference Motion

An expert maps observations to joint-position targets executed by a proportional–derivative controller, maximizing discounted reward. The one complete climbing reference is captured at 65.5°/24.8 cm, with four end-effector contact annotations made once; geometry supplies the corresponding target rung positions.

Table 3 preserves reference body/joint poses and phase, root angular velocity/orientation, joint-position history, previous actions and privileged geometry/body errors. Table 4 gives separate upper/lower tracking weights/temperatures, .3 hand/foot contact rewards and all regularizers.

Randomize mass, center of mass, joint offsets, friction/restitution and reference initialization; perturb each target ladder by ±1° inclination and ±1 cm spacing. Terminate local/global pose deviations above .5 m or 1.2 rad after 25 steps, or required contact loss after thirty.

PPO uses 4,096 environments on one L40S for 8,000 epochs. Actor/critic hidden sizes are [512,256,256]/[512,256,128]. Tables preserve exact ranges, including broad friction values; the overall expert grid, total compute and several PPO settings are not fully reported.

Equations 4–6
ℳ=⟨𝒪,𝒜,𝒯,ℛ,γ⟩,\mathcal{M}=\langle\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R},\gamma\rangle, (4)
rt=ℛ⁡(ot,at),r_{t}=\mathcal{R}(o_{t},a_{t}), (5)
𝔼⁡[∑t=1Tγt−1​rt].\mathbb{E}\left[\sum_{t=1}^{T}\gamma^{t-1}r_{t}\right]. (6)
Observation Group Description Equation
Reference Motion reference body local pose p^t\hat{p}_{t}
reference joint position q^t\hat{q}_{t}
reference motion phase t/Tt/T
Proprioception root angle velocity wtw_{t}
root orientation θt\theta_{t}
joint position qt−8:tq_{t-8:t}
previous actions at−3:t−1a_{t-3:t-1}
Observation Group Description Equation
Privilege State ladder relative pose ptroot−ptladderp_{t}^{\text{root}}-p_{t}^{\text{ladder}}
body local pose pt{p}_{t}
body velocity p˙t\dot{p}_{t}
root velocity vtv_{t}
body pose error pt−p^tp_{t}-\hat{p}_{t}
body velocity error p˙t−p˙^t\dot{p}_{t}-\hat{\dot{p}}_{t}
joint position error qt−q^tq_{t}-\hat{q}_{t}
Table 3: Observation space of the expert climbing policy.
Equations 7–8
r=ω​exp⁡(−Eτ),r=\omega\exp\left(-\frac{E}{\tau}\right), (7)
rcontact=ω​c^​exp⁡(−‖P−P^‖2τ)+(1−c^),r^{\text{contact}}=\omega\hat{c}\exp\left(-\frac{\|P-\hat{P}\|_{2}}{\tau}\right)+(1-\hat{c}), (8)
Reward Weight Temperature
Motion Tracking
Body Local Pose Upper Body 1.01.0 1.01.0
Lower Body 2.02.0 0.50.5
Body Velocity Upper Body 0.50.5 5.05.0
Lower Body 1.01.0 2.52.5
Joint Tracking Upper Body 0.50.5 0.50.5
Lower Body 1.01.0 0.250.25
Root Pose - 1.01.0 0.50.5
Contact Tracking
Hands Contact Position 0.30.3 0.30.3
Feet Contact Position 0.30.3 0.30.3
Reward Weight
Regularization Penalties
Action Rate L2 0.10.1
Joint Position Limits L1 10.010.0
Joint Velocity Penalty 5​e−45e^{-4}
Joint Torque Limits L1 0.010.01
Feet Impact Force L2 1.01.0
Feet Slip 0.50.5
Feet Air Time 5.05.0
Table 4: Reward terms used for expert policy learning.
Domain Randomization Sampling Distribution Domain Randomization Sampling Distribution
Physical parameters Reference initialization
Link mass mass∼𝒰⁡[0.9,1.1]×mass\text{mass}\sim\mathcal{U}[0.9,1.1]\times\text{mass} Reference root position [m] Δ​x​y​z∼𝒰⁡[−0.01,0.01]\Delta xyz\sim\mathcal{U}[-0.01,0.01]
Torso COM offset [m] Δ​x​y​z∼𝒰⁡[−0.02,0.02]\Delta xyz\sim\mathcal{U}[-0.02,0.02] Reference root rotation [rad] Δ​r​p​y∼𝒰⁡[−0.05,0.05]\Delta rpy\sim\mathcal{U}[-0.05,0.05]
Joint position offset [rad] Δ​q∼𝒰⁡[−0.01,0.01]\Delta q\sim\mathcal{U}[-0.01,0.01] Reference joint position [rad] Δ​q∼𝒰⁡[−0.05,0.05]\Delta q\sim\mathcal{U}[-0.05,0.05]
Static friction coefficient μstatic∼𝒰⁡[0.3,7.2]\mu_{\text{static}}\sim\mathcal{U}[0.3,7.2] Ladder Configuration
Dynamic friction coefficient μdynamic∼𝒰⁡[0.3,3.6]\mu_{\text{dynamic}}\sim\mathcal{U}[0.3,3.6] Ladder inclination [∘] Δ​ϕ∼𝒰⁡[−1.0,1.0]\Delta\phi\sim\mathcal{U}[-1.0,1.0]
Restitution coefficient erest∼𝒰⁡[0.0,0.2]e_{\text{rest}}\sim\mathcal{U}[0.0,0.2] Ladder rung spacing [m] Δ​z∼𝒰⁡[−0.01,0.01]\Delta z\sim\mathcal{U}[-0.01,0.01]
Table 5: Domain randomization parameters used during expert policy training.

A.2 Unified Climbing Policy via Hybrid Imitation and Reinforcement Learning

The unified actor receives climbing direction, proprioception and 60×80 depth. Its MLP is [2048,1024,512,256,128], its critic [512,256,128], and its depth encoder three convolutions with eight channels each and kernel size five.

Warp supplies simulation depth. Alongside .05 dropout, Gaussian noise and .1–2-m clipping, camera extrinsics vary ±1 cm and ±1.5° because the onboard camera is not rigidly mounted. The source does not give a complete depth-model latency or end-to-end control-delay budget.

A.3 Bridging the Sim-to-Real Gap

Cleaner stereo depth addresses perception; accurate collision geometry addresses contact dynamics. Simplified TWIST2/HDMI models poorly represent ladder contacts. LadderMan starts from OmniRetarget and emphasizes improved shin/hand/foot geometry beyond simple primitives.

The figures preserve the model comparisons, but the text does not specify a complete mesh-generation procedure or quantify a controlled collision-model ablation. Small contact discrepancies are presented as a practical transfer concern.

Complete original figure with every panel, label and legend.
Figure 7: Side-by-side comparison between a human and LadderMan performing the same ladder climbing task. LadderMan achieves human-comparable climbing speed.
Complete original figure with every panel, label and legend.
Figure 8: Comparison of Unitree G1 collision geometries used in different humanoid projects.

A.4 On-Ladder Manipulation via Dual-Agent Learning

The separate manipulation policy uses two actor/critic pairs with [512,256,256]/[512,256,128] hidden layers. Upper-body target tracking and lower-body stabilization are trained as distinct objectives; the appendix does not supply a separate quantitative manipulation benchmark.

Equation
πmanip​(st,gt)=[πu​(st,gt);πl​(st)],\pi^{\text{manip}}(s_{t},g_{t})=\left[\pi^{u}(s_{t},g_{t});\pi^{l}(s_{t})\right],

Appendix B Experimental Results

Supplementary deployment and visual comparisons document how policies, stereo inference and teleoperation are combined, and retain reference adaptation, depth and human timing examples.

B.1 Experiment Setup

Climbing and manipulation policies run at 50 Hz on the onboard Jetson Orin. PICO 4 Ultra streams target motion without an external motion-capture system at deployment; motion capture was used to collect the reference.

Fast-FoundationStereo processes stereo grayscale from the D435i with infrared emitter and automatic exposure disabled. Its inference runs on an RTX 4090 during deployment. Thus onboard policy execution is distinct from fully onboard perception; the paper gives no complete communication/latency characterization.

Complete original figure with every panel, label and legend.
Figure 9: Comparison between raw depth observations and depth predictions generated by Fast FoundationStereo. The foundation model significantly reduces sensor artifacts and improves geometric consistency, facilitating sim-to-real transfer.

B.2 Additional Results

At 55°/20 cm, the learned policy visibly adapts hand/foot contacts relative to the 65.5°/24.8-cm reference while retaining its overall pattern. This selected sequence complements, rather than replaces, the grid evaluation.

One three-rung ascent plus three-rung descent takes 19.3 s for the robot and 18.2 s for a human. Depth sequences show cleaner predicted rung structure than raw sensing, but no metric depth-error comparison or safety guarantee accompanies them.

Complete original figure with every panel, label and legend.
Figure 10: Comparison between the reference motion and the learned climbing behavior on a ladder different from that used during motion capture.
Concept demonstration

Open image ↗