At a glance
CCRL learns increasingly complex quadruped behaviors by reusing a library of frozen skills. Three networks learn how to weight those skills, give them intermediate goals, and supply residual actions. Penalizing the residual keeps new movements close to the parent skills. Simulation experiments cover locomotion, pushing, crawling, door opening, and interactive navigation; selected policies transfer to a Unitree A1 without hardware fine-tuning.
Introduction
Interactive navigation combines locomotion with contact-rich actions such as opening a door. Training every behavior from scratch is difficult, while merely selecting fixed skills limits what the robot can do. CCRL instead constructs a hierarchy recursively: each learned composite can become a parent of a more demanding skill.
The contribution is the combination of multiplicative policy composition, goal synthesis, and regularized residual learning on a 12-DOF quadruped. The skill-dependency graph is supplied manually; learning discovers how to combine its available parents. [12] [46]
Related work
The method connects legged locomotion, interactive navigation, and hierarchical RL.
Robotic Locomotion
Learned locomotion policies can replace substantial manual controller design, but simulation mismatch impairs deployment. Prior approaches use randomization, learned actuator models, adaptation, or real-world learning. CCRL adopts domain randomization to transfer interaction skills. [7] [9] [24]
Interactive Navigation
Interactive navigation requires changing the environment to reach a target. This study directly controls the joints of a legged robot to crawl, push, and open doors, extending beyond navigation with wheeled mobile manipulators. Object search itself is outside its scope: target locations are available.
Hierarchical Reinforcement Learning
Hierarchical RL commonly selects or modulates low-level primitives. CCRL builds multiple levels of composite skills and adds a trainable residual at each new level, so a new behavior can depart from the existing repertoire instead of only rearranging it.
Cascaded Compositional Residual Learning
The framework learns prerequisite skills first, freezes them, and trains a new policy from their outputs and a residual component.
Background: Markov Decision Process
Each motor task is an MDP with states, joint actions, transitions, rewards, and a discount factor. Long horizons and interaction dynamics make reward design and tuning difficult; a successful simulated behavior must also have a motion style suitable for hardware.
Background: Residual Learning
Residual learning begins with a frozen base policy and learns a correction to its actions. This transfers a useful behavior to a related task without relearning it entirely. CCRL extends the idea to multiple, recursively reused parents. [12]
Equation
Problem Definition: Multi-skill Learning
Tasks share a robot and action space but can require different observations and goals. A manually specified directed skill graph identifies prerequisites—for example, reaching builds on walking, turning, and standing. Previously learned policies form a library. Extra or imperfect parents can be downweighted, as tested in the ablations.
Cascaded Compositional Residual Learning
A new skill contains three trainable networks. A weight network combines parent action distributions through a normalized weighted product; a residual network contributes task-specific actions; a goal-synthesis network supplies intermediate targets to goal-conditioned parents. Observation selectors provide the inputs each parent needs. The parents themselves stay frozen. [46]
The residual is treated as another compositional policy component, extending the simple additive residual formulation. After training, this entire composite can serve as a parent in the next stage.
Equation
Equation 1
| (1) |
Auxiliary Loss
Unrestricted residuals can dominate the library and produce unstable motions. The training objective therefore penalizes both residual action magnitude and residual weight. PPO trains the new policy; value and entropy coefficients are 1.0 and 0.01, while the two residual coefficients are tuned per task. A new critic is trained for each skill.
The saved weight, goal, and residual networks become reusable components. These penalties influence motion style empirically; they are not a formal guarantee of hardware feasibility.
Equations 2
| (2) |
Experiments
Evaluation compares simulated task success and motion quality, tests redundant or poor parents, and demonstrates transfer to a physical quadruped.
Problem Formulation
The policy uses a 60-dimensional state including base motion, orientation, position, projected gravity, joint states, and relative object locations. Its 12 target joint angles feed PD controllers. Rewards combine tracking, stability, smoothness, contact, effort, and task-progress terms; the original ten terms are retained below. Their scale factors are 0.25, 0.25, and 2.0.
The ten skills are straight walking, left/right turning, standing, easy reaching, easy/hard door opening, object pushing, crawling, and interactive reaching. Reaching succeeds within 0.5 m; pushing requires the puck within 0.1 m of its target. Hard door opening includes approaching the door, while interactive reaching combines navigation and contact in a furnished two-room scene.
- (R1)
Linear velocity tracking:
- (R2)
Angular velocity tracking:
- (R3)
Pitch and roll penalty:
- (R4)
Joint acceleration penalty:
- (R5)
Collision penalty:
- (R6)
Action change penalty :
- (R7)
Torque penalty :
- (R8)
Door angle:
- (R9)
Distance to target:
- (R10)
Object-target distance:
Simulation Setup
Training uses 4,096 parallel Isaac Gym environments on one NVIDIA Titan X GPU. Ground friction is sampled from 0.5–1.25. Policies use two fully connected layers with 256–512 units and ELU activations; they optimize the modified PPO objective.
| Success Rate | ||||||||||||
|
|
|
| |||||||||
| CCRL | 0.90 | 0.98 | 0.89 | 0.72 | ||||||||
| Vanilla | 0.95 | 0 | 0.04 | 0.46 | ||||||||
| Curriculum | 0.92 | 0.98 | 0.52 | 0.48 | ||||||||
| Big Policy | 0.92 | 0 | 0.01 | 0.22 | ||||||||
Simulation Results
CCRL succeeds in 90% of easy-reaching trials, 98% of hard-door trials, 89% of pushing trials, and 72% of interactive-reaching trials. Curriculum learning ties its hard-door result but reaches 52% on pushing and 48% on interactive reaching. A larger policy trained from scratch does not explain the improvement. Easy reaching is an exception: the vanilla baseline reaches 95%.
Torque traces and videos show smoother, lower-effort movements than the baselines. The paper attributes this to anchoring new behavior in the parent library, with residual regularization limiting simulation-specific shortcuts.
Simulation Analysis
Adding a randomly initialized parent leaves easy-reaching success at 90%; adding unnecessary turning skills to straight walking leaves tracking error near the parent’s 16%. These are targeted robustness tests, not arbitrary-graph guarantees. Removing residual penalties still allows target reaching but produces poorer motion judged unsuitable for transfer.
Sim2real transfer
A physical Unitree A1 demonstrates pushing, crawling, door opening, and their use in interactive navigation without additional fine-tuning. Motion capture supplies robot and object states. The hardware evidence is qualitative demonstrations; the reported success-rate table is simulation-based, and this is not a vision-only autonomous deployment.
Conclusion
Recursive skill reuse and regularized residuals enable complex interaction behaviors while preserving useful locomotion patterns. The remaining manual skill graph, state instrumentation, and task-specific tuning limit generality. Proposed extensions include discovering skill dependencies, using visual observations, and adapting to varied object dynamics.