Literature
Display
← All papers

Cascaded Compositional Residual Learning for Complex Interactive Behaviors

K. Niranjan Kumar · Irfan Essa · Sehoon Ha

· IEEE Robotics and Automation Letters ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

CCRL learns increasingly complex quadruped behaviors by reusing a library of frozen skills. Three networks learn how to weight those skills, give them intermediate goals, and supply residual actions. Penalizing the residual keeps new movements close to the parent skills. Simulation experiments cover locomotion, pushing, crawling, door opening, and interactive navigation; selected policies transfer to a Unitree A1 without hardware fine-tuning.

Introduction

Interactive navigation combines locomotion with contact-rich actions such as opening a door. Training every behavior from scratch is difficult, while merely selecting fixed skills limits what the robot can do. CCRL instead constructs a hierarchy recursively: each learned composite can become a parent of a more demanding skill.

The contribution is the combination of multiplicative policy composition, goal synthesis, and regularized residual learning on a 12-DOF quadruped. The skill-dependency graph is supplied manually; learning discovers how to combine its available parents. [12] [46]

Refer to caption
Figure 1: Our framework allows robots to learn complex environment interactions with a recursive hierarchy. The image shows a motion frame of the robot crawling under a table, reaching the door, pushing it open, and walking through it to reach a target location.

Related work

The method connects legged locomotion, interactive navigation, and hierarchical RL.

Robotic Locomotion

Learned locomotion policies can replace substantial manual controller design, but simulation mismatch impairs deployment. Prior approaches use randomization, learned actuator models, adaptation, or real-world learning. CCRL adopts domain randomization to transfer interaction skills. [7] [9] [24]

Interactive Navigation

Interactive navigation requires changing the environment to reach a target. This study directly controls the joints of a legged robot to crawl, push, and open doors, extending beyond navigation with wheeled mobile manipulators. Object search itself is outside its scope: target locations are available.

Hierarchical Reinforcement Learning

Hierarchical RL commonly selects or modulates low-level primitives. CCRL builds multiple levels of composite skills and adds a trainable residual at each new level, so a new behavior can depart from the existing repertoire instead of only rearranging it.

Cascaded Compositional Residual Learning

The framework learns prerequisite skills first, freezes them, and trains a new policy from their outputs and a residual component.

Background: Markov Decision Process

Each motor task is an MDP with states, joint actions, transitions, rewards, and a discount factor. Long horizons and interaction dynamics make reward design and tuning difficult; a successful simulated behavior must also have a motion style suitable for hardware.

Background: Residual Learning

Residual learning begins with a frozen base policy and learns a correction to its actions. This transfers a useful behavior to a related task without relearning it entirely. CCRL extends the idea to multiple, recursively reused parents. [12]

Equation
π⁡(s)=π0​(s)+πk​(s)\pi(s)=\pi_{0}(s)+\pi_{k}(s)

Problem Definition: Multi-skill Learning

Tasks share a robot and action space but can require different observations and goals. A manually specified directed skill graph identifies prerequisites—for example, reaching builds on walking, turning, and standing. Previously learned policies form a library. Extra or imperfect parents can be downweighted, as tested in the ablations.

Refer to caption
Figure 2: Directed graph representing the relationship between skills. Each skill is built as a compositional policy over its parent policies and a learned residual specific to that skill.

Cascaded Compositional Residual Learning

A new skill contains three trainable networks. A weight network combines parent action distributions through a normalized weighted product; a residual network contributes task-specific actions; a goal-synthesis network supplies intermediate targets to goal-conditioned parents. Observation selectors provide the inputs each parent needs. The parents themselves stay frozen. [46]

The residual is treated as another compositional policy component, extending the simple additive residual formulation. After training, this entire composite can serve as a parent in the next stage.

Equation
πk=ℱk​(𝒞k∪{πkr})\pi_{k}=\mathcal{F}_{k}(\mathcal{C}_{k}\cup\{{\pi_{k}^{r}}\})
Original figure
Figure 3: Overview of our Cascaded Compositional Residual Learning, which consists of three learnable components: a synthetic goal network, a weight network, and a residual action network. The example illustrates the architecture for Interactive Reach, which uses Reach target (Easy) on top of other primitive skills.
Equation 1
π⁡(a∣s,g)=1Z⁡(s,g)​∏i=0kπi​(a∣s,g)wi​(s,g),\pi(a\mid s,g)=\frac{1}{Z(s,g)}\prod_{i=0}^{k}\pi_{i}(a\mid s,g)^{w_{i}(s,g)}, (1)

Auxiliary Loss

Unrestricted residuals can dominate the library and produce unstable motions. The training objective therefore penalizes both residual action magnitude and residual weight. PPO trains the new policy; value and entropy coefficients are 1.0 and 0.01, while the two residual coefficients are tuned per task. A new critic is trained for each skill.

The saved weight, goal, and residual networks become reusable components. These penalties influence motion style empirically; they are not a formal guarantee of hardware feasibility.

Refer to caption
(a) Straight walking
Refer to caption
(b) Turning
Refer to caption
(c) Reach Target (Easy)
Refer to caption
(d) Crawling
Refer to caption
(e) Door Open (Easy)
Refer to caption
(f) Door Open (Hard)
Refer to caption
(g) Push Object to Target
Refer to caption
(h) Interactive Reach
Figure 4: Overview of the tasks we train our robot to accomplish.
Equations 2
Ltr​w=𝔼t​[∣wr​e​s​(st,gt)∣1]L^{rw}_{t}=\mathbb{E}_{t}\left[{\mid w_{res}(s_{t},g_{t})\mid}_{1}\right]
Ltr​m=∣𝔼t​[πr​e​s​(at∣st,gt)]∣1L^{rm}_{t}={\mid\mathbb{E}_{t}\left[\pi_{res}(a_{t}\mid s_{t},g_{t})\right]\mid}_{1}
LtP​P​O​(θ)=𝔼^t​[LtC​L​I​P​(θ)−c1​LtV​F​(θ)+c2​S​[πθ]​(st)]L_{t}^{PPO}(\theta)=\hat{\mathbb{E}}_{t}\left[L_{t}^{CLIP}(\theta)-c_{1}L_{t}^{VF}(\theta)+c_{2}S\left[\pi_{\theta}\right]\left(s_{t}\right)\right]
Lt​(θ)=LtP​P​O​(θ)+c3​Ltr​w+c4​Ltr​m.L_{t}(\theta)=L_{t}^{PPO}(\theta)+c_{3}L^{rw}_{t}+c_{4}L^{rm}_{t}. (2)

Experiments

Evaluation compares simulated task success and motion quality, tests redundant or poor parents, and demonstrates transfer to a physical quadruped.

Problem Formulation

The policy uses a 60-dimensional state including base motion, orientation, position, projected gravity, joint states, and relative object locations. Its 12 target joint angles feed PD controllers. Rewards combine tracking, stability, smoothness, contact, effort, and task-progress terms; the original ten terms are retained below. Their scale factors are 0.25, 0.25, and 2.0.

The ten skills are straight walking, left/right turning, standing, easy reaching, easy/hard door opening, object pushing, crawling, and interactive reaching. Reaching succeeds within 0.5 m; pushing requires the puck within 0.1 m of its target. Hard door opening includes approaching the door, while interactive reaching combines navigation and contact in a furnished two-room scene.

  1. (R1)

    Linear velocity tracking: exp(−(𝐯t​a​r​g​e​t−𝐯)2/σ1)exp(-(\mathbf{v}_{target}-\mathbf{v})^{2}/\sigma_{1})

  2. (R2)

    Angular velocity tracking: exp(−(ωzt​a​r​g​e​t−ωz)2/σ2)exp(-(\omega_{z_{target}}-\omega_{z})^{2}/\sigma_{2})

  3. (R3)

    Pitch and roll penalty: ωx2+ωy2\omega_{x}^{2}+\omega_{y}^{2}

  4. (R4)

    Joint acceleration penalty: ∑(𝐪˙t−𝐪˙t−1Δ​t)2\sum(\frac{\dot{\mathbf{q}}_{t}-\dot{\mathbf{q}}_{t-1}}{\Delta t})^{2}

  5. (R5)

    Collision penalty: nc​o​n​t​a​c​tn_{contact}

  6. (R6)

    Action change penalty : ∑(𝐚t−𝐚t−1)2\sum(\mathbf{a}_{t}-\mathbf{a}_{t-1})^{2}

  7. (R7)

    Torque penalty : ∑𝝉2\sum\boldsymbol{\tau}^{2}

  8. (R8)

    Door angle: qd​o​o​rq_{door}

  9. (R9)

    Distance to target: exp(−‖𝐱t​a​r​g​e​t‖/σ3)exp(-\left\|\mathbf{x}_{target}\right\|/\sigma_{3})

  10. (R10)

    Object-target distance: exp(−‖𝐱t​2​o‖/σ3)exp(-\left\|\mathbf{x}_{t2o}\right\|/\sigma_{3})

Simulation Setup

Training uses 4,096 parallel Isaac Gym environments on one NVIDIA Titan X GPU. Ground friction is sampled from 0.5–1.25. Policies use two fully connected layers with 256–512 units and ELU activations; they optimize the modified PPO objective.

Refer to caption
Figure 5: Heatmap showing the weight magnitude on the Open door (Easy) skill. Notice the increased activation of the skill when the robot is close to the door.
Success Rate
Reach Target
(Easy)
Open Door
(Hard)
Push
Object
Interactive
Reach
CCRL 0.90 0.98 0.89 0.72
Vanilla 0.95 0 0.04 0.46
Curriculum 0.92 0.98 0.52 0.48
Big Policy 0.92 0 0.01 0.22
Table I: Performance comparison of our method against three baselines Vanilla, Curriculum, and Big Policy. Our method achieves significantly better success rates on harder tasks, such as Push Object or Interactive Reach.

Simulation Results

CCRL succeeds in 90% of easy-reaching trials, 98% of hard-door trials, 89% of pushing trials, and 72% of interactive-reaching trials. Curriculum learning ties its hard-door result but reaches 52% on pushing and 48% on interactive reaching. A larger policy trained from scratch does not explain the improvement. Easy reaching is an exception: the vanilla baseline reaches 95%.

Torque traces and videos show smoother, lower-effort movements than the baselines. The paper attributes this to anchoring new behavior in the parent library, with residual regularization limiting simulation-specific shortcuts.

Refer to caption
Figure 6: Torque curves of our approach compared to baselines. Although all techniques have a success rate of around 90−95%90-95\%, our approach has a much lower net torque magnitude and trajectories better suited for sim-to-real transfer.
Refer to caption
Figure 7: Trajectories taken by the robot from a fixed starting point in our Interactive Reach Environment. The target positions are sampled to be at grid corners inside the room. Our robot crawls under furniture, walks around obstacles and opens a door to get from one room to another.

Simulation Analysis

Adding a randomly initialized parent leaves easy-reaching success at 90%; adding unnecessary turning skills to straight walking leaves tracking error near the parent’s 16%. These are targeted robustness tests, not arbitrary-graph guarantees. Removing residual penalties still allows target reaching but produces poorer motion judged unsuitable for transfer.

Sim2real transfer

A physical Unitree A1 demonstrates pushing, crawling, door opening, and their use in interactive navigation without additional fine-tuning. Motion capture supplies robot and object states. The hardware evidence is qualitative demonstrations; the reported success-rate table is simulation-based, and this is not a vision-only autonomous deployment.

Refer to caption
(a) Push object
Refer to caption
(b) Open door
Refer to caption
(c) Crawling
Figure 8: We transfer the skills, (a) Push object, (b) Open door and (c) Crawling to the real world without any finetuning.

Conclusion

Recursive skill reuse and regularized residuals enable complex interaction behaviors while preserving useful locomotion patterns. The remaining manual skill graph, state instrumentation, and task-specific tuning limit generality. Proposed extensions include discovering skill dependencies, using visual observations, and adapting to varied object dynamics.

Concept demonstration

Open image ↗