Literature
Display
← All papers

ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids

Gechen Qu · Tong Zhang · Bike Zhang · Yen-Jen Wang · Koushil Sreenath · Claire Tomlin · Jason Jangho Choi

Preprint posted · CoRL 2026 (accepted) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

The paper studies residual reinforcement learning as an empirical safety filter for humanoid whole-body balance and motion tracking.

Refer to caption
Figure 1: Humanoid balance tasks under various payloads. We introduce ResSafe, a performance-safety decoupling framework based on residual reinforcement learning which learns safety filtering. (a)-(d) Successful experiments of balance motions under our policy. (e) Failure case of the baseline policy. See more details in Section 5.3 and Table 3.

At a glance

ResSafe freezes a task-tracking policy and trains a second policy to add small action corrections that prolong safe operation. Simulation and G1 hardware experiments show improved balance under disturbances, with trade-offs in tracking accuracy. The residual resembles a minimum-change safety filter, but has no certified invariant set or hard safety guarantee. Its demonstrated transfer is across balance motions and reference-policy checkpoints within the training setup, not arbitrary humanoid tasks.

1 Introduction

Tracking accuracy, robustness, and fall avoidance can conflict when optimized in one policy. ResSafe separates task learning from a residual correction stage, aiming to improve the empirical trade-off without online optimization. This design reduces interference between objectives but does not eliminate reward weights, domain randomization, or conservative behavior. A universal reusable safety policy is the longer-term motivation, not an achieved guarantee.

Refer to caption
Figure 2: ResSafe Overview. (a) By decoupling the learning of performance and safety, we aim to achieve a better performance–safety Pareto frontier. (b) To achieve this, we propose a residual RL-based architecture in which the residual RL policy learns a safety correction, resembling the structure of safety filters [3, 25]. (c) We train the residual RL safety filtering policy primarily through a maximum-survival-time reachability formulation. To improve generalization across various reference actions, we expose the residual RL policy during training to varying levels of reference motion difficulty, reference policy safety, and domain randomization.

2 Related Works

Classical control barrier functions [3] and reachability methods characterize safe actions through models and certificates. Learning-based methods can instead approximate safety quantities, while residual RL modifies an existing controller. ResSafe connects these strands by learning the correction directly and compares against discriminating-hyperplane filtering [25]. The theoretical filtering framework provides motivation; its exact guarantees do not automatically transfer to a neural residual trained from samples.

3 Background & Problem Formulation

The background formulates safety as remaining inside a state constraint set. A reachability value function characterizes the maximal controlled-invariant subset, and a minimum-norm filter projects a reference action into a safe action set. For suitable control-affine settings, a discriminating hyperplane represents a half-space constraint on actions. ResSafe later replaces explicit set construction and projection with a learned correction. The retained equations describe the ideal model-based objects separately from their learned approximation.

Equations 1–9
x˙​(t)=f⁡(x⁡(t),u⁡(t))​for​t≥0,x⁡(0)=x,\dot{\mathrm{x}}(t)=f(\mathrm{x}(t),\mathrm{u}(t))\;\;\text{for}\;t\geq 0,\;\;\mathrm{x}(0)=x, (1)
infu⁡(⋅)∈𝒰\displaystyle\inf_{\mathrm{u}(\cdot)\in\mathcal{U}} ∫0∞e−γ​t​g​(x⁡(t),u⁡(t))​𝑑t\displaystyle\int_{0}^{\infty}e^{-\gamma t}g(\mathrm{x}(t),\mathrm{u}(t))dt (2a)
s.t. x⁡(t)∈𝒞​∀t≥0,\displaystyle\mathrm{x}(t)\in\mathcal{C}\;\;\forall t\geq 0, (2b)
V⁡(x,t):=maxu⁡(⋅)⁡mins∈[0,t]⁡c⁡(x⁡(s)),V⁡(x):=limt→∞V⁡(x,t).\displaystyle V(x,t):=\max_{\mathrm{u}(\cdot)}\min_{s\in[0,t]}c(\mathrm{x}(s)),\quad\quad V(x):=\lim_{t\rightarrow\infty}V(x,t). (3)
𝒰safe∗​(x)={U,V⁡(x)>0,{u∈U|∇V(x)⊤f(x,u)≥0},V⁡(x)=0.\mathcal{U}_{\mathrm{safe}}^{*}(x)=\begin{cases}U,&V(x)>0,\\[5.69054pt] \left\{u\in U\;\middle|\;\nabla V(x)^{\top}f(x,u)\geq 0\right\},&V(x)=0.\end{cases} (4)
infu⁡(⋅)∈𝒰∫0∞e−γ​t​g​(x⁡(t),u⁡(t))​𝑑ts.t.u⁡(t)∈𝒰safe∗​(x⁡(t))​∀t≥0.\inf_{\mathrm{u}(\cdot)\in\mathcal{U}}\int_{0}^{\infty}e^{-\gamma t}g(\mathrm{x}(t),\mathrm{u}(t))dt\quad\text{s.t.}\quad\mathrm{u}(t)\in\mathcal{U}_{\mathrm{safe}}^{*}(\mathrm{x}(t))\;\;\forall t\geq 0. (5)
πref=min⁡∫0∞π∈Π⁡e−γ​t​g​(x⁡(t),u⁡(t))​𝑑ts.t.u⁡(t)=π⁡(x⁡(t))​∀t≥0,\pi_{\mathrm{ref}}=\min_{\pi\in\Pi}\int_{0}^{\infty}e^{-\gamma t}g(\mathrm{x}(t),\mathrm{u}(t))dt\quad\text{s.t.}\quad\mathrm{u}(t)=\pi(\mathrm{x}(t))\;\;\forall t\geq 0, (6)
πsafe​(x⁡(t),πref​(x⁡(t)))\displaystyle\pi_{\mathrm{safe}}(\mathrm{x}(t),\pi_{\mathrm{ref}}(\mathrm{x}(t))) =arg⁡minu∈U⁡‖u−πref​(x⁡(t))‖2\displaystyle=\arg\min_{u\in U}\|u-\pi_{\mathrm{ref}}(\mathrm{x}(t))\|^{2} (7a)
s.t. u∈𝒰safe​(x​(t)).\displaystyle\quad u\in\mathcal{U}_{\mathrm{safe}}(\mathrm{x}(t)). (7b)
πsafe​(x,u)=u+Δ​πsafe​(x,u).\pi_{\mathrm{safe}}(x,u)=u+\Delta\pi_{\mathrm{safe}}(x,u). (8)
𝒰safe​(x)={u∈U|a​(x)⊤​u≥b⁡(x)}.\mathcal{U}_{\mathrm{safe}}(x)=\{u\in U\;|\;a(x)^{\top}u\geq b(x)\}. (9)

Running Example (Cart-Pole).

The cart-pole example deliberately conflicts: the reference controller targets cart position 0.6 while safety requires position at most 0.5. A learned hyperplane filter prevents the violation and supplies a comparison for residual corrections. The example illustrates filtering behavior in a small system; it does not establish high-dimensional humanoid guarantees.

Refer to caption
Figure 3: Cart-Pole Running Example. (a) Reference policy is trained for a task positioning the cart position to p=0.6p=0.6, and violates the safety constraint, p≤0.5p\leq 0.5. (b) Learned safety filters can successfully prevent the state from safety violations. (c) πsafe\pi_{\mathrm{safe}} under the learned min-norm discriminating hyperplane safety filter (blue) and residual RL (proposed method, brown).

4 Learning Safety Filtering with Residual Reinforcement Learning

The proposed method learns the additive correction itself, using survival-oriented rewards and a penalty for deviating from the nominal action.

4.1 Learning Safety Filtering with Residual RL

The residual receives state or observations together with the reference action. PPO maximizes accumulated alive reward until termination while penalizing correction magnitude. Expected survival time is a practical training objective, but by itself is not a certificate of constraint satisfaction or generally identical to maximizing full-horizon survival probability. In the cart-pole illustration, the learned correction resembles the hyperplane filter while intervening somewhat more conservatively.

Equation 10
maxΔ​πsafeθ⁡𝔼at∼πsafeθ​(st,πref​(st))​[∑t=0Tsurv𝟏−λ​‖Δ​πsafeθ​(st,at)‖22],\max_{\Delta\pi_{\mathrm{safe}}^{\theta}}\;\mathbb{E}_{a_{t}\sim\pi_{\mathrm{safe}}^{\theta}(s_{t},\pi_{\mathrm{ref}}(s_{t}))}\;\Big[\sum_{t=0}^{T_{\mathrm{surv}}}\mathbf{1}-\lambda||\Delta\pi_{\mathrm{safe}}^{\theta}(s_{t},a_{t})||_{2}^{2}\Big], (10)

4.2 Learning Safety Filters for Humanoid Balance

For the 29-action G1, the nominal controller follows the HuB balance-tracking pipeline [53]. A collection of 10,000 generated motions supports broader balance training. The residual adds velocity-stabilization and action-smoothness terms to survival and minimal-intervention objectives. Thus the practical reward includes several designed terms, detailed in the restored appendix table.

An asymmetric actor–critic gives the deployable actor proprioception and the nominal action while the critic receives privileged simulator state. Payloads and pushes expose near-failure states. Training against multiple nominal-policy checkpoints improves transfer to unseen checkpoints; training against one checkpoint can overfit its action distribution.

5 Experimental Results

The evaluation compares tracking error and fall frequency under payload disturbances, then tests checkpoint transfer and physical balance motions. The trade-off studies use one reference motion, while broader generalization and hardware studies use the motion dataset.

5.1 Experimental Setup

Baselines are the nominal tracking policy, a nominal policy trained with payload randomization, and a PPO-trained discriminating-hyperplane filter. Tracking error is measured relative to the robot base; falls provide the safety outcome. These measures assess balance in the tested scenarios, not all possible hazards such as environmental collision or unsafe contact forces.

5.2 Simulation Results

In the robustness experiment, payload fall rates are 7.29% for the nominal policy, 1.20% for randomized nominal training, and 1.11% for ResSafe. ResSafe’s payload tracking error is 37.66 mm versus 43.28 mm for randomized training. Without payload, however, the nominal policy tracks more accurately: 24.95 mm versus ResSafe’s 34.04 mm.

The separate safety-weight study shows a trade-off rather than universal dominance. ResSafe reaches 0.5% falls at 68.65 mm error; a moderate safety-penalty baseline reaches zero observed falls at 101.68 mm, while the unaugmented randomized baseline has 1.17% falls at 59.89 mm. A larger safety weight worsens both metrics. Multi-checkpoint residual training reduces degradation on held-out checkpoints compared with single-checkpoint training, within the evaluated motion-policy family.

Table 1: Performance–Robustness Trade-off.
Policy 𝑬𝐩𝐨𝐬​-​𝒍\boldsymbol{E_{\mathrm{pos}\text{-}l}} ↓\downarrow (w. payload) 𝑬𝐩𝐨𝐬​-​𝒍\boldsymbol{E_{\mathrm{pos}\text{-}l}} ↓\downarrow (w/o payload) Fall Rate↓\downarrow (w. payload)
Baseline 38.6 mm 24.95 mm 7.29%
Baseline* 43.28 mm 44.40 mm 1.20%
ResSafe 37.66 mm 34.04 mm 1.11%
Table 2: Performance–Safety Trade-off.
Policy (safety-cost weight) 𝑬𝐩𝐨𝐬​-​𝒍\boldsymbol{E_{\mathrm{pos}\text{-}l}} ↓\downarrow Fall Rate↓\downarrow
Baseline* 59.89 mm 1.17%
Baseline* + Safety (50) 101.68 mm 0.00%
Baseline* + Safety (100) 353.55 mm 1.63%
ResSafe 68.65 mm 0.5%
Refer to caption
Figure 4: Generalization across different reference policies. (a) Both the residual policy and the hyperplane policy are trained on a single reference-policy checkpoint. When they are applied to out-of-distribution reference-policy checkpoints, the fall rate increases sharply. (b) Our method and the hyperplane policy are trained with half of the checkpoints in the dataset, which leads to a smaller safety degradation when evaluated with unseen checkpoints.

5.3 Hardware Results

Four hardware motions are tested ten times each. ResSafe records 2/10, 1/10, 0/10, and 0/10 falls; the nominal policy records 10/10, 8/10, 10/10, and 5/10. Improvements are substantial against that baseline but mixed against the stronger alternatives. On the first payload motion, randomized nominal training has zero falls and the hyperplane filter has one, both fewer than ResSafe’s two.

Randomized nominal training also has lower tracking error than ResSafe on all four listed motions. ResSafe’s advantage is lower fall counts on three of them, especially the third and fourth payload motions. The source prose’s broad claim of outperforming every baseline in both metrics is therefore not supported uniformly by Table 3. Zero falls in ten trials is an observed sample result, not a safety guarantee.

6 Conclusion

Separating task tracking from residual correction can improve useful regions of the balance-performance trade-off and transfer to hardware. The experiments support a practical learned intervention mechanism, while explicit guarantees and broad task-independent reuse remain unresolved.

7 Limitations

The learned filter has no hard safety guarantee. Its conservative behavior lacks a complete theoretical explanation, and hardware transfer still depends on standard identification and randomization techniques. Generalization is bounded by motion-data diversity. The filter is blind to the environment beyond proprioception, so adding perception and environmental affordances is an important proposed extension.

Table 3: Real-World Results (# indicates motion id in dataset).
Method Fall 𝑬𝐩𝐨𝐬​-​𝒍\boldsymbol{E_{\mathrm{pos}\text{-}l}} (mm) Method Fall 𝑬𝐩𝐨𝐬​-​𝒍\boldsymbol{E_{\mathrm{pos}\text{-}l}} (mm)
(a) #9350 (w/ payload) (b) #1665 (w/o payload)
Baseline 10/10 84.13 ±\pm 12.99 Baseline 8/10 80.42 ±\pm 23.16
Baseline* 0/10 47.73 ±\pm 4.01 Baseline* 2/10 73.67 ±\pm 13.53
DH 1/10 67.88 ±\pm 20.13 DH 3/10 87.95 ±\pm 21.10
Ours 2/10 49.81 ±\pm 11.65 Ours 1/10 76.43 ±\pm 6.22
(c) #6337 (w/ payload) (d) #8407 (w/ payload)
Baseline 10/10 73.56 ±\pm 14.68 Baseline 5/10 75.49 ±\pm 28.52
Baseline* 5/10 62.00 ±\pm 14.43 Baseline* 2/10 48.49 ±\pm 13.62
DH 9/10 82.62 ±\pm 22.98 DH 4/10 66.10 ±\pm 15.04
Ours 0/10 70.01 ±\pm 3.11 Ours 0/10 61.01 ±\pm 4.41

Acknowledgments

The authors thank their discussion and hardware collaborators and acknowledge VESSL AI compute support, NSF and DARPA programs, and the UCLA Samueli School of Engineering.

Appendix A Residual Reinforcement Learning Details

The appendix specifies the complete residual reward, actor and critic inputs, training hyperparameters, and disturbance curriculum. Tables 4–8 were absent from the source HTML except for captions and are restored below from the original PDF.

A.1 Reward Design

The practical reward weights are 1.6 for the alive bonus, 2.5 for horizontal velocity stabilization, and 4 for angular velocity stabilization, with scale parameters 0.3 and 1.2. Penalties are −60 for termination, −0.07 for squared deviation from the reference action, and −0.01 for action changes. These additional terms are important for reproducing the empirical filter and qualify the simplified survival-plus-minimum-change objective.

Table 4: Reward components and weights.
Original Table 4 restored from the paper PDF without rescaling.
Term Expression Weight Remarks
Reward
Alive bonus 11 1.61.6
Base linear velocity stabilization exp⁡(−‖vx​y,tbase‖22σlin2)\exp\!\left(-\dfrac{\|v_{xy,t}^{\text{base}}\|_{2}^{2}}{\sigma_{\text{lin}}^{2}}\right) 2.52.5 σlin=0.3\sigma_{\text{lin}}=0.3
Base angular velocity stabilization exp⁡(−‖ωx​y​z,tbase‖22σang2)\exp\!\left(-\dfrac{\|\omega_{xyz,t}^{\text{base}}\|_{2}^{2}}{\sigma_{\text{ang}}^{2}}\right) 44 σang=1.2\sigma_{\text{ang}}=1.2
Penalty
Termination 11 −60-60
Minimal intervention ‖atapplied−atreference‖22\left\|a_{t}^{\text{applied}}-a_{t}^{\text{reference}}\right\|_{2}^{2} −0.07-0.07
Action smoothness ‖atapplied−at−1applied‖22\left\|a_{t}^{\text{applied}}-a_{t-1}^{\text{applied}}\right\|_{2}^{2} −0.01-0.01

A.2 State Space Design

The critic uses 691 input dimensions: privileged rigid-body state and actions plus two history steps of joint state, angular velocity, projected gravity, and applied action. The deployable actor uses 122 dimensions comprising joint positions/velocities, base angular velocity, projected gravity, applied action, and reference-policy action. The original tables retain the exact dimension breakdown.

Table 5: State space information of the critic network input.
Original Table 5 restored from the paper PDF without rescaling.
State Term Dimensions
Rigid body position 87
Rigid body rotation 180
Rigid body velocity 90
Rigid body angular velocity 90
Applied action 29
Reference policy action 29
History State Term Dimensions
DoF position 29
DoF velocity 29
Base angular velocity 3
Projected gravity 3
Applied action 29
History steps 2
Total dim 𝟓𝟎𝟓+𝟗𝟑×𝟐=𝟔𝟗𝟏\mathbf{505+93\times 2=691}
Table 6: State space information of the actor network input.
Original Table 6 restored from the paper PDF without rescaling.
State Term Dimensions
DoF position 29
DoF velocity 29
Base angular velocity 3
Projected gravity 3
Applied action 29
Reference policy action 29
Total dim 𝟏𝟐𝟐\mathbf{122}

A.3 Training Parameters

PPO uses an Adam learning rate of 10⁻³, a 36-step rollout, twelve minibatches, five learning epochs, discount 0.99, clipping 0.2, and entropy coefficient 0.001. The MLP hidden widths are 512, 256, and 128. Table 7 preserves the remaining optimizer, gradient, value-loss, and policy-variance settings.

Table 7: Hyperparameters.
Original Table 7 restored from the paper PDF without rescaling.
Hyperparameter Value
Optimizer Adam
Adam coefficients (β1,β2)(\beta_{1},\beta_{2}) (0.9,0.999)(0.9,0.999)
Learning rate 1×10−31\times 10^{-3}
Rollout horizon NstepsN_{\mathrm{steps}} 3636
Number of mini-batches 1212
Discount factor γ\gamma 0.990.99
PPO clipping parameter 0.20.2
Entropy coefficient 0.0010.001
Gradient clipping norm 55
Value loss coefficient 11
Initial policy std. 0.50.5
Number of learning epochs 55
MLP hidden layers [512, 256, 128][512,\ 256,\ 128]

A.4 Domain Randomization

The method inherits HuB dynamics randomization and adds payloads on left/right wrists, elbows, and shoulders, with each listed link’s added mass sampled from 0–3 kg. Push velocity is sampled up to a curriculum maximum of 0.5 m/s every four seconds. The bound increases after episodes longer than 300 steps and decreases below 250 steps. This exposes the residual to challenging states while adapting disturbance severity.

Table 8: Domain-randomization settings.
Original Table 8 restored from the paper PDF without rescaling.
Term Value
Push Disturbance Curriculum Learning
Curriculum variable vx​y∈𝒰⁡(0,vmax)​m/s,interval=4​sv_{xy}\in\mathcal{U}(0,v_{\max})\,\mathrm{m/s},\quad\mathrm{interval}=4\,\mathrm{s}
Initial value vmax=0.0​m/sv_{\max}=0.0\,\mathrm{m/s}
Maximum value vmax=0.5​m/sv_{\max}=0.5\,\mathrm{m/s}
Update rule vmax←min⁡(vmax+0.01, 0.5),if episode length>300,vmax←max⁡(vmax−0.01, 0.0),if episode length<250,vmax←vmax,otherwise.\begin{aligned} v_{\max}&\leftarrow\min(v_{\max}+0.01,\ 0.5),&&\text{if episode length}>300,\\ v_{\max}&\leftarrow\max(v_{\max}-0.01,\ 0.0),&&\text{if episode length}<250,\\ v_{\max}&\leftarrow v_{\max},&&\text{otherwise}.\end{aligned}
Payload Disturbance
Payload links {L/R wrist,L/R elbow,L/R shoulder}\{\text{L/R wrist},\ \text{L/R elbow},\ \text{L/R shoulder}\}
Added mass per link madd∼𝒰⁡(0,3)​kgm_{\mathrm{add}}\sim\mathcal{U}(0,3)\,\mathrm{kg}

Appendix B Experimental Details

Experimental details distinguish privileged single-motion trade-off studies from the deployable student-policy generalization and hardware setup.

B.1 Performance–Robustness Trade-off

The robustness trade-off uses one motion and the teacher reference policy to avoid distillation confounds. Both actor and critic receive the privileged Table 5 state representation in this study. It should therefore not be treated as identical to the limited-observation hardware actor.

B.2 Performance–Safety Trade-off

The safety-weight study likewise uses a single motion, the teacher reference policy, and the Table 5 input for both actor and critic. This isolates the reward/training comparison within that controlled setting.

B.3 Generalization Across Different Reference Policies

Checkpoint generalization uses the student general reference policy and full motion dataset. The purpose is transfer across controller checkpoints and a broader balance-motion distribution, rather than a change to an unrelated task family.

B.4 Hardware Results

Hardware policies are trained with the student general reference controller on the full motion dataset and tested on different motions from that dataset. These tests demonstrate sim-to-real execution within the studied motion domain, not held-out task-family generalization.

Concept demonstration

Open image ↗