Literature
Display
← All papers

Stable Modular Control via Contraction Theory for Reinforcement Learning

Bing Song · Jean-Jacques Slotine · Quang-Cuong Pham

· Learning for Dynamics and Control 2024 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

The paper integrates contraction-based modular control into reinforcement learning. It transforms dynamics into an auxiliary coordinate system, constrains elementary feedback modules, and composes them so that the authors’ stability conditions can be enforced through network Jacobians. Simulated peg tasks illustrate improved robustness and exploration. The guarantees depend on model, transformation, and coupling assumptions; they do not certify arbitrary learned policies or contact environments.

I Introduction

Reward optimization alone can favor policies with unstable behavior for some system parameters. The proposed architecture separates signal composition, which defines the policy’s latent state, from dynamic decomposition, which exposes manageable feedback subsystems. Contraction constraints are imposed on the controller while the usual RL update optimizes rewards. Models or suitable task-space controllers are still needed to construct the transformations.

Refer to caption
Fig. 1: Modular control framework for RL. The modularity, for example, hierarchical combination of subsystems, is realized in the auxiliary space by coordinate transformation (Ty,TaT_{y},T_{a}). Stability can be automatically guaranteed provided each subsystem has stable self-feedbacks, yielding linear constraints on the input gradients of control networks that can be as simple as switching the signs of weights.
Original figure
Fig. 2: Signal flows across different spaces. Signal composition creates the latent space, within which neural control with RL applies to maximizing rewards. Provided that the composition of latent signals is a contracting system, the stability of latent signals implies the stability of physical dynamics. Dynamic decomposition in the auxiliary space further simplifies the stability of latent signals into algebraically solvable ones via modularity.

I-A Technical Contributions.

The central contribution is a coordinate-transformation framework that reduces stability analysis to elementary modules and constrains their input gradients. Additional constructions transfer latent convergence to physical tracking and use task-space controller gains to obtain a usable latent dynamics model.

I-B Related work.

The method connects contraction-based control with modular policy architectures.

I-B 1 Contraction theory and safe RL

Contraction analyzes convergence between trajectories without prescribing an equilibrium. Unlike methods that learn a separate certificate or solve a constrained policy optimization problem, this approach uses stability-preserving interconnections to derive algebraic constraints on the control network. [23], [24], [6].

I-B 2 Modularity

The modularity concerns differential dynamics, beyond simply dividing a network into functions. It is related to motor primitives, synergies, and geometric motion policies, but generalization to underactuated or learned latent modules is proposed rather than demonstrated.

I-C Paper organization

The paper develops decomposition, signal composition, and implementation before presenting simulation experiments and supporting derivations.

II Dynamic decomposition

Coordinate transformations make the closed-loop differential dynamics amenable to modular analysis.

II-A Contraction from auxiliary space to latent space

Contraction in auxiliary coordinates transfers to latent dynamics through the induced metric when the transformation has the required positive-definiteness properties. Euclidean trajectories may have bounded transients even when their auxiliary distance contracts.

Original figure
Fig. 3: By constraining the dynamics in the auxiliary space to converge exponentially, the dynamics in the Euclidean space is converging with bounded transients.

II-B Differential dynamics in the auxiliary space

State and action transformations create auxiliary signals; each policy can use both the current signal and its integral. The block differential model combines plant derivatives and policy Jacobians. The derivation explicitly neglects small changes in the policy input gradient, a restriction relevant to interpreting the resulting conditions.

Equations 1–7
[𝐳𝐚]=ΘM​[𝐲𝐮]whereΘM=[TyTa−1]\begin{bmatrix}\mathbf{z}\\ \mathbf{a}\end{bmatrix}=\Theta_{M}\begin{bmatrix}\mathbf{y}\\ \mathbf{u}\end{bmatrix}\ \ \text{where}\ \ \Theta_{M}=\begin{bmatrix}T_{y}&\\ &T_{a}^{-1}\end{bmatrix} (1)
[δ​𝐳˙δ​𝐚˙]=Θ˙M​[δ​𝐲δ​𝐮]+ΘM​[δ​𝐲˙δ​𝐮˙]\begin{bmatrix}\delta\dot{\mathbf{z}}\\ \delta\dot{\mathbf{a}}\end{bmatrix}=\dot{\Theta}_{M}\begin{bmatrix}\delta\mathbf{y}\\ \delta\mathbf{u}\end{bmatrix}+\Theta_{M}\begin{bmatrix}\delta\dot{\mathbf{y}}\\ \delta\dot{\mathbf{u}}\end{bmatrix} (2)
𝐚=𝝅⁡([𝐬1𝐬2])=𝝅⁡([𝐳∫0t𝐳​𝑑t])\mathbf{a}=\bm{\pi}(\begin{bmatrix}\mathbf{s}_{1}\\ \mathbf{s}_{2}\end{bmatrix})=\bm{\pi}(\begin{bmatrix}\mathbf{z}\\ \int^{t}_{0}\mathbf{z}dt\end{bmatrix}) (3)
δ​𝐚˙=∂𝝅∂𝐬1​δ​𝐳˙+∂𝝅∂𝐬2​δ​𝐳\delta\dot{\mathbf{a}}=\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\delta\dot{\mathbf{z}}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}\delta\mathbf{z} (4)
δ​𝐲˙=∂𝐟∂𝐲​δ​𝐲+∂𝐟∂𝐮​δ​𝐮\delta\dot{\mathbf{y}}=\frac{\partial\mathbf{f}}{\partial\mathbf{y}}\delta\mathbf{y}+\frac{\partial\mathbf{f}}{\partial\mathbf{u}}\delta\mathbf{u} (5)
[δ​𝐳˙δ​𝐚˙]=[ABCD]​[δ​𝐳δ​𝐚]\begin{bmatrix}\delta\dot{\mathbf{z}}\\ \delta\dot{\mathbf{a}}\end{bmatrix}=\begin{bmatrix}A&B\\ C&D\\ \end{bmatrix}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix} (6)
A\displaystyle A =[T˙y+Ty​∂𝐟∂𝐲]​Ty−1\displaystyle=[\dot{T}_{y}+T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}]T_{y}^{-1} (7)
B\displaystyle B =Ty​∂𝐟∂𝐮​Ta\displaystyle=T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}
C\displaystyle C =∂𝝅∂𝐬1​(T˙y+Ty​∂𝐟∂𝐲)​Ty−1+∂𝝅∂𝐬2\displaystyle=\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}(\dot{T}_{y}+T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}})T_{y}^{-1}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}
D\displaystyle D =∂𝝅∂𝐬1​Ty​∂𝐟∂𝐮​Ta\displaystyle=\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}

II-C Modular neural control via dynamic decomposition

Grouping each transformed state with its action yields elementary feedback modules. Triangular couplings permit hierarchical composition, while antisymmetric couplings motivate a feedback alternative. Finding suitable transformations is itself a design problem; the paper develops an explicit hierarchical construction rather than a general existence result.

Equations 8–11
[δ​z˙1δ​a˙1δ​z˙2δ​a˙2]=[a11b11a12b12c11d11c12d12a21b21a22b22c21d21c22d22]​[δ​z1δ​a1δ​z2δ​a2]\begin{bmatrix}\delta\dot{z}_{1}\\ \delta\dot{a}_{1}\\ \hline\cr\delta\dot{z}_{2}\\ \delta\dot{a}_{2}\end{bmatrix}=\begin{bmatrix}a_{11}&b_{11}&\!\!\!\framebox{}\!\!\!&a_{12}&b_{12}\\ c_{11}&d_{11}&\!\!\!\framebox{}\!\!\!&c_{12}&d_{12}\\ \hline\cr a_{21}&b_{21}&\!\!\!\framebox{}\!\!\!&a_{22}&b_{22}\\ c_{21}&d_{21}&\!\!\!\framebox{}\!\!\!&c_{22}&d_{22}\\ \end{bmatrix}\begin{bmatrix}\delta z_{1}\\ \delta a_{1}\\ \hline\cr\delta z_{2}\\ \delta a_{2}\end{bmatrix} (8)
[δ​z˙1δ​a˙1]=[a11b11c11d11]​[δ​z1δ​a1]+[a12b12c12d12]​[δ​z2δ​a2]\displaystyle\begin{bmatrix}\delta\dot{z}_{1}\\ \delta\dot{a}_{1}\end{bmatrix}=\begin{bmatrix}a_{11}&b_{11}\\ c_{11}&d_{11}\\ \end{bmatrix}\begin{bmatrix}\delta z_{1}\\ \delta a_{1}\end{bmatrix}+\begin{bmatrix}a_{12}&b_{12}\\ c_{12}&d_{12}\\ \end{bmatrix}\begin{bmatrix}\delta z_{2}\\ \delta a_{2}\end{bmatrix} (9)
[δ​z˙2δ​a˙2]=[a22b22c22d22]​[δ​z2δ​a2]+[a21b21c21d21]​[δ​z1δ​a1]\displaystyle\begin{bmatrix}\delta\dot{z}_{2}\\ \delta\dot{a}_{2}\end{bmatrix}=\begin{bmatrix}a_{22}&b_{22}\\ c_{22}&d_{22}\\ \end{bmatrix}\begin{bmatrix}\delta z_{2}\\ \delta a_{2}\end{bmatrix}+\begin{bmatrix}a_{21}&b_{21}\\ c_{21}&d_{21}\\ \end{bmatrix}\begin{bmatrix}\delta z_{1}\\ \delta a_{1}\end{bmatrix}
[a12b12c12d12]​and​[a21b21c21d21]\begin{bmatrix}a_{12}&b_{12}\\ c_{12}&d_{12}\\ \end{bmatrix}\ \text{and}\ \begin{bmatrix}a_{21}&b_{21}\\ c_{21}&d_{21}\\ \end{bmatrix} (10)
[a11b11c11d11]​and​[a22b22c22d22]\begin{bmatrix}a_{11}&b_{11}\\ c_{11}&d_{11}\\ \end{bmatrix}\ \text{and}\ \begin{bmatrix}a_{22}&b_{22}\\ c_{22}&d_{22}\\ \end{bmatrix} (11)
Original figure
(a) Combination of two subsystems.
Original figure
(b) Hierarchical combination

Original figure
(c) Feedback combination
Fig. 4: Combination examples.
Equations 12–13
[a12b12c12d12]=0\begin{bmatrix}a_{12}&b_{12}\\ c_{12}&d_{12}\\ \end{bmatrix}=0 (12)
[a12b12c12d12]=−[a21b21c21d21]T\begin{bmatrix}a_{12}&b_{12}\\ c_{12}&d_{12}\\ \end{bmatrix}=-\begin{bmatrix}a_{21}&b_{21}\\ c_{21}&d_{21}\\ \end{bmatrix}^{T} (13)
Original figure
Fig. 5: Convergence of hierarchical combinations, where 𝜹i=[δ​zi,δ​ai]T\bm{\delta}_{i}=[\delta z_{i},\delta a_{i}]^{T} denotes the differential displacements between two arbitrary trajectories at fixed time tt of the subsystem. When 𝜹i=0\bm{\delta}_{i}=0, all possible trajectories of the iith subsystem converge to its equilibrium. The subsystems converge recursively following their hierarchies, given stable self-feedback-loops and bounded couplings, as the lowest level with nonzero 𝜹\bm{\delta}’s is always independent.

II-D Explicit solution to the hierarchical-type modularity

The stated hierarchical theorem requires a diagonalizable dynamics Jacobian, suitable uniformly positive-definite transformations, independent per-coordinate policies, bounded couplings, and transformations that stop changing after a finite transient. Eigenvector and QR constructions yield linear inequalities on policy derivatives. These assumptions delimit the stability claim.

Equation 14
∂πi∂s1​i​Ri​i+Λi​i<−α,∂πi∂s2​i​Ri​i<−α\displaystyle\frac{\partial\pi^{i}}{\partial s_{1i}}R_{ii}+\Lambda_{ii}<-\alpha,\ \ \ \frac{\partial\pi^{i}}{\partial s_{2i}}R_{ii}<-\alpha (14)
Original figure
Fig. 6: Independent network πi\pi^{i} for each dimension ii.

III Signal composition

Stable latent signals alone do not ensure stable physical motion. The signal-composition map must also have stable dynamics. A weighted combination of position and velocity tracking errors provides the worked example; extension to learned latent signals requires additional derivative constraints.

Equation 15
𝐲˙−𝐟⁡(𝐲,𝐮)=0\displaystyle\mathbf{\dot{y}}-\mathbf{f}(\mathbf{y},\mathbf{u})=0 (15)
𝐠⁡(𝐱,𝐱˙)−𝐲=0\displaystyle\mathbf{g}(\mathbf{x},\mathbf{\dot{x}})-\mathbf{y}=0

IV For unknown environments

Unknown external environments can be handled using a task-space controller that shapes end-effector dynamics and admits an added RL command. Its gains then define the latent model. This avoids modeling every environment detail but still requires a robot model and the controller’s tracking assumptions.

V Implementation and limitations

Implementation modifies the policy architecture and controller interface while retaining the underlying RL update.

V-A Implementation

Use composite tracking errors as state, insert coordinate-transform layers, assign an independent network to each transformed coordinate, and enforce the derived Jacobian inequalities. Transformation quantities may be updated each time step.

V-B Further simplification

For already stable nominal dynamics, the constraints reduce to signs of policy derivatives. With nonsaturated tanh activations and suitable weight signs, an MLP can enforce the required monotonicity. This simplification relies on the architecture and baseline stability.

Equations 16–17
∂πi∂sj​i​Ri​i<0\frac{\partial\pi^{i}}{\partial s_{ji}}R_{ii}<0 (16)
∂πi∂si​1=WlMl−1Wl−1⋯M2W2M1W1,1\displaystyle\frac{\partial\pi^{i}}{\partial s_{i1}}=W_{l}M_{l-1}W_{l-1}\cdots M_{2}W_{2}M_{1}W_{1,1} (17)

V-C Limitations

Computational and expressivity costs remain.

Limited control frequency

Repeated eigenvalue and QR decompositions limit achievable control frequency. Designing latent dynamics with a diagonal Jacobian can reduce this cost.

Loss of optimality

Restricting couplings and signal composition may sacrifice control optimality. A separate planner can potentially compensate, but this is not an optimality guarantee.

VI Example 1: Necessity for robustness and generlization

A simulated two-dimensional peg-contact example compares PPO with its constrained version, C-PPO. Unconstrained training increasingly produces oscillation or drift despite good returns; constraints prevent those observed instabilities and improve tests with changed dynamics and surface parameters.

Original figure
Fig. 7: Intuition. Local optimal policies can contain unstable control because the stability boundaries may not align with the landscape of expected returns.

VII Example 2: Manipulation learning

Simulated Franka Panda peg tasks use a slow TD3 planner and a faster constrained TD3 controller. Compared with HIRO, the proposed implementation also changes planning coordinates and removes relabeling, so its advantage cannot be attributed solely to stability constraints. Separate ablations compare unconstrained neural control and fixed non-neural controllers. [17].

VII-A TD3/C-TD3 vs. HIRO on peg maze.

Peg Maze compares learning and contact behavior with HIRO.

Task setting.

The robot must pass a peg through fixed obstacles under a height-limiting hood to reach a sampled goal.

Experiment

With initial exploration directed toward the maze, the proposed method learns in about one million steps versus two million for HIRO. Contacts below 1 N occur on 99.93% versus 65.88% of sampled steps. The authors attribute benefits to both stable low-level control and easier task-space planning.

VII-B Ablation study: peg-push task

Peg Push tests the role of learned control and contraction constraints.

Task setting

Reaching the hidden target requires opening a sliding lid, entering a narrow slot, and moving an obstacle.

Experiment

Across five seeds, only TD3/C-TD3 learns the task within the reported 12-million-step budget; two runs converge near three million steps, producing large variation. Fixed controllers learn stable but incomplete routes, while unconstrained hierarchical control passes the goal unstably. Soft-contact penetration is an acknowledged simulator limitation.

VIII Concluding Remarks

Contraction-based modularity offers a structured way to combine control analysis and learned policies. The demonstrated benefits concern simulated manipulation under the specified controller and model assumptions.

IX Appendix

The retained result figures show peg-contact stability tests, parameter-shift robustness, maze contact-force distributions, and peg-push learning curves. The subsequent appendix supplies definitions, proofs, and implementation settings.

Refer to caption
(a) Peg touching by PPO.
Refer to caption
(b) Stability: mitigation of oscillation and drift by constraints
Original figure
(c) Generalization and robustness.
Fig. 8: Necessity for robustness and generalization (more results in [6]). (a) 2D peg-touching task. Both PPO and C-PPO can learn the task with similar return levels and with similar data efficiency. (b) Stability. Starting from the policy at Episode 5000, 8000 trajectories are tested every 1k episodes. In PPO, the amount of oscillation and drift is increasing as learning progresses, which implies overfitting, while in C-PPO, the fraction remains zero. (c) Generalization and robustness. The policies at Episode 5000 are tested for 8000 trajectories, with new task parameters with 50%50\% differences in τi\tau_{i} (i=x,zi=x,z), 100%100\% in Ks​u​rK_{sur} and 50%50\% in KiK_{i} (i=1,2i=1,2) compared to the training set. The nominal model from the training set is used in C-PPO. The histogram of tracking errors shows dramatic improvements in generalization in C-PPO.


Refer to caption
(a) Hierarchical RL with stable modular neural control in the low level (TD3/C-TD3).
Refer to caption
(b) Peg maze and peg push.
Original figure
(c) Peg maze: Contact force distributions and the success rates.
Refer to caption
(d) Peg push: Learning curves.
Fig. 9: Effectiveness in manipulation learning (trajectory examples in https://www.youtube.com/watch?v=nFLHwVfPIJw for peg maze and https://www.youtube.com/watch?v=RpGzpZPifUw for peg push.) (a) Hierarchical RL framework of TD3/C-TD3. (b) Peg maze and peg push tasks. Peg maze (top) is a goal reaching task with fixed obstacles both near the goal (randomly sampled in the green zone) and near the robot arm (the transparent hood). Peg push (bottom) is a goal reaching task that requires opening a sliding lid box, inserting into a deep slot, and pushing away the green obstacle to reach the small red spot underneath. (c) Peg maze results: TD3/C-TD3 vs. HIRO (the state-of-the-art data-efficient hierarchical RL). Both TD3/C-TD3 and HIRO can quickly learn the task (∼\sim1M and 2M steps). The contact force distribution are estimated from policies (success rates on top right) at 10k, 200k, 400k, 600k, 800k, and 1M steps for TD3/C-TD3, and at 10k, 500k, 1M, 1.5M, 2M, and 2.3M steps for HIRO. The distribution shows improvements in safe exploration that in TD3/C-TD3, 99.93%99.93\% are with less than 1 N, and 65.88%65.88\% with HIRO. (d) Peg push results: TD3/C-TD3 vs. RL with non-neural controllers (PPO/Control, TD3/Control) vs. TD3/TD3 without stability constraints. Only TD3/C-TD3 successfully learn the task within 12 M steps, with 2 trials learned the task around 3 M steps causing the large variation. TD3/control and PPO/control can learn stable trajectories but haven’t figured out the entire path to the goal. Without stability constraints, TD3/TD3 learns unstable trajectories passing by the goal.

Acknowledgments

Support came from the RIE2020 industry collaboration initiative and the HP–NTU Digital Manufacturing Corporate Lab.

X Appendix

The technical appendix makes the stability assumptions and experimental settings explicit.

X-A Incremental exponential stability

Incremental exponential stability bounds the distance between any two trajectories by an exponentially decaying multiple of their initial separation.

Equation 18
∥ξ1​(t)−ξ2​(t)∥≤C​e−β​t​∥ξ1​(0)−ξ2​(0)∥\lVert\xi_{1}(t)-\xi_{2}(t)\rVert\leq Ce^{-\beta t}\lVert\xi_{1}(0)-\xi_{2}(0)\rVert (18)

X-B Stability theorem for smooth coordinate transformation

The coordinate-transformation theorem requires a uniformly negative symmetric part of the transformed differential dynamics, with margin for time-varying transformation terms. It establishes the paper’s regional convergence claim under these stated conditions.

Equations 19–21
(F1T+F1)≺−[β+m​a​x​(ν+,0)]​I(F_{1}^{T}+F_{1})\prec-[\beta+max(\nu^{+},0)]I (19)
F1=[Ty​∂𝐟∂𝐲​Ty−1Ty​∂𝐟∂𝐮​Ta∂𝝅∂𝐬1​Ty​∂𝐟∂𝐲​Ty−1+∂𝝅∂𝐬2∂𝝅∂𝐬1​Ty​∂𝐟∂𝐮​Ta]F_{1}=\begin{bmatrix}T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}T_{y}^{-1}&T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}\\[8.0pt] \frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}T_{y}^{-1}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}&\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}\end{bmatrix} (20)
F2=[T˙y​Ty−10∂𝝅∂𝐬1​T˙y​Ty−10]F_{2}=\begin{bmatrix}\dot{T}_{y}T_{y}^{-1}&0\\[8.0pt] \frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\dot{T}_{y}T_{y}^{-1}&0\end{bmatrix} (21)

Differential dynamics in the auxiliary space

The transformed differential matrix includes plant derivatives, policy input gradients, and the derivative of the state transformation.

Equations 22–24
[δ​𝐳˙δ​𝐚˙]=F​[δ​𝐳δ​𝐚]\begin{bmatrix}\delta\dot{\mathbf{z}}\\ \delta\dot{\mathbf{a}}\end{bmatrix}=F\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix} (22)
F=[[T˙y+Ty​∂𝐟∂𝐲]​Ty−1Ty​∂𝐟∂𝐮​TaF21∂𝝅∂𝐬1​Ty​∂𝐟∂𝐮​Ta]F=\begin{bmatrix}[\dot{T}_{y}+T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}]T_{y}^{-1}&T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}\\[8.0pt] F_{21}&\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}\end{bmatrix} (23)
F21=∂𝝅∂𝐬1​(T˙y+Ty​∂𝐟∂𝐲)​Ty−1+∂𝝅∂𝐬2F_{21}=\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}(\dot{T}_{y}+T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}})T_{y}^{-1}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}} (24)

Sufficient condition that makes the squared distance between trajectories exponentially converge in the auxiliary space

Differentiating squared auxiliary distance gives an exponential bound when the symmetric part of the differential matrix is uniformly negative.

Equations 25–26
dd​t​([δ​𝐳δ​𝐚]T​[δ​𝐳δ​𝐚])=2​[δ​𝐳δ​𝐚]T​dd​t​[δ​𝐳δ​𝐚]\displaystyle\frac{d}{dt}\Big(\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}^{T}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}\Big)=2\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}^{T}\frac{d}{dt}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix} (25)
=\displaystyle= 2​[δ​𝐳δ​𝐚]T​F​[δ​𝐳δ​𝐚]≤2​λm​a​x​(F)​([δ​𝐳δ​𝐚]T​[δ​𝐳δ​𝐚])\displaystyle 2\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}^{T}F\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}\leq 2\lambda_{max}(F)\Big(\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}^{T}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix}\Big)
(δ​𝐳)T​(δ​𝐳)+(δ​𝐚)T​(δ​𝐚)\displaystyle(\delta\mathbf{z})^{T}(\delta\mathbf{z})+(\delta\mathbf{a})^{T}(\delta\mathbf{a}) (26)
≤\displaystyle\leq e∫0t2​λm​a​x​(F)​𝑑t​[(δ​𝐳0)T​(δ​𝐳0)+(δ​𝐚0)T​(δ​𝐚0)]\displaystyle e^{\int_{0}^{t}2\lambda_{max}(F)dt}[(\delta\mathbf{z}_{0})^{T}(\delta\mathbf{z}_{0})+(\delta\mathbf{a}_{0})^{T}(\delta\mathbf{a}_{0})]
=\displaystyle= e−2​β​t​[(δ​𝐳0)T​(δ​𝐳0)+(δ​𝐚0)T​(δ​𝐚0)]\displaystyle e^{-2\beta t}[(\delta\mathbf{z}_{0})^{T}(\delta\mathbf{z}_{0})+(\delta\mathbf{a}_{0})^{T}(\delta\mathbf{a}_{0})]

Distance between two arbitrary trajectories

A lower bound on the induced metric relates auxiliary path length to separation in the original coordinates, yielding a transient multiplier and exponential rate.

Equation 27
‖ξ1​(t)−ξ0​(t)‖=∫ξ0ξ1‖δ​𝐲‖\displaystyle\|\xi_{1}(t)-\xi_{0}(t)\|=\int_{\xi_{0}}^{\xi_{1}}\|\delta\mathbf{y}\| (27)
=\displaystyle= ∫ξ0ξ1(δ​𝐲)T​m¯​I​δ​𝐲m¯\displaystyle\int_{\xi_{0}}^{\xi_{1}}\frac{\sqrt{(\delta\mathbf{y})^{T}\underline{m}I\delta\mathbf{y}}}{\sqrt{\underline{m}}}
≤\displaystyle\leq ∫ξ0ξ1(δ​𝐲)T​TyT​Ty​δ​𝐲m¯\displaystyle\int_{\xi_{0}}^{\xi_{1}}\frac{\sqrt{(\delta\mathbf{y})^{T}T_{y}^{T}T_{y}\delta\mathbf{y}}}{\sqrt{\underline{m}}}
≤\displaystyle\leq ∫ξ0ξ1(δ​𝐲)T​TyT​Ty​δ​𝐲+(δ​𝐮)T​(Ta−1)T​Ta−1​(δ​𝐮)m¯\displaystyle\int_{\xi_{0}}^{\xi_{1}}\frac{\sqrt{(\delta\mathbf{y})^{T}T_{y}^{T}T_{y}\delta\mathbf{y}+(\delta\mathbf{u})^{T}(T_{a}^{-1})^{T}T_{a}^{-1}(\delta\mathbf{u})}}{\sqrt{\underline{m}}}
=\displaystyle= ∫ξ0ξ1[(δ​𝐳)T​(δ​𝐳)+(δ​𝐚)T​(δ​𝐚)]m¯\displaystyle\int_{\xi_{0}}^{\xi_{1}}\sqrt{\frac{[(\delta\mathbf{z})^{T}(\delta\mathbf{z})+(\delta\mathbf{a})^{T}(\delta\mathbf{a})]}{\underline{m}}}
≤\displaystyle\leq ∫ξ0ξ1e−2​β​t​[(δ​𝐳0)T​(δ​𝐳0)+(δ​𝐚0)T​(δ​𝐚0)]m¯\displaystyle\int_{\xi_{0}}^{\xi_{1}}\sqrt{\frac{e^{-2\beta t}[(\delta\mathbf{z}_{0})^{T}(\delta\mathbf{z}_{0})+(\delta\mathbf{a}_{0})^{T}(\delta\mathbf{a}_{0})]}{\underline{m}}}
=\displaystyle= C​e−β​t​∥ξ1​(0)−ξ2​(0)∥\displaystyle Ce^{-\beta t}\lVert\xi_{1}(0)-\xi_{2}(0)\rVert

Boundary for the constant part in F

The time-varying matrix contribution is bounded separately; the nominal contraction margin must dominate its largest symmetric-part eigenvalue.

Equation 28
λm​a​x​(F)<\displaystyle\lambda_{max}(F)< λm​a​x​(F1)+λm​a​x​(F2)\displaystyle\lambda_{max}(F_{1})+\lambda_{max}(F_{2}) (28)
<\displaystyle< −[β+max⁡(ν+,0)]+ν+\displaystyle-[\beta+\max(\nu^{+},0)]+\nu^{+}
≤\displaystyle\leq −β\displaystyle-\beta

X-C Proof of the theorem for the hierarchical combination.

The hierarchical construction simplifies these conditions using triangular interconnections.

X-C 1 Assumption that the term involving F 2 in Eq. ( 21 ) can be seen as bounded temporary disturbances

The proof assumes transformation changes end after a finite time and trajectory separation remains bounded before then. Arbitrarily persistent transformation variation is outside this simplification.

X-C 2 Differential dynamics in the auxiliary space

After the transient, the reduced differential system applies exactly. Before it, the bounded separation is absorbed into the exponential bound’s prefactor.

Equation 29
[δ​𝐳˙δ​𝐚˙]=F1​[δ​𝐳δ​𝐚]\begin{bmatrix}\delta\dot{\mathbf{z}}\\ \delta\dot{\mathbf{a}}\end{bmatrix}=F_{1}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix} (29)

X-C 3 Diagonal Λ via T y

A coordinate basis diagonalizes the latent-state Jacobian when the required eigendecomposition exists.

Equation 30
Λ=Ty​∂𝐟∂𝐲​Ty−1=[Λ11Λ22⋱Λn​n]\Lambda=T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}T_{y}^{-1}=\begin{bmatrix}\Lambda_{11}&&&\\ &\Lambda_{22}&&\\ &&\ddots&\\ &&&\Lambda_{nn}\end{bmatrix} (30)

X-C 4 Upper triangular R via T a

A QR factorization and reversal permutation construct an action transformation with upper-triangular input coupling.

Equations 31–33
R=Ty​∂𝐟∂𝐮​Ta=[R11R12⋯R1​nR22R2​n⋱Rn​n]R=T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}=\begin{bmatrix}R_{11}&R_{12}&\cdots&R_{1n}\\ &R_{22}&&R_{2n}\\ &&\ddots&\vdots\\ &&&R_{nn}\end{bmatrix} (31)
R=\displaystyle R= Ty​∂𝐟∂𝐮​Ta\displaystyle T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a} (32)
=\displaystyle= (P​P)​Ty​∂𝐟∂𝐮​(P​QT)−1\displaystyle(PP)T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}(PQ^{T})^{-1}
=\displaystyle= P​[(Ty​∂𝐟∂𝐮)T​P]T​(P​QT)−1\displaystyle P[(T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}})^{T}P]^{T}(PQ^{T})^{-1}
=\displaystyle= P​(Q​Rq​r)T​(QT)−1​P\displaystyle P(QR_{qr})^{T}(Q^{T})^{-1}P
=\displaystyle= P​Rq​rT​QT​(QT)−1​P\displaystyle PR_{qr}^{T}Q^{T}(Q^{T})^{-1}P
=\displaystyle= P​Rq​rT​P\displaystyle PR_{qr}^{T}P
P=[1⋰1]P=\begin{bmatrix}&&1\\ &\iddots&\\ 1&&\end{bmatrix} (33)

X-C 5 Hierarchical combination of subsystems

Independent coordinate policies preserve this triangular structure. Regrouping state and action displacements exposes a hierarchy of two-dimensional modules.

Equations 34–39
[δ​𝐳˙δ​𝐚˙]\displaystyle\begin{bmatrix}\delta\dot{\mathbf{z}}\\ \delta\dot{\mathbf{a}}\end{bmatrix} =[ΛR∂𝝅∂𝐬1​Λ+∂𝝅∂𝐬2∂𝝅∂𝐬1​R]​[δ​𝐳δ​𝐚]\displaystyle=\begin{bmatrix}\Lambda&R\\[8.0pt] \frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\Lambda+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}&\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}R\end{bmatrix}\begin{bmatrix}\delta\mathbf{z}\\ \delta\mathbf{a}\end{bmatrix} (34)
Λ\displaystyle\Lambda =Ty​∂𝐟∂𝐲​Ty−1=[Λ11Λ22⋱Λn​n]\displaystyle=T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}T_{y}^{-1}=\begin{bmatrix}\Lambda_{11}&&&\\ &\Lambda_{22}&&\\ &&\ddots&\\ &&&\Lambda_{nn}\end{bmatrix} (35)
R\displaystyle R =Ty​∂𝐟∂𝐮​Ta=[R11R12⋯R1​nR22R2​n⋱Rn​n]\displaystyle=T_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}T_{a}=\begin{bmatrix}R_{11}&R_{12}&\cdots&R_{1n}\\ &R_{22}&&R_{2n}\\ &&\ddots&\vdots\\ &&&R_{nn}\end{bmatrix}
[δ​z˙1δ​z˙2δ​z˙nδ​a˙1δ​a˙2δ​a˙n]=[F1​lF1​r]​[δ​z1δ​z2δ​znδ​a1δ​a2δ​an]\begin{bmatrix}\delta\dot{z}_{1}\\ \delta\dot{z}_{2}\\ \vdots\\ \delta\dot{z}_{n}\\ \delta\dot{a}_{1}\\ \delta\dot{a}_{2}\\ \vdots\\ \delta\dot{a}_{n}\end{bmatrix}=\begin{bmatrix}F_{1l}&F_{1r}\end{bmatrix}\begin{bmatrix}\delta z_{1}\\ \delta z_{2}\\ \vdots\\ \delta z_{n}\\ \delta a_{1}\\ \delta a_{2}\\ \vdots\\ \delta a_{n}\end{bmatrix} (36)
F1​l=[Λ11⋱Λn​n∂π1∂s11​Λ11+∂π1∂s21⋱∂πn∂s1​n​Λn​n+∂πn∂s2​n]\displaystyle F_{1l}=\begin{bmatrix}\Lambda_{11}&&\\ &\ddots&\\ &&\Lambda_{nn}\\[10.0pt] \frac{\partial\pi^{1}}{\partial s_{11}}\Lambda_{11}+\frac{\partial\pi^{1}}{\partial s_{21}}&&\\ &\ddots&\\ &&\frac{\partial\pi^{n}}{\partial s_{1n}}\Lambda_{nn}+\frac{\partial\pi^{n}}{\partial s_{2n}}\\ \end{bmatrix} (37)
F1​r=[R11R12⋯R1​nR22R2​n⋱Rn​n∂π1∂s11​R11∂π1∂s11​R12⋯∂π1∂s11​R1​n∂π2∂s12​R22∂π2∂s12​R2​n⋱∂πn∂s1​n​Rn​n]\displaystyle F_{1r}=\begin{bmatrix}R_{11}&R_{12}&\cdots&R_{1n}\\ &R_{22}&&R_{2n}\\ &&\ddots&\vdots\\ &&&R_{nn}\\[10.0pt] \frac{\partial\pi^{1}}{\partial s_{11}}R_{11}&\frac{\partial\pi^{1}}{\partial s_{11}}R_{12}&\cdots&\frac{\partial\pi^{1}}{\partial s_{11}}R_{1n}\\ &\frac{\partial\pi^{2}}{\partial s_{12}}R_{22}&&\frac{\partial\pi^{2}}{\partial s_{12}}R_{2n}\\ &&\ddots&\vdots\\ &&&\frac{\partial\pi^{n}}{\partial s_{1n}}R_{nn}\\ \end{bmatrix}
[δ​z˙1δ​a˙1δ​z˙2δ​a˙2δ​z˙nδ​a˙n]=\displaystyle\begin{bmatrix}\delta\dot{z}_{1}\\ \delta\dot{a}_{1}\\ \delta\dot{z}_{2}\\ \delta\dot{a}_{2}\\ \vdots\\ \delta\dot{z}_{n}\\ \delta\dot{a}_{n}\end{bmatrix}= [F11F12⋯F1​n0F22F2​n⋱0⋯0Fn​n]​[δ​z1δ​a1δ​z2δ​a2δ​znδ​an]\displaystyle\begin{bmatrix}F_{11}&F_{12}&\cdots&F_{1n}\\ 0&F_{22}&&F_{2n}\\ \vdots&&\ddots&\vdots\\ 0&\cdots&0&F_{nn}\end{bmatrix}\begin{bmatrix}\delta z_{1}\\ \delta a_{1}\\ \delta z_{2}\\ \delta a_{2}\\ \vdots\\ \delta z_{n}\\ \delta a_{n}\end{bmatrix} (38)
Fi​i\displaystyle F_{ii} =[Λi​iRi​i∂πi∂s1​i​Λi​i+∂πi∂s2​i∂πi∂s1​i​Ri​i]\displaystyle=\begin{bmatrix}\Lambda_{ii}&R_{ii}\\ \frac{\partial\pi^{i}}{\partial s_{1i}}\Lambda_{ii}+\frac{\partial\pi^{i}}{\partial s_{2i}}&\frac{\partial\pi^{i}}{\partial s_{1i}}R_{ii}\end{bmatrix} (39)
Fi​j\displaystyle F_{ij} =[0Ri​j0∂πi∂s1​i​Ri​j],(i<j)\displaystyle=\begin{bmatrix}0&R_{ij}\\ 0&\frac{\partial\pi^{i}}{\partial s_{1i}}R_{ij}\end{bmatrix},\ (i<j)

X-C 6 Linear constraints from the preservation of contraction

The paper derives derivative inequalities from each module’s characteristic polynomial and uses its hierarchical contraction argument to obtain the stated stability condition. This is a sufficient-design construction under the earlier assumptions, not an unrestricted test for nonlinear feedback stability.

Equations 40–43
[δ​z˙iδ​a˙i]=Fi​i​[δ​ziδ​ai]\begin{bmatrix}\delta\dot{z}_{i}\\ \delta\dot{a}_{i}\end{bmatrix}=F_{ii}\begin{bmatrix}\delta z_{i}\\ \delta a_{i}\end{bmatrix} (40)
[δ​ziδ​ai]=c1​𝐯1​e−λ1​t+c2​𝐯2​e−λ2​t\begin{bmatrix}\delta z_{i}\\ \delta a_{i}\end{bmatrix}=c_{1}\mathbf{v}_{1}e^{-\lambda_{1}t}+c_{2}\mathbf{v}_{2}e^{-\lambda_{2}t} (41)
s2−(Λi​i+∂πi∂s1​i​Ri​i)​s−Ri​i​∂πi∂s2​i=0s^{2}-(\Lambda_{ii}+\frac{\partial\pi^{i}}{\partial s_{1i}}R_{ii})s-R_{ii}\frac{\partial\pi^{i}}{\partial s_{2i}}=0 (42)
∂πi∂s1​i​Ri​i+Λi​i<−α,∂πi∂s2​i​Ri​i<−α\displaystyle\frac{\partial\pi^{i}}{\partial s_{1i}}R_{ii}+\Lambda_{ii}<-\alpha,\ \ \ \frac{\partial\pi^{i}}{\partial s_{2i}}R_{ii}<-\alpha (43)

X-D Integration of composite variables and task space controllers for unknown environments

Task-space control supplies a more tractable latent model when the external environment is not explicitly known.

X-D 1 Reference 𝐩 ⁡ ( 𝐱 , 𝐮 ) in task space control

Rigid-body compensation and a Jacobian-based controller shape end-effector errors into a second-order system with an added learned input.

Equations 44–49
H⁡(𝐪)​𝐪¨+C⁡(𝐪,𝐪˙)​𝐪˙+𝐠⁡(𝐪)=𝝉H(\mathbf{q})\ddot{\mathbf{q}}+C(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}+\mathbf{g}(\mathbf{q})=\bm{\tau} (44)
𝐱\displaystyle\mathbf{x} =𝐡⁡(𝐪)\displaystyle=\mathbf{h}(\mathbf{q}) (45)
𝐱˙\displaystyle\dot{\mathbf{x}} =J⁡(𝐪)​𝐪˙\displaystyle=J(\mathbf{q})\dot{\mathbf{q}}
𝐱¨\displaystyle\ddot{\mathbf{x}} =J​(𝐪)​𝐪¨+J˙​(𝐪)​𝐪˙\displaystyle=J(\mathbf{q})\ddot{\mathbf{q}}+\dot{J}(\mathbf{q})\dot{\mathbf{q}}
𝝉=\displaystyle\bm{\tau}= H⁡(𝐪)​J†​𝐩​(𝐱,𝐮)\displaystyle H(\mathbf{q})J^{\dagger}\mathbf{p}(\mathbf{x},\mathbf{u}) (46)
+H⁡(𝐪)​J†​(𝐱¨d−J˙​(𝐪)​𝐪˙)\displaystyle+H(\mathbf{q})J^{\dagger}(\ddot{\mathbf{x}}_{d}-\dot{J}(\mathbf{q})\dot{\mathbf{q}})
+C⁡(𝐪,𝐪˙)​𝐪˙+𝐠⁡(𝐪)\displaystyle+C(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}+\mathbf{g}(\mathbf{q})
𝐱¨d−𝐱¨+𝐩⁡(𝐱,𝐮)=0\mathbf{\ddot{x}}_{d}-\mathbf{\ddot{x}}+\mathbf{p}(\mathbf{x},\mathbf{u})=0 (47)
𝐩⁡(𝐱,𝐮)=Λd−1​Kd​(𝐱˙d−𝐱˙)+Λd−1​Kp​(𝐱d−𝐱)+Λd−1​𝐮\mathbf{p}(\mathbf{x},\mathbf{u})=\Lambda_{d}^{-1}K_{d}(\mathbf{\dot{x}}_{d}-\mathbf{\dot{x}})+\Lambda_{d}^{-1}K_{p}(\mathbf{x}_{d}-\mathbf{x})+\Lambda_{d}^{-1}\mathbf{u} (48)
Λd​𝐞¨+Kd​𝐞˙+Kp​𝐞+𝐮=0\Lambda_{d}\mathbf{\ddot{e}}+K_{d}\mathbf{\dot{e}}+K_{p}\mathbf{e}+\mathbf{u}=0 (49)

X-D 2 Composite variables for a given 𝐩 ⁡ ( 𝐱 , 𝐮 )

Combining position and velocity errors yields first-order latent dynamics. The displayed gain solution requires a nonnegative discriminant in each controlled dimension; a zero position gain instead permits a velocity-error signal.

Equations 50–54
𝐲=K1​𝐞+K2​𝐞˙\mathbf{y}=K_{1}\mathbf{e}+K_{2}\dot{\mathbf{e}} (50)
A​𝐲˙+B​𝐲+𝐮=0A\dot{\mathbf{y}}+B\mathbf{y}+\mathbf{u}=0 (51)
Λd​𝐲˙+Kd​𝐲+𝐮=0\Lambda_{d}\dot{\mathbf{y}}+K_{d}\mathbf{y}+\mathbf{u}=0 (52)
K1,i=Kp,iλd,i,K2,i=2​Kp,iKd,i±Kd,i2−4​Kp,i​λd,iK_{1,i}=\frac{K_{p,i}}{\lambda_{d,i}},\ \ \ \ K_{2,i}=\frac{2K_{p,i}}{K_{d,i}\pm\sqrt{K_{d,i}^{2}-4K_{p,i}\lambda_{d,i}}} (53)
Ai​i=λd,i2​Kp,i​(Kd,i±Kd,i2−4​Kp,i​λd,i),Bi​i=λd,iA_{ii}=\frac{\lambda_{d,i}}{2K_{p,i}}(K_{d,i}\pm\sqrt{K_{d,i}^{2}-4K_{p,i}\lambda_{d,i}}),B_{ii}=\lambda_{d,i} (54)

X-D 3 Stability constraints

Positive diagonal latent coefficients allow identity transformations and reduce the proposed inequalities to positive policy input gradients.

Equation 55
−∂πi∂s1​i​Ai​i−1−Ai​i−1​Bi​i<−α,−∂πi∂s2​i​Ai​i−1<−α\displaystyle-\frac{\partial\pi^{i}}{\partial s_{1i}}A^{-1}_{ii}-A^{-1}_{ii}B_{ii}<-\alpha,\ \ \ -\frac{\partial\pi^{i}}{\partial s_{2i}}A^{-1}_{ii}<-\alpha (55)

X-E Robustness

Two robustness analyses bound disturbances and model mismatch.

X-E 1 Robustness to unknown deterministic perturbation

For a contracting system with bounded metric and bounded disturbances, trajectory error approaches a ball whose size grows with disturbance magnitude and shrinks with contraction rate. Applying this result to the augmented state also requires bounded transformations and network gradients.

Equations 56–61
∥ξ1​(t)−ξ0​(t)∥≤Vl​(0)m¯​e−β​t+d¯β​m¯m¯​(1−e−β​t)\lVert\xi_{1}(t)-\xi_{0}(t)\rVert\leq\frac{V_{l}(0)}{\sqrt{\underline{m}}}e^{-\beta t}+\frac{\bar{d}}{\beta}\sqrt{\frac{\bar{m}}{\underline{m}}}(1-e^{-\beta t}) (56)
𝐲˙\displaystyle\dot{\mathbf{y}} =𝐟⁡(𝐲,𝐮,t)+𝐝⁡(𝐲,𝐮,t)\displaystyle=\mathbf{f}(\mathbf{y},\mathbf{u},t)+\mathbf{d}(\mathbf{y},\mathbf{u},t) (57)
𝐮\displaystyle\mathbf{u} =Ta​𝝅​(𝐬1,𝐬2)=Ta​𝝅​(𝐳,∫𝐳​dt)\displaystyle=T_{a}\bm{\pi}(\mathbf{s}_{1},\mathbf{s}_{2})=T_{a}\bm{\pi}(\mathbf{z},\smallint\mathbf{z}dt)
𝐲¯˙=𝐟¯​(𝐲¯,t)+𝐝¯\mathbf{\dot{\bar{y}}}=\mathbf{\bar{f}}(\mathbf{\bar{y}},t)+\mathbf{\bar{d}} (58)
𝐟¯​(𝐲¯,t)\displaystyle\mathbf{\bar{f}}(\mathbf{\bar{y}},t) (59)
=\displaystyle= [𝐟⁡(𝐲,𝐮,t)T˙a​𝝅+(Ta​∂𝝅∂𝐬1​T˙y+Ta​∂𝝅∂𝐬2​Ty)​𝐲+(Ta​∂𝝅∂𝐬1​Ty)​f​(𝐲,𝐮,t)]\displaystyle\begin{bmatrix}\mathbf{f}(\mathbf{y},\mathbf{u},t)\\ \dot{T}_{a}\bm{\pi}+(T_{a}\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\dot{T}_{y}+T_{a}\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{2}}T_{y})\mathbf{y}+(T_{a}\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y})f(\mathbf{y},\mathbf{u},t)\end{bmatrix}
𝐝¯=[𝐝⁡(𝐲,𝐮,t)(Ta​∂𝝅∂𝐬1​Ty)​𝐝​(𝐲,𝐮,t)]\mathbf{\bar{d}}=\begin{bmatrix}\mathbf{d}(\mathbf{y},\mathbf{u},t)\\ (T_{a}\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}T_{y})\mathbf{d}(\mathbf{y},\mathbf{u},t)\end{bmatrix} (60)
∥ξ1​(t)−ξ2​(t)∥≤C1​e−β​t+C2​sup𝐲,𝐮,t∥ΘM​𝐝¯∥​1−e−β​tβ\displaystyle\lVert\xi_{1}(t)-\xi_{2}(t)\rVert\leq C_{1}e^{-\beta t}+C_{2}\sup_{\mathbf{y},\mathbf{u},t}\lVert\Theta_{M}\mathbf{\bar{d}}\rVert\frac{1-e^{-\beta t}}{\beta} (61)

X-E 2 Robustness to model errors

Model errors preserve the argument when the real transformed symmetric dynamics remain dominated by the nominal stability bound. This is a conditional robustness criterion, not tolerance of arbitrary model error.

Equations 62–63
(F1,r​e​a​lT+F1,r​e​a​l)⪯(F^1T+F^1)≺−[β+m​a​x​(ν+,0)]​I(F_{1,real}^{T}+F_{1,real})\preceq(\hat{F}_{1}^{T}+\hat{F}_{1})\prec-[\beta+max(\nu^{+},0)]I (62)
F1,r​e​a​l\displaystyle F_{1,real} =[T^y​∂𝐟∂𝐲​T^y−1T^y​∂𝐟∂𝐮​T^a∂𝝅∂𝐬1​T^y​∂𝐟∂𝐲​T^y−1+∂𝝅∂𝐬2∂𝝅∂𝐬1​T^y​∂𝐟∂𝐮​T^a]\displaystyle=\begin{bmatrix}\hat{T}_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}\hat{T}_{y}^{-1}&\hat{T}_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}\hat{T}_{a}\\[8.0pt] \frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\hat{T}_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{y}}\hat{T}_{y}^{-1}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}&\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\hat{T}_{y}\frac{\partial\mathbf{f}}{\partial\mathbf{u}}\hat{T}_{a}\end{bmatrix} (63)
F^1\displaystyle\hat{F}_{1} =[T^y​∂𝐟^∂𝐲​T^y−1T^y​∂𝐟^∂𝐮​T^a∂𝝅∂𝐬1​T^y​∂𝐟^∂𝐲​T^y−1+∂𝝅∂𝐬2∂𝝅∂𝐬1​T^y​∂𝐟^∂𝐮​T^a]\displaystyle=\begin{bmatrix}\hat{T}_{y}\frac{\partial\hat{\mathbf{f}}}{\partial\mathbf{y}}\hat{T}_{y}^{-1}&\hat{T}_{y}\frac{\partial\hat{\mathbf{f}}}{\partial\mathbf{u}}\hat{T}_{a}\\[8.0pt] \frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\hat{T}_{y}\frac{\partial\hat{\mathbf{f}}}{\partial\mathbf{y}}\hat{T}_{y}^{-1}+\frac{\partial\bm{\pi}}{{\partial\mathbf{s}_{2}}}&\frac{\partial\bm{\pi}}{\partial\mathbf{s}_{1}}\hat{T}_{y}\frac{\partial\hat{\mathbf{f}}}{\partial\mathbf{u}}\hat{T}_{a}\end{bmatrix}

X-F Implementation in the PPO example

The PPO example uses first-order peg dynamics and a random sinusoidal surface with sampled stiffness and goals.

Equations 64–65
τx​x˙+x=ux,τz​z˙+z=uz\tau_{x}\dot{x}+x=u_{x},\ \ \tau_{z}\dot{z}+z=u_{z} (64)
f=Ks​u​r​min⁡(z−g⁡(x), 0)f=K_{sur}\min{(z-g(x),\ 0)} (65)

Computation of T y , T a , Λ i ​ i , and R i ​ i

Contact dynamics are expressed in horizontal position and force. Approximate plant derivatives determine the transformations and constraint coefficients at each step.

Equations 66–68
x˙=\displaystyle\dot{x}= −1τx​x+1τx​ux\displaystyle-\frac{1}{\tau_{x}}x+\frac{1}{\tau_{x}}u_{x} (66)
f˙=\displaystyle\dot{f}= Ks​u​r​(z˙−d​gd​x​x˙)\displaystyle K_{sur}(\dot{z}-\frac{dg}{dx}\dot{x})
=\displaystyle= −1τz​f+[Ks​u​rτx​d​gd​x​x−Ks​u​rτz​g​(x)]\displaystyle-\frac{1}{\tau_{z}}f+[\frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}x-\frac{K_{sur}}{\tau_{z}}g(x)]
−Ks​u​rτx​d​gd​x​ux+Ks​u​rτz​uz\displaystyle-\frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}u_{x}+\frac{K_{sur}}{\tau_{z}}u_{z}
𝐱˙=dd​t​[xf]=\displaystyle\dot{\mathbf{x}}=\frac{d}{dt}\begin{bmatrix}x\\ f\end{bmatrix}= [−1τx​x[Ks​u​rτx​d​gd​x​x−Ks​u​rτz​g​(x)]−1τz​f]\displaystyle\begin{bmatrix}-\frac{1}{\tau_{x}}x\\[7.0pt] [\frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}x-\frac{K_{sur}}{\tau_{z}}g(x)]-\frac{1}{\tau_{z}}f\end{bmatrix} (67)
+[1τx0−Ks​u​rτx​d​gd​xKs​u​rτz]​𝐮\displaystyle+\begin{bmatrix}\frac{1}{\tau_{x}}&0\\[7.0pt] -\frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}&\frac{K_{sur}}{\tau_{z}}\end{bmatrix}\mathbf{u}
∂𝐟∂𝐱\displaystyle\frac{\partial\mathbf{f}}{\partial\mathbf{x}} ≈[−1τx0Ks​u​rτx​d​gd​x−Ks​u​rτz​g⁡(x)x−1τz]\displaystyle\approx\begin{bmatrix}-\frac{1}{\tau_{x}}&0\\[7.0pt] \frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}-\frac{K_{sur}}{\tau_{z}}\frac{g(x)}{x}&-\frac{1}{\tau_{z}}\end{bmatrix} (68)
∂𝐟∂𝐮\displaystyle\frac{\partial\mathbf{f}}{\partial\mathbf{u}} =[1τx0−Ks​u​rτx​d​gd​xKs​u​rτz]\displaystyle=\begin{bmatrix}\frac{1}{\tau_{x}}&0\\[7.0pt] -\frac{K_{sur}}{\tau_{x}}\frac{dg}{dx}&\frac{K_{sur}}{\tau_{z}}\end{bmatrix}

Simplification into constraints on MLP weights

Three-layer tanh networks use positive weights and a conditional output sign flip to obtain the required derivative signs. The retained tables provide maze hyperparameters, peg-push geometry, controller gains, and TD3 settings.

TABLE I: Hyperparameters in Peg Maze
Parameter TD3/C-TD3 HIRO
Gaussian noise std HL:1.0, LL:0.2 HL:1.0, LL:1.0
gradient clipping 0.1 0.1
target update period 100 100
train steps HL:100, LL:200 HL:100, LL:200
TABLE II: Peg Push Task Settings
Geometry Length (y-axis) Width (x-axis) Depth (z-axis)
Box 36 mm 36 mm 20 mm
Sliding cover 12 mm 36 mm 4 mm
Slot - 4 mm 7 mm
Obstacle 2 mm 2 mm 2 mm
Target 1 mm 1 mm 12.5* mm
Peg - radius=0.8 mm 20 mm
TABLE III: Parameters of TD3-based Algorithms
critic learning rate 0.001
target learning rate 0.0001
adam beta1 0.9
adam beta2 0.999
high-level dqda clipping None
low-level dqda clipping 0.1
target q clipping None
target tau 0.005
target update period 50
action regularizer None
observation normalization None
reward scaling 1
high-level discount factor 0.5
low-level discount factor 1
high-level exploration noise 0.1𝒩⁡(0,0.1)\mathcal{N}(0,0.1) mm
low-level exploration noise 𝒩⁡(0,0.1)\mathcal{N}(0,0.1)
batch size 128
buffer size 65536
samples to buffer per update 1024
train step per update 100
Translation Kp 20
Translation Kd 30
Orientation Kp 30
Orientation Kd 40
high-level critic network. (400, 300) with tanh
low-level critic network. (400, 300) with tanh
high-level actor network. (400, 300) with tanh
low-level actor network. (200, 200) with W>0W>0 and tanh

X-G Peg Maze Example

Maze success requires position error below 5 mm. Task-space and joint-space baselines use separately tuned controller and action ranges; these implementation differences matter when interpreting the comparison.

X-H Peg-Push Example

Peg Push uses MuJoCo soft-contact settings, 50 Hz control, a 2.5 Hz planner, and 20-second trajectories. Planner increments are bounded by 5 mm and 0.5 radians; the appendix retains geometry, rewards, gains, and learning settings.

Concept demonstration

Open image ↗