Literature
Display
← All papers

Globally Stable Neural Imitation Policies

Amin Abyaneh · Mariana Sosa Guzmán · Hsiu-Chin Lin

· IEEE International Conference on Robotics and Automation (ICRA 2024) ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

SNDS learns an autonomous velocity policy and a convex Lyapunov candidate together. A projection constrains the policy’s energy change, while a short-horizon trajectory loss improves imitation beyond pointwise velocity fitting. Handwriting experiments and a Kinova arm demonstrate recovery and accurate reproduction with few demonstrations. The paper claims global asymptotic stability; the displayed projection and numerical implementation require care when interpreting that claim.

I Introduction

Behavior cloning can reproduce familiar demonstrations yet behave unpredictably after a perturbation. Stable dynamical-system policies aim to converge to the same target from unfamiliar states, but restrictive model classes and expensive fitting can limit their accuracy and scalability. SNDS combines a neural policy, a learned convex potential, and differentiable training to address those limitations.

The contribution is a policy architecture plus a trajectory-alignment objective, evaluated on planar handwriting and pose trajectories. Convergence to a target should be distinguished from collision avoidance or compliance with robot limits, which this work does not enforce.

Refer to caption
Fig. 1: Overview of the proposed stable neural policy learning method. Policy learning (top) optimizes a Lyapunov-stable neural policy over the expert demonstration data. The optimized policy is then deployed (bottom) to plan globally stable trajectories resistant to unpredictable perturbations.

II Background

The policy is an autonomous differential equation whose output is a state derivative. Training demonstrations share one terminal state.

II-A Preliminaries

A DS maps the current state to velocity without explicit time dependence. Lyapunov analysis relates convergence to a positive-definite potential that decreases along trajectories. An ICNN supplies a convex function with architectural weight and activation restrictions; this potential is learned alongside the motion field.

Equations
∀𝐱0∈𝒳,𝐱t=πθ​(𝐱t−1,𝐚),limt→∞𝐱t=𝐱∗\forall\mathbf{{x}}_{0}\in\mathcal{X},\;\;\mathbf{{x}}_{t}=\pi_{\theta}(\mathbf{{x}}_{t-1},\mathbf{a}),\;\;\lim_{t\rightarrow\infty}\;\mathbf{{x}}_{t}=\mathbf{{x}}^{*}
v^​(𝐱)=zL,zl+1=σl​(Ul​x+Wl​zl+bl),\displaystyle\hat{v}(\mathbf{{x}})=z_{L},\quad z_{l+1}=\sigma_{l}(U_{l}x+W_{l}z_{l}+b_{l}),

II-B Problem statement

Training data contain sampled position–velocity pairs from several demonstrations, all ending at the same target with zero terminal velocity. The objective is a deterministic state-to-derivative mapping that reproduces these motions and generalizes away from them. Rollouts use forward Euler integration, so the simulation step size also affects the realized trajectory.

Equations 1–2
𝐃≜{(𝐱d[s],𝐚d[s])|d∈[Nd],s∈[Ns]},\displaystyle\mathbf{D}\triangleq\Bigl\{\big(\mathbf{{x}}^{d}[s],\;{\mathbf{a}}^{d}[s]\big)\;\big|\;d\in[N_{d}],\;s\in[N_{s}]\Bigr\}, (1)
πθ​(a∣x),πθ:𝒳→𝒜,s.t.,a=∂x∂t\displaystyle\pi_{\theta}(\textbf{a}\mid\textbf{x}),\;\;\pi_{\theta}:\mathcal{X}\xrightarrow{}\mathcal{A},\;\text{s.t.},\;\textbf{a}=\frac{\partial\textbf{x}}{\partial t} (2)

III Methodology

The method builds a projected neural vector field, analyzes its Lyapunov behavior, and trains it using velocity and short-horizon position errors.

III-A Dynamical system policy formulation

An unrestricted feedforward network proposes a velocity. A second network defines the potential by subtracting its target value and adding a positive quadratic term. Equation (4) removes the component of the proposed velocity that violates the prescribed Lyapunov half-space. Automatic differentiation permits joint optimization.

The paper uses ReLU for this projection and relaxes an earlier exponentially stable formulation. The exact displayed equations are preserved because the distinction between nonincreasing and strictly decreasing potential matters to the subsequent guarantee.

Equations 3–4
v⁡(𝐱)=[v^​(𝐱)−v^​(𝐱∗)]+δ​‖𝐱−𝐱∗‖22,\displaystyle v(\mathbf{{x}})=\big[\hat{v}(\mathbf{{x}})-\hat{v}(\mathbf{{x}}^{*})\big]+\delta||\mathbf{{x}}-\mathbf{{x}}^{*}||_{2}^{2}, (3)
πθ​(𝐱)=π^​(𝐱)−∇v​(𝐱)​σ(∇v(𝐱)Tπ^(𝐱))∥∇v​(𝐱)∥22.\displaystyle\pi_{\theta}(\mathbf{{x}})=\hat{\pi}(\mathbf{{x}})-\nabla v(\mathbf{{x}})\frac{\sigma(\nabla v(\mathbf{{x}})^{T}\hat{\pi}(\mathbf{{x}}))}{\lVert\nabla v(\mathbf{{x}})\rVert_{2}^{2}}. (4)
Refer to caption
Fig. 2: An example of unstable (left) vs. a stable (right) policies optimized on expert’s data from the handwriting dataset [24]. While both policy rollouts can reproduce the expert motion, an unstable policy cannot recover from perturbation that push the robot to unknown state space regions.

III-B Global asymptotic stability guarantees

Proposition 1 states that the constructed policy is globally asymptotically stable. Its contradiction argument assumes a positive-definite potential and a uniformly negative derivative on a compact set away from the target, then integrates that decrease until it contradicts nonnegativity.

Reading note: with the stated ReLU, equation (4) gives dV/dt = min(∇V·π̂, 0) wherever its denominator is nonzero. This ensures a nonpositive derivative, but does not by itself establish the strict decrease used in the proof. Convexity and subtracting a target value likewise do not alone make that target the minimum of an arbitrary convex network. The convergence claim therefore needs these additional conditions to hold; the empirical recovery results remain separate evidence.

Refer to caption
Fig. 3: Joint training of globally stable neural policies (left) and the corresponding LPF (right) optimized on expert’s data from the handwriting dataset [24]. The contours for the LPF (green) illustrate both positive-definiteness and convexity of the trained function.
Equations
v⁡(𝐱⁡[T])=v⁡(𝐱⁡[0])+∫0Tv˙​(𝐱⁡[t])​𝑑t≤v⁡(𝐱⁡[0])−α​T,\displaystyle v(\mathbf{{x}}[T])=v(\mathbf{{x}}[0])+\int_{0}^{T}\dot{v}(\mathbf{{x}}[t])dt\leq v(\mathbf{{x}}[0])-\alpha T,
v⁡(𝐱⁡[T])≤v⁡(𝐱⁡[0])−α⁡(v⁡(𝐱⁡[0])α+β)=−α​β<0,\displaystyle v(\mathbf{{x}}[T])\leq v(\mathbf{{x}}[0])-\alpha(\frac{v(\mathbf{{x}}[0])}{\alpha}+\beta)=-\alpha\beta<0,

III-C SRVF training loss

Velocity MSE can be small even when integration accumulates substantial position error. The proposed objective combines velocity fitting with discounted short-horizon Euler position errors and SRVF-based trajectory features. A short horizon keeps gradients and memory manageable; the experiments use two steps with weights γᵢ = 2⁻ⁱ. The retained equations give the precise feature transform and loss.

Equations 5–6
SRVF​(𝐱kd)=𝐱kd​[s+Δ​t]−𝐱kd​[s]‖𝐱kd​[s+Δ​t]−𝐱kd​[s]‖,Δ​t=1.\displaystyle\texttt{SRVF}(\mathbf{{x}}^{d}_{k})=\sqrt{\frac{\mathbf{{x}}^{d}_{k}[s+\Delta t]-\mathbf{{x}}^{d}_{k}[s]}{\|\mathbf{{x}}^{d}_{k}[s+\Delta t]-\mathbf{{x}}^{d}_{k}[s]\|}},\quad\Delta t=1. (5)
ℒ⁡(𝐱,θ)=γ0​𝔼[(πθ​(𝐱d​[s])−𝐱˙d​[s])2]+∑i∈[Nw]γi𝔼[(𝐱d​[s+i−1]+πθ​(𝐱d​[s+i−1])​Δ​t−𝐱d​[s+i])2],\displaystyle\begin{aligned} &\mathcal{L}(\mathbf{{x}};\;\theta)=\;\gamma_{0}\mathop{\mathbb{E}}\Big[\big(\pi_{\theta}(\mathbf{{x}}^{d}[s])-\dot{\mathbf{{x}}}^{d}[s]\big)^{2}\Big]\;+\\ \sum_{i\in[N_{w}]}\gamma_{i}\mathop{\mathbb{E}}&\Big[\big(\mathbf{{x}}^{d}[s+i-1]+\pi_{\theta}(\mathbf{{x}}^{d}[s+i-1])\Delta t-\mathbf{{x}}^{d}[s+i]\big)^{2}\Big],\end{aligned} (6)

IV Experiments

Evaluation uses LASA handwriting and a more complex Messy Snake dataset. The latter also supplies the trajectory for simulation and hardware deployment.

IV-A Evaluation

Baselines are ordinary behavior cloning, LPV-DS, and SDS-EF. The experiments use an 80/20 train/test split and ten random seeds. MSE measures velocity prediction, while DTW compares complete generated and demonstrated trajectories; these metrics capture different failure modes. The figures also report learning time.

Refer to caption
Fig. 4: Comparing the reproduction accuracy (top) and computational cost (bottom) of SNDS against baseline methods using the MSE and DTW metrics introduced in Equation 7 and Equation 8, respectively. The accuracy of policy actions and the learning time against other stable methods remain comparably lower for SNDS across most designated trajectories.
Equations 7–8
M​S​E​(𝐱˙g,𝐱˙d)=∑d=1Ndt​e​s​t∑s=1Ns(πθ​(𝐱d​[s])−𝐱˙d​[s])22​Ndt​e​s​t​Ns,MSE(\dot{\mathbf{{x}}}^{g},\dot{\mathbf{{x}}}^{d})=\frac{\sum_{d=1}^{N_{d}^{test}}\sum_{s=1}^{N_{s}}(\pi_{\theta}(\mathbf{{x}}^{d}[s])-\dot{\mathbf{{x}}}^{d}[s])^{2}}{2N_{d}^{test}N_{s}}, (7)
D​T​W​(𝐱g,𝐱d)=minp∈𝒫⁡(𝐱g,𝐱d)⁡(∑(i,j)∈π𝐸𝑢𝑐​(𝐱g​[i],𝐱d​[j])q)1q,DTW(\mathbf{{x}}^{g},\mathbf{{x}}^{d})=\min_{p\in\mathcal{P}({\mathbf{{x}}}^{g},{\mathbf{{x}}}^{d})}\left(\sum_{(i,j)\in\pi}\mathit{Euc}(\mathbf{{x}}^{g}[i],\mathbf{{x}}^{d}[j])^{q}\right)^{\frac{1}{q}}, (8)
Refer to caption
Fig. 5: Policy rollouts for SNDS and other baselines. Notice the highlighted inaccuracies or instabilities (yellow) for other methods. The acquired policies are optimized using the N-shaped data of the handwriting dataset.

IV-B Handwriting dataset policies

SNDS generally provides favorable imitation error and training time on the selected motions, though it does not win every case. Streamlines and perturbed rollouts illustrate recovery outside the demonstrated region. A single-demonstration Sine experiment tests data efficiency against mixture-model and flow-based policies. These comparisons concern the tested implementations and motions, not all possible models in each family.

Refer to caption
Fig. 6: Policy rollouts when training on a single Sine-shaped demonstration of the handwriting data. Inaccuracies or instabilities are highlighted. SNDS performs better owing to the expressive architecture and customized loss.

IV-C SE(3) policy training

The state is extended to position and orientation for pose imitation. The orientation demonstrations are synthetic additions to translational data. The authors evaluate Sine, G, and Messy Snake motions, deploy the learned field in PyBullet, and transfer it to a Kinova Jaco2 arm. The robot reproduces the demonstrated pattern and recovers in the shown trials. This does not establish global guarantees for every orientation representation or low-level robot controller.

Refer to caption
Fig. 7: DTW between policy rollouts and expert’s data for the complex Messy Snake demonstrations in SE(3) (left). SNDS can better extend to higher state space dimensions compared to former baselines. The trained SNDS policy over the expert demonstrations is depicted on the right.

V Discussion

The architecture offers expressive, efficiently trained policies with a learned stability-related structure. Remaining practical gaps include obstacles, joint and torque limits, and physically infeasible generated paths. The authors propose barrier constraints and more flexible Lyapunov representations as future extensions.

VI Conclusion

Joint policy–potential learning and a short-horizon imitation objective produce promising stable-looking rollouts in simulation and on hardware. The method’s formal claims should be read with the conditions discussed above, and kept separate from the demonstrated task performance.

Refer to caption
Fig. 8: Policy deployment in simulation (top) and real-world Kinova Jaco2 arm (bottom) scenarios. The policy is trained on a single Messy Snake demonstration. The simulation rollouts (red) are close to the expert’s demonstration (blue), and are similar to the rollouts in Figure 7.

VII Reproducability

The policy network has widths 2–256–256–128–128–2 with Leaky ReLU activations; the ICNN has widths 2–128–128–128–1 with the specified convex activations. The loss uses a two-step horizon. The selected LASA motions come from a dataset with seven demonstrations per motion; Messy Snake has three demonstrations, with one used for the robot experiment.

The timing comparison restricts computation to a single CPU, while the broader implementation reports an RTX 4050 machine. Architecture, data, and executable experiments are available in the authors’ repository.

Acknowledgment

Anya Forestell is acknowledged for hardware support.

Appendix

The appendix explains trajectory differentiation, practical numerical choices, and additional multimodal handwriting experiments.

VII-A Efficient trajectory generation

Short explicit Euler rollouts are the chosen differentiable training procedure. The appendix also discusses implicit differentiation as a memory-conscious alternative, but reports that explicit steps are faster for the small horizons used here. Long-horizon memory advantages are not an evaluated contribution of this implementation.

Equation
∑i∈[Nw]γi​𝔼[(𝐱d​[s+i−1]+πθ​(𝐱d​[s+i−1])​Δ​t−𝐱d​[s+i])2],\sum_{i\in[N_{w}]}\gamma_{i}\mathop{\mathbb{E}}\Big[\big(\mathbf{{x}}^{d}[s+i-1]+\pi_{\theta}(\mathbf{{x}}^{d}[s+i-1])\Delta t-\mathbf{{x}}^{d}[s+i]\big)^{2}\Big],

VII-B Smoothness and numerical stability

ReLU makes the projection nonsmooth, and a near-zero potential gradient makes its denominator numerically problematic. Suggested remedies include smoother activations, denominator regularization, gradient clipping, and a smoother or Lipschitz-bounded potential network. These changes modify the exact projected field, so the displayed continuous-time argument cannot automatically certify every regularized implementation or Euler step size.

VII-C Additional experiments

Additional handwriting experiments include multiple approach directions toward a shared equilibrium. Figure 9 compares the resulting SNDS, SDS-EF, and behavior-cloning vector fields and shows the learned potential.

Refer to caption
Fig. 9: SNDS is compared against with SDS-EF (stable) and BC (unstable) learning methods. This experiment uses the multimodel motions in the handwriting motions. The trained Lyapunov function is displayed in the last row.
Concept demonstration

Open image ↗