Literature
Display
← All papers

Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning

Urain, Julen · Li, Anqi · Liu, Puze · D'Eramo, Carlo · Peters, Jan

· Robotics: Science and Systems 2021 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

Refer to caption
Fig. 1: Composable Energy Policies generate trajectories reactively in a wide set of environments, as the ones in the figure. The generated trajectories are attracted to a target pose, visualized by the green sphere, and avoid a set of obstacles, shown as red spheres.

At a glance

Composable Energy Policies (CEP) select robot actions by maximizing a product of stochastic component policies. This can retain several acceptable actions for each objective, instead of combining independently chosen deterministic actions that may conflict. An energy tree brings objectives from different task spaces into a common optimization, and a hierarchical variant combines fixed priors with reinforcement learning. Experiments improve reactive reaching and puck-hitting performance, while difficult cages and long-horizon conflicts remain limitations.

I Introduction

Reaching a target, avoiding obstacles, and respecting joint limits must often be satisfied simultaneously. Global planning reasons over trajectories but costs more computation; reactive controllers respond quickly but can oscillate or become trapped. CEP seeks actions jointly favored by all components. Multiplying policy distributions acts like a probabilistic conjunction, allowing multiple acceptable avoidance directions to survive until the objectives are combined. [20] [32]

Equation 1
𝒂∗=arg​max𝐚∏kπk(𝐚|𝐬).\displaystyle{\bm{a}}^{*}=\argmax_{\bm{a}}\prod_{k}\pi_{k}({\bm{a}}|{\bm{s}}). (1)
Refer to caption
Fig. 2: Visual Representation of modular control for Goto + Obstacle Avoidance. In the top box, we show Artificial Potential Field (APF) [20] policy. In the bottom box, we show Composable Energy Policies (CEP). In contrast, with artificial potential field method, that sums deterministic actions (goto, avoid obstacle), CEP computes the product of the policy distributions. The composition will provide high probability to those actions that satisfy both components and low to the rest. This approach helps avoiding conflicts between different components. Robot: blue circle, obstacle: red circle and target: green cross. Thick dotted line: performed trajectory, light dotted line: possible future trajectories.

Contribution

The contribution is a modular energy representation, optimization over its composed policy, and a hierarchical learning interface for priors from different sources. The appendix relates composition to control as inference and explains why locally combined objectives need not recover a globally optimal trajectory. Hard-constraint behavior requires suitable priors and successful optimization; the empirical method is not a general collision-free planning guarantee.

Notation

Superscripts identify the space in which a policy is represented, subscripts identify policy components, and mapping functions transform states and actions between configuration and task spaces.

II Preliminaries

II-A Artificial Potential Fields

Artificial potential fields combine task-space dynamical systems into a robot action through weighted projections. Independently preferred accelerations can conflict. CEP replaces each deterministic proposal with an energy distribution over possible actions, then selects an action from the combined objective. Gaussian component energies recover a form of potential-field composition, while richer energies can represent constraints and multiple suitable directions.

Equation 2
𝒈x=∑k=0K𝑱zk+​𝚲k​(𝒛k)​𝒈zk\displaystyle{\bm{g}}^{x}=\sum_{k=0}^{K}{\bm{J}}^{z_{k}^{+}}{\bm{\Lambda}}_{k}({\bm{z}}_{k}){\bm{g}}^{z_{k}} (2)

III Composable Energy Policies

A Boltzmann policy assigns probability proportional to an exponentiated energy. A weighted product of policies becomes a weighted sum of energies in log space. Normalizing constants independent of the candidate action need not be computed for action selection, and components can be evaluated in parallel.

Equations 3–4
πi​(𝒂|𝒔)=exp⁡(Ei​(𝒂,𝒔))​1Zi​(𝒔)\displaystyle\pi_{i}({\bm{a}}|{\bm{s}})=\exp(E_{i}({\bm{a}},{\bm{s}}))\frac{1}{Z_{i}({\bm{s}})} (3)
π⁡(𝒂|𝒔)=∏k=0Kπk​(𝒂|𝒔)βk∝exp⁡(∑kβk​Ek​(𝒂,𝒔)).\displaystyle\pi({\bm{a}}|{\bm{s}})=\prod_{k=0}^{K}\pi_{k}({\bm{a}}|{\bm{s}})^{\beta_{k}}\propto\,\exp\left(\sum_{k}\beta_{k}E_{k}({\bm{a}},{\bm{s}})\right). (4)

III-A Energy Tree

An energy tree maps a candidate configuration-space action into each component’s task space, evaluates the corresponding energies, and combines them. The resulting controller is implicit: it optimizes a score instead of directly mapping each component’s preferred action back through an inverse. The paper’s density change-of-variables equality requires bijective action maps; its extension to injective or surjective maps is a practical approximation explicitly left for further study.

Equations 5–10
πx​(𝒂x|𝒔x)\displaystyle\pi^{x}({\bm{a}}^{x}|{\bm{s}}^{x}) =πz​(𝒂z|𝒔z)​det(𝑱⊺​𝑱)\displaystyle=\pi^{z}({\bm{a}}^{z}|{\bm{s}}^{z})\sqrt{\det({\bm{J}}^{\intercal}{\bm{J}})}
=πz​(𝒇xz​(𝒔x,𝒂x))​det(𝑱⊺​𝑱).\displaystyle=\pi^{z}({\bm{f}}_{x}^{z}({\bm{s}}^{x},{\bm{a}}^{x}))\sqrt{\det({\bm{J}}^{\intercal}{\bm{J}})}. (5)
πx​(𝒂x|𝒔x)=exp⁡(Ez​(𝒇xz​(𝒂x,𝒔x)))​1Z⁡(𝒔x)\displaystyle\pi^{x}({\bm{a}}^{x}|{\bm{s}}^{x})=\exp\left(E^{z}({\bm{f}}_{x}^{z}({\bm{a}}^{x},{\bm{s}}^{x}))\right)\frac{1}{Z({\bm{s}}^{x})} (6)
Z=∫𝒂zexp⁡(Ez​(𝒂z,𝒔z))​d​𝒂zdet(𝑱⊺​𝑱).\displaystyle Z=\frac{\int_{{\bm{a}}^{z}}\exp(E^{z}({\bm{a}}^{z},{\bm{s}}^{z}))d{\bm{a}}^{z}}{\sqrt{\det({\bm{J}}^{\intercal}{\bm{J}})}}. (7)
𝒂x=𝒇xz−1​(𝒂z)=𝒇xz−1​(𝒈⁡(𝒔z))=𝒇xz−1​(𝒈⁡(𝒇xz​(𝒔x)))\displaystyle{\bm{a}}^{x}={\bm{f}}_{x}^{z-1}({\bm{a}}^{z})={\bm{f}}_{x}^{z-1}({\bm{g}}({\bm{s}}^{z}))={\bm{f}}_{x}^{z-1}({\bm{g}}({\bm{f}}_{x}^{z}({\bm{s}}^{x}))) (8)
𝒂x=arg​max𝐚x⁡Ez​(𝐟xz​(𝐚x,𝐬x)).\displaystyle{\bm{a}}^{x}=\argmax_{{\bm{a}}^{x}}E^{z}({\bm{f}}_{x}^{z}({\bm{a}}^{x},{\bm{s}}^{x})). (9)
πx​(𝒂x|𝒔x)=exp⁡(∑k=1Kβk​Ezk​(𝒇xzk​(𝒂x,𝒔x)))​1Z⁡(𝒔x)\displaystyle\pi^{x}({\bm{a}}^{x}|{\bm{s}}^{x})=\exp\left(\sum_{k=1}^{K}\beta_{k}E^{z_{k}}({\bm{f}}_{x}^{z_{k}}({\bm{a}}^{x},{\bm{s}}^{x}))\right)\frac{1}{Z({\bm{s}}^{x})} (10)
Original diagram from PDF page 4, including all mathematical labels.
Fig. 3: Bayesian network for the energy tree. The policy distribution πx​(𝒂x|𝒔x)\pi^{x}({\bm{a}}^{x}|{\bm{s}}^{x}) is computed w.r.t. the policies in the latent spaces 𝒵i{\mathcal{Z}}_{i}. Additionally, a latent policy πz​(𝒂z|𝒔z)\pi^{z}({\bm{a}}^{z}|{\bm{s}}^{z}) can be represented by a set of policies in a different latent spaces 𝒲i{\mathcal{W}}_{i}. White diamonds represent deterministic transformations and black square is a probabilistic mapping.

III-B Maximum Likelihood Estimation on CEP

Cross-entropy optimization repeatedly samples candidate actions, scores their combined energies, and updates a proposal distribution. Algorithm 1 preserves the complete procedure. Batched GPU evaluation of candidates and independent components makes this stochastic search practical at the reported control frequencies.

Equations 11–12
𝒂MLx=arg​max𝐚x∏k=0Kπk(𝐚x|𝐬x)βk,βk>0.\displaystyle{\bm{a}}_{\textrm{ML}}^{x}=\argmax_{{\bm{a}}^{x}}\prod_{k=0}^{K}\pi_{k}({\bm{a}}^{x}|{\bm{s}}^{x})^{\beta_{k}}\,,\,\beta_{k}>0. (11)
𝜽t+1\displaystyle{\bm{\theta}}_{t+1} =arg​max𝜽t⁡𝔼p⁡(𝐚x|𝜽t)​[log⁡(∏k=0Kπk​(𝐚x|𝐬x)βk)]\displaystyle=\argmax_{{\bm{\theta}}_{t}}\mathbb{E}_{p({\bm{a}}^{x}|{\bm{\theta}}_{t})}\left[\log\left(\prod_{k=0}^{K}\pi_{k}({\bm{a}}^{x}|{\bm{s}}^{x})^{\beta_{k}}\right)\right]
∝𝔼p⁡(𝒂x|𝜽t)​[∑k=1Kβk​Ezk​(𝒇xzk​(𝒂x,𝒔x))].\displaystyle\propto\mathbb{E}_{p({\bm{a}}^{x}|{\bm{\theta}}_{t})}\left[\sum_{k=1}^{K}\beta_{k}E^{z_{k}}({\bm{f}}_{x}^{z_{k}}({\bm{a}}^{x},{\bm{s}}^{x}))\right]. (12)
Algorithm 1 Composable Energy Policies
Given : NN: Number of samples;
𝒔x{\bm{s}}^{x}: Current state in common space;
KK: Number of Energy Components;
(𝒇xz1,Ez1,βz1),…,(𝒇xzK,EzK,βzK))({\bm{f}}_{x}^{z_{1}},E^{z_{1}},\beta^{z_{1}}),\dots,({\bm{f}}_{x}^{z_{K}},E^{z_{K}},\beta^{z_{K}})): Maps, energies and inverse temperatures;
II: Optimization steps;
(μ0,Σ0)(\mu_{0},\Sigma_{0}): Initial sampling distribution mean and variance;
(𝒂∗,e∗)({\bm{a}}^{*},e^{*}): Initial optimal action and energy;
for i←0i\leftarrow 0 to I−1I-1 do
   for n←0n\leftarrow 0 to N−1N-1 do
      𝒂nx∼𝒩⁡(μi,Σi){\bm{a}}_{n}^{x}\sim{\mathcal{N}}(\mu_{i},\Sigma_{i});
      for k←1k\leftarrow 1 to KK do
         𝒔zk,𝒂nzk=𝒇xzk​(𝒔x,𝒂nx){\bm{s}}^{z_{k}},{\bm{a}}_{n}^{z_{k}}={\bm{f}}_{x}^{z_{k}}({\bm{s}}^{x},{\bm{a}}_{n}^{x});
         enzk=Ezk​(𝒔zk,𝒂nzk)e_{n}^{z_{k}}=E^{z_{k}}({\bm{s}}^{z_{k}},{\bm{a}}_{n}^{z_{k}});
      enx=∑k=1Kβzk​enzke_{n}^{x}=\sum_{k=1}^{K}\beta^{z_{k}}e_{n}^{z_{k}}
   μi+1←Updateμ(μi,𝒂0:Nx,e0:Nx)\mu_{i+1}\leftarrow\textrm{Update}_{\mu}(\mu_{i},{\bm{a}}_{0:N}^{x},e_{0:N}^{x});
   Σi+1←UpdateΣ(Σi,𝒂0:Nx,e0:Nx)\Sigma_{i+1}\leftarrow\textrm{Update}_{\Sigma}(\Sigma_{i},{\bm{a}}_{0:N}^{x},e_{0:N}^{x});
   𝒂i∗,ei∗←arg𝒂maxe(e0:Nx){\bm{a}}_{i}^{*},e_{i}^{*}\leftarrow\arg_{\bm{a}}\max_{e}(e_{0:N}^{x});
   if e∗<ei∗e^{*}<e_{i}^{*} then
      𝒂∗←𝒂i∗{\bm{a}}^{*}\leftarrow{\bm{a}}_{i}^{*};
      e∗←ei∗e^{*}\leftarrow e_{i}^{*};
return 𝒂∗{\bm{a}}^{*};

IV CEP for Robot Motion Generation

Robot state is joint position and velocity, and the action is joint acceleration. Forward kinematics and Jacobians map candidates to task-space accelerations. The approach avoids explicitly inverting the task map during action selection, although its performance still depends on the chosen energies and numerical search.

Equations 13–15
𝒒¨=g⁡(𝒒,𝒒˙)\displaystyle\ddot{{\bm{q}}}=g({\bm{q}},\dot{{\bm{q}}}) (13)
𝒙\displaystyle{\bm{x}} =fkin​(𝒒)\displaystyle=\textbf{f}_{\textrm{kin}}({\bm{q}})
𝒙˙\displaystyle\dot{{\bm{x}}} =𝑱⁡(𝒒)​𝒒˙\displaystyle={\bm{J}}({\bm{q}})\dot{{\bm{q}}} (14)
𝒙¨\displaystyle\ddot{{\bm{x}}} =𝑱⁡(𝒒)​𝒒¨+𝑱˙​(𝒒)​𝒒˙≈𝑱⁡(𝒒)​𝒒¨\displaystyle={\bm{J}}({\bm{q}})\ddot{{\bm{q}}}+\dot{{\bm{J}}}({\bm{q}})\dot{{\bm{q}}}\approx{\bm{J}}({\bm{q}})\ddot{{\bm{q}}}
πq​(𝒒¨|𝒒,𝒒˙)∝exp⁡(∑k=1Kβk​Exk​(𝑱k​(𝒒)​𝒒¨,fkin​k​(𝒒),𝑱k​(𝒒)​𝒒˙)).\displaystyle\pi^{q}(\ddot{{\bm{q}}}|{\bm{q}},\dot{{\bm{q}}})\propto\exp\left(\sum_{k=1}^{K}\beta_{k}E^{x_{k}}({\bm{J}}_{k}({\bm{q}})\ddot{{\bm{q}}},\textbf{f}_{\textrm{kin}k}({\bm{q}}),{\bm{J}}_{k}({\bm{q}})\dot{{\bm{q}}})\right). (15)

Go To a target

A Gaussian goal-reaching policy is centered on a proportional–derivative controller. Its covariance shrinks with distance to the target, making the goal component increasingly selective near the desired state.

Equations 16–18
π⁡(𝒙¨|𝒙,𝒙˙)=𝒩⁡(𝝁⁡(𝒙,𝒙˙),𝚺⁡(𝒙))\displaystyle\pi(\ddot{{\bm{x}}}|{\bm{x}},\dot{{\bm{x}}})={\mathcal{N}}({\bm{\mu}}({\bm{x}},\dot{{\bm{x}}}),{\bm{\Sigma}}({\bm{x}})) (16)
𝝁⁡(𝒙,𝒙˙)=−𝑲p​(𝒙−𝒙∗)−𝑲v​𝒙˙\displaystyle{\bm{\mu}}({\bm{x}},\dot{{\bm{x}}})=-{\bm{K}}_{p}({\bm{x}}-{\bm{x}}^{*})-{\bm{K}}_{v}\dot{{\bm{x}}} (17)
𝚺⁡(𝒙)=α​(𝒙−𝒙∗)⊺​(𝒙−𝒙∗).\displaystyle{\bm{\Sigma}}({\bm{x}})=\alpha({\bm{x}}-{\bm{x}}^{*})^{\intercal}({\bm{x}}-{\bm{x}}^{*}). (18)

Obstacle Avoidance

Obstacle avoidance projects velocity and acceleration onto the robot-to-obstacle direction. Far from an obstacle, or while moving away, the component is unrestrictive. Close to an obstacle while approaching it, the allowed acceleration interval enforces braking away from the obstacle. This represents a set of acceptable actions rather than a single repulsive vector.

Equations 19–21
𝒗r,o=𝒙r−𝒙o\displaystyle{\bm{v}}_{r,o}={\bm{x}}_{r}-{\bm{x}}_{o} (19)
x˙p\displaystyle\dot{x}_{p} =𝒙˙​𝒙˙⋅𝒗r,o‖𝒙˙‖​‖𝒗r,o‖\displaystyle=\dot{{\bm{x}}}\frac{\dot{{\bm{x}}}\cdot{\bm{v}}_{r,o}}{||\dot{{\bm{x}}}||||{\bm{v}}_{r,o}||}
x¨p\displaystyle\ddot{x}_{p} =𝒙¨​𝒙¨⋅𝒗r,o‖𝒙¨‖​‖𝒗r,o‖\displaystyle=\ddot{{\bm{x}}}\frac{\ddot{{\bm{x}}}\cdot{\bm{v}}_{r,o}}{||\ddot{{\bm{x}}}||||{\bm{v}}_{r,o}||} (20)
E⁡(x¨p,x˙p,𝒗r,o)={0if x˙p<0 or ‖𝒗r,o‖>γ −∞if x¨p>−α​x˙p−β0otherwise\displaystyle E(\ddot{x}_{p},\dot{x}_{p},{\bm{v}}_{r,o})=\begin{cases}0&\text{{if} $\dot{x}_{p}<0$ {or} $||{\bm{v}}_{r,o}||>\gamma$ }\\ -\infty&\text{{if} $\ddot{x}_{p}>-\alpha\dot{x}_{p}-\beta$}\\ 0&\textrm{otherwise}\end{cases} (21)

Avoid Joint Limits

A similar interval-based energy activates near joint limits and favors accelerations directed back toward the allowed region. The displayed equations retain the distance, activation threshold, and braking parameters.

Equations 22–23
dl=q−ql\displaystyle d_{l}=q-q_{l} (22)
E⁡(q¨,q˙,q)={0if dl|dl|​q˙<0 or |dl|>γ −∞if dl|dl|​q¨>−α​dl|dl|​q˙−β0otherwise\displaystyle E(\ddot{q};\dot{q},q)=\begin{cases}0&\text{{if} $\frac{d_{l}}{|d_{l}|}\dot{q}<0$ {or} $|d_{l}|>\gamma$ }\\ -\infty&\text{{if} $\frac{d_{l}}{|d_{l}|}\ddot{q}>-\alpha\frac{d_{l}}{|d_{l}|}\dot{q}-\beta$}\\ 0&\text{otherwise}\end{cases} (23)

Imitation Learning, Control-as-Inference and RL

Component distributions can also come from imitation learning, probabilistic control inference, or reinforcement learning. Multimodal policies are allowed; the architecture is not limited to Gaussian heuristic controllers.

V CEP for Hierarchical Reinforcement Learning

The high-level learned policy supplies the mean and covariance of a Gaussian component, which the low-level CEP optimizer combines with fixed priors. A prior can exclude actions by assigning zero probability or encourage exploration in several regions. The learned covariance controls how strongly the new policy competes with these priors.

This differs from adding an unrestricted residual to a safe action: composition can preserve explicit exclusions represented by the prior. In practice, numerical optimization, prior validity, and the covariance choice matter; the ablation observes more collisions when the learned component reduces the prior’s influence.

Equations 24–26
minθDKL(πθ(𝒂|𝒔)||πq(𝒂|𝒔))\displaystyle\min_{\theta}D_{\mathrm{KL}}(\pi_{\theta}({\bm{a}}|{\bm{s}})||\pi_{q}({\bm{a}}|{\bm{s}}))
maxθ⁡𝔼pπθ​(𝒔,𝒂)​[ℛ⁡(𝒔,𝒂)].\displaystyle\max_{\theta}\mathbb{E}_{p_{\pi_{\theta}}({\bm{s}},{\bm{a}})}[{\mathcal{R}}({\bm{s}},{\bm{a}})].
πL​(𝒂|𝒔)=∫𝒂Hπq​(𝒂|𝒔,𝒂H)​πθH​(𝒂H|𝒔)​d​𝒂H\displaystyle\pi^{L}({\bm{a}}|{\bm{s}})=\int_{{\bm{a}}_{H}}\pi_{q}({\bm{a}}|{\bm{s}},{\bm{a}}_{H})\pi_{\theta}^{H}({\bm{a}}_{H}|{\bm{s}})d{\bm{a}}_{H} (24)
𝝁H,𝚺H\displaystyle{\bm{\mu}}_{H},{\bm{\Sigma}}_{H} ∼πθH​(𝝁H,𝚺H|𝒔)\displaystyle\sim\pi_{\theta}^{H}({\bm{\mu}}_{H},{\bm{\Sigma}}_{H}|{\bm{s}}) (25)
πL​(𝒂|𝒔,𝝁H,𝚺H)\displaystyle\pi^{L}({\bm{a}}|{\bm{s}},{\bm{\mu}}_{H},{\bm{\Sigma}}_{H}) =πq​(𝒂|𝒔)​𝒩​(𝒂|𝝁H,𝚺H)\displaystyle=\pi_{q}({\bm{a}}|{\bm{s}}){\mathcal{N}}({\bm{a}}|{\bm{\mu}}_{H},{\bm{\Sigma}}_{H})
𝒂=arg​max𝐚⁡πL​(𝐚|𝐬,𝝁H,𝚺H).\displaystyle{\bm{a}}=\argmax_{{\bm{a}}}\pi^{L}({\bm{a}}|{\bm{s}},{\bm{\mu}}_{H},{\bm{\Sigma}}_{H}). (26)

VI Experimental Evaluation

Evaluation separates reactive motion generation from prior-guided learning. Comparisons target reactive controllers rather than global planners, and the learning comparison uses the same prior across methods.

Refer to caption
Fig. 4: Reactive Motion Generation Environments. In the top, from left to right: 1 Obstacle, 3 Obstacles, Cross. In the bottom, from left to right: Cage I, Cage II, Double Cross. Green Sphere: target pose, red spheres: obstacles.

Reactive Motion Generation

A seven-degree-of-freedom KUKA-LWR arm reaches position-only or full-pose targets in increasingly difficult obstacle arrangements. CEP combines target, link-obstacle, and joint-limit energies and is compared with potential fields and Riemannian motion policies. The reported implementation evaluates 10,000 action candidates per search step and uses two steps with GPU parallelization to target control at at least 500 Hz.

Methods 1 Obstacle 3 Obstacles Cross Double Cross Cage I Cage II
Success Collide Success Collide Success Collide Success Collide Success Collide Success Collide
Riemannian Motion Policies [32] 100/100 0/100 98/100 0/100 94/100 0/100 84/100 0/100 55/100 0/100 5/100 0/100
Artificial Potential Fields [20] 100/100 0/100 100/100 0/100 90/100 0/100 76/100 0/100 43/100 0/100 4/100 2/100
Composable Energy Policies(Ours) 100/100 0/100 100/100 0/100 98/100 0/100 91/100 0/100 89/100 0/100 46/100 0/100
TABLE I: Results for 3D GoTo + Obstacle Avoidance Task
Methods 1 Obstacle 3 Obstacles Cross Double Cross Cage I Cage II
Success Collide Success Collide Success Collide Success Collide Success Collide Success Collide
Riemannian Motion Policies [32] 100/100 0/100 91/100 0/100 75/100 0/100 63/100 0/100 18/100 0/100 1/100 0/100
Artificial Potential Fields [20] 100/100 0/100 90/100 0/100 78/100 0/100 60/100 0/100 21/100 0/100 1/100 0/100
Composable Energy Policies(Ours) 100/100 0/100 100/100 0/100 91/100 0/100 87/100 0/100 43/100 0/100 14/100 0/100
TABLE II: Results for 6D GoTo + Obstacle Avoidance Task

Results and analysis

For position reaching, CEP succeeds in 89 of 100 trials in Cage I and 46 in Cage II, compared with 55/5 for RMP and 43/4 for potential fields. No CEP collisions are recorded in these trials. Full-pose reaching is harder: CEP’s cage success falls to 43% and 14%. It still exceeds the tested baselines but frequently fails to reach the goal, supporting the need for a global planner in difficult environments.

Refer to caption
Fig. 5: Puck hitting environment. CEP is tested in a RL environment where the robot needs to learn how to hit the puck. The reward is defined by the euclidean distance between the puck and the green target. An additional reward penalizes if the robot hits the table.

Prior Based Policies for RL

A seven-degree-of-freedom arm must strike a puck toward a target while avoiding the table. The learned action is three-dimensional end-effector velocity. PPO compares behavior-cloning initialization, additive residual learning, and CEP composition, all using the same goal-reaching/table-avoidance prior. Rewards combine puck-target distance with a large table-collision penalty. [37] [39]

Equations 27–33
r=−‖𝒙puck−𝒙target‖2+rT​(𝒙ee)\displaystyle r=-||{\bm{x}}_{\textrm{puck}}-{\bm{x}}_{\textrm{target}}||^{2}+r_{T}({\bm{x}}_{\textrm{ee}}) (27)
𝜽∗=arg​min𝜽𝔼p⁡(𝐬)[DKL(πq(𝐚|𝐬)||πθ(𝐚|𝐬))]\displaystyle{\bm{\theta}}^{*}=\argmin_{{\bm{\theta}}}\mathbb{E}_{p({\bm{s}})}\left[D_{\mathrm{KL}}(\pi_{q}({\bm{a}}|{\bm{s}})||\pi_{\theta}({\bm{a}}|{\bm{s}}))\right] (28)
𝒂∼πθ​(𝒂|𝒔).\displaystyle{\bm{a}}\sim\pi_{\theta}({\bm{a}}|{\bm{s}}). (29)
𝒂=𝒂∗+𝒂H,𝒂H∼πθ​(𝒂H|𝒔)\displaystyle{\bm{a}}={\bm{a}}^{*}+{\bm{a}}_{H}\,,\,{\bm{a}}_{H}\sim\pi_{\theta}({\bm{a}}_{H}|{\bm{s}}) (30)
𝒂∗=arg​max𝐚⁡π𝐪​(𝐚|𝐬).\displaystyle{\bm{a}}^{*}=\argmax_{{\bm{a}}}\pi_{{\bm{q}}}({\bm{a}}|{\bm{s}}). (31)
𝒂H∼π𝜽​(𝒂H|𝒔)\displaystyle{\bm{a}}_{H}\sim\pi_{\bm{\theta}}({\bm{a}}_{H}|{\bm{s}}) (32)
𝒂=arg​max𝐚⁡πq​(𝐚|𝐬,𝐚H).\displaystyle{\bm{a}}=\argmax_{{\bm{a}}}\pi_{q}({\bm{a}}|{\bm{s}},{\bm{a}}_{H}). (33)

Results and analysis

CEP-PPO improves reward more consistently and records fewer table collisions than the tested alternatives. The behavior-cloned policy can learn to avoid the table by abandoning useful striking behavior, while unrestricted residual actions disturb the prior’s avoidance behavior. The result supports preserving useful prior constraints during exploration rather than treating the comparison as a universal ranking of learning algorithms.

Refer to caption
Refer to caption
Fig. 6: Training curves for Hitting a puck environment. CEP-PPO performs consistenly better than other prior + RL methods. Collision avoidance prior is stronger in CEP-PPO and provides a safer training.

CEP ablation study

Learning the Gaussian covariance outperforms the tested fixed-covariance choices in reward. A large fixed covariance lets the prior dominate; a small one gives the learned policy more influence. Adapting this tradeoff improves hitting behavior but can increase collisions, so the most flexible variant does not provide unconditional safety.

Refer to caption
Refer to caption
Fig. 7: Training curves for Hitting a puck environment. Learning the variance of the policy improves the learning curve, but also increases the probability of collisions.

VII Conclusions

CEP provides a common optimization interface for simultaneously active motion objectives and learned priors. Its benefits come from retaining distributions over acceptable actions before composition. The experiments support improved local control, while incomplete cage success, long-horizon interactions, and approximate treatment of non-bijective mappings limit stronger claims.

Appendix A A Control as Inference view for Reactive motion generation

Control as inference introduces optimality variables whose likelihood depends on reward, producing a posterior over action trajectories. A reactive controller instead optimizes only the next action, integrating over future states and actions under an assumed future policy. Its local energy depends on that policy’s action-value function; global optimality requires an appropriate optimal value function, not merely a short computation horizon.

Fig. 8: Graphical model for the Optimal Control problem. 𝒔t{\bm{s}}_{t} denoted the state, 𝒂t{\bm{a}}_{t} denotes the action and oto_{t} is an additional variable representing the optimality of the state and action for a given reward.
Original diagram from PDF page 11, including all mathematical labels.
Equations 34–37
p⁡(A0T|𝒔0,O0T)=p⁡(O0T|A0T,𝒔0)​q​(A0T|𝒔0)p⁡(O0T|𝒔0)\displaystyle p(A_{0}^{T}|{\bm{s}}_{0},O_{0}^{T})=\frac{p(O_{0}^{T}|A_{0}^{T},{\bm{s}}_{0})q(A_{0}^{T}|{\bm{s}}_{0})}{p(O_{0}^{T}|{\bm{s}}_{0})} (34)
p(O0T|A0T,𝒔0)=∫𝒔1:Tp(O0T|S0T,A0T)p(S1T|A0T,𝒔0)dS1T\displaystyle p(O_{0}^{T}|A_{0}^{T},{\bm{s}}_{0})=\int_{{\bm{s}}_{1:T}}p(O_{0}^{T}|S_{0}^{T},A_{0}^{T})p(S_{1}^{T}|A_{0}^{T},{\bm{s}}_{0})dS_{1}^{T} (35)
p⁡(𝒂0|𝒔0,O0T)=p⁡(O0T|𝒂0,𝒔0)​q​(𝒂0|𝒔0)p⁡(O0T|𝒔0)\displaystyle p({\bm{a}}_{0}|{\bm{s}}_{0},O_{0}^{T})=\frac{p(O_{0}^{T}|{\bm{a}}_{0},{\bm{s}}_{0})q({\bm{a}}_{0}|{\bm{s}}_{0})}{p(O_{0}^{T}|{\bm{s}}_{0})} (36)
p⁡(O0T|𝒂0,𝒔0)=\displaystyle p(O_{0}^{T}|{\bm{a}}_{0},{\bm{s}}_{0})=
∫S1:T∫A1:Tp(O0T|S0T,A0T)p(S1T|𝒂0,𝒔0)π(A1T|S1T)dS1TdA1T.\displaystyle\int_{S_{1:T}}\int_{A_{1:T}}p(O_{0}^{T}|S_{0}^{T},A_{0}^{T})p(S_{1}^{T}|{\bm{a}}_{0},{\bm{s}}_{0})\pi(A_{1}^{T}|S_{1}^{T})dS_{1}^{T}dA_{1}^{T}. (37)
Fig. 9: Graphical model for one-step ahead optimal control problem. In this approach, the actions A1TA_{1}^{T} are dependant on S1TS_{1}^{T} given a policy π\pi.
Original diagram from PDF page 11, including all mathematical labels.
Equations 38–39
𝔼S0T,A0T∼pπ(S0T,A0T|𝒔0,𝒂0)[p(O0T|S0T,A0T)]∝\displaystyle\mathbb{E}_{S_{0}^{T},A_{0}^{T}\sim p^{\pi}(S_{0}^{T},A_{0}^{T}|{\bm{s}}_{0},{\bm{a}}_{0})}\left[p(O_{0}^{T}|S_{0}^{T},A_{0}^{T})\right]\propto
𝔼S0T,A0T∼pπ(S0T,A0T|𝒔0,𝒂0)[exp(∑t=0Tr(𝒔t,𝒂t))]=\displaystyle\mathbb{E}_{S_{0}^{T},A_{0}^{T}\sim p^{\pi}(S_{0}^{T},A_{0}^{T}|{\bm{s}}_{0},{\bm{a}}_{0})}\left[\exp(\sum_{t=0}^{T}r({\bm{s}}_{t},{\bm{a}}_{t}))\right]=
exp⁡(Qrπ​(𝒔0,𝒂0))\displaystyle\exp(Q_{r}^{\pi}({\bm{s}}_{0},{\bm{a}}_{0})) (38)
p⁡(𝒂0|𝒔0,O0T)∝exp⁡(Qrπ​(𝒔0,𝒂0))​q​(𝒂0|𝒔0).\displaystyle p({\bm{a}}_{0}|{\bm{s}}_{0},O_{0}^{T})\propto\exp(Q_{r}^{\pi}({\bm{s}}_{0},{\bm{a}}_{0}))q({\bm{a}}_{0}|{\bm{s}}_{0}). (39)

A-A Optimality Guarantees

The sum of independently optimal component action-value functions generally differs from the optimal value for the combined reward. The derivation recovers equality for the one-step case and analyzes how disagreement accumulates over longer horizons. Identical component values eliminate the composition discrepancy; conflicting objectives increase it. The retained recurrence and related maximum-entropy bounds explain why reactive composition becomes less reliable when success depends on long-term coordination.

Equations 40–54
Q⁡(𝒔,𝒂)\displaystyle Q({\bm{s}},{\bm{a}}) =r⁡(𝒔,𝒂)+𝔼p⁡(𝒔′|𝒔,𝒔)​[V⁡(𝒔′)]\displaystyle=r({\bm{s}},{\bm{a}})+\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{s}})}\left[V({\bm{s}}^{\prime})\right]
V⁡(𝒔)\displaystyle V({\bm{s}}) =log∫𝒜exp(Q(𝒂,𝒔)).\displaystyle=\log\int_{\mathcal{A}}\exp(Q({\bm{a}},{\bm{s}})). (40)
Q∗T=12​(r1+r2)\displaystyle Q^{*T}=\frac{1}{2}(r_{1}+r_{2}) (41)
Q1∗T=r1,Q2∗T=r2.\displaystyle Q_{1}^{*T}=r_{1}\,,\,Q_{2}^{*T}=r_{2}. (42)
Q∗T=12​(Q1∗T+Q2∗T)\displaystyle Q^{*T}=\frac{1}{2}(Q_{1}^{*T}+Q_{2}^{*T}) (43)
Δ​QT=Q∗T−QΣT=0.\displaystyle\Delta Q^{T}=Q^{*T}-Q_{\Sigma}^{T}=0. (44)
Q∗T−1​(𝒔,𝒂)\displaystyle Q^{*T-1}({\bm{s}},{\bm{a}}) =r⁡(𝒔,𝒂)+𝔼p⁡(𝒔′|𝒔,𝒔)​[V∗T​(𝒔′)]\displaystyle=r({\bm{s}},{\bm{a}})+\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{s}})}\left[V^{*T}({\bm{s}}^{\prime})\right]
V∗T​(𝒔)\displaystyle V^{*T}({\bm{s}}) =log∫𝒜exp(Q∗T(𝒂,𝒔)).\displaystyle=\log\int_{\mathcal{A}}\exp(Q^{*T}({\bm{a}},{\bm{s}})). (45)
Δ​QT−1=Q∗T−1−QΣT−1=𝔼p⁡(𝒔′|𝒔,𝒂)​[V∗T−VΣT]\displaystyle\Delta Q^{T-1}=Q^{*T-1}-Q_{\Sigma}^{T-1}=\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})}\left[V^{*T}-V_{\Sigma}^{T}\right]
V∗T−VΣT=log⁡∫𝒜exp⁡(Q∗T)∫𝒜(exp⁡(Q1∗T)​∫𝒜exp⁡(Q2∗T))12\displaystyle V^{*T}-V_{\Sigma}^{T}=\log\frac{\int_{\mathcal{A}}\exp(Q^{*T})}{\int_{\mathcal{A}}(\exp(Q_{1}^{*T})\int_{\mathcal{A}}\exp(Q_{2}^{*T}))^{\frac{1}{2}}}
=log⁡∫𝒜exp⁡(12​Q1∗T)​exp⁡(12​Q2∗T)(∫𝒜exp⁡(Q1∗T)​∫𝒜exp⁡(Q2∗T))12.\displaystyle=\log\frac{\int_{\mathcal{A}}\exp(\frac{1}{2}Q_{1}^{*T})\exp(\frac{1}{2}Q_{2}^{*T})}{(\int_{\mathcal{A}}\exp(Q_{1}^{*T})\int_{\mathcal{A}}\exp(Q_{2}^{*T}))^{\frac{1}{2}}}. (47)
Q∗T−1\displaystyle Q^{*T-1} =QΣT−1+Δ​QT−1\displaystyle=Q_{\Sigma}^{T-1}+\Delta Q^{T-1}
=QΣT−1+𝔼p⁡(𝒔′|𝒔,𝒂)​[log⁡∫𝒜exp⁡(12​Q1∗T)​exp⁡(12​Q2∗T)(∫𝒜exp⁡(Q1∗T)​∫𝒜exp⁡(Q2∗T))12]\displaystyle=Q_{\Sigma}^{T-1}+\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})}\left[\log\frac{\int_{\mathcal{A}}\exp(\frac{1}{2}Q_{1}^{*T})\exp(\frac{1}{2}Q_{2}^{*T})}{(\int_{\mathcal{A}}\exp(Q_{1}^{*T})\int_{\mathcal{A}}\exp(Q_{2}^{*T}))^{\frac{1}{2}}}\right] (48)
Δ​Qt−1=𝔼p⁡(𝒔′|𝒔,𝒂)​[log⁡∫𝒜exp⁡(12​Q1∗t)​exp⁡(12​Q2∗t)​exp⁡(Δ​Qt)(∫𝒜exp⁡(Q1∗t)​∫𝒜exp⁡(Q2∗t))12].\displaystyle\Delta Q^{t-1}=\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})}\left[\log\frac{\int_{\mathcal{A}}\exp(\frac{1}{2}Q_{1}^{*t})\exp(\frac{1}{2}Q_{2}^{*t})\exp(\Delta Q^{t})}{(\int_{\mathcal{A}}\exp(Q_{1}^{*t})\int_{\mathcal{A}}\exp(Q_{2}^{*t}))^{\frac{1}{2}}}\right]. (49)
(∫𝒜exp⁡(Q1∗t)​∫𝒜exp⁡(Q2∗t))12>∫𝒜exp⁡(12​Q1∗t)​exp⁡(12​Q2∗t)\displaystyle(\int_{\mathcal{A}}\exp(Q_{1}^{*t})\int_{\mathcal{A}}\exp(Q_{2}^{*t}))^{\frac{1}{2}}>\int_{\mathcal{A}}\exp(\frac{1}{2}Q_{1}^{*t})\exp(\frac{1}{2}Q_{2}^{*t}) (50)
Δ​Q<0,∀Q1∗,Q2∗\displaystyle\Delta Q<0,\forall\,Q_{1}^{*},Q_{2}^{*} (51)
J(π)=𝔼π,p⁡(𝒔0)[∑t=0Tr(𝒔t,𝒂t)+αℋ(π)|𝒔t+1∼p(⋅|𝒔t,𝒂t)]\displaystyle J(\pi)=\mathbb{E}_{\pi,p({\bm{s}}_{0})}\left[\sum_{t=0}^{T}r({\bm{s}}_{t},{\bm{a}}_{t})+\alpha{\mathcal{H}}(\pi)|{\bm{s}}_{t+1}\sim p(\cdot|{\bm{s}}_{t},{\bm{a}}_{t})\right] (52)
QΣ​(𝒔,𝒂)≥Q∗​(𝒔,𝒂)≥QΣ​(𝒔,𝒂)−𝒞∗​(𝒔,𝒂)\displaystyle Q_{\Sigma}({\bm{s}},{\bm{a}})\geq Q^{*}({\bm{s}},{\bm{a}})\geq Q_{\Sigma}({\bm{s}},{\bm{a}})-{\mathcal{C}}^{*}({\bm{s}},{\bm{a}}) (53)
𝒞(𝒔,𝒂)←γ𝔼p⁡(𝒔′|𝒔,𝒂)[𝒟12(π1∗(𝒂|𝒔)||π2∗(𝒂|𝒔))+max𝒂′𝒞(𝒔′,𝒂′)]\displaystyle{\mathcal{C}}({\bm{s}},{\bm{a}})\leftarrow\gamma\mathbb{E}_{p({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})}\left[{\mathcal{D}}_{\frac{1}{2}}(\pi_{1}^{*}({\bm{a}}|{\bm{s}})||\pi_{2}^{*}({\bm{a}}|{\bm{s}}))+\max_{{\bm{a}}^{\prime}}{\mathcal{C}}({\bm{s}}^{\prime},{\bm{a}}^{\prime})\right] (54)

Appendix B Composable Energy Policies as Inference

Independent optimality factors conditioned on the same state and action yield a posterior proportional to the product of their likelihoods and an action prior. With a uniform prior, maximizing this posterior recovers the summed-energy CEP objective. This is an alternative interpretation of the energy tree’s product of policies.

Fig. 10: Graphical model for Composable Energy Policies. oko_{k} is an auxiliary variable that represents the optimality of 𝒔0{\bm{s}}_{0} and 𝒂0{\bm{a}}_{0} for a particular energy.
Original diagram from PDF page 13, including all mathematical labels.
Equations 55–56
p(𝒔0,𝒂0,o0:2)=q(𝒂0)p(𝒔0)∏k=02p(ok|𝒔0,𝒂0).\displaystyle p({\bm{s}}_{0},{\bm{a}}_{0},o_{0:2})=q({\bm{a}}_{0})p({\bm{s}}_{0})\prod_{k=0}^{2}p(o_{k}|{\bm{s}}_{0},{\bm{a}}_{0}). (55)
𝒂0∗\displaystyle{\bm{a}}_{0}^{*} =arg​max𝐚0p(𝐚0|𝐬0,o0:2=1)\displaystyle=\argmax_{{\bm{a}}_{0}}p({\bm{a}}_{0}|{\bm{s}}_{0},o_{0:2}=1)
∝exp⁡(∑k=02Ek​(𝒂0,𝒔0))​q​(𝒂0).\displaystyle\propto\exp\left(\sum_{k=0}^{2}E_{k}({\bm{a}}_{0},{\bm{s}}_{0})\right)q({\bm{a}}_{0}). (56)

Appendix C Composable Energy Policies and Artificial Potential Fields Method

If all components are Gaussian, their product is Gaussian and its maximizer is a precision-weighted combination of component means. This recovers a potential-field-style acceleration rule. Gaussian assumptions are restrictive for avoidance: several qualitatively different actions may be safe, and quality need not decay smoothly with distance from a single preferred action.

Equations 57–64
𝒒¨=∑k=0K𝚲k​𝒈k​(𝒒,𝒒˙).\displaystyle\ddot{{\bm{q}}}=\sum_{k=0}^{K}{\bm{\Lambda}}_{k}{\bm{g}}_{k}({\bm{q}},\dot{{\bm{q}}}). (57)
πk​(𝒒¨|𝒒,𝒒˙)=𝒩⁡(𝒈k​(𝒒,𝒒˙),Σk).\displaystyle\pi_{k}(\ddot{{\bm{q}}}|{\bm{q}},\dot{{\bm{q}}})={\mathcal{N}}({\bm{g}}_{k}({\bm{q}},\dot{{\bm{q}}}),\Sigma_{k}). (58)
π⁡(𝒒¨|𝒒,𝒒˙)\displaystyle\pi(\ddot{{\bm{q}}}|{\bm{q}},\dot{{\bm{q}}}) =∏k=0K𝒩⁡(𝒈k​(𝒒,𝒒˙),Σk)=𝒩⁡(𝒈Σ​(𝒒,𝒒˙),ΣΣ)\displaystyle=\prod_{k=0}^{K}{\mathcal{N}}({\bm{g}}_{k}({\bm{q}},\dot{{\bm{q}}}),\Sigma_{k})={\mathcal{N}}({\bm{g}}_{\Sigma}({\bm{q}},\dot{{\bm{q}}}),\Sigma_{\Sigma}) (59)
𝒈Σ=(∑k=0KΣk−1)−1​(∑k=0KΣk−1​𝒈k)\displaystyle{\bm{g}}_{\Sigma}=\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}\right)^{-1}\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}{\bm{g}}_{k}\right) (60)
ΣΣ=(∑k=0KΣk−1)−1.\displaystyle\Sigma_{\Sigma}=\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}\right)^{-1}. (61)
𝒒¨=arg​max𝐪¨⁡π​(𝐪¨|𝐪,𝐪˙).\displaystyle\ddot{{\bm{q}}}=\argmax_{\ddot{{\bm{q}}}}\pi(\ddot{{\bm{q}}}|{\bm{q}},\dot{{\bm{q}}}). (62)
𝒒¨=(∑k=0KΣk−1)−1​(∑k=0KΣk−1​𝒈k​(𝒒,𝒒˙)).\displaystyle\ddot{{\bm{q}}}=\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}\right)^{-1}\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}{\bm{g}}_{k}({\bm{q}},\dot{{\bm{q}}})\right). (63)
𝚲k=(∑k=0KΣk−1)−1.Σk−1\displaystyle{\bm{\Lambda}}_{k}=\left(\sum_{k=0}^{K}\Sigma_{k}^{-1}\right)^{-1}.\Sigma_{k}^{-1} (64)

Appendix D Experiments

D-A Reactive Motion Generation

Table III preserves the goal-attraction gains and covariance scale, plus obstacle and joint-limit activation/braking parameters. Baseline components follow the cited potential-field and RMPflow formulations. [3]

Energy Modules Parameters
Reach Target 𝑲p=20.{\bm{K}}_{p}=20. 𝑲v=30.{\bm{K}}_{v}=30. α=10.\alpha=10.
Obstacle Avoidance γ=0.2\gamma=0.2 α=4.\alpha=4. β=0.1\beta=0.1
Joint Limits avoidance γ=0.3\gamma=0.3 α=4.\alpha=4. β=0.1\beta=0.1
TABLE III: Component parameters for Composable Energy Policies in reactive motion generation

D-B Prior based policies for RL

Table IV retains the shared PPO networks and optimization settings, including a 1,400-step horizon, discount factor 0.997, and separate actor/critic learning rates. The same settings are used for the three prior-based learning methods.

Hyperparameters
horizon 1400
Policy Net 18-128-128-3
policy std0 1.
policy batch 256
n_epochs_policy 2
policy learn rate 1e-4
Critic Net 18-256-256-1
critic batch 256
critic learn rate 3e-4
n_steps_per_fit horizon x 30
Discount factor .997
TABLE IV: Component parameters for Composable Energy Policies in reactive motion generation
Concept demonstration

Open image ↗