Literature
Display
← All papers

Learning from Demonstration Using Products of Experts: Applications to Manipulation and Task Prioritization

Emmanuel Pignat · João Silvério · Sylvain Calinon

· The International Journal of Robotics Research ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

Products of experts combine task-space preferences into a normalized distribution over robot configurations. Learning the factors jointly helps separate intended goals from kinematic restrictions and competing tasks. Variational mixtures approximate the otherwise difficult normalization.

A null-space extension recovers secondary objectives hidden by higher-priority tasks. Evidence includes planar simulations, Panda configuration experiments, and physical tray serving. The framework uses structured task transformations and kinematics; it does not learn arbitrary robot skills directly from images.

1 Introduction

Learning each task-space model independently can confuse demonstrated variability with what a task allows. Joint normalization asks which complete configurations the combined model predicts, allowing constraints and dependencies to influence learning.

Refer to caption
Figure 1: Product of experts can be used to set up virtual guides on robotic manipulators. A user demonstrates kinesthetically several configurations fulfilling the task objectives (left). In this example, the end-effector of the robot should stay close to a horizontal plane, while pointing to a desired target. The objectives are then inferred by the robot as target distributions under several transformations (task spaces). By exploiting the torque control capabilities of recent robots, the different objectives are tracked. Using an optimal control strategy, feedback gains can be computed according to the precision of the different objectives. As a resulting behavior, the robot is free to move along directions that are not constrained by the objectives (center), while preventing deviations from the objectives (right).

Organization of the article

The paper develops product-of-experts estimation, a null-space extension, a toolkit of robot transformations and distributions, two control strategies, and experiments testing their advantages.

Refer to caption
Figure 2: Product of experts can learn complex distributions of joint angles as a fusion of simple distributions within several task spaces. (a) 30 samples of the configuration of a 2-DoF planar robot are displayed. The end-effector follows an elongated Gaussian distribution. The distribution in configuration space has a more complex shape, which it is difficult to represent. (b) With our approach, we learn configuration-space distributions as a product of simple densities in different task spaces. It is thus possible to represent this sharp distribution easily with few parameters. (c) A standard approach like a Gaussian mixture model, directly applied in configuration space, needs much more samples and parameters to represent this distribution. Following Occam’s razor principle, finding simpler explanations (fewer parameters of the model) to understand complex dataset leads to better generalization.

1.1 Related work

The related work connects density estimation, compositional probabilistic models, robot imitation, and inverse optimal control.

Estimating distributions

General density estimators can represent complex data but often need many examples. Robot kinematics supplies useful structure: a complicated joint-space distribution may correspond to a simple target in task space.

Product of experts

In a mixture, any component can explain an observation. In a product, all experts jointly constrain it. The main difficulty is estimating the product’s normalization and learning its parameters.

Distributions for learning in robotics

Earlier robot methods frequently fit independent distributions and combine them only during control. Joint fitting instead accounts for reachable configurations and task interactions. It can infer hidden target preferences from static configurations, without requiring demonstrated control commands. This extends earlier task-priority learning, which required known task references. [57]

Inverse optimal control

Negative expert log-likelihoods act like task costs. Learning them resembles inverse optimal control, with more explicit structure and smaller demonstration datasets than general neural cost learning. The experiments primarily concern static configuration distributions.

Equation 1
p⁡(𝝉)∝exp⁡(−c𝜽​(𝝉)).\displaystyle p(\bm{\tau})\propto\exp\big(-c_{\bm{\theta}}(\bm{\tau})\big). (1)

2 Product of Experts

Each expert scores a transformation of the joint configuration. Multiplying these scores and integrating over configurations defines one distribution. Simple experts can produce sharply constrained, multimodal joint-space distributions because robot kinematics is nonlinear.

Equations 2–3
p⁡(𝒒|𝜽1,…,𝜽M)\displaystyle p(\bm{q}|\bm{\theta}_{1},\ldots,\bm{\theta}_{M}) =∏mpm​(𝒯m​(𝒒)|𝜽m)∫z∏mpm​(𝒯m​(𝒛)|𝜽m).\displaystyle=\frac{\prod_{m}p_{m}(\mathcal{T}_{m}(\bm{q})\,|\bm{\theta}_{m})}{\int_{z}\prod_{m}p_{m}(\mathcal{T}_{m}(\bm{z})\,|\bm{\theta}_{m})}. (2)
p~​(𝒒)\displaystyle\tilde{p}(\bm{q}) =∏mpm​(𝒯m​(𝒒)|𝜽m),\displaystyle=\prod_{m}p_{m}(\mathcal{T}_{m}(\bm{q})\,|\bm{\theta}_{m}), (3)

2.1 Importance and implications of the normalization constant

Joint normalization accounts for the robot’s reachable support and the interaction between experts. An irrelevant expert can become nearly constant without being penalized for failing to fit its transformed data alone. Unnormalized orientation factors are also usable because normalization occurs at the product level.

This distinction matters when goals are unreachable, secondary objectives are masked, or models transfer between different kinematic chains. Independent expert fits need not reproduce the demonstrated configuration distribution.

2.2 Estimating PoE

Nonlinear transformations generally prevent closed-form normalization. The approximation must support sampling, parameter gradients, and representation of separated modes.

2.2.1 Markov Chain Monte Carlo (MCMC)

MCMC approximates the density with samples, but narrow correlated regions and widely separated modes can cause slow mixing. Short chains can miss entire solutions. Gradient-based stochastic methods also permit transformations and hierarchy filters that are awkward for likelihood-based sampling.

2.2.2 Variational inference

VI minimizes reverse KL divergence from a tractable approximation to the target. The unknown normalization cancels from optimization. Reparameterized samples provide gradients for fitting means, covariances, and richer transformations.

Equations 4–8
DKL(q~||p)=∫𝒒q~(𝒒;𝝀)logq~​(𝒒,𝝀)p⁡(𝒒)d𝒒.\displaystyle D_{\mathrm{KL}}(\tilde{q}||p)=\int_{\bm{q}}\tilde{q}(\bm{q};\bm{\lambda})\log\frac{\tilde{q}(\bm{q};\bm{\lambda})}{p(\bm{q})}d\bm{q}. (4)
DKL(q~||p)=∫𝒒q~(𝒒;𝝀)logq~​(𝒒,𝝀)p~​(𝒒)d𝒒+log𝒞,\displaystyle D_{\mathrm{KL}}(\tilde{q}||p)=\int_{\bm{q}}\tilde{q}(\bm{q};\bm{\lambda})\log\frac{\tilde{q}(\bm{q};\bm{\lambda})}{\tilde{p}(\bm{q})}d\bm{q}+\log\mathcal{C}, (5)
ℒ⁡(𝝀)\displaystyle\mathcal{L}(\bm{\lambda}) =∫𝒒q~​(𝒒,𝝀)​log⁡q~​(𝒒,𝝀)p~​(𝒒)​𝑑𝒒\displaystyle=\int_{\bm{q}}\tilde{q}(\bm{q};\bm{\lambda})\log\frac{\tilde{q}(\bm{q};\bm{\lambda})}{\tilde{p}(\bm{q})}d\bm{q} (6)
=𝔼q~​[log⁡q~​(𝒒,𝝀)−log⁡p~​(𝒒)]\displaystyle=\mathbb{E}_{\tilde{q}}[\log\tilde{q}(\bm{q};\bm{\lambda})-\log\tilde{p}(\bm{q})] (7)
≈1N​∑n=1N(log⁡q~​(𝒒(n),𝝀)−log⁡p~​(𝒒(n))),\displaystyle\approx\frac{1}{N}\sum_{n=1}^{N}\big(\log\tilde{q}(\bm{q}^{(n)};\bm{\lambda})-\log\tilde{p}(\bm{q}^{(n)})\big), (8)
with𝒒(n)∼q~(⋅|𝝀).\displaystyle\mathrm{with}\quad\bm{q}^{(n)}\sim\tilde{q}(\,\cdot\,|\bm{\lambda}).

Zero forcing properties of minimizing D KL ( q | | p )

Reverse KL penalizes assigning probability where the target has little mass. An insufficiently expressive approximation may therefore omit valid modes. Variational fitting does not itself guarantee complete coverage.

Mixture model variational distribution

A Gaussian mixture represents disconnected solution families. Optimizing expectations separately within components improves sampling efficiency and permits different means, covariances, and mixture weights. More components increase computation.

Equations 9–11
q~(𝒒|𝝀)=∑k=1Kπkq~k(𝒒|𝝀k),∑k=1Kπk=1,\displaystyle\tilde{q}(\bm{q}\arrowvert\bm{\lambda})=\sum_{k=1}^{K}\pi_{k}\,\tilde{q}_{k}(\bm{q}\arrowvert\bm{\lambda}_{k}),\quad\sum_{k=1}^{K}\pi_{k}=1, (9)
ℒ⁡(𝝀)\displaystyle\mathcal{L}(\bm{\lambda}) =𝔼q~​[log⁡q~​(𝒒,𝝀)−log⁡p~​(𝒒)]\displaystyle=\mathbb{E}_{\tilde{q}}[\log\tilde{q}(\bm{q};\bm{\lambda})-\log\tilde{p}(\bm{q})] (10)
=∑k=1Kπk​𝔼q~k​[log⁡q~​(𝒒,𝝀)−log⁡p~​(𝒒)],\displaystyle=\sum_{k=1}^{K}\pi_{k}\mathbb{E}_{\tilde{q}_{k}}[\log\tilde{q}(\bm{q};\bm{\lambda})-\log\tilde{p}(\bm{q})], (11)
Refer to caption
Figure 3: The product of experts is an unnormalized density. As in Fig. 2, the density shown in (a) results from an elongated Gaussian defined on the end-effector of a 2-DoF planar robot. For control applications, such as keeping the robot in this distribution, accessing the unnormalized density is sufficient. For other applications, such as training a PoE, sampling from this distribution, evaluating the normalizing constant and its gradient might be needed. Using variational inference, the PoE can be approximated by a tractable distribution, as a mixture of banana-shaped distributions (b) or Gaussians (c). On purpose, only 5 mixture components were used in the illustrations. A more precise approximation can be achieved with a higher number.

2.3 Training PoEs

Learning compares expert statistics on demonstrated configurations with statistics under the model. Contrastive divergence approximates the latter using short data-initialized chains; the proposed method alternates fitting a variational mixture and updating expert parameters. Importance weights reduce approximation bias, and optional sampling refinements improve robustness.

Equations 12–16
∂log⁡p⁡(𝒒|𝜽1,…,𝜽M)∂𝜽𝒎=∂log⁡pm​(𝒒|𝜽m)∂𝜽𝒎−∂log⁡𝒞⁡(𝜽1,…,𝜽M)∂𝜽m.\displaystyle\begin{split}\frac{\partial\log p(\bm{q}|\bm{\theta}_{1},\ldots,\bm{\theta}_{M})}{\partial\bm{\theta_{m}}}=&{}\frac{\partial\log p_{m}(\bm{q}|\bm{\theta}_{m})}{\partial\bm{\theta_{m}}}-\\ &{}\frac{\partial\log\mathcal{C}(\bm{\theta}_{1},\ldots,\bm{\theta}_{M})}{\partial\bm{\theta}_{m}}.\end{split} (12)
∂log⁡𝒞⁡(𝜽1,…,𝜽M)∂𝜽m=∫cp(𝒄|𝜽1,…,𝜽M)​∂log⁡pm​(𝒄|𝜽m)∂𝜽𝒎​d​𝒄,\displaystyle\begin{split}\frac{\partial\log\mathcal{C}(\bm{\theta}_{1},\ldots,\bm{\theta}_{M})}{\partial\bm{\theta}_{m}}&{}=\\ \int_{c}p(\bm{c}|&{}\bm{\theta}_{1},\ldots,\bm{\theta}_{M})\,\frac{\partial\log p_{m}(\bm{c}|\bm{\theta}_{m})}{\partial\bm{\theta_{m}}}\,d\bm{c},\end{split} (13)
⟨∂log⁡p⁡(𝒒)∂𝜽⟩𝑸\displaystyle\Big<\frac{\partial\log p(\bm{q})}{\partial\bm{\theta}}\Big>_{\bm{Q}} =⟨∂log⁡p~​(𝒒)∂𝜽⟩𝑸−⟨∂log⁡p~​(𝒒)∂𝜽⟩p⁡(𝒒),\displaystyle=\Big<\frac{\partial\log\tilde{p}(\bm{q})}{\partial\bm{\theta}}\Big>_{\bm{Q}}\,-\Big<\frac{\partial\log\tilde{p}(\bm{q})}{\partial\bm{\theta}}\Big>_{p(\bm{q})}, (14)
⟨∂log⁡p~​(𝒒)∂𝜽⟩p⁡(𝒒)≈∑n=1Nwn∂log⁡p~​(𝒒(n))∂𝜽/∑n=1Nwn\displaystyle\Big<\frac{\partial\log\tilde{p}(\bm{q})}{\partial\bm{\theta}}\Big>_{p(\bm{q})}\approx\sum_{n=1}^{N}w_{n}\frac{\partial\log\tilde{p}(\bm{q}^{(n)})}{\partial\bm{\theta}}/\sum_{n=1}^{N}w_{n} (15)
with𝒒(n)∼q~(.|𝝀)andwn=p~​(𝒒(n))q~​(𝒒(n)|𝝀)\displaystyle\mathrm{with}\quad\bm{q}^{(n)}\sim\tilde{q}(\,.\,|\bm{\lambda})\quad\mathrm{and}\quad w_{n}=\frac{\tilde{p}(\bm{q}^{(n)})}{\tilde{q}(\bm{q}^{(n)}|\bm{\lambda})} (16)

Initializing PoEs

Experts with tractable likelihoods are first fitted independently. Those fits initialize the subsequent joint learning, reducing the search burden without replacing product normalization.

3 Product of Experts with nullspace filter (PoENS)

PoENS projects secondary expert gradients into the primary task’s Jacobian null space. This prevents secondary updates from directly changing the protected task directions and helps explain why a desired secondary goal was not reached.

Unlike an ordinary PoE, this construction is supplied as a filtered gradient field, without an evaluated scalar log-density. The paper uses stochastic variational updates compatible with that field. Its examples support the proposed approximation; they are not a general proof that arbitrary filtered fields define a normalized density.

Equations 17–21
𝒚m=𝒯m​(𝒒),\displaystyle\bm{y}_{m}=\mathcal{T}_{m}(\bm{q}), (17)
𝒚˙m=𝑱m​(𝒒)​𝒒˙,\displaystyle\dot{\bm{y}}_{m}=\bm{J}_{m}(\bm{q})\,\dot{\bm{q}}, (18)
𝒒˙=𝑱m†​(𝒒)​𝒚˙m+(𝑰−𝑱m†​(𝒒)​𝑱m​(𝒒))​𝒛,\displaystyle\dot{\bm{q}}=\bm{J}_{m}^{\dagger}(\bm{q})\ \dot{\bm{y}}_{m}+\big(\bm{I}-\bm{J}_{m}^{\dagger}(\bm{q})\bm{J}_{m}(\bm{q})\big)\bm{z}, (19)
𝑱m​(𝒒)​(𝑰−𝑱m†​(𝒒)​𝑱m​(𝒒))​𝒛=(𝑱m​(𝒒)−𝑱m​(𝒒))​𝒛=𝟎.\bm{J}_{m}(\bm{q})\big(\bm{I}-\bm{J}_{m}^{\dagger}(\bm{q})\bm{J}_{m}(\bm{q})\big)\,\bm{z}=\big(\bm{J}_{m}(\bm{q})-\bm{J}_{m}(\bm{q})\big)\,\bm{z}=\bm{0}.
∂log⁡p~​(𝒒|𝜽1,…,𝜽M)∂𝒒\displaystyle\frac{\partial\log\tilde{p}(\bm{q}|\bm{\theta}_{1},\ldots,\bm{\theta}_{M})}{\partial\bm{q}} =∑m∂log⁡p⁡(𝒯m​(𝒒)|𝜽m)∂𝒒\displaystyle=\sum_{m}\frac{\partial\log p(\mathcal{T}_{m}(\bm{q})|\bm{\theta}_{m})}{\partial\bm{q}}
=∑m∂log⁡p⁡(𝒚|𝜽m)∂𝒚​𝑱m​(𝒒).\displaystyle=\sum_{m}\frac{\partial\log p(\bm{y}|\bm{\theta}_{m})}{\partial\bm{y}}\bm{J}_{m}(\bm{q}). (20)
∂log⁡p⁡(𝒯m​(𝒒)|𝜽1,𝜽2)∂𝒒=\displaystyle\frac{\partial\log p(\mathcal{T}_{m}(\bm{q})|\bm{\theta}_{1},\bm{\theta}_{2})}{\partial\bm{q}}\qquad=
∂log⁡p⁡(𝒚|𝜽1)∂𝒚​𝑱1​(𝒒)+∂log⁡p⁡(𝒚|𝜽2)∂𝒚​𝑱2​(𝒒)​𝑵1​(𝒒).\displaystyle\frac{\partial\log p(\bm{y}|\bm{\theta}_{1})}{\partial\bm{y}}\bm{J}_{1}(\bm{q})+\frac{\partial\log p(\bm{y}|\bm{\theta}_{2})}{\partial\bm{y}}\bm{J}_{2}(\bm{q})\bm{N}_{1}(\bm{q}). (21)
Refer to caption
Figure 4: Densities and derivatives of a two-expert problem. A 2-DoF planar robot is considered as in Fig. 2. The first expert is an elongated Gaussian distribution on its end-effector. The second expert acts in joint angles and defines a preferred configuration. (a) The density of the expert log⁡p1​(𝒯1​(𝒒)|𝜽1)\log p_{1}(\mathcal{T}_{1}(\bm{q})|\bm{\theta}_{1}) is shown as a colormap. The derivative ∂log⁡p⁡(𝒚|𝜽1)/∂𝒚​𝑱1​(𝒒)\partial\log p(\bm{y}|\bm{\theta}_{1})/\partial\bm{y}\,\bm{J}_{1}(\bm{q}) is displayed as streamlines. (b) The same is done with the second objective in configuration space. (c) The derivative of the second experts is this time filtered as ∂log⁡p⁡(𝒚|𝜽2)/∂𝒚​𝑱2​(𝒒)​𝑵1​(𝒒)\partial\log p(\bm{y}|\bm{\theta}_{2})/\partial\bm{y}\,\bm{J}_{2}(\bm{q})\bm{N}_{1}(\bm{q}).
Refer to caption
Figure 5: 5-DoF bimanual planar robot with two forward kinematics objectives. Variational inference is used to generate the displayed samples. (a) The two tasks are compatible and the distribution of solution is approximated. (b) No nullspace, the two tasks are of the same importance. (c) The gradient of the objective of the orange arm is projected onto the nullspace of the Jacobian of the forward kinematics of the blue arm. This is achieved with stochastic variational inference that only requires to evaluate the gradient of the unnormalized density.

4 Useful distributions and transformations for robotics

The following toolkit supplies transformations and expert scores for common robotic constraints.

4.1 Transformations

Transformations may be known from kinematics, partly unknown, or learned. Joint training can estimate transformation parameters together with expert preferences.

Forward kinematics (FK)

Forward kinematics maps joints into link poses, possibly relative to an object. Unknown object frames or a held tool’s fixed offset can be learned when they provide a compact explanation of demonstrated configurations.

Equation 22
𝒯m​(𝒒)=𝑭R​(𝒒)​𝒅+𝑭x​(𝒒),\displaystyle\mathcal{T}_{m}(\bm{q})=\bm{F}_{R}(\bm{q})\bm{d}+\bm{F}_{x}(\bm{q}), (22)

Manipulability

Manipulability descriptors encode directions in which motion or force is easy. Their physically attainable values depend on kinematics and higher-priority tasks. Joint learning can account for that restricted support instead of treating every mathematical matrix as attainable.

Equations 23–24
𝒯m​(𝒒)=det(𝑱⁡(𝒒)​𝑱​(𝒒)⊤),\displaystyle\mathcal{T}_{m}(\bm{q})=\sqrt{\det(\bm{J}(\bm{q})\bm{J}(\bm{q})^{\scriptscriptstyle\top})}, (23)
𝒯m​(𝒒)=(𝑱⁡(𝒒)​𝑱​(𝒒)⊤)−1,\displaystyle\mathcal{T}_{m}(\bm{q})=\big(\bm{J}(\bm{q})\bm{J}(\bm{q})^{\scriptscriptstyle\top}\big)^{-1}, (24)

Relative distances

Distances between robot points and targets or obstacles offer an alternative task representation. They can complement pose constraints in cluttered environments.

Center of Mass (CoM)

Link masses and kinematics determine the center of mass. For static equilibrium on level ground, its projection should lie within the support polygon.

Jacobian pseudoinverse iterations

Repeated Jacobian-pseudoinverse corrections can be included inside a transformation to approximate very tight constraints. This improves concentration near a solution manifold but can bias mass toward locally linear regions and adds differentiation cost.

Equations 25–28
𝒫⁡(𝒒)\displaystyle\mathcal{P}(\bm{q}) =𝒒+J​(𝒒)†​(𝒑^−𝑭⁡(𝒒)),\displaystyle=\bm{q}+J(\bm{q})^{\dagger}\Big(\hat{\bm{p}}-\bm{F}(\bm{q})\Big), (25)
𝒫0​(𝒒)\displaystyle\mathcal{P}_{0}(\bm{q}) =𝒫⁡(𝒒),\displaystyle=\mathcal{P}(\bm{q}), (26)
𝒫n+1​(𝒒)\displaystyle\mathcal{P}_{n+1}(\bm{q}) =𝒫​(𝒫n−1​(𝒒)).\displaystyle=\mathcal{P}\Big(\mathcal{P}_{n-1}(\bm{q})\Big). (27)
pm​(𝒒)∝𝒩⁡(𝒫N​(𝒒)|𝒑^,σ​𝑰)\displaystyle p_{m}(\bm{q})\propto\mathcal{N}\Big(\mathcal{P}_{N}(\bm{q})\Big|\,\hat{\bm{p}},\sigma\bm{I}\Big) (28)

Neural network

Invertible neural transformations can map complex demonstrated shapes to simple expert distributions. Change-of-variable likelihoods provide initialization, while regularizing the inverse Jacobian discourages abrupt distortion. A diffeomorphism preserves the underlying expert’s isolated optimum; the full multi-expert robot problem can still have multiple solutions.

Equations 29–30
p⁡(𝒒)=pm​(𝒯m​(𝒒))​|det(∂𝒯m​(𝒒)∂𝒒⊤)|,\displaystyle p(\bm{q})=p_{m}\big(\mathcal{T}_{m}(\bm{q})\big)\,\Big|\det\Big(\frac{\partial\mathcal{T}_{m}(\bm{q})}{\partial\bm{q}^{\scriptscriptstyle\top}}\Big)\Big|, (29)
𝔼pr​(𝒚)​[∫𝒓|∂𝒯m−1​(𝒚)∂𝒚⊤−∂𝒯m−1​(𝒚+α​𝒘)∂(𝒚+α​𝒘)⊤|​𝑑𝒘],\displaystyle\mathbb{E}_{p_{r}(\bm{y})}\Bigg[\int_{\bm{r}}\,\Big|\frac{\partial\mathcal{T}_{m}^{-1}(\bm{y})}{\partial\bm{y}^{\scriptscriptstyle\top}}-\frac{\partial\mathcal{T}_{m}^{-1}(\bm{y}+\alpha\bm{w})}{\partial(\bm{y}+\alpha\bm{w})^{\scriptscriptstyle\top}}\Big|\,d\bm{w}\Bigg], (30)
Refer to caption
Figure 6: For greater expressiveness, an invertible neural network can be used as an expert transformation 𝒯m\mathcal{T}_{m}. Instead of optimizing the parameters of the expert density pmp_{m}, which can be set as a zero-centered and unit covariance Gaussian (right), the parameters of the network can be trained. The neural network becomes a transformation under which the dataset (N shape, left) becomes a simple distribution (Gaussian, right). The densities are shown as a colormap with isolines. A grid that undergoes the inverse transformation is also shown. This approach allows complex attractors or guiding distributions to be learned, where it is easy to control the robot in the transformed space.

4.2 Distributions

Different expert families express point targets, orientations, trajectories, inequality constraints, and optional objectives.

Multivariate normal distribution (MVN)

Gaussian experts have quadratic negative log-likelihoods, making them convenient for inverse kinematics and quadratic control. Their covariance determines directional precision.

Equation 31
𝒩⁡(𝒒|𝝁,𝚺)∝exp⁡(−12​(𝒒−𝝁)⊤​𝚺−1​(𝒒−𝝁)),\mathcal{N}(\bm{q}|\bm{\mu},\bm{\Sigma})\propto\exp\Big(-\frac{1}{2}(\bm{q}-\bm{\mu})^{{\scriptscriptstyle\top}}\bm{\Sigma}^{-1}(\bm{q}-\bm{\mu})\Big), (31)

Matrix Bingham—von Mises—Fisher distributions (BMF)

Matrix Bingham–von Mises–Fisher scores describe rotations without requiring a separate orientation normalizer. Structured quadratic forms can couple positions and vectorized rotation matrices, with positive-definiteness constraints controlling valid covariance parametrizations.

Equations 32–36
pBMF​(𝑸|𝑨,𝑩,𝑪)∝exp⁡(tr⁡(𝑪⊤​𝑸+𝑩​𝑸⊤​𝑨​𝑸)),p_{\mathrm{BMF}}(\bm{Q}|\bm{A},\bm{B},\bm{C})\propto\exp\Big(\operatorname{tr}(\bm{C}^{{\scriptscriptstyle\top}}\bm{Q}+\bm{B}\,{\bm{Q}}^{{\scriptscriptstyle\top}}\bm{A}\,\bm{Q})\Big), (32)
∂l⁡(𝑸)∂𝑸=𝑪T+2​𝑨​𝑸​𝑩=0,\displaystyle\frac{\partial l(\bm{Q})}{\partial\bm{Q}}=\bm{C}^{T}+2\bm{A}\bm{Q}\bm{B}=0, (33)
𝒩⁡(vec⁡(𝑸)|vec⁡(𝑨−1​𝑪T​𝑩−1),(𝑨⊗𝑩)−1).\mathcal{N}\Big(\mathrm{vec}(\bm{Q})\Big|\,\mathrm{vec}(\bm{A}^{-1}\bm{C}^{T}\bm{B}^{-1}),(\bm{A}\otimes\bm{B})^{-1}\Big). (34)
𝒩⁡([𝒒vec⁡(𝑸)]|[𝝁vec⁡(𝑨−1​𝑪T​𝑩−1)]⊤,[𝚺𝒙​𝒙𝚺𝒙​𝑿𝚺𝑿​𝒙(𝑨⊗𝑩)−1]),\footnotesize\mathcal{N}\Bigg(\begin{bmatrix}\bm{q}\\ \mathrm{vec}({\bm{Q}})\end{bmatrix}\Bigg|\\ \begin{bmatrix}\bm{\mu}\\ \mathrm{vec}(\bm{A}^{-1}\bm{C}^{T}\bm{B}^{-1})\end{bmatrix}^{{\scriptscriptstyle\top}}\!,\!\begin{bmatrix}\bm{\Sigma}_{\bm{xx}}&\bm{\Sigma}_{\bm{xX}}\\ \bm{\Sigma}_{\bm{Xx}}&(\bm{A}\otimes\bm{B})^{-1}\\ \end{bmatrix}\Bigg),\normalsize (35)
𝚺≻0⇔𝚺𝒙​𝒙≻0,(𝑨⊗𝑩)−1−𝚺𝑿​𝒙​𝚺𝒙​𝒙−1​𝚺𝒙​𝑿≻0.\displaystyle\bm{\Sigma}\succ 0\iff\bm{\Sigma}_{\bm{xx}}\succ 0,(\bm{A}\otimes\bm{B})^{-1}-\bm{\Sigma}_{\bm{Xx}}\bm{\Sigma}_{\bm{xx}}^{-1}\bm{\Sigma}_{\bm{xX}}\succ 0. (36)

Matrix normal distribution

Matrix-normal experts reduce parameter count by separating row and column covariance. They are useful when a transformation naturally produces a matrix.

Equations 37–38
ℳ​𝒩​(𝑸|𝑴,𝑼,𝑽)∝exp⁡(−12​tr⁡[𝑽−𝟏​(𝑸−𝑴)⊤​𝑼−1​(𝑸−𝑴)]),\mathcal{MN}(\bm{Q}|\bm{M},\bm{U},\bm{V}){}\propto\\ \exp\Big(-\frac{1}{2}\operatorname{tr}\big[\bm{V^{-1}}(\bm{Q}-\bm{M})^{{\scriptscriptstyle\top}}\bm{U}^{-1}(\bm{Q}-\bm{M})\big]\Big), (37)
vec⁡(𝑸)∝𝒩⁡(vec⁡(𝑴),(𝑽⊗𝑼)−1),\operatorname{vec}(\bm{Q})\propto\mathcal{N}\big(\!\operatorname{vec}(\bm{M}),(\bm{V}\otimes\bm{U})^{-1}\big), (38)

Probabilistic movement primitives

ProMP experts extend the framework from static configurations to trajectory distributions. A mixture of joint-space ProMPs can approximate products involving task-space and joint-space trajectories, including multiple kinematic solution families. [47]

Equations 39–40
pProMP​(𝒒t|𝚽t,𝒘,𝚺𝒙)\displaystyle p_{\textsc{ProMP}}(\bm{q}_{t}|\bm{\Phi}_{t},\bm{w},\bm{\Sigma_{x}}) =𝒩⁡(𝒒t|𝚿t⊤​𝒘,𝚺𝒙),\displaystyle=\mathcal{N}(\bm{q}_{t}|\bm{\Psi}_{t}^{\scriptscriptstyle\top}\bm{w},\bm{\Sigma_{x}}), (39)
pProMP​(𝒒t|𝚽t,𝝁𝒘,𝚺𝒘,𝚺𝒙)=∫𝒩⁡(𝒒t|𝚿t⊤​𝒘,𝚺𝒙)​𝒩​(𝒘|𝝁𝒘,𝚺𝒘)​d𝒘=𝒩⁡(𝒒t|𝚿t⊤​𝝁𝒘,𝚿t⊤​𝚺𝒘​𝚿t+𝚺𝒙).\small p_{\textsc{ProMP}}(\bm{q}_{t}|\bm{\Phi}_{t},\bm{\mu}_{\bm{w}},\bm{\Sigma}_{\bm{w}},\bm{\Sigma_{x}})=\\ \int\mathcal{N}(\bm{q}_{t}|\bm{\Psi}_{t}^{\scriptscriptstyle\top}\bm{w},\bm{\Sigma_{x}})\,\mathcal{N}(\bm{w}|\bm{\mu}_{\bm{w}},\bm{\Sigma}_{\bm{w}})d\bm{w}=\\ \mathcal{N}(\bm{q}_{t}|\bm{\Psi}_{t}^{\scriptscriptstyle\top}\bm{\mu}_{\bm{w}},\bm{\Psi}_{t}^{\scriptscriptstyle\top}\bm{\Sigma}_{\bm{w}}\bm{\Psi}_{t}+\bm{\Sigma_{x}}). (40)
Refer to caption
Figure 7: To encode distributions of trajectories, probabilistic movement primitives (ProMP) can be used as experts in a PoE. In this example, a product of two ProMP experts is considered: one defines a joint angle trajectory, while the other defines the trajectory of the end-effector. A 3-DoF planar robot is used in this example. The product of ProMPs is approximated using variational inference as a mixture of ProMPs in configuration space. Samples of trajectories of the end-effector are displayed, as well as a full sequence of configurations. Even if each expert is Gaussian, the product of ProMPs is multimodal because of the planar robot kinematics.

Approximate distributions in the tangent space of manifolds

Gaussian scores in a manifold’s tangent space offer a generic treatment of orientations and other geometric data. The logarithmic map must be differentiable for the proposed training procedure.

Equation 41
𝒩⁡(𝒒|𝝁,𝚺)∝exp⁡(−12​Log𝝁​(𝒒)⊤​𝚺−1​Log𝝁​(𝒒)).\mathcal{N}(\bm{q}|\bm{\mu},\bm{\Sigma})\propto\exp\Big(-\frac{1}{2}\mathrm{Log}_{\bm{\mu}}(\bm{q})^{{\scriptscriptstyle\top}}\,\bm{\Sigma}^{-1}\,\mathrm{Log}_{\bm{\mu}}(\bm{q})\Big). (41)

Cumulative distribution functions (CDF)

Cumulative Gaussian probabilities provide smooth inequality scores for joint limits, obstacles, and equilibrium. Their scale controls the transition margin. These probabilistic penalties alone are not hard safety guarantees.

Equation 42
p⁡(x≤b),withx∼𝒩⁡(𝒯⁡(𝒒),σ2),\displaystyle p(x\leq b),\quad\text{with}\quad x\sim\mathcal{N}\Big(\mathcal{T}(\bm{q}),\sigma^{2}\,\Big), (42)

Uni-Gauss distributions

A uniform–Gaussian mixture allows an objective to be effectively inactive. It gives softer prioritization than null-space projection. Enumerating active-task combinations is practical only for small task sets; marginalized mixtures avoid that combinatorial expansion.

Equation 43
pUG,m​(𝒒)=πm​pm​(𝒒)+1−πmc,\displaystyle p_{\mathrm{UG},m}(\bm{q})=\pi_{m}p_{m}(\bm{q})+\frac{1-\pi_{m}}{c}, (43)

5 Control

One controller seeks and maintains high-density configurations. Another plans motion whose spatial coverage approximates the desired distribution.

5.1 Optimal control

LQT augments the robot state with expert task coordinates and linearizes their dynamics. Gaussian expert costs become quadratic; other scores require local approximation. Learned precision sets feedback strength, allowing compliance in unconstrained directions.

A slower loop recomputes the controller in 2–10 milliseconds, while a faster loop executes the latest command in under one millisecond in the reported implementation. Both algorithms and the disturbance-response example are preserved.

Equations 44–58
𝝃t+1=𝑨t​𝝃t+𝑩t​𝒖t,\displaystyle\bm{\xi}_{t+1}=\bm{A}_{t}\bm{\xi}_{t}+\bm{B}_{t}\bm{u}_{t}, (44)
J=12​∑t=1N−1((𝝃t−𝒛t)⊤​𝑸t​(𝝃k−𝒛t)+𝒖t⊤​𝑹​𝒖t),\displaystyle J=\frac{1}{2}\sum_{t=1}^{N-1}\Big((\bm{\xi}_{t}-\bm{z}_{t})^{\scriptscriptstyle\top}\bm{Q}_{t}(\bm{\xi}_{k}-\bm{z}_{t})+\bm{u}_{t}^{\scriptscriptstyle\top}\bm{R}\bm{u}_{t}\Big), (45)
𝒖t=−𝑲t​𝝃t+𝑲tv​𝒗t+1,\displaystyle\bm{u}_{t}=-\bm{K}_{t}\bm{\xi}_{t}+\bm{K}^{v}_{t}\bm{v}_{t+1}, (46)
[𝒒t+1𝒒˙t+1]=𝑨t​[𝒒t𝒒˙t]+𝑩t​𝒖t,\displaystyle\begin{bmatrix}\bm{q}_{t+1}\\ \dot{\bm{q}}_{t+1}\\ \end{bmatrix}=\bm{A}_{t}\begin{bmatrix}\bm{q}_{t}\\ \dot{\bm{q}}_{t}\\ \end{bmatrix}+\bm{B}_{t}\bm{u}_{t}, (47)
𝝃t=[𝒯1​(𝒒t)𝒯M​(𝒒t)𝒒t𝒒˙t]=[𝒚1,t𝒚M,t𝒒t𝒒˙t].\displaystyle\bm{\xi}_{t}=\begin{bmatrix}\mathcal{T}_{1}(\bm{q}_{t})\\ \vdots\\ \mathcal{T}_{M}(\bm{q}_{t})\\ \hline\cr\bm{q}_{t}\\ \dot{\bm{q}}_{t}\end{bmatrix}=\begin{bmatrix}\bm{y}_{1,t}\\ \vdots\\ \bm{y}_{M,t}\\ \hline\cr\bm{q}_{t}\\ \dot{\bm{q}}_{t}\\ \end{bmatrix}. (48)
𝒚m,t+1\displaystyle\bm{y}_{m,t+1} ≈𝒚m,t+𝑱m​(𝒒t)​(𝒒t+1−𝒒t)\displaystyle\approx\bm{y}_{m,t}+\bm{J}_{m}(\bm{q}_{t})\,(\bm{q}_{t+1}-\bm{q}_{t})
≈𝒚m,t+𝑱m​(𝒒t)​𝒒˙t​Δ​t,\displaystyle\approx\bm{y}_{m,t}+\bm{J}_{m}(\bm{q}_{t})\,\dot{\bm{q}}_{t}\,\Delta t, (49)
[𝒚1,t+1𝒚M,t+1𝒒t+1𝒒˙t+1]⏞𝝃t+1=[𝑰d1⋯𝟎 0Δ​t​𝑱^1⋮⋱⋮⋮⋮𝟎⋯𝑰dM 0Δ​t​𝑱^M𝟎⋯𝟎𝑨t]⏞𝑨t~​[𝒚1,t𝒚M,t𝒒t𝒒˙t]⏞𝝃t+\displaystyle\overbrace{\begin{bmatrix}\bm{y}_{1,t+1}\\ \vdots\\ \bm{y}_{M,t+1}\\ \hline\cr\bm{q}_{t+1}\\ \dot{\bm{q}}_{t+1}\\ \end{bmatrix}}^{\bm{\xi}_{t+1}}=\overbrace{\left[\begin{array}[]{ccc|cc}\bm{I}_{d_{1}}&\cdots&\bm{0}&\,\bm{0}&\,\Delta t\hat{\bm{J}}_{1}\\ \vdots&\ddots&\vdots&\vdots&\vdots\\ \bm{0}&\cdots&\bm{I}_{d_{M}}&\,\bm{0}&\,\Delta t\hat{\bm{J}}_{M}\\ \hline\cr\bm{0}&\cdots&\bm{0}&\lx@intercol\hfil\bm{A}_{t}\hfil\lx@intercol\end{array}\right]}^{\tilde{\bm{A}_{t}}}\overbrace{\begin{bmatrix}\bm{y}_{1,t}\\ \vdots\\ \bm{y}_{M,t}\\ \hline\cr\bm{q}_{t}\\ \dot{\bm{q}}_{t}\\ \end{bmatrix}}^{\bm{\xi}_{t}}+
[[𝟎𝑱^1]​𝑩t[𝟎𝑱^M]​𝑩t𝑩t]⏞𝑩~t​𝒖t.\displaystyle\overbrace{\begin{bmatrix}\begin{bmatrix}\bm{0}&\hat{\bm{J}}_{1}\end{bmatrix}\bm{B}_{t}\\ \vdots\\ \begin{bmatrix}\bm{0}&\hat{\bm{J}}_{M}\end{bmatrix}\bm{B}_{t}\\ \hline\cr\bm{B}_{t}\end{bmatrix}}^{\tilde{\bm{B}}_{t}}\bm{u}_{t}. (54)
J=∑t=1N−1(−∑m=1Mlogpm(𝒚m,t)+12𝒒˙t⊤𝑸~𝒒˙t+12𝒖t⊤𝑹𝒖t),\displaystyle J=\sum_{t=1}^{N-1}\Big(-\sum_{m=1}^{M}\log p_{m}(\bm{y}_{m,t})+\frac{1}{2}\dot{\bm{q}}_{t}^{\scriptscriptstyle\top}\tilde{\bm{Q}}\dot{\bm{q}}_{t}+\frac{1}{2}\,\bm{u}_{t}^{\scriptscriptstyle\top}\bm{R}\bm{u}_{t}\Big), (55)
𝑸~=[ς1−2⋯0⋱0⋯ςP−2],𝑹=[τ1−2⋯0⋱0⋯τP−2],\displaystyle\tilde{\bm{Q}}=\begin{bmatrix}\varsigma_{1}^{-2}&\cdots&0\\ \vdots&\ddots&\vdots\\ 0&\cdots&\varsigma_{P}^{-2}\\ \end{bmatrix},\;\bm{R}=\begin{bmatrix}\tau_{1}^{-2}&\cdots&0\\ \vdots&\ddots&\vdots\\ 0&\cdots&\tau_{P}^{-2}\\ \end{bmatrix}, (56)
𝑸=[𝚺1−1⋯𝟎𝟎⋱𝟎⋯𝚺M−1𝟎𝟎⋯𝟎𝑸~],𝒛𝒕=[𝝁1𝝁M𝟎],\displaystyle\bm{Q}=\begin{bmatrix}\bm{\Sigma}_{1}^{-1}&\cdots&\bm{0}&\bm{0}\\ \vdots&\ddots&\vdots&\vdots\\ \bm{0}&\cdots&\bm{\Sigma}_{M}^{-1}&\bm{0}\\ \bm{0}&\cdots&\bm{0}&\tilde{\bm{Q}}\\ \end{bmatrix},\;\bm{z_{t}}=\begin{bmatrix}\bm{\mu}_{1}\\ \vdots\\ \bm{\mu}_{M}\\ \bm{0}\end{bmatrix}, (57)
𝒖t=∑m=1M−𝑲m,t𝒯m(𝝃t)+𝑲m,tv𝒗m,t+1.\displaystyle\bm{u}_{t}=\sum_{m=1}^{M}-\bm{K}_{m,t}\mathcal{T}_{m}(\bm{\xi}_{t})+\bm{K}^{v}_{m,t}\bm{v}_{m,t+1}. (58)
Algorithm 1 High-level loop (coded in Python)
1 set 𝑹\bm{R} and 𝑸~\tilde{\bm{Q}} as in (56)
2 compute 𝑸\bm{Q} and 𝒛\bm{z}, LQT cost from the expert distributions as in (57)
3 while controlling do
    4 (loop duration : 2−10​[ms]2-10[\mathrm{ms}])
    5 compute 𝑨\bm{A} and 𝑩\bm{B}, linearized model of the manipulator around current state
    6 get 𝒒\bm{q} current configuration of the robot
    7 foreach expert mm do
       8 compute 𝑱^m​(𝒒)\hat{\bm{J}}_{m}(\bm{q})
    9 end foreach
    10 compute 𝑨^\hat{\bm{A}} and 𝑩^\hat{\bm{B}}, augmented linear model as in (54)
    11 solve finite or infinite horizon LQT with constant parameters over the horizon
    12 publish 𝑲1\bm{K}_{1}, 𝑲1v\bm{K}^{v}_{1}, 𝒗2\bm{v}_{2}, LQT controller parameters
13 end while
Algorithm 2 Low-level control loop (coded in C++)
1 while controlling do
    2 (loop duration : <1​[ms]<1[\mathrm{ms}])
    3 read 𝑲1\bm{K}_{1}, 𝑲1v\bm{K}^{v}_{1}, 𝒗2\bm{v}_{2}, current LQT controller parameters
    4 get 𝒒\bm{q}, 𝒒˙\dot{\bm{q}} current configuration and velocity of the robot
    5 foreach expert mm do
       6 compute 𝒯m​(𝒒)\mathcal{T}_{m}(\bm{q}) (call Python functions)
    7 end foreach
    8 compute 𝝃\bm{\xi} as in (48)
    9 compute and apply torques or accelerations as 𝒖=−𝑲t​𝝃1+𝑲1v​𝒗2\bm{u}=-\bm{K}_{t}\bm{\xi}_{1}+\bm{K}^{v}_{1}\bm{v}_{2}
10 end while
Refer to caption
Figure 8: Example of 7-DoF manipulator controlled with an LQT. To show the time evolution, only the kinematic chain is displayed. The PoE is the intersection between a vertical plane (e1=0e_{1}=0) and a sphere. The two experts act on the end-effector. They are defined by a Gaussian distribution on e1e_{1} around 00 and by a Gaussian distribution on the log-distance to a target (black square). On the left, the evolution of the values of the joint angles is displayed over time. At t=2.35​[s]t=2.35[s], a perturbation in configuration space occurs. The robot converges to a new local mode of the PoE.

5.2 Ergodic control

Ergodic control instead matches the trajectory’s time-averaged coverage to the target density. A mixture around trajectory points models sensing or influence radius; velocity and acceleration penalties make the path executable. The same variational objective supports optimization without a normalized target.

Equations 59–60
DKL(Γ||p)\displaystyle D_{\mathrm{KL}}(\Gamma||p) =∫𝒒Γ⁡(𝒒)​log⁡Γ⁡(𝒒)p⁡(𝒒)​𝑑𝒒.\displaystyle=\int_{\bm{q}}\Gamma(\bm{q})\log\frac{\Gamma(\bm{q})}{p(\bm{q})}d\bm{q}. (59)
Γ⁡(𝒒)≈1N​∑t=0N−1𝒩⁡(𝒒|𝒒t,𝚺),\displaystyle\Gamma(\bm{q})\approx\frac{1}{N}\sum_{t=0}^{N-1}\mathcal{N}(\bm{q}|\bm{q}_{t},\bm{\Sigma}), (60)
Refer to caption
Figure 9: The trajectory of the robot is optimized to cover the PoE. The density is defined by a position constraint of the end-effector (expert 0) and a distance constraint (expert 1).

6 Experiments

The experiments test density estimation, hidden priorities, learned task transformations, and pose correlations. Distribution discrepancy is the main quantitative metric; it is not a task-success percentage.

6.1 Multimodal distributions

Three two-joint planar datasets contain thirty configurations each. Across random initializations, variational learning gives lower alpha-divergence than contrastive divergence: 0.067 versus 0.837 for the joint-target case, 0.016 versus 0.039 for the task-target case, and 0.014 versus 0.063 for their combination. Missing alternative kinematic modes explains much of the sampling baseline’s error.

Refer to caption
Figure 10: Samples of configurations for three tasks that illustrate the advantages of using variational inference with a mixture model for training a PoE. A 2-DoF planar robot is considered. (a) Joint angle target. (b) Operational space target (c) Operational space target and joint angle target.
Equations 61
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ2,\displaystyle\bm{q}\in\mathbb{R}^{2},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭𝒙​(𝒒)∈ℝ2,\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{\bm{x}}(\bm{q})\in\mathbb{R}^{2},
𝒚2=𝒯2​(𝒒)=𝒒∈ℝ2,\displaystyle\bm{y}_{2}=\mathcal{T}_{2}(\bm{q})=\bm{q}\in\mathbb{R}^{2},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁1,σ1​𝑰),\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{1},\sigma_{1}\bm{I}),
𝒚2∼𝒩⁡(𝝁2,σ2​𝑰),\displaystyle\bm{y}_{2}\sim\mathcal{N}(\bm{\mu}_{2},\sigma_{2}\bm{I}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad 𝝁1,σ1,𝝁2,σ2,\displaystyle\bm{\mu}_{1},\sigma_{1},\bm{\mu}_{2},\sigma_{2},
Dα=1/2(p||q~)=−2log∫p⁡(𝒒)​q~​(𝒒)d𝒒.\displaystyle D_{\alpha=1/2}(p||\tilde{q})=-2\log\int\sqrt{p(\bm{q})\tilde{q}(\bm{q})}d\bm{q}. (61)
Table 1: Quantitative evaluations of the learned distribution using contrastive divergence (CD) and our approach based on variational inference with a Gaussian mixture model (VI). The table shows alpha-divergence Dα=1/2D_{\alpha=1/2} measures between the different approximations and the ground-truth distribution, computed for the three tasks shown in Fig. 10.
Task (a) (b) (c)
CD 0.837 ±\pm 0.597 0.039 ±\pm 0.030 0.063 ±\pm 0.026
VI 0.067 ±\pm 0.060 0.016 ±\pm 0.022 0.014 ±\pm 0.004

6.2 Hierarchical tasks

These tests ask whether a model can recover a secondary target that demonstrations cannot satisfy because a primary objective takes precedence.

Planar robot

Planar bimanual reaching uses thirty samples for each of three primary targets. A separate three-joint test combines reaching with manipulability, using four target conditions. Providing the hierarchy lets PoENS recover goals more accurately than independent fits or an unprioritized product.

Table 2 reports alpha-divergence 0.094 for bimanual PoENS versus 0.258 for PoE and 1.814 for independent experts; corresponding manipulability values are 0.202, 0.630, and 0.812. The scalar manipulability comparison simplifies the richer baseline representation.

Refer to caption
Figure 11: Dataset for the bimanual task with a hierarchy. Three situations are given, where each end-effector should track the target of the same color (displayed as a square).
Refer to caption
Figure 12: Samples from the different models learned for the bimanual task with a hierarchy. Left: Using independent experts, the secondary task is not well understood and its variance is over-estimated. Center: Using a PoE, the target of the primary task is inferred to be further away. Right: Using a PoE with a nullspace structure, the targets are well recovered.
Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ5,\displaystyle\bm{q}\in\mathbb{R}^{5},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭R,𝒙​(𝒒)∈ℝ2,\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{R,\bm{x}}(\bm{q})\in\mathbb{R}^{2},
𝒚2=𝒯2​(𝒒)=𝑭L,𝒙​(𝒒)∈ℝ2,\displaystyle\bm{y}_{2}=\mathcal{T}_{2}(\bm{q})=\bm{F}_{L,\bm{x}}(\bm{q})\in\mathbb{R}^{2},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁1(i),σ1​𝑰)|i =0,…,2,\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{1}^{(i)},\sigma_{1}\bm{I})\,|\,\textit{i }=0,\dots,2,
𝒚2∼𝒩⁡(𝝁2,σ2​𝑰),\displaystyle\bm{y}_{2}\sim\mathcal{N}(\bm{\mu}_{2},\sigma_{2}\bm{I}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad {𝝁1(i)}i=0,…,2,𝝁2,σ1,σ2.\displaystyle\{\bm{\mu}_{1}^{(i)}\}_{i=0,\dots,2},\bm{\mu}_{2},\sigma_{1},\sigma_{2}.
Refer to caption
Figure 13: Dataset for the hierarchical task with a manipulability objective. The robot should track the target (displayed as a blue square) and maximize a manipulability measure. Four different targets are given.
Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ3,\displaystyle\bm{q}\in\mathbb{R}^{3},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭𝒙​(𝒒)∈ℝ2,\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{\bm{x}}(\bm{q})\in\mathbb{R}^{2},
y2=𝒯2​(𝒒)=\displaystyle y_{2}=\mathcal{T}_{2}(\bm{q})=
logdet(𝑱(𝒒)𝑱(𝒒)⊤)∈ℝ+,\displaystyle\qquad\log\det\big(\bm{J}(\bm{q})\bm{J}(\bm{q})^{{\scriptscriptstyle\top}}\big)\in\mathbb{R}^{+},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁1(i),σ1​𝑰)|i=0,…,3,\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{1}^{(i)},\sigma_{1}\bm{I})\,|\,i=0,\dots,3,
y2∼𝒩⁡(μ2,σ2),\displaystyle y_{2}\sim\mathcal{N}(\mu_{2},\sigma_{2}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad {𝝁1(i)}i=0,…,3,μ2,σ1,σ2,\displaystyle\{\bm{\mu}_{1}^{(i)}\}_{i=0,\dots,3},\mu_{2},\sigma_{1},\sigma_{2},
Refer to caption
Figure 14: Samples from the different models learned for the task with a manipulability objective. Left: Using independent experts, the secondary task is not well understood (the manipulability measure has a big variance in the dataset). Center: Using a PoE, the estimation of the principal objective is biased. Right: By providing the hierarchical structure, the two objectives are well understood.
Table 2: Quantitative evaluations of the quality of the learned distribution for the two planar tasks. The table shows alpha-divergence Dα=1/2D_{\alpha=1/2} measures between the different approximations and the ground-truth distribution are computed. (a) Bimanual task (b) Manipulability objective
Task (a) (b)
Independent experts 1.814 ±\pm 0.055 0.812 ±\pm 0.117
PoE 0.258 ±\pm 0.101 0.630 ±\pm 0.086
PoENS 0.094 ±\pm 0.024 0.202 ±\pm 0.067

Welding task

The Panda welding-like dataset prioritizes end-effector position over a preferred vertical orientation, with thirty configurations in each of three target conditions. PoENS gives the lowest reported discrepancy in all three. Unbiased squared-MMD estimates may be slightly negative near zero; these are estimator fluctuations, not negative distances.

Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ7,\displaystyle\bm{q}\in\mathbb{R}^{7},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭x​(𝒒)∈ℝ3,\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{x}(\bm{q})\in\mathbb{R}^{3},
𝒚2=𝒯2​(𝒒)=vec⁡(𝑭R​(𝒒))∈ℝ9,\displaystyle\bm{y}_{2}=\mathcal{T}_{2}(\bm{q})=\mathrm{vec}(\bm{F}_{R}(\bm{q}))\in\mathbb{R}^{9},
𝒚3=𝒯3​(𝒒)=𝒒∈ℝ7,\displaystyle\bm{y}_{3}=\mathcal{T}_{3}(\bm{q})=\bm{q}\in\mathbb{R}^{7},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁x(i),σx​𝑰)|i=0,…,3,\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{x}^{(i)},\sigma_{x}\bm{I})\,|\,i=0,\dots,3,
𝒚2∼𝒩⁡(𝝁R,σR​𝑰),\displaystyle\bm{y}_{2}\sim\mathcal{N}(\bm{\mu}_{R},\sigma_{R}\bm{I}),
𝒚3∼𝒩⁡(𝝁𝒒,σ𝒒​𝑰),\displaystyle\bm{y}_{3}\sim\mathcal{N}(\bm{\mu}_{\bm{q}},\sigma_{\bm{q}}\bm{I}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad {𝝁x(i)}i=0,…,3,σx,𝝁R,σR,𝝁𝒒,σ𝒒\displaystyle\{\bm{\mu}_{x}^{(i)}\}_{i=0,\dots,3},\sigma_{x},\bm{\mu}_{R},\sigma_{R},\bm{\mu}_{\bm{q}},\sigma_{\bm{q}}
Table 3: Quantitative evaluations of the quality of the learned distribution for the 7-DoF Panda robot (welding task). The table shows maximum mean discrepancy MMDu2\mathrm{MMD}_{u}^{2} measures between the dataset and different models for the 3 cases displayed in Fig. 17.
Case (0) (1) (2)
Independent experts 1.3​e−3{1.3}\mathrm{e}^{-3} 1.3​e−2{1.3}\mathrm{e}^{-2} 8.2​e−3{8.2}\mathrm{e}^{-3}
PoE 7.0​e−4{7.0}\mathrm{e}^{-4} 1.7​e−3{1.7}\mathrm{e}^{-3} 1.4​e−3{1.4}\mathrm{e}^{-3}
PoENS −9.8​𝐞−𝟔\bm{{-9.8}\mathrm{e}^{-6}} −8.5​𝐞−𝟓\bm{{-8.5}\mathrm{e}^{-5}} 4.5​𝐞−𝟒\bm{{4.5}\mathrm{e}^{-4}}

Robot waiter task

In physical tray serving, the robot prioritizes a level tray over placing a bottle within the other gripper. Ten samples in each of three hand-position conditions teach both target relationships and priority. A second gravity-compensated robot stands in for the person. The demonstration shows compliant reproduction, without reporting a large-scale success-rate evaluation.

Refer to caption
Refer to caption
Figure 15: Several demonstrations of the desired priority behavior are given.Top: Both position and orientation are achievable. Bottom: Position is only partially fulfilled, to allow the tray to remain parallel to the floor. The left robot arm is playing the role of a person who should grasp the bottle.
Refer to caption
Figure 16: The right arm should keep the tray horizontal and the bottle between the gripper fingers. With a PoENS model, the robot learns i) the tray pose with respect to the gripper and ii) that the tray orientation has a higher priority than the position. Top row: Reproduction when both position and orientation can be fulfilled. Bottom row: Reproduction when position reference is too far so the robot prioritizes orientation.
Refer to caption
Figure 17: Dataset and retrieved samples for the hierarchical task with the 7-DoF Panda robot (welding task). Only the kinematic chain of the robot is displayed, such that multiple samples can be shown. The dataset provides three different situations (displayed in blue, yellow and red) in which the robot should track a different target with its end-effector. In each situation, the orientation should be held vertical as a secondary task.

6.3 Learning transformations and conditional PoEs

Further experiments estimate partly unknown transformations and test whether explicit kinematic structure reduces sample requirements.

New end-effector

A seven-joint Panda learns the offset of a new tool and its target distribution from datasets of 3, 10, 30, 100, or 3,000 configurations. The paper compares PoE with a VAE, mixture model, invertible network, and two GAN variants. Structured PoE performs best in the reported distribution comparisons; task-space features also help the GAN.

The text describes this evaluation as squared MMD, while Figure 19’s caption calls it alpha-divergence. That inconsistency limits interpretation of the plotted metric label.

Refer to caption
Figure 18: Dataset and samples from the different models for a task involving an unknown end-effector. In this illustration, the models were trained with 3000 samples.
Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ7,\displaystyle\bm{q}\in\mathbb{R}^{7},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭R​(𝒒)​𝒅+𝑭x​(𝒒)\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{R}(\bm{q})\bm{d}+\bm{F}_{x}(\bm{q})
=𝑭d​(𝒒)∈ℝ3,\displaystyle\hskip 46.94704pt=\bm{F}_{d}(\bm{q})\in\mathbb{R}^{3},
𝒚2=𝒯2​(𝒒)=𝒒∈ℝ7,\displaystyle\bm{y}_{2}=\mathcal{T}_{2}(\bm{q})=\bm{q}\in\mathbb{R}^{7},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁x,σx​𝑰),\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{x},\sigma_{x}\bm{I}),
𝒚2∼𝒩⁡(𝝁𝒒,σ𝒒​𝑰),\displaystyle\bm{y}_{2}\sim\mathcal{N}(\bm{\mu}_{\bm{q}},\sigma_{\bm{q}}\bm{I}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad 𝒅,𝝁x,σx,𝝁𝒒,σ𝒒,\displaystyle\bm{d},\bm{\mu}_{x},\sigma_{x},\bm{\mu}_{\bm{q}},\sigma_{\bm{q}},
Refer to caption
Figure 19: Quantitative results for a task involving an unknown end-effector. The graphs show alpha-divergence measures (with α=1/2\alpha=1/2) between the dataset and the different models for the different sizes of the dataset.

Conditional distributions

Conditional learning uses three known targets with 1,000 configurations each. PoE transfers to a seen, interpolated, and distant target with much lower reported discrepancy than the VAE and GMR baselines. Sampling a new PoE condition requires a fresh approximation; evaluating its unnormalized control cost does not.

Refer to caption
Figure 20: Dataset and samples from the different models for a task involving an unknown end-effector. Three cases are reproduced. Left column: The distribution is sampled on a known target. Middle column: The target is in-between two given targets. Right column: The target is far from the given targets. The first and second rows show results for a conditional variational autoencoder and a PoE, respectively.
Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ7,\displaystyle\bm{q}\in\mathbb{R}^{7},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭R​(𝒒)​𝒅+𝑭x​(𝒒)\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{R}(\bm{q})\bm{d}+\bm{F}_{x}(\bm{q})
=𝑭d​(𝒒)∈ℝ3,\displaystyle\hskip 46.94704pt=\bm{F}_{d}(\bm{q})\in\mathbb{R}^{3},
𝒚2=𝒯2​(𝒒)=𝒒∈ℝ7,\displaystyle\bm{y}_{2}=\mathcal{T}_{2}(\bm{q})=\bm{q}\in\mathbb{R}^{7},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁x(i),σx​𝑰)|i=0,…,3,\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu}_{x}^{(i)},\sigma_{x}\bm{I})\,|\,i=0,\dots,3,
𝒚2∼𝒩⁡(𝝁𝒒,σ𝒒​𝑰),\displaystyle\bm{y}_{2}\sim\mathcal{N}(\bm{\mu}_{\bm{q}},\sigma_{\bm{q}}\bm{I}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad 𝒅,σx,𝝁𝒒,σ𝒒.\displaystyle\bm{d},\sigma_{x},\bm{\mu}_{\bm{q}},\sigma_{\bm{q}}.
Table 4: Quantitative results for the task with the new end-effector. The table shows maximum mean discrepancy MMDu2\mathrm{MMD}_{u}^{2} measures between the dataset and the different models for the different cases. (a) The distribution is sampled on a known target. (b) The target is in-between two given targets. (c) Generalization with respect to a new target far from the given targets.
Case (a) (b) (c)
VAE 8.8​e−4{8.8}\mathrm{e}^{-4} 2.2​e−4{2.2}\mathrm{e}^{-4} 1.0​e−2{1.0}\mathrm{e}^{-2}
GMR 2.7​e−6{2.7}\mathrm{e}^{-6} 5.1​e−6{5.1}\mathrm{e}^{-6} 9.0​e−2{9.0}\mathrm{e}^{-2}
PoE 3.5​𝐞−𝟗\bm{{3.5}\mathrm{e}^{-9}} −2.2​𝐞−𝟕\bm{{-2.2}\mathrm{e}^{-7}} 3.4​𝐞−𝟕\bm{{3.4}\mathrm{e}^{-7}}

6.4 End-effector position and rotation correlations

Three synthetic datasets of 3,000 configurations couple end-effector height and orientation. The model uses a twelve-dimensional pose representation with diagonal plus rank-one covariance. It compares against independently fitted vectorized Gaussians and a Riemannian representation.

PoE has the highest reported test p-values and the lowest squared MMD in cases (b) and (c). In case (a), Table 5 gives the Riemannian model a smaller MMD, 0.00103 versus 0.0015, despite the prose claiming PoE wins every case. The preserved table supports this narrower conclusion.

Refer to caption
Figure 21: Samples where the height of the end-effector is correlated with its rotation along the e1e_{1}-axis. The transformation matrix of the end-effector is displayed.
Equation
state:\displaystyle\mathrm{state:}\quad 𝒒∈ℝ7,\displaystyle\bm{q}\in\mathbb{R}^{7},
transformations:\displaystyle\mathrm{transformations:}\quad 𝒚1=𝒯1​(𝒒)=𝑭𝒙,vec⁡(R)​(𝒒)∈ℝ12,\displaystyle\bm{y}_{1}=\mathcal{T}_{1}(\bm{q})=\bm{F}_{\bm{x},\mathrm{vec}(R)}(\bm{q})\in\mathbb{R}^{12},
experts:\displaystyle\mathrm{experts:}\quad 𝒚1∼𝒩⁡(𝝁,diag⁡(𝝈)+𝑳​𝑳T),\displaystyle\bm{y}_{1}\sim\mathcal{N}(\bm{\mu},\mathrm{diag}(\bm{\sigma})+\bm{L}\bm{L}^{T}),
parameters​to​learn:\displaystyle\mathrm{parameters\,to\,learn:}\quad 𝝁,σ,𝑳,\displaystyle\bm{\mu},\sigma,\bm{L},
Refer to caption
Figure 22: Scatter plot showing the correlation between the height of the end-effector and the first element of the first row of the rotation matrix for case (a). The dataset is shown in black and the samples from the different models in blue.
Table 5: Quantitative results for the task with a correlation between position and orientation. The table shows pvaluep_{\mathrm{value}} and maximum mean discrepancy MMDu2\mathrm{MMD}_{u}^{2} measures between the dataset and the different models for the different cases.
Case (a) (b) (c)
pvaluep_{\mathrm{value}} MMDu2\mathrm{MMD}_{u}^{2} pvaluep_{\mathrm{value}} MMDu2\mathrm{MMD}_{u}^{2} pvaluep_{\mathrm{value}} MMDu2\mathrm{MMD}_{u}^{2}
Vectorized Gaussian 0.0001 2.0​e−2{2.0}\mathrm{e}^{-2} 0.10 2.15​e−4{2.15}\mathrm{e}^{-4} 0.0011 1.0​e−2{1.0}\mathrm{e}^{-2}
Riemannian manifold 0.10 1.03​e−3{1.03}\mathrm{e}^{-3} 0.17 1.5​e−4{1.5}\mathrm{e}^{-4} 0.29 2.03​e−4{2.03}\mathrm{e}^{-4}
PoE 0.44\bm{0.44} 1.5​𝐞−𝟑\bm{{1.5}\mathrm{e}^{-3}} 0.38\bm{0.38} −3.9​𝐞−𝟓\bm{{-3.9}\mathrm{e}^{-5}} 0.68\bm{0.68} −1.04​𝐞−𝟑\bm{{-1.04}\mathrm{e}^{-3}}

7 Conclusion

Jointly normalized expert models exploit robot structure to learn useful constraints from small datasets, while null-space filtering helps recover masked secondary goals. Results support the studied tasks and model families. Expressive learned transformations and broader robotics applications remain directions for extending the framework.

Concept demonstration

Open image ↗