Literature
Display
← All papers

Active Inverse Learning in Stackelberg Trajectory Games

William Ward · Yue Yu · Jacob Levy · Negar Mehr · David Fridovich-Keil · Ufuk Topcu

· ACC 2025 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

A leader actively chooses its trajectory to reveal which of finitely many known objectives governs a boundedly rational follower. The follower uses a maximum-entropy linear quadratic controller; the leader maximizes separation among its predicted Gaussian state distributions through a difference-of-convex program. In two simulated receding-horizon games, the posterior identifies the true hypothesis faster than a feasible random-trajectory baseline. The evidence is numerical and Gazebo simulation, not human or physical-robot testing.

I Introduction

Passive inverse-game methods infer objectives from already collected behavior. This paper instead chooses actions that make self-interested follower responses informative, without requiring cooperative preference labels. It combines a maximum-entropy game model with finite-hypothesis identification and a convex-concave optimization procedure. The leader already knows the candidate cost/observation models; the task is to distinguish their types online, not discover an arbitrary unknown reward. Human-driver interaction motivates the setup, but demonstrations use synthetic rationality models.

II Linear quadratic Stackelberg trajectory games

Each player controls a stochastic linear time-invariant system. The leader commits to a trajectory; the follower responds by tracking a linear transformation of it while balancing control effort and entropy. This structure makes conditional behavior and state moments analytically tractable.

II-A The dynamics of the players’ systems

Leader and follower states follow separate known linear dynamics with independent, identically distributed, zero-mean Gaussian process disturbances and positive-definite covariances. Their state and input dimensions may differ. In the experiments, double-integrator trajectories approximate ground-rover motion, then a separate controller tracks those references in Gazebo.

Equations 1–3
xt+1L=AL​xtL+BL​utL+wtLx_{t+1}^{\texttt{L}}=A^{\texttt{L}}x_{t}^{\texttt{L}}+B^{\texttt{L}}u_{t}^{\texttt{L}}+w_{t}^{\texttt{L}} (1)
xt+1F=AF​xtF+BF​utF+wtFx_{t+1}^{\texttt{F}}=A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}u_{t}^{\texttt{F}}+w_{t}^{\texttt{F}} (2)
wtL∼𝒩⁡(0nL,ΩL),wtF∼𝒩⁡(0nF,ΩF).w_{t}^{\texttt{L}}\sim\mathcal{N}(0_{n_{\texttt{L}}},\Omega^{\texttt{L}}),\,w_{t}^{\texttt{F}}\sim\mathcal{N}(0_{n_{\texttt{F}}},\Omega^{\texttt{F}}). (3)

II-B The players’ objectives

The follower minimizes quadratic tracking error and input effort with a negative log-determinant term that rewards Gaussian action entropy. Proposition 1 supplies backward Riccati-style recursions for its optimal action covariance and affine state-feedback mean, followed by forward recursions for Gaussian state moments. Bounded-rationality noise is part of this assumed probabilistic model, not fitted evidence about actual human behavior.

The leader’s general bilevel objective combines expected joint trajectory cost with an input-only regularizer, subject to follower optimality. The proposed information-gathering formulation subsequently replaces the joint trajectory objective with distribution separation; it does not simultaneously establish an arbitrary task-performance guarantee.

Equations 4–8
minimizeμ0:τ−1,Σ0:τ−1∑t=0τ𝔼⁡[12​∥xtF−MF​xtL∥QF2]+12∑t=0τ−1(𝔼[∥utF∥RF2]−logdetΣt)subject toxt+1F=AFxtF+BFutF+wtF,x0F=x^0F,utF|xtF∼𝒩(μt,Σt),wtF∼𝒩(0nF,ΩF),t∈[0,τ−1],\begin{array}[]{ll}\lx@intercol\underset{\mu_{0:\tau-1},\Sigma_{0:\tau-1}}{\mbox{minimize}}&\sum_{t=0}^{\tau}\mathds{E}\left[\frac{1}{2}\left\lVert x_{t}^{\texttt{F}}-M^{\texttt{F}}x_{t}^{\texttt{L}}\right\rVert_{Q^{\texttt{F}}}^{2}\right]\\ &+\frac{1}{2}\sum_{t=0}^{\tau-1}\left(\mathds{E}\left[\left\lVert u_{t}^{\texttt{F}}\right\rVert_{R^{\texttt{F}}}^{2}\right]-\log\det\Sigma_{t}\right)\\ \lx@intercol\mbox{subject to}&x_{t+1}^{\texttt{F}}=A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}u_{t}^{\texttt{F}}+w_{t}^{\texttt{F}},\,x_{0}^{\texttt{F}}=\hat{x}_{0}^{\texttt{F}},\\ &u_{t}^{\texttt{F}}|x_{t}^{\texttt{F}}\sim\mathcal{N}(\mu_{t},\Sigma_{t}),\,w_{t}^{\texttt{F}}\sim\mathcal{N}(0_{n_{\texttt{F}}},\Omega^{\texttt{F}}),\\ &t\in[0,\tau-1],\end{array} (4)
PτF=QF,PtF=QF+(AF)⊤​Pt+1F​EtF,\displaystyle P^{\texttt{F}}_{\tau}=Q^{\texttt{F}},\,P^{\texttt{F}}_{t}=Q^{\texttt{F}}+(A^{\texttt{F}})^{\top}P^{\texttt{F}}_{t+1}E^{\texttt{F}}_{t}, (5a)
FtF=BF​(RF+(BF)⊤​Pt+1F​BF)−1​(BF)⊤,\displaystyle F^{\texttt{F}}_{t}=B^{\texttt{F}}(R^{\texttt{F}}+(B^{\texttt{F}})^{\top}P^{\texttt{F}}_{t+1}B^{\texttt{F}})^{-1}(B^{\texttt{F}})^{\top}, (5b)
EtF=AF−FtF​Pt+1F​AF,\displaystyle E^{\texttt{F}}_{t}=A^{\texttt{F}}-F^{\texttt{F}}_{t}P^{\texttt{F}}_{t+1}A^{\texttt{F}}, (5c)
qτF=−QF​MF​xτL,\displaystyle q_{\tau}^{\texttt{F}}=-Q^{\texttt{F}}M^{\texttt{F}}x_{\tau}^{\texttt{L}}, (5d)
qtF=(EtF)⊤​qt+1F−QF​MF​xtL,\displaystyle q_{t}^{\texttt{F}}=(E^{\texttt{F}}_{t})^{\top}q_{t+1}^{\texttt{F}}-Q^{\texttt{F}}M^{\texttt{F}}x_{t}^{\texttt{L}}, (5e)
Σt\displaystyle\Sigma_{t} =(RF+(BF)⊤​Pt+1F​BF)−1,\displaystyle=(R^{\texttt{F}}+(B^{\texttt{F}})^{\top}P^{\texttt{F}}_{t+1}B^{\texttt{F}})^{-1}, (6a)
μt\displaystyle\mu_{t} =−Σt​(BF)⊤​(Pt+1F​AF​xtF+qt+1F).\displaystyle=-\Sigma_{t}(B^{\texttt{F}})^{\top}(P^{\texttt{F}}_{t+1}A^{\texttt{F}}x_{t}^{\texttt{F}}+q_{t+1}^{\texttt{F}}). (6b)
ξt+1\displaystyle\xi_{t+1} =EtF​ξt−FtF​qt+1F,\displaystyle=E^{\texttt{F}}_{t}\xi_{t}-F^{\texttt{F}}_{t}q_{t+1}^{\texttt{F}}, (7a)
Λt+1\displaystyle\Lambda_{t+1} =EtF​Λt​(EtF)⊤+FtF+ΩF,\displaystyle=E^{\texttt{F}}_{t}\Lambda_{t}(E^{\texttt{F}}_{t})^{\top}+F^{\texttt{F}}_{t}+\Omega^{\texttt{F}}, (7b)
minimizeu0:τ−1L𝔼[f(xL0:τ,xF0:τ)]+g(uL0:τ)subject toxt+1L=ALxtL+BLutL+wtL,x0L=x^L0,xt+1F=AFxtF+BFutF+wtF,x0F=x^0F,wtL∼𝒩(0nL,ΩL),wtF∼𝒩(0nF,ΩF),ut∈𝕌,utF|xtF∼𝒩⁡(μt,Σt),t∈[0,τ−1],(μ0:τ−1,Σ0:τ−1) is optimal for (4),\begin{array}[]{ll}\lx@intercol\underset{u_{0:\tau-1}^{\texttt{L}}}{\mbox{minimize}}&\mathds{E}[f(x^{\texttt{L}}_{0:\tau},x^{\texttt{F}}_{0:\tau})]+g(u^{\texttt{L}}_{0:\tau})\\ \lx@intercol\mbox{subject to}&x_{t+1}^{\texttt{L}}=A^{\texttt{L}}x_{t}^{\texttt{L}}+B^{\texttt{L}}u_{t}^{\texttt{L}}+w_{t}^{\texttt{L}},\,x_{0}^{\texttt{L}}=\hat{x}^{\texttt{L}}_{0},\\ &x_{t+1}^{\texttt{F}}=A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}u_{t}^{\texttt{F}}+w_{t}^{\texttt{F}},\,x_{0}^{\texttt{F}}=\hat{x}_{0}^{\texttt{F}},\\ &w_{t}^{\texttt{L}}\sim\mathcal{N}(0_{n_{\texttt{L}}},\Omega^{\texttt{L}}),\,w_{t}^{\texttt{F}}\sim\mathcal{N}(0_{n_{\texttt{F}}},\Omega^{\texttt{F}}),\\ &u_{t}\in\mathbb{U},\,u_{t}^{\texttt{F}}|x_{t}^{\texttt{F}}\sim\mathcal{N}(\mu_{t},\Sigma_{t}),\,t\in[0,\tau-1],\\ &(\mu_{0:\tau-1},\Sigma_{0:\tau-1})\text{ is optimal for \eqref{opt: f-lqr},}\end{array} (8)

III Active inverse learning via difference maximization

The unknown follower type must equal one member of a finite list of known triples: state-tracking weight, input-cost weight, and leader-to-follower reference mapping. Bayes’ rule compares trajectory likelihoods under those hypotheses. If two types induce nearly identical observations, posterior odds remain close to prior odds, motivating deliberate excitation. True-in-list and correct-model assumptions are central; out-of-set types and misspecified noise are not evaluated.

Equations 9–11
(QF,RF,MF)=(Qi,Ri,Mi)(Q^{\texttt{F}},R^{\texttt{F}},M^{\texttt{F}})=(Q^{i},R^{i},M^{i}) (9)
ℙ(θF=θi|xF0:τ)ℙ(θF=θj|xF0:τ)=ℙ(θF=θi|xF0)ℙ(xF1:τ|θF=θi)ℙ(θF=θj|xF0)ℙ(xF1:τ|θF=θj).\frac{\mathds{P}(\theta^{\texttt{F}}=\theta^{i}|x^{\texttt{F}}_{0:\tau})}{\mathds{P}(\theta^{\texttt{F}}=\theta^{j}|x^{\texttt{F}}_{0:\tau})}=\frac{\mathds{P}(\theta^{\texttt{F}}=\theta^{i}|x^{\texttt{F}}_{0})\mathds{P}(x^{\texttt{F}}_{1:\tau}|\theta^{\texttt{F}}=\theta^{i})}{\mathds{P}(\theta^{\texttt{F}}=\theta^{j}|x^{\texttt{F}}_{0})\mathds{P}(x^{\texttt{F}}_{1:\tau}|\theta^{\texttt{F}}=\theta^{j})}. (10)
ℙ(x1:τF|θF=θi)≈ℙ(x1:τF|θF=θj)\mathds{P}(x^{\texttt{F}}_{1:\tau}|\theta^{\texttt{F}}=\theta^{i})\approx\mathds{P}(x^{\texttt{F}}_{1:\tau}|\theta^{\texttt{F}}=\theta^{j}) (11)

III-A The follower’s trajectories under different hypotheses

The follower solution gives a Gaussian state distribution for every time and hypothesis. Pairwise separation is the squared difference of state means weighted by the sum of inverse covariances. It equals the leader-dependent part of a symmetrized Gaussian KL divergence up to scaling and additive terms. The omitted covariance terms do not vary with the leader’s trajectory in this model. This is a tractable separation surrogate, not exact posterior entropy or full correlated-trajectory information gain.

Equations 12a–15
Eti=AF−Fti​Pt+1i​AF,\displaystyle E^{i}_{t}=A^{\texttt{F}}-F^{i}_{t}P^{i}_{t+1}A^{\texttt{F}}, (12a)
Fti=BF​(Ri+(BF)⊤​Pt+1i​BF)−1​(BF)⊤,\displaystyle F^{i}_{t}=B^{\texttt{F}}(R^{i}+(B^{\texttt{F}})^{\top}P^{i}_{t+1}B^{\texttt{F}})^{-1}(B^{\texttt{F}})^{\top}, (12b)
Pτi=Qi,Pti=Qi+(AF)⊤​Pt+1i​Eti,\displaystyle P_{\tau}^{i}=Q^{i},\,P^{i}_{t}=Q^{i}+(A^{\texttt{F}})^{\top}P^{i}_{t+1}E^{i}_{t}, (12c)
Λ0i=0nF×nF,Λt+1i=Eti​Λti​(Eti)⊤+Fti+ΩF,\displaystyle\Lambda_{0}^{i}=0_{n_{\texttt{F}}\times n_{\texttt{F}}},\,\Lambda_{t+1}^{i}=E^{i}_{t}\Lambda_{t}^{i}(E^{i}_{t})^{\top}+F^{i}_{t}+\Omega^{\texttt{F}}, (12d)
qτi=−Qi​Mi​xτL,\displaystyle q_{\tau}^{i}=-Q^{i}M^{i}x_{\tau}^{\texttt{L}}, (12e)
qti=(Eti)⊤​qt+1i−Qi​Mi​xtL,\displaystyle q_{t}^{i}=(E^{i}_{t})^{\top}q_{t+1}^{i}-Q^{i}M^{i}x_{t}^{\texttt{L}}, (12f)
ξ0i=x^0F,ξt+1i=Eti​ξti+Fti​qt+1i,\displaystyle\xi_{0}^{i}=\hat{x}^{\texttt{F}}_{0},\,\xi_{t+1}^{i}=E^{i}_{t}\xi_{t}^{i}+F^{i}_{t}q_{t+1}^{i}, (12g)
xtF∼𝒢ti≔𝒩⁡(ξti,Λti).x^{\texttt{F}}_{t}\sim\mathcal{G}_{t}^{i}\coloneqq\mathcal{N}(\xi^{i}_{t},\Lambda^{i}_{t}). (13)
𝔻≔{(i,j)|i<j,i,j∈[1,d]}.\mathbb{D}\coloneqq\{(i,j)~|~i<j,\,i,j\in[1,d]\}. (14)
D⁡(𝒢ti,𝒢tj)≔∥ξti−ξtj∥(Λti)−1+(Λtj)−12D(\mathcal{G}^{i}_{t},\mathcal{G}^{j}_{t})\coloneqq\left\lVert\xi^{i}_{t}-\xi^{j}_{t}\right\rVert^{2}_{(\Lambda^{i}_{t})^{-1}+(\Lambda^{j}_{t})^{-1}} (15)

III-B Maximizing the worst-case pairwise distance

The leader maximizes the minimum time-summed separation across all hypothesis pairs, with an input-smoothness penalty. An epigraph reformulation expresses that objective as a difference of convex functions. The planner replaces the leader’s stochastic state by its mean, using averaged dynamics. The convex-concave procedure supplies a stationary-point method, not a certificate of global optimality or of identification for indistinguishable hypotheses.

The published formulas remain intact. The PDF itself has apparent typographical inconsistencies: (12g) uses a plus sign where (7a) and (17) use minus, and parts of (16) repeat the same mean index in a difference. These notes describe the stated pairwise-separation construction without silently correcting the formulas; implementation should be checked against the released code.

Equations 16
min(i,j)∈𝔻​{∑t=1τ∥ξti−ξti∥(Λti)−1+(Λtj)−12}\displaystyle\textstyle\displaystyle\underset{(i,j)\in\mathbb{D}}{\min}\left\{\sum_{t=1}^{\tau}\left\lVert\xi_{t}^{i}-\xi_{t}^{i}\right\rVert_{(\Lambda^{i}_{t})^{-1}+(\Lambda^{j}_{t})^{-1}}^{2}\right\} (16)
=∑(i,j)∈𝔻∑t=1τ∥ξti−ξti∥(Λti)−1+(Λtj)−12\displaystyle=\textstyle\sum\limits_{(i,j)\in\mathbb{D}}\sum_{t=1}^{\tau}\left\lVert\xi_{t}^{i}-\xi_{t}^{i}\right\rVert_{(\Lambda^{i}_{t})^{-1}+(\Lambda^{j}_{t})^{-1}}^{2}
−max(i,j)∈𝔻​{∑(k,l)∈𝔻∖{(i,j)}∑t=1τ∥ξtk−ξtl∥(Λk)t−1+(Λl)t−12}.\displaystyle\textstyle\displaystyle-\underset{(i,j)\in\mathbb{D}}{\max}\left\{\sum\limits_{(k,l)\in\mathbb{D}\setminus\{(i,j)\}}\sum_{t=1}^{\tau}\left\lVert\xi_{t}^{k}-\xi_{t}^{l}\right\rVert_{(\Lambda^{k})^{-1}_{t}+(\Lambda^{l})^{-1}_{t}}^{2}\right\}.
Original unnumbered equation in the published PDF.
Equation 17
minimizes,u0:τ−1Ls−∑(i,j)∈𝔻∑t=0τ∥ξti−ξtj∥(Λti)−1+(Λtj)−12+g(u0:τ−1)subject toηt+1L=ALηtL+BLutL,η0L=x^L0,qti=(Eti)⊤qt+1i−QiMiηt,qτi=−QiMiητ,ξt+1i=Etiξti−Ftiqt+1i,ξ0i=x^0F,∑(k,l)∈𝔻∖{(i,j)}∑t=1τ∥ξtk−ξtl∥(Λk)t−1+(Λl)t−12≤s,∀t∈[0,τ−1],i∈[1,d],(i,j)∈𝔻.\begin{array}[]{ll}\lx@intercol\underset{s,u_{0:\tau-1}^{\texttt{L}}}{\mbox{minimize}}&s-\sum_{(i,j)\in\mathbb{D}}\sum_{t=0}^{\tau}\left\lVert\xi_{t}^{i}-\xi_{t}^{j}\right\rVert_{(\Lambda^{i}_{t})^{-1}+(\Lambda^{j}_{t})^{-1}}^{2}+g(u_{0:\tau-1})\\ \lx@intercol\mbox{subject to}&\eta_{t+1}^{\texttt{L}}=A^{\texttt{L}}\eta_{t}^{\texttt{L}}+B^{\texttt{L}}u_{t}^{\texttt{L}},\,\eta_{0}^{\texttt{L}}=\hat{x}^{\texttt{L}}_{0},\\ &q_{t}^{i}=(E_{t}^{i})^{\top}q_{t+1}^{i}-Q^{i}M^{i}\eta_{t},\,q_{\tau}^{i}=-Q^{i}M^{i}\eta_{\tau},\,\xi_{t+1}^{i}=E_{t}^{i}\xi_{t}^{i}-F_{t}^{i}q_{t+1}^{i},\,\xi_{0}^{i}=\hat{x}_{0}^{\texttt{F}},\\ &\sum_{(k,l)\in\mathbb{D}\setminus\{(i,j)\}}\sum_{t=1}^{\tau}\left\lVert\xi_{t}^{k}-\xi_{t}^{l}\right\rVert_{(\Lambda^{k})^{-1}_{t}+(\Lambda^{l})^{-1}_{t}}^{2}\leq s,\,\forall t\in[0,\tau-1],\,i\in[1,d],\,(i,j)\in\mathbb{D}.\end{array} (17)
 
Equation
max(i,j)∈𝔻​{∑(k,l)∈𝔻∖{(i,j)}∑t=1τ∥ξtk−ξtl∥(Λk)t−1+(Λl)t−12}≤s.\textstyle\underset{(i,j)\in\mathbb{D}}{\max}\left\{\sum\limits_{(k,l)\in\mathbb{D}\setminus\{(i,j)\}}\sum\limits_{t=1}^{\tau}\left\lVert\xi_{t}^{k}-\xi_{t}^{l}\right\rVert_{(\Lambda^{k})^{-1}_{t}+(\Lambda^{l})^{-1}_{t}}^{2}\right\}\leq s.

IV Numerical Experiments and Simulations

Two repeated games each have three hypotheses: identifying which of three leader-controlled rovers is pursued, and identifying a simulated follower’s response to a driving assistant. Julia, JuMP, and MOSEK solve the optimizations; splines transmit trajectories through ROS to virtual TurtleBots in Gazebo. Proportional controllers track them under tuned speed and maneuverability limits. No physical deployment or solver-time study is reported.

IV-A Leader-follower pursuit game parameters

The pursuit setup uses planar double integrators and two-second discretization. The three follower types share tracking weights diag(20,20,0,0), input weight 30,000 times identity, and process covariance 10⁻⁵ times identity; their mappings select different leader rovers. Leader acceleration inputs are bounded by 0.005 and velocities by 0.1 m/s. The regularizer penalizes consecutive control differences. Identifying the type mainly requires separating candidate target trajectories.

Equation
Original unnumbered equation in the published PDF.

IV-B Driving assistant game parameters

The driving task also uses two-second double-integrator dynamics, with follower process covariance 10⁻⁴ and leader covariance 10⁻⁷ times identity. Three mappings scale state/velocity reference components down, leave them unchanged, or scale them up; cost weights are shared. Leader input bounds are 0.05. A sideways L-shaped road has width 1.3 m and short-segment length 3 m; road and non-reversal constraints reduce feasible excitation. These are stylized simulated driver types, not empirically validated human categories.

IV-C Experimental procedure

At each two-second update the leader observes both states, optimizes a finite-horizon input sequence, shares its simulated trajectory, and executes the first input. The follower solves its maximum-entropy tracking problem for that reference, samples the first Gaussian action, and executes it. Both replan. This assumes state observation and communication of the planned leader trajectory, rather than inference from noisy partial visual observations alone.

IV-D Results

With a uniform prior and the first hypothesis as ground truth, evaluation measures the ℓ₁ distance between the posterior and the correct one-hot distribution. Figure 2 reports medians and first/third quartiles over one hundred simulations. The baseline samples allowed state trajectories uniformly and projects them through feasible tracking optimization; it is not uncontrolled random action noise.

Active planning spreads pursuit targets in different directions and gives orders-of-magnitude smaller posterior error in the shown horizon. Driving gains become clear only after about six seconds, with less excitation available under road constraints. Figures 1 and 3 show responses and virtual-warehouse execution. Alternative information-gain planners, real humans, and collision safety are not evaluated. The authors provide a simulation video, optimization code, and ROS simulation code.

Equation 18
pit≔ℙ((QF,RF,MF)=(Qi,Ri,Mi)|x¯0:tF)p_{i}^{t}\coloneqq\mathds{P}((Q^{\texttt{F}},R^{\texttt{F}},M^{\texttt{F}})=(Q^{i},R^{i},M^{i})~|~\overline{x}^{\texttt{F}}_{0:t}) (18)
Original Figure 1 with both complete panels and labels.
Fig. 1: The trajectories of the leader (black) and followers under different hypotheses. Each color indicates one of the hypotheses. The semi-axis of the shaded ellipses represents the directions of the eigenvectors of the corresponding covariance matrices.
Original Figure 2 with both complete panels and labels.
Fig. 2: Comparing the convergence of ptp^{t} when using the proposed method (red) versus sampling a random trajectory (blue). The solid lines show the median over 100 simulations, while the boundaries of the colored areas mark the corresponding first and third quartiles.
Original Figure 3 with both complete panels and labels.
Fig. 3: Simulating the results on TurtleBots in a virtual warehouse environment in Gazebo.

V Conclusion

Finite-model, linear-quadratic excitation improves identification in these two games. The authors leave nonlinear dynamics, collision avoidance, and deceptive followers unresolved, and propose broader Bayesian-game information gathering as future work. The method also depends on known candidate objectives and noise models; faster concentration in this synthetic setting does not establish calibrated inference under model mismatch.

APPENDIX

The appendix derives the follower’s optimal Gaussian policy and state-moment recursions. It supports the closed-form model, rather than proving global optimality of the difference-of-convex planner.

Proof of Proposition Proposition 1 .

Dynamic programming preserves a quadratic value function. Expected quadratic control and future-value costs introduce covariance trace terms; differentiation with respect to the Gaussian mean and covariance gives the feedback law and backward recursions. Gaussian propagation gives forward state moments using the standard joint Gaussian construction. The published derivation and final unnumbered block are retained. Some appendix indices/covariance terms appear inconsistent with surrounding definitions; they remain visible as published rather than being rewritten into an unverified corrected proof.

Equations 19–23
ψt(y,μt:τ,Σt:τ)\displaystyle\psi_{t}(y,\mu_{t:\tau},\Sigma_{t:\tau}) (19)
≔∑j=tτ𝔼⁡[12​∥xjF−MF​xjL∥QF2|xtF=y]\displaystyle\coloneqq\textstyle\sum_{j=t}^{\tau}\mathds{E}\left[\frac{1}{2}\left\lVert x_{j}^{\texttt{F}}-M^{\texttt{F}}x_{j}^{\texttt{L}}\right\rVert_{Q^{\texttt{F}}}^{2}|x_{t}^{\texttt{F}}=y\right]
+12∑j=tτ−1(𝔼[∥ujF∥RF2|xtF=y]−logdetΣj).\displaystyle\textstyle\displaystyle+\frac{1}{2}\sum_{j=t}^{\tau-1}\left(\mathds{E}\left[\left\lVert u_{j}^{\texttt{F}}\right\rVert_{R^{\texttt{F}}}^{2}|x_{t}^{\texttt{F}}=y\right]\textstyle-\log\det\Sigma_{j}\right).
Vt​(y)=12​∥y−MF​xtF∥QF2−12​log​detΣt\displaystyle V_{t}(y)=\textstyle\frac{1}{2}\left\lVert y-M^{\texttt{F}}x_{t}^{\texttt{F}}\right\rVert^{2}_{Q^{\texttt{F}}}-\frac{1}{2}\log\det\Sigma_{t} (20)
+minμt,Σt⁡𝔼⁡[12​∥utF∥RF2+Vt+1​(AF​xtF+BF​utF+wtF)|xtF=y],\displaystyle+\min_{\mu_{t},\Sigma_{t}}\textstyle\mathds{E}\left[\frac{1}{2}\left\lVert u_{t}^{\texttt{F}}\right\rVert_{R^{\texttt{F}}}^{2}+V_{t+1}(A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}u_{t}^{\texttt{F}}+w_{t}^{\texttt{F}})|x_{t}^{\texttt{F}}=y\right],
𝔼⁡[∥utF∥RF2|xtF=y]\displaystyle\mathds{E}[\left\lVert u_{t}^{\texttt{F}}\right\rVert_{R^{\texttt{F}}}^{2}|x_{t}^{\texttt{F}}=y] (21)
=μt⊤​RF​μt+𝔼⁡[tr((utF−μt)​(utF−μt)⊤​RF)|xtF=y]\displaystyle=\mu_{t}^{\top}R^{\texttt{F}}\mu_{t}+\mathds{E}[\mathop{\rm tr}((u_{t}^{\texttt{F}}-\mu_{t})(u_{t}^{\texttt{F}}-\mu_{t})^{\top}R^{\texttt{F}})|x_{t}^{\texttt{F}}=y]
=μt⊤​RF​μt+tr(Σt​RF).\displaystyle=\mu_{t}^{\top}R^{\texttt{F}}\mu_{t}+\mathop{\rm tr}(\Sigma_{t}R^{\texttt{F}}).
𝔼⁡[Vt+1​(AF​xtF+BF​utF+wtF)|xtF=y]\displaystyle\mathds{E}[V_{t+1}(A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}u_{t}^{\texttt{F}}+w_{t}^{\texttt{F}})|x_{t}^{\texttt{F}}=y] (22)
=12​(AF​y+BF​μt)⊤​Pt+1F​(AF​y+BF​μt)\displaystyle=\textstyle\frac{1}{2}(A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t})^{\top}P_{t+1}^{\texttt{F}}(A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t})
+𝔼⁡[(AF​xtF+BF​μt)⊤​Pt+1F​BF​(utF−μt)|xtF=y]\displaystyle+\mathds{E}[(A^{\texttt{F}}x_{t}^{\texttt{F}}+B^{\texttt{F}}\mu_{t})^{\top}P_{t+1}^{\texttt{F}}B^{\texttt{F}}(u_{t}^{\texttt{F}}-\mu_{t})|x_{t}^{\texttt{F}}=y]
+12​𝔼​[(BF​(utF−μt))⊤​Pt+1F​BF​(utF−μt)|xtF=y]\displaystyle\textstyle\displaystyle+\frac{1}{2}\mathds{E}[(B^{\texttt{F}}(u_{t}^{\texttt{F}}-\mu_{t}))^{\top}P_{t+1}^{\texttt{F}}B^{\texttt{F}}(u_{t}^{\texttt{F}}-\mu_{t})|x_{t}^{\texttt{F}}=y]
+⟨qt+1F,AF​y+BF​μt⟩+12​𝔼​[tr(wtF​(wtF)⊤​Pt+1)]+νt+1F\displaystyle+\langle q_{t+1}^{\texttt{F}},A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t}\rangle+\frac{1}{2}\mathds{E}[\mathop{\rm tr}(w_{t}^{\texttt{F}}(w_{t}^{\texttt{F}})^{\top}P_{t+1})]+\nu_{t+1}^{\texttt{F}}
=12​(AF​y+BF​μt)⊤​Pt+1F​(AF​y+BF​μt)+12​tr(ΩF​Pt+1)\displaystyle=\textstyle\frac{1}{2}(A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t})^{\top}P_{t+1}^{\texttt{F}}(A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t})+\frac{1}{2}\mathop{\rm tr}(\Omega^{\texttt{F}}P_{t+1})
+12tr(Σt(BF)⊤Pt+1FBF)+⟨qt+1F,AFy+BFμt⟩+νt+1F.\displaystyle\textstyle\displaystyle+\frac{1}{2}\mathop{\rm tr}(\Sigma_{t}(B^{\texttt{F}})^{\top}P_{t+1}^{\texttt{F}}B^{\texttt{F}})+\langle q_{t+1}^{\texttt{F}},A^{\texttt{F}}y+B^{\texttt{F}}\mu_{t}\rangle+\nu_{t+1}^{\texttt{F}}.
Vt(y)=+12tr(ΣtRF+ΩFPt+1+(BF)⊤Pt+1FBF))+νt+1F\displaystyle V_{t}(y)=\textstyle+\frac{1}{2}\mathop{\rm tr}(\Sigma_{t}R^{\texttt{F}}+\Omega^{\texttt{F}}P_{t+1}+(B^{\texttt{F}})^{\top}P_{t+1}^{\texttt{F}}B^{\texttt{F}}))+\nu_{t+1}^{\texttt{F}} (23)
+12​y⊤​(QF+(AF)⊤​Pt+1F​AF)​y+⟨A⊤​qt+1F−QF​MF​xtL,y⟩\displaystyle+\textstyle\frac{1}{2}y^{\top}(Q^{\texttt{F}}+(A^{\texttt{F}})^{\top}P_{t+1}^{\texttt{F}}A^{\texttt{F}})y+\langle A^{\top}q_{t+1}^{\texttt{F}}-Q^{\texttt{F}}M^{\texttt{F}}x_{t}^{\texttt{L}},y\rangle
+12​μt⊤​(RF+(BF)⊤​Pt+1F​BF)​μt+12​(xtL)⊤​(MF)⊤​QF​MF​xtL\displaystyle\textstyle\displaystyle+\frac{1}{2}\mu_{t}^{\top}(R^{\texttt{F}}+(B^{\texttt{F}})^{\top}P_{t+1}^{\texttt{F}}B^{\texttt{F}})\mu_{t}+\frac{1}{2}(x_{t}^{\texttt{L}})^{\top}(M^{\texttt{F}})^{\top}Q^{\texttt{F}}M^{\texttt{F}}x_{t}^{\texttt{L}}
+⟨(BF)⊤​qt+1F+(BF)⊤​Pt+1F​AF​y,μt⟩−12​log​detΣt.\displaystyle\textstyle\displaystyle+\langle(B^{\texttt{F}})^{\top}q_{t+1}^{\texttt{F}}+(B^{\texttt{F}})^{\top}P_{t+1}^{\texttt{F}}A^{\texttt{F}}y,\mu_{t}\rangle-\frac{1}{2}\log\det\Sigma_{t}.
Original unnumbered equation in the published PDF.

ACKNOWLEDGMENT

The work acknowledges a National Science Foundation CAREER award and three additional NSF grants, plus Office of Naval Research subcontracts with Phillips and the University of Pennsylvania.

Concept demonstration

Open image ↗