Literature
Display
← All papers

Parallel Stochastic Gradient-Based Planning for World Models

Michael Psenka · Michael Rabbat · Aditi Krishnapriyan · Yann LeCun · Amir Bar

· arXiv (preprint) ↗

World models7 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition. Instead of guessing every action and repeatedly simulating the whole journey, imagine placing tentative waypoints first and adjusting all neighboring transitions together. GRASP treats these intermediate latent states as optimization variables, adds a penalty when neighboring states disagree with the learned dynamics, and injects noise to escape poor routes. It also blocks unreliable gradients through state inputs while keeping useful action gradients, then periodically refines the plan with an ordinary rollout-based update. The method is primarily a planner for an existing differentiable world model, not a new JEPA representation-learning objective. Visual-control experiments report its strongest advantage at long horizons. Its theoretical analyses use restricted smooth or linear settings; practical stochastic, detached updates do not guarantee a globally feasible optimum.

1 Introduction

Long rollouts create long chains of derivatives and difficult local minima. Classical multiple shooting already introduces intermediate states; this paper adapts that idea to learned visual dynamics whose state derivatives may be brittle. GRASP combines lifted states, stochastic exploration, detached state-input gradients, and occasional full-rollout synchronization. Its gains depend on these components working together and on the quality of the underlying world model.

Refer to caption
Figure 1: Difficulty of the planning problem. Subfigure (a) shows the distance to the goal in L2L^{2} norm throughout a successful trajectory. This illustrates the difficulty of planning optimization away from a minimizer: successful trajectories often have to first move away from the goal to successfully plan towards it later, resulting in greedy strategies failing. Subfigures (b)-(c) depict the loss landscape at convergence of standard rollout-based planners vs. our planner. The example given is in the Push-T environment at horizon length 50. The axes plotted over are with respect to two random, orthogonal, unit-norm directions in the full action space ℝ50×2\mathbb{R}^{50\times 2}. Our planner loss is taken as in Eq. (10), and for GD the loss is taken as in Eq. (3).

2 Problem formulation

The task is to choose a continuous action sequence that makes a predicted final state approach a specified goal. Ordinary shooting generates every intermediate state serially from the first. This is computationally expensive and can produce poorly conditioned derivatives as the horizon grows. The paper instead introduces auxiliary state variables so that neighboring transitions can be evaluated in parallel.

Refer to caption
Figure 2: Graphical depiction of (a) a standard serial-based setup for optimization-based planning, where states are rolled out using the actions and the loss is evaluated on the goal state, (b) our setup, which parallelizes the world model evaluations by optimizing “virtual states” directly and only supervising pairwise dynamics satisfaction. The crossed lines and skipped connections for our method’s depiction (b) are detailed in Section 3.3, which keeps the full planning graph connected while not requiring state gradients of the dynamics FθF_{\theta}. For our planner, we find it helpful to alternate between (a) and (b) throughout the planning optimization.
Equations 1–3
ℱθT​(𝐚,s0)=g,\mathcal{F}_{\theta}^{T}(\mathbf{a},s_{0})=g, (1)
ℱθT(𝐚,s0)≔Fθ(…Fθ(Fθ(⏟T timess0,a0),a1),…aT−1).\mathcal{F}_{\theta}^{T}(\mathbf{a},s_{0})\coloneqq\underbrace{F_{\theta}(\dots F_{\theta}(F_{\theta}(}_{\text{$T$ times}}s_{0},a_{0}),a_{1}),\dots a_{T-1}). (2)
𝐚∗=arg​min𝐚⁡‖ℱθT​(𝐚,s0)−g‖22.\mathbf{a}^{*}=\argmin_{\mathbf{a}}\|\mathcal{F}_{\theta}^{T}(\mathbf{a},s_{0})-g\|_{2}^{2}. (3)

3 Decoupling dynamics for gradient-based planning

A soft dynamics penalty links free intermediate states to one-step predictions. This replaces a deeply composed rollout objective with many local consistency terms. The relaxation improves parallelism but temporarily permits imagined states that no action sequence can realize, so obtaining low relaxed loss and obtaining an executable plan must be distinguished.

3.1 Parallelized planning

All adjacent transition penalties can be evaluated simultaneously. Their global solutions correspond to the intended planning formulation under its stated conditions, but local optimization may still settle on implausible paths through obstacles. Learned high-dimensional dynamics also offer adversarial state directions. Subsequent components specifically address these two failures rather than assuming lifting solves them.

Equation 4
min𝐬,𝐚⁡ℒdyn​(𝐬,𝐚)≔∑t=0T−1‖Fθ​(st,at)−st+1‖22,with ​s0​ fixed,sT=g.\begin{split}\min_{\mathbf{s},\mathbf{a}}\ \mathcal{L}_{\mathrm{dyn}}(\mathbf{s},\mathbf{a})\coloneqq\sum_{t=0}^{T-1}\big\|F_{\theta}(s_{t},a_{t})-s_{t+1}\big\|_{2}^{2},\\ \text{with }s_{0}\text{ fixed},\ s_{T}=g.\end{split} (4)

3.2 Exploration via Langevin state updates

Gaussian noise perturbs the optimized intermediate states to explore alternatives around poor local routes. This is exploration during planning optimization, not a stochastic environment model and not additional training data.

Langevin dynamics on state iterates.

Each state update combines a descent step and an independent Gaussian perturbation. The motivation is to escape narrow basins in the relaxed loss landscape. The later detached-gradient variant changes the vector field, so its complete practical update should not be identified with exact Langevin sampling.

Equations 5–6
stk+1\displaystyle s_{t}^{k+1} ←stk−ηs​∇stℒdyn​(𝐬k,𝐚k)+σstate​ξtk,\displaystyle\leftarrow s_{t}^{k}-\eta_{s}\nabla_{s_{t}}\mathcal{L}_{\mathrm{dyn}}(\mathbf{s}^{k},\mathbf{a}^{k})+\sigma_{\text{state}}\xi_{t}^{k}, (5)
atk+1\displaystyle a_{t}^{k+1} ←atk−ηa​∇atℒdyn​(𝐬k,𝐚k),\displaystyle\leftarrow a_{t}^{k}-\eta_{a}\nabla_{a_{t}}\mathcal{L}_{\mathrm{dyn}}(\mathbf{s}^{k},\mathbf{a}^{k}), (6)

Noise on states vs. actions.

Direct state perturbations can discover an intermediate waypoint without first finding the precise action sequence leading there. Dynamics penalties encourage those exploratory states to become locally consistent. This is different from injecting noise only into actions and propagating it through a long rollout.

Refer to caption
(a) Expected behavior
Refer to caption
(b) Adversarial example
Figure 3: Sensitivity of state gradient structure. Examples of three states far away from the goal on the right (either in-distribution or out-of-distribution), such that taking a small step along the gradient s′=s−ϵ​∇sℒ​(s),ℒ⁡(s)=‖Fθ​(s,a=𝟎)−g‖22s^{\prime}=s-\epsilon\nabla_{s}\mathcal{L}(s),\mathcal{L}(s)=\|F_{\theta}(s,a=\mathbf{0})-g\|_{2}^{2}, leads to a nearby state s′s^{\prime} that solves the planning problem in a single step: Fθ​(s′,𝟎)=gF_{\theta}(s^{\prime},\mathbf{0})=g. Thus, optimizing states directly through the world model FθF_{\theta} can be quite challenging.

3.3 Sensitivity to state gradients

A note on adversarial robustness of state gradients.

The optimizer can exploit tiny off-manifold state changes that make a learned model predict a desired output, rather than finding a physically meaningful transition. The paper therefore removes derivatives through state inputs. Its impossibility result explains the resulting compromise: for a general dynamics family, exact dynamics-enforcing minimizers cannot simultaneously ignore all such state derivatives.

Equation 7
arg​mins⁡‖y−Fθ​(s,a)‖22,\argmin_{s}\|y-F_{\theta}(s,a)\|_{2}^{2}, (7)
Refer to caption
Figure 4: Virtual states learned through planning. All examples are instantiations of our planner at horizon 50 in the Point-Maze, Wall-Single, and Push-T environments. Regardless of the dynamics constraint relaxation and state noising, directly optimized states find realistic, non-greedy paths towards the goal.
1: Initial observation o0o_{0}, goal ogo_{g}, world model FθF_{\theta}, horizon TT, steps KK, learning rates ηa,ηs\eta_{a},\eta_{s}.
2: 𝐚∗\mathbf{a}^{*}
3: s0←encode​(o0)s_{0}\leftarrow\text{encode}(o_{0}), sT←encode​(og)s_{T}\leftarrow\text{encode}(o_{g})
4: 𝐚←𝐚0\mathbf{a}\leftarrow\mathbf{a}^{0}; 𝐬←init_states​(s0,sT)\mathbf{s}\leftarrow\text{init\_states}(s_{0},s_{T})
5: for k=0k=0 to K−1K-1 do
6:    Compute ℒ\mathcal{L} as in Eq. (10)
7:    Joint step: (𝐚,𝐬)←(𝐚,𝐬)−(ηa​∇𝐚ℒ,ηs​∇𝐬ℒ)(\mathbf{a},\mathbf{s})\leftarrow(\mathbf{a},\mathbf{s})-(\eta_{a}\nabla_{\mathbf{a}}\mathcal{L},\eta_{s}\nabla_{\mathbf{s}}\mathcal{L})
8:    Stochastic state: st←st+σstate​ξt,ξt∼𝒩⁡(0,I)s_{t}\leftarrow s_{t}+\sigma_{\text{state}}\xi_{t},\xi_{t}\sim\mathcal{N}(0,I) for t=1,…,T−1t=1,\dots,T-1
9:    Sync (periodic): if kmodKsync=0k\bmod K_{\mathrm{sync}}=0, rollout from s0s_{0};  st+1←Fθ​(st,at)s_{t+1}\leftarrow F_{\theta}(s_{t},a_{t}) for t=0,…,T−1t=0,\dots,T-1
10:         then take a GD step 𝐚←𝐚−ηsync​∇𝐚‖sT−g‖22\mathbf{a}\leftarrow\mathbf{a}-\eta_{\mathrm{sync}}\nabla_{\mathbf{a}}\|s_{T}-g\|_{2}^{2}
11: end for
12: return 𝐚∗←𝐚\mathbf{a}^{*}\leftarrow\mathbf{a}
 {algorithm}
0 GRASP planner: parallel stochastic gradient-based planning

Grad-cut dynamics loss.

A stop-gradient copy has the same numerical state value but contributes no derivative through the dynamics input. Gradients still update actions and the next-state consistency target. This deliberately changes the optimization rule; it is not ordinary gradient descent on the original planning objective.

Equation 8
ℒdynsg​(𝐬,𝐚)=∑t=0T−1‖Fθ​(s¯t,at)−st+1‖22.\mathcal{L}_{\mathrm{dyn}}^{\mathrm{sg}}(\mathbf{s},\mathbf{a})=\sum_{t=0}^{T-1}\big\|F_{\theta}(\bar{s}_{t},a_{t})-s_{t+1}\big\|_{2}^{2}. (8)

Dense goal shaping on one-step predictions.

Without an additional goal signal, detached updates tend to stay near a current rollout. A goal penalty on every one-step prediction supplies action-directed progress at all time steps. Combined noisy updates favor locally consistent, goal-directed trajectories, but the states no longer follow a true gradient field or an exact Boltzmann sampler.

Equations 9–12
ℒgoalsg​(𝐬,𝐚)=∑t=0T−1‖Fθ​(s¯t,at)−g‖22.\mathcal{L}_{\mathrm{goal}}^{\mathrm{sg}}(\mathbf{s},\mathbf{a})=\sum_{t=0}^{T-1}\big\|F_{\theta}(\bar{s}_{t},a_{t})-g\big\|_{2}^{2}. (9)
ℒ⁡(𝐬,𝐚)=∑t=0T−1‖Fθ​(s¯t,at)−st+1‖22+γ∑t=0T−1∥Fθ(s¯t,at)−g∥22,\displaystyle\begin{split}\mathcal{L}(\mathbf{s},\mathbf{a})=&\sum_{t=0}^{T-1}\big\|F_{\theta}(\bar{s}_{t},a_{t})-s_{t+1}\big\|_{2}^{2}\\ &\quad+\gamma\sum_{t=0}^{T-1}\big\|F_{\theta}(\bar{s}_{t},a_{t})-g\big\|_{2}^{2},\end{split} (10)
stk+1\displaystyle s_{t}^{k+1} ←stk−2​ηs​(st−Fθ​(st−1,at−1))+σstate​ξtk,\displaystyle\leftarrow s_{t}^{k}-2\eta_{s}\Bigl(s_{t}-F_{\theta}(s_{t-1},a_{t-1})\Bigr)+\sigma_{\text{state}}\xi_{t}^{k}, (11)
atk+1\displaystyle a_{t}^{k+1} ←atk−ηa​∇atℒ​(𝐬k,𝐚k),\displaystyle\leftarrow a_{t}^{k}-\eta_{a}\nabla_{a_{t}}\mathcal{L}(\mathbf{s}^{k},\mathbf{a}^{k}), (12)

3.4 Full-rollout synchronization

Periodic synchronization addresses the gap between the relaxed stochastic search and an ordinary rollout. A brief refinement phase optimizes actions through the full model composition, using the original goal-reaching loss.

Full-gradient rollout step.

At a chosen cadence, a small number of full-backpropagation action updates refine the trajectory. The practical algorithm therefore mixes robust local search with occasional exposure to the brittle long-rollout derivatives it otherwise avoids. Both cadence and refinement count are consequential hyperparameters.

Equations 13–15
min𝐚⁡‖sT​(𝐚,s0)−g‖22,\min_{\mathbf{a}}\|s_{T}(\mathbf{a},s_{0})-g\|_{2}^{2}, (13)
st+1=Fθ​(st,at),s0​ given.s_{t+1}=F_{\theta}(s_{t},a_{t}),\qquad s_{0}\text{ given}. (14)
𝐚←𝐚−ηsync​∇𝐚‖sT​(𝐚,s0)−g‖22,\mathbf{a}\leftarrow\mathbf{a}-\eta_{\mathrm{sync}}\nabla_{\mathbf{a}}\|s_{T}(\mathbf{a},s_{0})-g\|_{2}^{2}, (15)

4 Results

The experiments ask whether lifting and noise escape greedy solutions, remain effective at long horizons, and improve time to a successful plan. The retained trajectories and ablations show why success alone is insufficient: an implausible relaxed path must eventually agree with executable dynamics.

H=40H{=}40 H=50H{=}50 H=60H{=}60 H=70H{=}70 H=80H{=}80
Succ. / Time Succ. / Time Succ. / Time Succ. / Time Succ. / Time
CEM 61.4 / 35.3s 30.2 / 96.2s 7.2 / 83.1s 7.8 / 156.1s 2.8 / 132.2s
LatCo 15.0 / 598.0s 4.2 / 1114.7s 2.0 / 231.5s 0.0 / — 0.0 / —
GD 51.0 / 18.0s 37.6 / 76.3s 16.4 / 146.5s 12.0 / 103.1s 6.4 / 161.3s
GRASP (Ours) 59.0 / 8.5s 43.4 / 15.2s 26.2 / 49.1s 16.0 / 79.9s 10.4 / 58.9s
Table 1: Open-loop planning results on long range Push-T. Reported are success rate (%) and median success time (seconds; successful trials only) across planning horizons. 500 trials per setting. Each cell reports Success / Time.

4.1 Baselines

Comparators are cross-entropy sampling, direct action-gradient descent, and a lifted LatCo-style optimizer. Hyperparameters are searched separately by task and horizon. GRASP initializes virtual states near an interpolated start–goal path and actions near zero; this initialization is part of the evaluated recipe.

Original figure
(a) Point Maze (H=50H=50)
Original figure
(b) Push-T (H=50H=50)
Figure 5: Success rate over time at a fixed horizon. Success rate over fixed set of open-loop planning tasks for CEM, GD, LatCo (Rybkin et al., 2021), and our planner for a fixed horizon of 50. Curves summarize how quickly each planner makes progress under the learned world model setting when evaluated at a fixed planning horizon. Shaded regions are Wald 95% confidence intervals.

4.2 Environments and evaluation protocol

PointMaze, Wall-Single, and Push-T use visual world models trained through the DINO-WM framework. A model step consumes action chunks and predicts several physical steps. The reported protocol defines goal-reaching over these learned-model planning tasks; it should not be generalized into a claim of real-robot deployment.

4.3 Long-term planning and horizon scaling

The main gains occur when longer horizons require non-greedy routes. GRASP solves more evaluated tasks and reaches successful plans faster than the selected baselines in those regimes. The success-versus-time plots complement median times, since a median over successful trials alone hides failures.

Table 2: Ablation Studies over the GD sync steps, level of noise for Langevin dynamics, and whether we use our detached gradient approach with the goal-reaching objective or not. GD sync happens every 100 stochastic steps. Ablations done on the Push-T environment at horizon H=40H=40. Time reported is median over successful trials, which only beats our method when the accuracy is much smaller.
Setting Accuracy (%) Time (s)
Sync Steps
No Sync 1.6 4.5
10 48.0 11.2
25 (ours) 59.0 8.5
50 58.0 9.8
Noise Level
σ=0.0\sigma=0.0 54.8 10.4
σ=0.5\sigma=0.5 (ours) 59.0 8.5
σ=1.0\sigma=1.0 50.6 8.4
Detach ∇sF​(s,a)\nabla_{s}F(s,a)
Stop (ours) 59.0 8.5
Flow 46.646.6 10.3
Push-T PointMaze WallSingle
Planner H=10H{=}10 H=20H{=}20 H=30H{=}30 H=10H{=}10 H=20H{=}20 H=30H{=}30 H=10H{=}10 H=20H{=}20 H=30H{=}30
CEM 100.0 96.0 76.0 100.0 100.0 100.0 98.0 100.0 100.0
LatCo 67.4 55.2 35.4 100.0 99.8 99.0 91.4 81.6 62.8
GD 99.2 88.4 69.0 94.6 91.8 94.6 91.8 92.8 91.6
GRASP (Ours) 100.0 89.2 75.2 100.0 100.0 99.8 95.2 91.8 95.0
Table 3: Short Term Planning. Success rate (%) for Push-T, PointMaze, and WallSingle. 500 trials per setting. Our method has comparable success rate while having a consistently low completion time (Table 4).
Push-T PointMaze WallSingle
Planner H=10H{=}10 H=20H{=}20 H=30H{=}30 H=10H{=}10 H=20H{=}20 H=30H{=}30 H=10H{=}10 H=20H{=}20 H=30H{=}30
CEM 1.3 7.5 23.6 0.7 2.0 6.3 0.9 1.5 2.1
LatCo 39.9 437.2 1800.0 1.6 13.7 36.6 12.1 23.8 111.1
GD 0.5 2.4 25.3 0.3 0.6 1.2 0.4 0.5 0.7
GRASP (Ours) 1.5 2.1 9.1 0.7 1.8 2.1 1.6 1.8 2.1
Table 4: Median completion times (seconds) across short-term experiments. 500 trials per setting. Our method has a consistently low completion time with a comparable success rate (Table 3).

4.4 Short-term planning

At shorter horizons, the method is generally competitive rather than uniformly superior. Its extra exploration can cost time relative to a simpler gradient planner while improving success. The full success and timing tables should be read together rather than reducing the result to one universal speedup.

5 Related work

The method connects sampling planners, differentiable trajectory optimization, multiple shooting, and noisy optimization. Its distinctive contribution is the combination suited to imperfect learned visual dynamics. It does not claim to introduce state lifting itself, and unlike world-model adversarial training it leaves the existing predictor unchanged.

6 Limitations and future work

Benefits shrink on short easy tasks, and several stabilizing components require tuning. A smoother learned state geometry could reduce the need for detached gradients and synchronization. Hybridizing GRASP with other planners is left open. No empirical result establishes global optimality or removes errors in the learned dynamics.

7 Conclusion

The paper demonstrates that planner design can substantially change the usefulness of the same learned world model. Time-parallel virtual states and action-focused gradient updates are especially useful in its long-horizon experiments; successful application still depends on representation geometry, model fidelity, and optimization settings.

Appendix A Theory

The appendix separates several restricted analytical arguments: conditioning of lifted objectives, Gaussian smoothing, the distribution induced by state-only noise, and the consequences of stop-gradient updates. These motivate individual components but do not combine into a global convergence proof for the full nonlinear planner.

A.1 Convergence of various methods in the convex setting

A linear-system analysis compares the horizon dependence of shooting and lifted optimization. The useful distinction is the smoothness of the optimization landscape, not a guarantee that every learned visual dynamics model satisfies the convex assumptions.

Equations 16–20
st+1=A​st+B​at,s_{t+1}=As_{t}+Ba_{t}, (16)
ℱ⁡(s0,𝐚)=AT​s0+∑t=0T−1AT−1−t​B​at.\mathcal{F}(s_{0},\mathbf{a})=A^{T}s_{0}+\sum_{t=0}^{T-1}A^{T-1-t}Ba_{t}. (17)
min𝐚∈ℝm​T⁡JS​(𝐚)=‖ℱ⁡(si​n​i​t,𝐚)−g‖22\min_{\mathbf{a}\in\mathbb{R}^{mT}}J_{S}(\mathbf{a})=\|\mathcal{F}(s_{init},\mathbf{a})-g\|_{2}^{2} (18)
min𝐚,𝐬JL​(𝐚,𝐬)\displaystyle\min_{\mathbf{a},\mathbf{s}}\quad J_{L}(\mathbf{a},\mathbf{s}) =∑t=0T−1‖A​st+B​at−st+1‖22\displaystyle=\sum_{t=0}^{T-1}\|As_{t}+Ba_{t}-s_{t+1}\|_{2}^{2} (19)
subject to s0=si​n​i​t,sT=g\displaystyle s_{0}=s_{init},\quad s_{T}=g (20)

Shooting Method.

Composing an unstable linear dynamics model can make the shooting Hessian grow exponentially with horizon under the stated reachability assumptions. This explains why a step size safe for a short rollout can fail on a much longer one.

Equations 21–22
sT=AT​si​n​i​t+𝒞T​𝐚s_{T}=A^{T}s_{init}+\mathcal{C}_{T}\mathbf{a} (21)
JS​(𝐚)=‖𝒞T​𝐚−(g−AT​si​n​i​t)‖22J_{S}(\mathbf{a})=\|\mathcal{C}_{T}\mathbf{a}-(g-A^{T}s_{init})\|_{2}^{2} (22)

Lifted Method.

The lifted objective produces a sparse, locally coupled quadratic system. Its Hessian norm is bounded using the one-step state and action matrices rather than their long products. The resulting upper bound is independent of horizon, although conditioning and feasibility still require their own assumptions.

Equations 23–38
LS=λmax​(HS)≥2​μ2​|λ|2​(T−1).L_{S}=\lambda_{\max}(H_{S})\geq 2\mu^{2}|\lambda|^{2(T-1)}. (23)
‖𝒞T‖2=sup𝐚≠0‖𝒞T​𝐚‖2‖𝐚‖2.\|\mathcal{C}_{T}\|_{2}=\sup_{\mathbf{a}\neq 0}\frac{\|\mathcal{C}_{T}\mathbf{a}\|_{2}}{\|\mathbf{a}\|_{2}}. (24)
𝒞T​𝐚t​e​s​t=AT−1​B​w,\mathcal{C}_{T}\mathbf{a}_{test}=A^{T-1}Bw, (25)
‖𝒞T​𝐚t​e​s​t‖2\displaystyle\|\mathcal{C}_{T}\mathbf{a}_{test}\|_{2} ≥|⟨v,AT−1​B​w⟩|,\displaystyle\geq|\langle v,A^{T-1}Bw\rangle|, (26)
=|⟨(A⊤)T−1​v,B​w⟩|,\displaystyle=|\langle(A^{\top})^{T-1}v,Bw\rangle|, (27)
=|⟨λT−1​v,B​w⟩|,\displaystyle=|\langle\lambda^{T-1}v,Bw\rangle|, (28)
=|λ|T−1​|⟨v,B​w⟩|,\displaystyle=|\lambda|^{T-1}|\langle v,Bw\rangle|, (29)
=μ​|λ|T−1.\displaystyle=\mu|\lambda|^{T-1}. (30)
LL=λmax​(HL)≤6​(1+‖A‖22+‖B‖22).L_{L}=\lambda_{\max}(H_{L})\leq 6\bigl(1+\|A\|_{2}^{2}+\|B\|_{2}^{2}\bigr). (31)
rt=A​st+B​at−st+1,r_{t}\;=\;As_{t}+Ba_{t}-s_{t+1}, (32)
‖rt‖22\displaystyle\|r_{t}\|_{2}^{2} =‖−st+1+A​st+B​at‖22\displaystyle=\|-s_{t+1}+As_{t}+Ba_{t}\,\|_{2}^{2} (33)
≤3​(‖st+1‖22+‖A‖22​‖st‖22+‖B‖22​‖at‖22).\displaystyle\leq 3\Bigl(\|s_{t+1}\|_{2}^{2}+\|A\|_{2}^{2}\|s_{t}\|_{2}^{2}+\|B\|_{2}^{2}\|a_{t}\|_{2}^{2}\Bigr). (34)
‖M​𝐳‖22=∑t=0T−1‖rt‖22≤ 3​∑t=0T−1(‖st+1‖22+‖A‖22​‖st‖22+‖B‖22​‖at‖22).\|M\mathbf{z}\|_{2}^{2}\;=\;\sum_{t=0}^{T-1}\|r_{t}\|_{2}^{2}\;\leq\;3\sum_{t=0}^{T-1}\Bigl(\|s_{t+1}\|_{2}^{2}+\|A\|_{2}^{2}\|s_{t}\|_{2}^{2}+\|B\|_{2}^{2}\|a_{t}\|_{2}^{2}\Bigr). (35)
‖M​𝐳‖22\displaystyle\|M\mathbf{z}\|_{2}^{2} ≤3​((1+‖A‖22)​∑t=1T−1‖st‖22+‖B‖22​∑t=0T−1‖at‖22),\displaystyle\leq 3\Bigl((1+\|A\|_{2}^{2})\sum_{t=1}^{T-1}\|s_{t}\|_{2}^{2}\;+\;\|B\|_{2}^{2}\sum_{t=0}^{T-1}\|a_{t}\|_{2}^{2}\Bigr), (36)
≤3​(1+‖A‖22+‖B‖22)​‖𝐳‖22.\displaystyle\leq 3\bigl(1+\|A\|_{2}^{2}+\|B\|_{2}^{2}\bigr)\,\|\mathbf{z}\|_{2}^{2}. (37)
LL= 2​‖M‖22≤ 6​(1+‖A‖22+‖B‖22),L_{L}\;=\;2\|M\|_{2}^{2}\;\leq\;6\bigl(1+\|A\|_{2}^{2}+\|B\|_{2}^{2}\bigr), (38)

Interpretation.

Lifting acts as a form of optimization preconditioning by replacing long dependencies with neighboring constraints. The horizon-independent smoothness result belongs to the analyzed linear objective; it is motivation for, not a blanket guarantee about, the nonlinear experiments.

A.2 Gaussian noise regularization

Gaussian convolution smooths a loss, with gradient bounds controlled by noise variance under integrability assumptions. Noisy updates consequently see smoothed gradients in expectation. Annealing noise trades broad exploration for more precise local refinement; the theorem does not choose an optimal finite noise schedule.

Equations 39–55
ϕσ(ϵ)=(2πσ2)−d/2exp(−∥ϵ∥2/(2σ2))\phi_{\sigma}(\bm{\epsilon})=(2\pi\sigma^{2})^{-d/2}\exp\!\big(-\|\bm{\epsilon}\|^{2}/(2\sigma^{2})\big) (39)
Lσ​(𝐬)=(ϕσ∗L)​(𝐬)=∫ℝdϕσ​(ϵ)​L​(𝐬−ϵ)​𝑑ϵ.L_{\sigma}(\mathbf{s})=(\phi_{\sigma}*L)(\mathbf{s})=\int_{\mathbb{R}^{d}}\phi_{\sigma}(\bm{\epsilon})\,L(\mathbf{s}-\bm{\epsilon})\,d\bm{\epsilon}. (40)
‖∇sLσ‖Lp≤‖∇sL‖Lp.\|\nabla_{s}L_{\sigma}\|_{L^{p}}\ \leq\ \|\nabla_{s}L\|_{L^{p}}. (41)
‖∇aLσ‖Lp≤cdσ​‖L‖Lp,cd:=𝔼​‖Z‖=2​Γ⁡(d+12)Γ⁡(d2),Z∼𝒩⁡(0,𝐈d).\|\nabla_{a}L_{\sigma}\|_{L^{p}}\ \leq\ \frac{c_{d}}{\sigma}\,\|L\|_{L^{p}},\qquad c_{d}:=\mathbb{E}\|Z\|=\sqrt{2}\,\frac{\Gamma\big(\frac{d+1}{2}\big)}{\Gamma\big(\frac{d}{2}\big)},\ \ Z\sim\mathcal{N}(0,\mathbf{I}_{d}). (42)
∇sLσ\displaystyle\nabla_{s}L_{\sigma} =∇s(ϕσ∗L),\displaystyle=\nabla_{s}(\phi_{\sigma}*L), (43)
=(∇ϕσ)∗L,\displaystyle=(\nabla\phi_{\sigma})*L, (44)
=ϕσ∗(∇sL),\displaystyle=\phi_{\sigma}*(\nabla_{s}L), (45)
‖g∗ϕσ‖Lp≤‖g‖Lp.\|g*\phi_{\sigma}\|_{L^{p}}\leq\|g\|_{L^{p}}. (46)
‖∇sLσ‖Lp=‖ϕσ∗(∇sL)‖Lp≤‖∇sL‖Lp.\|\nabla_{s}L_{\sigma}\|_{L^{p}}=\|\phi_{\sigma}*(\nabla_{s}L)\|_{L^{p}}\leq\|\nabla_{s}L\|_{L^{p}}. (47)
‖∇aLσ‖Lp=‖(∇ϕσ)∗L‖Lp≤‖∇ϕσ‖L1​‖L‖Lp.\|\nabla_{\!a}L_{\sigma}\|_{L^{p}}=\|(\nabla\phi_{\sigma})*L\|_{L^{p}}\leq\|\nabla\phi_{\sigma}\|_{L^{1}}\,\|L\|_{L^{p}}. (48)
‖∇ϕσ‖L1\displaystyle\|\nabla\phi_{\sigma}\|_{L^{1}} =∫ℝd‖ϵ‖2σ2​ϕσ​(ϵ)​𝑑ϵ\displaystyle=\int_{\mathbb{R}^{d}}\frac{\|\bm{\epsilon}\|_{2}}{\sigma^{2}}\,\phi_{\sigma}(\bm{\epsilon})\,d\bm{\epsilon} (49)
=1σ2​𝔼​‖X‖2\displaystyle=\frac{1}{\sigma^{2}}\,\mathbb{E}\|X\|_{2} (50)
=1σ​𝔼​‖Z‖2\displaystyle=\frac{1}{\sigma}\,\mathbb{E}\|Z\|_{2} (51)
=cdσ,\displaystyle=\frac{c_{d}}{\sigma}, (52)
Lσ​(𝐬):=𝔼𝝃​[L⁡(𝐬+𝝃)].L_{\sigma}(\mathbf{s}):=\mathbb{E}_{\bm{\xi}}\big[L(\mathbf{s}+\bm{\xi})\big]. (53)
𝔼𝝃​[∇L​(𝐬+𝝃)]=∇𝐬Lσ​(𝐬).\mathbb{E}_{\bm{\xi}}\big[\nabla L(\mathbf{s}+\bm{\xi})\big]=\nabla_{\!\mathbf{s}}L_{\sigma}(\mathbf{s}). (54)
‖∇𝐬Lσ‖L∞≤cdσ​‖L‖L∞,and hence‖𝔼𝝃​[∇L​(𝐬+𝝃)]‖≤cdσ​‖L‖L∞.\|\nabla_{\!\mathbf{s}}L_{\sigma}\|_{L^{\infty}}\leq\frac{c_{d}}{\sigma}\,\|L\|_{L^{\infty}},\qquad\text{and hence}\qquad\Big\|\mathbb{E}_{\bm{\xi}}\big[\nabla L(\mathbf{s}+\bm{\xi})\big]\Big\|\leq\frac{c_{d}}{\sigma}\,\|L\|_{L^{\infty}}. (55)

A.3 Connection to Boltzmann sampling, state-only noising

Adding properly matched noise to all variables can yield a Gibbs stationary distribution under suitable conditions. GRASP perturbs states but not actions, so that standard Boltzmann interpretation does not apply directly. The appendix describes its more constrained action-conditioned behavior rather than claiming exact equilibrium sampling.

Equations 56–57
p⁡(𝐬,𝐚)∝exp⁡(−β​ℒdyn​(𝐬,𝐚)),p(\mathbf{s},\mathbf{a})\ \propto\ \exp\big(-\beta\,\mathcal{L}_{\mathrm{dyn}}(\mathbf{s},\mathbf{a})\big), (56)
p⁡(𝐬,𝐚)∝δ⁡(𝐚=𝐚∗​(𝐬))⋅exp⁡(−β​ℒdyn​(𝐬,𝐚)),p(\mathbf{s},\mathbf{a})\propto\delta(\mathbf{a}=\mathbf{a}^{*}(\mathbf{s}))\cdot\exp\big(-\beta\,\mathcal{L}_{\mathrm{dyn}}(\mathbf{s},\mathbf{a})\big), (57)

A.4 State-gradient-free Dynamics

The appendix proves that a general exact dynamics-enforcing objective cannot have state gradients independent of the dynamics Jacobian. For a linear detached update with positive goal weights and full-column-rank action matrix, it establishes convergence to a unique fixed point. That fixed point need not be a globally optimal feasible trajectory for the original nonlinear problem.

Equations 58–91
LF​(s1,a0,a1)\displaystyle L_{F}(s_{1},a_{0},a_{1}) =Φ⁡(s1,a0,a1,y0,y1),\displaystyle=\Phi(s_{1},a_{0},a_{1},y_{0},y_{1}), (58)
y0\displaystyle y_{0} =F⁡(s0,a0),\displaystyle=F(s_{0},a_{0}), (59)
y1\displaystyle y_{1} =F⁡(s1,a1),\displaystyle=F(s_{1},a_{1}), (60)
ℳ⁡(F)={(s1,a0,a1):s1=F⁡(s0,a0)​and​g=F⁡(s1,a1)}\mathcal{M}(F)=\{(s_{1},a_{0},a_{1}):s_{1}=F(s_{0},a_{0})\ \text{and}\ g=F(s_{1},a_{1})\} (61)
∇s1LF​(s1,a0,a1)=∂Φ∂s1​(s1,a0,a1,y0,y1)+(∇sF​(s1,a1))⊤​∂Φ∂y1​(s1,a0,a1,y0,y1).\nabla_{s_{1}}L_{F}(s_{1},a_{0},a_{1})=\frac{\partial\Phi}{\partial s_{1}}(s_{1},a_{0},a_{1},y_{0},y_{1})+\big(\nabla_{s}F(s_{1},a_{1})\big)^{\top}\frac{\partial\Phi}{\partial y_{1}}(s_{1},a_{0},a_{1},y_{0},y_{1}). (62)
∇sF​(s1,a1)=JF,∇sG​(s1,a1)=JG.\nabla_{s}F(s_{1},a_{1})=J_{F},\qquad\nabla_{s}G(s_{1},a_{1})=J_{G}. (63)
F⁡(s,a)=H⁡(s,a)+ψ⁡(s,a)​((y1−H⁡(s1,a1))+(JF−∇sH​(s1,a1))​(s−s1)),F(s,a)=H(s,a)+\psi(s,a)\Big((y_{1}-H(s_{1},a_{1}))+(J_{F}-\nabla_{s}H(s_{1},a_{1}))(s-s_{1})\Big), (64)
(JF−JG)⊤​∂Φ∂y1​(s1,a0,a1,y0,y1)=0.(J_{F}-J_{G})^{\top}\frac{\partial\Phi}{\partial y_{1}}(s_{1},a_{0},a_{1},y_{0},y_{1})=0. (65)
∂Φ∂y1​(s1,a0,a1,y0,y1)=0.\frac{\partial\Phi}{\partial y_{1}}(s_{1},a_{0},a_{1},y_{0},y_{1})=0. (66)
LF​(s1,a0,a1)=Φ~​(s1,a0,a1,F⁡(s0,a0)).L_{F}(s_{1},a_{0},a_{1})=\widetilde{\Phi}(s_{1},a_{0},a_{1},F(s_{0},a_{0})). (67)
arg⁡min⁡LF=arg⁡min⁡LG.\arg\min L_{F}=\arg\min L_{G}. (68)
F⁡(s0,a)=G⁡(s0,a) for all ​a∈𝒜,F(s_{0},a)=G(s_{0},a)\quad\text{ for all }a\in\mathcal{A}, (69)
F⁡(s0,u)\displaystyle F(s_{0},u) =G⁡(s0,u)=sA,\displaystyle=G(s_{0},u)=s_{A}, (70)
F⁡(s0,v)\displaystyle F(s_{0},v) =G⁡(s0,v)=sB,\displaystyle=G(s_{0},v)=s_{B}, (71)
F⁡(sA,a⋆)\displaystyle F(s_{A},a^{\star}) =g,\displaystyle=g, F⁡(sB,a⋆)\displaystyle F(s_{B},a^{\star}) ≠g,\displaystyle\neq g, (72)
G⁡(sA,a⋆)\displaystyle G(s_{A},a^{\star}) ≠g,\displaystyle\neq g, G⁡(sB,a⋆)\displaystyle G(s_{B},a^{\star}) =g.\displaystyle=g. (73)
arg⁡min⁡LF=ℳ⁡(F)andarg⁡min⁡LG=ℳ⁡(G),\arg\min L_{F}=\mathcal{M}(F)\quad\text{and}\quad\arg\min L_{G}=\mathcal{M}(G), (74)
ℒs​g​(𝐬,𝐚)=\displaystyle\mathcal{L}_{sg}(\mathbf{s},\mathbf{a})= ∑t=0T−1‖st+1−A​sg​(st)−B​at‖22\displaystyle\sum_{t=0}^{T-1}\|s_{t+1}-A\text{sg}(s_{t})-Ba_{t}\|_{2}^{2}
+∑t=0T−1βt∥g−Asg(st)−Bat∥22\displaystyle+\sum_{t=0}^{T-1}\beta_{t}\|g-A\text{sg}(s_{t})-Ba_{t}\|_{2}^{2} (75)
(𝐬k+1,𝐚k+1)=(𝐬k,𝐚k)−η∇ℒs​g(𝐬k,𝐚k)(\mathbf{s}^{k+1},\mathbf{a}^{k+1})=(\mathbf{s}^{k},\mathbf{a}^{k})-\eta\nabla\mathcal{L}_{sg}(\mathbf{s}^{k},\mathbf{a}^{k})
βt∈[βmin,βmax]for all ​t,βmin>0,\beta_{t}\in[\beta_{\min},\beta_{\max}]\quad\text{for all }t,\qquad\beta_{\min}>0,
‖𝐳k−𝐳⋆‖≤C​qk​‖𝐳0−𝐳⋆‖,for some ​q∈(0,1)​ and ​C>0.\|\mathbf{z}^{k}-\mathbf{z}^{\star}\|\leq Cq^{k}\|\mathbf{z}^{0}-\mathbf{z}^{\star}\|,\qquad\text{for some }q\in(0,1)\text{ and }C>0.
ℒs​g​(𝐬,𝐚)=∑t=0T−1‖st+1−A​sg​(st)−B​at‖22+∑t=0T−1βt​‖g−A​sg​(st)−B​at‖22.\mathcal{L}_{sg}(\mathbf{s},\mathbf{a})=\sum_{t=0}^{T-1}\|s_{t+1}-A\,\mathrm{sg}(s_{t})-Ba_{t}\|_{2}^{2}\;+\;\sum_{t=0}^{T-1}\beta_{t}\|g-A\,\mathrm{sg}(s_{t})-Ba_{t}\|_{2}^{2}. (76)
rt:=st+1−A​st−B​at,et:=g−A​st−B​at.r_{t}:=s_{t+1}-As_{t}-Ba_{t},\qquad e_{t}:=g-As_{t}-Ba_{t}. (77)
∇stℒs​g=2rt−1,t=1,…,T,\nabla_{s_{t}}\mathcal{L}_{sg}=2r_{t-1},\qquad t=1,\dots,T, (78)
∇atℒs​g=−2B⊤rt−2βtB⊤et,t=0,…,T−1.\nabla_{a_{t}}\mathcal{L}_{sg}=-2B^{\top}r_{t}-2\beta_{t}B^{\top}e_{t},\qquad t=0,\dots,T-1. (79)
stk+1=stk−2η(stk−Ast−1k−Bat−1k),t=1,…,T,s_{t}^{k+1}=s_{t}^{k}-2\eta\big(s_{t}^{k}-As_{t-1}^{k}-Ba_{t-1}^{k}\big),\qquad t=1,\dots,T, (80)
atk+1=atk+2ηB⊤((st+1k−Astk−Batk)+βt(g−Astk−Batk)),t=0,…,T−1.a_{t}^{k+1}=a_{t}^{k}+2\eta B^{\top}\Big(\big(s_{t+1}^{k}-As_{t}^{k}-Ba_{t}^{k}\big)+\beta_{t}\big(g-As_{t}^{k}-Ba_{t}^{k}\big)\Big),\qquad t=0,\dots,T-1. (81)
ϕt​(st+1,at,st)=‖st+1−A​st−B​at‖22+βt​‖g−A​st−B​at‖22,\phi_{t}(s_{t+1},a_{t};s_{t})=\|s_{t+1}-As_{t}-Ba_{t}\|_{2}^{2}+\beta_{t}\|g-As_{t}-Ba_{t}\|_{2}^{2}, (82)
Ht=2​[I−B−B⊤(1+βt)​B⊤​B].H_{t}=2\begin{bmatrix}I&-B\\[1.99997pt] -B^{\top}&(1+\beta_{t})B^{\top}B\end{bmatrix}. (83)
(1+βt)​B⊤​B−B⊤​I−1​B=βt​B⊤​B≻0,(1+\beta_{t})B^{\top}B-B^{\top}I^{-1}B=\beta_{t}B^{\top}B\succ 0, (84)
0<μ​I⪯Ht⪯L​I<∞,0<\mu I\preceq H_{t}\preceq LI<\infty, (85)
ρ⁡(I−η​Ht)≤max⁡{|1−η​μ|,|1−η​L|}=:q<1,\rho(I-\eta H_{t})\leq\max\{|1-\eta\mu|,\ |1-\eta L|\}=:q<1, (86)
ρ⁡(J)=maxt⁡ρ⁡(I−η​Ht)≤q0<1.\rho(J)=\max_{t}\rho(I-\eta H_{t})\leq q_{0}<1. (87)
‖J‖†≤q<1.\|J\|_{\dagger}\leq q<1. (88)
‖𝐳k+1−𝐳⋆‖†=‖J⁡(𝐳k−𝐳⋆)‖†≤‖J‖†​‖𝐳k−𝐳⋆‖†≤q​‖𝐳k−𝐳⋆‖†,\|\mathbf{z}^{k+1}-\mathbf{z}^{\star}\|_{\dagger}=\|J(\mathbf{z}^{k}-\mathbf{z}^{\star})\|_{\dagger}\leq\|J\|_{\dagger}\,\|\mathbf{z}^{k}-\mathbf{z}^{\star}\|_{\dagger}\leq q\,\|\mathbf{z}^{k}-\mathbf{z}^{\star}\|_{\dagger}, (89)
‖𝐳k−𝐳⋆‖†≤qk​‖𝐳0−𝐳⋆‖†.\|\mathbf{z}^{k}-\mathbf{z}^{\star}\|_{\dagger}\leq q^{k}\|\mathbf{z}^{0}-\mathbf{z}^{\star}\|_{\dagger}. (90)
‖𝐳k−𝐳⋆‖2≤C​qk​‖𝐳0−𝐳⋆‖2.\|\mathbf{z}^{k}-\mathbf{z}^{\star}\|_{2}\leq Cq^{k}\|\mathbf{z}^{0}-\mathbf{z}^{\star}\|_{2}. (91)

Notes on the stopgrad optimization.

With one-step predictions held fixed, noisy virtual-state updates form a Gaussian neighborhood around those predictions. Action updates move that neighborhood toward the goal through controllable directions. Ordinary noisy rollouts instead propagate randomness through time and can drift or spread. These conditional calculations clarify the mechanism while retaining their fixed-center, linearization, and stability assumptions.

Equations 92–105
μt≔Fθ​(s¯t,at),s¯t=stopgrad⁡(st).\mu_{t}\;\coloneqq\;F_{\theta}(\bar{s}_{t},a_{t}),\qquad\bar{s}_{t}=\mathrm{stopgrad}(s_{t}).
ℒ⁡(𝐬,𝐚)=∑t=0T−1‖μt−st+1‖22+γ​∑t=0T−1‖μt−g‖22,\mathcal{L}(\mathbf{s},\mathbf{a})=\sum_{t=0}^{T-1}\|\mu_{t}-s_{t+1}\|_{2}^{2}\;+\;\gamma\sum_{t=0}^{T-1}\|\mu_{t}-g\|_{2}^{2}, (92)
st+1k+1\displaystyle s_{t+1}^{k+1} =st+1k−2​ηs​(st+1k−μtk)+σ​ξt+1k,ξt+1k∼𝒩⁡(0,I),\displaystyle=s_{t+1}^{k}-2\eta_{s}\bigl(s_{t+1}^{k}-\mu_{t}^{k}\bigr)+\sigma\,\xi_{t+1}^{k},\qquad\xi_{t+1}^{k}\sim\mathcal{N}(0,I), (93)
atk+1\displaystyle a_{t}^{k+1} =atk−ηa​∇atℒ​(𝐬k,𝐚k).\displaystyle=a_{t}^{k}-\eta_{a}\nabla_{a_{t}}\mathcal{L}(\mathbf{s}^{k},\mathbf{a}^{k}). (94)
𝔼⁡[st+1k+1∣μtk]=(1−2​ηs)​𝔼​[st+1k∣μtk]+2​ηs​μtk.\mathbb{E}\!\left[s_{t+1}^{k+1}\mid\mu_{t}^{k}\right]=(1-2\eta_{s})\,\mathbb{E}\!\left[s_{t+1}^{k}\mid\mu_{t}^{k}\right]+2\eta_{s}\,\mu_{t}^{k}. (95)
st+1∞∼𝒩⁡(μt,Σtube),Σtube=σ21−(1−2​ηs)2​I=σ24​ηs​(1−ηs)​I.s_{t+1}^{\infty}\ \sim\ \mathcal{N}\!\bigl(\mu_{t},\ \Sigma_{\mathrm{tube}}\bigr),\qquad\Sigma_{\mathrm{tube}}=\frac{\sigma^{2}}{1-(1-2\eta_{s})^{2}}\,I=\frac{\sigma^{2}}{4\eta_{s}(1-\eta_{s})}\,I. (96)
d​st+1​(τ)=−λ⁡(st+1​(τ)−μt)​d​τ+σ​d​Wt+1​(τ),λ>0,ds_{t+1}(\tau)=-\lambda\bigl(s_{t+1}(\tau)-\mu_{t}\bigr)d\tau+\sigma\,dW_{t+1}(\tau),\quad\lambda>0,
st+1k+1=(1−2​ηs)​st+1k+2​ηs​μtk+σ​ξt+1k.s_{t+1}^{k+1}=(1-2\eta_{s})s_{t+1}^{k}+2\eta_{s}\mu_{t}^{k}+\sigma\xi_{t+1}^{k}.
Jtk≔∇atFθ​(s¯tk,atk)∈ℝds×da,Ptk≔Jtk​(Jtk)⊤⪰0.J_{t}^{k}\;\coloneqq\;\nabla_{a_{t}}F_{\theta}(\bar{s}_{t}^{k},a_{t}^{k})\in\mathbb{R}^{d_{s}\times d_{a}},\qquad P_{t}^{k}\;\coloneqq\;J_{t}^{k}(J_{t}^{k})^{\top}\succeq 0.
μtk+1≈μtk+Jtk​(atk+1−atk).\mu_{t}^{k+1}\approx\mu_{t}^{k}+J_{t}^{k}\,(a_{t}^{k+1}-a_{t}^{k}). (97)
μtk+1≈(I−α​γ​Ptk)​μtk+α​γ​Ptk​g⏟goal-directed averaging (drift)+α​Ptk​εt+1k⏟exploration forcing,α≔2​ηa,\mu_{t}^{k+1}\;\approx\;\underbrace{\bigl(I-\alpha\gamma P_{t}^{k}\bigr)\mu_{t}^{k}+\alpha\gamma P_{t}^{k}g}_{\textbf{goal-directed averaging (drift)}}\;+\;\underbrace{\alpha\,P_{t}^{k}\,\varepsilon_{t+1}^{k}}_{\textbf{exploration forcing}},\qquad\alpha\coloneqq 2\eta_{a}, (98)
𝔼[μtk+1∣μtk,stk,atk]≈(I−αγPtk)μtk+αγPtkg,\mathbb{E}[\mu_{t}^{k+1}\mid\mu_{t}^{k},s_{t}^{k},a_{t}^{k}]\;\approx\;\bigl(I-\alpha\gamma P_{t}^{k}\bigr)\mu_{t}^{k}+\alpha\gamma P_{t}^{k}g, (99)
∇atℒ=2​(Jt)⊤​(μt−st+1)+2​γ​(Jt)⊤​(μt−g).\nabla_{a_{t}}\mathcal{L}=2(J_{t})^{\top}(\mu_{t}-s_{t+1})+2\gamma(J_{t})^{\top}(\mu_{t}-g). (100)
atk+1−atk=−ηa​∇atℒ=−2​ηa​(Jtk)⊤​(μtk−st+1k)−2​ηa​γ​(Jtk)⊤​(μtk−g).a_{t}^{k+1}-a_{t}^{k}=-\eta_{a}\nabla_{a_{t}}\mathcal{L}=-2\eta_{a}(J_{t}^{k})^{\top}(\mu_{t}^{k}-s_{t+1}^{k})-2\eta_{a}\gamma(J_{t}^{k})^{\top}(\mu_{t}^{k}-g).
μtk+1\displaystyle\mu_{t}^{k+1} ≈μtk−2​ηa​Jtk​(Jtk)⊤​(μtk−st+1k)−2​ηa​γ​Jtk​(Jtk)⊤​(μtk−g)\displaystyle\approx\mu_{t}^{k}-2\eta_{a}J_{t}^{k}(J_{t}^{k})^{\top}(\mu_{t}^{k}-s_{t+1}^{k})-2\eta_{a}\gamma J_{t}^{k}(J_{t}^{k})^{\top}(\mu_{t}^{k}-g)
=μtk−2​ηa​Ptk​(μtk−(μtk+εt+1k))−2​ηa​γ​Ptk​(μtk−g),\displaystyle=\mu_{t}^{k}-2\eta_{a}P_{t}^{k}(\mu_{t}^{k}-(\mu_{t}^{k}+\varepsilon_{t+1}^{k}))-2\eta_{a}\gamma P_{t}^{k}(\mu_{t}^{k}-g),
st+1=Fθ(st,at)+σenvζt,ζt∼𝒩(0,I),s0fixed,s_{t+1}=F_{\theta}(s_{t},a_{t})+\sigma_{\mathrm{env}}\zeta_{t},\qquad\zeta_{t}\sim\mathcal{N}(0,I),\qquad s_{0}\ \text{fixed}, (101)
mt+1=𝔼⁡[Fθ​(st,at)].m_{t+1}=\mathbb{E}\!\left[F_{\theta}(s_{t},a_{t})\right]. (102)
mt+1=Fθ​(mt,at).m_{t+1}=F_{\theta}(m_{t},a_{t}). (103)
mt+1≈Fθ​(mt,at)+12​(Hesss​Fθ​(mt,at)):Σt,m_{t+1}\approx F_{\theta}(m_{t},a_{t})+\frac{1}{2}\,\big(\mathrm{Hess}_{s}F_{\theta}(m_{t},a_{t})\big):\Sigma_{t}, (104)
Σt+1≈Gt​Σt​Gt⊤+σenv2​I,Gt≔∇sFθ​(μt,at).\Sigma_{t+1}\approx G_{t}\,\Sigma_{t}\,G_{t}^{\top}+\sigma_{\mathrm{env}}^{2}I,\qquad G_{t}\coloneqq\nabla_{s}F_{\theta}(\mu_{t},a_{t}). (105)

Appendix B Ablations

The Push-T ablations isolate noise, detached state gradients, and synchronization, and also compare simpler action- or rollout-noise variants. Their combination is strongest in the tested setup. Noise cannot be treated as interchangeable across state optimization and action rollout, and the ablation does not establish one setting for every horizon.

Equations 106–107
st+1\displaystyle s_{t+1} =F⁡(st,at)+z,\displaystyle=F(s_{t},a_{t})+z, (106)
z\displaystyle z ∼𝒩⁡(0,σs​I).\displaystyle\sim\mathcal{N}(0,\sigma_{s}I). (107)

Appendix C Additional experiments

The additional visual examples expose both successful relaxed trajectories and a bad local minimum that crosses an obstacle. They are useful checks of what the optimizer is doing internally; selected examples supplement the quantitative evaluation rather than replacing it.

Table 5: Stochastic GD Baselines Ablation Study, showing alternatives from our method for a stochastic gradient-based planner. For rollouts of the world model Fθ​(s,a)F_{\theta}(s,a), values of σa\sigma_{a} correspond to variance of noise added to actions throughout optimization, and values of σs\sigma_{s} correspond to noise added to states directly through the rollout; that is, for rollouts of the world model, st+1=Fθ​(st,ai)+ξ,ξ∼𝒩⁡(0,I)s_{t+1}=F_{\theta}(s_{t},a_{i})+\xi,\xi\sim\mathcal{N}(0,I). None beat our method’s performance, shown in Table 2. Reported time is average time over all experiments, with reported intervals as 95% CI.
Setting Accuracy (%) Time (s)
Default Mode (objective.mode=default)
σs=0.0\sigma_{s}=0.0 53.0 ±\pm 4.4 864 ±\pm 77
σs=0.5\sigma_{s}=0.5 50.6 ±\pm 4.4 913.97 ±\pm 78
σs=1.0\sigma_{s}=1.0 37.2 ±\pm 4.3 1149 ±\pm 75
σa=0.0\sigma_{a}=0.0 47.6 ±\pm 4.4 1056 ±\pm 73
σa=0.5\sigma_{a}=0.5 53.0 ±\pm 4.4 864 ±\pm 77
σa=1.0\sigma_{a}=1.0 33.4 ±\pm 4.2 1208 ±\pm 73
σs=0.5,σa=0.5\sigma_{s}=0.5,\sigma_{a}=0.5 53.8 ±\pm 4.4 854 ±\pm 78
Mode All (objective.mode=all)
σs=0.0\sigma_{s}=0.0 54.0 ±\pm 4.4 865 ±\pm 77
σs=0.5\sigma_{s}=0.5 55.8 ±\pm 4.4 838 ±\pm 77
σs=1.0\sigma_{s}=1.0 41.0 ±\pm 4.4 1111 ±\pm 75
σa=0.0\sigma_{a}=0.0 46.6 ±\pm 4.4 1067 ±\pm 73
σa=0.5\sigma_{a}=0.5 52.8 ±\pm 4.4 873 ±\pm 77
σa=1.0\sigma_{a}=1.0 52.4 ±\pm 4.4 878 ±\pm 77
σs=0.5,σa=0.5\sigma_{s}=0.5,\sigma_{a}=0.5 55.8 ±\pm 4.4 838 ±\pm 77
Refer to caption
Figure 6: Example of converged plans for our planner on the Push-T environment at horizon 80, with goals depicted to the right of the dashed line. The parallel optimization is able to converge more consistently at longer horizons, and find the required non-greedy paths to the goal.
Refer to caption
Figure 7: Example of a bad local minima when relaxing the dynamics constraint into a penalty function. The dynamics are nonzero for the middle transition, but both the states and actions are at a local minima; there’s no direction they can move locally to reduce the dynamics loss further.
Concept demonstration

Open image ↗