Literature
Display
← All papers

Temporal Straightening for Latent Planning

Ying Wang · Oumayma Bounou · Gaoyue Zhou · Randall Balestriero · Tim G. J. Rudner · Yann LeCun · Mengye Ren

· International Conference on Machine Learning (ICML 2026) ↗

World models10 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition: a route around a wall can look short in a visual feature space even though it is physically far away. This paper trains a JEPA encoder and predictor while encouraging consecutive latent movements to point in similar directions. The resulting geometry gives gradient-based planners a more useful goal-distance signal in simulated navigation and pushing. A linear-systems analysis connects near-identity transitions to the conditioning of action optimization. That theory is conditional: the cosine penalty only controls visited directions under additional assumptions, and it does not prove global convexity for nonlinear models. Experiments show stronger planning with fewer search computations, while long horizons and asymmetric dynamics remain difficult.

1 Introduction

A good image representation need not be a good planning representation. Visual similarity can ignore walls or make progress toward a goal hard to measure. The proposed regularizer reshapes the representation during predictive training, aiming to make local trajectory directions more consistent and action optimization easier.

Refer to caption
Figure 1: Latent trajectories encoded by a pretrained visual encoder are usually highly curved, increasing the difficulty of prediction and planning. We learn a representation space where feasible trajectories are straighter to facilitate latent planning.
Refer to caption
Refer to caption
Figure 2: Latent trajectories before vs. after straightening. The upper PushT example is a rotation and the bottom UMaze example shows the agent traveling from the top-left to the top-right, with the star denoting the target. Straightening yields less curved and smoother trajectories, and makes Euclidean distance a more faithful proxy for geodesic progress towards the goal. More examples are in Section E.2.
Refer to caption
Figure 3: During training, we minimize the prediction loss between the predicted embedding z^t\hat{z}_{t} and the target embedding zt{z}_{t} with stop-grad in the target branch, and minimize the local curvature of embeddings. During planning, we roll out for the horizon T using the trained predictor and select optimal actions that minimize the cost between the predicted terminal state z^T\hat{z}_{T} and the goal embedding zgz_{g}.

2 Related Work

The method is part of the JEPA world-model family and directly extends frozen-feature planning. Unlike a temporal contrastive objective, it does not push distant-in-time frames apart or require negative pairs. Unlike pixel reconstruction, it targets trajectory geometry. These differences motivate the design; measured comparisons are needed to establish when it helps.

3 Temporal Straightening

The system learns observation features, action features and a future predictor from offline trajectories. Straightening is an additional geometric objective, separate from prediction and collapse prevention.

3.1 World Model

Three modules connect observations and actions to future latent states. Encoders may retain a spatial feature grid or output one global vector.

Sensory encoder.

The observation encoder chooses the state space in which both prediction and planning occur. A learned projector can adapt frozen visual features, or an image encoder can be trained from scratch.

Equation 1
zt∈ℝd=ℰϕs​(ot).\displaystyle z_{t}\in\mathbb{R}^{d}=\mathcal{E}^{s}_{\phi}(o_{t}). (1)

Action encoder.

An action encoder transforms physical control inputs into features suitable for the predictor. It does not itself choose actions.

Equation
ℰψa:ℝna→ℝda.\mathcal{E}^{a}_{\psi}:\mathbb{R}^{n_{a}}\to\mathbb{R}^{d_{a}}.

Predictor.

The predictor uses a history of encoded states and actions to forecast the next state. At deployment its own predicted states supply later context, so long-rollout errors can accumulate.

Equation 2
z^t=fθ​({zi}i=t−Kt−1,{ℰψa​(ai)}i=t−Kt−1).\hat{z}_{t}=f_{\theta}\left(\{z_{i}\}_{i=t-K}^{t-1},\{\mathcal{E}^{a}_{\psi}(a_{i})\}_{i=t-K}^{t-1}\right). (2)

3.2 Straightening Latent Trajectories

Subtract adjacent latent states to obtain approximate movement vectors, then maximize the cosine between consecutive movements. This penalizes changes in direction rather than simply shrinking every step. The desired straightness is local along observed trajectories, not a requirement that all possible routes lie on one line.

Equations 3–4
vt=zt+1−zt,vt+1=zt+2−zt+1,v_{t}=z_{t+1}-z_{t},\quad v_{t+1}=z_{t+2}-z_{t+1}, (3)
𝒞=vt⋅vt+1‖vt‖2⋅‖vt+1‖2.\mathcal{C}=\frac{v_{t}\cdot v_{t+1}}{||v_{t}||_{2}\cdot||v_{t+1}||_{2}}. (4)

3.3 Training Objective.

Prediction and curvature losses jointly train the modules. Their relative weight controls how strongly the state geometry is reshaped.

Prediction objective.

Squared error matches predicted next features to target features whose gradients are stopped. The targets are produced by the learned representation rather than annotated physical states.

Equation 5
ℒp​r​e​d=||z^t+1−sg(zt+1)||22 ,\mathcal{L}_{pred}=||\hat{z}_{t+1}-\operatorname{sg}(z_{t+1})||_{2}^{2}\\ \text{ }, (5)

Straightening objective.

One minus cosine similarity is small when adjacent latent displacements point the same way. It is invariant to uniform scaling of those displacement vectors and therefore does not independently require a large informative representation.

Equation 6
ℒc​u​r​v=1−𝒞.\mathcal{L}_{curv}=1-\mathcal{C}. (6)

Overall objective.

A nonnegative weight combines curvature and predictive accuracy. Excessive geometric pressure could remove useful changes; the empirical setting of this weight matters.

Equation 7
ℒt​o​t​a​l=ℒp​r​e​d+λ​ℒc​u​r​v,\mathcal{L}_{total}=\mathcal{L}_{pred}+\lambda\mathcal{L}_{curv}, (7)

Collapse prevention.

Stop-gradient is used in most experiments, but a constant representation remains theoretically possible. Straightening is not an anti-collapse guarantee. It can be combined with other regularization or contrastive methods; successful training here is empirical.

Refer to caption
(a) DINOv2
Refer to caption
(b) Straightened
Figure 4: Action-Space Loss Landscape. We pick one test sample from PushT with a planning horizon of 25 steps. For each coordinate (ax,ay)(a_{x},a_{y}) in the grid, we fix the first action and optimize the remaining actions in the planning horizon to minimize the terminal goal cost. The heatmap represents the minimum attainable loss for each initial action choice, with darker colors indicating lower loss. The loss landscape is closer to being convex after straightening.

4 Planning with Straightened Dynamics

For linear latent dynamics, the terminal goal loss is quadratic in the action sequence. Its nonzero Hessian spectrum matches a controllability Gramian. If the transition is close to identity and actions are well conditioned, the derived bound on optimization conditioning improves. This first assumes square invertible action mapping; low-dimensional actions require controllability conditions on the reachable subspace. High cosine similarity is only a proxy along visited velocities, and nonlinear predictors need further Jacobian and higher-order analysis. The original assumptions and statements are retained.

Equation 8
ℒ⁡(𝐚)=‖zK−zg‖22,zK=Φ⁡(𝐚),\mathcal{L}(\mathbf{a})=\|z_{K}-z_{g}\|_{2}^{2},\qquad z_{K}=\Phi(\mathbf{a}), (8)
Assumption 4.1 (Linear latent dynamics).

For analysis, we consider linear latent dynamics

Equation 9
f:(zt,at)→A​zt+B​at,s.t.zt+1=A​zt+B​at,f:(z_{t},a_{t})\to Az_{t}+Ba_{t},\qquad\text{s.t.}\quad z_{t+1}=Az_{t}+Ba_{t}, (9)

where A∈ℝd×dA\in\mathbb{R}^{d\times d} and B∈ℝd×daB\in\mathbb{R}^{d\times d_{a}}. We first state results for da=dd_{a}=d and BB invertible; see Remark 4.5 for da<dd_{a}<d.

Definition 4.2 (ε\varepsilon-straight transition).

Under the linear dynamics, we call ff ε\varepsilon-straight if

Equation 10
‖A−I‖2≤ε.\|A-I\|_{2}\leq\varepsilon. (10)

The term “straight” reflects that, as ε\varepsilon tends to 00, the dynamics of ff approach those of the reference function g:(zt,at)→zt+B​atg:(z_{t},a_{t})\to z_{t}+Ba_{t}, where the state evolves linearly along a straight trajectory modified only by the control input. We are primarily interested in the regime where ε\varepsilon is small.

Remark 4.3 (Cosine similarity as a practical proxy).

In practice, we regularize temporal straightness using the cosine similarity between consecutive latent velocities (4). Under mild bounded-variation assumptions on velocity magnitudes and smooth actions, a large cosine similarity implies that (A−I)(A-I) is small along visited velocity directions. Detailed proofs are in Section C.3.

Theorem 4.4 (Conditioning of the planning Hessian).

Under Assumption 4.1 with da=dd_{a}=d and BB invertible, unrolling (9) yields

Equation
zK=AK​z0+∑t=0K−1AK−1−t​B​at,z_{K}=A^{K}z_{0}+\sum_{t=0}^{K-1}A^{K-1-t}Ba_{t},

so zKz_{K} is affine in 𝐚\mathbf{a} and the planning Hessian is

Equation
H:=∇𝐚2​ℒ​(𝐚)=2​JΦ⊤​JΦ⪰0,H:=\nabla_{\mathbf{a}}^{2}\mathcal{L}(\mathbf{a})=2J_{\Phi}^{\top}J_{\Phi}\succeq 0,

where JΦ=[AK−1​B,AK−2​B,⋯,B]∈ℝd×K​d.J_{\Phi}=\bigl[\,A^{K-1}B,\;A^{K-2}B,\;\cdots,\;B\,\bigr]\in\mathbb{R}^{d\times Kd}. Let 𝒲K:=JΦ​JΦ⊤=∑k=0K−1Ak​B​B⊤​(A⊤)k\mathcal{W}_{K}:=J_{\Phi}J_{\Phi}^{\top}=\sum_{k=0}^{K-1}A^{k}BB^{\top}(A^{\top})^{k} be the finite-horizon controllability Gramian (Kailath, 1980; Sontag, 1998; Chen, 1999). Then the effective condition number κeff​(H):=σmax​(H)/σmin+​(H)\kappa_{\mathrm{eff}}(H):=\sigma_{\max}(H)/\sigma_{\min}^{+}(H) satisfies

Equation 11
κeff​(H)=κ⁡(𝒲K)\displaystyle\kappa_{\mathrm{eff}}(H)=\kappa(\mathcal{W}_{K}) ≤κ​(B)2​∑k=0K−1σmax​(A)2​k∑k=0K−1σmin​(A)2​k\displaystyle\leq\kappa(B)^{2}\,\frac{\sum_{k=0}^{K-1}\sigma_{\max}(A)^{2k}}{\sum_{k=0}^{K-1}\sigma_{\min}(A)^{2k}} (11)
≤κ​(B)2​κ​(A)2​(K−1),\displaystyle\leq\kappa(B)^{2}\,\kappa(A)^{2(K-1)},

where κ⁡(A):=σmax​(A)/σmin​(A)\kappa(A):=\sigma_{\max}(A)/\sigma_{\min}(A). Moreover, if the transition is ε\varepsilon-straight with ε=‖A−I‖2<1\varepsilon=\|A-I\|_{2}<1, then

Equation 12
κeff​(H)≤κ​(B)2​(1+ε1−ε)2​(K−1).\kappa_{\mathrm{eff}}(H)\;\leq\;\kappa(B)^{2}\left(\frac{1+\varepsilon}{1-\varepsilon}\right)^{2(K-1)}. (12)

For ε≤12\varepsilon\leq\tfrac{1}{2}, this gives κeff​(H)≤κ​(B)2​e6​ε​K\kappa_{\mathrm{eff}}(H)\;\leq\;\kappa(B)^{2}e^{6\varepsilon K}.

Proofs are in Section C.2.

Remark 4.5 (Low-dimensional actions).

When da<dd_{a}<d, BB is not invertible and 𝒲K\mathcal{W}_{K} (hence HH) may be singular outside the controllable subspace. Theorem 4.4 holds on 𝒮K=range⁡(𝒲K)\mathcal{S}_{K}=\mathrm{range}(\mathcal{W}_{K}) when κeff\kappa_{\mathrm{eff}} is computed using σmin+​(𝒲K)\sigma_{\min}^{+}(\mathcal{W}_{K}); see Section C.2.

5 Experiments

Four simulated environments cover a wall, two maze complexities and contact-rich pushing. The main comparison adapts DINO-WM-style encoders and uses the same recorded datasets. No real-robot deployment is demonstrated.

5.1 Architecture details

The experiments instantiate the idea with frozen DINOv2 plus a learned projector, or with a learned ResNet, and an action-conditioned transformer predictor.

Visual encoder.

The projector can compress channels or spatial resolution while adapting the features for dynamics. A from-scratch encoder offers more freedom but must learn useful perception from the task data. These are distinct representation setups, not equivalent amounts of pretraining.

Equation 13
ztv=𝒫ϕ​(et)∈ℝmv×dv,z_{t}^{v}=\mathcal{P}_{\phi}(e_{t})\in\mathbb{R}^{m_{v}\times d_{v}}, (13)

Predictor.

Actions and available proprioception are concatenated to visual features. A causal attention mask prevents later observations leaking into earlier predictions. The availability of proprioception changes what physical information the model receives.

Cosine similarity computation.

For spatial grids, straightness can be measured patch by patch, after flattening, after pooling or through a learned aggregation head. The main experiments use the last option. The prediction model retains spatial detail while the regularizer shapes a global trajectory signal.

Original figure
Figure 5: Latent Curvature and Open-Loop GD Success Rate for Different Encoders. Higher cosine similarity indicates lower curvature. Here, we compare models with spatial features and report the average patch-wise cosine similarity. Given the same type of encoder, reduced curvature generally leads to higher success rates.
Refer to caption
(a) DINOv2 CLS embedding
Refer to caption
(b) Straightened (agg head)
Refer to caption
(c) Straightened (spatial features)
Refer to caption
(d) Ground-Truth using A-star
Figure 6: Distance heatmaps of PointMaze (blue indicates small values, and red indicates large values). The yellow star represents the target, and we compute the Euclidean distance between its embedding and those of all other states in the maze. Figures 6(b) and 6(c) use ResNet with output features z∈ℝ14×14×8z\in\mathbb{R}^{14\times 14\times 8}, trained with straightening regularization. After straightening, the latent distance accurately reflects the minimum number of steps required to reach the target.

5.2 How good is the embedding space?

The diagnostics examine curvature, distance-to-goal maps and decoded images. They help explain planning performance but do not themselves prove a representation is sufficient for every task.

Refer to caption
(a) Wall: DINO
Refer to caption
(b) Wall: Ours
Refer to caption
(c) UMaze: DINO
Refer to caption
(d) UMaze: Ours
Refer to caption
(e) Medium: DINO
Refer to caption
(f) Medium: Ours
Figure 7: Comparison of Open-loop GD Planning. The star denotes the target. For each subfigure, the upper row shows the overlaid rendered images from the simulator by executing the actions, and the bottom shows imaginary rollouts (with a frameskip of 5) decoded using a trained decoder. GD planners can easily get stuck with pretrained DINOv2 features, while straightening significantly increases success. More examples of open-loop planning are in Section E.3.

Reduced curvature.

Predictive training alone often reduces curvature, and the explicit penalty strengthens that trend. Aggregation allows complex local patch motion without requiring each patch to follow an overly straight path. The explanation connecting this geometry to better search is supported by controlled comparisons, with task-dependent effects.

Faithful distance.

Distance heatmaps are compared with shortest paths on discretized mazes. Straightened features better reflect obstacle-aware progress in those examples. This is empirical alignment with the tested transition graph, not a global theorem identifying geodesic distance from arbitrary videos.

Sufficient information.

A separately trained detached decoder reconstructs observations from compressed latents. Accurate reconstruction shows retained visual information in these simple scenes, but is neither necessary nor sufficient for successful control. Table 1 reports actual planning outcomes, with regularization strengths selected using held-out validation seeds.

Table 1: Success Rate of 50 Test Episodes (%) using the GD planner. Values are mean ±\pm std over three eval data seeds. The best values are bold. The shaded rows are ours while the rest is DINO-WM (Zhou et al., 2025). For global features, the straightening strength λ\lambda is selected using MPC on two held-out validation seeds (λ=0.1\lambda{=}0.1 unless marked: †λ=0.01{}^{{\dagger}}\lambda{=}0.01, ‡λ=0.001{}^{{\ddagger}}\lambda{=}0.001). All spatial features use λ=0.1\lambda{=}0.1.
Encoder Config Wall PointMaze – UMaze PointMaze – Medium PushT
dim ℒc​u​r​v\mathcal{L}_{curv} Open-loop MPC Open-loop MPC Open-loop MPC Open-loop MPC
DINOv2 (CLS) 1×3841\times 384 ✗ 28.67 ±\pm 12.68 66.67 ±\pm 10.50 25.33 ±\pm 0.94 82.67 ±\pm 9.98 20.00 ±\pm 8.16 67.50 ±\pm 3.54 19.33 ±\pm 8.22 28.00 ±\pm 1.63
DINOv2 (patch) + proj 1×3841\times 384 ✗ 28.67 ±\pm 0.94 76.00 ±\pm 4.90 34.67 ±\pm 1.89 79.33 ±\pm 2.49 18.00 ±\pm 1.63 46.00 ±\pm 3.27 2.00 ±\pm 1.63 11.33 ±\pm 3.40
DINOv2 (patch) + proj 1×3841\times 384 ✓ 32.00‡ ±\pm 7.12 77.33‡ ±\pm 0.94 38.67 ±\pm 3.40 96.00 ±\pm 0.00 22.67† ±\pm 5.73 78.00† ±\pm 2.83 2.00 ±\pm 1.63 8.67 ±\pm 1.89
ResNet (from scratch) 1×3841\times 384 ✗ 58.67 ±\pm 4.99 68.00 ±\pm 4.32 58.00 ±\pm 4.90 88.67 ±\pm 0.94 73.33 ±\pm 6.60 89.33 ±\pm 2.49 40.00 ±\pm 4.32 70.00 ±\pm 1.63
ResNet (from scratch) 1×3841\times 384 ✓ 86.67 ±\pm 5.25 97.33 ±\pm 2.49 63.33 ±\pm 1.89 97.33 ±\pm 1.89 79.33‡ ±\pm 5.25 93.33‡ ±\pm 2.49 58.67† ±\pm 5.73 83.33† ±\pm 2.49
DINOv2 (patch) 14×14×38414\times 14\times 384 ✗ 52.67 ±\pm 5.73 76.67 ±\pm 6.18 35.33 ±\pm 4.11 80.67 ±\pm 6.18 40.83 ±\pm 10.07 76.67 ±\pm 5.14 56.00 ±\pm 4.32 66.00 ±\pm 4.90
DINOv2 (patch) + proj 14×14×814\times 14\times 8 ✗ 80.00 ±\pm 7.12 90.67 ±\pm 3.77 44.00 ±\pm 7.12 81.33 ±\pm 6.80 72.00 ±\pm 4.32 96.67 ±\pm 0.94 70.00 ±\pm 1.63 78.67 ±\pm 0.94
DINOv2 (patch) + proj 14×14×814\times 14\times 8 ✓ 90.67 ±\pm 0.94 100.00 ±\pm 0.00 94.00 ±\pm 1.63 100.00 ±\pm 0.00 82.67 ±\pm 3.77 98.67 ±\pm 0.94 77.33 ±\pm 6.18 85.33 ±\pm 4.99
ResNet (from scratch) 14×14×814\times 14\times 8 ✗ 1.33 ±\pm 1.89 6.67 ±\pm 1.89 14.67 ±\pm 4.99 66.00 ±\pm 9.09 18.67 ±\pm 4.11 57.33 ±\pm 4.71 71.33 ±\pm 7.36 70.67 ±\pm 10.50
ResNet (from scratch) 14×14×814\times 14\times 8 ✓ 84.67 ±\pm 2.49 100.00 ±\pm 0.00 64.67 ±\pm 8.38 98.67 ±\pm 1.89 80.67 ±\pm 0.94 99.33 ±\pm 0.94 70.67 ±\pm 0.94 91.33 ±\pm 2.49
Original figure
(a) Wall
Original figure
(b) PointMaze-UMaze
Original figure
(c) PointMaze-Medium
Original figure
(d) PushT
Figure 8: Success Rate over MPC Steps. The dashed black lines represent DINO-WM with frozen DINOv2 patch features. The solid lines represent frozen DINOv2 patch features with a trainable channel projector (with resulting features z∈ℝ14×14×8z\in\mathbb{R}^{14\times 14\times 8}) with different strengths of straightening. Our model reaches 100% success rates very quickly as shown in Figures 8(a) and 8(b).

5.3 Planning

Planning tests compare open-loop execution with feedback replanning and examine both success and runtime.

Setup.

The usual goal is sampled from a later test state reachable within twenty-five physical actions. Five-action chunks require only five predictor calls. PushT supplies target proprioception as well as an image. Maze MPC uses intermediate goal losses, while PushT uses a terminal loss inside the short horizon; the resulting curves therefore reflect different cost choices.

Results.

Learning a projector or encoder already improves several baselines, and explicit straightening often adds another gain. Reported perfect success on some easy MPC tasks is scoped to the selected goals and seeds. The original table retains cases where encoder choice or feature geometry remains poor.

(a) Success Refer to caption

(b) Failure Refer to caption

Figure 9: Examples of Long-Horizon Open-Loop GD Planning on PushT. For each example, the top row shows simulator-rendered images and the bottom row shows decoded images, with the last column being the target. The failure example shows a case where the long-horizon imagined rollout does not match the real dynamics.

Comparison of CEM and GD.

CEM often achieves higher absolute success but evaluates many more candidate rollouts. Straightening improves both planners and reduces the gap, producing a favorable gradient-planning latency tradeoff in the measured implementation.

Effect of feature dimensions.

Keeping spatial tokens matters more than preserving every channel in these tasks. Moderate channel compression works well; two channels can be too few and very wide features can make planning harder. These are task-specific dimension ablations, not a universal optimal latent size.

Long horizon.

With goals fifty steps away, success drops and imagined trajectories can drift from the simulator. Adding an aggregated global goal cost to the spatial cost helps several long-horizon settings. Easier mazes are omitted from this test because a long random trajectory does not necessarily create a distant shortest-path goal.

Teleported-PointMaze.

A modified maze teleports the agent from the right boundary to the left. A selected successful plan shows that learned temporal geometry can exploit a transition that is visually discontinuous. This is a diagnostic example, not a broad success-rate claim for all teleportation tasks.

Limitations and future directions.

The planner uses a symmetric Euclidean cost, which may be unsuitable for irreversible or asymmetric reachability. The local regularizer does not require reversibility, but directional costs may still be needed. Prediction features and the best planning-cost features can also be different spaces.

Table 2: Longer-Horizon Success Rate (%) w/ Spatial Features.
Model ℒc​u​r​v\mathcal{L}_{curv} PushT PointMaze – Medium
Open-loop MPC Open-loop MPC
DINO-WM – 3.33 ±\pm 2.36 27.33 ±\pm 6.66 35.00 ±\pm 2.35 65.33 ±\pm 3.13
+ Proj ✗ 6.67 ±\pm 3.77 26.67 ±\pm 9.98 60.00 ±\pm 3.27 72.00 ±\pm 0.00
+ Proj ✓ 13.33 ±\pm 3.77 24.00 ±\pm 6.53 68.00 ±\pm 8.64 88.00 ±\pm 3.27
ResNet ✗ 13.33 ±\pm 3.77 29.33 ±\pm 9.43 14.67 ±\pm 6.80 48.00 ±\pm 9.80
ResNet ✓ 10.67 ±\pm 4.99 33.33 ±\pm 4.99 76.00 ±\pm 6.53 98.67 ±\pm 1.89
Combined planning cost: ℒplan=ℒspatial+0.1​ℒagg\mathcal{L}_{\mathrm{plan}}=\mathcal{L}_{\mathrm{spatial}}+0.1\mathcal{L}_{\mathrm{agg}}
+ Proj ✓ 20.00 ±\pm 0.00 33.33 ±\pm 4.16 66.67 ±\pm 7.57 92.00 ±\pm 5.29
ResNet ✓ 13.33 ±\pm 4.62 36.00 ±\pm 5.29 68.67 ±\pm 4.16 98.67 ±\pm 1.15

6 Conclusion

The contribution is a simple training-time geometric objective that improves several latent planners. Its strength is making action search easier without a new pixel generator or reward model. The remaining limits are learned-model error, task-dependent geometry and the gap between linear analysis and nonlinear deployment.

Impact Statement

The authors identify potential robotics and autonomous-system uses and call for attention to deployment safety. The experiments themselves remain simulated.

Acknowledgments

The source acknowledges research discussions, grants and computing support.

Appendix

Appendix A Data and Environments

A.1 Wall

Wall uses image-based two-room navigation with direct displacement actions. Its simple transitions differ from inertial physics in PointMaze.

A.2 PointMaze (UMaze and Medium-Maze)

The two PointMaze tasks use force inputs and include velocity and inertia. Separate datasets cover U-shaped and more complex layouts. These dynamics make short visual history important.

A.3 PushT

PushT requires both the pusher and T-block to match a feasible goal. The fixed green marker is a visual reference, not the target. Its reused dataset contains expert-derived noisy trajectories, so the evaluation is not uniformly based on random behavior.

Appendix B Experiments

B.1 Model Predictive Control (MPC)

Gradient descent differentiates goal error through an imagined rollout, updates the candidate actions, executes a prefix and replans from observed feedback. The main text adds task-specific intermediate losses beyond the simplified terminal-cost outline here.

Equations
z0=ℰs​(o0),zg=ℰs​(og).z_{0}=\mathcal{E}^{s}(o_{0}),\quad z_{g}=\mathcal{E}^{s}(o_{g}).
L=‖z^T−zg‖22L=\|\hat{z}_{T}-z_{g}\|_{2}^{2}
at←at−η∂L∂at,for t=0,…,T−1,a_{t}\leftarrow a_{t}-\eta\frac{\partial L}{\partial a_{t}},\quad\text{for }t=0,\dots,T-1,

B.2 Hyperparameters

The source tables retain training and planning hyperparameters. Reproduction requires matching encoder setup, feature shape, loss weight and action-chunk conventions.

Table 3: Training Hyperparameters.
Name Value
Spatial Projector/ResNet lr 1e-522 2 We observe severe performance degradation when training without straightening and decreasing the learning rate helps. We thus use l​r=1​e−6lr=1e-6 for no straightening.
Global Projector/ResNet lr 1e-6
Predictor lr 5e-4
Action/Prop encoder lr 5e-4
Batch size 32
History frames 3
Frameskip 5
Table 4: Planning Hyperparameters.
Name Value
Subplanner horizon 25
# Executed actions 2533 3 This is for open-loop. If using MPC, we execute the first 5 actions (or the first chunk of actions if using a frameskip of 5).
Optimizer Adam
Action Initialization Zero
Learning rate 0.1
#opt steps 100

B.3 Planning: GD vs. CEM

The timing comparison uses a single L40S and fifty PushT episodes. CEM’s candidate population provides more search effort, while gradient descent uses forward and backward passes on a proposed sequence. The observed speed ratio is an implementation-and-budget result, not a fixed complexity ratio for every device.

Original figure
Figure 10: Success rate versus wall-clock planning time for open-loop GD and CEM planning after straightening.
Table 5: Open-Loop Planning Success Rate of 50 Test Trajectories (%). We compare GD and CEM planners. For CEM, we use 200 samples and 10 iterations. Values are mean ±\pm std over three data seeds. The best value is bold.
Method Config Wall PointMaze – UMaze PointMaze – Medium PushT
dim ℒc​u​r​v\mathcal{L}_{curv} GD CEM GD CEM GD CEM GD CEM
DINOv2 (patch) 14×14×38414\times 14\times 384 ✗ 52.67 ±\pm 5.73 87.33 ±\pm 4.99 35.33 ±\pm 4.11 88.00 ±\pm 1.63 40.83 ±\pm 10.07 88.00 ±\pm 1.63 56.00 ±\pm 4.32 71.33 ±\pm 7.72
DINOv2 (patch) + proj 14×14×814\times 14\times 8 ✗ 80.00 ±\pm 7.12 92.00 ±\pm 0.00 44.00 ±\pm 7.12 75.33 ±\pm 4.99 72.00 ±\pm 4.32 92.67 ±\pm 4.71 70.00 ±\pm 1.63 71.33 ±\pm 6.18
DINOv2 (patch) + proj 14×14×814\times 14\times 8 ✓ 90.67 ±\pm 0.94 100.00 ±\pm 0.00 94.00 ±\pm 1.63 94.00 ±\pm 1.63 82.67 ±\pm 3.77 86.67 ±\pm 1.89 77.33 ±\pm 6.18 80.00 ±\pm 4.32
ResNet 14×14×814\times 14\times 8 ✗ 1.33 ±\pm 1.89 1.33 ±\pm 0.94 14.67 ±\pm 4.99 20.67 ±\pm 0.94 18.67 ±\pm 4.11 24.00 ±\pm 4.32 71.33 ±\pm 7.36 56.00 ±\pm 0.00
ResNet 14×14×814\times 14\times 8 ✓ 84.67 ±\pm 2.49 90.00 ±\pm 5.89 64.67 ±\pm 8.38 83.33 ±\pm 2.49 80.67 ±\pm 0.94 89.33 ±\pm 6.18 70.67 ±\pm 0.94 72.67 ±\pm 6.60

B.4 Effect of Feature Dimensions

Dimension ablations retain the full patch grid while reducing channels. The strongest results use moderate compression. They isolate channel count more directly than comparisons that also collapse the spatial grid.

Original figure
(a) Wall
Original figure
(b) PointMaze-UMaze
Original figure
(c) PointMaze-Medium
Original figure
(d) PushT
Figure 11: Comparison of Different Dimensions. The line plots show the success rate changes with increasing channels. Too small dimensions (e.g. dv=2d_{v}=2) are unable to encode sufficient planning-relevant information, while unnecessarily high dimensions (e.g. dv=128d_{v}=128) hinder the planning performance.

B.5 Comparison to Smoothness and Temporal Contrastive Objectives

Smoothness shrinks adjacent latent distances; temporal contrastiveness separates distant-in-time observations. Neither matches straightening in this PushT comparison, and large weights can hurt. This does not show that either objective is useless in other settings.

Original figure
Figure 12: Comparison with Other Regularizations.
Equations
ℒsmooth=𝔼t​[‖zt+1−zt‖22].\mathcal{L}_{\mathrm{smooth}}=\mathbb{E}_{t}\left[\|z_{t+1}-z_{t}\|_{2}^{2}\right].
ℒtc=InfoNCE⁡(zpos,zneg).\mathcal{L}_{\mathrm{tc}}=\mathrm{InfoNCE}(z_{\mathrm{pos}},z_{\mathrm{neg}}).

B.6 Cosine Similarity Variants for Spatial Features

The aggregation head supplies a global trajectory direction while spatial features retain local detail for prediction. Its best reported regularization weight differs from those used for the other cosine variants, so the comparison includes that tuning choice.

Refer to caption
Figure 13: Aggregation Head for Straightening during Training. The prediction loss is applied to the spatial features, while the curvature loss is applied to the aggregated features.
Equations
𝒞t=1mv​∑i=1mvcos⁡(vt,i,vt+1,i).\mathcal{C}_{t}=\frac{1}{m_{v}}\sum_{i=1}^{m_{v}}\cos\!\left(v_{t,i},\,v_{t+1,i}\right).
v¯t=1mv​∑i=1mvvt,i,𝒞t=cos⁡(v¯t,v¯t+1).\bar{v}_{t}=\frac{1}{m_{v}}\sum_{i=1}^{m_{v}}v_{t,i},\qquad\mathcal{C}_{t}=\cos(\bar{v}_{t},\bar{v}_{t+1}).
𝒞t=cos⁡(vec⁡(vt),vec⁡(vt+1)),\mathcal{C}_{t}=\cos\!\left(\mathrm{vec}(v_{t}),\,\mathrm{vec}(v_{t+1})\right),
𝒞t=cos⁡(hϕ​(vt),hϕ​(vt+1)),\mathcal{C}_{t}=\cos\!\left(h_{\phi}(v_{t}),\,h_{\phi}(v_{t+1})\right),
Original figure
(a) Wall
Original figure
(b) PointMaze-UMaze
Original figure
(c) PointMaze-Medium
Original figure
(d) PushT
Figure 14: Comparison of Different Straightening Strategies. The bar charts show the planning success rates. While all cosine similarity variants lead to better performance than no straightening, adding a learnable aggregation head gives the best performance.

Appendix C Theoretical Analysis

C.1 Setup and notation

The analysis defines linear dynamics and uses an effective condition number based only on nonzero singular values. This is necessary because many action sequences can produce the same terminal state, leaving a nullspace in the action Hessian.

Equation 14
ℒ⁡(𝐚)=‖zK−zg‖22,zK=Φ⁡(𝐚),\mathcal{L}(\mathbf{a})=\|z_{K}-z_{g}\|_{2}^{2},\qquad z_{K}=\Phi(\mathbf{a}), (14)
Assumption C.1 (Linear latent dynamics).

We assume linear latent dynamics

Equation 15
zt+1=A​zt+B​at,A∈ℝd×d,B∈ℝd×da.z_{t+1}=Az_{t}+Ba_{t},\qquad A\in\mathbb{R}^{d\times d},\;B\in\mathbb{R}^{d\times d_{a}}. (15)
Definition C.2 (Effective condition number).

For a PSD matrix H⪰0H\succeq 0 with a nontrivial nullspace, define

Equation
κeff​(H):=σmax​(H)σmin+​(H),\kappa_{\mathrm{eff}}(H):=\frac{\sigma_{\max}(H)}{\sigma_{\min}^{+}(H)},

where σmin+​(H)\sigma_{\min}^{+}(H) is the smallest nonzero singular value.

Definition C.3 (ε\varepsilon-straight transition).

In the linear model (15), define

Equation
ε:=‖A−I‖2.\varepsilon:=\|A-I\|_{2}.

C.2 Conditioning of the planning Hessian

Unrolling the linear system gives a terminal-state Jacobian built from successive transition powers and action matrices. Its Gramian shares nonzero eigenvalues with the action Hessian. This converts the optimization question into a standard reachability-conditioning question.

Equations 16–18
zK=AK​z0+∑t=0K−1AK−1−t​B​at.z_{K}=A^{K}z_{0}+\sum_{t=0}^{K-1}A^{K-1-t}Ba_{t}. (16)
JΦ:=∂zK∂𝐚=[AK−1B,AK−2B,⋯,B]∈ℝd×(K​da).J_{\Phi}:=\frac{\partial z_{K}}{\partial\mathbf{a}}=\bigl[\,A^{K-1}B,\quad A^{K-2}B,\quad\cdots,\quad B\,\bigr]\in\mathbb{R}^{d\times(Kd_{a})}. (17)
𝒲K:=JΦ​JΦ⊤=∑k=0K−1Ak​B​B⊤​(A⊤)k∈ℝd×d,\mathcal{W}_{K}:=J_{\Phi}J_{\Phi}^{\top}=\sum_{k=0}^{K-1}A^{k}BB^{\top}(A^{\top})^{k}\in\mathbb{R}^{d\times d}, (18)
Lemma C.4 (Hessian form and Gramian equivalence).

Under (14)–(15), the planning Hessian satisfies

Equation 19
H:=∇𝐚2​ℒ​(𝐚)=2​JΦ⊤​JΦ⪰0.H:=\nabla^{2}_{\mathbf{a}}\mathcal{L}(\mathbf{a})=2J_{\Phi}^{\top}J_{\Phi}\succeq 0. (19)

Moreover, the nonzero singular values of JΦ⊤​JΦJ_{\Phi}^{\top}J_{\Phi} equal those of JΦ​JΦ⊤J_{\Phi}J_{\Phi}^{\top}, hence

Equation 20
κeff​(H)=κ⁡(𝒲K).\kappa_{\mathrm{eff}}(H)=\kappa(\mathcal{W}_{K}). (20)
Theorem C.5 (Conditioning bound).

Assume (15). Consider first the square-action case da=dd_{a}=d with BB invertible. Then

Equation 21
κeff​(H)=κ⁡(𝒲K)≤κ​(B)2​∑k=0K−1σmax​(A)2​k∑k=0K−1σmin​(A)2​k≤κ​(B)2​κ​(A)2​(K−1),\kappa_{\mathrm{eff}}(H)=\kappa(\mathcal{W}_{K})\;\leq\;\kappa(B)^{2}\,\frac{\sum_{k=0}^{K-1}\sigma_{\max}(A)^{2k}}{\sum_{k=0}^{K-1}\sigma_{\min}(A)^{2k}}\;\leq\;\kappa(B)^{2}\,\kappa(A)^{2(K-1)}, (21)

where κ⁡(A):=σmax​(A)/σmin​(A)\kappa(A):=\sigma_{\max}(A)/\sigma_{\min}(A). If additionally ε=‖A−I‖2<1\varepsilon=\|A-I\|_{2}<1, then

Equation 22
κeff​(H)≤κ​(B)2​(1+ε1−ε)2​(K−1)≤κ​(B)2​e6​ε​K(ε≤12).\kappa_{\mathrm{eff}}(H)\;\leq\;\kappa(B)^{2}\left(\frac{1+\varepsilon}{1-\varepsilon}\right)^{2(K-1)}\;\leq\;\kappa(B)^{2}e^{6\varepsilon K}\quad(\varepsilon\leq\tfrac{1}{2}). (22)

Upper bound.

The upper bound combines the largest action gain with the largest transition gain at each horizon step. It limits how steep the strongest action direction can become.

Equations
x⊤​𝒲K​x=∑k=0K−1‖B⊤​(A⊤)k​x‖22≤∑k=0K−1‖B‖22​‖Ak‖22​‖x‖22≤σmax​(B)2​∑k=0K−1σmax​(A)2​k.x^{\top}\mathcal{W}_{K}x=\sum_{k=0}^{K-1}\|B^{\top}(A^{\top})^{k}x\|_{2}^{2}\leq\sum_{k=0}^{K-1}\|B\|_{2}^{2}\|A^{k}\|_{2}^{2}\|x\|_{2}^{2}\leq\sigma_{\max}(B)^{2}\sum_{k=0}^{K-1}\sigma_{\max}(A)^{2k}.
λmax​(𝒲K)≤σmax​(B)2​∑k=0K−1σmax​(A)2​k.\lambda_{\max}(\mathcal{W}_{K})\leq\sigma_{\max}(B)^{2}\sum_{k=0}^{K-1}\sigma_{\max}(A)^{2k}.

Lower bound.

The lower bound depends on the action matrix being invertible, ensuring every state direction receives some action influence. This assumption cannot be silently applied to underactuated systems.

Equations
‖B⊤​(A⊤)k​x‖2≥σmin​(B)​‖(A⊤)k​x‖2≥σmin​(B)​σmin​(Ak)​‖x‖2≥σmin​(B)​σmin​(A)k,\|B^{\top}(A^{\top})^{k}x\|_{2}\geq\sigma_{\min}(B)\,\|(A^{\top})^{k}x\|_{2}\geq\sigma_{\min}(B)\,\sigma_{\min}(A^{k})\,\|x\|_{2}\geq\sigma_{\min}(B)\,\sigma_{\min}(A)^{k},
x⊤​𝒲K​x≥σmin​(B)2​∑k=0K−1σmin​(A)2​k.x^{\top}\mathcal{W}_{K}x\geq\sigma_{\min}(B)^{2}\sum_{k=0}^{K-1}\sigma_{\min}(A)^{2k}.
λmin​(𝒲K)≥σmin​(B)2​∑k=0K−1σmin​(A)2​k.\lambda_{\min}(\mathcal{W}_{K})\geq\sigma_{\min}(B)^{2}\sum_{k=0}^{K-1}\sigma_{\min}(A)^{2k}.

Combine.

Dividing the bounds controls the ratio of steepest to weakest nonzero directions. The expression also depends on action conditioning; straight transitions alone cannot repair an arbitrarily ill-conditioned action map.

Equation
∑k=0K−1σmax​(A)2​k∑k=0K−1σmin​(A)2​k≤max0≤k≤K−1⁡σmax​(A)2​kσmin​(A)2​k=κ​(A)2​(K−1).\frac{\sum_{k=0}^{K-1}\sigma_{\max}(A)^{2k}}{\sum_{k=0}^{K-1}\sigma_{\min}(A)^{2k}}\leq\max_{0\leq k\leq K-1}\frac{\sigma_{\max}(A)^{2k}}{\sigma_{\min}(A)^{2k}}=\kappa(A)^{2(K-1)}.

ε -specialization.

Near-identity transitions bound their largest and smallest singular values, yielding the stated horizon-dependent estimate. For low-dimensional actions, extra controllability assumptions are needed to bound the smallest positive Gramian eigenvalue. Merely restricting to a reachable subspace does not supply that missing quantitative bound.

Remark C.6 (Low-dimensional actions da<dd_{a}<d).

If da<dd_{a}<d, then BB is not invertible and 𝒲K\mathcal{W}_{K} may be singular. All statements hold on the controllable subspace 𝒮K=range⁡(𝒲K)\mathcal{S}_{K}=\mathrm{range}(\mathcal{W}_{K}) by replacing λmin​(𝒲K)\lambda_{\min}(\mathcal{W}_{K}) with λmin+​(𝒲K)\lambda_{\min}^{+}(\mathcal{W}_{K}) and interpreting κ⁡(𝒲K)\kappa(\mathcal{W}_{K}) as an effective condition number. In this case, additional controllability assumptions are needed to lower bound σmin+​(𝒲K)\sigma_{\min}^{+}(\mathcal{W}_{K}).

C.3 Cosine similarity as a proxy

The cosine connection assumes constant nonzero latent speed and bounded changes in actions. It bounds deviation from identity along visited movement directions. Turning that into a global spectral bound needs adequate directional coverage. Thus high cosine on training trajectories is evidence for a useful proxy, not an unconditional guarantee over all latent states.

Assumption C.7 (Constant velocity and smooth actions).

Define latent velocities vt:=zt+1−ztv_{t}:=z_{t+1}-z_{t}. Assume there exists a constant c>0c>0 such that

Equation
‖vt‖2=cfor all ​t=0,…,K−1.\|v_{t}\|_{2}=c\qquad\text{for all }t=0,\dots,K-1.

Assume action smoothness Δa:=maxt⁡‖at+1−at‖2<∞\Delta_{a}:=\max_{t}\|a_{t+1}-a_{t}\|_{2}<\infty.

Definition C.8 (Cosine similarity).

For t=0,…,K−2t=0,\dots,K-2, define

Equation
𝒞t:=cos⁡(vt,vt+1)=vt⊤​vt+1‖vt‖2​‖vt+1‖2,𝒞¯:=1K−1​∑t=0K−2𝒞t.\mathcal{C}_{t}:=\cos(v_{t},v_{t+1})=\frac{v_{t}^{\top}v_{t+1}}{\|v_{t}\|_{2}\|v_{t+1}\|_{2}},\qquad\bar{\mathcal{C}}:=\frac{1}{K-1}\sum_{t=0}^{K-2}\mathcal{C}_{t}.
Proposition C.9 (Cosine proxy ⇒\Rightarrow small (A−I)(A-I) along visited directions).

Under linear dynamics (15), let v^t:=vt/‖vt‖2\hat{v}_{t}:=v_{t}/\|v_{t}\|_{2}. Under Assumption C.7, for each t=0,…,K−2t=0,\dots,K-2,

Equation 23
‖(A−I)​v^t‖2≤2​(1−𝒞t)+σmax​(B)​Δac.\|(A-I)\hat{v}_{t}\|_{2}\;\leq\;\sqrt{2(1-\mathcal{C}_{t})}\;+\;\frac{\sigma_{\max}(B)\Delta_{a}}{c}. (23)

If 𝒞¯≥1−η\bar{\mathcal{C}}\geq 1-\eta, then

Equation 24
1K−1​∑t=0K−2‖(A−I)​v^t‖2≤2​η+σmax​(B)​Δac.\frac{1}{K-1}\sum_{t=0}^{K-2}\|(A-I)\hat{v}_{t}\|_{2}\;\leq\;\sqrt{2\eta}\;+\;\frac{\sigma_{\max}(B)\Delta_{a}}{c}. (24)
Equations
vt+1−vt=(zt+2−zt+1)−(zt+1−zt)=(A−I)​(zt+1−zt)+B⁡(at+1−at)=(A−I)​vt+B⁡(at+1−at).v_{t+1}-v_{t}=(z_{t+2}-z_{t+1})-(z_{t+1}-z_{t})=(A-I)(z_{t+1}-z_{t})+B(a_{t+1}-a_{t})=(A-I)v_{t}+B(a_{t+1}-a_{t}).
‖(A−I)​v^t‖2=‖(A−I)​vt‖2‖vt‖2≤‖vt+1−vt‖2‖vt‖2+‖B⁡(at+1−at)‖2‖vt‖2≤‖vt+1−vt‖2c+σmax​(B)​Δac.\|(A-I)\hat{v}_{t}\|_{2}=\frac{\|(A-I)v_{t}\|_{2}}{\|v_{t}\|_{2}}\leq\frac{\|v_{t+1}-v_{t}\|_{2}}{\|v_{t}\|_{2}}+\frac{\|B(a_{t+1}-a_{t})\|_{2}}{\|v_{t}\|_{2}}\leq\frac{\|v_{t+1}-v_{t}\|_{2}}{c}+\frac{\sigma_{\max}(B)\Delta_{a}}{c}.
‖vt+1−vt‖22=‖vt+1‖22+‖vt‖22−2​vt+1⊤​vt=2​c2​(1−𝒞t),\|v_{t+1}-v_{t}\|_{2}^{2}=\|v_{t+1}\|_{2}^{2}+\|v_{t}\|_{2}^{2}-2v_{t+1}^{\top}v_{t}=2c^{2}(1-\mathcal{C}_{t}),
1K−1​∑t=0K−21−𝒞t≤1−𝒞¯≤η,\frac{1}{K-1}\sum_{t=0}^{K-2}\sqrt{1-\mathcal{C}_{t}}\leq\sqrt{1-\bar{\mathcal{C}}}\leq\sqrt{\eta},
Remark C.10 (Directional vs. spectral control).

Proposition C.9 bounds (A−I)(A-I) only along visited directions {v^t}\{\hat{v}_{t}\}. Upgrading this to a uniform spectral bound ε=‖A−I‖2\varepsilon=\|A-I\|_{2} requires an additional coverage condition so that visited directions span the latent space. This is not an unreasonable assumption since training trajectories are typically collected to be diverse. Under such regimes, maximizing cosine similarity provides a meaningful proxy for making AA close to II in spectral norm.

Appendix D Related Work (Cont.)

Straightening is distinct from forcing all dynamics to be linear: even a linear rotation traces curved paths. It also differs from locally linear control and Koopman models. The method allows a nonlinear predictor while directly shaping observed trajectory geometry.

Appendix E Visualizations

E.1 Distance Heatmaps

The heatmaps compare learned goal distances with grid-search distances. Four-neighbor and eight-neighbor graphs imply different legal routes, so their reference maps differ. The plots illustrate local versus global geometry; their particular discretization is not the continuous system itself.

Refer to caption
(a) Ground-Truth using A-star (4 neighbors)
Refer to caption
(b) Ground-Truth using A-star (8 neighbors)
Refer to caption
(c) DINOv2 CLS embedding
Refer to caption
(d) DINOv2 patch embedding
Refer to caption
(e) DINOv2 + spatial proj [straightening=False]
Refer to caption
(f) ResNet - spatial [straightening=False]
Refer to caption
(g) ResNet - global [straightening=False]
Refer to caption
(h) ResNet - global [straightening=True]
Refer to caption
(i) DINOv2 + spatial proj (agg head) [straightening=True]
Refer to caption
(j) DINOv2 + spatial proj (spatial) [straightening=True]
Refer to caption
(k) ResNet - spatial (agg head) [straightening=True]
Refer to caption
(l) ResNet - spatial (spatial) [straightening=True]
Figure 15: Distance heatmaps of PointMaze-UMaze.
Refer to caption
(a) Ground-Truth using A-star (4 neighbors)
Refer to caption
(b) Ground-Truth using A-star (8 neighbors)
Refer to caption
(c) DINOv2 CLS embedding
Refer to caption
(d) DINOv2 patch embedding
Refer to caption
(e) DINOv2 + spatial proj [straightening=False]
Refer to caption
(f) ResNet - spatial [straightening=False]
Refer to caption
(g) ResNet - global [straightening=False]
Refer to caption
(h) ResNet - global [straightening=True]
Refer to caption
(i) DINOv2 + spatial proj (agg head) [straightening=True]
Refer to caption
(j) DINOv2 + spatial proj (spatial) [straightening=True]
Refer to caption
(k) ResNet - spatial (agg head) [straightening=True]
Refer to caption
(l) ResNet - spatial (spatial) [straightening=True]
Figure 16: Distance heatmaps of PointMaze-Medium.

E.2 Visualization of Latent Trajectories

PCA plots make the change in trajectory geometry visible, while latent goal-error curves show whether distance tracks progress. A two-dimensional projection is illustrative and can hide structure in the remaining feature dimensions.

Refer to caption
Refer to caption
Figure 17: PCA of Trajectories of Wall.
Refer to caption
Refer to caption
Figure 18: PCA of Trajectories of PointMaze-UMaze.
Refer to caption
Refer to caption
Refer to caption
Figure 19: PCA of Trajectories of PointMaze-Medium.
Refer to caption
Refer to caption
Refer to caption
Figure 20: PCA of Trajectories of PushT. The overlaid figures only include five samples for readability.

E.3 Planning Trajectories

The retained panels pair simulator execution with decoded imagined trajectories. Agreement and drift should be assessed separately; the latter reveals predictive-model failures that a low optimized latent cost can conceal.

Refer to caption
Refer to caption
Refer to caption
Figure 21: Open-Loop Planning Trajectories of Wall. The first row is from the simulator and the second from the decoder. The last column is the goal image.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 22: Open-Loop Planning Trajectories of PointMaze-UMaze. The first row is from the simulator and the second from the decoder. The last column is the goal image.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 23: Open-Loop Planning Trajectories of PointMaze-Medium. The first row is from the simulator and the second from the decoder. The last column is the goal image.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 24: Open-Loop Planning Trajectories of PushT. The first row is from the simulator and the second from the decoder.

Appendix F Teleported-PointMaze

Teleportation changes position discontinuously and forces horizontal velocity to point right after the jump. This creates asymmetric temporal reachability. The example supports learning transition-sensitive geometry while reinforcing the limitation of using symmetric distance as a universal planning cost.

Refer to caption
Figure 25: Teleported-PointMaze. Note that the teleportation happens inside the red box.
Refer to caption
(a) DINOv2 patch embedding
Refer to caption
(b) Trained projector with straightening
Refer to caption
(c) Trained projector without straightening
Refer to caption
(d) Ground-Truth using A-star
Figure 26: Distance heatmaps of Teleported-PointMaze (blue indicates small values, red indicates large values). The state marked by the yellow star is used as the target, and we compute the MSE between its embedding and those of all other states in the maze. Since MSE is symmetric, this visualization does not fully capture directional reachability in the asymmetric teleportation dynamics. Nevertheless, with straightening, the resulting heatmaps are significantly closer to the transition-aware distances obtained using A-star.
Refer to caption
(a) With straightening, the agent reaches the target within given step limit.
Refer to caption
(b) Without straightening, the agent gets stuck at the corner.
Figure 27: Comparison of Planning Trajectories in Teleported-PointMaze. The frames were masked by black after reaching the target.
Concept demonstration

Open image ↗