Literature
Display
← All papers

LpWM: A Case for Sparse Representations in World Models

Yilun Kuang · Yash Dagade · Quentin Le Lidec · Lucas Maes · Randall Balestriero · Yann LeCun

Preprint posted · arXiv (preprint) ↗

World models7 min text · CondensedOriginal paper ↗
Contents

At a glance

Think of describing a scene with many available features but activating only a few at once. LpWM asks whether these sparse codes make a world model’s job easier. An encoder produces nonnegative features, a predictor forecasts them after an action, and a distribution-matching loss encourages exact zeros without collapsing every observation to the same code. A quantization theorem motivates the idea, but practical learned codes are an approximation to that ideal. In the tested simulations, sparse codes help most when the predictor has intermediate capacity. Which features are active can reveal dynamical regimes, although meaningful contact tracking additionally needs a temporal prior.

1 Introduction

Preventing constant embeddings is necessary for JEPA training, but it does not determine the best representation for predicting motion. Dense Gaussian-like codes are one choice. This paper tests an alternative: use sparse codes so that a simpler predictor may be sufficient, and inspect whether the active-feature pattern reflects how the environment changes.

Original figure
(a) LpWM Schematic Diagram
Original figure
(b) PushT Planning Performance.
Figure 1: LpWorldModel: a JEPA model with sparse representations. (a) An encoder ff maps observation 𝐨t\mathbf{o}_{t} to non-negative, sparse latents 𝐳t\mathbf{z}_{t} via a terminal (Rep)ReLU function. The predictor gg takes inputs 𝐳t\mathbf{z}_{t} and actions 𝐚t\mathbf{a}_{t} to predict 𝐳^t+1\hat{\mathbf{z}}_{t+1}, compared against the encoded target 𝐳t+1=f⁡(𝐨t+1)\mathbf{z}_{t+1}=f(\mathbf{o}_{t+1}). RDMReg (24) matches per-timestep latent marginals to a Rectified Generalized Gaussian, preventing collapse and inducing sparsity. (b). Arrows indicate the success-rate difference between LpWM and LeWM: ↑\uparrow denotes higher performance for LpWM, while ↓\downarrow denotes lower performance. LpWM outperforms LeWM with more linear predictor variants (MLP∘\circLTV(k)(k), MLP∘\circLTI(k)(k), and LTI(k)(k). See full specifications in Table 1) by up to 57%57\% in success rate, while performing on par with LeWM using the high-capacity DiT predictors. This suggests that sparse representations induce a latent space with simpler dynamics to learn compared to their dense counterparts.

Contributions.

The contributions are a sparse action-conditioned world model, an idealized one-hot linearization result, predictor-capacity comparisons, and probes of discrete versus continuous information in the learned code. The theory motivates the experiments; it does not prove that training the practical network reaches the constructed encoding.

2 Method

2.1 LpWM: JEPA with Rectified Distribution Matching Regularization (RDMReg)

Training balances future-feature prediction with a constraint on the feature distribution. Rectified Distribution Matching Regularization compares random one-dimensional projections with samples from a target distribution. Rectification maps negative values to zero, producing nonnegative sparse features; the default target is a rectified Laplace distribution. Matching a finite set of projections is a practical distribution-matching procedure, not an exact finite-sample equality guarantee.

Equations 1–2
min𝜽,ϕ‖𝐳^t+1−𝐳t+1‖2+λRDMReg⋅ℛ⁡(𝐳t+1)\displaystyle\min_{\boldsymbol{\theta},\boldsymbol{\phi}}\quad\|\hat{\mathbf{z}}_{t+1}-\mathbf{z}_{t+1}\|_{2}+\lambda_{\operatorname{RDMReg}}\cdot\mathcal{R}(\mathbf{z}_{t+1}) (1)
ℛ(𝐳)=𝔼𝐜∼Unif⁡(𝕊ℓ2d−1)[ℒ(ℙ𝐜⊤​𝐳∥ℙ𝐜⊤​𝐲)]\displaystyle\mathcal{R}(\mathbf{z})=\mathbb{E}_{\mathbf{c}\sim\operatorname{Unif}(\mathbb{S}^{d-1}_{\ell_{2}})}\big[\mathcal{L}(\mathbb{P}_{\mathbf{c}^{\top}\mathbf{z}}\|\mathbb{P}_{\mathbf{c}^{\top}\mathbf{y}})\big] (2)
𝐳t+1=f𝜽​(𝐱t+1)∈ℝd\mathbf{z}_{t+1}=f_{\boldsymbol{\theta}}(\mathbf{x}_{t+1})\in\mathbb{R}^{d}
𝐳^t+1=gϕ​(𝐳t,𝐚t)\hat{\mathbf{z}}_{t+1}=g_{\boldsymbol{\phi}}(\mathbf{z}_{t},\mathbf{a}_{t})
𝕊ℓ2d−1:={𝐱∈ℝd\mathbb{S}_{\ell_{2}}^{d-1}:=\{\mathbf{x}\in\mathbb{R}^{d}
𝐲∼∏i=1dReLU⁡(𝒢​𝒩p​(μ,σ))\mathbf{y}\sim\prod_{i=1}^{d}\operatorname{ReLU}(\mathcal{GN}_{p}(\mu,\sigma))

2.2 Architecture and Rectification

A vision transformer and projection network encode the image; an action-conditioned predictor estimates the next code. The RepReLU link outputs ordinary rectified features but routes gradients through a smoother function. This helps avoid inactive units during optimization. The distribution constraint, rather than this gradient-routing device alone, supplies the anti-collapse mechanism.

Equations
RepReLU⁡(x)=sg⁡(ReLU⁡(x))+GeLU⁡(x)−sg⁡(GeLU⁡(x))\operatorname{RepReLU}(x)=\operatorname{sg}(\operatorname{ReLU}(x))+\operatorname{GeLU}(x)-\operatorname{sg}(\operatorname{GeLU}(x))
RepReLU⁡(x)=ReLU⁡(x)\operatorname{RepReLU}(x)=\operatorname{ReLU}(x)

2.3 Planning with the Learned World Model

Encode the current and goal images, imagine candidate action sequences, and score the predicted final feature distance to the goal. The cross-entropy planner repeatedly samples and refines candidate sequences. Closed-loop execution observes the result and replans, so good planning depends on both the learned representation and prediction accuracy.

Refer to caption
(a) Support Jaccard Heatmap.
Original figure
(b) State Decoding.
Original figure
(c) Planning Evaluation.
Figure 2: The sparse support recovers the piecewise structure of the dynamics. (a) Support Jaccard heatmap over a 20×2020\times 20 grid: the Jaccard index of each cell’s support with a fixed reference cell (white ++) is high within the reference zone and drops across zone boundaries (dashed), for a model trained with rendered zones (top) and one with no visual cues (bottom). (b) The binary support decodes the ground-truth zone identity as accurately as the full embedding, while agent position is decoded better from the magnitudes than from the support. (c) Planning success vs. horizon (closed-loop, 3-seed mean±\pmstd). Goal locations of the dot agent are sampled randomly. Both sparse variants beat dense LeWM at every horizon.
Equation
𝒞=‖𝐳^T−𝐳g‖2\mathcal{C}=\|\hat{\mathbf{z}}_{T}-\mathbf{z}_{g}\|_{2}

3 Does Sparsity Simplify Latent Dynamics?

The key experiment varies predictor complexity while holding the representation-learning comparison as consistent as possible. If sparse geometry is easier to predict, its advantage should appear before the predictor becomes powerful enough to solve both versions.

3.1 Linearization of Nonlinear Dynamics with Sparse Representations

Imagine dividing the state space into many tiny cells and representing a state by the identity of its cell. For a fixed action, moving between cell labels is a matrix multiplication on one-hot vectors. Under compactness and Lipschitz assumptions, the decoded approximation becomes accurate as the cells shrink, for a fixed rollout horizon. The matrix depends on the action; this is not a claim of one globally linear system in state and action. Required code size grows exponentially with state dimension.

Proposition 1 (Linear Approximation of Nonlinear Dynamics with Sparse Representations).

Let 𝒳⊂ℝd\mathcal{X}\subset\mathbb{R}^{d} be a compact state space and let 𝒜⊂ℝp\mathcal{A}\subset\mathbb{R}^{p} be a compact continuous action space. Consider a deterministic discrete-time controlled dynamical system 𝐱t+1=f⁡(𝐱t,𝐚t)\mathbf{x}_{t+1}=f(\mathbf{x}_{t},\mathbf{a}_{t}) where f:𝒳×𝒜→𝒳.f:\mathcal{X}\times\mathcal{A}\rightarrow\mathcal{X}. Assume that ff is uniformly Lipschitz in the state variable, i.e. there exists a finite constant L≥0L\geq 0 such that ‖f⁡(𝐱,𝐚)−f⁡(𝐲,𝐚)‖≤L​‖𝐱−𝐲‖\left\|f(\mathbf{x},\mathbf{a})-f(\mathbf{y},\mathbf{a})\right\|\leq L\|\mathbf{x}-\mathbf{y}\| for every 𝐱,𝐲∈𝒳\mathbf{x},\mathbf{y}\in\mathcal{X} and 𝐚∈𝒜\mathbf{a}\in\mathcal{A}. Then, for every ε>0\varepsilon>0, there exist 1) a finite latent dimension NN; 2) a one-hot sparse encoder Eε:𝒳→ℝNE_{\varepsilon}:\mathcal{X}\rightarrow\mathbb{R}^{N}; 3) a decoder Dε:{𝐞1,…,𝐞N}→𝒳D_{\varepsilon}:\{\mathbf{e}_{1},\ldots,\mathbf{e}_{N}\}\rightarrow\mathcal{X}; and 4) an action-conditioned family of matrices Pε:𝒜→{0,1}N×N,P_{\varepsilon}:\mathcal{A}\rightarrow\{0,1\}^{N\times N}, 𝐚↦𝐏ε​(𝐚),\mathbf{a}\mapsto\mathbf{P}_{\varepsilon}(\mathbf{a}), such that the latent dynamics 𝐳t+1=𝐏ε​(𝐚t)​𝐳t\mathbf{z}_{t+1}=\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})\mathbf{z}_{t} are exactly linear in 𝐳t\mathbf{z}_{t}, and the decoded one-step prediction satisfies

Equation 3
sup𝐱t∈𝒳,𝐚t∈𝒜‖f⁡(𝐱t,𝐚t)−Dε​(𝐏ε​(𝐚t)​Eε​(𝐱t))‖≤(L+1)​ε.\displaystyle\sup_{\mathbf{x}_{t}\in\mathcal{X},\,\mathbf{a}_{t}\in\mathcal{A}}\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-D_{\varepsilon}\!\left(\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})E_{\varepsilon}(\mathbf{x}_{t})\right)\right\|\leq(L+1)\varepsilon. (3)
Corollary 1 (Sparsity in High-Dimensions).

Let the state space be 𝒳=[0,1]d\mathcal{X}=[0,1]^{d} and consider dividing each coordinate into nn equal intervals, producing N=ndN=n^{d} grid cells. We represent each cell by its center with the covering radius εN=d2N−1/d\varepsilon_{N}=\frac{\sqrt{d}}{2}N^{-1/d}. Hence the one-step prediction error satisfies

Equation 4
sup𝐱,𝐚‖f(𝐱,𝐚)−DεN(𝐏εN(𝐚)EεN(𝐱))‖≤d2(L+1)N−1/d=O(N−1/d).\displaystyle\sup_{\mathbf{x},\mathbf{a}}\left\|f(\mathbf{x},\mathbf{a})-D_{\varepsilon_{N}}\!\left(\mathbf{P}_{\varepsilon_{N}}(\mathbf{a})E_{\varepsilon_{N}}(\mathbf{x})\right)\right\|\leq\frac{\sqrt{d}}{2}(L+1)N^{-1/d}=O(N^{-1/d}). (4)

For any fixed rollout horizon HH, ‖𝐱H−𝐱^H‖\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\| ≤d2N−1/d∑k=0HLk\leq\frac{\sqrt{d}}{2}N^{-1/d}\sum_{k=0}^{H}L^{k} and limN→∞\lim_{N\rightarrow\infty} ‖𝐱H−𝐱^H‖=0\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|=0.

3.2 Experiment Designs

Practical models learn distributed sparse codes rather than an infeasibly large one-hot lookup table. Tests compare transformer, state-dependent and fixed linear-core predictors across feature widths, with dense Gaussian and VICReg baselines. Width-dependent parameterization is used to make the capacity comparison more stable. These learned models are empirical relaxations of the theorem.

Table 1: Predictors with Different Levels of Complexity. 𝐳^t+1\hat{\mathbf{z}}_{t+1} is the predicted latent, 𝐳t−i\mathbf{z}_{t-i} past latents, and 𝐚t\mathbf{a}_{t} the action. AdaLN predictors are Transformer-based models with Adaptive Layer Normalization for action conditioning, i.e. DiT architecture (30). The Deep-AdaLN(k)(k) predictor is the one used in LeWM (28). LTI denotes a linear time-invariant predictor with fixed operators 𝐀i,𝐁\mathbf{A}_{i},\mathbf{B}, while LTV denotes a linear time-varying predictor with state-dependent operators 𝐀i​(𝐳t),𝐁⁡(𝐳t)\mathbf{A}_{i}(\mathbf{z}_{t}),\mathbf{B}(\mathbf{z}_{t}). We use MLP∘\circ to denote one extra nonlinear transform. The history kk denotes context length, with 11 the single-frame case. The output link is σ=identity\sigma=\mathrm{identity} for dense LeWM and σ=RepReLU\sigma=\mathrm{RepReLU} for sparse LpWM. See Appendix B for details.
Predictor Layers History (kk) Functional form
Deep-AdaLN(k)(k) 66 33 𝐳^t+1=σ(AdaLN(6)(𝐳t−k+1:t,𝐚t))\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\mathrm{AdaLN}^{(6)}(\mathbf{z}_{t-k+1:t},\,\mathbf{a}_{t})\right)
Shallow-AdaLN(k)(k) 11 33 𝐳^t+1=σ(AdaLN(1)(𝐳t−k+1:t,𝐚t))\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\mathrm{AdaLN}^{(1)}(\mathbf{z}_{t-k+1:t},\,\mathbf{a}_{t})\right)
MLP∘\circLTV(k)(k) 11 33 𝐳^t+1=σ⁡(𝐖​ReLU​(∑i=0k−1𝐀i​(𝐳t)​𝐳t−i+𝐁⁡(𝐳t)​𝐚t))\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\mathbf{W}\,\mathrm{ReLU}(\textstyle\sum_{i=0}^{k-1}\mathbf{A}_{i}(\mathbf{z}_{t})\,\mathbf{z}_{t-i}+\mathbf{B}(\mathbf{z}_{t})\,\mathbf{a}_{t})\right)
MLP∘\circLTI(k)(k) 11 33 𝐳^t+1=σ⁡(𝐖​ReLU​(∑i=0k−1𝐀i​𝐳t−i+𝐁​𝐚t))\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\mathbf{W}\,\mathrm{ReLU}(\textstyle\sum_{i=0}^{k-1}\mathbf{A}_{i}\,\mathbf{z}_{t-i}+\mathbf{B}\,\mathbf{a}_{t})\right)
LTI(k)(k) 11 33 𝐳^t+1=σ⁡(∑i=0k−1𝐀i​𝐳t−i+𝐁​𝐚t)\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\textstyle\sum_{i=0}^{k-1}\mathbf{A}_{i}\,\mathbf{z}_{t-i}+\mathbf{B}\,\mathbf{a}_{t}\right)
LTI(1)(1) 11 11 𝐳^t+1=σ⁡(𝐀​𝐳t+𝐁​𝐚t)\hat{\mathbf{z}}_{t+1}=\sigma\!\left(\mathbf{A}\,\mathbf{z}_{t}+\mathbf{B}\,\mathbf{a}_{t}\right)
Table 2: Predictor parameter counts across latent dimensions DD. Counts are for the predictor alone for each DD used in our experiments. LTI, MLP∘\circLTI, and MLP∘\circLTV are attention-free and far smaller than the AdaLN Transformers.
Predictor D=384D{=}384 D=768D{=}768 D=1536D{=}1536 D=2048D{=}2048 D=4096D{=}4096
Deep-AdaLN(k)(k) 25.825.8M 62.262.2M 166.9166.9M 260.2260.2M 822.4822.4M
Shallow-AdaLN(k)(k) 5.65.6M 13.013.0M 33.133.1M 50.450.4M 151.1151.1M
MLP∘\circLTV(k)(k) 0.810.81M 3.103.10M 12.112.1M 21.421.4M 84.784.7M
MLP∘\circLTI(k)(k) 0.740.74M 2.952.95M 11.811.8M 21.021.0M 83.983.9M
LTI(k)(k) 0.590.59M 2.362.36M 9.449.44M 16.816.8M 67.167.1M
LTI(1)(1) 0.300.30M 1.181.18M 4.724.72M 8.398.39M 33.633.6M

3.3 Results

The easy Wall task is already solved by simple predictors for both code types. On PushT, the weakest predictor fails for both, strong predictors reduce the gap, and intermediate-capacity predictors benefit most from sparsity. The result is a capacity-dependent advantage in these settings, rather than a universal ranking of sparse over dense representations.

Original figure
(a) Wall Environment. LeWM vs. LpWM
Original figure
(b) PushT Environment. VICReg vs. LpWM
Figure 3: Linear predictor is sufficient for the wall environment regardless of dense or sparse embeddings, and LpWM outperforms VICReg-based JEPA across predictor settings in PushT. (a) We plot the planning success rate of the linear LTI(1)(1) predictor against the nonlinear Deep-AdaLN(k)(k), where each dot denotes one setting of method and feature dimension. For simplicity, we use μ=∗\mu=* to denote LpWM with the target distribution parameter μ\mu. We show the open-loop evaluation on top and closed-loop results at bottom. Both are ran using CEM planner. We observe that the closed-loop success rate for LTI(1)(1) predictor using SIGReg is already approaching 100%100\%, indicating that the wall environment can already be described well by a linear dynamics model in the latent space. (b) We have already shown LeWM vs. LpWM in Figure 1(b). Here we compare LpWM with a variant of LeWM where we replaced SIGReg with VICReg in the PushT environment across predictor settings. As oppposed to SIGReg, we observe consistent improvement from LpWM over VICReg even for the Deep-AdaLN(k)(k) predictor, and the gap between sparse LpWM and dense methods (i.e. VICReg here) continues to hold for MLP∘\circLTV(k)(k) and MLP∘\circLTI(k)(k) predictors. See full hyperparameter sweeps in Appendix H.2.

4 Does Sparsity Lead to Interpretable Latent Dynamics?

A sparse representation has two parts: support identifies which coordinates are nonzero; magnitudes describe their values. The paper asks whether support marks discrete regimes while magnitudes retain continuous state within a regime.

Original figure
(a) Temporal Jaccard Regularization
Original figure
(b) Example Episode.
Refer to caption
(c) Support Raster Plot.
Figure 4: Temporal-Jaccard (TJ) makes the cube support track contact, not effector motion. (a) As TJ strength increases, support-instability correlation with effector motion drops (0.87→0.400.87{\to}0.40) while cube-motion (0.26→0.800.26{\to}0.80) and gripper-contact (0.05→0.610.05{\to}0.61) rise—at no change in planning success. (b) One example episode: without TJ (top) support instability tracks effector displacement (reff=0.86r_{\mathrm{eff}}{=}0.86); with TJ (bottom) it tracks cube displacement (rcube=0.80r_{\mathrm{cube}}{=}0.80), spiking inside the shaded contact windows. (c) Support raster: without TJ the support rapidly transitions across frame; with TJ it is smoothly varying in time, reorganizing almost during the shaded contact window.

4.1 Discovering Piecewise-Affine Dynamics

The controlled Piecewise environment has zones with different force biases. Sparse support identifies those zones even when the background does not reveal them visually; feature magnitudes better recover continuous position. Overlap maps, linear probes and planning tests support this separation in the constructed environment, where the regime labels are known.

Equation 5
J⁡(𝐱,𝐲)=∑i=1D𝟙𝐱i=1∧𝐲i=1∑i=1D𝟙𝐱i=1∨𝐲i=1,\displaystyle J(\mathbf{x},\mathbf{y})=\frac{\sum_{i=1}^{D}\mathbbm{1}_{\mathbf{x}_{i}=1\land\mathbf{y}_{i}=1}}{\sum_{i=1}^{D}\mathbbm{1}_{\mathbf{x}_{i}=1\lor\mathbf{y}_{i}=1}}, (5)

4.2 From Navigation to Contact-Rich Dynamics

On cube manipulation, unregularized support mainly tracks arm movement rather than contact. Adding a temporal Jaccard loss makes the active-feature pattern change more slowly and align better with cube interaction. Sparsity alone therefore does not guarantee a human-meaningful regime decomposition.

5 Conclusion

Sparse feature geometry can reduce the predictor capacity needed for the tested planning tasks. The strongest evidence is the controlled predictor-capacity comparison. Interpretable support is conditional on the environment and temporal constraints; broad conclusions about harder robotics and longer horizons remain open.

Acknowledgement

The authors acknowledge discussions, research support and computing resources that supported the experiments.

Appendix A Related Work

Latent world models for planning.

Latent world models plan by predicting encoded observations rather than reconstructing every pixel. Frozen-feature and jointly trained encoders make different representation-learning choices; this paper focuses on controlling geometry during joint training.

Feature Regularization for Latent World Models.

Anti-collapse methods constrain representation variation or match a target distribution. Their geometry can affect the later dynamics problem. The sparse-versus-dense comparison concerns that geometry as well as the prediction objective.

Sparse and Discrete Representations in World Models.

Sparse and discrete codes restrict which latent patterns are available. LpWM uses distributed nonnegative sparsity, which differs from assigning each observation one discrete token or one-hot identity.

Linearizing Nonlinear Dynamics.

Lifting a nonlinear system into a larger representation can simplify its dynamics. The theorem uses finite state-space quantization; the learned sparse representation is motivated by this idea without inheriting its exact construction or guaranteed accuracy.

Table 3: Sparsity of the LpWM encoder representations across predictor types and latent dimensions. Each entry is the fraction of non-zero coordinates 𝔼⁡[∥𝐳∥0]/D\mathbb{E}[\lVert\mathbf{z}\rVert_{0}]/D in the sparse code for the target distribution with {μ=0,σ=1/2,p=1}\{\mu=0,\sigma=\sqrt{1/2},p=1\}, measured on the validation set at the tuned-best cell for the LpWM run paired with each predictor at latent dimension DD. Dense LeWM has active fraction 1.01.0 by construction. The learned code stays ∼30\sim\!30–65%65\% active regardless of predictor or dimension, confirming the codes underlying the planning comparison are genuinely sparse.
DD Deep-AdaLN(k)(k) Shallow-AdaLN(k)(k) MLP∘\circLTV(k)(k) MLP∘\circLTI(k)(k) LTI(k)(k) LTI(1)(1)
384384 0.340.34 0.330.33 0.370.37 0.280.28 0.410.41 0.630.63
768768 0.440.44 0.370.37 0.540.54 0.530.53 0.410.41 0.630.63
15361536 0.490.49 0.390.39 0.490.49 0.540.54 0.410.41 0.600.60
20482048 0.420.42 0.460.46 0.510.51 0.460.46 0.430.43 0.610.61
40964096 0.480.48 0.490.49 0.470.47 0.430.43 0.440.44 0.560.56
Table 4: Success Rate (%) over 50 Test Samples in the Piecewise Environments with Varying Numbers of Zones. Piecewise (n×nn\times n) denotes the environment with varying numbers of zones. Goal images are either sampled from the evaluation dataset or randomly sampled from all possible locations of the dot agent. HH denotes the horizon and RR represents the receding-horizon. Results are shown as mean±\pmstd over 3 seeds. Bold denotes the best performance.
Goals from Evaluation Set Random Goals
Env Method H=R=5H=R=5 H=5,R=1H=5,R=1 H=R=5H=R=5
Piecewise (2×22\times 2) LpWM (Ours) 99.33±1.15\mathbf{99.33{\pm}1.15} 84.67±4.16\mathbf{84.67{\pm}4.16} 59.33±2.3159.33{\pm}2.31
LpWM+TJ (Ours) 99.33±1.15\mathbf{99.33{\pm}1.15} 82.67±5.0382.67{\pm}5.03 68.67±4.62\mathbf{68.67{\pm}4.62}
LeWM 99.33±1.15\mathbf{99.33{\pm}1.15} 65.33±4.1665.33{\pm}4.16 36.00±3.4636.00{\pm}3.46
Piecewise (3×33\times 3) LpWM (Ours) 100.00±0.00\mathbf{100.00{\pm}0.00} 58.67±1.1558.67{\pm}1.15 56.00±4.0056.00{\pm}4.00
LpWM+TJ (Ours) 100.00±0.00\mathbf{100.00{\pm}0.00} 60.00±2.00\mathbf{60.00{\pm}2.00} 58.00±5.29\mathbf{58.00{\pm}5.29}
LeWM 100.00±0.00\mathbf{100.00{\pm}0.00} 50.00±5.2950.00{\pm}5.29 47.33±6.1147.33{\pm}6.11

Appendix B Predictor Designs

B.1 Overall Designs

The predictor ladder ranges from several transformer blocks to fixed linear state-space maps, sometimes followed by a nonlinear readout. The sparse models also retain a rectifying output link, so even a nominally linear predictor need not be an entirely linear end-to-end dynamics model.

B.2 MLP ∘ LTV ( k )

A state-dependent gate adds low-rank corrections to fixed transition matrices. This supplies some local flexibility without generating a full matrix with a cubic parameter cost. The action enters the dynamics core affinely, before the shared nonlinear readout. Zero-initialized corrections start from the simpler fixed-operator model.

Equation 6
𝐀i​(𝐳t)=𝐀i+𝐔i​diag​(𝐠i​(𝐳t))​𝐕i⊤,𝐠⁡(𝐳t)=sigmoid⁡(𝐆​𝐳t)∈[0,1]r,\displaystyle\mathbf{A}_{i}(\mathbf{z}_{t})\;=\;\mathbf{A}_{i}\;+\;\mathbf{U}_{i}\,\mathrm{diag}\!\big(\mathbf{g}_{i}(\mathbf{z}_{t})\big)\,\mathbf{V}_{i}^{\top},\qquad\mathbf{g}(\mathbf{z}_{t})=\mathrm{sigmoid}(\mathbf{G}\,\mathbf{z}_{t})\in[0,1]^{r}, (6)
Refer to caption
(a) Wall
Refer to caption
(b) PushT
Refer to caption
(c) Piecewise
Refer to caption
(d) OGBench-Cube
Figure 5: Environments. (a) Wall: 2D navigation through a doorway (red agent). (b) PushT: push a T-shaped block to a target pose (green). (c) Piecewise: an open room partitioned into force-field zones (colored) with a known ground-truth segmentation. (d) OGBench-Cube: a robot arm manipulating a cube.
Equation
𝐳^t+1=σ⁡(𝐖​ReLU​(∑i=0k−1𝐀i​(𝐳t)​𝐳t−i+𝐁⁡(𝐳t)​𝐚t))\hat{\mathbf{z}}_{t+1}=\sigma\!\big(\mathbf{W}\,\mathrm{ReLU}(\sum_{i=0}^{k-1}\mathbf{A}_{i}(\mathbf{z}_{t})\,\mathbf{z}_{t-i}+\mathbf{B}(\mathbf{z}_{t})\,\mathbf{a}_{t})\big)

Appendix C Environment

C.1 TwoRoom, PushT, and OGBench Cube

TwoRoom, PushT and cube-manipulation experiments reuse the cited datasets. Goal images are generally selected 25 raw steps ahead, with five-action blocks. Those choices define relatively short-horizon planning tests.

C.2 Piecewise

Piecewise.

Piecewise divides an otherwise open room into zones, each adding a different motion bias. The resulting dynamics are simple within a zone but change across boundaries. Because the true zones and motion rule are known, the experiment can directly test whether the learned support recovers them.

Equations
𝐩t+1=𝐩t+𝐚t⋅speed+𝐛⁡[zone​(𝐩t)]\mathbf{p}_{t+1}=\mathbf{p}_{t}+\mathbf{a}_{t}\cdot\text{speed}+\mathbf{b}[\text{zone}(\mathbf{p}_{t})]
𝐛i=bias_scale⋅(cos⁡θi,sin⁡θi)\mathbf{b}_{i}=\text{bias\_scale}\cdot(\cos\theta_{i},\sin\theta_{i})

C.3 Visualization of Environments

The original environment snapshots are retained. A version without colored zone backgrounds checks whether regime-sensitive features can emerge without an obvious visual label for each zone.

Appendix D Experimental Details

Two pipelines study different questions: predictor complexity on Wall/PushT, and support interpretation on Piecewise/cube manipulation. They share the LpWM objective but differ in architecture and training. Comparisons should be made within their respective pipeline, using the retained configuration table.

Table 5: Experimental configurations for Section 3 and Section 4. We list out the experimental pipelines in the following table.
Section 3 Section 4
Wall PushT Piecewise OGBench-Cube
Encoder from-scratch ViT, 1212 layers, width 384384 from-scratch ViT-Tiny, width 192192
Predictor See Table 1 Deep-AdaLN(k)(k)
Dimension (DD) {384,768,1536,2048,4096}\{384,768,1536,2048,4096\} 192192
History kk 11 33 33 33
Frameskip 55 55 55 55
Batch size 128128 6464 128128 128128
Epochs 2020 22 1010 1010
Optimizer Adam, μ\muP (with base width 384384) AdamW
Base LR See Appendix H ×10−55\!\times\!10^{-5}
Grad. clip 0.0 1.01.0
Reg. weight λ\lambda See Appendix H 2525 1010
Number of projections 81928192 10241024
Parameter μ\mu {0,−1,−2}\{0,-1,-2\} 00 00
Temporal Jaccard off off {0.0,0.01}\{0.0,0.01\} {0.0,0.005,0.01,0.05,0.1}\{0.0,0.005,0.01,0.05,0.1\}

Planning and Evaluation.

The cross-entropy planner uses 300 candidate sequences, keeps 30, and performs 30 refinement iterations. A five-step planning horizon with frameskip five reaches 25 raw steps. Closed-loop tests replan after each action block; open-loop tests execute one plan. Short cube horizons can saturate performance, limiting what those results establish about difficult long-horizon manipulation.

Appendix E Jaccard Index

E.1 Hard and Soft Jaccard index

Jaccard similarity measures the fraction of active coordinates shared by two supports, relative to the coordinates active in either. A continuous relaxation replaces set operations with elementwise minimum and maximum, allowing a differentiable overlap loss for nonnegative codes.

Equations 7
JS​(𝐱,𝐲)=∑i=1Dmin⁡(𝐱i,𝐲i)∑i=1Dmax⁡(𝐱i,𝐲i)+ϵ,\displaystyle J_{S}(\mathbf{x},\mathbf{y})=\frac{\sum_{i=1}^{D}\min(\mathbf{x}_{i},\mathbf{y}_{i})}{\sum_{i=1}^{D}\max(\mathbf{x}_{i},\mathbf{y}_{i})+\epsilon}, (7)
Jac⁡(A,B)=|A∩B|/|A∪B|\operatorname{Jac}(A,B)=|A\cap B|/|A\cup B|
J⁡(𝐱,𝐲)=∑i=1D𝟙𝐱i=1∧𝐲i=1/∑i=1D𝟙𝐱i=1∨𝐲i=1J(\mathbf{x},\mathbf{y})=\sum_{i=1}^{D}\mathbbm{1}_{\mathbf{x}_{i}=1\land\mathbf{y}_{i}=1}\,/\,\sum_{i=1}^{D}\mathbbm{1}_{\mathbf{x}_{i}=1\lor\mathbf{y}_{i}=1}

E.2 Temporal Jaccard Loss

Matching the sparse distribution separately at each time does not specify how support should evolve. The optional temporal loss penalizes rapid changes between neighboring frames. It adds a preference for stable support; it does not itself identify which physical events should define a regime.

Equation 8
ℒTJ=1B⁡(T−1)∑b=1B∑t=1T−1(1−JS(𝐙[b,t,:],𝐙[b,t+1,:]))\displaystyle\mathcal{L}_{\operatorname{TJ}}=\frac{1}{B(T-1)}\sum_{b=1}^{B}\sum_{t=1}^{T-1}\Big(1-J_{S}\big(\mathbf{Z}[b,t,:],\mathbf{Z}[b,t+1,:]\big)\Big) (8)

Appendix F Maximal-Update Parameterization ( μ P) for JEPA

Maximum-update parameterization rescales initialization and learning rates as feature width changes. The goal is stable feature learning across capacities, rather than proving that one tuned learning rate always transfers. The authors report occasional transfer exceptions and use explicit sweeps.

Equations 9–10
η𝐖∝1din.\displaystyle\eta_{\mathbf{W}}\propto\frac{1}{d_{\mathrm{in}}}. (9)
η𝐖​(D)=ηbase​D0D.\displaystyle\eta_{\mathbf{W}}(D)=\eta_{\mathrm{base}}\frac{D_{0}}{D}. (10)
𝐡=𝐖𝐱\mathbf{h}=\mathbf{W}\mathbf{x}
Θ⁡(dout)\Theta(\sqrt{d_{\mathrm{out}}})
‖𝐖‖∗=Θ⁡(dout/din)\|\mathbf{W}\|_{*}=\Theta(\sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}})

Appendix G Proofs of Linearization

G.1 Proof of Proposition 1

The proof covers the compact state space by finitely many small cells, encodes each representative with one active coordinate, and defines each action’s transition matrix by the representative’s next cell. The error combines the original quantization error, Lipschitz propagation and quantization of the next state. Repeating this argument gives the finite-horizon bound. Original proposition, lemma and equations are retained.

Equations 11–23
𝒳⊆⋃𝐜∈𝒳Bε​(𝐜)\displaystyle\mathcal{X}\subseteq\bigcup_{\mathbf{c}\in\mathcal{X}}B_{\varepsilon}(\mathbf{c}) (11)
𝒳⊆⋃𝐜∈CεBε​(𝐜)\displaystyle\mathcal{X}\subseteq\bigcup_{\mathbf{c}\in C_{\varepsilon}}B_{\varepsilon}(\mathbf{c}) (12)
𝐳t+1=𝐏ε​(𝐚t)​𝐳t\displaystyle\mathbf{z}_{t+1}=\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})\mathbf{z}_{t} (13)
𝐏ε​(𝐚t)​Eε​(𝐱t)\displaystyle\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})E_{\varepsilon}(\mathbf{x}_{t}) =𝐏ε​(𝐚t)​𝐞i=𝐞τ𝐚t​(i)\displaystyle=\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})\mathbf{e}_{i}=\mathbf{e}_{\tau_{\mathbf{a}_{t}}(i)} (14)
𝐱^t+1\displaystyle\hat{\mathbf{x}}_{t+1} :=Dε​(𝐏ε​(𝐚t)​Eε​(𝐱t))=𝐜Qε​(f⁡(𝐜i,𝐚t)).\displaystyle:=D_{\varepsilon}\!\left(\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})E_{\varepsilon}(\mathbf{x}_{t})\right)=\mathbf{c}_{Q_{\varepsilon}(f(\mathbf{c}_{i},\mathbf{a}_{t}))}. (15)
𝐱t+1=f⁡(𝐱t,𝐚t).\displaystyle\mathbf{x}_{t+1}=f(\mathbf{x}_{t},\mathbf{a}_{t}). (16)
‖𝐱t+1−𝐱^t+1‖\displaystyle\|\mathbf{x}_{t+1}-\hat{\mathbf{x}}_{t+1}\| =‖f⁡(𝐱t,𝐚t)−𝐜Qε​(f⁡(𝐜i,𝐚t))‖\displaystyle=\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-\mathbf{c}_{Q_{\varepsilon}(f(\mathbf{c}_{i},\mathbf{a}_{t}))}\right\| (17)
≤‖f⁡(𝐱t,𝐚t)−f⁡(𝐜i,𝐚t)‖+‖f⁡(𝐜i,𝐚t)−𝐜Qε​(f⁡(𝐜i,𝐚t))‖.\displaystyle\leq\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-f(\mathbf{c}_{i},\mathbf{a}_{t})\right\|+\left\|f(\mathbf{c}_{i},\mathbf{a}_{t})-\mathbf{c}_{Q_{\varepsilon}(f(\mathbf{c}_{i},\mathbf{a}_{t}))}\right\|. (18)
‖f⁡(𝐱t,𝐚t)−f⁡(𝐜i,𝐚t)‖\displaystyle\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-f(\mathbf{c}_{i},\mathbf{a}_{t})\right\| ≤L​‖𝐱t−𝐜i‖≤L​ε.\displaystyle\leq L\|\mathbf{x}_{t}-\mathbf{c}_{i}\|\leq L\varepsilon. (19)
‖f⁡(𝐜i,𝐚t)−𝐜Qε​(f⁡(𝐜i,𝐚t))‖≤ε.\displaystyle\left\|f(\mathbf{c}_{i},\mathbf{a}_{t})-\mathbf{c}_{Q_{\varepsilon}(f(\mathbf{c}_{i},\mathbf{a}_{t}))}\right\|\leq\varepsilon. (20)
‖𝐱t+1−𝐱^t+1‖\displaystyle\|\mathbf{x}_{t+1}-\hat{\mathbf{x}}_{t+1}\| ≤L​ε+ε=(L+1)​ε.\displaystyle\leq L\varepsilon+\varepsilon=(L+1)\varepsilon. (21)
sup𝐱t∈𝒳,𝐚t∈𝒜‖f⁡(𝐱t,𝐚t)−Dε​(𝐏ε​(𝐚t)​Eε​(𝐱t))‖≤(L+1)​ε.\displaystyle\sup_{\mathbf{x}_{t}\in\mathcal{X},\,\mathbf{a}_{t}\in\mathcal{A}}\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-D_{\varepsilon}\!\left(\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})E_{\varepsilon}(\mathbf{x}_{t})\right)\right\|\leq(L+1)\varepsilon. (22)
limε→0sup𝐱t∈𝒳,𝐚t∈𝒜‖f⁡(𝐱t,𝐚t)−Dε​(𝐏ε​(𝐚t)​Eε​(𝐱t))‖=0.\displaystyle\lim_{\varepsilon\rightarrow 0}\sup_{\mathbf{x}_{t}\in\mathcal{X},\,\mathbf{a}_{t}\in\mathcal{A}}\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-D_{\varepsilon}\!\left(\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})E_{\varepsilon}(\mathbf{x}_{t})\right)\right\|=0. (23)
Lemma 1 (Finite-horizon rollout bound).

Under the assumptions of Proposition 1, let 𝐱t+1=f⁡(𝐱t,𝐚t)\mathbf{x}_{t+1}=f(\mathbf{x}_{t},\mathbf{a}_{t}) for t=0,…,H−1t=0,\ldots,H-1 be the ground-truth trajectory under an arbitrary action sequence 𝐚0,…,𝐚H−1∈𝒜\mathbf{a}_{0},\ldots,\mathbf{a}_{H-1}\in\mathcal{A}. Initialize 𝐳0=Eε​(𝐱0)\mathbf{z}_{0}=E_{\varepsilon}(\mathbf{x}_{0}) and recursively define the latent transition as 𝐳t+1=𝐏ε​(𝐚t)​𝐳t.\mathbf{z}_{t+1}=\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})\mathbf{z}_{t}. Let 𝐱^t=Dε​(𝐳t)\hat{\mathbf{x}}_{t}=D_{\varepsilon}(\mathbf{z}_{t}) be the decoded state. Then

Equation 24
‖𝐱H−𝐱^H‖≤ε​∑k=0HLk.\displaystyle\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|\leq\varepsilon\sum_{k=0}^{H}L^{k}. (24)
Equations 25–30
𝐳t+1\displaystyle\mathbf{z}_{t+1} =𝐏ε​(𝐚t)​𝐞i=𝐞Qε​(f⁡(𝐜i,𝐚t)).\displaystyle=\mathbf{P}_{\varepsilon}(\mathbf{a}_{t})\mathbf{e}_{i}=\mathbf{e}_{Q_{\varepsilon}(f(\mathbf{c}_{i},\mathbf{a}_{t}))}. (25)
𝐱^t+1\displaystyle\hat{\mathbf{x}}_{t+1} =Dε​(𝐳t+1)=qε​(f⁡(𝐱^t,𝐚t)).\displaystyle=D_{\varepsilon}(\mathbf{z}_{t+1})=q_{\varepsilon}\!\left(f(\hat{\mathbf{x}}_{t},\mathbf{a}_{t})\right). (26)
et+1\displaystyle e_{t+1} =‖f⁡(𝐱t,𝐚t)−qε​(f⁡(𝐱^t,𝐚t))‖\displaystyle=\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-q_{\varepsilon}\!\left(f(\hat{\mathbf{x}}_{t},\mathbf{a}_{t})\right)\right\| (27)
≤‖f⁡(𝐱t,𝐚t)−f⁡(𝐱^t,𝐚t)‖+‖f⁡(𝐱^t,𝐚t)−qε​(f⁡(𝐱^t,𝐚t))‖\displaystyle\leq\left\|f(\mathbf{x}_{t},\mathbf{a}_{t})-f(\hat{\mathbf{x}}_{t},\mathbf{a}_{t})\right\|+\left\|f(\hat{\mathbf{x}}_{t},\mathbf{a}_{t})-q_{\varepsilon}\!\left(f(\hat{\mathbf{x}}_{t},\mathbf{a}_{t})\right)\right\|
≤L​et+ε.\displaystyle\leq Le_{t}+\varepsilon. (28)
eH\displaystyle e_{H} ≤LH​e0+ε​∑k=0H−1Lk≤LH​ε+ε​∑k=0H−1Lk=ε​∑k=0HLk.\displaystyle\leq L^{H}e_{0}+\varepsilon\sum_{k=0}^{H-1}L^{k}\leq L^{H}\varepsilon+\varepsilon\sum_{k=0}^{H-1}L^{k}=\varepsilon\sum_{k=0}^{H}L^{k}. (29)
‖𝐱H−𝐱^H‖≤ε​∑k=0HLk.\displaystyle\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|\leq\varepsilon\sum_{k=0}^{H}L^{k}. (30)
Cε={𝐜1,…,𝐜N}⊂𝒳C_{\varepsilon}=\{\mathbf{c}_{1},\ldots,\mathbf{c}_{N}\}\subset\mathcal{X}
Bε​(𝐜):={𝐱∈𝒳:‖𝐱−𝐜‖<ε}.B_{\varepsilon}(\mathbf{c}):=\left\{\mathbf{x}\in\mathcal{X}:\|\mathbf{x}-\mathbf{c}\|<\varepsilon\right\}.
Eε​(𝐱)=𝐞Qε​(𝐱),E_{\varepsilon}(\mathbf{x})=\mathbf{e}_{Q_{\varepsilon}(\mathbf{x})},
‖Eε​(𝐱)‖0=1\|E_{\varepsilon}(\mathbf{x})\|_{0}=1
Dε​(𝐞i)=𝐜i.D_{\varepsilon}(\mathbf{e}_{i})=\mathbf{c}_{i}.
τ𝐚​(i):=Qε​(f⁡(𝐜i,𝐚))\tau_{\mathbf{a}}(i):=Q_{\varepsilon}\!\left(f(\mathbf{c}_{i},\mathbf{a})\right)
𝐏ε​(𝐚)​𝐞i=𝐞τ𝐚​(i).\mathbf{P}_{\varepsilon}(\mathbf{a})\mathbf{e}_{i}=\mathbf{e}_{\tau_{\mathbf{a}}(i)}.
i:=Qε​(𝐱t).i:=Q_{\varepsilon}(\mathbf{x}_{t}).
Eε​(𝐱t)=𝐞iE_{\varepsilon}(\mathbf{x}_{t})=\mathbf{e}_{i}
qε​(𝐱):=Dε​(Eε​(𝐱))=𝐜Qε​(𝐱).q_{\varepsilon}(\mathbf{x}):=D_{\varepsilon}(E_{\varepsilon}(\mathbf{x}))=\mathbf{c}_{Q_{\varepsilon}(\mathbf{x})}.
𝐳t=𝐞i,\mathbf{z}_{t}=\mathbf{e}_{i},
𝐱^t=Dε​(𝐳t)=𝐜i.\hat{\mathbf{x}}_{t}=D_{\varepsilon}(\mathbf{z}_{t})=\mathbf{c}_{i}.
et:=‖𝐱t−𝐱^t‖.e_{t}:=\|\mathbf{x}_{t}-\hat{\mathbf{x}}_{t}\|.
e0=‖𝐱0−𝐱^0‖≤ε.e_{0}=\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}\|\leq\varepsilon.

G.2 Proof of Corollary 1

For a uniform grid on a bounded cube, increasing the number of cells reduces their radius and therefore the approximation error at any fixed horizon. In high dimensions, obtaining small cells needs exponentially many codes—the curse of dimensionality acknowledged by the paper.

Equations 31–39
|𝐱j−𝐜i,j|≤h2,j=1,…,d.\displaystyle|\mathbf{x}_{j}-\mathbf{c}_{i,j}|\leq\frac{h}{2},\qquad j=1,\ldots,d. (31)
‖𝐱−𝐜i‖2\displaystyle\|\mathbf{x}-\mathbf{c}_{i}\|^{2} =∑j=1d|𝐱j−𝐜i,j|2≤∑j=1d(h2)2=d​h24.\displaystyle=\sum_{j=1}^{d}|\mathbf{x}_{j}-\mathbf{c}_{i,j}|^{2}\leq\sum_{j=1}^{d}\left(\frac{h}{2}\right)^{2}=d\frac{h^{2}}{4}. (32)
εN=d2​n=d2N−1/d.\displaystyle\varepsilon_{N}=\frac{\sqrt{d}}{2n}=\frac{\sqrt{d}}{2}N^{-1/d}. (33)
sup𝐱,𝐚‖f⁡(𝐱,𝐚)−DεN​(𝐏εN​(𝐚)​EεN​(𝐱))‖≤(L+1)​εN.\displaystyle\sup_{\mathbf{x},\mathbf{a}}\left\|f(\mathbf{x},\mathbf{a})-D_{\varepsilon_{N}}\!\left(\mathbf{P}_{\varepsilon_{N}}(\mathbf{a})E_{\varepsilon_{N}}(\mathbf{x})\right)\right\|\leq(L+1)\varepsilon_{N}. (34)
sup𝐱,𝐚‖f⁡(𝐱,𝐚)−DεN​(𝐏εN​(𝐚)​EεN​(𝐱))‖\displaystyle\sup_{\mathbf{x},\mathbf{a}}\left\|f(\mathbf{x},\mathbf{a})-D_{\varepsilon_{N}}\!\left(\mathbf{P}_{\varepsilon_{N}}(\mathbf{a})E_{\varepsilon_{N}}(\mathbf{x})\right)\right\| ≤(L+1)d2N−1/d=d2(L+1)N−1/d.\displaystyle\leq(L+1)\frac{\sqrt{d}}{2}N^{-1/d}=\frac{\sqrt{d}}{2}(L+1)N^{-1/d}. (35)
sup𝐱,𝐚‖f(𝐱,𝐚)−DεN(𝐏εN(𝐚)EεN(𝐱))‖=O(N−1/d).\displaystyle\sup_{\mathbf{x},\mathbf{a}}\left\|f(\mathbf{x},\mathbf{a})-D_{\varepsilon_{N}}\!\left(\mathbf{P}_{\varepsilon_{N}}(\mathbf{a})E_{\varepsilon_{N}}(\mathbf{x})\right)\right\|=O(N^{-1/d}). (36)
‖𝐱H−𝐱^H‖≤εN​∑k=0HLk.\displaystyle\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|\leq\varepsilon_{N}\sum_{k=0}^{H}L^{k}. (37)
∥𝐱H−𝐱^H∥≤d2N−1/d∑k=0HLk.\displaystyle\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|\leq\frac{\sqrt{d}}{2}N^{-1/d}\sum_{k=0}^{H}L^{k}. (38)
limN→∞‖𝐱H−𝐱^H‖=0.\displaystyle\lim_{N\rightarrow\infty}\|\mathbf{x}_{H}-\hat{\mathbf{x}}_{H}\|=0. (39)
‖𝐱−𝐜i‖≤d2​h.\|\mathbf{x}-\mathbf{c}_{i}\|\leq\frac{\sqrt{d}}{2}h.

Appendix H Hyperparameter Sweeps

H.1 LpWM and LeWM Sweeps in the PushT Environment

The sweeps vary feature width, regularization weight and learning rate for both sparse and dense targets. Both use sliced-Wasserstein matching here, rather than using the published Gaussian baseline’s different distribution test. Training uses one seed; planning is averaged over three seeds. Later grids omit settings found unsuccessful earlier, so the sweep coverage is not identical for every predictor.

Equation
∏i=1dReLU⁡(Laplace⁡(μ=0,σ=1/2))\prod_{i=1}^{d}\operatorname{ReLU}(\operatorname{Laplace}(\mu=0,\sigma=\sqrt{1/2}))

H.2 VICReg Sweep in the PushT Environment

The extra sweeps replace Gaussian distribution matching with VICReg while keeping the world-model setup. “VICReg” names that dense baseline variant in these plots, not a separate general-purpose world-model architecture.

Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 6: Deep-AdaLN(k)(k) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 7: Shallow-AdaLN(k)(k) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 8: MLP∘\circLTV(k)(k) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 9: MLP∘\circLTI(k)(k) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 10: LTI(1)(1) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 11: LTI(k)(k) Predictor. Hyperparameter Sweeps for the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 12: Deep-AdaLN(k)(k) Predictor. Hyperparameter Sweeps for VICReg-based JEPA models in the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 13: MLP∘\circLTV(k)(k) Predictor. Hyperparameter Sweeps for VICReg-based JEPA models in the PushT Environment.
Refer to caption
(a) Open-Loop.
Refer to caption
(b) Closed-Loop.
Figure 14: MLP∘\circLTI(k)(k) Predictor. Hyperparameter Sweeps for VICReg-based JEPA models in the PushT Environment.
Concept demonstration

Open image ↗