Literature
Display
← All papers

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Lucas Maes · Quentin Le Lidec · Damien Scieur · Yann LeCun · Randall Balestriero

· arXiv (preprint) ↗

World models8 min text · CondensedOriginal paper ↗
Contents

Overview

The training and planning diagrams distinguish the learned encoder/predictor from the cross-entropy optimizer used only at deployment.

Website Code
Refer to caption
Figure 1: LeWorldModel Training Pipeline. Given frame observations 𝒐1:T{\bm{o}}_{1:T} and actions 𝒂1:T{\bm{a}}_{1:T}, the encoder maps frames into low-dimensional latent representations 𝒛1:T{\bm{z}}_{1:T}. The predictor models the environment dynamics by autoregressively predicting the next latent state 𝒛t+1{\bm{z}}_{t+1} from the current latent state 𝒛t{\bm{z}}_{t} and action 𝒂t{\bm{a}}_{t}. The encoder and predictor are jointly optimized using a mean-squared error (MSE) prediction loss. LeWM does not rely on any training heuristics, such as stop-gradient, exponential moving averages, or pre-trained representations. To prevent trivial collapse, the SIGReg regularization term enforces Gaussian-distributed latent embeddings, promoting feature diversity. For tractability, latent embeddings are projected onto multiple random directions, and a normality test is applied to each one-dimensional projection. Aggregating these statistics encourages the full embedding distribution to match an isotropic Gaussian.

At a glance

Intuition. A compact world model should remember enough about a scene to predict what an action will change, while ignoring unnecessary image detail. Predicting its own learned representations creates a trap: every image could map to the same constant. LeWorldModel pairs next-latent prediction with SIGReg, which encourages a diverse, approximately isotropic Gaussian representation distribution. It trains the visual encoder and action-conditioned predictor together from pixels, without rewards, pixel reconstruction, a frozen foundation encoder, or an EMA teacher. A roughly 15-million-parameter model supports rapid simulated control and physical-variable probing. Its reported 48-fold planning speedup is tied to the chosen foundation-model baseline and planning settings; it is not a universal speedup or a guarantee that all task-relevant physical information survives.

1 Introduction

The work simplifies end-to-end JEPA world-model training to a prediction term and one anti-collapse regularizer. Compact global state vectors reduce the cost of repeatedly imagining candidate actions. The contribution combines a practical training recipe with control, probing and surprise evaluations, rather than claiming that low prediction loss alone proves physical understanding.

Refer to caption
Figure 2: Characteristics of latent world model approaches. Methods are grouped by training paradigm. End-to-end methods (PLDM) learn the encoder and predictor jointly from pixels without pre-trained representations or heuristics like stop-gradient or EMAs, but require many hyperparameters and lack collapse guarantees. Foundation-based methods (DINO-WM) avoid collapse by freezing a pre-trained vision encoder, forgoing end-to-end learning. Task-specific methods (Dreamer, TD-MPC) require reward signals or privileged state access. LeWM combines the strengths of all three: end-to-end, task-agnostic, pixel-based, reconstruction- and reward-free, with a single hyperparameter with provable anti-collapse guarantees.

2 Related Work

The paper distinguishes generative simulators, frozen-feature latent models, and end-to-end latent models. Generative approaches reconstruct more visual detail and may use rewards; frozen encoders inherit a representation learned elsewhere. LeWorldModel instead learns a reward-free control representation jointly with its dynamics predictor, accepting the resulting collapse-prevention problem.

Original figure
Original figure
Original figure
Figure 3: Planning time and performance under fixed compute. Left: Planning time comparison averaged over 50 runs. Encoding observations with ∼200×\sim 200\times fewer tokens than DINO-WM allows LeWM to achieve planning speeds comparable to PLDM while being up to ∼50×\sim 50\times faster than DINO-WM. Center–Right: Planning performance under the same computational budget (fixed FLOPs). LeWM significantly outperforms DINO-WM on Push-T (center) and OGBench-Cube (right). See App. D for planning setup details.
Refer to caption
Figure 4: Latent Planning with LeWorldModel . Given an initial observation 𝒐1{\bm{o}}_{1} and a goal 𝒐g{\bm{o}}_{g}, the world model learned in Fig. 2 performs planning in the LeWM latent space. The initial state embedding 𝒛1{\bm{z}}_{1} and the goal embedding 𝒛g{\bm{z}}_{g} are obtained from the encoder. The predictor then rolls out future latent states up to a horizon HH. A latent cost between the final predicted state and the goal embedding guides a solver to optimize the action sequence. This prediction–optimization loop is repeated until convergence to a good plan candidate.

3 Method: LeWorldModel

Training learns the encoder and predictor from offline action–observation trajectories. Deployment freezes them and searches for actions that bring the predicted latent state near an encoded goal.

3.1 Learning the Latent World Model

Offline Dataset.

The data contains observations and actions but no task rewards or optimality labels. Coverage of relevant transitions still matters: reward-free training does not remove the need for informative interaction data.

Model Architecture.

A small vision transformer produces a projected global state vector, and a causal transformer predicts its successor from recent states and actions. Adaptive layer normalization injects actions into the predictor. Projection after normalization is important for allowing the Gaussian-distribution objective to operate on the learned representation.

Equation LeWM
Encoder:\displaystyle\text{Encoder:} 𝒛t=encθ​(𝒐t)\displaystyle{\displaystyle\bm{z}}_{t}={\rm enc}_{\theta}({\bm{o}}_{t}) (LeWM)
Predictor:\displaystyle\text{Predictor:} 𝒛^t+1=predϕ​(𝒛t,𝒂t)\displaystyle\hat{{\bm{z}}}_{t+1}={\rm pred}_{\phi}({\bm{z}}_{t},{\bm{a}}_{t})

Training Objective.

Teacher-forced latent prediction encourages temporal predictability; SIGReg discourages a constant representation by testing random one-dimensional projections against a Gaussian reference. A single weight balances these terms. Finite random projections and empirical optimization approximate the distributional target; they do not prove exact Gaussianity or preservation of every physical factor.

Equations 1–3
ℒpred≜‖𝒛^t+1−𝒛t+1‖22,𝒛^t+1=predϕ​(𝒛t,𝒂t).\mathcal{L}_{\rm pred}\triangleq\|\hat{{\bm{z}}}_{t+1}-{{\bm{z}}}_{t+1}\|^{2}_{2},\quad\quad\hat{{\bm{z}}}_{t+1}={\rm pred}_{\phi}({\bm{z}}_{t},{\bm{a}}_{t}). (1)
SIGReg⁡(𝒁)≜1M​∑m=1MT⁡(𝒉(m)).{\rm SIGReg}({\bm{Z}})\triangleq\frac{1}{M}\sum_{m=1}^{M}T({\bm{h}}^{(m)}). (2)
ℒLeWM≜ℒpred+λ​SIGReg​(𝒁).{\mathcal{L}}_{\rm LeWM}\triangleq{\mathcal{L}}_{\rm pred}+\lambda\,{\rm SIGReg}({\bm{Z}}). (3)

3.2 Latent Planning

Cross-entropy search optimizes candidate action sequences under a terminal latent-goal cost. Only an initial action segment is executed before replanning from a new observation. This receding horizon limits the practical effect of accumulated model error without eliminating it.

Equations 4–5
𝒛^t+1=predϕ​(𝒛^t,𝒂t),𝒛^1=encθ​(𝒐1),\hat{{\bm{z}}}_{t+1}={\rm pred}_{\phi}(\hat{{\bm{z}}}_{t},{\bm{a}}_{t}),\quad\hat{{\bm{z}}}_{1}={\rm enc}_{\theta}({\bm{o}}_{1}),
𝒞⁡(𝒛^H)=‖𝒛^H−𝒛g‖22,𝒛g=encθ​(𝒐g),{\mathcal{C}}(\hat{{\bm{z}}}_{H})=\|\hat{{\bm{z}}}_{H}-{\bm{z}}_{g}\|_{2}^{2},\quad{\bm{z}}_{g}={\rm enc}_{\theta}({\bm{o}}_{g}), (4)
𝒂1:H∗=argmin𝒂1:H𝒞(𝒛^H),{\bm{a}}^{*}_{1:H}=\arg\min_{{\bm{a}}_{1:H}}{\mathcal{C}}(\hat{{\bm{z}}}_{H}), (5)

4 Latent Planning Performance

4.1 Planning evaluation setup

Refer to caption
Figure 5: Environments used for evaluation. Left: Push-T, a 2D manipulation task where the agent must push a block toward a target configuration, commonly used as a robotics benchmark. Center (1): OGBench-Cube, a visually richer 3D manipulation environment where a robotic arm interacts with a cube to reach a target position. Center (2): Two-Room, a simple 2D navigation environment where an agent moves between rooms to reach target positions. Right: Reacher, a task where a 2-joint arm needs to reach a target configuration in a 2D plane. All environments have a continuous action space. More details on environment and datasets are available in appendix E.

Environments.

The simulations span navigation, arm motion and contact-rich manipulation in two and three dimensions. They test controlled settings with offline data, not open-world real-robot generalization.

Baselines.

PLDM is the closest end-to-end latent baseline; DINO-WM uses a pretrained visual encoder and many patch features. Goal-conditioned behavioral cloning and offline value-learning baselines supply policy-based comparisons. Proprioception is controlled where specified because extra state information changes the problem.

4.2 Towards Efficient Planning with WMs

LeWorldModel improves several difficult tasks over PLDM and is competitive with DINO-WM while planning much faster in the reported configuration. It is worse on Two-Room. The suggested explanation—low dataset complexity conflicting with a high-dimensional Gaussian target—is plausible but not isolated as a universal cause.

4.3 Towards Stable Training of World Models

Ablations.

Projection count, representation dimension and architecture are ablated. Performance often saturates across a useful range, while the regularization weight remains consequential. Fewer tuned loss coefficients simplify experimentation; the paper’s search-complexity comparison should not be read as a general theorem that tuning one parameter is logarithmic.

Training Curves.

The displayed training curves are smoother for this two-term recipe than the compared multi-term PLDM run. Smooth optimization traces support the empirical stability claim for these settings, not a guarantee across arbitrary datasets and seeds.

Original figure
Original figure
Original figure
Original figure
Figure 6: Planning performance across environments. Results are shown for Two-Room (left), Reacher (center 1), PushT (center-2) and OGBench-Cube (right). LeWM consistently outperforms PLDM and DINO-WM on Push-T and Reacher. On OGBench-Cube, DINO-WM slightly outperforms LeWM, possibly due to the higher visual complexity and the 3D nature of the environment, which makes encoder training more challenging. In the simpler Two-Room environment, PLDM and DINO-WM outperform LeWM, which may be explained by the SIGReg regularization encouraging a Gaussian distribution in a high-dimensional latent space, while the intrinsic dimensionality of the environment is much lower.

5 Quantifying Physical Understanding in LeWM

Three diagnostics probe what the latent model retains: predicting physical variables, decoding visualizations, and detecting unexpected changes. They measure different properties and do not independently establish an identified physical state model.

5.1 Physical Structure of the Latent Space

Probing physical quantities.

Supervised linear and nonlinear probes predict physical quantities from frozen latents. Competitive probing means the information is recoverable by those probes; it does not prove that the planning procedure actually uses it.

Table 1: Physical latent probing results on Push-T. LeWM consistently outperforms PLDM while remaining competitive with DINO-WM. The strong probing performance of DINO-WM on certain properties may stem from its foundation-model pretraining: the DINOv2 encoder is trained on two orders of magnitude more data (∼\sim124M images) spanning a far more diverse distribution, which likely allows it to capture some physical properties in its embeddings by default.
Linear MLP
Property Model MSE ↓\downarrow r ↑\uparrow MSE ↓\downarrow r ↑\uparrow
Agent Location DINO-WM 1.888±0.5001.888\pm 0.500 0.977\mathbf{0.977} 0.003±0.022\mathbf{0.003\pm 0.022} 0.999\mathbf{0.999}
PLDM 0.090±0.3110.090\pm 0.311 0.9550.955 0.014±0.1190.014\pm 0.119 0.9930.993
LeWM 0.052±0.149\mathbf{0.052\pm 0.149} 0.9740.974 0.004±0.0560.004\pm 0.056 0.9980.998
Block Location DINO-WM 0.006±0.007\mathbf{0.006\pm 0.007} 0.997\mathbf{0.997} 0.002±0.0030.002\pm 0.003 0.999\mathbf{0.999}
PLDM 0.122±0.3410.122\pm 0.341 0.9380.938 0.011±0.0660.011\pm 0.066 0.9940.994
LeWM 0.029±0.0730.029\pm 0.073 0.9860.986 0.001±0.006\mathbf{0.001\pm 0.006} 0.999\mathbf{0.999}
Block Angle DINO-WM 0.050±0.101\mathbf{0.050\pm 0.101} 0.979\mathbf{0.979} 0.009±0.052\mathbf{0.009\pm 0.052} 0.995\mathbf{0.995}
PLDM 0.446±0.6250.446\pm 0.625 0.7450.745 0.056±0.1840.056\pm 0.184 0.9720.972
LeWM 0.187±0.3590.187\pm 0.359 0.9020.902 0.021±0.1390.021\pm 0.139 0.9900.990
Refer to caption
Figure 7: Predictor rollout on OGBench-Cube. We visualize decoded latent plans from LeWM given context and action sequences. Each rollout encodes three image observations as context, then autoregressively generates future latents conditioned on the actions in an open-loop fashion. Latents are decoded via a decoder trained a posteriori. The imagined rollout suggests that latent representations capture the global scene structure while finer details like end-effector angle are not fully preserved. Additional rollouts for Push-T and OGBench-Cube are provided in Fig. 9.

Decoding Latent Space.

A separate visualization decoder reconstructs observations from frozen representations. Reconstruction is not part of world-model training. The decoded images reveal retained information and prediction errors without changing the control model.

Visualizing Latent Space.

A t-SNE plot visually relates latent neighborhoods to positions in Push-T. This is a qualitative projection, whose appearance cannot by itself establish a globally faithful or Euclidean physical geometry.

Temporal Latent Path Straightening.

Consecutive latent velocity vectors become more aligned during training on the studied trajectories. This temporal straightening is observed without a dedicated smoothness term. It is a descriptive statistic and does not imply globally straight optimal routes around obstacles.

5.2 Violation-of-expectation Framework

Surprise is prediction discrepancy on perturbed trajectories. Physical changes are compared with cosmetic color changes and unperturbed controls. Higher response to the former suggests sensitivity to learned dynamics; the designed perturbations remain a limited test of general physical reasoning.

Original figure
Original figure
Original figure
Figure 8: Violation-of-expectation evaluation across three environments. Each plot shows LeWM surprise along three trajectories: an unperturbed reference, a visually perturbed trajectory with abrupt object color change, and a physically perturbed trajectory where objects teleport to random positions. Teleportation violates physical continuity and produces a pronounced surprise spike, while the unperturbed trajectory stays at a low baseline. The increase is significant for teleportation across all environments (paired t-test, p<0.01p<0.01) but weaker and non-significant for color changes, indicating greater sensitivity to physical than visual perturbations. Environments, left to right: TwoRoom, PushT, OGBench Cube.

6 Conclusion

A small end-to-end representation can be useful for fast latent-space control with a simpler objective than the compared recipe. The paper’s practical evidence combines planning performance and diagnostic probes; broader generalization and representation guarantees remain open.

Limitations & Future Work.

The tested domains are relatively controlled, and low-complexity data can challenge the Gaussian target. Future work includes larger and more diverse settings and extensions beyond the evaluated control tasks.

Appendix A SIGReg

SIGReg evaluates random scalar projections of embeddings using an Epps–Pulley normality statistic. Matching all projections has a distributional justification, but the implementation uses finitely many sampled directions and integration points. The appendix preserves the exact statistic and training pseudocode.

Equations 6–Cramer-Wold
𝒉(m)\displaystyle{\bm{h}}^{(m)} ≜𝒁​𝒖(m),𝒖(m)∈𝕊D−1,\displaystyle\triangleq{\bm{Z}}{\bm{u}}^{(m)},{\bm{u}}^{(m)}\in\mathbb{S}^{D-1}, (6)
SIGReg⁡(𝒁)≜1M​∑m=1MT(m),\displaystyle{\rm SIGReg}({\bm{Z}})\triangleq\frac{1}{M}\sum_{m=1}^{M}T^{(m)}, (SIGReg)
T(m)=∫−∞∞w⁡(t)​|ϕN​(t,𝒉(m))−ϕ0​(t)|2​𝑑t,T^{(m)}=\int_{-\infty}^{\infty}w(t)\left|\phi_{N}(t;{\bm{h}}^{(m)})-\phi_{0}(t)\right|^{2}dt, (EP)
SIGReg⁡(𝒁)→0⇔ℙ𝒁→N⁡(0,𝑰).\displaystyle{\rm SIGReg}({\bm{Z}})\rightarrow 0\iff\mathbb{P}_{\bm{Z}}\rightarrow N(0,{\bm{I}}). (Cramer-Wold)

Appendix B Cross-Entropy Method

Cross-entropy planning iteratively samples actions, scores predicted endpoints, and updates the candidate distribution from elites. Planning hyperparameters and replanning frequency are part of the efficiency/accuracy comparison.

Algorithm 2 Cross-Entropy Method (CEM) for Action Sequence Optimization
1: World model ff, planning horizon HH, number of samples NN, number of elites KK, number of iterations TT
2: Initialize sampling distribution parameters μ0=𝟎\mu_{0}=\mathbf{0}, Σ0=I\Sigma_{0}=I
3: for t=1t=1 to TT do
4:   Sample NN candidate action sequences {a1:H(i)}i=1N∼𝒩(μt−1,Σt−1)\{a_{1:H}^{(i)}\}_{i=1}^{N}\sim\mathcal{N}(\mu_{t-1},\Sigma_{t-1})
5:   for i=1i=1 to NN do
6:    Roll out a1:H(i)a_{1:H}^{(i)} in the world model ff
7:    Compute cost J(i)J^{(i)}
8:   end for
9:   Select the KK sequences with lowest cost (elites)
10:   Update distribution parameters using elite set:
11:   μt←1K∑i∈ℰa1:H(i)\mu_{t}\leftarrow\frac{1}{K}\sum_{i\in\mathcal{E}}a_{1:H}^{(i)}
12:   Σt←Vari∈ℰ(a1:H(i))\Sigma_{t}\leftarrow\text{Var}_{i\in\mathcal{E}}\left(a_{1:H}^{(i)}\right)
13: end for
14: return best action sequence found or first action of μT\mu_{T}

Appendix C Baselines

C.1 DINO-WM

DINO-WM predicts frozen pretrained patch features. Its larger token set increases planning cost and can preserve detail absent from a small global vector, so speed and information content should be compared together.

Equation 7
ℒDINO-WM=1B​T​∑iB∑tT‖𝒛^t+1(i)−𝒛t+1(i)‖22\mathcal{L}_{\text{DINO-WM}}=\frac{1}{BT}\sum_{i}^{B}\sum_{t}^{T}\|\hat{{\bm{z}}}^{(i)}_{t+1}-{\bm{z}}^{(i)}_{t+1}\|_{2}^{2} (7)

C.2 PLDM

PLDM learns perception and dynamics jointly with several variance, covariance, temporal and related regularization terms. Its coefficient search and architecture are specified to make the end-to-end baseline reproducible rather than assuming all such methods share one stability profile.

Equations 8
ℒPLDM=ℒpred+α​ℒvar+β​ℒcov+γ​ℒtime-sim+ζ​ℒtime-var+ν​ℒtime-cov+μ​ℒIDM\mathcal{L}_{\text{PLDM}}=\mathcal{L}_{\text{pred}}+\alpha\mathcal{L}_{\text{var}}+\beta\mathcal{L}_{\text{cov}}+\gamma\mathcal{L}_{\text{time-sim}}+\zeta\mathcal{L}_{\text{time-var}}+\nu\mathcal{L}_{\text{time-cov}}+\mu\mathcal{L}_{\text{IDM}} (8)
ℒpred=1B​T​∑iB∑tT‖𝒛^t+1(i)−𝒛t+1(i)‖22\mathcal{L}_{\text{pred}}=\frac{1}{BT}\sum_{i}^{B}\sum_{t}^{T}\|\hat{{\bm{z}}}^{(i)}_{t+1}-{\bm{z}}^{(i)}_{t+1}\|_{2}^{2}
ℒvar=1T​D∑tT∑dDmax(0,1−Var(𝒛(:)t,d)+ϵ)\mathcal{L}_{\text{var}}=\frac{1}{TD}\sum_{t}^{T}\sum_{d}^{D}\max\left(0,1-\sqrt{\text{Var}({\bm{z}}^{(:)}_{t,d})}+\epsilon\right)
ℒcov=1T​∑tT1D​∑i≠jD[Cov​(𝒁t)]i​j\mathcal{L}_{\text{cov}}=\frac{1}{T}\sum_{t}^{T}\frac{1}{D}\sum_{i\neq j}^{D}\left[\text{Cov}({\bm{Z}}_{t})\right]_{ij}
ℒtime-sim=1B​T​∑iB∑tT‖𝒛t(i)−𝒛t+1(i)‖22\mathcal{L}_{\text{time-sim}}=\frac{1}{BT}\sum_{i}^{B}\sum_{t}^{T}\|{\bm{z}}^{(i)}_{t}-{\bm{z}}^{(i)}_{t+1}\|_{2}^{2}
ℒtime-var=1B​D∑iB∑dDmax(0,1−Var(𝒛(i):,d)+ϵ)\mathcal{L}_{\text{time-var}}=\frac{1}{BD}\sum_{i}^{B}\sum_{d}^{D}\max\left(0,1-\sqrt{\text{Var}({\bm{z}}^{(i)}_{:,d})}+\epsilon\right)
ℒtime-cov=1B​∑bB1D​∑i≠jD[Cov​(𝒁)]i​j\mathcal{L}_{\text{time-cov}}=\frac{1}{B}\sum_{b}^{B}\frac{1}{D}\sum_{i\neq j}^{D}\left[\text{Cov}({\bm{Z}})\right]_{ij}
ℒIDM=1B​T​∑iB∑tT‖𝒂^t(i)−𝒂t(i)‖22\mathcal{L}_{\text{IDM}}=\frac{1}{BT}\sum_{i}^{B}\sum_{t}^{T}\|\hat{{\bm{a}}}^{(i)}_{t}-{\bm{a}}^{(i)}_{t}\|_{2}^{2}
𝒁t=[(𝒛t(1))⊤(𝒛t(B))⊤],{\bm{Z}}_{t}=\begin{bmatrix}({\bm{z}}_{t}^{(1)})^{\top}\\ \vdots\\ ({\bm{z}}_{t}^{(B)})^{\top}\end{bmatrix},
𝒁¯t=𝒁t−1B​𝟏𝟏⊤​𝒁t.\bar{{\bm{Z}}}_{t}={\bm{Z}}_{t}-\frac{1}{B}\mathbf{1}\mathbf{1}^{\top}{\bm{Z}}_{t}.
Var(𝒛t,d(:))=1B−1∑i=1B(zt,d(i)−1B∑i′=1Bzt,d(i′))2,\mathrm{Var}({\bm{z}}^{(:)}_{t,d})=\frac{1}{B-1}\sum_{i=1}^{B}\left(z^{(i)}_{t,d}-\frac{1}{B}\sum_{i^{\prime}=1}^{B}z^{(i^{\prime})}_{t,d}\right)^{2},
Cov⁡(𝒁t)=1B−1​𝒁¯t⊤​𝒁¯t∈ℝD×D.\mathrm{Cov}({\bm{Z}}_{t})=\frac{1}{B-1}\bar{{\bm{Z}}}_{t}^{\top}\bar{{\bm{Z}}}_{t}\in\mathbb{R}^{D\times D}.
𝒁¯(i)=𝒁(i)−1T​𝟏𝟏⊤​𝒁(i).\bar{{\bm{Z}}}^{(i)}={\bm{Z}}^{(i)}-\frac{1}{T}\mathbf{1}\mathbf{1}^{\top}{\bm{Z}}^{(i)}.
Var(𝒛:,d(i))=1T−1∑t=1T(zt,d(i)−1T∑t′=1Tzt′,d(i))2,\mathrm{Var}({\bm{z}}^{(i)}_{:,d})=\frac{1}{T-1}\sum_{t=1}^{T}\left(z^{(i)}_{t,d}-\frac{1}{T}\sum_{t^{\prime}=1}^{T}z^{(i)}_{t^{\prime},d}\right)^{2},
Cov⁡(𝒁(i))=1T−1​(𝒁¯(i))⊤​𝒁¯(i)∈ℝD×D.\mathrm{Cov}({\bm{Z}}^{(i)})=\frac{1}{T-1}(\bar{{\bm{Z}}}^{(i)})^{\top}\bar{{\bm{Z}}}^{(i)}\in\mathbb{R}^{D\times D}.
Loss coefficient Initial value
α\alpha 18.0
β\beta 12
γ\gamma 0.2
ζ\zeta 0.7
ν\nu 0.0
μ\mu 0.0
Table 2: Best coefficient found from grid search.

C.3 GC-RL

Goal-conditioned offline reinforcement learning learns value information and an action-selection mechanism from fixed trajectories. It is a distinct route to control from optimizing an action sequence through a predictive model.

GCIQL

Goal-conditioned implicit Q-learning uses expectile-based value fitting on offline transitions. Its learned policy extraction and value assumptions differ from the latent endpoint-matching cost used by LeWorldModel.

Equations
ℒQ=𝔼(st,at,st+1,g)∼𝒟​[(Qψ​(st,at,g)−(r⁡(st,g)+γ​mt​Vθ¯​(st+1,g)))2],\mathcal{L}_{Q}=\mathbb{E}_{(s_{t},a_{t},s_{t+1},g)\sim\mathcal{D}}\left[\left(Q_{\psi}(s_{t},a_{t},g)-\left(r(s_{t},g)+\gamma m_{t}V_{\bar{\theta}}(s_{t+1},g)\right)\right)^{2}\right],
ℒV=𝔼(st,at,g)∼𝒟​[Lτ2​(Qψ¯​(st,at,g)−Vθ​(st,g))],\mathcal{L}_{V}=\mathbb{E}_{(s_{t},a_{t},g)\sim\mathcal{D}}\left[L_{\tau}^{2}\left(Q_{\bar{\psi}}(s_{t},a_{t},g)-V_{\theta}(s_{t},g)\right)\right],
Lτ2​(u)=|τ−𝟙​(u<0)|​u2.L_{\tau}^{2}(u)=|\tau-\mathbbm{1}(u<0)|u^{2}.
ℒcritic=ℒQ+ℒV.\mathcal{L}_{\text{critic}}=\mathcal{L}_{Q}+\mathcal{L}_{V}.

GCIVL

The alternative goal-conditioned value-learning baseline changes how the value is estimated. The appendix preserves that objective rather than treating every offline value method as the same comparator.

Equation
ℒV=𝔼(st,st+1,g)∼𝒟​[Lτ2​(r⁡(st,g)+γ​Vθ¯​(st+1,g)−Vθ​(st,g))].\mathcal{L}_{V}=\mathbb{E}_{(s_{t},s_{t+1},g)\sim\mathcal{D}}\left[L_{\tau}^{2}\left(r(s_{t},g)+\gamma V_{\bar{\theta}}(s_{t+1},g)-V_{\theta}(s_{t},g)\right)\right].

Policy extraction.

A policy is extracted from the fitted goal-conditioned value representation. Its deployment avoids repeated world-model rollout but inherits the offline data and value-learning limitations.

Equations
ℒπ=𝔼(st,at,g)∼𝒟​[exp⁡(β​A​(st,at,g))​‖πθ​(st,g)−at‖22],\mathcal{L}_{\pi}=\mathbb{E}_{(s_{t},a_{t},g)\sim\mathcal{D}}\left[\exp\left(\beta A(s_{t},a_{t},g)\right)\|\pi_{\theta}(s_{t},g)-a_{t}\|_{2}^{2}\right],
A⁡(st,at,g)=r⁡(st,g)+γ​V​(st+1,g)−V⁡(st,g),A(s_{t},a_{t},g)=r(s_{t},g)+\gamma V(s_{t+1},g)-V(s_{t},g),

C.4 GCBC

Goal-conditioned behavioral cloning predicts dataset actions conditioned on goals. This baseline directly learns behavior instead of modeling dynamics; its performance depends on how useful the observed trajectories are for the evaluation goals.

Equation
ℒGCBC=𝔼(st,at,g)∼𝒟​[‖πθ​(st,g)−at‖22],\mathcal{L}_{\text{GCBC}}=\mathbb{E}_{(s_{t},a_{t},g)\sim\mathcal{D}}\left[\|\pi_{\theta}(s_{t},g)-a_{t}\|_{2}^{2}\right],

Appendix D Implementation details

Implementation details retain the model architectures, optimization settings and planning configuration needed to distinguish representation compactness from other recipe changes.

Figure 9: Algorithm 9. Pseudo-code for the training procedure of LeWorldModel. Pixel observations are encoded into latent embeddings, and a predictor estimates the dynamics by predicting the next-step embedding conditioned on actions. The model is optimized end-to-end using a next-embedding prediction loss together with a step-wise SIGReg regularization term to prevent representation collapse.
⬇ def LeWorldModel(obs,actions,lambd=0.1): """ obs: (B, T, C, H, W) raw pixels sequence actions: (B, T, A) action sequence lambd: (float) SIGReg loss weight """ emb = encoder(obs) # (B, T, D) next_emb = predictor(emb,actions) #(B, T, D) # -- LeWorldModel training loss # next-embedding prediction loss pred_loss = F.mse_loss(emb[:, 1:] - next_emb[:, :-1]) # step-wise sigreg (anti-collapse) sigreg_loss = mean(SIGReg(emb.transpose(0, 1)) return pred_loss + lambd * sigreg_loss

Encoder Architecture.

The visual encoder uses the specified small transformer and projected global token. This bottleneck is central to fast rollout computation.

Predictor Architecture.

The causal transformer predictor combines recent latent states with action conditioning. It forecasts latent state rather than pixels.

Decoder (Visualization Only).

The separately trained decoder is used only to inspect reconstructions and imagined futures. It contributes no reconstruction loss to the learned dynamics representation.

Planning solver.

Candidate counts, elite selection, horizon and executed-action chunk specify the cross-entropy solver. Changing them changes both runtime and control success.

Appendix E Environment & Dataset

Environment descriptions explain dataset generation, action spaces and observations for Two-Room, Reacher, Push-T and OGBench Cube. Dataset diversity and complexity differ across tasks, limiting a single interpretation of all gains and failures.

Appendix F Evaluation Details

Refer to caption
Refer to caption
Refer to caption
Figure 9: Additional predictor rollouts on PushT (top) and OGBench-Cube (bottom). Same setup as Fig. 7: three context frames are encoded into latent representations, and the predictor autoregressively generates future latent states conditioned on the action sequence. All predictions are decoded using a decoder not used during training. On PushT, the imagined trajectory closely tracks the real one, accurately capturing both agent and block motion. On OGBench-Cube, the model preserves the overall scene layout and cube displacement but loses finer details such as end-effector orientation at longer horizons, consistent with the lower probing accuracy on rotational quantities reported in Tab. 4.

F.1 Control

Control evaluation fixes the initial–goal tasks and success definitions, with matched planning settings where specified. A task-specific success rate is more informative about control than prediction error alone.

F.2 Probing

Probe data and supervised targets are separated from representation training. The retained tables distinguish positional, velocity and other physical quantities rather than reducing them to one aggregate score.

Refer to caption
Figure 10: Decoder visualization during training. As training progresses, the latent representation increasingly captures the information required to reconstruct the visual scene, even though no reconstruction loss is used during training. Early in training, the decoded images correspond to slow features, a phenomenon previously reported [46].
Table 3: Physical Latent Probing results on TwoRoom. Although LeWM underperforms PLDM in downstream planning on this environment, it matches or outperforms PLDM across all probing metrics, and both methods substantially outperform DINO-WM on the linear probe. This suggests that the learned latent space captures the underlying physical state equally well and that the planning gap is not due to a less informative representation but rather to other factors such as the dynamics model or the planning procedure itself.
Agent Position
Linear MLP
Model MSE ↓\downarrow r ↑\uparrow MSE ↓\downarrow r ↑\uparrow
DINO-WM 0.488±0.4510.488\pm 0.451 0.8240.824 0.000±0.0000.000\pm 0.000 0.9990.999
PLDM 0.008±0.0410.008\pm 0.041 0.9960.996 0.000±0.0000.000\pm 0.000 1.0001.000
LeWM 0.008±0.0180.008\pm 0.018 0.9960.996 0.000±0.0000.000\pm 0.000 1.0001.000
Table 4: Physical latent probing results on OGBench-Cube. LeWM matches or outperforms PLDM on most properties and achieves the best results on positional quantities such as block position and end-effector position. DINO-WM retains a clear advantage on dynamic and rotational properties (joint velocity, end-effector yaw), likely because such quantities benefit from the richer visual priors learned during large-scale pretraining. All three methods struggle to recover block orientation (quaternion and yaw), suggesting that fine-grained rotational information remains difficult to encode in compact latent spaces regardless of the training strategy.
Linear MLP
Property Model MSE ↓\downarrow r ↑\uparrow MSE ↓\downarrow r ↑\uparrow
Joint Position DINO-WM 0.960±1.1500.960\pm 1.150 0.8080.808 0.200±0.9670.200\pm 0.967 0.8700.870
PLDM 0.372±1.1720.372\pm 1.172 0.6950.695 0.340±1.1640.340\pm 1.164 0.7280.728
LeWM 0.352±1.1730.352\pm 1.173 0.7060.706 0.330±1.1570.330\pm 1.157 0.7420.742
Joint Velocity DINO-WM 0.792±0.7480.792\pm 0.748 0.7630.763 0.263±0.6830.263\pm 0.683 0.8520.852
PLDM 1.016±0.9051.016\pm 0.905 0.1150.115 0.661±0.8300.661\pm 0.830 0.5360.536
LeWM 1.021±0.9021.021\pm 0.902 0.0950.095 0.818±0.8990.818\pm 0.899 0.3860.386
End-Effector Position DINO-WM 0.024±0.0100.024\pm 0.010 0.9960.996 0.004±0.0030.004\pm 0.003 0.9990.999
PLDM 0.052±0.0730.052\pm 0.073 0.9740.974 0.013±0.0290.013\pm 0.029 0.9930.993
LeWM 0.018±0.0250.018\pm 0.025 0.9910.991 0.003±0.0040.003\pm 0.004 0.9980.998
End-Effector Yaw DINO-WM 3.317±1.0163.317\pm 1.016 0.8280.828 0.167±0.1680.167\pm 0.168 0.9170.917
PLDM 0.996±0.1650.996\pm 0.165 0.0560.056 0.985±0.2070.985\pm 0.207 0.1170.117
LeWM 0.980±0.2950.980\pm 0.295 0.1240.124 0.952±0.3690.952\pm 0.369 0.2130.213
Gripper DINO-WM 0.114±0.0950.114\pm 0.095 0.9430.943 0.038±0.0600.038\pm 0.060 0.9820.982
PLDM 0.234±0.1690.234\pm 0.169 0.8760.876 0.066±0.1110.066\pm 0.111 0.9670.967
LeWM 0.121±0.1110.121\pm 0.111 0.9380.938 0.048±0.0790.048\pm 0.079 0.9760.976
Block Position DINO-WM 0.085±0.0290.085\pm 0.029 0.9910.991 0.007±0.0070.007\pm 0.007 0.9980.998
PLDM 0.031±0.0230.031\pm 0.023 0.9850.985 0.003±0.0040.003\pm 0.004 0.9990.999
LeWM 0.007±0.0100.007\pm 0.010 0.9970.997 0.002±0.0030.002\pm 0.003 0.9990.999
Block Quaternion DINO-WM 1.596±10.4571.596\pm 10.457 0.2570.257 0.769±8.0460.769\pm 8.046 0.4110.411
PLDM 1.021±12.6001.021\pm 12.600 0.0660.066 0.989±12.1400.989\pm 12.140 0.2180.218
LeWM 1.019±12.5961.019\pm 12.596 0.0870.087 0.963±11.4500.963\pm 11.450 0.2240.224
Block Yaw DINO-WM 4.223±2.5304.223\pm 2.530 0.1760.176 0.916±0.2780.916\pm 0.278 0.3040.304
PLDM 0.996±0.0880.996\pm 0.088 0.0610.061 0.989±0.1400.989\pm 0.140 0.1060.106
LeWM 0.996±0.0940.996\pm 0.094 0.0620.062 0.973±0.1990.973\pm 0.199 0.1640.164
Overall DINO-WM 1.162±1.5791.162\pm 1.579 0.7250.725 0.290±1.2020.290\pm 1.202 0.7990.799
PLDM 0.611±1.8750.611\pm 1.875 0.4640.464 0.503±1.8090.503\pm 1.809 0.6000.600
LeWM 0.592±1.8740.592\pm 1.874 0.4770.477 0.525±1.7140.525\pm 1.714 0.5840.584

F.3 Violation-of-expectation

The surprise protocol inserts controlled visual or physical changes into otherwise comparable trajectories. The examples and discrepancy curves show both the timing and type of detected perturbation.

TwoRoom.

Two-Room perturbations test whether the model responds differently to cosmetic changes and changes affecting navigation dynamics. The environment’s simplicity also contributes to its special behavior elsewhere in the study.

PushT.

Push-T perturbs the manipulation scene and compares surprise against unperturbed motion. These tests target the learned relation between action and object motion.

OGBench-Cube.

Cube experiments extend the same logic to three-dimensional manipulation. Their controlled violations do not cover every possible visual distraction or physical change.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 11: Example of trajectories used for the Violation of Expectation experiments (Sec. 5.2). For each environment, the first row corresponds to the unperturbed trajectory, the second row corresponds to a trajectory where a visual perturbation occurs and the third row displays trajectories where the state of the system is randomly reset in the middle of the trajectory. The frame where the perturbation occurs is highlighted in red.
Original figure
Original figure
Original figure
Figure 12: Violation-of-expectation evaluation with PLDM. From left to right: TwoRoom, PushT, and OGBench-Cube. Surprise is plotted over time for unperturbed, visually perturbed, and physically perturbed trajectories. In TwoRoom and PushT, the model assigns significantly higher surprise to both visual and physical perturbations. In OGBench-Cube, the increase in surprise is weaker and not consistently significant.
Original figure
Figure 13: Visualization of the latent space obtained with LeWM for the PushT environment. On the left, the grid of states is obtained by moving the agent and the block in the x-y plane. On the right, the embeddings of these states are visualized using a t-SNE.
Original figure
Original figure
Original figure
Figure 14: Violation-of-expectation evaluation with DINO-WM. From left to right: TwoRoom, PushT, and OGBench-Cube. Surprise is plotted over time for unperturbed, visually perturbed, and physically perturbed trajectories. While the model detects both perturbations in TwoRoom and PushT, surprise does not increase significantly for either perturbation in OGBench-Cube.

Appendix G Ablations.

Original figure
Original figure
Original figure
Figure 15: Ablation studies of key design choices in LeWM. Left: effect of the embedding dimension; performance improves with larger embeddings but quickly saturates beyond a certain threshold. Center: effect of the number of random projections used in SIGReg; performance remains stable, indicating that this parameter is not critical. Right: effect of the number of integration knots used to compute the SIGReg loss; results are similarly insensitive to this parameter.

Training variance.

Training-seed variation checks whether the reported behavior depends on one initialization. The measured variation is specific to these tasks and budgets.

Table 5: Training Variance. We report the mean success rate across three training seeds and the corresponding variance, evaluated over the same set of 50 trajectories on Push-T. The goal configuration is reachable within 25 steps, and we allow a planning budget of 50 steps. PLDM exhibits higher variance compared to DINO-WM and LeWM.
Model Push-T (SR ↑\uparrow)
DINO-WM 92.0±1.6392.0\pm 1.63
PLDM 78.0±5.078.0\pm 5.0
LeWM (ours) 96.0±2.83\bm{96.0}\pm\bm{2.83}

Embedding dimensions.

Too small a bottleneck loses useful state information; larger dimensions eventually provide diminishing benefit. Capacity therefore remains a real design choice even with a simpler loss.

Number of projections in SIGReg.

The projection-count ablation tests the cost and adequacy of the sampled Gaussian tests. Empirical robustness over tested counts does not make a finite test exact.

Weight of SIGReg regularization.

The balance between prediction and regularization is tuned. Too little regularization risks collapse; too much can prioritize the distributional target over task-relevant predictability.

Original figure
Figure 16: Effect of the SIGReg regularization weight λ\lambda on Push-T planning performance. Success rate remains above 80% across a wide range of values (λ∈[0.01,0.2]\lambda\in[0.01,0.2]), peaking near λ=0.09\lambda=0.09. Performance degrades sharply only at λ=0.5\lambda=0.5, where the regularizer dominates the prediction loss and hinders dynamics modeling. Since λ\lambda is the only effective hyperparameter of LeWM, the SIGReg loss coefficient is easy to tune via a simple bisection search.

Predictor Size.

Predictor capacity affects dynamics quality and planning cost. The retained comparison separates that choice from encoder dimension.

Table 6: Effect of the predictor size on planning performance in the Push-T environment. We report the success rate (SR). The ViT-S predictor achieves the best performance.
pred. size Push-T (SR ↑\uparrow)
tiny 80.67±6.5480.67\pm 6.54
small 96.0±2.83\bm{96.0}\pm\bm{2.83}
base 86.7±3.0686.7\pm 3.06

Decoder.

A decoder is evaluated as an auxiliary design choice. The final claim about a reconstruction-free training recipe should be read against this explicit ablation.

Table 7: Effect of adding a reconstruction loss during training. We report the success rate (SR) on the Push-T planning task. The model trained without the decoder loss achieves higher performance.
Push-T (SR ↑\uparrow)
LeWM w/o decoder loss 96.0±2.83\bm{96.0}\pm\bm{2.83}
LeWM with decoder loss 86.0±7.5486.0\pm 7.54

Architecture.

A residual convolutional encoder supplies an architectural control. Competitive results suggest the recipe is not limited to the default transformer, within the examined settings.

Table 8: Encoder Architecture Effect. We report the success rate (SR) on the Push-T planning task. LeWM achieves competitive performance across encoder architectures, with ViT holding a slight edge.
Push-T (SR ↑\uparrow)
LeWM ViT 96.0±2.83\bm{96.0}\pm\bm{2.83}
LeWM ResNet-18 94.0±3.2794.0\pm 3.27

Predictor Dropout.

Predictor dropout is varied to assess its contribution to generalization and optimization. The selected rate remains a recipe choice, not a mathematical requirement of JEPA.

Table 9: Effect of predictor dropout during training on Push-T planning performance. We report the success rate (SR). A small amount of dropout (p=0.1p=0.1) yields the best results.
pp Push-T (SR ↑\uparrow)
0.00.0 78±6.5478\pm 6.54
0.10.1 96.0±2.83\bm{96.0}\pm\bm{2.83}
0.20.2 85.33±5.7485.33\pm 5.74
0.50.5 66.67±4.1166.67\pm 4.11

Planning Solver.

Planning solvers are compared separately from model training. Better control need not imply that the representation itself changed.

Table 10: Planning Solver Performance. We report the success rate (SR) on the Push-T planning task.
LeWM PLDM
CEM 96.0±2.83\bm{96.0}\pm\bm{2.83} 78.0±5.078.0\pm 5.0
SGD (β1\beta_{1}) 26±4.3226\pm 4.32 4.67±0.064.67\pm 0.06
RMSProp (β2\beta_{2}) 67.33±2.4967.33\pm 2.49 49.33±8.2649.33\pm 8.26
Adam (β1,β2\beta_{1},\beta_{2}) 84±7.1284\pm 7.12 80±3.2780\pm 3.27

Appendix H Temporal Latent Path Straightening.

Temporal straightness is measured by alignment of successive latent differences, with controls against alternative models. The statistic tracks a training-time geometric trend; it is neither a route-length metric nor a theorem about all latent trajectories.

Equation 9
𝒮straight=1B⁡(T−2)​∑i=1B∑t=1T−2⟨𝐯t(i),𝐯t+1(i)⟩‖𝐯t(i)‖​‖𝐯t+1(i)‖.\mathcal{S}_{\text{straight}}=\frac{1}{B(T-2)}\sum_{i=1}^{B}\sum_{t=1}^{T-2}\frac{\langle\mathbf{v}_{t}^{(i)},\,\mathbf{v}_{t+1}^{(i)}\rangle}{\|\mathbf{v}_{t}^{(i)}\|\,\|\mathbf{v}_{t+1}^{(i)}\|}. (9)
Original figure
Figure 17: Temporal Latent Straightening on Push-T. Mean cosine similarity between consecutive latent velocity vectors (Eq. 9) over training. Higher values indicate straighter latent trajectories. PLDM explicitly encourages temporal regularity through a dedicated temporal smoothness loss (ℒtime-sim\mathcal{L}_{\text{time-sim}}), yet LeWM achieves substantially straighter latent paths as a purely emergent phenomenon, without any temporal regularization term in its objective.

Appendix I Training Curves

Complete loss traces retain both prediction and anti-collapse components for the compared models. They support a qualitative stability comparison under the stated runs and hyperparameters.

Original figure
Figure 18: Push-T Training curves for LeWM.
Original figure
Figure 19: Push-T Training curves for PLDM.
Concept demonstration

Open image ↗