Literature
Display
← All papers

Value-guided action planning with JEPA world models

Matthieu Destrade · Oumayma Bounou · Quentin Le Lidec · Jean Ponce · Yann LeCun

· arXiv (preprint) ↗

World models5 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition. Two locations can look close in a picture while a wall makes the route between them long. A useful planning representation should reflect the effort of reaching the goal, rather than visual similarity alone. This paper trains a JEPA encoder so that latent distance approximates the negative of a goal-conditioned value function. An asymmetric quasi-distance can represent situations where going from A to B differs from returning from B to A. The learned dynamics model then supports model-predictive control in this reshaped space. Experiments on wall and maze tasks favor the value-guided objectives, especially the quasi-distance variant, over prediction-only and simple geometric objectives. The evidence is from small simulated settings. A good value fit, global planning reliability, and robustness to stochastic transitions remain separate questions.

1 Introduction

The proposed intervention changes the geometry used to score imagined plans. A prediction loss can learn a useful state representation without making distance-to-goal an easy optimization landscape. Training latent distance to approximate a reaching value function aims to make that distance decrease along useful routes. Planning still requires an action-conditioned dynamics predictor and an optimizer; the encoder does not itself become a policy.

2 Related work

2.1 JEPA world models

A JEPA predicts future encoded states from the current state and actions. Variance/covariance regularization or a slowly updated target encoder can discourage trivial constant representations. At execution time, model-predictive control repeatedly searches for an action sequence whose predicted endpoint resembles the encoded goal, executes an initial portion, and replans. The paper targets the quality of that resemblance measure.

2.2 Learning a value function

A goal-conditioned value estimates the discounted cost of reaching a particular goal. Implicit Q-learning learns this information from offline trajectories using expectile regression. Euclidean distance imposes symmetry, whereas reaching costs can be directional. A quasi-distance relaxes symmetry while retaining useful distance-like structure, motivating the two representation families compared here.

3 Value-guided JEPA for action planning

The encoder is trained to make negative latent distance behave like a reaching value. This supplies the planning optimizer with information about routes and obstacles that plain appearance distance can miss. The mechanism is a learned objective geometry, not an extra terminal-value network attached only at deployment.

3.1 Baseline loss functions

Two simpler alternatives test whether local geometric constraints suffice: contrastive learning pulls consecutive states together relative to unrelated pairs, and regression encourages successive-state distance to equal one. These objectives describe local transitions but do not explicitly solve the discounted goal-reaching value problem.

3.2 IQL for JEPA models

The value is parameterized as negative encoded-state distance. A discounted Bellman-style target and asymmetric squared expectile loss favor favorable transitions supported by the dataset. Goals include trajectory endpoints and randomly sampled batch states. The authors compare Euclidean and learned quasi-metric distances, and either pretrain the encoder before fitting dynamics or optimize representation and prediction together. The discount and expectile coefficients control both long-range information and training stability.

Equation 1
∀((st),(gn))∈𝒟,ℒVFθ​((st),(gn))=∑n=0N∑t=0T−1Lτ2​(−𝟏st≠gn+γ​Vθ¯​(st+1,gn)−Vθ​(st,gn)),\forall((s_{t}),(g_{n}))\in\mathcal{D},\quad\mathcal{L}_{\text{VF}}^{\theta}((s_{t}),(g_{n}))=\sum_{n=0}^{N}\sum_{t=0}^{T-1}L_{\tau}^{2}\Big(-\mathbf{1}_{s_{t}\neq g_{n}}+\gamma V_{\bar{\theta}}(s_{t+1},g_{n})-V_{\theta}(s_{t},g_{n})\Big), (1)

4 Experiments

4.1 Experiment settings

Offline random trajectories supply all training data. In Wall, an agent must reach a point across a wall through its door; small- and large-step datasets differ in exploration. In Maze, velocity commands and inertia make a single image insufficient, so velocity is supplied to the encoder. Evaluation uses maze layouts distinct from the training layouts.

4.2 Planning with the representations

The retained tables compare prediction, geometric, and value-based representation objectives under the same planning procedure. Value-guided approaches generally improve success, and quasi-distance is strongest in the reported settings. Jointly adding prediction or variance/covariance regularization is not automatically beneficial. Larger-step wall data also helps, illustrating that the objective and dataset jointly determine the learned planning geometry.

Name State encoder loss Sep
Contrastive ℒcontrastive\mathcal{L}_{\text{contrastive}} ✓
Regressive ℒregressive\mathcal{L}_{\text{regressive}} & ℒVCReg\mathcal{L}_{\text{VCReg}} ✓
pred_VCReg ℒVCReg\mathcal{L}_{\text{VCReg}} ×\times
pred_EMA EMA procedure ×\times
VF ℒVF\mathcal{L}_{\text{VF}} ✓
Name State encoder loss Sep
VF_pred ℒVF\mathcal{L}_{\text{VF}} ×\times
VF_quasi ℒVF\mathcal{L}_{\text{VF}} & quasi-distance ✓
VF_quasi_pred ℒVF\mathcal{L}_{\text{VF}} & quasi-distance ×\times
VF_VCReg ℒVF\mathcal{L}_{\text{VF}} & ℒVCReg\mathcal{L}_{\text{VCReg}} ✓
VF_VCReg_pred ℒVF\mathcal{L}_{\text{VF}} & ℒVCReg\mathcal{L}_{\text{VCReg}} ×\times
Table 1: Training approaches
Type WS WB Maze
Contrastive 0.49 0.59 0.50
Regressive 0.54 0.57 0.46
pred_VCReg 0.55 0.89 0.54
pred_EMA 0.46 0.43 0.04
VF 0.63 0.94 0.49
Type WS WB Maze
VF_pred 0.55 0.75 0.49
VF_quasi 0.71 0.96 0.63
VF_quasi_pred 0.61 0.85 0.43
VF_VCReg 0.49 0.75 0.39
VF_VCReg_pred 0.47 0.67 0.39
Table 2: Planning results in the different environments

5 Discussion

Distant state–goal relationships are sparsely sampled, and discounted values provide weak gradients far from the goal. Thus local accuracy need not produce a reliable global route map. Hierarchical representations are suggested as one remedy. Trajectory quality also matters: highly suboptimal paths can distort apparent reaching costs, while narrowly expert data can lose coverage. Neither more data nor expert demonstrations alone resolves every tradeoff.

6 Conclusion

Value-guided representation learning is a promising complement to JEPA prediction for these deterministic control tasks. The main result is improved planning success, not a general replacement for predictive learning. Stochastic environments are an important limitation because the implicit-Q-learning objective can be biased there; broader evaluations are left open.

7 Appendix

7.1 Datasets

7.1.1 Wall

Wall observations separate agent and obstacle information into two image channels. The datasets contain 1,000 trajectories of length 64, with half crossing a doorway. Persistent random directions encourage broader movement than independent Gaussian actions. The larger-step dataset explores more space per trajectory, so its better performance should be interpreted together with this coverage difference.

Refer to caption
Crossing trajectory, WS
Refer to caption
Non-crossing trajectory, WB
Figure 1: Examples of trajectories from the wall datasets

7.1.2 Maze

Maze uses randomized connected layouts and velocity-controlled motion. Each training set has 1,000 trajectories of length 101 across five layouts, while evaluation uses different layouts and well-separated start–goal pairs. Supplying velocity addresses partial observability from inertia; the evaluation therefore is not vision-only state inference.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 2: Examples of states of the maze environment (the agent is the green point)

7.2 Experiment settings

The implementation uses 512-dimensional state representations, a convolutional residual encoder, and a multilayer-perceptron predictor. The appendix preserves optimization settings and the distinct model-predictive path-integral planning budgets for Wall and Maze. Euclidean and quasi-distance variants use separately selected discount and expectile settings, which matters when attributing differences solely to representation geometry.

7.3 Additional experiments

Increasing the discount or expectile can strengthen distant-goal information until instability reverses the gain. Parameters are chosen on a separate wall dataset. The additional study considers separating predictive and planning representations rather than assuming one space must optimize both objectives equally well. These ablations qualify the main claim: useful planning geometry requires tuning, and remains an empirical approximation.

Original figure
Results for γ\gamma (τ=0.7\tau=0.7)
Original figure
Results for τ\tau (VF: γ=0.98\gamma=0.98, VF_quasi: γ=0.93\gamma=0.93)
Figure 3: Evolution of planning accuracy with respect to hyperparameters
Concept demonstration

Open image ↗