Literature
Display
← All papers

A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures

Basile Terver · Randall Balestriero · Megi Dervishi · David Fan · Quentin Garrido · Tushar Nagarajan · Koustuv Sinha · Wancong Zhang · Mike Rabbat · Yann LeCun · Amir Bar

Preprint posted · ICLR 2026 World Models Workshop (accepted) ↗

Contents

Overview

At a glance

Intuition. Encode an image, predict how its representation changes, then try candidate actions inside that learned model. EB-JEPA packages these steps into small examples that can be trained on one accelerator, so individual design choices are easier to inspect than in a billion-parameter training system.

The library progresses from image views to video dynamics and goal-conditioned navigation. Its headline results are approximately 90–91% linear-probe accuracy on CIFAR-10 and 97% planning success in a particular Two Rooms setup. These demonstrate the included examples, not a general-purpose robot world model.

1 Introduction

The contribution is an accessible implementation and controlled experiments. Image, video and action-conditioned variants share modular encoders, predictors, regularizers and planners. Small examples make it possible to change one component and inspect the result before attempting a larger system. The paper’s representation-learning motivation does not make pixel reconstruction categorically useless; the experiments test particular objectives and tasks.

Original figure
Figure 1: EB-JEPA is a modular code base and tutorial, providing self-contained implementations of Joint-Embedding Predictive Architecture for (a) self-supervised image representation learning, (b) video prediction in latent space, and (c) action-conditioned world models that enable goal-directed planning (d).

2 Related Work

Joint-Embedding methods.

The library focuses on JEPAs that prevent collapse through explicit regularization, rather than representing every JEPA design. VICReg and SIGReg are implemented examples. Teacher/stop-gradient approaches and pixel-prediction methods form related alternatives, not interchangeable implementations of the same loss.

World models for planning.

Related latent world models learn dynamics for planning or reinforcement learning. EB-JEPA aims at readable, self-contained experiments rather than state-of-the-art results on complicated benchmarks.

3 Preliminaries: A Unified JEPA Framework

An energy is a compatibility score: an observed context and a plausible target should receive a low value. Here it is computed from error between a predicted embedding and an encoded target. A regularizer stops the encoder from making every target the same easy-to-predict vector. For image views, the predictor can be identity. Video introduces temporal prediction. Action-conditioned video additionally encodes the control input, allowing different actions from the same state to lead to different predicted states.

Equations 1–4
ℒ=ℒpred​(gϕ​(z,u),z′)+λ​ℛ​(z),\mathcal{L}=\mathcal{L}_{\text{pred}}(g_{\phi}(z,u),z^{\prime})+\lambda\mathcal{R}(z), (1)
ℒimage=‖z−z′‖22+λ​ℛ​(z,z′).\mathcal{L}_{\text{image}}=\|z-z^{\prime}\|_{2}^{2}+\lambda\mathcal{R}(z,z^{\prime}). (2)
ℒvideo=∑t=1T−1∥gϕ(zt−v:t)−zt+1∥22+λℛ(z1:T).\mathcal{L}_{\text{video}}=\sum_{t=1}^{T-1}\|g_{\phi}(z_{t-v:t})-z_{t+1}\|_{2}^{2}+\lambda\mathcal{R}(z_{1:T}). (3)
ℒworld=∑t=1T−1∥gϕ(zt−v:t,ut−v:t)−zt+1∥22+λℛ(z1:T,u1:T).\mathcal{L}_{\text{world}}=\sum_{t=1}^{T-1}\|g_{\phi}(z_{t-v:t},u_{t-v:t})-z_{t+1}\|_{2}^{2}+\lambda\mathcal{R}(z_{1:T},u_{1:T}). (4)

A unified energy formulation.

A common expression combines an encoder, a predictor and an optional conditioning encoder. Removing action conditioning gives a video predictor; removing temporal prediction gives an image-view example. The shared equation is a software and conceptual interface, while the data and regularization needs differ.

Equation 5
E⁡(x,x′,a)=ℒpred​(gϕ​(fθ​(x),qω​(a)),fθ​(x′)).E(x,x^{\prime},a)=\mathcal{L}_{\text{pred}}(g_{\phi}(f_{\theta}(x),q_{\omega}(a)),f_{\theta}(x^{\prime})). (5)

Regularization: Preventing Collapse.

VICReg encourages variation across examples and reduces covariance between coordinates. SIGReg instead tests whether randomly projected embeddings resemble an isotropic Gaussian distribution. Image and video examples may apply these losses after a learned projection head. A spread-out embedding avoids a constant-vector shortcut, but useful dynamics can still require additional constraints; Gaussianity alone does not guarantee a suitable representation for every task.

Equations 6–8
ℒvar(Z)=1D∑j=1Dmax(0,γ−Var(Z:,j)+ϵ),\mathcal{L}_{\text{var}}(Z)=\frac{1}{D}\sum_{j=1}^{D}\max\left(0,\gamma-\sqrt{\text{Var}(Z_{:,j})+\epsilon}\right), (6)
ℒcov​(Z)=1D⁡(D−1)​∑i≠j[C⁡(Z)]i,j2,C⁡(Z)=1N−1​(Z−Z¯)⊤​(Z−Z¯).\mathcal{L}_{\text{cov}}(Z)=\frac{1}{D(D-1)}\sum_{i\neq j}[C(Z)]^{2}_{i,j},\quad C(Z)=\frac{1}{N-1}(Z-\bar{Z})^{\top}(Z-\bar{Z}). (7)
ℛSIGReg​(Z)=1P​∑p=1P𝒢⁡(Z​ξp),\mathcal{R}_{\text{SIGReg}}(Z)=\frac{1}{P}\sum_{p=1}^{P}\mathcal{G}(Z\xi_{p}), (8)
Original figure
Figure 2: Hyperparameter sensitivity comparison between SIGReg and VICReg on CIFAR-10. SIGReg demonstrates greater stability across different hyperparameter configurations, while VICReg achieves similar peak performance but requires more careful tuning.
Original figure
(a)
Original figure
(b)
Figure 3: Video-JEPA training dynamics and multistep rollout ablation. (a) Training dynamics over 50 epochs: variance-covariance regularization loss ℛ\mathcal{R} (left), prediction loss ℒpred\mathcal{L}_{\text{pred}} (center), and mean Average Precision (right). (b) Training with kk-step recursive predictions achieves significantly better Average Precision compared to single-step predictions, demonstrating improved temporal understanding, with a Pareto optimum around k=4k=4 rollout steps.
Refer to caption
Figure 4: Video JEPA visualization on Moving MNIST. From left to right: input frames, 1-step prediction visualization, and full autoregressive rollout. The model maintains coherent predictions of digit motion over extended horizons, correctly capturing trajectory and dynamics.

4 Training and Planning with World Models

Multistep Rollout Training.

A one-step predictor trained only on freshly encoded true states can fail when repeatedly fed its own predictions. Multistep training includes recursively predicted states in the loss, bringing training closer to rollout-time use. The horizon and treatment of gradients through successive steps are important implementation choices.

Equations 9–11
ℒk=∑t=1T−k∥gϕ(zt−v:t(k−1),ut−v:t)−zt+1∥22,\mathcal{L}_{k}=\sum_{t=1}^{T-k}\|g_{\phi}(z^{(k-1)}_{t-v:t},u_{t-v:t})-z_{t+1}\|_{2}^{2}, (9)
zt+1(k)=gϕ(zt−v:t(k−1),ut−v:t),zt(0)=fθ(xt−w:t).z^{(k)}_{t+1}=g_{\phi}(z^{(k-1)}_{t-v:t},u_{t-v:t}),\quad z^{(0)}_{t}=f_{\theta}(x_{t-w:t}). (10)
ℒvideo=ℒpred+λℛ(z1:T),ℒworld=ℒpred+λℛ(z1:T,u1:T),ℒpred=∑k=1Kℒk.\mathcal{L}_{\text{video}}=\mathcal{L}_{{\text{pred}}}+\lambda\mathcal{R}(z_{1:T}),\quad\mathcal{L}_{\text{world}}=\mathcal{L}_{{\text{pred}}}+\lambda\mathcal{R}(z_{1:T},u_{1:T}),\quad\mathcal{L}_{{\text{pred}}}=\sum_{k=1}^{K}\mathcal{L}_{k}. (11)

Additional Regularizers for World Models.

The action-conditioned example adds temporal smoothness and inverse dynamics, which predicts an action from consecutive states. The latter discourages encoding only a static background that is predictable but unrelated to the controls. These constraints address a different shortcut from total representation collapse.

Equations 12–13
ℒsim=∑t‖zt−zt+1‖22,ℒIDM=∑t‖at−MLP​(zt,zt+1)‖22.\mathcal{L}_{\text{sim}}=\sum_{t}\|z_{t}-z_{t+1}\|_{2}^{2},\quad\mathcal{L}_{\text{IDM}}=\sum_{t}\|a_{t}-\text{MLP}(z_{t},z_{t+1})\|_{2}^{2}. (12)
ℒ=ℒpred+α​ℒvar+β​ℒcov+δ​ℒsim+ω​ℒIDM.\mathcal{L}=\mathcal{L}_{\text{pred}}+\alpha\mathcal{L}_{\text{var}}+\beta\mathcal{L}_{\text{cov}}+\delta\mathcal{L}_{\text{sim}}+\omega\mathcal{L}_{\text{IDM}}. (13)

Goal-Conditioned Planning.

To reach a goal image, the planner samples action sequences, rolls them through the latent predictor and scores their distances from the goal representation. Model Predictive Path Integral planning refines the proposals by favoring lower-cost sequences. This implementation sums costs across the horizon rather than checking only the final predicted state, and replans as new observations arrive.

Equation 14
Eplan(a0:H;x0,xg)=∑t=1H∥fθ(xg)−z^t∥2,where z^t+1=gϕ(z^t−v:t,ut−v:t),z^0=fθ(x0).E_{\text{plan}}(a_{0:H};x_{0},x_{g})=\sum_{t=1}^{H}\|f_{\theta}(x_{g})-\hat{z}_{t}\|_{2},\quad\text{where }\hat{z}_{t+1}=g_{\phi}(\hat{z}_{t-v:t},u_{t-v:t}),\quad\hat{z}_{0}=f_{\theta}(x_{0}). (14)

5 Experiments

Experimental Setup.

The three examples are CIFAR-10 image representation learning, MovingMNIST video prediction and Two Rooms goal-conditioned navigation. The code includes alternative encoders, predictors, regularizers and sampling-based planners. Appendix tables fix the actual training and planning settings for each comparison.

Image Representation Learning.

Linear probes reach roughly 90–91% on CIFAR-10. A learned projection head improves the tested regularizers by about three percentage points, and its preferred dimensions differ between VICReg and SIGReg. The comparison uses a simple hyperparameter search; it illustrates tuning behavior rather than proving one regularizer universally superior.

Table 1: Image-JEPA Linear probing accuracy on CIFAR-10 with ResNet-18 backbone trained for 300 epochs, comparing regularizers (SIGReg and VICReg) and the impact of using a projector.
Best acc. ↑\uparrow Average acc. ↑\uparrow w/o Projector Hyperparams Best projector
SIGReg 91.02% 89.22% -3.3 points 1 2048×\times128
VICReg 90.12% 84.90% -2.9 points 2 2048×\times1024
Table 2: Ablation of Image-JEPA on loss hyperparameters when training on CIFAR-10 with ResNet-18 backbone trained for 300 epochs.
SIGReg VICReg
Rank Hyperparameters Accuracy ↑\uparrow Hyperparameters Accuracy ↑\uparrow
1 λ=10\lambda=10 90.88% std = 1, cov = 100 90.12%
2 λ=1\lambda=1 86.94% std = 1, cov = 10 89.93%
3 λ=100\lambda=100 80.86% std = 10, cov = 10 89.20%
-1 λ=0.1\lambda=0.1 27.20% std = 100, cov = 100 10.00%
Table 3: Image-JEPA ablation of projector design when training on CIFAR-10 with ResNet-18 backbone trained for 300 epochs.
SIGReg VICReg
Rank Dimensions Accuracy ↑\uparrow Dimensions Accuracy ↑\uparrow
1 2048×\times128 91.02% 2048×\times1024 90.12%
2 4096×\times1024 91.00% 4096×\times512 90.10%
3 2048×\times64 90.99% 1024×\times1024 90.05%
4 512×\times256 90.99% 2048×\times512 90.03%
5 4096×\times64 90.96% 4096×\times1024 90.02%
N/A None 87.75% None 87.27%

Video Prediction.

MovingMNIST results favor training on longer recursive prediction horizons over a one-step-only setup. Predicted embeddings are evaluated through a downstream detection probe, with reconstructed clips providing a visual check. The measured improvement is specific to the included dataset and setup.

Table 4: AC-video-JEPA planning ablations on Two Rooms with randomized wall positions. Each result averages over 3 seeds ×\times 3 checkpoints ×\times 20 episodes. Left: Planner configuration comparison. Right: Regularization term ablation; removing IDM causes collapse.
Configuration Success ↑\uparrow Time
MPPI (full cost) 97±297\pm 2% 37s
CEM (full cost) 96±296\pm 2% 37s
MPPI (last state) 89±289\pm 2% 37s
Ablated Term Success ↑\uparrow
None (full model) 97±297\pm 2%
Variance (α=0\alpha=0) 47±347\pm 3%
Covariance (β=0\beta=0) 46±346\pm 3%
Temporal Sim. (δ=0\delta=0) 61±261\pm 2%
IDM (ω=0\omega=0) 1±11\pm 1%
Refer to caption
Refer to caption
Refer to caption
Figure 5: Visualization of three successful planning evaluation episodes of our AC-video-JEPA on the Two Rooms environment with random wall. From left to right: initial frame (red), full episode outputted by the planning optimization procedure, goal frame used to define planning cost (red). Each episode allows a maximum of 200 steps in the environment.

Action-Conditioned Video-JEPA.

In random-wall Two Rooms navigation, the best configuration reaches 97% success. Removing inverse dynamics reduces success to 1%, consistent with the static-shortcut issue. Other regularization ablations also reduce success. Cumulative goal distance beats final-state-only distance in this setup, even though a successful path may initially need to move away from the goal to pass through a doorway.

6 Future Directions

The small-scale examples are intended as a testbed. Scaling an observed improvement to natural videos or robots requires additional evaluation; the library does not establish that transfer automatically.

Advancing Regularization Theory.

Regularization combinations remain an open research question. Controlled ablations can help determine when coordinate diversity, temporal smoothness and inverse dynamics contribute distinct information, and when their weights need retuning.

Hierarchical World Models.

Hierarchical prediction could connect short control steps to longer-term abstractions. This is a proposed future extension here, not an evaluated hierarchy already supplied by these experiments.

Learned Cost and Value Functions.

Distances between learned states may be a poor proxy for useful progress in complicated tasks. Learned costs or values are suggested extensions for directing the planner toward task-specific outcomes.

Complementary to Large-Scale Codebases.

The intended workflow is to test a mechanism in a compact example and then investigate it in infrastructure supporting larger datasets and distributed training. EB-JEPA complements those systems rather than replacing their scaling and evaluation machinery.

7 Conclusion

EB-JEPA connects explicit regularization, latent prediction and planning in one small codebase. Its ablations show why a usable world model needs more than a low prediction loss. Its strongest contribution is an inspectable experimental starting point with stated datasets and settings.

Ethics statement

Experiments use public image/video benchmarks and procedurally generated navigation scenes. The ethics statement identifies no newly collected personal data or human-subject experiments.

Reproducibility statement

Self-contained training scripts, model implementations, evaluation code and hyperparameter tables support reproducing the examples. Two Rooms generation also records seeds; these details matter when comparing the reported success rates.

Acknowledgments

The authors acknowledge discussions and conceptual contributions from Adrien Bardes and Gaoyue Zhou.

Appendix A Hyperparameters

The preserved tables state training hyperparameters, rollout horizons, sequence lengths and planning configurations. Consult these before interpreting an ablation as changing only one factor.

Table 5: Training hyperparameters for image-JEPA examples on CIFAR-10. The “ViT Image-JEPA” column documents hyperparameters for a ViT-based example provided in the codebase, which achieves 87% linear probing accuracy; all image results reported in the main paper (Tables 1, 2, and 3) use the ResNet-18 backbone.
Group Hyperparameter VICReg ViT Image-JEPA SIGReg
Optimization Optimizer LARS AdamW LARS
LR schedule 10-epoch warmup (from 3×10−53\times 10^{-5}) + cosine to 0
Learning rate 0.3 0.001 0.3
Epochs 300 300 300
Batch size 256 512 256
Weight decay 10−410^{-4} 10−410^{-4} 10−410^{-4}
Data Dataset CIFAR-10 CIFAR-10 CIFAR-10
Image size 32×3232\times 32 32×3232\times 32 32×3232\times 32
Architecture Encoder ResNet-18 ViT-S ResNet-18
Predictor Identity Identity Identity
Encoder output dim 512 384 512
Projector hidden dim 2048 2048 2048
Projector output dim 2048 2048 128
Projector layers 3 3 3
Loss Loss type VICReg VICReg SIGReg
Variance coeff. α\alpha 1 1 –
Covariance coeff. β\beta 80 80 –
SIGReg coeff. λ\lambda – – 10
Table 6: Training hyperparameters for video-JEPA examples. KK denotes the number of training rollout steps (multistep prediction), and TT denotes the training trajectory slice length.
Group Hyperparameter Video-JEPA AC-Video-JEPA
Optimization Learning rate 0.001 0.001
Epochs 50 12
Batch size 64 384
Weight decay – 10−510^{-5}
Data Dataset Moving MNIST Two Rooms
Trajectory length TT 10 17
Image size 64×6464\times 64 65×6565\times 65
Architecture Encoder ResNet-5 IMPALA
Predictor ResUNet GRU
Latent dimension dd 16 32
Hidden dimension 32 32
Encoder receptive field ww 1 1
Predictor receptive field vv 2 1
Loss Rollout steps KK 4 8
Variance coeff. α\alpha 10 16
Covariance coeff. β\beta 100 8
Time similarity coeff. δ\delta – 12
IDM coeff. ω\omega – 1
Table 7: Planning hyperparameters for the action-conditioned video-JEPA example using MPPI, corresponding to the notations of Algorithm 1. The total number of replanning steps for an evaluation episode is Mm\frac{M}{m}.
Hyperparameter Symbol Value
Planning horizon HH 90
Number of parallel samples NN 200
Number of iterations JJ 20
Number of elites QQ 20
Noise scale σ\sigma 2
Temperature τ\tau 0.005
Actions stepped per plan mm 1
Max steps per episode MM 200

Appendix B Planning Algorithm

The planner is a practical hybrid: it first selects a subset of low-cost trajectories, then applies exponential cost weights within that subset. Temperature controls concentration. This differs from both weighting every sampled trajectory in a pure MPPI description and uniformly fitting only elites in a basic cross-entropy method. The original algorithm is retained below.

Algorithm 1 Model Predictive Path Integral (MPPI)
1:  Input: Initial observation x0x_{0}, goal observation xgx_{g}, initial mean μ∈ℝH×A\mu\in\mathbb{R}^{H\times A}, noise scale σ\sigma, temperature τ\tau, number of samples NN, number of iterations JJ, number of elites QQ, max steps per episode MM
2:  Encode initial and goal: z^0=fθ​(x0)\hat{z}_{0}=f_{\theta}(x_{0}), zg=fθ​(xg)z_{g}=f_{\theta}(x_{g})
3:  for j=1j=1 to JJ do
4:   Sample NN noise perturbations: ϵi∼𝒩⁡(0,σ2​I)\epsilon_{i}\sim\mathcal{N}(0,\sigma^{2}\textrm{I}) for i=1,…,Ni=1,\ldots,N
5:   Compute candidate action sequences: a(i)0:H−1=μ+ϵia^{(i)}_{0:H-1}=\mu+\epsilon_{i}
6:   Unroll predictor for each trajectory: z^t+1(i)=gϕ(z^t−v:t(i),ut−v:t(i))\hat{z}^{(i)}_{t+1}=g_{\phi}(\hat{z}^{(i)}_{t-v:t},u^{(i)}_{t-v:t}) for t=0,…,H−1t=0,\ldots,H-1
7:   Compute trajectory costs: Si=∑t=1H‖zg−z^t(i)‖2S_{i}=\sum_{t=1}^{H}\|z_{g}-\hat{z}^{(i)}_{t}\|_{2}
8:   Select top QQ elite samples with lowest costs
9:   Compute weights over elites: wi=exp(−Si/τ)∑q=1Qexp(−Sq/τ)w_{i}=\frac{\exp(-S_{i}/\tau)}{\sum_{q=1}^{Q}\exp(-S_{q}/\tau)}
10:   Update mean: μ←∑i=1Qwi⋅a(i)0:H−1\mu\leftarrow\sum_{i=1}^{Q}w_{i}\cdot a^{(i)}_{0:H-1}
11:  end for
12:  Return: Execute first mm actions of μ\mu, then replan from new observation until MM steps reached

Appendix C Extended Related Work

Diffusion-Based Planning.

Diffusion-based planners learn to generate trajectories by iterative denoising; JEPA world models instead support evaluating candidate actions through latent dynamics. The methods can be combined, but their objectives and online computations differ.

Concept demonstration

Open image ↗