Literature
Display
← All papers

AdaJEPA: An Adaptive Latent World Model

Ying Wang · Oumayma Bounou · Yann LeCun · Mengye Ren

Preprint posted · arXiv (preprint) ↗

World models7 min text · CondensedOriginal paper ↗
Contents

At a glance

A planner can keep making poor choices if its internal model stays wrong. AdaJEPA closes that feedback loop: plan with the current world model, execute a short action chunk, observe what actually happened, update the model, and plan again. The update uses prediction error from the new transition, without a new expert demonstration or reward label. In the tested simulated tasks, updating only a few layers for one gradient step often improves goal reaching at small additional cost. Each episode starts from the same pretrained checkpoint. This is lightweight within-episode adaptation, not a guarantee of unlimited continual learning or arbitrary distribution-shift recovery.

1 Introduction

A frozen world model can mispredict the consequences of actions, and planning may amplify those errors. The proposed remedy uses the very transitions produced during control as new self-supervised training examples. It complements offline pretraining by correcting the model locally while the agent acts.

Refer to caption
(a) AdaJEPA: Plan–Act–Adapt–Replan Loop
Refer to caption
(b) World Model for Planning
Figure 1:AdaJEPA performs test-time adaptation during closed-loop MPC. At each MPC step, we plan with the current model, execute the first action ata_{t}, collect observation ot+1o_{t+1} from the environment, and update the model to minimize the prediction error on the newly observed transition {ot,at,ot+1}\{o_{t},a_{t},o_{t+1}\} before replanning. This yields a simple plan–execute–adapt–replan loop that continually recalibrates the model to transitions encountered in the current environment.

2 Related Work

JEPA world models.

JEPA world models predict future features instead of pixels. Goal-conditioned control then searches for actions that bring those features near an encoded goal. Their learned representations, dynamics predictors and anti-collapse objectives can vary.

Test-time training and adaptation.

Test-time adaptation changes a trained model using information available during deployment. AdaJEPA applies this idea to action-conditioned feature prediction, using the observed next state as a target.

Adaptation in planning and control.

Adaptive control and online model-based learning provide earlier versions of the same broad feedback principle. AdaJEPA places small model updates directly inside a pretrained latent model’s planning loop, without a separate target-domain demonstration phase.

3 AdaJEPA: An Adaptive Latent World Model

The method repeats plan, execute, observe, adapt and replan. Updating the dynamics or visual representation changes the model that the next planning problem uses. The original algorithm gives the full episode procedure.

3.1 Background: JEPA World Models

An image encoder describes the state, an action encoder describes the command, and a predictor forecasts the next representation. Offline training minimizes future-feature error with a stabilizer against collapse. Planning rolls out candidate actions and minimizes a goal-distance cost. Prediction quality and the geometry of that cost both affect success.

Equations 1–3
zt=ℰϕs​(ot),ut=ℰψa​(at),z^t+1=fθ​(zt,ut).z_{t}=\mathcal{E}^{s}_{\phi}(o_{t}),\qquad u_{t}=\mathcal{E}^{a}_{\psi}(a_{t}),\qquad\hat{z}_{t+1}=f_{\theta}(z_{t},u_{t}). (1)
ℒpred=1K​∑k=1Kℓ⁡(z^t+k,zt+k),\mathcal{L}_{\rm pred}=\frac{1}{K}\sum_{k=1}^{K}\ell(\hat{z}_{t+k},z_{t+k}), (2)
at:t+H−1∗=argminat:t+H−1∑k=1Hαkd(z^t+k,zg),a^{*}_{t:t+H-1}=\arg\min_{a_{t:t+H-1}}\sum_{k=1}^{H}\alpha_{k}\,d(\hat{z}_{t+k},z_{g}), (3)
𝒟off={(ot,at,ot+1)}\mathcal{D}_{\rm off}=\{(o_{t},a_{t},o_{t+1})\}
zg=ℰϕs​(og)z_{g}=\mathcal{E}^{s}_{\phi}(o_{g})
d⁡(z^,zg)=‖z^−zg‖22d(\hat{z},z_{g})=\|\hat{z}-z_{g}\|_{2}^{2}

3.2 Closed-Loop Plan-and-Adapt

After each executed action chunk, the observed transition supplies a small training update before the next plan. Each episode has its own model copy and replay buffer, reset from the pretrained checkpoint at the start. Adaptation uses interaction data, even though it requires no additional expert answers.

Online buffer.

A bounded buffer keeps recent transitions or transitions with the largest prediction error. Recent data focus the update on the local situation the controller is currently encountering. The buffer is an adaptation memory, separate from the offline training dataset.

Adaptation loss.

The adaptation loss again compares predicted and observed latent targets. The default stops gradients through the target branch, while updating a restricted parameter subset. This is a practical stabilization recipe; the limited-update ablation is evidence about the tested models, not a general proof that online collapse cannot occur.

Equation 4
ℒada​(ℬ)=1|ℬ|​∑(oi,ai,oi+1)∈ℬℓ⁡(fθ​(zi,ℰψa​(ai)),sg⁡(zi+1)),\mathcal{L}_{\rm ada}(\mathcal{B})=\frac{1}{|\mathcal{B}|}\sum_{(o_{i},a_{i},o_{i+1})\in\mathcal{B}}\ell\!\left(f_{\theta}\!\left(z_{i},\mathcal{E}^{a}_{\psi}(a_{i})\right),\operatorname{sg}(z_{i+1})\right), (4)
Algorithm 1 AdaJEPA: Closed-Loop Plan-and-Adapt
1:  Input: pretrained world model (ℰϕs,ℰψa,fθ)(\mathcal{E}^{s}_{\phi},\mathcal{E}^{a}_{\psi},f_{\theta}), trainable parameters Ω\Omega, goal observation ogo_{g}, planning horizon HH, adaptation steps UU, buffer size NN, max steps TT
2:  Initialize buffer ℬ←∅\mathcal{B}\leftarrow\emptyset; observe o0o_{0}
3:  for t=0,1,…,T−1t=0,1,\ldots,T-1 do
4:    Plan with the current model: optimizes an action sequence by minimizing a latent goal-reaching cost (eq (3))
5:    Execute the first action ata_{t} and observe ot+1o_{t+1}
6:    Add (ot,at,ot+1)(o_{t},a_{t},o_{t+1}) to ℬ\mathcal{B} and trim (if necessary) to keep NN transitions at maximum
7:    for u=1,…,Uu=1,\ldots,U do
8:     Ω←Ω−η​∇Ωℒada​(ℬ)\Omega\leftarrow\Omega-\eta\nabla_{\Omega}\mathcal{L}_{\rm ada}(\mathcal{B})
9:    end for
10:  end for
Equation
zi=ℰϕs​(oi)z_{i}=\mathcal{E}^{s}_{\phi}(o_{i})

Adapted parameters.

Only a selected subset of encoder and predictor parameters needs to change. A few gradient updates are then immediately reused for planning. Restricting the update makes adaptation cheaper and reduces how far it can move from the pretrained model.

Equation 5
Ω←Ω−η​∇Ωℒada​(ℬ).\Omega\leftarrow\Omega-\eta\nabla_{\Omega}\mathcal{L}_{\rm ada}(\mathcal{B}). (5)

4 Experiments

Experiments compare adaptive and frozen versions of the same world model under controlled changes in geometry, appearance, physics and maze layout. The evidence concerns these simulated goal-reaching tasks and their planning protocols.

4.1 Setup

Environments.

PushObj changes object shape while retaining the underlying contact setting. PushT tests image corruptions and color changes. PointMaze changes mass, damping or maze layout. Training and test sets separate the specified conditions; these are distinct kinds of shift, not one unrestricted out-of-distribution benchmark.

Plan-and-Adapt.

The default update changes the last visual-encoder and predictor layers once per replan, using five recent transitions and the training learning rates. Results average several test seeds and episodes. The source uses different 20- and 30-replan budgets across experiments; its captions and appendix settings should accompany reproduced scores.

Equation
ηpred=5×10−4\eta_{\text{pred}}=5\times 10^{-4}

Architectures.

The main models combine a small residual encoder, action and proprioceptive inputs (measurements of the agent’s own state), and a transformer predictor. Stop-gradient and a regularizer favoring straighter latent trajectories are part of the pretraining recipe. Additional comparisons apply the adaptation loop to other world-model implementations.

Refer to caption
Refer to caption
Figure 2: Planning Success under Shape Shifts (top) and Visual Shifts (bottom). The ★\bigstar denotes unseen shapes and configurations. AdaJEPA consistently improves planning success across all settings, using only a single adaptation step per MPC replanning step. We extend the maximum number of steps to 30 to show the increasing trend of planning success of AdaJEPA. Comparison between frozen and AdaJEPA planning trajectories is in Appendix C, showing AdaJEPA consistently decreases prediction loss and leads to better planning.

4.2 Results

In-distribution performance.

Adaptation improves several in-distribution cases where the frozen model is imperfect and preserves performance in the strong tested baseline. This supports applying the method in those experiments; it does not establish that adaptation is universally harmless.

Distribution shifts.

The reported shifts generally benefit from adaptation, including new shapes and maze layouts. Some color changes yield only modest gains because color helps distinguish the object from the reference. Recovering from a mismatch in visual features can require encoder updates as well as predictor updates.

Table 1: Planning Success under Dynamics and Layout Shifts for PointMaze. Note that low mass causes faster movement under same force and high damping causes velocity decay faster.
Dynamics Shift (PointMaze-Medium) Layout Shift
default low mass high damping Unseen Layouts
Adapt GD CEM GD CEM GD CEM GD CEM
Frozen 82.7  ±\pm 6.8 84.0  ±\pm 3.3 77.3  ±\pm 8.2 82.0  ±\pm 2.8 77.3  ±\pm 5.0 76.0  ±\pm 2.8 53.3  ±\pm 8.2 49.3  ±\pm 6.2
predlast + enclast 83.3  ±\pm 6.6 ↑\uparrow0.7 83.3  ±\pm 3.4 ↓\downarrow0.7 80.0  ±\pm 3.3 ↑\uparrow2.7 86.7  ±\pm 2.5 ↑\uparrow4.7 77.3  ±\pm 10.5 78.7  ±\pm 3.4 ↑\uparrow2.7 66.0  ±\pm 7.1 ↑\uparrow12.7 55.3  ±\pm 5.0 ↑\uparrow6.0
predfirst + enclast 84.0  ±\pm 1.6 ↑\uparrow1.3 84.0  ±\pm 4.3 82.0  ±\pm 1.6 ↑\uparrow4.7 82.7  ±\pm 3.4 ↑\uparrow0.7 78.7  ±\pm 4.7 ↑\uparrow1.3 82.0  ±\pm 3.3 ↑\uparrow6.0 78.7  ±\pm 5.0 ↑\uparrow25.3 70.7  ±\pm 3.8 ↑\uparrow21.3
Refer to caption
a1 Default Dyn.
frozen →\rightarrow success
Refer to caption
a2 Low Mass
frozen →\rightarrow failure
Refer to caption
a3 Low Mass
adapt →\rightarrow success
Refer to caption
b1 Default Dyn.
frozen →\rightarrow success
Refer to caption
b2 High Damping
frozen →\rightarrow failure
Refer to caption
b3 High Damping
adapt →\rightarrow success
Figure 4:PointMaze-Medium Dynamics-Shift Planning Trajectories. The green polylines trace the agent’s position over time, the blue square marks the end, and the gold star marks the goal. Under in-distribution dynamics the model reaches the goal. However, frozen world model mispredicts and planning fails under dynamics shifts, while test-time adaptation realigns with the new environment and recovers success.
Refer to caption
a1 Maze1
frozen →\rightarrow failure
Refer to caption
a2 Maze1
adapt →\rightarrow success
Refer to caption
b1 Maze2
frozen →\rightarrow failure
Refer to caption
b2 Maze2
adapt →\rightarrow success
Refer to caption
c1 Maze3
frozen →\rightarrow failure
Refer to caption
c2 Maze3
adapt →\rightarrow success
Figure 5:Diverse Maze Planning Trajectories. We use the same visual conventions as in Figure 4. While the frozen model fails on unseen mazes, AdaJEPA succeeds with trajectories close to the shortest paths.

Latency.

A small selected-layer update adds little measured time per replan in the tested implementations. Reaching the goal with fewer replans can compensate for that overhead. The latency result depends on the model, planner, hardware and number of update steps.

Visualization.

A separate decoder visualizes imagined latent rollouts. It can reconstruct training-like colors or shapes even when the observed test image differs. These examples suggest the updates reuse existing latent structure; decoding is an analysis tool, not part of the goal-reaching objective.

Table 2: AdaJEPA across different implementations. The planning success is evaluated using PushT validation trajectories from Zhou et al. (2025). The reported values are the average MPC success rate (%) and per-replan time using one H200. Test-time adaptation consistently improves and introduces almost negligible latency.
Encoder / Predictor Latent Dim Anti-collapse Setting GD (%) Time (s) CEM (%) Time (s)
WM w/ Temporal Straightening 1×3841{\times}384 stop-grad Frozen 84.0 ±\pm 2.0 3.14 74.0 ±\pm 3.5 0.24
(global feat.) (Wang et al., 2026) Adapt 85.3 ±\pm 3.1 ↑\uparrow1.3 3.17 ↑\uparrow0.03 81.3 ±\pm 6.4 ↑\uparrow7.3 0.27 ↑\uparrow0.03
WM w/ Temporal Straightening 196×384196{\times}384 stop-grad Frozen 91.3 ±\pm 4.2 3.37 89.3 ±\pm 3.1 5.37
(spatial feat.) (Wang et al., 2026) Adapt 92.0 ±\pm 3.5 ↑\uparrow0.7 3.38 ↑\uparrow0.01 93.3 ±\pm 2.3 ↑\uparrow4.0 5.39 ↑\uparrow0.02
DINO-WM (patch) 196×384196{\times}384 - Frozen 68.0 ±\pm 10.6 3.66 86.7 ±\pm 6.1 9.53
(spatial feat.) (Zhou et al., 2025) Adapt 70.0 ±\pm 4.0 ↑\uparrow2.0 3.68 ↑\uparrow0.02 90.0 ±\pm 3.5 ↑\uparrow3.3 9.56 ↑\uparrow0.03

4.3 Ablations

Which parameters to adapt?

Several selected-layer and low-rank-adapter updates improve on the frozen baseline. Shape changes can often be handled mainly by the predictor, whereas visual and layout changes benefit from encoder adaptation too. The best target depends on the environment; the experiment does not identify one universally optimal layer.

How should adaptation hyperparameters be chosen?

One step at the training learning rate is a useful tested default. Larger rates can work with one step but overshoot with repeated updates; smaller rates may need more steps. Buffer design has a smaller effect in these ablations. Stronger adaptation trades extra cost and possible instability for more correction.

AdaJEPA improves over various JEPA world models.

The loop improves the evaluated world models across several architectures, representation layouts and planners. Reported added time is about 0.01–0.03 seconds per replan for these models. This shows portability across the tested implementations rather than an architecture-independent guarantee.

4.4 Discussion

More training data and greater shape diversity help both frozen and adaptive models. Adaptation is particularly valuable when coverage is limited, but cannot reliably manufacture features the pretrained representation never learned. The data-scale study therefore supports combining pretraining and online correction, rather than replacing either with the other.

Original figure
Figure 6: Effect of training data scale on PushObj planning success: shape diversity KK and trajectories per shape NN.

(a) PushT-RedAnchor Refer to caption

(b) PushObj-Square Refer to caption

Figure 7:Examples of AdaJEPA Planning Trajectories under Visual Shifts and Shape Shifts. The decoder is trained on pretrained representations; i.e. (a) original PushT data with a gray pushed block; (b) PushObj data which only includes {T, L, Z, +} while square is an unseen shape.

5 Conclusion

Small self-supervised updates inside the control loop can correct useful prediction errors while the agent acts. The demonstrated benefit is within-episode simulated planning. Extending representation coverage through longer-term active or continual learning remains a proposed next step.

Acknowledgments

The authors acknowledge research funding, computing support and discussions that contributed to the work.

Appendix A Environments and Data

A.1 PushT (DINO-WM)

The PushT task moves a T-shaped block toward a target pose. In the DINO-WM evaluation setup, the fixed green T is a visual reference rather than the manipulated target object. This distinction matters for interpreting the color-shift experiments.

A.2 PushObj

PushObj replaces the block with several shapes. Replaying old actions can miss a new shape, so the dataset is biased toward contact and test trajectories require at least one contact. Shape-shift and visual-shift experiments use different training subsets. Their results depend on this deliberately contact-rich sampling.

A.3 PointMaze-Medium

PointMaze-Medium applies forces to a moving agent in a fixed maze. Tests alter mass or damping and choose separated start/goal cells, reducing trivial short-distance cases. The experiment reuses the cited pretrained model and data.

A.4 Diverse PointMaze

The layout test trains on 25 generated mazes and holds out five. Goals are selected three to five shortest-path cells away using breadth-first search. This tests new layouts with the specified generation and distance rules.

Appendix B Experiments

B.1 Hyperparameters

The retained tables give the full model, training and planning hyperparameters. Keep these together with the data and evaluation budget when reproducing an adaptive-versus-frozen comparison.

Table 3: Training Hyperparameters.
Name Value
Encoder lr 1e-5
Predictor lr 5e-4
Action/Prop encoder lr 5e-4
Batch size 64
History frames 3
Frameskip 5
Table 4: Planning Hyperparameters.
Name Value
Subplanner horizon 25
# Executed actions 5
GD Optimizer Adam
GD Action Initialization Zero
GD Learning rate 0.1
GD #opt steps 100
CEM # samples 200
CEM #opt steps 10

B.2 Ablations

These ablations vary the parameter subset, learning rate, update count and replay strategy while keeping the remaining protocol as consistent as specified.

Original figure
Figure 8: Effect of Adaptation Layers to Planning Success Rates. The reported values are the per-shift success rates (%) averaged over all setups within each shift. Test-time adaptation improves planning across all distribution shifts and is largely insensitive to which layers are adapted.
Original figure
(a) T (seen shape)
Original figure
(b) Square (unseen shape)
Figure 9:Effect of Adaptation Hyperparameters and Replay Buffers to Planning Success Rates.

Setup.

The encoder’s last stage is its projection head; its first stage extracts low-level visual features. Predictor variants update selected transformer blocks. The appendix uses a shorter replanning budget for these ablations than the longer-budget main plots.

What to adapt.

The listed variants change early or late predictor blocks, early or late encoder stages, or low-rank adapters while freezing the remaining parameters. Several choices work; stronger layout adaptation from early predictor changes suggests sensitivity near the state/action input interface.

Learning rates, optimization steps, and replay buffers.

Learning rate and update count interact: a large step can help quickly but repeated large steps may overshoot. Recent-transition buffers are a stable default in the tested grid, although adaptation also helps without a buffer. Best settings remain model- and environment-dependent.

B.3 Additional Experiments: Training Data Scale for PushObj

The data study varies both the number of shapes and trajectories per shape, balances which shapes appear in each training set, and holds optimization-step count fixed. Diversity improves generalization, while adaptation adds gains across the examined scales. Comparisons to larger frozen datasets use this specific training and evaluation design.

Refer to caption
Figure 10: Heatmap of offline training data scale vs. PushObj planning success. Rows vary shape diversity KK and columns vary trajectories per shape NN.

Appendix C Visualization of Planning Trajectories

The original panels compare observed trajectories, decoded imagined rollouts, goal success and prediction error over successive replans. Adaptive runs are grayed after reaching the goal; frozen runs are capped for display. A decoder trained on default images may retain their colors even when the real test observations change.

Refer to caption
Figure 11:The model is trained on shape {T,L,+,Z}, and tested on a seen shape + here.
Refer to caption
Figure 12:Shape Shifts: The model is trained on shapes {T, L, Z, +}, and tested on an unseen shape smallT. The decoder tries to decode to the closest seen shapes.
Refer to caption
Figure 13:Visual Shifts: The model is trained on the default PushT data, but the test-time observations have salt-and-pepper noise. AdaJEPA consistently decreases prediction loss.
Refer to caption
Figure 14:Visual Shifts: The model is trained on the default PushT data (agent is blue), but the agent is red at test time. AdaJEPA consistently decreases prediction loss.
Concept demonstration

Open image ↗