Literature
Display
← All papers

Learning and Leveraging World Models in Visual Representation Learning

Quentin Garrido · Mahmoud Assran · Nicolas Ballas · Adrien Bardes · Laurent Najman · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Contents

At a glance

Intuition. Given a picture and the instruction “increase its brightness,” a useful predictor should change the representation in the corresponding way. If it simply removes all brightness information, it cannot perform that instruction. Image World Models extend JEPA beyond filling masked regions by conditioning predictions on image transformations.

The paper distinguishes keeping information about a transformation from becoming invariant to it. Conditioning, sufficiently difficult transformations and adequate predictor capacity determine the observed behavior. A central downstream result is that the pretrained predictor can be adapted for recognition instead of always being discarded after pretraining.

1 Introduction

Self-supervised image transformations can be interpreted as actions inside a small world model. The encoder represents an image and the predictor models what a specified transformation does to that representation. This does not mean the system has learned physical dynamics from static images. It tests whether transformation modeling learns useful visual features and whether the predictor itself can later help solve classification or segmentation tasks.

Refer to caption
Figure 1: Visualisation of predictions in latent space with a learned Image World Model. We apply an action on a source image in latent space and retrieve the nearest neighbour of the predicted representation in a bank of 256 images. We see that IWM is capable of modeling transformations and undo corruptions, showing an understanding of the underlying image transformations. Image from: ai.meta.com/blog/yann-lecun-advances-in-ai-research/
Original figure
Figure 2: Multiple families of methods with related architectures can be distinguished, in which the conditioning or not of their world model is a key distinction. Generative World Models are trained to invert a transformation in input space, leveraging an autoencoder framework. Methods for world modeling and representation learning can be instantiated in this way. Joint Embedding methods get rid of the world model but operate in latent space by encoding what is common between transformed inputs. It is the main class of SSL methods. JEPA World Models can be seen as a more general framework where a world model is trained in latent space. This family has been very successful both in reinforcement learning and in representation learning, and is where Image World Models (IWM) falls.

2 Related works

2.1 Augmentation invariant Self-Supervised Learning

Augmentation-invariant learning makes related image views share a representation. That can improve recognition by suppressing irrelevant details, but may also remove information useful for another task. Different contrastive and noncontrastive objectives provide different protections against a constant representation.

2.2 World modeling in visual representation learning

Masked image modeling, equivariant representation learning and generative prediction can all be viewed as learning effects of transformations. IWM emphasizes two differences: explicitly condition the latent predictor on what changed, and investigate reusing that predictor downstream rather than evaluating only the encoder.

3 Method

The method constructs source and target views, including masked source patches and different photometric transformations. An encoder represents the source; a momentum-updated target encoder supplies the training target. The predictor receives both the source representation and transformation information, then predicts target patch features. The paper calls a predictor “equivariant” when it can apply the controlled change. Some transformations are destructive rather than invertible group actions, so this term describes the tested predictive behavior rather than a universal mathematical symmetry guarantee.

Equation
L⁡(x,y)=∑i∈MxC‖pϕ​(fθ​(x),ax→y,ma)i−fθEMA​(y)i‖22.L(x,y)=\sum_{i\in M_{x}^{C}}\|p_{\phi}\left(f_{\theta}(x),a_{x\rightarrow y},m_{a}\right)_{i}-f_{\theta}^{\text{EMA}}(y)_{i}\|_{2}^{2}.

3.1 Architecture and nomenclature

The main experiments use a ViT-B/16 encoder and Transformer predictors of different depths and widths. Model names record predictor depth, embedding dimension and whether its observed behavior is invariant or transformation-sensitive. A larger predictor is not merely an evaluation head added after training: its pretraining task shapes the resulting encoder.

4 Learning an Image World Model for representation learning

4.1 Evaluating the quality of the world model

To test whether a transformation was predicted correctly, the authors rank the actual target representation among a bank of 256 transformed candidates. Mean Reciprocal Rank is high when the intended transformed target is near the top. This measures the specific transformation task; it is not classification accuracy or a physical-planning success rate.

4.2 Learning a strong Image World Model

Three choices interact. The predictor needs unambiguous transformation conditioning, an adequately challenging range of transformations and enough capacity to model their effects. Without conditioning it can instead remove transformation information. Weak transformations or an undersized predictor also encourage invariant behavior. The favored implementation mixes transformation parameters with mask-token features; its ablations compare this with sequence-token conditioning.

Table 1: Influence of predictor conditioning on the quality of the world model. Both Sequence and Feature conditioning lead to good world models .Gray is our default setting.
Conditioning: None Sequence Feature
MRR 0.00 0.82 0.79
Table 2: Impact of predictor architecture and transformations on MRR. Learning an effective world model requires complex transformations and adequate predictor capacity. Gray is our default setting. Red and Green respectively indicate invariant and equivariant behavior.
Predictor: I-JEPA IWM
(depth, dim.): (12,384) (12,384) (18,384)
Jitter 0.00 0.11 0.25
+ Destructive 0.00 0.09 0.79
+ Strong Jitter 0.00 0.81 0.85

4.3 Visualizing predictions.

Nearest-neighbor images visualize predicted embeddings by retrieving transformed candidates. They show learned photometric changes but also ambiguity when information has been destroyed, such as recovering color from grayscale. These are representation-space retrieval visualizations, not direct pixel generation by the JEPA predictor.

5 Leveraging world models for downstream tasks

Predicting color changes is not itself the end application. The downstream experiments ask whether the world-model computation can be repurposed for image classification and semantic segmentation.

5.1 Predictor finetuning

Predictor finetuning places the pretrained predictor above a frozen encoder and adapts it, with an evaluation head, to the supervised downstream task. Its pretraining output is another representation, so simply reading it without adaptation is not the same protocol. Experiments vary the target prediction task, teacher versus student features, and whether to use a pretrained or random predictor. Reusing a capable pretrained predictor can match or exceed tested encoder-finetuning baselines while avoiding encoder gradients. Comparisons must account for both predictor capacity and which parameters are trained.

Table 3: How to predict for predictor finetuning. Using the teacher improves performance, and the exact prediction task is not crucial. Null latents are more flexible and perform better. For better efficiency, a full prediction is not needed but leads to a small drop in performance. Gray is our default setting.
Setting ImageNet Top-1 (%) Gap
Default 82.9 -
+ Teacher 83.2 + 0.3
+ Null latents 83.3 + 0.1
+ Pred only one token 82.8 -0.5
Table 4: Finetuning evaluations on ImageNet-1k. We evaluate prediction based methods by finetuning their encoder, by keeping the encoder frozen and finetuning their predictive world model or by finetuning both. Finetuning the world model is highly effective with IWM when it exhibits an equivariant behavior. This behavior is absent or less clear with other methods, showing the importance of a strong world model.
Method Epochs No predictor Frozen encoder, tuned predictor End to end
Encoder Random Init. Pretrained
MAE 300 82.7 82.4 82.7 (+0.3) 82.3
1600 83.6 83.0 83.1 (+0.1) 83.3
I-JEPA 300 83.0 79.1 80.0 (+0.9) 82.0
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 300 83.3 80.5 81.3 (+0.8) 82.7
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 300 82.9 81.5 83.3 (+1.8) 84.4
Original figure
Figure 3: Finetuning efficiency. When taking into account the number of finetuned parameters, predictor finetuning is significantly more efficient than finetuning the encoder.
Table 5: Peak performance achieved from a single pretraining instance. We compare ImageNet Top-1 accuracy with a frozen encoder or when allowing any evaluation head with any protocol, finetuning or not, with a predictor on top of the encoder or not.
Method Epochs Frozen Encoder Any protocol
DINO 1600 82.0 82.8
MOCOv3 300 76.4 83.2
iBOT 1600 83.0 84.0
MAE 1600 83.1 83.6
I-JEPA 300 80.0 82.0
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 300 81.3 83.3
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 300 83.3 84.4
Table 6: Finetuning for segmentation on ADE20k. Similar to image classification, we observe that predictor finetuning improves performance and outperforms encoder finetuning.
Method Encoder Predictor End to end
I-JEPA 44.2 45.4 45.1
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 45.6 45.7 46.5
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 44.2 46.8 47.0

5.2 Multitask predictor tuning

Task-specific tokens and output heads allow one predictor to be finetuned for several datasets at once. The minibatch is divided between tasks and their losses update the shared predictor. The experiment tests a particular sampling and training budget; it does not establish that adding arbitrary tasks is free or always beneficial.

Table 7: Multi-task finetuning. Finetuning the predictor on multiple tasks at once performs similarly as finetuning it on each task separately. This enables the use of a single prediction head for multiple task, amortizing its cost.
Dataset Single-task Multi-task Difference
ImageNet 80.8 79.6 -1.2
iNat18 72.4 72.0 -0.4
SUN397 75.6 78.2 +2.6
Places205 64.8 64.1 -0.7
Average 73.4 73.5 +0.1

6 Image World Models enable flexible representations

Invariant representations tend to be easier to read with a linear probe. Transformation-sensitive representations can retain richer information that an attentive head or finetuned model uses better. The paper therefore treats evaluation protocol and representation abstraction as linked choices: one linear-probe score does not determine which representation is best for every downstream use.

Table 8: Linear and attentive probing performance on ImageNet-1k. IWMInv\text{IWM}^{\text{Inv}} performs similarly to contrastive methods and IWMEqui\text{IWM}^{\text{Equi}} to mask modeling ones.
Method Effective Epochs Linear Attentive
MoCoV3 300 76.3 76.4
MAE 300 60.2 73.5
MAE 1600 68.0 76.0
I-JEPA 300 70.0 75.0
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 300 74.5 77.0
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 300 67.5 75.1
Original figure
Figure 4: While the level of equivariance influences performance in Linear and Predictor finetuning setting, it is hardly correlated to Attentive probing. This suggests that there is a trade-off in terms of the level of abstraction of the representation, and that different evaluation protocols evaluate different properties.
Refer to caption
Figure 5: Image World Models allow representation modularity. Different families of methods offer representations with different properties, but IWM allows exploring the whole spectrum.

7 Conclusion and future perspectives

The main design lesson is to specify which variations a representation should retain and give the predictor the information and capacity needed to model them. The demonstrated world is one of image transformations. Extending this argument to video, control or uncertain physical events requires new training data and tests.

8 Broader impact statement

The broader-impact statement presents general visual representation research and identifies no paper-specific societal consequence. This is a disclosure statement, not a deployment safety evaluation.

Appendix A Experimental details

A.1 Pretraining

Most models pretrain on ImageNet for 300 epochs with a ViT-B/16 encoder. The appendix fixes optimization, masking and augmentation schedules. Targets keep more visual information, while source views may receive destructive grayscale, blur or solarization. Momentum-target and schedule settings are part of the recipe.

Refer to caption
Figure S1: IWM (Image World Model). Starting from an image, two augmented views are produced: the source and the target. The source view is partially masked to form the context and then encoded to be used as conditioning for the world model, instantiated by the predictor. The target is encoded through an exponential moving average of the encoder, and target positions are sampled as the masked patches of the source image. Conditioned on the transformation parameters between the source and target, the encoded source image, and the positions to predict, the predictor is trained to predict the target representations.

A.2 Evaluation

Classification validation uses resize and center-crop preprocessing. Hyperparameters are separately tuned for the methods and evaluation protocols, so reported differences are not all from a single common optimizer setting.

Linear.

Linear evaluation average-pools frozen encoder features and trains one linear classification layer. This is the simplest readout and favors features that already make the labels linearly accessible.

Attentive.

The attentive probe learns a cross-attention pooling operation over unpooled features. It has more capacity to extract useful information than average pooling plus a linear layer, while still keeping the representation backbone frozen.

Encoder finetuning.

Encoder finetuning updates the encoder and classification head with the stated augmentation and optimizer recipe. It is a different resource and adaptation setting from training a lightweight readout.

Predictor finetuning.

Predictor finetuning uses a frozen teacher encoder, asks the predictor for the full image representation with null transformation parameters and trains an attentive task head. Pretrained predictor parameters receive a smaller learning rate than a randomly initialized predictor.

Original figure
Figure S2: Multitask tuning of the predictor. We sample a batch uniformly across task which is then fed through the predictor with an additional task token, indicating which task is being solved. The predictions are then fed through a task specific head and losses are summed.

Multitask predictor finetuning.

The multitask experiment shares predictor parameters across task-specific tokens and heads. The appendix gives the balanced minibatch allocation and training budget needed to reproduce that comparison.

End to end finetuning.

End-to-end finetuning updates both encoder and predictor, with separate learning-rate scaling for pretrained components. It tests the value of further adaptation, rather than the efficiency of keeping the encoder frozen.

Segmentation.

Segmentation uses an UperNet head and ADE20K, with features from the last four encoder or predictor layers. The protocol specifies training resolution, positional interpolation at evaluation and learning-rate searches; these details affect the reported intersection-over-union scores.

Appendix B Complete finetuning results

The full table includes predictor sizes and additional baselines omitted from compact main-text comparisons. Some methods respond differently to attaching a random predictor. The proposed explanation for one weak baseline remains a hypothesis, not a verified causal finding.

Table S1: Complete results of tables 4 and 5.
Method Epochs No predictor Frozen encoder, tuned predictor End to end
Encoder Random Init. Pretrained
DINO 1600 82.8 82.0 N/A 82.1
MOCOv3 300 83.2 56.4 N/A 79.4
iBOT 1600 84.0 83.0 N/A 82.8
MAE 300 82.7 82.4 82.7 (+0.3) 82.3
600 83.2 82.8 83.0 (+0.2) 83.1
1600 83.6 83.0 83.1 (+0.1) 83.3
I-JEPA 300 83.0 79.1 80.0 (+0.9) 82.0
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 300 83.3 80.5 81.3 (+0.8) 82.7
IWM12,384Equi\text{IWM}_{12,384}^{\text{Equi}} 300 82.7 81.3 82.7 (+1.4) 83.3
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 300 82.9 81.5 83.3 (+1.8) 84.4

Appendix C Impact of data augmentation

Depth and augmentation difficulty jointly affect whether a predictor learns transformation-sensitive behavior. A deeper predictor succeeds under more tested color-transformation settings; very weak augmentations can remove the pressure to learn the transformation.

Table S2: Impact of data augmentation strategy on IWM’s performance. In all settings, destructive augmentations are never applied to the target.
Predictor Strengths Probabilities Performance
Bright. Contrast Sat. Hue Jitter Blur Gray. Solarize MRR Linear Attentive Pred. ft.
IWM12,384\text{IWM}_{12,384} 0.4 0.4 0.2 0.1 0.8 0.2 0.2 0.1 0.09 74.5 77.0 81.3
0.4 0.4 0.2 0.1 0.8 0.11 72.1 76.5 80.5
0.4 0.4 0.2 0.1 0.8 0.4 0.4 0.2 0.22 71.0 74.9 80.9
0.5 0.5 0.4 0.2 0.8 0.2 0.2 0.1 0.81 69.3 75.5 82.7
0.5 0.5 0.4 0.2 0.8 0.07 73.3 76.3 80.1
0.2 0.2 0.1 0.02 72.9 76.3 80.7
IWM18,384\text{IWM}_{18,384} 0.4 0.4 0.2 0.1 0.8 0.2 0.2 0.1 0.79 67.5 75.1 83.3
0.4 0.4 0.2 0.1 0.8 0.25 70.1 74.8 81.4
0.4 0.4 0.2 0.1 0.8 0.4 0.4 0.2 0.85 56.1 74.5 83.1
0.5 0.5 0.4 0.2 0.8 0.2 0.2 0.1 0.85 34.3 71.0 81.7
0.5 0.5 0.4 0.2 0.8 0.83 69.2 75.8 83.3
0.2 0.2 0.1 0.02 70.9 74.8 81.5

On the asymmetry of augmentations.

Using more destructive augmentation on the source than the target helps the tested invariant models. Transformation-sensitive models are less affected in these experiments. This suggests a robustness benefit under the tested augmentation choices, not invariance to every augmentation design.

Appendix D Impact of the prediction task on predictor finetuning performance

Additional combinations support predicting a full target representation from teacher features with flexible null conditioning during finetuning. Predicting only one token is slightly worse in the tested comparisons.

Table S3: Predictor finetuning performance which different prediction tasks.
Method Null latents On teacher Pred only one token Accuracy
IWM12,384\text{IWM}_{12,384} ✓ ✓ 83.3
✓ ✓ ✓ 82.8
✓ 83.1
✓ ✓ 82.6
✓ 83.2
✓ ✓ 82.8
82.9
✓ 82.9

Appendix E Scaling to larger models

Scaling to a larger encoder also requires a sufficiently deep and wide predictor. Wider predictors introduce stability issues and sensitivity to the momentum schedule. The authors report a useful predictor-to-encoder parameter ratio for their runs; it is an empirical recipe rather than a universal scaling law.

Table S4: With a ViT-L/16 encoder, we observe a similar trend as with the base model. Significant gains are observed with a good world model, allowing it to surpass encoder finetuning.
Method Epochs Encoder Predictor End to end
I-JEPA 300 84.1 79.9
IWM18,384Inv\text{IWM}_{18,384}^{\text{Inv}} 300 84.3 81.5
IWM36,512Equi\text{IWM}_{36,512}^{\text{Equi}} 300 83.7 85.0 85.4

Appendix F Evaluation on downstream datasets beyond ImageNet

Attentive probes on iNaturalist18, SUN397 and Places205 test transfer beyond ImageNet. Transformation-sensitive IWM can improve over its invariant counterpart even when it is not the best ImageNet model, reinforcing the distinction between in-domain readout simplicity and retained information useful for transfer.

Table S5: When evaluating with attentive probing on downstream task, being equivariant improves performance across the board. All methods use ViT-B/16 encoders and were pretrained for 300 epochs on ImageNet
Method ImageNet iNat18 SUN397 Places205
MAE 73.5 50.1 70.2 60.3
I-JEPA 75.0 50.4 69.2 58.3
IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}} 77.0 51.6 71.0 59.4
IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}} 75.1 54.2 71.7 60.5

Appendix G Visualizing representation differences between invariant and equivariant behavior

Representation-similarity visualizations show that an invariant model maps augmented views close together, while a transformation-sensitive model retains differences. I-JEPA lies between these tested behaviors rather than being explicitly optimized for either extreme.

{subfigure}

0.3 Refer to caption {subfigure}0.3 Refer to caption {subfigure}0.3 Refer to caption

Figure S3: IWM12,384Inv\text{IWM}_{12,384}^{\text{Inv}}
Figure S4: IWM18,384Equi\text{IWM}_{18,384}^{\text{Equi}}
Figure S5: IJEPA
Figure S6: Difference in embedding space between invariant and equivariant behaviours. Each image is augmented 16 times and we compute the similarity matrix between all images. The yellow regions indicate high similarities between samples originating from the same image. We can see more variations in the equivariant model, or in I-JEPA where invariance is not enforced. This suggests that augmentations influence the representation more in these models.

Appendix H On the meaning and role of invariance in Self-Supervised learning

Invariance can arise by throwing away a changing attribute or by retaining enough information to normalize over its changes. The appendix uses IWM’s predictor to average transformed representations efficiently in latent space. Its retrieval and evaluation experiments explore this alternative while noting that invariance alone does not specify how much information was retained.

Equation
zxInv=1N​∑i=1Npϕ​(fθ​(x),Ai,mAi)z_{x}^{\text{Inv}}=\frac{1}{N}\sum_{i=1}^{N}p_{\phi}\left(f_{\theta}(x),A_{i},m_{A_{i}}\right)
Refer to caption
Figure S7: Retrieval of invariant representations computed using 256 augmentations in latent space. In the top row we visualize some of the corresponding image and on the bottom the nearest neighbours of the invariant representation. We can notice that the nearest neighbour is the original non-augmented image, followed by images with small transformations.
Table S6: Linear evaluation on marginalized representations. Using more augmented prediction to create an invariant representation does not improve performance.
Number of predictions (NN) 1 (default) 8 16 32 128
ImageNet Top-1 accuracy (%) 64.5 64.3 64.6 64.6 64.4

Appendix I Additional qualitative evaluations of the world model

The preserved additional qualitative figures compare predicted representations with transformed image candidates. They complement the ranking metric and show both successful transformations and ambiguous destructive cases.

Refer to caption
Figure S8: Randomly selected retrieval samples of our world model. For each image, we generate 256 augmented views and apply transformations in latent space. We then retrieve the nearest neighbor of the prediction and visualize whether it is close to the groundtruth or not. The learned world model performs well in most settings but has some inaccuracies with inverting grayscale.
Refer to caption
Figure S9: Randomly selected retrieval samples of our world model. For each image, we generate 256 augmented views and apply transformations in latent space. We then retrieve the nearest neighbor of the prediction and visualize whether it is close to the groundtruth or not. The learned world model performs well in most settings but has some inaccuracies with inverting grayscale.
Refer to caption
Figure S10: Randomly selected retrieval samples of our world model. For each image, we generate 256 augmented views and apply transformations in latent space. We then retrieve the nearest neighbor of the prediction and visualize whether it is close to the groundtruth or not. The learned world model performs well in most settings but has some inaccuracies with inverting grayscale.
Refer to caption
Figure S11: Randomly selected retrieval samples of our world model. For each image, we generate 256 augmented views and apply transformations in latent space. We then retrieve the nearest neighbor of the prediction and visualize whether it is close to the groundtruth or not. The learned world model performs well in most settings but has some inaccuracies with inverting grayscale.
Refer to caption
Figure S12: Application of the world model on precise transformations. For each parameter, we vary its value on a grid to see whether the model is able to predict small changes. The model is able to show the gradient of transformations, highlighting again the capabilities of the world model. We can still notice some imperfections however, as the model was only trained on combinations of augmentations. To make changes more visible, we used a model trained with a strong color jitter for this figure.
Refer to caption
Figure S13: Application of the world model on precise transformations. For each parameter, we vary its value on a grid to see whether the model is able to predict small changes. The model is able to show the gradient of transformations, highlighting again the capabilities of the world model. We can still notice some imperfections however, as the model was only trained on combinations of augmentations. To make changes more visible, we used a model trained with a strong color jitter for this figure.
Refer to caption
Figure S14: Application of the world model on precise transformations. For each parameter, we vary its value on a grid to see whether the model is able to predict small changes. The model is able to show the gradient of transformations, highlighting again the capabilities of the world model. We can still notice some imperfections however, as the model was only trained on combinations of augmentations. To make changes more visible, we used a model trained with a strong color jitter for this figure.
Concept demonstration

Open image ↗