Literature
Display
← All papers

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

Mahmoud Assran · Quentin Duval · Ishan Misra · Piotr Bojanowski · Pascal Vincent · Michael Rabbat · Yann LeCun · Nicolas Ballas

· CVPR 2023 ↗

World models7 min text · CondensedOriginal paper ↗
Contents

Overview

Original figure
Figure 1: ImageNet Linear Evaluation. The I-JEPA method learns semantic image representations without using any view data augmentations during pretraining. By predicting in representation space, I-JEPA produces semantic representations while using less compute than previous methods.
Original figure
(a) Joint-Embedding Architecture
Original figure
(b) Generative Architecture
Original figure
(c) Joint-Embedding Predictive Architecture
Figure 2: Common architectures for self-supervised learning, in which the system learns to capture the relationships between its inputs. The objective is to assign a high energy (large scaler value) to incompatible inputs, and to assign a low energy (low scaler value) to compatible inputs. (a) Joint-Embedding Architectures learn to output similar embeddings for compatible inputs x,yx,y and dissimilar embeddings for incompatible inputs. (b) Generative Architectures learn to directly reconstruct a signal yy from a compatible signal xx, using a decoder network that is conditioned on additional (possibly latent) variables zz to facilitate reconstruction. (c) Joint-Embedding Predictive Architectures learn to predict the embeddings of a signal yy from a compatible signal xx, using a predictor network that is conditioned on additional (possibly latent) variables zz to facilitate prediction.

At a glance

Imagine hiding several regions of a photograph and asking what belongs there. I-JEPA predicts a learned description of each hidden region, rather than its exact pixels. A context encoder sees only the visible patches; a predictor uses their features and target positions; a slowly updated target encoder supplies the training targets. Large target regions and a context spread across the image make the task require useful scene information. The learned encoder transfers to classification, counting and depth tasks. The experiments establish an effective image-pretraining recipe, rather than a complete predictive world model or a guarantee of human-like understanding.

1 Introduction

Pixel reconstruction spends effort on details that may not help recognition. View-invariance training can learn semantic features but relies on choices about which image changes should be ignored. I-JEPA instead predicts missing-region embeddings. Its contribution is the combination of representation-space targets, an asymmetric encoder–predictor design, and a masking strategy that makes semantic prediction useful. Comparisons cover frozen features, limited-label adaptation and local prediction tasks.

2 Background

Self-supervision learns relationships within unlabeled data. In the energy-based view used here, compatible inputs should receive lower mismatch scores than incompatible ones. The following architectures differ in what they compare or predict.

Joint-Embedding Architectures.

A joint-embedding architecture makes two related image views have similar representations. Its obvious failure is collapse: every image receives the same vector. Contrastive examples, distributional regularization and asymmetric network updates are different ways of addressing that problem.

Generative Architectures.

A generative masked-image method predicts the hidden pixels or tokens themselves. Mask positions identify what must be reconstructed. This supplies a direct training target, but also requires modeling low-level details that may be irrelevant to a later task.

Joint-Embedding Predictive Architectures.

A JEPA predicts the representation of a target from the representation of its context, conditioned on information such as target position. Prediction replaces direct view matching, but collapse remains possible. I-JEPA uses a slowly updated target branch and a separate predictor; it does not derive a universal anti-collapse theorem.

3 Method

Three vision transformers share the work: a context encoder processes visible patches, a target encoder describes the full image, and a narrower predictor estimates selected target features. Figure 3 shows their connections; Figure 4 shows how masking removes overlap while leaving a distributed context.

Refer to caption
Figure 3: I-JEPA. The Image-based Joint-Embedding Predictive Architecture uses a single context block to predict the representations of various target blocks originating from the same image. The context encoder is a Vision Transformer (ViT), which only processes the visible context patches. The predictor is a narrow ViT that takes the context encoder output and, conditioned on positional tokens (shown in color), predicts the representations of a target block at a specific location. The target representations correspond to the outputs of the target-encoder, the weights of which are updated at each iteration via an exponential moving average of the context encoder weights.
Refer to caption
Figure 4: Examples of our context and target-masking strategy. Given an image, we randomly sample 44 target blocks with scale in the range (0.15,0.2)(0.15,0.2) and aspect ratio in the range (0.75,1.5)(0.75,1.5). Next, we randomly sample a context block with scale in the range (0.85,1.0)(0.85,1.0) and remove any overlapping target blocks. Under this strategy, the target-blocks are relatively semantic, and the context-block is informative, yet sparse (efficient to process).

Targets.

The target encoder first processes the complete image. Four blocks are then sampled from its output, each covering roughly 15–20% of the image with varying aspect ratio. Masking after encoding lets each target feature use full-image context, unlike encoding an isolated crop.

Context.

Start from a large context block covering 85–100% of the image, then remove every patch overlapping a target. The context encoder therefore receives a sparse collection of visible patches rather than the full image. Keeping enough distributed context makes the prediction task informative.

Prediction.

For each target block, the predictor combines visible-context features with learned mask tokens carrying the target locations. Running it separately for the four blocks produces a feature estimate for every requested target patch.

Loss.

Training minimizes squared feature error between predictions and target-encoder outputs. Gradients update the context encoder and predictor. The target encoder instead follows an exponential moving average of context-encoder weights, providing a slower-changing learning target.

Equation
1M​∑i=1MD⁡(𝒔^y​(i),𝒔y​(i))=1M​∑i=1M∑j∈Bi∥𝒔^yj−𝒔yj∥22.\frac{1}{M}\sum^{M}_{i=1}{D}\left(\hat{{\bm{s}}}_{y}(i),{\bm{s}}_{y}(i)\right)=\frac{1}{M}\sum^{M}_{i=1}\sum_{j\in B_{i}}\lVert\hat{{\bm{s}}}_{y_{j}}-{\bm{s}}_{y_{j}}\rVert^{2}_{2}.

4 Related Work

I-JEPA belongs to masked representation learning, alongside methods using pixels, discrete tokens or online feature targets. Its distinguishing goal is useful frozen semantic features without extensive handcrafted view augmentations. Data2vec and context autoencoders are particularly close comparisons; the retained table gives the actual pretraining and evaluation configurations.

Method Arch. Epochs Top-1
Methods without view data augmentations
data2vec [8] ViT-L/16 1600 77.3
MAE [36] ViT-B/16 1600 68.0
ViT-L/16 1600 76.0
ViT-H/14 1600 77.2
CAE [22] ViT-B/16 1600 70.4
ViT-L/16 1600 78.1
I-JEPA ViT-B/16 600 72.9
ViT-L/16 600 77.5
ViT-H/14 300 79.3
ViT-H/1644816_{448} 300 81.1
Methods using extra view data augmentations
SimCLR v2 [21] RN152 (2×2\times) 800 79.1
DINO [18] ViT-B/8 300 80.1
iBOT [79] ViT-L/16 250 81.0
Table 1: ImageNet. Linear-evaluation on ImageNet-1k (the ViT-H/16448 is pretrained at at a resolution of 448×448448\times 448). I-JEPA improves linear probing performance compared to other methods that do not rely on hand-crafted view data-augmentations during pretraining. Moreover, I-JEPA demonstrates good scalability — the larger I-JEPA model matches the performance of view-invariance approaches without requiring view data-augmentations.

5 Image Classification

Classification tests ask whether the pretrained features already organize images usefully. A frozen linear probe tests this directly; low-shot adaptation additionally tests whether a small labeled sample can make the representation useful.

Method Arch. Epochs Top-1
Methods without view data augmentations
data2vec [8] ViT-L/16 1600 73.3
MAE [36] ViT-L/16 1600 67.1
ViT-H/14 1600 71.5
I-JEPA ViT-L/16 600 69.4
ViT-H/14 300 73.3
ViT-H/1644816_{448} 300 77.3
Methods using extra view data augmentations
iBOT [79] ViT-B/16 400 69.7
DINO [18] ViT-B/8 300 70.0
SimCLR v2 [35] RN151 (2×2\times) 800 70.2
BYOL [35] RN200 (2×2\times) 800 71.2
MSN [4] ViT-B/4 300 75.7
Table 2: ImageNet-1%. Semi-supervised evaluation on ImageNet-1K using only 1% of the available labels. Models are adapted via fine-tuning or linear-probing, depending on whichever works best for each respective method. ViT-H/16448 is pretrained at at a resolution of 448×448448\times 448. I-JEPA pretraining outperforms MAE which also does not rely on hand-crafted data-augmentations during pretraining. Moreover, I-JEPA benefits from scale. A ViT-H/16 trained at resolution 448448 surpasses previous methods including methods that leverage extra hand-crafted data-augmentations.

ImageNet-1K.

The encoder stays frozen while a linear classifier learns from ImageNet labels. I-JEPA improves on the compared pixel-reconstruction methods. Larger models and higher input resolution narrow the gap to augmentation-heavy joint-embedding methods; the headline scores compare different model sizes and training budgets.

Low-Shot ImageNet-1K.

The 1%-label benchmark supplies only about 12–13 images per class. Each method uses its stronger adaptation protocol, either linear probing or fine-tuning. I-JEPA performs well in this setting, but those scores are not a uniform frozen-feature comparison across all methods.

Transfer learning.

Frozen features transfer to several image-classification datasets. I-JEPA improves over the compared non-augmentation methods and approaches the strongest view-invariance baselines, with performance depending on the dataset.

Method Arch. CIFAR100 Places205 iNat18
Methods without view data augmentations
data2vec [8] ViT-L/16 81.6 54.6 28.1
MAE [36] ViT-H/14 77.3 55.0 32.9
I-JEPA ViT-H/14 87.5 58.4 47.6
Methods using extra view data augmentations
DINO [18] ViT-B/8 84.9 57.9 55.9
iBOT [79] ViT-L/16 88.3 60.4 57.3
Table 3: Linear-probe transfer for image classification. Linear-evaluation on downstream image classification tasks. I-JEPA significantly outperforms previous methods that also do not use augmentations (MAE and data2vec), and decreases the gap with the best view-invariance-based methods that leverage hand-crafted data augmentations during pretraining.

6 Local Prediction Tasks

CLEVR counting and depth tasks probe information beyond image category. A linear model trained above a frozen encoder benefits from I-JEPA features relative to the tested view-invariance methods. These controlled tasks support the claim that the representation retains useful local structure.

Method Arch. Clevr/Count Clevr/Dist
Methods without view data augmentations
data2vec [8] ViT-L/16 85.3 71.3
MAE [36] ViT-H/14 90.5 72.4
I-JEPA ViT-H/14 86.7 72.4
Methods using extra data augmentations
DINO [18] ViT-B/8 86.6 53.4
iBOT [79] ViT-L/16 85.7 62.8
Table 4: Linear-probe transfer for low-level tasks. Linear-evaluation on downstream low-level tasks consisting of object counting (Clevr/Count) and depth prediction (Clevr/Dist). The I-JEPA method effectively captures low-level image features during pretraining and outperforms view-invariance based methods on tasks such object counting and depth prediction.
Original figure
Figure 5: Scaling. Semi-supervised evaluation on ImageNet-1K 1% as a function of pretraining GPU hours. I-JEPA requires less compute than previous methods to achieve strong performance. Compared to MAE and data2vec, I-JEPA obtains a significant speedup by requiring fewer pretraining epochs. Compared to iBOT, which relies on hand-crafted data-augmentations, a huge I-JEPA model (ViT-H/14) requires less compute than their smallest model (ViT-S/16).
Pretrain Arch. CIFAR100 Place205 INat18 Clevr/Count Clevr/Dist
IN1k ViT-H/14 87.5 58.4 47.6 86.7 72.4
IN22k ViT-H/14 89.5 57.8 50.5 88.6 75.0
IN22k ViT-G/16 89.5 59.1 55.3 86.7 73.0
Table 5: Ablating dataset and model size. Evaluating impact of pre-training dataset size and model size on transfer tasks. I-JEPA benefits from larger more diverse datasets. When increasing the size of the pretraining dataset (IN1k versus IN22k) we see an performance improvement for the ViT-H/14 model. We observe a further performance improvement on semantic tasks by training a larger model ViT-G/16 model on ImageNet-22k. The ViT-H/14 is trained for 300 epochs on IN1k and the equivalent of 900 IN1K epochs on IN22k. The ViT-H/16 is trained for the equivalent of 600 IN1k epochs.
Refer to caption
Figure 6: Visualization of I-JEPA predictor representations. For each image: first column contains the original image; second column contains the context image, which is processed by a pretrained I-JEPA ViT-H/14 encoder. Green bounding boxes in subsequent columns contain samples from a generative model decoding the output of the pretrained I-JEPA predictor, which is conditioned on positional mask tokens corresponding to the location of the green bounding box. Qualities that are common across samples represent information that contained is in the I-JEPA prediction. The I-JEPA predictor is correctly capturing positional uncertainty and producing high-level object parts with a correct pose (e.g., the back of the bird and top of a car). Qualities that vary across samples represent information that is not contained in the representation. In this case, the I-JEPA predictor discards the precise low-level details as well as background information.

7 Scalability

Model Efficiency.

Computing feature targets adds per-step cost relative to MAE, but the reported recipe needs many fewer training steps. The resulting compute savings are measured for these implementations and schedules. Processing one sparse context also avoids the multiple augmented views used by some baselines.

Scaling data size.

Pretraining on ImageNet-22K instead of ImageNet-1K improves both semantic and local downstream tasks in the reported comparisons. More diverse pretraining data is useful alongside the training objective.

Scaling model size.

Increasing model size improves several classification tasks. The largest model does not improve every local task: its larger input patches can discard spatial detail. Model capacity and spatial resolution therefore need separate consideration.

Targets Context
Mask Type Freq. Type Avg. Ratio∗ Top-1
multi-block Block(0.15,0.2)(0.15,0.2) 4 Block(0.85,1.0)(0.85,1.0) ×\times Complement 0.25 54.2
rasterized Quadrant 3 Complement 0.25 15.5
block Block(0.6)(0.6) 1 Complement 0.4 20.2
random Random(0.6) 1 Complement 0.4 17.6
∗Avg. Ratio is the average number of patches in the context block relative to the total number of patches in the image.
Table 6: Ablating masking strategy. Linear evaluation on ImageNet-1K using only 1% of the available labels after I-JEPA pretraining of a ViT-B/16 for 300 epochs. Comparison of proposed multi-block masking strategy. In rasterized masking the image is split into four large quadrants; one quadrant is used as a context to predict the other three quadrants. In block masking, the target is a single image block and the context is the image complement. In random masking, the target is a set of random image patches and the context is the image complement. The proposed multi-block masking strategy is helpful for guiding I-JEPA to learn semantic representations.

8 Predictor Visualizations

A separately trained conditional diffusion decoder turns predicted embeddings into illustrative images. Features shared across generated samples suggest information retained by the embedding; varying details suggest ambiguity. This visualization decoder is an analysis tool added after pretraining, not part of I-JEPA’s training objective.

9 Ablations

Targets Arch. Epochs Top-1
Target-Encoder Output ViT-L/16 500 66.9
Pixels ViT-L/16 800 40.7
Table 7: Ablating targets. Linear evaluation on ImageNet-1K using only 1% of the available labels. The semantic level of the I-JEPA representations degrades significantly when the loss is applied in pixel space, rather than representation space, highlighting the importance of the target-encoder during pretraining.

Predicting in representation space.

Replacing learned feature targets with pixel targets reduces low-shot linear-probe performance in the tested ablation. This isolates evidence for representation-space prediction within the chosen training setup.

Masking strategy.

Multi-block targets outperform the compared random, single-block and quadrant strategies. The result concerns the combined information available in context and targets: simply hiding more pixels does not reproduce the successful recipe.

10 Conclusion

I-JEPA demonstrates efficient masked-feature pretraining with useful semantic and local image features. It is an image representation-learning result; downstream labels, task-specific probes and the chosen masking design remain part of its empirical story.

Appendix A Implementation Details

A.1 Pretraining

Architectures.

The predictor has width 384 and grows in depth with the encoder. Pretraining omits a classification token; evaluation averages target-encoder patch features. This distinguishes the representation being evaluated from a generic classification-token output.

Optimization.

AdamW trains the context encoder and predictor using a warmup and cosine learning-rate schedule. The default batch is 2048. Target-encoder averaging becomes progressively slower during training; the appendix specifies the learning-rate, weight-decay and momentum schedules.

Masking.

Four independently sampled target blocks overlap a large initial context mask; their overlap is removed from context. Equal token counts within each device’s batch make processing efficient. The data loader supplies masks as patch indices.

A.2 Downstream Tasks

Linear evaluation.

Frozen evaluation searches over the last layer or the last four layers and over alternative linear heads. I-JEPA uses pooled patch features where comparison methods may use classification tokens. Counting and depth avoid random cropping because it can remove the objects whose number or distance must be predicted.

ImageNet evaluations.

ImageNet linear probing follows a LARS-based recipe with a large batch and a small hyperparameter sweep. The encoder remains fixed, so this experiment measures information accessible through a shallow classifier.

Low-shot evaluation.

Low-shot evaluation fine-tunes the pretrained model for 50 epochs with AdamW, label smoothing and selected image augmentations. The absence of extensive view augmentations refers to pretraining; downstream adaptation still uses an explicitly chosen recipe.

Appendix B Broader Related Work

The broader discussion distinguishes information-maximizing regularization, architectural anti-collapse mechanisms and masked reconstruction. Maximizing retained information need not prefer task-relevant information over noise. I-JEPA’s position-conditioned predictor is a different local-prediction construction from contrastive patch discrimination.

Appendix C Additional Ablations

The extra ablations use a frozen linear probe on 1% of ImageNet labels. Their scores should be read separately from the main low-shot results that may use end-to-end fine-tuning.

Multiblock masking strategy.

Several sufficiently large target blocks and a broad initial context work best in these sweeps. Smaller contexts or fewer targets reduce accuracy, showing that the geometry of the prediction problem strongly affects learning.

Targets Context
Scale Freq. Scale Top-1
(0.075, 0.2) 4 (0.85, 1.0) 19.2
(0.1, 0.2) 4 (0.85, 1.0) 39.2
(0.125, 0.2) 4 (0.85, 1.0) 42.4
(0.15, 0.2) 4 (0.85, 1.0) 54.2
(0.2, 0.25) 4 (0.85, 1.0) 38.9
(0.2, 0.3) 4 (0.85, 1.0) 33.6
Table 8: Ablation of the target block size for multi-block masking. Linear evaluation on 1% ImageNet-1K (using only 1% of the available labels); ablating the multi-block target size during I-JEPA pretraining of a ViT-B/16 for 300 epochs. Predicting larger (semantic) blocks improves the low-shot accuracy as long as the context is sufficiently informative.
Targets Context
Scale Freq. Scale Top-1
(0.15, 0.2) 4 (0.40, 1.0) 31.2
(0.15, 0.2) 4 (0.65, 1.0) 47.1
(0.15, 0.2) 4 (0.75, 1.0) 49.3
(0.15, 0.2) 4 (0.85, 1.0) 54.2
Table 9: Ablation of the context size for multi-block masking. Linear evaluation on 1% ImageNet-1K (using only 1% of the available labels); ablating the multi-block target size during I-JEPA pretraining of a ViT-B/16 for 300 epochs. Reducing the multi-block context size degrades the low-shot performance.
Targets Context
Scale Freq. Scale Top-1
(0.15, 0.2) 1 (0.85, 1.0) 9.0
(0.15, 0.2) 2 (0.85, 1.0) 22.0
(0.15, 0.2) 3 (0.85, 1.0) 48.5
(0.15, 0.2) 4 (0.85, 1.0) 54.2
Table 10: Ablation of the targets number for multi-block masking. Linear evaluation on 1% ImageNet-1K (using only 1% of the available labels); ablating the multi-block number of targets during I-JEPA pretraining of a ViT-B/16 for 300 epochs. Increasing the number of target blocks improve the low-shot accuracy.

Masking at the output of the target-encoder.

Selecting targets after full-image encoding outperforms separately encoding isolated target crops. The context available to the target encoder influences what its features ask the predictor to learn.

Target Masking Arch. Epochs Top-1
Output ViT-H/16 300 67.3
Input ViT-H/16 300 56.1
Table 11: Ablating masking output of target encoder. Linear evaluation on ImageNet-1K using only 1% of the available labels; ablating the effect of masking the target-encoder output during I-JEPA pretraining of a ViT-H/16 for 300 epochs. Masking the output of the target-encoder during pretraining significantly improves the linear probing performance of the pretrained representations.

Predictor depth.

A 12-layer predictor improves the tested low-shot probe over a 6-layer predictor. This is a specific capacity ablation, not evidence that arbitrary additional depth always helps.

Predictor Depth Arch. Epochs Top-1
6 ViT-L/16 500 64.0
12 ViT-L/16 500 66.9
Table 12: Ablating the predictor depth. Linear evaluation on ImageNet-1K using only 1% of the available labels; ablating the effect of masking the predictor depth for a ViT-L/16 pretrained for 500 epochs. Increasing the predictor depth leads to significant improvement of the linear probe performance of the pretrained representations.

Weight decay.

Lower fixed weight decay improves low-shot fine-tuning but worsens full-label linear evaluation. The preferred pretraining recipe depends partly on which downstream protocol is valued.

Weight Decay Arch. Epochs ImageNet-1% ImageNet Linear-Eval
0.04→0.40.04\rightarrow 0.4 ViT-L/16 600 69.4 77.8
0.050.05 ViT-L/16 600 70.7 76.4
Table 13: Ablating the pretraining weight-decay. We compare our default pretraining weight decay strategy where we linearly increase the weight-decay from 0.040.04 to 0.40.4 to using a fix weight decay of 0.050.05. Using a smaller weight-decay during pretraining can improve the fine-tuning performance on ImageNet-1%, However, it also leads to a drop of performance in linear evaluation.

Predictor width.

A narrow 384-channel predictor outperforms a 1024-channel predictor in the reported fine-tuning comparison. A bottleneck can improve learning rather than merely save computation.

Predictor Width Arch. Epochs Top-1
384 ViT-L/16 600 70.7
1024 ViT-L/16 600 68.4
Table 14: Ablating the predictor width. We reports results on ImageNet-1K 1% using fine-tuning. We compare two predictors having a width of either 384384 or 10241024. Note the I-JEPA encoder is a ViT-L with 10241024 channels. Having a width bottleneck in the predictor improves the downstream performances.

Appendix D Finetuning on the full ImageNet

With full ImageNet fine-tuning, the high-resolution I-JEPA model reaches 87.1% top-1 accuracy, compared with 87.8% for the cited MAE configuration. Pretraining schedules differ substantially; the result demonstrates competitive adaptation rather than a matched-compute universal ranking.

Method Arch. Epochs Top-1
MAE [36] ViT-H/14448 1600 87.8
I-JEPA ViT-H/16448 300 87.1
Table 15: Finetuning on the full ImageNet dataset. I-JEPA achieves competitive performance. I-JEPA is close to MAE approach despite I-JEPA being trained for 5.35.3 times less epochs than MAE.

Appendix E RCDM Visualizations

RCDM learns a separate diffusion decoder conditioned on frozen representation vectors. Sampling it with different noise seeds visualizes what those vectors constrain. Its training and generated images must not be mistaken for an image-generation capability trained inside I-JEPA.

E.1 Encoder Visualization

The I-JEPA visualizations tend to retain object identity, pose and layout while varying fine detail and background. The MSN comparison varies more of the object’s local structure. These examples are qualitative probes affected by a separately learned decoder, rather than direct reconstructions or definitive measurements of every represented feature.

Refer to caption
Figure 7: Visualization of I-JEPA target-encoder representations. For each image: first column contains the original image; subsequent columns contain samples from a generative model decoding the average-pooled output of a pretrained I-JEPA target-encoder. Qualities that are common across samples represent information that contained is in the I-JEPA representation. I-JEPA is able to correctly capture the high-level information regarding objects and their poses. Qualities that vary across samples represent information that is not contained in the representation. I-JEPA encoder discards the precise low-level details as well as background information.
Refer to caption
Figure 8: Visualization of MSN target-encoder representations. For each image: first column contains the original image; subsequent columns contain samples from a generative model decoding the output of a frozen MSN encoder [4]. Qualities that are common across samples represent information that is contained in the representation. Qualities that vary across samples represent information that is not captured by MSN. Compared to I-JEPA, MSN samples show higher variability. MSN retains less information from the input. In particular, it discards global structure information such as the object pose or even number of instances.
Concept demonstration

Open image ↗