Literature
Display
← All papers

MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features

Adrien Bardes · Jean Ponce · Yann LeCun

Preprint posted · arXiv (preprint) ↗

World models5 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

Recognizing a car and tracking which way it moves are related but different jobs. MC-JEPA trains one encoder to support both. One branch learns optical flow: how pixels correspond between successive frames. Another learns image content by making augmented views of the same image agree. Combining their training losses improves the shared features for motion and segmentation tasks. Unlike the later masked-feature-only V-JEPA recipe, this motion branch also uses photometric reconstruction and geometric consistency losses. The contribution is a tested multi-task representation-learning system, not a universal recipe for learning every aspect of a video or a planner for controlling actions.

1 Introduction

Most image self-supervision emphasizes semantic identity, while optical-flow systems emphasize precise correspondence. MC-JEPA connects them with shared encoder weights. Its motion-only component, M-JEPA, improves a coarse-to-fine flow estimator; combining it with VICReg image training yields MC-JEPA. Evaluation asks whether one encoder can support optical flow, image segmentation and video-object tracking rather than specializing entirely in one task.

Refer to caption
Figure 1: Multi-task self-supervised learning of content and motion features. MC-JEPA combines a self-supervised features learning and optical flow estimation approach in a multi-task setup where with a single shared encoder. The self-supervised learning of content features objective is trained on ImageNet and the self-supervised flow estimation task is trained on various videos datasets. Our final encoder produces features that have motion and content information, and that can be used to estimate optical flow in videos or for content understanding downstream tasks.
Refer to caption
Figure 2: MC-JEPA architecture. Our method learns motion through optical flow estimation on videos and content through joint-embedding of views of images, in a multi-task way with a shared encoder. Our optical flow estimation architecture is based on PWC-Net (Sun et al., 2018) and works as follows. Given a pair of consecutive frames ItI_{t}, It+1I_{t+1} in a video, an encoder produces a set of pyramidal features {Xt(l)}\{X_{t}^{(l)}\} and {Xt+1(l)}\{X_{t+1}^{(l)}\}. The flow is estimated in a coarse-to-fine manner, starting at the lowest resolution features X(1)X^{(1)}. A first flow ft,t+12f_{t,t+1}^{2} is estimated by the flow estimator network, then used to warp the features Xt(2)X_{t}^{{(2)}}, which is compared to Xt+1(2)X_{t+1}^{{(2)}} with a regression loss. The flow is then iteratively refined at every layer by predicting the residual flow and adding it to the previous layer flow. The final flow is used to warp ItI_{t} and compare the warped image with It+1I_{t+1} using a reconstruction loss. Forward-backward flow consistency is encouraged with the cycle consistency losses, which minimizes the distance between Xt(l)X_{t}^{(l)} and ft,t+1(l)​(ft+1,t(l)​(Xt(l)))f_{t,t+1}^{(l)}(f_{t+1,t}^{(l)}(X_{t}^{(l)})) at every layer. When the encoder is trained in the multi-task setup with a standard self-supervised learning criterion, the training is very unstable, which is prevented by the variance-covariance regularization term on every feature layer.

2 Related Work

The related work separates global semantic learning, dense local learning, optical flow and video correspondence. Multi-task learning supplies the connection: motion supervision can improve localization, while semantic features can improve correspondence. The approach does not require that these objectives always agree; their relative weights and training stability are explicit engineering issues.

3 Proposed approach

The method first builds a motion learner, then adds a semantic-learning branch. Both update the same encoder, while retaining their own prediction networks and losses.

3.1 Optical flow

Optical flow assigns a two-dimensional displacement to each pixel. A hierarchical estimator predicts coarse motion and refines it at progressively finer scales, using feature warping and correlations between frames. The losses check whether the predicted movement aligns features and reconstructs the corresponding image. Smoothness regularization helps where texture gives little evidence.

Forward–backward cycle consistency asks whether following predicted motion out and back returns to the original features. Occlusion handling excludes unreliable correspondences from the regression term. Variance–covariance penalties encourage informative, nonredundant encoder features. These pieces work together: low feature-prediction loss alone could otherwise be achieved with uninformative representations.

Equations 1–4
ft,t+1(l+1)=Fθ​(Xt(l),Xt+1(l),ft,t+1(l)).f_{t,t+1}^{(l+1)}=F_{\theta}(X_{t}^{(l)},X_{t+1}^{(l)},f_{t,t+1}^{(l)}). (1)
ℒreg=∑l=1L‖Xt+1(l)−X^t+1(l)‖22,\mathcal{L}_{\mathrm{reg}}=\sum_{l=1}^{L}\|X^{(l)}_{t+1}-\hat{X}_{t+1}^{(l)}\|^{2}_{2}, (2)
ℒrec=d⁡(It+1,I^t+1),\mathcal{L}_{\mathrm{{rec}}}=d(I_{t+1},\hat{I}_{t+1}), (3)
ℒsmooth=∑d∈{x,y}∑pexp⁡(−λ​∇dI)​‖∇dft,t+1‖1,\mathcal{L}_{\mathrm{smooth}}=\sum_{d\in\{x,y\}}\sum_{p}\exp({-\lambda\nabla_{d}I})\|\nabla_{d}f_{t,t+1}\|_{1}, (4)
Table 1: Quantitative results. Comparison of the performance of our model on: (1) Sintel (Butler et al., 2012) clean and final, and KITTI 2015 (Menze & Geiger, 2015) optical flow estimation benchmarks; (2) Pascal VOC (Everingham et al., 2010), Cityscapes (Cordts et al., 2016) and ADE20k (Zhou et al., 2019), both frozen and fine-tune linear segmentation benchmarks; (3) DAVIS-2017 (Pont-Tuset et al., 2017) and video object segmentation benchmark, against several self-supervised methods optimized for a single task specifically. EPE is the average end-point-error (↓\downarrow Lower is better). F1 is the average-f1 error in (%) (↑\uparrow Lower is better). mIoU is the mean intersection-over-union (↑\uparrow Higher is better). (𝒥&ℱ)m(\mathcal{J}\&\mathcal{F})_{m} is the average between mean region similarity and mean contour-based accuracy (↑\uparrow Higher is better). MC-JEPA is our full model trained in multi-task way on ImageNet and flow estimation. M-JEPA is our model without content learning, trained only on flow estimation. The best and second best result for each benchmark are bold and underlined.
Optical Flow Estimation Image Segmentation Video Seg.
Sintel Clean Sintel Final KITTI 2015 Pascal VOC CityScapes ADE20k Davis 2017
Method Backbone train test train test train test Frozen FT Frozen FT Frozen FT
EPE EPE EPE EPE EPE F1 mIoU mIoU mIoU mIoU mIoU mIoU (𝒥&ℱ)m(\mathcal{J}\&\mathcal{F})_{m}
Rand. weights CNX-T 23.71 - 24.02 - 24.88 - 0.5 - - - - - -
flow methods
UFlow (Jonschkowski et al., 2020) PWC 2.50 5.21 3.39 6.50 2.71 11.13 7.8 - - - - - 42.0
ARFLow (Liu et al., 2020) PWC 2.79 4.78 3.73 5.89 2.85 11.80 7.9 - - - - - -
UPFlow (Luo et al., 2021) PWC 2.33 4.68 2.67 5.32 2.45 9.38 8.8 - - - - - -
SMURF (Stone et al., 2021) RAFT 1.71 3.15 2.58 4.18 2.00 6.83 10.4 - - - - - -
correspondence methods
VFS (Xu & Wang, 2021) R50 - - - - - - 51.2 - - - - - 68.9
MCRW (Bian et al., 2022) PWC 2.84 5.68 3.82 6.72 2.81 11.67 39.8 - - - - - 57.9
content methods
VICReg (Bardes et al., 2022a) CNX-T - - - - 13.5 - 60.1 77.8 59.8 76.3 28.6 41.1 58.1
VICRegL (Bardes et al., 2022b) CNX-T - - - - 11.4 - 66.8 79.7 64.9 78.3 30.6 44.1 66.7
MoCo v3 (Chen et al., 2021) ViT-S - - - - 12.9 - 57.1 75.9 56.5 74.0 23.7 39.8 -
DINO (Caron et al., 2021) ViT-S - - - - 11.8 - 65.2 79.5 64.8 78.1 30.5 43.5 69.9
ours
M-JEPA CNX-T 2.98 - 3.82 - 3.01 - 9.4 - - - - - -
MC-JEPA CNX-T 2.81 5.01 3.51 6.12 2.67 11.33 67.1 79.9 65.5 78.4 30.8 44.2 70.5
Equations 5–6
ℒcycle=∑l=1L‖Xt(l)−ft+1,t​(ft,t+1​(Xt(l)))‖22,\mathcal{L}_{\mathrm{cycle}}=\sum_{l=1}^{L}\|X_{t}^{(l)}-f_{t+1,t}(f_{t,t+1}(X_{t}^{(l)}))\|_{2}^{2}, (5)
ℒvc=∑l=1L1d​∑j=1dmax⁡(0,γ−Var⁡(Xt,j(l))+ϵ)+1d∑i≠j[C(Xt(l))]i,j2.\begin{split}\mathcal{L}_{\mathrm{vc}}&=\sum_{l=1}^{L}\frac{1}{d}\sum_{j=1}^{d}\max(0,\gamma-\sqrt{\mathrm{Var}(X_{t,j}^{(l)})+\epsilon})\\ &+\frac{1}{d}\sum_{i\neq j}[C(X_{t}^{(l)})]_{i,j}^{2}.\end{split} (6)

3.2 Multi-task self-supervised learning

A second branch applies VICReg to two randomly cropped and color-transformed views of an ImageNet image. For each training iteration, one image batch and one video batch provide their respective losses; their weighted sum updates the shared encoder. Task-specific coefficients balance content invariance against precise motion. MC-JEPA therefore uses both image and video datasets, rather than extracting every training signal from a single video collection.

Equation 7
∑𝒟1ℒrec+ℒreg+ℒsmooth+ℒcycle+ℒvc+∑𝒟2ℒssl,\sum_{\mathcal{D}_{1}}\mathcal{L}_{\mathrm{{rec}}}+\mathcal{L}_{\mathrm{reg}}+\mathcal{L}_{\mathrm{smooth}}+\mathcal{L}_{\mathrm{cycle}}+\mathcal{L}_{\mathrm{vc}}+\sum_{\mathcal{D}_{2}}\mathcal{L}_{\mathrm{ssl}}, (7)
Refer to caption
Figure 3: Qualitative visualization: optical flow. We compare our results of our complete model (MC-JEPA) and our model only pretrained on flow (M-JEPA) with ARFlow. Top 2 rows are from KITTI-15, bottom 2 rows are from Sintel clean and Sintel final.
Refer to caption
Figure 4: Qualitative visualization: video segmentation. We visualize the segmentation maps obtained by the frozen features learnt with MC-JEPA on the video instance tracking task on DAVIS 2017, for several video sequences, at frames t=1,10,25,50. Frame 1 is given as ground truth, and the others are predicted by our model.

4 Experiments

4.1 Datasets

Pretraining combines ImageNet-1K with synthetic and real optical-flow datasets, including KITTI, Sintel, FlyingChairs, FlyingThings and HD1K. Evaluation covers flow error, frozen and fine-tuned image segmentation, and DAVIS video segmentation. The differing metrics measure complementary capabilities and should not be collapsed into one generic accuracy claim.

4.2 Main results

MC-JEPA reaches competitive flow quality while supporting strong image and video segmentation with the same encoder. It improves over its VICReg content-only baseline on localization tasks. The specialized SMURF flow method remains better on the reported flow benchmarks; MC-JEPA’s advantage is breadth across the tested tasks, not best-in-class optical flow alone.

4.3 Ablations

The ablations expose a multi-task tradeoff. More diverse flow data primarily improves flow metrics, whereas semantic benefits are less domain-specific. Layer normalization stabilizes the estimator, and the modified ConvNeXt backbone works better here than the compared alternatives.

Training flow only above frozen image features is restrictive; optimizing the combined objectives or alternating at batch scale is more effective than large alternating phases. Starting the flow task after an initial content-learning warmup saves work and improves the tested recipe. Cycle consistency helps, but its coefficient matters. Too much motion-loss weight damages segmentation, so sharing an encoder does not remove the need to balance objectives.

Table 2: Ablation: flow datasets. Impact on performance when varying the set of pretraining datasets. KITTI means pretraining on KITTI raw, 2012 and 2015. Sintel means pretraining Sintel raw, clean and final. FT/FC are FlyingThings and FlyingChairs. The metric for K15 (KITTI 2015), clean and final is the EPE. ISeg is the linear frozen evaluation on Pascal VOC, in mIoU, VSeg is the evaluation on DAVIS 2017, in (𝒥&ℱ)m(\mathcal{J}\&\mathcal{F})_{m}.
KITTI Sintel FT/FC HD1k K15 clean final ISeg VSeg
✓ 2.93 3.23 3.96 66.8 70.0
✓ 3.78 2.95 3.61 66.4 69.9
✓ ✓ 2.91 2.99 3.70 67.2 70.4
✓ ✓ ✓ 2.88 2.93 3.66 67.1 70.3
✓ ✓ ✓ ✓ 2.67 2.81 3.51 67.1 70.5
Table 3: Ablation: estimator architecture. Comparison between different flow estimator size form of normalization. The factor size influences the number of filters in each convolution of the estimator. LN means layer norm means usage of layer norm after every layer of the estimator, except the last one. l2 means l2-normalization before the last layer of the estimator.
Factor size #Params LN l2 K15 clean final ISeg VSeg
1 2M crashed
1 2M ✓ 2.68 2.88 3.57 67.0 70.2
1 2M ✓ 6.21 6.04 6.99 53.2 47.9
1 2M ✓ ✓ 4.55 4.47 5.66 62.3 63.6
2 8M ✓ 2.67 2.81 3.51 67.1 70.5
Table 4: Ablation: backbone. Comparison of the performance of MC-JEPA when using different backbones.
Backbone #Params K15 clean final ISeg VSeg
PWC-Net 8M 2.66 2.80 3.47 14.8 10.1
ResNet-50 21M 2.71 2.85 3.59 55.8 60.1
ConvNeXt-T 23M 2.67 2.81 3.51 67.1 70.5
Table 5: Ablation: data sampling. Comparison between different training order and data sampling strategies.
Strategy K15 clean final ISeg VSeg
Flow estimator training 13.52 13.82 14.81 60.1 65.2
Flow estimator fine-tuning 2.71 2.82 3.77 61.3 62.3
Epoch alternation 4.54 4.91 5.57 63.5 66.9
Batch alternation 2.78 2.95 3.62 67.1 70.5
Combined loss 2.67 2.81 3.51 67.1 70.5
Original figure
Original figure
Original figure
Figure 5: (1) Ablation: flow start epoch. Flow estimation performance as a function of the ImageNet training epoch from which flow estimation starts. There are 100 pretraining epochs in total. (2) Ablation: cycle consistency coefficient. Flow estimation performance as a function of the coefficient used to balance the cycle consistency loss of Eq (5). (3) Ablation: multi-task balancing coefficient. Flow estimation and segmentation performance as a function of the balancing coefficient between flow losses and SSL loss in Eq (7).

5 Conclusion

Joint content and motion learning produces useful representations across the evaluated flow and segmentation tasks. Extending this result to larger natural-video collections, a shared training domain and longer-range dynamics remains future work. No action-conditioned planning result is claimed here.

A Implementation details

The training mixture samples video datasets throughout the run instead of repeatedly fine-tuning to each benchmark. Image self-supervision starts first, followed by flow learning. Layer normalization, bounded flow outputs and separate optimizer settings prevent unstable flow predictions from corrupting the combined loss.

A modified ConvNeXt-T encoder supplies six feature-pyramid levels. The reported recipe uses AdamW and eight V100 GPUs; the retained hyperparameter tables provide the exact batch sizes, schedules and separate encoder/estimator learning rates.

B Hyper-parameters

The appendix records dataset repetitions, input resolutions and loss coefficients. Motion-only tuning establishes flow settings before content-learning settings are tuned. Deeper feature levels receive stronger variance–covariance regularization. These tuned choices are part of the result and should be retained when attempting reproduction.

Table 6: Hyper-parameters. List of all the hyper-parameters used for MC-JEPA and M-JEPA training. The values that are noted ”-” indicate that the corresponding parameter is not used.
Hyper-parameter MC-JEPA M-JEPA
data
ImageNet res. (224, 224) -
min_scale_crops 0.08 -
max_scale_crops 1.0 -
Sintel res. (384, 832) (384, 832)
KITTI res. (256, 832) (256, 832)
FlyingX res. (384, 512) (384, 512)
optimization
num_gpus 8 8
epochs 100 100
warmump_epochs 10 10
batch_size 384 -
optimizer AdamW AdamW
lr 3×10−43\text{\times}{10}^{-4} -
scheduler cosine cosine
end_lr 3×10−83\text{\times}{10}^{-8} -
weight_decay 1×10−61\text{\times}{10}^{-6} -
betas (0.9, 0.999) (0.9, 0.999)
architecture
drop_path_rate 0.1 0.1
layer_scale_init_value 0.0 0.0
expander dims. 8192-8192-8192 -
flow
flow_alpha 0.1 1.0
flow_coeff 1.0 1.0
flow_clip_value 128.0 256.0
flow_start_epoch 10 0
flow_loss_smooth_factor 75.0 75.0
flow_cycle_consistency_coeff 0.2 0.1
flow_batch_size 8 8
flow_lr 1×10−41\text{\times}{10}^{-4} 1×10−41\text{\times}{10}^{-4}
flow_weigth_decay 1×10−61\text{\times}{10}^{-6} 1×10−61\text{\times}{10}^{-6}
Table 7: Flow datasets. List of all the datasets that we use for training the flow estimation objective. The size is the total number of pairs on which the optical flow is estimated. Repetition is the number of time the dataset is repeated to build the final dataset from which a random batch is sampled at each iteration.
Dataset Size Repetition
FlyingThings 40302 1
FlyingChairs 22232 1
KITTI raw 42382 1
KITTI 2012 train 200 100
KITTI 2012 multiview train 3800 5
KITTI 2012 val 198 100
KITTI 2012 multiview val 3762 5
KITTI 2015 train 200 100
KITTI 2015 multiview train 3800 5
KITTI 2015 val 198 100
KITTI 2015 multiview val 3762 5
Sintel raw. 27858 1
Sintel clean 1041 5
Sintel final 1041 5
HD1k 1047 5
Table 8: Loss coefficients. The coefficients for each loss at each specific layer of the architecture. Var and Cov are the variance and covariance regularization losses of Eq. (6). Invaraince is the invariance loss of VICReg used only for self-supervised training on ImageNet. Flow is the sum of all the flow losses, without the SSL loss, in Eq. (7). L1 to L6 are the features stages from lowest to highest resolution of our modified ConvNeXt-T backbone.
Layer Var Cov Invariance Flow
Expander output 25.0 1.0 1.0 -
Encoder output (L1) 0.01 0.04 - 1.0
L2 0.01 0.04 - 1.0
L3 0.01 0.001 - 1.0
L4 0.01 0.0 - 1.0
L5 0.001 0.0 - 0.1
L6 0.0001 0.0 - 0.01

C Architecture details

The encoder stem is modified to expose a finer feature pyramid for flow refinement. The flow estimator adds layer normalization after intermediate convolutions and increases channel capacity. Its final output layer avoids normalization, which could otherwise distort the range of physical displacement values. The original diagrams specify the architecture.

Original figure
Figure 6: Backbone stem modifications. The stem convolutional layer in the ConvNeXt-T architecture is replaced in our backbone with two convolutional layers by reducing the kernel size and stride. Blue boxes are convolutional layers. I: input channels, O: output channels, K: kernel_size, S: stride, P: padding.
Original figure
Figure 7: PWC estimator modifications. A layer norm layer is added after each convolutional layer of the PWC estimator architecture, the coefficient CC corresponds to the size factor parameter discussed in Table 3. We use C=2C=2 in practice. Blue boxes are convolutional layers. I: input channels, O: output channels.

D Additional results

Additional tables separate the flow and video-segmentation metrics; the figures show qualitative successes and errors on both tasks. These supplement the numerical comparisons rather than establishing robustness outside the evaluated datasets.

Table 9: Additional optical flow results. We report additionally to Table 1, EPE performance on non-occluded (noc) and occluded (occ) pixels.
KITTI 2015 Sintel Clean Sintel Final
Method all noc occ all noc occ all noc occ
M-JPEA 3.01 2.26 6.98 2.98 1.54 23.99 3.82 2.17 24.68
MC-JEPA 2.67 2.08 6.24 2.81 1.25 23.82 3.51 1.99 24.23
Table 10: Additional video segmentation results. We report additionally to Table 1, mean region similarity 𝒥m\mathcal{J}_{m} and mean contour-based accuracy ℱm\mathcal{F}_{m}.
Method (𝒥&ℱ)m(\mathcal{J}\&\mathcal{F})_{m} 𝒥m\mathcal{J}_{m} ℱm\mathcal{F}_{m}
VICReg 58.1 56.4 59.8
VICRegL 66.7 64.5 68.9
DINO 69.9 66.6 73.1
MC-JEPA 70.5 67.0 74.0

E Additional ablation

Variance–covariance regularization is tested at the final layer and throughout the pyramid. Applying it at every level improves the reported motion and segmentation outcomes. A short regularization warmup also helps. Preventing complete collapse at one layer is therefore not the only concern: intermediate feature quality and stable optimization matter.

Table 11: Ablation: variance-covariance. Influence of variance-covariance regularization (VC) on performance. During warmup epochs, only the variance-covariance criteria are trained; this helps stabilizing the training. VC is applied in the last layer, in the expander output, or in every layer, with carefully chosen coefficients as described in Appendix B.
Warmup ep. Setup K15 clean final ISeg VSeg
- None 3.41 3.37 4.45 47.3 37.8
0 Last layer 2.77 2.88 3.55 65.6 69.2
0 All layers 2.65 2.80 3.48 66.2 69.4
1 All layers 2.67 2.81 3.51 67.1 70.5
2 All layers 2.91 2.99 3.78 62.5 64.1
Refer to caption
Figure 8: Qualitative visualization: optical flow. We compare our results of our complete model (MC-JEPA) and our model only pretrained on flow (M-JEPA) with ARFlow. Top 5 rows are from KITTI-15, bottom 5 rows are from Sintel.
Refer to caption
Figure 9: Qualitative visualization: video segmentation. We visualize the segmentation maps obtained by the frozen features learnt with MC-JEPA on the video instance tracking task on DAVIS 2017, for several video sequences, at frames t=1,10,25,50. Frame 1 is given as ground truth, and the others are predicted by our model.
Concept demonstration

Open image ↗