Literature
Display
← All papers

Video Representation Learning with Joint-Embedding Predictive Architectures

Katrina Drozdov · Ravid Shwartz-Ziv · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Contents

At a glance

Intuition. If a digit may turn left or right after the visible frames, one prediction cannot specify which random turn actually occurs. VJ-VCR separates the predictable representation from an extra variable that selects an alternative future. Variance–covariance penalties keep the learned representation varied instead of allowing every clip to become the same vector.

This is a separate experimental video-JEPA branch, not a new name for the main V-JEPA release. It compares latent prediction with pixel reconstruction on small synthetic datasets. Its action-recognition evaluation infers the extra variable using observed target frames; it does not forecast an unknowable future action from the past alone.

1 Introduction

The study asks whether predicting video embeddings retains useful dynamics more efficiently than optimizing pixel reconstruction. Its main changes are an explicit variance–covariance collapse-prevention loss and optional constrained latent variables for unpredictable events. Speed probes, inferred-action probes and embedding spectra assess different kinds of information; improved reconstruction quality is not assumed to imply a better dynamics representation.

Refer to caption
(a) Video JEPA with Variance-Covariance Regularization.
Refer to caption
(b) Generative model.
Figure 1: Models for self-supervised video representation learning. Inputs xx and targets yy denote input and target frames coming from the same video, respectively. The optional latent variable zz is intended to capture information about the targets yy not present in xx. In the case of VJ-VCR, the Decoder module is optional. DD denotes the MSE loss function in the hidden representation space or in the input (pixel) space. VC\mathrm{VC} denotes variance-covariance regularization.

2 Related Literature

Joint-embedding Predictive Architectures

JEPA predicts an encoded target from an encoded context. A pixel decoder is optional. The intended advantage is to spend representation capacity on useful shared structure rather than require every target pixel to be predictable.

Representation Collapse

A constant vector would be easy to predict but useless. Related joint-embedding methods avoid this with negative examples, asymmetric training or direct representation regularization. This paper uses coordinate variation and decorrelation as explicit penalties.

Video Representation Learning

The work builds on video prediction and temporal representation learning. Unlike the main V-JEPA design’s momentum teacher and stop-gradient asymmetry, VJ-VCR directly regularizes a shared encoder. The experiments establish a result for the tested synthetic settings, not a comprehensive comparison of large video-model families.

3 Method

The encoder represents context and target frames; a predictor maps context representations toward target representations. A variance term discourages constant coordinates and a covariance term discourages redundant coordinates. Optional latent variables select uncertain outcomes, and an optional decoder makes the representation’s retained information visible.

3.1 The VJ-VCR Model

A shared encoder produces hidden representations of the input and target clips. The predictor maps the input representation toward the target. When the future has alternatives, an additional latent variable selects among them. A decoder can reconstruct pixels from the prediction, either during joint training or after freezing the learned representation. These components separate representation quality, uncertainty handling and visual reconstruction.

3.2 Variance-Covariance Regularization

The variance penalty raises each coordinate’s standard deviation toward a threshold across examples, separately by time step. The covariance penalty discourages different coordinates from carrying the same variation. Both act on the encoder’s top-layer embeddings for context and target frames. This prevents particular forms of representational collapse; it does not itself specify which varying features are useful for a downstream task.

Equations 1–4
lvar​(H)=1T​d​∑t=1T∑k=1dmax⁡(0,τ−Var⁡(Ht,k)+ε),l_{\mathrm{var}}(\mathrm{H})=\frac{1}{Td}\sum_{t=1}^{T}\sum_{k=1}^{d}\max\Big(0,\tau-\sqrt{\mathrm{Var}(\mathrm{H}_{t,k})+\varepsilon}\Big), (1)
lcov(H)=1T​d∑t=1T∑i≠j[Cov(Ht,:)]i,j2.l_{\mathrm{cov}}(\mathrm{H})=\frac{1}{Td}\sum_{t=1}^{T}\sum_{i\neq j}\Big[\mathrm{Cov}(\mathrm{H}_{t,:})\Big]_{i,j}^{2}. (2)
Cov(Ht,:)=1N−1∑i=1N(hi(t,:)−h¯(t,:))(hi(t,:)−h¯(t,:))T\mathrm{Cov}(\mathrm{H}_{t,:})=\frac{1}{N-1}\sum_{i=1}^{N}(h_{i}^{(t,:)}-\overline{h}^{(t,:)})(h_{i}^{(t,:)}-\overline{h}^{(t,:)})^{\mathrm{T}} (3)
lvcr​(H)=α​lvar​(H)+β​lcov​(H).l_{\mathrm{vcr}}(\mathrm{H})=\alpha l_{\mathrm{var}}(\mathrm{H})+\beta l_{\mathrm{cov}}(\mathrm{H}). (4)

3.3 Training and Inference

The training energy combines embedding prediction error and variance–covariance regularization, optionally adding pixel reconstruction. Unless stated otherwise, the reconstruction coefficient is zero. The optional latent is optimized together with the representation-learning procedure to accommodate future changes that context cannot determine.

Equation 5
EθVJ-VCR​(x,y,z)\displaystyle\mathrm{E}_{\theta_{\text{VJ-VCR}}}(x,y,z) =D⁡(h~y,hy)+lvcr​([hx,hy])+γ​D​(y~,y)\displaystyle=D(\tilde{h}_{y},h_{y})+l_{\mathrm{vcr}}([h_{x},h_{y}])+\gamma D(\tilde{y},y)
=‖Pred⁡(hx,z)−hy‖22+α​lvar​([hx,hy])+β​lcov​([hx,hy])+γ​‖Dec⁡(Pred⁡(hx,z))−y‖22,\displaystyle=\|\mathrm{Pred}(h_{x},z)-h_{y}\|_{2}^{2}+\alpha l_{\mathrm{var}}(\mathrm{[h_{x},h_{y}]})+\beta l_{\mathrm{cov}}(\mathrm{[h_{x},h_{y}]})+\gamma\|\mathrm{Dec}(\mathrm{Pred}(h_{x},z))-y\|_{2}^{2}, (5)

Inference

To infer a latent variable at evaluation, the method receives both context and observed target frames, freezes the trained model and minimizes their compatibility energy over the latent. This explains an observation after seeing it; it is different from predicting which random future will occur without observing that future.

Equation 6
z∗=arg​minz⁡EθVJ-VCR​(x,y,z)=arg​minz⁡(‖Pred⁡(hx,z)−hy‖22+γ​‖Dec⁡(Pred⁡(hx,z))−y‖22).z^{*}=\argmin_{z}\mathrm{E}_{\theta_{\text{VJ-VCR}}}(x,y,z)=\argmin_{z}\big(\|\mathrm{Pred}(h_{x},z)-h_{y}\|_{2}^{2}+\gamma\|\mathrm{Dec}(\mathrm{Pred}(h_{x},z))-y\|_{2}^{2}\Big). (6)

4 Experimental Setup

4.1 Generative Model Baseline

The generative baseline uses comparable encoder and predictor components but normally includes a decoder and learns through pixel reconstruction. Variance–covariance regularization and uncertainty latents can also be added, allowing the comparisons to distinguish the prediction space from those additional choices.

Equation 7
EθGEN​(x,y,z)\displaystyle\mathrm{E}_{\theta_{\mathrm{GEN}}}(x,y,z) =D⁡(y~,y)+lvcr​([hx,hy])=‖Dec⁡(Pred⁡(hx,z))−y‖22+α​lvar​([hx,hy])+β​lcov​([hx,hy])\displaystyle=D(\tilde{y},y)+l_{\mathrm{vcr}}([h_{x},h_{y}])=\|\mathrm{Dec}(\mathrm{Pred}(h_{x},z))-y\|_{2}^{2}+\alpha l_{\mathrm{var}}(\mathrm{[h_{x},h_{y}]})+\beta l_{\mathrm{cov}}(\mathrm{[h_{x},h_{y}]}) (7)

4.2 Datasets

The datasets separate deterministic motion from random future events. This distinction determines whether the evaluation reads predictable speed from an embedding or explains an observed event through an inferred latent.

Deterministic Setting

Deterministic MovingMNIST uses constant digit motion and predictable wall bounces. CLEVRER contains simulated collisions; the chosen clip pairs exclude new objects entering the target segment. These settings support evaluating predicted motion from past observations.

Non-deterministic Setting

The stochastic MovingMNIST variant introduces a random direction switch after the context. CATER contains object actions that can begin randomly. Context alone does not determine these outcomes, so the latent mechanism is evaluated as a representation of the observed future change.

4.3 Evaluation

Frozen-model evaluations use linear probes for speed or action information, train decoders to visualize what embeddings retain, and inspect their effective dimensionality. These measurements answer different questions and should not be treated as interchangeable model-quality scores.

4.3.1 Deterministic Setting

For deterministic clips, a trained linear regression reads object speed from predicted future embeddings, with encoder and predictor frozen. It targets the digit speed or the fastest object’s speed in the final target frame.

4.3.2 Non-deterministic Setting

MovingMNIST

The random turn can be represented by a five-way discrete choice or a sparse 20-dimensional vector. Inference uses context and target observations to select a discrete branch or optimize a sparse latent. Linear probes test whether this latent captures the turn while avoiding static digit identity. Sparsity controls a tradeoff between capturing the event and leaking other target information.

CATER

On CATER, training supplies time-indexed ground-truth action indicators to the predictor through the latent. At evaluation, a latent is inferred from context and target frames, then a linear classifier predicts the observed actions. This is an action-recognition evaluation with action information in pretraining, rather than wholly label-free future-action anticipation. The source’s displayed explanation of average precision is internally inconsistent; interpret the reported metric alongside the implementation and benchmark protocol.

4.4 Architecture

MovingMNIST uses a convolutional encoder and a multilayer predictor. CLEVRER and CATER use a SimVP encoder and Swin Transformer predictor. Their context and target lengths differ: 6 → 20 frames for CLEVRER and 50 → 50 for CATER after temporal subsampling. A decoder mirrors the encoder when visual reconstruction is required.

4.5 Implementation and Hardware

The implementation builds on OpenSTL and PyTorch. Reported experiments fit on one Nvidia RTX 8000 accelerator, with each run taking less than 48 hours.

Table 1: Evaluation of self-supervised models trained on MovingMNIST and CLEVRER with two different types of losses, namely prediction loss in the hidden space, Loss(hy)(h_{y}), and reconstruction loss in pixel space, Loss(y)(y), and with or without variance-covariance regularization (VCR). In the case of MovingMNIST, we report the MSE of a linear regression predicting the speed of the moving digit and the average reconstruction quality in terms of PSNR. In the case of CLEVRER, we report the MSE of a linear regression predicting the speed of the fastest object in the last predicted frame as well as the estimated rank of the learned hidden representations in terms of RankMe. All metrics are measured on the validation set.
Model Loss(hyh_{y}) Loss(yy) VCR MovingMNIST CLEVRER
MSE in vv ↓\downarrow PSNR ↑\uparrow MSE in vv ↓\downarrow RankMe ↑\uparrow
VJ-VCR w/o Decoder ✓ ✗ ✓ 0.04 19.5 0.19 423.7
VJ-VCR with Decoder ✓ ✓ ✓ 0.04 21.2 0.19 359.8
Generative with VCR ✗ ✓ ✓ 0.10 22.9 0.22 427.4
Generative w/o VCR ✗ ✓ ✗ 0.15 22.8 0.23 160.2

5 Results

5.1 Deterministic Setting: Speed Probing

The deterministic experiments compare what predicted representations reveal about motion, separately from whether their pixels can be reconstructed sharply.

MovingMNIST (deterministic)

On MovingMNIST, both embedding-prediction variants reach speed-probe mean squared error 0.04, compared with 0.10 for pixel prediction plus regularization and 0.15 for pixel prediction alone. The reconstruction ranking differs. Adding a decoder to the JEPA objective improves visual detail in this experiment without changing the best speed-probe error.

Original figure
Figure 2: Multi-label action recognition performed on the CATER dataset. The aggregated set of actions aya_{y} in the target frames is predicted from the inferred latent variable z∗z^{*} using a linear classifier. Latent variables z∗z^{*} computed from our VJ-VCR pre-trained model are more informative about the underlying actions than those from the pre-trained generative-based models using mAP as an evaluation metric on the validation set. The performance of a linear classifier trained on top of randomly generated latent variables z∗z^{*} in this multi-label setting is 39.6%39.6\%.

CLEVRER

On CLEVRER, the tested JEPA variants also give better speed probes than the generative baselines. RankMe suggests that regularization spreads information across more embedding directions, but the ranking by effective dimension does not simply reproduce the ranking by speed error. Representation diversity is one diagnostic, not the task itself.

5.2 Non-deterministic Setting: Action Recognition

Stochastic-setting evaluations test whether inferred latents encode the random turn or the observed set of actions. Their inference procedure has access to the target clip.

MovingMNIST (non-deterministic)

A discrete latent identifies the random direction switch at about 80% accuracy while carrying little digit identity. Sparse latents identify it more accurately, but also leak static identity: weakening sparsity increases both types of information. Visualizations show that including an inferred latent selects a particular future, while omitting it mixes alternatives. The results motivate more precise control over what the extra variable may contain.

Table 2: Evaluation of latent variables from VJ-VCR pre-trained on our non-deterministic MovingMNIST dataset. We report the average accuracy of predicting switches in trajectory ψ\psi from the inferred latent z∗z^{*} on the validation set in the case when zz is modeled as a discrete or a sparse latent variable with different levels of sparsity regularization. We also report the accuracy of a linear classifier trained to predict the digit identity from the inferred z∗z^{*}.
Latent Type z∗→ψz^{*}\to\psi z∗→z^{*}\to~digit
discrete (one-hot vector of dimension 5) 79.7% 11.4%
sparse: high (80%) sparsity 94.7% 31.6%
sparse: low (20%) sparsity 99.5% 57.6%
Refer to caption
(a) Reconstructions from a VJ-VCR model trained with a latent variable.
Refer to caption
(b) Reconstructions from a VJ-VCR model trained without a latent variable.
Figure 3: Reconstructions from our VJ-VCR model trained with and without a latent variable on the MovingMNIST dataset with a random switch in the digit trajectory after the third frame. The first three columns show the original target frames, the last three columns show the model’s predictions for the target frames and the middle three columns show the overlap between the original and predicted frames (the latter are displayed in green). The model that does not incorporate a latent variable predicts all possible switches in trajectory of the digit, while the one that uses a latent variable can correctly identify the actual switch in digit trajectory.

CATER

For CATER’s inferred-latent recognition probe, the paper reports 67.4% mean average precision for VJ-VCR versus 54.8% for its generative comparator, averaged across three seeds. The difference is 12.6 percentage points; the prose’s claim of 13.6 is an arithmetic inconsistency. This comparison concerns the stated supervised-latent training and target-conditioned inference protocol.

Original figure
(a) Distribution of the singular values of the hidden representations over the validation set coming from a VJ-VCR and generative-based models pre-trained on CATER in a self-supervised way. The singular values of the VJ-VCR model are more uniformly distributed than those of coming from the generative-based model.
Original figure
(b) Cumulative explained variance of hidden representations coming from a VJ-VCR and a generative model at different points during training (beginning, middle, and end). The curve of the VJ-VCR model at the last epoch is rising slower than that of the generative-based model, indicating a lower level of dimensional collapse.
Figure 4: Analysis of the informational content of the learned hidden representations of a VJ-VCR and a generative model through singular value decomposition.

6 Analyzing the Information Content of the Learned Representations

The authors examine singular values of encoder embeddings on CATER. A more even spectrum means variation occupies more directions instead of concentrating in a few. VJ-VCR’s spectrum is more balanced than the tested generative model’s, consistent with its regularization objective. This is evidence about dimensional collapse in these embeddings, rather than a theorem that a higher-dimensional representation always generalizes better.

7 Conclusion

The experiments support latent prediction with explicit regularization as a useful video representation-learning design. They also show why uncertainty latents need constraints: a variable intended for unpredictable events can absorb static information. The principal limitation is the reliance on small synthetic datasets; extension to large natural-video settings remains a proposed next step.

Appendix A Datasets

MovingMNIST

MovingMNIST uses separate MNIST training and validation digits. The deterministic version produces 20-frame sequences with a fixed velocity; the stochastic version shows three horizontal-motion frames followed by three frames after a randomly chosen direction change.

CLEVRER

CLEVRER uses simulated object collisions, downsampled to 64 × 64 for these experiments, with the official training and validation split.

CATER

CATER uses the provided max2action subset, resized to 128 × 128 and temporally subsampled from 24 to 8 frames per second. Its action vocabulary has 14 possibilities, with multiple actions allowed in a clip.

Appendix B Training and Evalutation Details

Hyperparameters are selected by grid search on validation loss. The retained settings are therefore part of the comparison and should not be assumed universal across datasets.

B.1 Training Details

MovingMNIST is trained for 100 epochs; CLEVRER and CATER for up to 20. Models use Adam, with task-specific batch sizes, context lengths and regularization coefficients. Reconstruction is enabled only in the variants whose objectives include it. The appendix states the distinct settings needed to reproduce each row.

B.2 Evaluation Details

Frozen representations feed speed regression, random-turn classification or multilabel action classification. The sparse MovingMNIST latent has dimension 20; CATER’s action latent spans actions and target times before aggregation. The appendix gives separate optimization settings for these downstream probes.

Appendix C Additional Visualizations on MovingMNIST

The additional reconstructions illustrate a useful distinction: a model can generate sharper pixels while encoding motion less accessibly. The accompanying comparison should be read together with the speed-probe results, not as an independent quality ranking.

Refer to caption
Figure 5: Reconstructions from a generative model trained only with loss in pixel space (left) and a VJ-VCR model trained with loss in pixel space, loss in the hidden representation space, and variance-covariance regularization (right). Odd rows display 9 ground truth frames. Even rows display the first 3 ground truth frames which are the input to the model followed by the first 6 (out of 12) reconstructed frames. The model on the left has PSNR of 22.8 and the one on the right has PSNR of 21.2. Both models can predict the trajectories of the digits. Hidden representations from the VJ-VCR model can be used to predict the actual speed of the digits more accurately.
Concept demonstration

Open image ↗