At a glance
Intuition. If a digit may turn left or right after the visible frames, one prediction cannot specify which random turn actually occurs. VJ-VCR separates the predictable representation from an extra variable that selects an alternative future. Variance–covariance penalties keep the learned representation varied instead of allowing every clip to become the same vector.
This is a separate experimental video-JEPA branch, not a new name for the main V-JEPA release. It compares latent prediction with pixel reconstruction on small synthetic datasets. Its action-recognition evaluation infers the extra variable using observed target frames; it does not forecast an unknowable future action from the past alone.
1 Introduction
The study asks whether predicting video embeddings retains useful dynamics more efficiently than optimizing pixel reconstruction. Its main changes are an explicit variance–covariance collapse-prevention loss and optional constrained latent variables for unpredictable events. Speed probes, inferred-action probes and embedding spectra assess different kinds of information; improved reconstruction quality is not assumed to imply a better dynamics representation.
2 Related Literature
Joint-embedding Predictive Architectures
JEPA predicts an encoded target from an encoded context. A pixel decoder is optional. The intended advantage is to spend representation capacity on useful shared structure rather than require every target pixel to be predictable.
Representation Collapse
A constant vector would be easy to predict but useless. Related joint-embedding methods avoid this with negative examples, asymmetric training or direct representation regularization. This paper uses coordinate variation and decorrelation as explicit penalties.
Video Representation Learning
The work builds on video prediction and temporal representation learning. Unlike the main V-JEPA design’s momentum teacher and stop-gradient asymmetry, VJ-VCR directly regularizes a shared encoder. The experiments establish a result for the tested synthetic settings, not a comprehensive comparison of large video-model families.
3 Method
The encoder represents context and target frames; a predictor maps context representations toward target representations. A variance term discourages constant coordinates and a covariance term discourages redundant coordinates. Optional latent variables select uncertain outcomes, and an optional decoder makes the representation’s retained information visible.
3.1 The VJ-VCR Model
A shared encoder produces hidden representations of the input and target clips. The predictor maps the input representation toward the target. When the future has alternatives, an additional latent variable selects among them. A decoder can reconstruct pixels from the prediction, either during joint training or after freezing the learned representation. These components separate representation quality, uncertainty handling and visual reconstruction.
3.2 Variance-Covariance Regularization
The variance penalty raises each coordinate’s standard deviation toward a threshold across examples, separately by time step. The covariance penalty discourages different coordinates from carrying the same variation. Both act on the encoder’s top-layer embeddings for context and target frames. This prevents particular forms of representational collapse; it does not itself specify which varying features are useful for a downstream task.
Equations 1–4
| (1) |
| (2) |
| (3) |
| (4) |
3.3 Training and Inference
The training energy combines embedding prediction error and variance–covariance regularization, optionally adding pixel reconstruction. Unless stated otherwise, the reconstruction coefficient is zero. The optional latent is optimized together with the representation-learning procedure to accommodate future changes that context cannot determine.
Equation 5
| (5) |
Inference
To infer a latent variable at evaluation, the method receives both context and observed target frames, freezes the trained model and minimizes their compatibility energy over the latent. This explains an observation after seeing it; it is different from predicting which random future will occur without observing that future.
Equation 6
| (6) |
4 Experimental Setup
4.1 Generative Model Baseline
The generative baseline uses comparable encoder and predictor components but normally includes a decoder and learns through pixel reconstruction. Variance–covariance regularization and uncertainty latents can also be added, allowing the comparisons to distinguish the prediction space from those additional choices.
Equation 7
| (7) |
4.2 Datasets
The datasets separate deterministic motion from random future events. This distinction determines whether the evaluation reads predictable speed from an embedding or explains an observed event through an inferred latent.
Deterministic Setting
Deterministic MovingMNIST uses constant digit motion and predictable wall bounces. CLEVRER contains simulated collisions; the chosen clip pairs exclude new objects entering the target segment. These settings support evaluating predicted motion from past observations.
Non-deterministic Setting
The stochastic MovingMNIST variant introduces a random direction switch after the context. CATER contains object actions that can begin randomly. Context alone does not determine these outcomes, so the latent mechanism is evaluated as a representation of the observed future change.
4.3 Evaluation
Frozen-model evaluations use linear probes for speed or action information, train decoders to visualize what embeddings retain, and inspect their effective dimensionality. These measurements answer different questions and should not be treated as interchangeable model-quality scores.
4.3.1 Deterministic Setting
For deterministic clips, a trained linear regression reads object speed from predicted future embeddings, with encoder and predictor frozen. It targets the digit speed or the fastest object’s speed in the final target frame.
4.3.2 Non-deterministic Setting
MovingMNIST
The random turn can be represented by a five-way discrete choice or a sparse 20-dimensional vector. Inference uses context and target observations to select a discrete branch or optimize a sparse latent. Linear probes test whether this latent captures the turn while avoiding static digit identity. Sparsity controls a tradeoff between capturing the event and leaking other target information.
CATER
On CATER, training supplies time-indexed ground-truth action indicators to the predictor through the latent. At evaluation, a latent is inferred from context and target frames, then a linear classifier predicts the observed actions. This is an action-recognition evaluation with action information in pretraining, rather than wholly label-free future-action anticipation. The source’s displayed explanation of average precision is internally inconsistent; interpret the reported metric alongside the implementation and benchmark protocol.
4.4 Architecture
MovingMNIST uses a convolutional encoder and a multilayer predictor. CLEVRER and CATER use a SimVP encoder and Swin Transformer predictor. Their context and target lengths differ: 6 → 20 frames for CLEVRER and 50 → 50 for CATER after temporal subsampling. A decoder mirrors the encoder when visual reconstruction is required.
4.5 Implementation and Hardware
The implementation builds on OpenSTL and PyTorch. Reported experiments fit on one Nvidia RTX 8000 accelerator, with each run taking less than 48 hours.
| Model | Loss() | Loss() | VCR | MovingMNIST | CLEVRER | ||
|---|---|---|---|---|---|---|---|
| MSE in | PSNR | MSE in | RankMe | ||||
| VJ-VCR w/o Decoder | ✓ | ✗ | ✓ | 0.04 | 19.5 | 0.19 | 423.7 |
| VJ-VCR with Decoder | ✓ | ✓ | ✓ | 0.04 | 21.2 | 0.19 | 359.8 |
| Generative with VCR | ✗ | ✓ | ✓ | 0.10 | 22.9 | 0.22 | 427.4 |
| Generative w/o VCR | ✗ | ✓ | ✗ | 0.15 | 22.8 | 0.23 | 160.2 |
5 Results
5.1 Deterministic Setting: Speed Probing
The deterministic experiments compare what predicted representations reveal about motion, separately from whether their pixels can be reconstructed sharply.
MovingMNIST (deterministic)
On MovingMNIST, both embedding-prediction variants reach speed-probe mean squared error 0.04, compared with 0.10 for pixel prediction plus regularization and 0.15 for pixel prediction alone. The reconstruction ranking differs. Adding a decoder to the JEPA objective improves visual detail in this experiment without changing the best speed-probe error.
CLEVRER
On CLEVRER, the tested JEPA variants also give better speed probes than the generative baselines. RankMe suggests that regularization spreads information across more embedding directions, but the ranking by effective dimension does not simply reproduce the ranking by speed error. Representation diversity is one diagnostic, not the task itself.
5.2 Non-deterministic Setting: Action Recognition
Stochastic-setting evaluations test whether inferred latents encode the random turn or the observed set of actions. Their inference procedure has access to the target clip.
MovingMNIST (non-deterministic)
A discrete latent identifies the random direction switch at about 80% accuracy while carrying little digit identity. Sparse latents identify it more accurately, but also leak static identity: weakening sparsity increases both types of information. Visualizations show that including an inferred latent selects a particular future, while omitting it mixes alternatives. The results motivate more precise control over what the extra variable may contain.
| Latent Type | digit | |
|---|---|---|
| discrete (one-hot vector of dimension 5) | 79.7% | 11.4% |
| sparse: high (80%) sparsity | 94.7% | 31.6% |
| sparse: low (20%) sparsity | 99.5% | 57.6% |
CATER
For CATER’s inferred-latent recognition probe, the paper reports 67.4% mean average precision for VJ-VCR versus 54.8% for its generative comparator, averaged across three seeds. The difference is 12.6 percentage points; the prose’s claim of 13.6 is an arithmetic inconsistency. This comparison concerns the stated supervised-latent training and target-conditioned inference protocol.
6 Analyzing the Information Content of the Learned Representations
The authors examine singular values of encoder embeddings on CATER. A more even spectrum means variation occupies more directions instead of concentrating in a few. VJ-VCR’s spectrum is more balanced than the tested generative model’s, consistent with its regularization objective. This is evidence about dimensional collapse in these embeddings, rather than a theorem that a higher-dimensional representation always generalizes better.
7 Conclusion
The experiments support latent prediction with explicit regularization as a useful video representation-learning design. They also show why uncertainty latents need constraints: a variable intended for unpredictable events can absorb static information. The principal limitation is the reliance on small synthetic datasets; extension to large natural-video settings remains a proposed next step.
Appendix A Datasets
MovingMNIST
MovingMNIST uses separate MNIST training and validation digits. The deterministic version produces 20-frame sequences with a fixed velocity; the stochastic version shows three horizontal-motion frames followed by three frames after a randomly chosen direction change.
CLEVRER
CLEVRER uses simulated object collisions, downsampled to 64 × 64 for these experiments, with the official training and validation split.
CATER
CATER uses the provided max2action subset, resized to 128 × 128 and temporally subsampled from 24 to 8 frames per second. Its action vocabulary has 14 possibilities, with multiple actions allowed in a clip.
Appendix B Training and Evalutation Details
Hyperparameters are selected by grid search on validation loss. The retained settings are therefore part of the comparison and should not be assumed universal across datasets.
B.1 Training Details
MovingMNIST is trained for 100 epochs; CLEVRER and CATER for up to 20. Models use Adam, with task-specific batch sizes, context lengths and regularization coefficients. Reconstruction is enabled only in the variants whose objectives include it. The appendix states the distinct settings needed to reproduce each row.
B.2 Evaluation Details
Frozen representations feed speed regression, random-turn classification or multilabel action classification. The sparse MovingMNIST latent has dimension 20; CATER’s action latent spans actions and target times before aggregation. The appendix gives separate optimization settings for these downstream probes.
Appendix C Additional Visualizations on MovingMNIST
The additional reconstructions illustrate a useful distinction: a model can generate sharper pixels while encoding motion less accessibly. The accompanying comparison should be read together with the speed-probe results, not as an independent quality ranking.