The overview compares compact world-model planning with larger feature-based baselines. The goal-conditioned evaluation uses recorded future observations as targets.
Figure 1: AD-E2E-JEPA is an action-conditioned, JEPA-based world model for end-to-end autonomous driving. Without training any driving policy, it enables driving via goal-conditioned zero-shot planning, where future sensor observations are specified as the goal and a trajectory is selected from the trajectory vocabulary solely via world-model rollouts. AD-E2E-JEPA demonstrates strong zero-shot driving performance, efficient planning, low displacement errors, high trajectory hit rates, and representations that transfer effectively to downstream imitation learning. Zero-shot planning metrics are evaluated on 100 subsampled test scenes for a fair comparison with the time-consuming DINO-WM/JEPA-WM. See Table 2 for AD-E2E-JEPA’s results on the full set of 12,146 test scenes.
At a glance
Intuition. Searching over future drives is costly when every imagined frame contains hundreds of high-dimensional patch vectors. AD-E2E-JEPA learns a compact projector on frozen DINOv3 features and trains an action-conditioned latent predictor, using SIGReg to discourage collapse of the projected space. It selects from a finite vocabulary of candidate trajectories by comparing their predicted future features with a goal image. Crucially, that goal is the recorded future image: this isolates world-model planning quality but gives privileged goal information, rather than solving unconstrained autonomous driving. The reported large speedup is relative to dense patch models under matched candidate settings. A separate imitation-learning experiment tests transfer of the pretrained projector; driving safety and full EPDMS must be distinguished from the safety-stripped score.
1 Introduction
The paper studies the accuracy–cost tradeoff of JEPA world models for driving. Dense patch representations can plan accurately but slowly, while a small global token can be fast and lose necessary detail. Its projector aims for a middle ground, evaluated through privileged goal-conditioned search and separately through downstream imitation learning.
2 Related Work
2.1 World Models
World models can reconstruct images, predict latent states, or train policies using imagined rollouts. This work follows the latent-prediction route with a frozen visual backbone and learnable compact features. It does not claim that every generative or policy-based world model solves the same evaluation problem.
2.2 End-to-End Autonomous Driving
Earlier driving systems use supervised trajectories, video generation, or JEPA-style representation learning. The distinctive diagnostic here searches with a known future image as its target, separating model-based trajectory ranking from learning a driving policy. That protocol narrows what its zero-shot label establishes.
3 Method
A frozen visual encoder, shared learnable projector and action-conditioned predictor support the planning experiment. The projector is also reused in a separately trained imitation model.
Figure 2: AD-E2E-JEPA architecture: (a) A self-supervised, action-conditioned JEPA world model for E2EAD. The DINOv3 encoder is frozen, while (a.1) a learnable projector is shared by the context-history and target-future branches, compressing the representation by and its dimensionality by , with SIGReg maximizing information while avoiding collapse and improving planning performance. This enables a speedup in zero-shot goal-conditioned planning in (b), where trajectories are selected via world-model rollouts over a driving vocabulary. (c) The projector pretrained through self-supervision also improves downstream imitation learning.
The front-camera setting predicts future planar ego poses relative to the current vehicle. Historical images and poses provide context; actions are pose increments. A safe trajectory must satisfy additional driving constraints beyond matching a future visual embedding.
The dense JEPA-WM baseline predicts frozen DINOv3 patch features using an AdaLN-conditioned predictor. Since its target encoder is fixed, its basic regression target cannot collapse through encoder co-adaptation. This baseline supplies the accuracy and runtime comparison for feature compression.
Two stride-two convolutional layers reduce spatial patch count by 16 and feature dimension by 4. A learned projector creates a new collapse risk even when the underlying encoder is frozen, so the training recipe adds detached targets and SIGReg. Temporal rollout consistency can be added to address self-generated prediction context.
Equations 7–12
(7)
(8)
(9)
(10)
(11)
(12)
3.3 Zero-Shot Goal-Conditioned Planning
The zero-shot experiment means no task-specific driving policy is trained for candidate selection. It still uses training interaction data, a trajectory vocabulary and ground-truth future goal images.
Candidate pose sequences come from a driving-trajectory vocabulary. Each is rolled out through the latent model, and the lowest terminal distance to the encoded goal is selected. Larger candidate vocabularies improve coverage at increased cost; the method is discrete trajectory search, not a general continuous optimal-control solver.
Equations 13–14
(13)
(14)
3.3.2 Metrics
Full EPDMS includes multiplicative safety factors; the daggered version omits them. Endpoint displacement, heading errors and whether the known target trajectory ranks highly diagnose different aspects of model quality. High safety-stripped progress does not establish safe driving.
3.4 Downstream Transfer for Imitation Learning
For imitation learning, the future predictor is removed and the frozen-backbone/projector representation feeds a trainable trajectory decoder. Comparing a pretrained and random projector tests representation transfer separately from goal-conditioned search.
4 Experiments
4.1 Datasets
Matched model comparisons use the 10-hour NAVSIM training split, while a scaled variant uses 70 hours. The larger-data result should be separated from the matched-data architectural comparison.
4.2 Implementation Details
The main recipe uses four context frames, an eight-frame future, DINOv3 features and a shared training schedule where specified. The projector’s compactness reduces training resources as well as planning cost. SIGReg weighting is adjusted heuristically in larger-batch settings.
Dense DINO-WM and JEPA-WM are accurate but slow on the 100-scene subset; the compact model lowers runtime substantially. More data, rollout training and more candidate trajectories improve several metrics. Full-set results are reported only where computationally feasible, so subset and full-set scores must not be mixed.
Figure 3: Zero-shot goal-conditioned planning qualitative comparison. LeWM is efficient but insufficient for accurate planning, resulting in a shifted selected trajectory. DINO-WM and JEPA-WM better match the ground-truth trajectory. AD-E2E-JEPA achieves comparable planning quality to DINO-WM and JEPA-WM while being 100 faster.
Table 2: Zero-shot goal-conditioned planning on the NAVSIM dataset, evaluated on 100 subsampled test scenes and the full set of 12,146 test scenes, with the ground-truth future frame used as the target. Unless specified, the world model selects from 256 candidate trajectories. The NAVSIM-v2 navtest-stage-1 metrics EPDMS and EPDMS measure driving performance, where EPDMS includes safety metrics and EPDMS† excludes them. Per-scene planning time (s) measures efficiency on an A100 GPU. FDE (m), (m), (m), and (degrees) measure geodesic accuracy. Top-1/top-5 hit rate (%) measures world-model reliability. : higher is better; : lower is better. For the 100 subsampled-test scene setting, the extended comfort (EC) metrics for EPDMS and EPDMS† are excluded because the subsampled set does not contain temporally adjacent scenes. See Appendix A.4 for details of each submetric of EPDMS.
Pretraining the projector improves the reported downstream imitation score compared with random projector initialization. Large external driving models are contextual references rather than matched baselines: they differ in pretraining scale, sensors and architectures.
Table 3:
NAVSIMv2 navtest-stage-1 benchmark. V: single-view (SV) or multi-view (MV); Type: perception-free (PF) or perception-based (PB); Fr.: number of input frames. PB methods use perception annotations or trajectory-score labels. ∗ denotes results obtained before the human-filter bug fix in commit 359c7f7 in NAVSIM’s codebase.
The projector offers a favorable cost–quality tradeoff for the stated latent driving-planning protocol and useful transfer in imitation learning. The benchmark is not evidence of safe deployment without future goal images, and its best result uses a larger candidate budget than the fast 256-candidate configuration.
SIGReg uses the retained numerical integration and random-projection settings. A finite projected-distribution test discourages collapse but does not prove that all driving-relevant distinctions remain represented.
A.2 Rollout Setting
The rollout variant extends one-step training across the full future window while maintaining the fixed history length. Teacher-forced and self-generated contexts receive separate losses.
A.2.1 Teacher-Forcing Loss
Teacher forcing supplies real encoded context. The first window supervises all outputs, while later overlapping windows supervise only their newest output to control repeated contributions.
Equations 15–18
(15)
(16)
(17)
(18)
A.2.2 Rollout Consistency Loss
Autoregressive training feeds predicted latents back into a sliding context. Truncated backpropagation detaches this context between calls, controlling derivative depth while still training predictions against future targets.
The Gaussian regularizer covers the retained context and future projected embeddings. It complements prediction consistency, rather than defining driving safety or task success.
A.3 Metrics
The appendix defines each multiplicative safety component and weighted driving-quality component, along with endpoint and trajectory-ranking diagnostics. The distinction between EPDMS and its safety-stripped variant is essential when interpreting the high reported numbers.
Equations 24–28
(24)
(25)
(26)
(27)
(28)
A.4 EPDMS Details for Zero-Shot Goal-Conditioned Planning
Complete component tables are retained for the 100-scene subset and the full 12,146-scene set. They show where aggregate driving scores can conceal failures in individual safety or quality terms.
Table 4: EPDMS details for 100 subsampled test scenes.
The imitation decoder combines visual features, the learned projector, positional/temporal embeddings and a driving command to predict a future trajectory. It is trained with supervised trajectory error. This downstream stage uses a learned policy, unlike the earlier goal-conditioned search evaluation.
Assran et al. (2023)M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. BallasSelf-supervised learning from images with a joint-embedding predictive architecture.
In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 15619–15629.
Assran et al. (2025)M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al.V-jepa 2: self-supervised video models enable understanding, prediction and planning.
arXiv preprint arXiv:2506.09985.
Balestriero and LeCun (2025)R. Balestriero and Y. LeCunLejepa: provable and scalable self-supervised learning without the heuristics.
arXiv preprint arXiv:2511.08544.
Bardes et al. (2024)A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. BallasRevisiting feature prediction for learning visual representations from video.
arXiv preprint arXiv:2404.08471.
Bardes et al. (2021)A. Bardes, J. Ponce, and Y. LeCunVicreg: variance-invariance-covariance regularization for self-supervised learning.
arXiv preprint arXiv:2105.04906.
Bojarski et al. (2016)M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al.End to end learning for self-driving cars.
arXiv preprint arXiv:1604.07316.
Cao et al. (2025)W. Cao, M. Hallgarten, T. Li, D. Dauner, X. Gu, C. Wang, Y. Miron, M. Aiello, H. Li, I. Gilitschenski, et al.Pseudo-simulation for autonomous driving.
arXiv preprint arXiv:2506.04218.
Chen et al. (2024a)L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. LiEnd-to-end autonomous driving: challenges and frontiers.
IEEE Transactions on Pattern Analysis and Machine Intelligence.
Chen et al. (2024b)S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. WangVadv2: end-to-end vectorized autonomous driving via probabilistic planning.
arXiv preprint arXiv:2402.13243.
Chen and He (2021)X. Chen and K. HeExploring simple siamese representation learning.
In 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR),
pp. 15745–15753.
Chitta et al. (2022)K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. GeigerTransfuser: imitation with transformer-based sensor fusion for autonomous driving.
IEEE transactions on pattern analysis and machine intelligence45 (11), pp. 12878–12895.
Dauner et al. (2024)D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, et al.Navsim: data-driven non-reactive autonomous vehicle simulation and benchmarking.
Advances in Neural Information Processing Systems37, pp. 28706–28719.
Finn and Levine (2017)C. Finn and S. LevineDeep visual foresight for planning robot motion.
In 2017 IEEE international conference on robotics and automation (ICRA),
pp. 2786–2793.
Garrido et al. (2025)Q. Garrido, N. Ballas, M. Assran, A. Bardes, L. Najman, M. Rabbat, E. Dupoux, and Y. LeCunIntuitive physics understanding emerges from self-supervised pretraining on natural videos.
arXiv preprint arXiv:2502.11831.
Grill et al. (2020)J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al.Bootstrap your own latent-a new approach to self-supervised learning.
Advances in neural information processing systems33, pp. 21271–21284.
Hu et al. (2023a)A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. CorradoGaia-1: a generative world model for autonomous driving.
arXiv preprint arXiv:2309.17080.
Hu et al. (2023b)Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al.Planning-oriented autonomous driving.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp. 17853–17862.
Jaeger (2002)H. JaegerTutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the echo state network approach.
GMD-Forschungszentrum Informationstechnik Bonn.
Jiang et al. (2023)B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. WangVad: vectorized scene representation for efficient autonomous driving.
In 2023 IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 8306–8316.
Kang et al. (2024)B. Kang, Y. Yue, R. Lu, Z. Lin, Y. Zhao, K. Wang, G. Huang, and J. FengHow far is video generation from world model: a physical law perspective.
arXiv preprint arXiv:2411.02385.
Kirby et al. (2026)E. Kirby, A. Boulch, Y. Xu, Y. Yin, G. Puy, É. Zablocki, A. Bursuc, S. Gidaris, R. Marlet, F. Bartoccioni, et al.Driving on registers.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 32058–32069.
Kuang et al. (2026a)Y. Kuang, Y. Dagade, Q. L. Lidec, L. Maes, R. Balestriero, and Y. LeCunLpWM: a case for sparse representations in world models.
arXiv preprint arXiv:2608.22764.
Kuang et al. (2026b)Y. Kuang, Y. Dagade, T. G. Rudner, R. Balestriero, and Y. LeCunRectified lpjepa: joint-embedding predictive architectures with sparse and maximum-entropy representations.
arXiv preprint arXiv:2602.01456.
Kuhn et al. (2026)L. Kuhn, L. Maes, G. Serra, Q. L. Lidec, Y. LeCun, R. Balestriero, and F. BuettnerLeVJEPA: efficient & scalable video pretraining without the heuristics.
arXiv preprint arXiv:2608.27395.
Li et al. (2025)K. Li, Z. Li, S. Lan, Y. Xie, Z. Zhang, J. Liu, Z. Wu, Z. Yu, and J. M. AlvarezHydra-mdp++: advancing end-to-end driving via expert-guided hydra-distillation.
arXiv preprint arXiv:2503.12820.
Li et al. (2024a)Y. Li, L. Fan, J. He, Y. Wang, Y. Chen, Z. Zhang, and T. TanEnhancing end-to-end autonomous driving with latent world model.
arXiv preprint arXiv:2406.08481.
Li et al. (2024b)Z. Li, K. Li, S. Wang, S. Lan, Z. Yu, Y. Ji, Z. Li, Z. Zhu, J. Kautz, Z. Wu, et al.Hydra-mdp: end-to-end multimodal planning with multi-target hydra-distillation.
arXiv preprint arXiv:2406.06978.
Liao et al. (2025)B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, et al.Diffusiondrive: truncated diffusion model for end-to-end autonomous driving.
In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 12037–12047.
Maes et al. (2026)L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. BalestrieroLeworldmodel: stable end-to-end joint-embedding predictive architecture from pixels.
arXiv preprint arXiv:2603.19312.
Mur-Labadia et al. (2026)L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, and N. BallasV-jepa 2.1: unlocking dense features in video self-supervised learning.
In European Conference on Computer Vision,
pp. 671–689.
Naeinian et al. (2026)F. Naeinian, A. Hamza, H. Zhu, and A. ChoromanskaZero-shot cross-city generalization in end-to-end autonomous driving: self-supervised versus supervised representations.
arXiv preprint arXiv:2603.11417.
Oquab et al. (2023)M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.Dinov2: learning robust visual features without supervision.
arXiv preprint arXiv:2304.07193.
Peebles and Xie (2023)W. Peebles and S. XieScalable diffusion models with transformers.
In 2023 IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 4172–4182.
Perez et al. (2018)E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. CourvilleFilm: visual reasoning with a general conditioning layer.
In Proceedings of the AAAI conference on artificial intelligence,
Siméoni et al. (2025)O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al.Dinov3.
arXiv preprint arXiv:2508.10104.
Sobal et al. (2026)U. Sobal, W. Zhang, K. Cho, R. Balestriero, T. G. Rudner, and Y. LeCunLearning from reward-free offline data: a case for planning with latent dynamics models.
Advances in Neural Information Processing Systems38, pp. 43905–43941.
Su et al. (2024)J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. LiuRoformer: enhanced transformer with rotary position embedding.
Neurocomputing568, pp. 127063.
Terver et al. (2025)B. Terver, T. Yang, J. Ponce, A. Bardes, and Y. LeCunWhat drives success in physical planning with joint-embedding predictive world models?.
arXiv preprint arXiv:2512.24497.
Wang et al. (2026a)L. Wang, Y. Zheng, Q. Chen, S. Li, Y. Zhang, Z. Xing, Q. Zhang, X. Li, D. Qian, P. Yang, et al.Latent-wam: latent world action modeling for end-to-end autonomous driving.
arXiv preprint arXiv:2603.24581.
Wang et al. (2026b)L. Wang, Z. Yang, C. Bai, G. Zhang, X. Liu, X. Zheng, X. Long, C. Lu, and C. LuDrive-jepa: video jepa meets multimodal trajectory distillation for end-to-end driving.
arXiv preprint arXiv:2601.22032.
Wang et al. (2026c)X. Wang, Y. Xiang, Y. Zhou, J. Wang, M. Huang, J. Huang, D. Wei, T. Zhou, X. Wang, G. Chen, et al.WA-jepa: rethinking the video jepa paradigm for world-action modeling in autonomous driving.
arXiv preprint arXiv:2608.20974.
Wang et al. (2026e)Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. J. Rudner, Y. LeCun, and M. RenTemporal straightening for latent planning.
In Forty-third International Conference on Machine Learning,
External Links: Link
Wu et al. (2026)H. Wu, R. Balestriero, and M. LevineVISReg: variance-invariance-sketching regularization for jepa training.
arXiv preprint arXiv:2606.02572.
Yang et al. (2026)J. Yang, Z. Chen, C. Huang, and J. LiAuto-jepa: a latent world model of continuous intent for end-to-end autonomous driving.
arXiv preprint arXiv:2607.29031.
Zhang et al. (2025)K. Zhang, Z. Tang, X. Hu, X. Pan, X. Guo, Y. Liu, J. Huang, L. Yuan, Q. Zhang, X. Long, et al.Epona: autoregressive diffusion world model for autonomous driving.
In 2025 IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 27220–27230.
Zhang et al. (2026)W. Zhang, B. Terver, A. Zholus, S. Chitnis, H. Sutaria, M. Assran, R. Balestriero, A. Bar, A. Bardes, Y. LeCun, et al.Hierarchical planning with latent world models.
arXiv preprint arXiv:2604.03208.
Zhong et al. (2026)R. Zhong, B. Ma, X. Chen, L. Zhang, M. Feng, Y. Wang, P. Liu, and J. MaDA-wam: decision-aligned future latents for driving world models.
arXiv preprint arXiv:2608.19085.
Zhou et al. (2024)G. Zhou, H. Pan, Y. LeCun, and L. PintoDino-wm: world models on pre-trained visual features enable zero-shot planning.
arXiv preprint arXiv:2411.04983.
Zhu and Choromanska (2026)H. Zhu and A. ChoromanskaSelf-supervised jepa-based world models for lidar occupancy completion and forecasting.
arXiv preprint arXiv:2602.12540.
Zhu et al. (2026)H. Zhu, Z. Dong, K. Topollai, B. Sha, and A. E. ChoromanskaSelf-supervised representation learning with joint embedding predictive architecture for automotive lidar object detection.
In Proceedings of the AAAI Conference on Artificial Intelligence,