Literature
Display
← All papers

AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

Haoran Zhu · Wancong Zhang · Yann LeCun · Anna Choromanska

· arXiv (preprint) ↗

World models5 min text · CondensedOriginal paper ↗
Contents

Overview

The overview compares compact world-model planning with larger feature-based baselines. The goal-conditioned evaluation uses recorded future observations as targets.

Refer to caption
Figure 1: AD-E2E-JEPA is an action-conditioned, JEPA-based world model for end-to-end autonomous driving. Without training any driving policy, it enables driving via goal-conditioned zero-shot planning, where future sensor observations are specified as the goal and a trajectory is selected from the trajectory vocabulary solely via world-model rollouts. AD-E2E-JEPA demonstrates strong zero-shot driving performance, efficient planning, low displacement errors, high trajectory hit rates, and representations that transfer effectively to downstream imitation learning. Zero-shot planning metrics are evaluated on 100 subsampled test scenes for a fair comparison with the time-consuming DINO-WM/JEPA-WM. See Table 2 for AD-E2E-JEPA’s results on the full set of 12,146 test scenes.

At a glance

Intuition. Searching over future drives is costly when every imagined frame contains hundreds of high-dimensional patch vectors. AD-E2E-JEPA learns a compact projector on frozen DINOv3 features and trains an action-conditioned latent predictor, using SIGReg to discourage collapse of the projected space. It selects from a finite vocabulary of candidate trajectories by comparing their predicted future features with a goal image. Crucially, that goal is the recorded future image: this isolates world-model planning quality but gives privileged goal information, rather than solving unconstrained autonomous driving. The reported large speedup is relative to dense patch models under matched candidate settings. A separate imitation-learning experiment tests transfer of the pretrained projector; driving safety and full EPDMS must be distinguished from the safety-stripped score.

1 Introduction

The paper studies the accuracy–cost tradeoff of JEPA world models for driving. Dense patch representations can plan accurately but slowly, while a small global token can be fast and lose necessary detail. Its projector aims for a middle ground, evaluated through privileged goal-conditioned search and separately through downstream imitation learning.

2 Related Work

2.1 World Models

World models can reconstruct images, predict latent states, or train policies using imagined rollouts. This work follows the latent-prediction route with a frozen visual backbone and learnable compact features. It does not claim that every generative or policy-based world model solves the same evaluation problem.

2.2 End-to-End Autonomous Driving

Earlier driving systems use supervised trajectories, video generation, or JEPA-style representation learning. The distinctive diagnostic here searches with a known future image as its target, separating model-based trajectory ranking from learning a driving policy. That protocol narrows what its zero-shot label establishes.

3 Method

A frozen visual encoder, shared learnable projector and action-conditioned predictor support the planning experiment. The projector is also reused in a separately trained imitation model.

Refer to caption
Figure 2: AD-E2E-JEPA architecture: (a) A self-supervised, action-conditioned JEPA world model for E2EAD. The DINOv3 encoder is frozen, while (a.1) a learnable projector is shared by the context-history and target-future branches, compressing the representation by 16×16\times and its dimensionality by 4×4\times, with SIGReg maximizing information while avoiding collapse and improving planning performance. This enables a 100×100\times speedup in zero-shot goal-conditioned planning in (b), where trajectories are selected via world-model rollouts over a driving vocabulary. (c) The projector pretrained through self-supervision also improves downstream imitation learning.

3.1 Preliminary: Task formulation of E2EAD

The front-camera setting predicts future planar ego poses relative to the current vehicle. Historical images and poses provide context; actions are pose increments. A safe trajectory must satisfy additional driving constraints beyond matching a future visual embedding.

Equations 1–2
Historical context: (𝐈t−K:t,𝐏t−K:t)\displaystyle(\mathbf{I}_{t-K:t},\,\mathbf{P}_{t-K:t}) (1)
Trajectory prediction: 𝐏^t+1:t+F=fϕ(𝐈t−K:t,𝐏t−K:t)\displaystyle\hat{\mathbf{P}}_{t+1:t+F}=f_{\phi}(\mathbf{I}_{t-K:t},\,\mathbf{P}_{t-K:t}) (2)

3.2 World Model Architecture

3.2.1 JEPA-WM Adaptation for E2EAD

The dense JEPA-WM baseline predicts frozen DINOv3 patch features using an AdaLN-conditioned predictor. Since its target encoder is fixed, its basic regression target cannot collapse through encoder co-adaptation. This baseline supplies the accuracy and runtime comparison for feature compression.

Equations 3–6
𝐚t=Relative⁡(𝐏t,𝐏t+1)=[Δ​xt→t+1,Δ​yt→t+1,Δ​θt→t+1]⊤.\mathbf{a}_{t}=\operatorname{Relative}(\mathbf{P}_{t},\mathbf{P}_{t+1})=\left[\Delta x_{t\rightarrow t+1},\Delta y_{t\rightarrow t+1},\Delta\theta_{t\rightarrow t+1}\right]^{\top}. (3)
st−K:t+1\displaystyle s_{t-K:t+1} =Enc(𝐈t−K:t+1),\displaystyle=\operatorname{Enc}(\mathbf{I}_{t-K:t+1}), (4)
s^t−K+1:t+1\displaystyle\hat{s}_{t-K+1:t+1} =Pred(st−K:t,Ea(𝐚t−K:t)),\displaystyle=\operatorname{Pred}\!\left(s_{t-K:t},E_{a}(\mathbf{a}_{t-K:t})\right), (5)
ℒpred\displaystyle\mathcal{L}_{\mathrm{pred}} =MSE(s^t−K+1:t+1,st−K+1:t+1).\displaystyle=\operatorname{MSE}\!\left(\hat{s}_{t-K+1:t+1},s_{t-K+1:t+1}\right). (6)

3.2.2 AD-E2E-JEPA

Two stride-two convolutional layers reduce spatial patch count by 16 and feature dimension by 4. A learned projector creates a new collapse risk even when the underlying encoder is frozen, so the training recipe adds detached targets and SIGReg. Temporal rollout consistency can be added to address self-generated prediction context.

Equations 7–12
zt−K:t+1=Proj(st−K:t+1)z_{t-K:t+1}=\operatorname{Proj}(s_{t-K:t+1}) (7)
z^t−K+1:t+1=Pred′(zt−K:t,Ea(𝐚t−K:t))\hat{z}_{t-K+1:t+1}=\operatorname{Pred}^{\prime}\!\left(z_{t-K:t},E_{a}(\mathbf{a}_{t-K:t})\right) (8)
ℒpredproj=MSE(z^t−K+1:t+1,sg(zt−K+1:t+1))\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}=\operatorname{MSE}\!\left(\hat{z}_{t-K+1:t+1},\operatorname{sg}\left(z_{t-K+1:t+1}\right)\right) (9)
ℒSIGRegt−K:t+1=1N​H′​W′​M∑l=1N​H′​W′∑m=1MT({⟨zl,b,𝒖(m)⟩}b=1B).\mathcal{L}^{t-K:t+1}_{\mathrm{SIGReg}}=\frac{1}{NH^{\prime}W^{\prime}M}\sum_{l=1}^{NH^{\prime}W^{\prime}}\sum_{m=1}^{M}T\left(\left\{\left\langle z_{l,b},\bm{u}^{(m)}\right\rangle\right\}_{b=1}^{B}\right). (10)
ℒ=ℒprojpred+λℒt−K:t+1SIGReg.\mathcal{L}=\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}+\lambda\mathcal{L}^{t-K:t+1}_{\mathrm{SIGReg}}. (11)
ℒmulti=ℒ¯TF+ℒ¯2+⋯+ℒ¯FF+λℒt−K:t+FSIGReg.\mathcal{L}^{\mathrm{multi}}=\frac{\overline{\mathcal{L}}_{\mathrm{TF}}+\overline{\mathcal{L}}_{2}+\cdots+\overline{\mathcal{L}}_{F}}{F}+\lambda\mathcal{L}^{t-K:t+F}_{\mathrm{SIGReg}}. (12)

3.3 Zero-Shot Goal-Conditioned Planning

The zero-shot experiment means no task-specific driving policy is trained for candidate selection. It still uses training interaction data, a trajectory vocabulary and ground-truth future goal images.

3.3.1 Planning with AD-E2E-JEPA

Candidate pose sequences come from a driving-trajectory vocabulary. Each is rolled out through the latent model, and the lowest terminal distance to the encoded goal is selected. Larger candidate vocabularies improve coverage at increased cost; the method is discrete trajectory search, not a general continuous optimal-control solver.

Equations 13–14
𝒞i=‖zt+F−z^t+Fi‖22,\mathcal{C}^{i}=\left\|z_{t+F}-\hat{z}_{t+F}^{\,i}\right\|_{2}^{2}, (13)
i∗=argmini∈{1,…,|Vsampled|}𝒞i=argmini∈{1,…,|Vsampled|}‖zt+F−z^t+Fi‖22,Pt:t+F∗=Pt:t+Fi∗.i^{*}=\operatorname*{argmin}_{i\in\{1,\ldots,|V_{\text{sampled}}|\}}\mathcal{C}^{i}=\operatorname*{argmin}_{i\in\{1,\ldots,|V_{\text{sampled}}|\}}\left\|z_{t+F}-\hat{z}_{t+F}^{\,i}\right\|_{2}^{2},\qquad P^{*}_{t:t+F}=P^{i^{*}}_{t:t+F}. (14)

3.3.2 Metrics

Full EPDMS includes multiplicative safety factors; the daggered version omits them. Endpoint displacement, heading errors and whether the known target trajectory ranks highly diagnose different aspects of model quality. High safety-stripped progress does not establish safe driving.

3.4 Downstream Transfer for Imitation Learning

For imitation learning, the future predictor is removed and the frozen-backbone/projector representation feeds a trainable trajectory decoder. Comparing a pretrained and random projector tests representation transfer separately from goal-conditioned search.

4 Experiments

4.1 Datasets

Matched model comparisons use the 10-hour NAVSIM training split, while a scaled variant uses 70 hours. The larger-data result should be separated from the matched-data architectural comparison.

4.2 Implementation Details

The main recipe uses four context frames, an eight-frame future, DINOv3 features and a shared training schedule where specified. The projector’s compactness reduces training resources as well as planning cost. SIGReg weighting is adjusted heuristically in larger-batch settings.

Table 1: Training configurations.
Split Variant GPUs Batch size Learning rate λ\lambda Training time
navtrain LeWM 4×4\timesA100 8 1×10−41\times 10^{-4} 0.09 1 d
DINO-WM 4×4\timesA100 64 1×10−41\times 10^{-4} – 11 h
JEPA-WM 4×4\timesA100 64 1×10−41\times 10^{-4} – 13 h
AD-E2E-JEPA 1×1\timesA100 128 1×10−41\times 10^{-4} 0.09 20 h
    + rollout 1×1\timesA100 128 1×10−41\times 10^{-4} 0.09 1 d 22 h
trainval AD-E2E-JEPA 4×4\timesA100 512 2×10−42\times 10^{-4} 0.025 2 d 5 h
    + rollout 4×4\timesA100 256 1.4×10−41.4\times 10^{-4} 0.09 4 d 2 h

4.3 Zero-Shot Planning Performance

Dense DINO-WM and JEPA-WM are accurate but slow on the 100-scene subset; the compact model lowers runtime substantially. More data, rollout training and more candidate trajectories improve several metrics. Full-set results are reported only where computationally feasible, so subset and full-set scores must not be mixed.

Refer to caption
Figure 3: Zero-shot goal-conditioned planning qualitative comparison. LeWM is efficient but insufficient for accurate planning, resulting in a shifted selected trajectory. DINO-WM and JEPA-WM better match the ground-truth trajectory. AD-E2E-JEPA achieves comparable planning quality to DINO-WM and JEPA-WM while being 100×\times faster.
Table 2: Zero-shot goal-conditioned planning on the NAVSIM dataset, evaluated on 100 subsampled test scenes and the full set of 12,146 test scenes, with the ground-truth future frame used as the target. Unless specified, the world model selects from 256 candidate trajectories. The NAVSIM-v2 navtest-stage-1 metrics EPDMS↑\uparrow and EPDMS†↑{}^{\dagger}\uparrow measure driving performance, where EPDMS includes safety metrics and EPDMS† excludes them. Per-scene planning time (s) measures efficiency on an A100 GPU. FDE (m), Δ​x\Delta x (m), Δ​y\Delta y (m), and Δ​θ\Delta\theta (degrees) measure geodesic accuracy. Top-1/top-5 hit rate (%) measures world-model reliability. ↑\uparrow: higher is better; ↓\downarrow: lower is better. For the 100 subsampled-test scene setting, the extended comfort (EC) metrics for EPDMS and EPDMS† are excluded because the subsampled set does not contain temporally adjacent scenes. See Appendix A.4 for details of each submetric of EPDMS.
Driving performance Efficiency Geodesic accuracy Reliability
Method Split EPDMS↑\uparrow EPDMS†↑{}^{\dagger}\uparrow Time ↓\downarrow FDE ↓\downarrow Δ​x\Delta x ↓\downarrow Δ​y\Delta y ↓\downarrow Δ​θ\Delta\theta ↓\downarrow Hit rate ↑\uparrow
100 subsampled test scenes
LeWM navtrain 48.3 73.9 0.7 12.4 11.3 2.7 13.6 6/18
DINO-WM 68.3 91.4 91.8 3.9 3.4 1.1 5.9 40/73
JEPA-WM 74.2 90.9 101.0 4.0 3.4 1.2 5.2 45/75
AD-E2E-JEPA 76.6 92.4 0.8 4.2 3.9 1.0 4.7 27/59
+ rollout 70.4 92.2 0.8 3.5 3.2 1.0 4.1 34/67
AD-E2E-JEPA trainval 72.1 89.8 0.8 4.7 4.4 0.9 3.4 33/55
+ rollout 72.3 91.9 0.8 3.2 3.0 0.7 3.6 47/78
Full 12,146 test scenes
LeWM navtrain 39.8 66.7 0.7 14.6 13 3.5 17.2 5.7/11.3
AD-E2E-JEPA 63.5 80.1 0.8 6.3 5.9 1.2 6.3 32.9/65.3
+ rollout 64.9 83.1 0.8 4.5 4.0 1.2 4.1 42.6/71.0
AD-E2E-JEPA trainval 63.2 80.1 0.8 6.2 5.8 1.1 4.5 31.0/64.8
+ rollout 67.3 84.1 0.8 4.0 3.6 1.1 3.5 53.8/82.7
+ rollout, 512 traj. 69.2 85.0 1.4 3.6 3.3 0.9 2.9 45.2/73.9
+ rollout, 1024 traj. 70.5 85.5 2.5 3.2 2.9 0.8 2.5 36.3/64.3
+ rollout, 2048 traj. 71.5 86.0 4.7 3.0 2.7 0.8 2.2 27.4/53.2
+ rollout, 4096 traj. 72.1 86.3 9.3 2.9 2.6 0.8 2.1 20.7/43.3
+ rollout, 8192 traj. 72.9 86.5 18.2 2.8 2.5 0.8 2.0 15.3/33.8

4.4 Transfer Learning Performance

Pretraining the projector improves the reported downstream imitation score compared with random projector initialization. Large external driving models are contextual references rather than matched baselines: they differ in pretraining scale, sensors and architectures.

Table 3: NAVSIMv2 navtest-stage-1 benchmark. V: single-view (SV) or multi-view (MV); Type: perception-free (PF) or perception-based (PB); Fr.: number of input frames. PB methods use perception annotations or trajectory-score labels. ∗ denotes results obtained before the human-filter bug fix in commit 359c7f7 in NAVSIM’s codebase.
NAVSIMv2 stage 1 driving metrics
Method V Type Fr. NC↑\uparrow DAC↑\uparrow DDC↑\uparrow TLC↑\uparrow EP↑\uparrow TTC↑\uparrow LK↑\uparrow HC↑\uparrow EC↑\uparrow EPDMS∗↑{}^{*}\uparrow EPDMS↑\uparrow
Transfuser MV PF 1 96.9 89.9 97.8 99.7 87.1 95.4 92.7 98.3 87.2 76.7 –
Latent-WAM MV PF 4 98.1 97.3 99.6 99.8 87.7 97.3 97.6 98.1 72.4 – 89.3
WA-JEPA MV PF 4 99.4 98.2 99.7 99.9 87.8 98.9 98.3 98.3 88.1 88.0 91.7
Drive-JEPA SV PB 2 98.4 98.6 99.1 99.8 88.4 97.8 97.6 97.9 84.8 87.8 –
DINOv3
+ rand. proj. SV PF 4 96.8 89.8 98.3 99.7 87.1 95.7 94.8 98.3 84.1 – 80.2
+ AD-E2E-JEPA proj. SV PF 4 97.7 93.7 99.2 99.8 87.3 96.8 97.0 98.4 88.7 – 85.4

5 Conclusion

The projector offers a favorable cost–quality tradeoff for the stated latent driving-planning protocol and useful transfer in imitation learning. The benchmark is not evidence of safe deployment without future goal images, and its best result uses a larger candidate budget than the fast 256-candidate configuration.

Appendix A Appendix

A.1 SIGReg Configuration Details

SIGReg uses the retained numerical integration and random-projection settings. A finite projected-distribution test discourages collapse but does not prove that all driving-relevant distinctions remain represented.

A.2 Rollout Setting

The rollout variant extends one-step training across the full future window while maintaining the fixed history length. Teacher-forced and self-generated contexts receive separate losses.

A.2.1 Teacher-Forcing Loss

Teacher forcing supplies real encoded context. The first window supervises all outputs, while later overlapping windows supervise only their newest output to control repeated contributions.

Equations 15–18
z^t+k−K+1:t+k+1TF=Pred′(zt+k−K:t+k,Ea(𝐚t+k−K:t+k)),k=0,…,F−1.\hat{z}^{\mathrm{TF}}_{t+k-K+1:t+k+1}=\operatorname{Pred}^{\prime}\!\left(z_{t+k-K:t+k},E_{a}(\mathbf{a}_{t+k-K:t+k})\right),\qquad k=0,\ldots,F-1. (15)
ℒpredproj[0]=MSE(z^t−K+1:t+1TF,sg(zt−K+1:t+1)).\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}[0]=\operatorname{MSE}\!\left(\hat{z}^{\mathrm{TF}}_{t-K+1:t+1},\operatorname{sg}\!\left(z_{t-K+1:t+1}\right)\right). (16)
ℒpredproj[k]=MSE(z^t+k+1TF,sg(zt+k+1)),k=1,…,F−1.\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}[k]=\operatorname{MSE}\!\left(\hat{z}^{\mathrm{TF}}_{t+k+1},\operatorname{sg}\!\left(z_{t+k+1}\right)\right),\qquad k=1,\ldots,F-1. (17)
ℒ¯TF=(K+1)​ℒpredproj​[0]+∑k=1F−1ℒpredproj​[k]K+F.\overline{\mathcal{L}}_{\mathrm{TF}}=\frac{(K+1)\,\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}[0]+\sum_{k=1}^{F-1}\mathcal{L}^{\mathrm{proj}}_{\mathrm{pred}}[k]}{K+F}. (18)

A.2.2 Rollout Consistency Loss

Autoregressive training feeds predicted latents back into a sliding context. Truncated backpropagation detaches this context between calls, controlling derivative depth while still training predictions against future targets.

Equations 19–23
z~AR,(0)t−K:t=zt−K:t.\tilde{z}^{\mathrm{AR},(0)}_{t-K:t}=z_{t-K:t}. (19)
z^t−K+1:t+1AR,(1)=Pred′(z~t−K:tAR,(0),Ea(𝐚t−K:t)).\hat{z}^{\mathrm{AR},(1)}_{t-K+1:t+1}=\operatorname{Pred}^{\prime}\!\left(\tilde{z}^{\mathrm{AR},(0)}_{t-K:t},E_{a}(\mathbf{a}_{t-K:t})\right). (20)
z^t−K+j:t+jAR,(j)=Pred′(sg(z~t−K+j−1:t+j−1AR,(j−1)),Ea(𝐚t−K+j−1:t+j−1)).\hat{z}^{\mathrm{AR},(j)}_{t-K+j:t+j}=\operatorname{Pred}^{\prime}\!\left(\operatorname{sg}\!\left(\tilde{z}^{\mathrm{AR},(j-1)}_{t-K+j-1:t+j-1}\right),E_{a}(\mathbf{a}_{t-K+j-1:t+j-1})\right). (21)
z~t−K+j:t+jAR,(j)=[z~t−K+j:t+j−1AR,(j−1),z^t+jAR,(j)],j=1,…,F.\tilde{z}^{\mathrm{AR},(j)}_{t-K+j:t+j}=\left[\tilde{z}^{\mathrm{AR},(j-1)}_{t-K+j:t+j-1},\hat{z}^{\mathrm{AR},(j)}_{t+j}\right],\qquad j=1,\ldots,F. (22)
ℒ¯k=MSE(z^t+kAR,(k),sg(zt+k)),k=2,…,F.\overline{\mathcal{L}}_{k}=\operatorname{MSE}\!\left(\hat{z}^{\mathrm{AR},(k)}_{t+k},\operatorname{sg}\!\left(z_{t+k}\right)\right),\qquad k=2,\ldots,F. (23)

A.2.3 SIGReg

The Gaussian regularizer covers the retained context and future projected embeddings. It complements prediction consistency, rather than defining driving safety or task success.

A.3 Metrics

The appendix defines each multiplicative safety component and weighted driving-quality component, along with endpoint and trajectory-ranking diagnostics. The distinction between EPDMS and its safety-stripped variant is essential when interpreting the high reported numbers.

Equations 24–28
bi={1,if a valid neighboring scene is available for computing EC,0,otherwise.b_{i}=\begin{cases}1,&\text{if a valid neighboring scene is available for computing EC},\\ 0,&\text{otherwise}.\end{cases} (24)
EPDMSi=NCi⋅DACi⋅DDCi⋅TLCi⋅5​EPi+5​TTCi+2​LKi+2​HCi+2​bi​ECi14+2​bi.\mathrm{EPDMS}_{i}=\mathrm{NC}_{i}\cdot\mathrm{DAC}_{i}\cdot\mathrm{DDC}_{i}\cdot\mathrm{TLC}_{i}\cdot\frac{5\,\mathrm{EP}_{i}+5\,\mathrm{TTC}_{i}+2\,\mathrm{LK}_{i}+2\,\mathrm{HC}_{i}+2\,b_{i}\,\mathrm{EC}_{i}}{14+2\,b_{i}}. (25)
EPDMSi†=5​EPi+5​TTCi+2​LKi+2​HCi+2​bi​ECi14+2​bi.\mathrm{EPDMS}^{\dagger}_{i}=\frac{5\,\mathrm{EP}_{i}+5\,\mathrm{TTC}_{i}+2\,\mathrm{LK}_{i}+2\,\mathrm{HC}_{i}+2\,b_{i}\,\mathrm{EC}_{i}}{14+2\,b_{i}}. (26)
hn@​k=𝕀[rank(Cngt)≤k],h_{n}^{@k}=\mathbb{I}\left[\operatorname{rank}\!\left(C_{n}^{\mathrm{gt}}\right)\leq k\right], (27)
HitRate​@​k=1Q​∑n=1Qhn@​k,k∈{1,5},\mathrm{HitRate}@k=\frac{1}{Q}\sum_{n=1}^{Q}h_{n}^{@k},\qquad k\in\{1,5\}, (28)

A.4 EPDMS Details for Zero-Shot Goal-Conditioned Planning

Complete component tables are retained for the 100-scene subset and the full 12,146-scene set. They show where aggregate driving scores can conceal failures in individual safety or quality terms.

Table 4: EPDMS details for 100 subsampled test scenes.
Method Split NC↑\uparrow DAC↑\uparrow DDC↑\uparrow TLC↑\uparrow EP↑\uparrow TTC↑\uparrow LK↑\uparrow HC↑\uparrow EC↑\uparrow EPDMS↑\uparrow EPDMS†↑\uparrow
LeWM navtrain 78.5 77.0 84.0 97.0 68.6 76.0 86.0 70.0 – 48.3 73.9
DINO-WM 96.0 79.0 96.5 99.0 87.8 93.0 92.0 96.0 – 68.3 91.4
JEPA-WM 94.5 87.0 96.5 99.0 87.6 92.0 94.0 93.0 – 74.2 90.9
AD-E2E-JEPA 96.0 87.0 95.5 99.0 88.0 95.0 94.0 95.0 – 76.6 92.4
+ rollout 96.0 77.0 95.5 100.0 87.9 95.0 93.0 95.0 – 70.4 92.2
AD-E2E-JEPA trainval 94.0 82.0 96.5 100.0 87.2 92.0 91.0 90.0 – 72.1 89.8
+ rollout 95.5 82.0 98.0 99.0 87.3 93.0 97.0 96.0 – 72.3 91.9
Table 5: EPDMS details for the full set of 12,146 test scenes.
Method Split NC↑\uparrow DAC↑\uparrow DDC↑\uparrow TLC↑\uparrow EP↑\uparrow TTC↑\uparrow LK↑\uparrow HC↑\uparrow EC↑\uparrow EPDMS↑\uparrow EPDMS†↑\uparrow
LeWM navtrain 82.0 67.5 81.4 98.5 66.1 79.5 79.0 66.7 13.7 39.8 66.7
AD-E2E-JEPA 93.1 82.4 95.6 99.5 80.6 90.9 89.4 89.6 21.7 63.5 80.1
+ rollout 94.7 80.9 93.8 99.7 84.0 92.5 87.9 91.8 34.6 64.9 83.1
AD-E2E-JEPA trainval 92.8 82.8 95.3 99.4 82.0 90.6 89.8 88.5 19.5 63.2 80.1
+ rollout 95.2 82.0 94.4 99.6 85.2 93.3 89.3 92.3 35.5 67.3 84.1
+ rollout, 512 traj. 95.9 83.6 95.5 99.6 85.6 94.4 90.4 93.5 36.9 69.2 85.0
+ rollout, 1024 traj. 96.4 84.2 96.1 99.6 85.6 95.0 91.0 94.1 38.5 70.5 85.5
+ rollout, 2048 traj. 96.7 84.8 96.4 99.7 85.6 95.4 91.5 94.5 40.9 71.5 86.0
+ rollout, 4096 traj. 96.7 85.4 96.4 99.7 85.5 95.6 91.5 95.0 42.5 72.1 86.3
+ rollout, 8192 traj. 96.9 86.0 96.7 99.7 85.4 95.7 91.8 95.1 43.8 72.9 86.5

A.5 Downstream Imitation Learning

The imitation decoder combines visual features, the learned projector, positional/temporal embeddings and a driving command to predict a future trajectory. It is trained with supervised trajectory error. This downstream stage uses a learned policy, unlike the earlier goal-conditioned search evaluation.

Refer to caption
Figure 4: Downstream imitation learning architecture.
Concept demonstration

Open image ↗