Literature
Display
← All papers

OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision

Ruixun Liu · Lingyu Kong · Derun Li · Hang Zhao

Preprint posted · arXiv (preprint) ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

Overview

A camera-based autonomous-driving model uses dense occupancy labels during training to improve spatial features, then predicts semantic driving intent and continuous coordinates. Ruixun Liu, Lingyu Kong and Derun Li contributed equally; Hang Zhao is the corresponding author.

At a glance

OccVLA trains a vision-language model to predict dense 3D occupancy alongside text, then converts language-level driving intentions into future vehicle coordinates. Occupancy queries extract spatial information from image features during training; the occupancy branch can be omitted during language/planning inference.

On nuScenes, the model reports 0.28 m average trajectory error with ego-history input and 59.5% visual-question-answering accuracy. The evidence concerns offline autonomous-driving benchmarks, not robot manipulation or closed-loop driving.

1 Introduction

Text-only 3D labels sparsely supervise geometry, while explicit 3D model inputs introduce representation and pretraining mismatches. OccVLA instead uses automatically obtainable semantic occupancy grids as dense output supervision, keeping multi-view camera images as the visual input.

Shared transformer layers learn image/text features and a compact occupancy representation. A decoder can expose the predicted grid for inspection, but downstream text generation need not consume it. The contribution is training-time spatial supervision with an optional prediction branch; it does not remove the cost of the VLM or establish zero total inference overhead.

Complete original Figure 1.
Figure 1: Comparison of autonomous driving VLM architectures. (a) VLM (2D): Takes only 2D visual inputs and relies solely on textual supervision, lacking explicit 3D spatial grounding. (b) 3D-VLM: Consumes explicit 3D inputs (e.g., Occupancy, LiDAR) for reasoning, but the absence of large-scale 3D vision–language pretraining often leads to loss of fine-grained spatial details and limits generalization. (c) OccVLA (ours): Predicts dense 3D occupancy from 2D images and uses it as both an output and a dense 3D supervisory signal, enhancing fine-grained spatial understanding while preserving rich 2D visual details.

2 Related Work

The comparison separates language-based driving systems from occupancy representations used for 3D perception and world modeling.

Complete original Figure 2.
Figure 2: Overview of the proposed OccVLA architecture. The framework unifies dense 3D occupancy (occ) prediction and language modeling within a shared Vision–Language–Occupancy (V-L-O) backbone. The model is jointly trained with LN​A​Ro​c​cL_{NAR}^{occ} (a non-autoregressive loss for occupancy prediction) and LA​Rt​e​x​tL_{AR}^{text} (an autoregressive loss for textual outputs). (a) In the V-L-O backbone, occupancy tokens query visual features from visual tokens through cross-attention, while visual tokens are modeled via self-attention. (b) After predicting meta actions through the VLM, a planning head (MLP) generates the future trajectory.

2.1 MLLMs in Autonomous Driving

Prior approaches such as DriveVLM and EMMA connect language reasoning with driving actions and spatial labels. OmniDrive supplies a further 3D-aware comparison. OccVLA aims to provide denser spatial supervision without requiring explicit 3D sensor inputs at inference.

2.2 Occupancy for 3D Perception

Semantic occupancy assigns labels to spatial voxels, retaining geometry that scene captions may omit. OccWorld, OccLLaMA and Occ-LLM learn from occupancy representations for prediction or action generation. OccVLA uses occupancy as a supervised output rather than requiring it as the language model’s input.

3 Method

The architecture separates spatial representation learning, semantic driving-intent prediction and continuous trajectory regression.

3.1 Overview

A camera-based VLM shares intermediate features with an occupancy-prediction branch. A separate planning head consumes the generated meta action and visual/vehicle-state features. Three training stages adapt the VLM to driving, add occupancy supervision, and train the coordinate regressor.

3.2 Occupancy Prediction

Learnable occupancy queries share the VLM’s query/key/value projections, feed-forward layers and normalization. Cross-attention reads image keys/values, while the image stream retains self-attention. Residual adapters provide parameter-efficient adaptation. Equations 1–2 preserve the two attention paths.

A 200×200×16 grid is compressed to an H/r×W/r×F latent because most voxels are empty. Occupancy hidden states are linearly projected, decoded with an OccWorld-initialized VQ-VAE decoder and classified into voxel labels. This avoids directly predicting the full-resolution grid inside the transformer.

The authors report little difference after convergence when text can versus cannot access occupancy features. They therefore omit occupancy computation during language inference. No dedicated numerical ablation or latency table quantifies that particular claim in this version.

Equations 1–2
hOo​c​c=O⁡(s​o​f​t​m​a​x​(hQo​c​c​[hKi​m​g]Td)​[hVi​m​g])h_{O}^{occ}=O(softmax(\frac{h_{Q}^{occ}[h_{K}^{img}]^{T}}{\sqrt{d}})[h_{V}^{img}]) (1)
hOi​m​g=O⁡(s​o​f​t​m​a​x​(hQi​m​g​[hKi​m​g]Td)​[hVi​m​g])h_{O}^{img}=O(softmax(\frac{h_{Q}^{img}[h_{K}^{img}]^{T}}{\sqrt{d}})[h_{V}^{img}]) (2)

3.3 Motion Planning

The VLM predicts a discrete meta action before a small MLP generates coordinates over the next three seconds. Speed classes are maintain, accelerate and decelerate; the listed directional classes are straight, left/right turn, left/right lane change and stop. A later sentence calls these five categories despite listing six.

Inputs include six camera views and past meta actions. Training descriptions connect the observed scene and historical intent to the future meta-action label. Acceleration thresholds produce speed labels; GPT-4o classifies trajectories projected onto HD-map lanes and generates label-conditioned explanatory text.

Although called fully automated, the pipeline manually inspects every nuScenes meta-action label and refines about 20%. The generated explanation is supervision constructed using the known answer, not independent evidence of faithful reasoning.

The planning head receives the meta-action embedding, vehicle velocity and VLM visual tokens. Training and main results also use ego-trajectory history, as clarified by Table 5. No high-level navigation command is supplied, but predictions are not based on scene images alone.

Complete original Figure 3.
Figure 3: Overview of the meta action and CoT data generation pipeline. (a) Meta Action Pipeline: Vehicle trajectory coordinates are processed to compute accelerations for velocity action classification, and matched to HD map lanes for trajectory action classification via GPT-4o, followed by human refinement. The two components are combined to produce final meta actions. (b) Training Data Pipeline: Multi-view images and related meta actions are provided to GPT-4o to generate scene descriptions, infer historical motion patterns, and perform future reasoning, forming CoT training data.

3.4 Training Stage

Stage 1 fine-tunes the VLM on driving image/text data sampled from OmniDrive. Stage 2 jointly optimizes autoregressive text cross-entropy and a λ-weighted, non-autoregressive cross-entropy over final occupancy labels. The authors prefer categorical supervision to latent alignment, which they find affected by VQ-VAE encoding bias.

Adapters tune transformer blocks. The occupancy decoder has zero learning rate, keeping its parameters fixed while allowing gradients through its computations; the other trainable components share a learning rate. Stage 3 trains the MLP on predicted meta actions and state/visual features with mean squared error against future coordinates. Equation 3 retains the original joint objective.

Complete original Figure 4.
Figure 4: Overview of the training pipeline. Stage 1: Pretraining the VLM on autonomous driving scenarios using visual and text inputs. Stage 2: Occupancy-language joint training to enhance 3D scene understanding. Stage 3: Planning head training where the planning head predicts future coordinates from VLM-generated meta actions.
Equation 3
ℒ=ℒA​Rt​e​x​t+λ​ℒN​A​Ro​c​c\mathcal{L}=\mathcal{L}_{AR}^{text}+\lambda\mathcal{L}_{NAR}^{occ} (3)

4 Experiment

Evaluation covers offline trajectory prediction, driving question answering and current-frame occupancy reconstruction on nuScenes-derived data.

4.1 Experiment Settings

The paper uses nuScenes’ 700 training and 150 validation scenes, Occ3D occupancy labels and nuScenes-QA, which contains 460,000 question-answer pairs across existence, counting, object, status and comparison categories.

The backbone is PaliGemma2-3B-224px, with OccWorld’s scene VQ-VAE initialization. Training uses AdamW on eight NVIDIA A800 GPUs. The image–occupancy–text corpus is described without a precise total size; detailed schedules, λ and several architecture settings are not supplied.

4.2 Results and Analysis

With ego history, OccVLA reports L2 errors 0.18/0.26/0.40 m at one/two/three seconds, averaging 0.28 m. This improves the tabulated average over DriveVLM-Dual (0.31), EMMA (0.32) and camera-based Occ-LLM (0.43), but it is worse at one second than several baselines. Occupancy-input Occ-LLM ties the rounded 0.28 m average and is better at one/two seconds.

Inputs, supervision and history use differ across comparison methods. Table 1 labels EMMA supervision “None,” whereas the accompanying prose says it uses 3D-coordinate/box supervision. The original values and labels are retained; these tables are not a controlled isolation of every architectural difference.

The 3B model reaches 59.5% nuScenes-QA accuracy versus OpenDriveVLA’s 58.2% and OccLLaMA3.1’s 54.5%. It improves object/status accuracy strongly, but trails OpenDriveVLA in counting (21.9 versus 22.7%) and comparison (67.2 versus 68.8%). Larger-model comparisons do not by themselves quantify runtime efficiency.

Current-frame occupancy prediction reports about 10% mIoU, with no detailed quantitative baseline table. Figure 5 qualitatively shows near-road objects while exposing missed occluded structure. Its caption calls this forecasting, but the method description reconstructs the current moment from current multi-view images, without temporal features.

Table 1: End-to-end motion planning experiments on nuScenes Caesar et al. (2020) with different input and supervision. L denotes LiDAR input and C denotes camera input.
Method Input Supervision L2(m)↓\downarrow
1s 2s 3s Avg.
NMP L Box & Motion 0.53 1.25 2.67 1.48
FF L Freespace 0.55 1.20 2.54 1.43
ST-P3 C Map & Box & Depth 1.33 2.11 2.90 2.11
UniAD C Map & Box & Motion & Track & Occ 0.48 0.96 1.65 1.03
VAD C Map & Box & Motion 0.54 1.15 1.98 1.22
DriveVLM-Dual C Map & Box & Motion 0.15 0.29 0.48 0.31
EMMA C None 0.14 0.29 0.54 0.32
OmniDrive C & L None 0.14 0.29 0.55 0.33
Ours C Occ 0.18 0.26 0.40 0.28
Table 2: End-to-end motion planning experiments on nuScenes Caesar et al. (2020) compared with models like OccNet Liu et al. (2024a), OccWorld Zheng et al. (2024), and others that use occupancy as LLM input.
Method Input Supervision L2(m)↓\downarrow
1s 2s 3s Avg.
OccNet Occ Map & Box 1.29 2.31 2.98 2.25
OccWorld-O Occ None 0.43 1.08 1.99 1.17
OccLLAMA-O Occ None 0.37 1.02 2.03 1.14
Occ-LLM Occ None 0.12 0.24 0.49 0.28
OccWorld-F C Occ 0.45 1.33 2.25 1.34
OccLLama-F C Occ 0.38 1.07 2.15 1.20
Occ-LLM C Occ 0.21 0.40 0.67 0.43
Ours C Occ 0.18 0.26 0.40 0.28
Table 3: Quantitative results on Nuscenes-QA(Qian et al., 2023) compared with models that using different input like LLAVA (Liu et al., 2023), LiDAR-LLM(Yang et al., 2023), OccLLaMA(Wei et al., 2024) and OpenDriveVLA(Zhou et al., 2025b).
Model Size Input exist(%)↑\uparrow count(%)↑\uparrow object(%)↑\uparrow status(%)↑\uparrow comparison(%)↑\uparrow acc(%)↑\uparrow
h0 h1 all h0 h1 all h0 h1 all h0 h1 all h0 h1 all
LLaVA 7B C 74.8 72.9 73.8 14.9 14.3 14.6 57.7 34.5 37.9 48.6 44.5 45.9 65.9 52.1 53.3 47.4
LiDAR-LLM 7B L 79.1 70.6 74.5 15.3 14.7 15.0 59.6 34.1 37.8 53.4 42.0 45.9 67.0 57.0 57.8 48.6
OccLLaMA3.1 8B Occ 82.9 79.2 80.9 19.2 19.2 19.2 64.8 43.1 46.3 51.0 46.1 47.8 76.5 65.6 66.6 54.5
OpenDriveVLA 7B C - - 84.2 - - 22.7 - - 49.6 - - 54.5 - - 68.8 58.2
Ours 3B C 87.4 81.7 84.3 22.6 21.2 21.9 73.6 51.2 54.5 62.6 57.9 59.5 79.2 66.0 67.2 59.5
Complete original Figure 5.
Figure 5: The 3D occupancy forecasting results of our OccVLA, which demonstrate accurate estimation for critical objects (e.g., vehicles, roads, etc.).

4.3 Ablation Study

Occupancy supervision raises speed accuracy 53.77→54.83%, directional accuracy 77.24→77.95%, and overall joint meta-action accuracy 41.48→43.08%. The overall gain is 1.60 percentage points; the mean of component accuracies improves 0.87 points. Table 4 does not separately report VQA or trajectory-error ablations despite the prose discussing both.

Removing ego-trajectory history changes average L2 from 0.28 to 0.48 m, and three-second error from 0.40 to 0.80 m. Thus the headline 0.28 m uses history. The authors acknowledge that history can provide a strong prior, consistent with earlier critiques of open-loop evaluation. This ablation does not establish generalization under closed-loop distribution shift.

Table 4: Ablation study of the occupancy supervision. The ✗ indicates that the model corresponds to the original VLM without occupancy integration, whereas the ✓ denotes that the model is trained through joint occupancy–vision–language learning.
Method Occupancy speed (%) trajectory (%) Avg. (%) Overall. (%)
OccVLA ✗ 53.77 77.24 65.50 41.48
OccVLA ✓ 54.83 77.95 66.37 43.08
Table 5: Ablation study on Ego Trajectory. The ✗ symbol denotes that the model has no access to Ego Trajectory information.
Method Ego Trajectory L2(m)↓\downarrow
1s 2s 3s Avg.
OccVLA ✗ 0.28 0.35 0.80 0.48
OccVLA ✓ 0.18 0.26 0.40 0.28

5 Conclusion

Training a camera-based VLM with an auxiliary occupancy branch provides a compact way to add spatial supervision and modestly improves joint meta-action accuracy in the reported ablation. The branch remains available for interpretable grid output or can be skipped for language/planning execution.

The evidence is limited to nuScenes-derived offline benchmarks. No closed-loop rollout, collision-rate evaluation, independent safety validation, repeated-seed uncertainty or measured latency comparison is reported. Modest occupancy quality, annotation refinement and dependence on ego history qualify the broader scalability and deployment claims.

Concept demonstration

Open image ↗