At a glance
DepthVLA adds a pretrained monocular-depth expert to a vision-language model and a continuous-action expert. Intermediate geometric features condition action generation, while an auxiliary depth loss helps preserve geometry during policy training. Reported averages improve from 58.8% to 74.8% on Simpler and from 93.6% to 94.9% on LIBERO relative to the reimplemented baseline. The real-robot result is a progress score of 78.5% versus 65.0%, not whole-task success probability.
I INTRODUCTION
Web-trained vision-language representations supply semantics but can miss precise geometry needed for grasping and collision avoidance. DepthVLA transfers a separate depth model’s prior into robot control, rather than relying exclusively on action-data pretraining. The resulting policy uses ordinary camera inputs and incurs a modest measured latency increase. Evidence concerns specific manipulation tasks; geometric representation is inferred from performance and depth predictions rather than directly established as general spatial reasoning.
II Related Work
Related work covers generalist robot policies, geometric enhancements to action models, and pretrained 3D perception.
II-A Generalist Robot Manipulation Policies
Generalist policies transfer visual-language knowledge into action tokens or continuous action chunks. Large robot datasets help, but do not automatically provide the fine geometric cues required by every manipulation task. DepthVLA adds a separate source of pretrained geometric supervision.
II-B VLAs with Spatial Awareness
Earlier approaches use depth sensors, pseudo-point clouds, future-state prediction, or autoregressively generated spatial tokens. DepthVLA instead gives the action stream access to intermediate depth features within the transformer. This avoids generating a long spatial-token sequence before each action, although it still adds model computation.
II-C 3D Geometry Perception
Monocular and multiview geometry foundation models learn useful priors from diverse 3D data. DepthVLA initializes its depth branch from Depth Anything V2 and further trains it for the architecture used in the policy.
III Method
The method specifies three expert streams and a joint action-and-depth training objective.
III-A Problem Formulation and Model Overview
The policy predicts an action chunk from current camera observations, language, and proprioception. Its mixture of transformers combines separately parameterized semantic, depth, and action streams. A block mask restricts semantic and depth tokens to their own streams while allowing action tokens to attend to both. Thus “fully shared attention” means a shared attention computation with explicit restrictions, not unrestricted cross-attention among all tokens.
Equation
III-B Depth Expert
A DINOv2-based encoder initialized from Depth Anything V2 feeds a transformer decoder and linear depth head. Before robot training, the depth branch learns on diverse 3D datasets with a scale-invariant logarithmic loss, using a scale coefficient of 0.5. The action expert attends to intermediate features rather than only the final depth map, retaining richer geometric information.
Equations
III-C DepthVLA Policy Training
Flow matching learns continuous action chunks from demonstrations. A retained depth-prediction loss jointly regularizes the geometric branch during policy training. The source equations specify noisy action interpolation, the target flow, and combined objective. Robot datasets without measured depth receive pseudo-labels from existing depth estimators, so geometric supervision depends partly on those models.
Equations
IV Experiments
Evaluation compares simulation success, real-robot subtask progress, and architectural ablations. These measure different outcomes and are kept separate.
IV-A Implementation Details
The implementation uses PaliGemma-3B and approximately 300M-parameter depth and action decoders. The depth encoder adds another roughly 300M parameters. Depth pretraining lasts 50,000 steps; policy training uses AdamW on 32 H100 GPUs. On an RTX 4090, memory rises from 6.7 to 8.0 GB and latency from 190 to 210 ms per chunk. Actions span about one second, with 16 steps on a 15 Hz platform. This amortizes the 20 ms increase but does not mean a new policy prediction arrives every control tick.
IV-B Simulation Benchmarking
BridgeData V2 training transfers zero-shot to Simpler WidowX, where DepthVLA averages 74.8% versus 58.8% for the matched baseline. Gains are largest on stacking and eggplant picking, while spoon placement decreases from 81.7% to 75.8%. On LIBERO, one model is trained jointly across all four suites and evaluated over 2,000 trials; the average reaches 94.9% versus 93.6%. Some compared literature models use different suite training and robot-pretraining resources. “No pretraining” in these tables means no additional robot action pretraining, not absence of pretrained vision, language, or depth knowledge.
| Model | Pretrained | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|---|
| Octo-Base [4] | ✓ | 78.9% | 85.7% | 84.6% | 51.1% | 75.1% |
| OpenVLA [1] | ✓ | 84.7% | 88.4% | 79.2% | 53.7% | 76.5% |
| SpatialVLA [9] | ✓ | 88.2% | 89.9% | 78.6% | 55.5% | 78.1% |
| CoT-VLA [11] | ✓ | 81.5% | 91.6% | 87.6% | 69.0% | 83.9% |
| MolmoACT [17] | ✓ | 87.0% | 95.4% | 87.6% | 77.2% | 86.6% |
| DreamVLA [15] | ✓ | 97.5% | 94.0% | 89.5% | 89.5% | 92.6% |
| (re-implemented) [3] | 95.8% | 96.4% | 94.8% | 87.4% | 93.6% | |
| (reported) [3]* | ✓ | 96.8% | 98.8% | 95.8% | 85.2% | 94.2% |
| DepthVLA (Ours) | 96.4% | 98.0% | 95.8% | 89.2% | 94.9% |
* Reported in official JAX implementation.
IV-C Real-World Benchmarking
Both matched policies first train on 100,000 Galaxea trajectories, then fine-tune on 100 demonstrations per real task, or 20 in the few-shot setting. Twenty runs per task score completed substeps in table bussing, microwave operation, and block stacking. The paper reports approximately 79% versus 65% progress with standard tuning and 63% versus 45% with few-shot tuning. Table bussing is comparable; benefits concentrate on stacking and microwave handling. The restored opening diagram labels the real-world baseline 63%, inconsistent with the abstract and experiment text’s 65%; this summary follows the explicit experiment account.
IV-D Ablation Studies
Removing depth pretraining, the auxiliary depth loss, or the attention separation reduces Simpler averages to 51.0%, 56.9%, and 55.6%, respectively, from 74.8%. Freezing the depth expert retains 71.9%, supporting useful transferable geometry even without continued depth adaptation. In a separate LIBERO ablation, direct ground-truth-depth input gives 93.3% versus 94.9% for predicted-depth features. This is an architectural comparison, not evidence that predicted depth is intrinsically more accurate than ground truth; the proposed modality-dominance explanation remains a hypothesis.
| Model | Spoon | Carrot | Block | Eggplant | Average |
|---|---|---|---|---|---|
| (i) | 60.0% | 60.8% | 43.3% | 40.0% | 51.0% |
| (ii) | 69.2% | 60% | 28.3% | 70.0% | 56.9% |
| (iii) | 65.8% | 69.2% | 74.2% | 78.3% | 71.9% |
| (iv) | 66.7% | 65.0% | 2.5% | 88.3% | 55.6% |
| DepthVLA | 75.8% | 71.7% | 62.5% | 89.2% | 74.8% |
| Model | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| (v) | 94.0% | 97.6% | 95.0% | 86.4% | 93.3% |
| DepthVLA | 96.4% | 98.0% | 95.8% | 89.2% | 94.9% |
IV-E Visualization of Depth Prediction
Depth visualizations retain all real and simulated examples, including clutter and object boundaries. They illustrate the branch’s outputs, but do not quantify depth accuracy or independently prove collision safety. Policy gains and ablations provide the stronger evidence for its practical value.
V Conclusion
Depth supervision and intermediate geometric features improve the tested manipulation policies, with extra memory and inference cost. Monocular estimation remains ill posed and can fail at small edges, reflective or transparent objects, and textureless surfaces. Such errors can affect action generation. Multiview depth or pointmap prediction is proposed as future work.