Literature
Display
← All papers

DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoning

Tianyuan Yuan · Yicheng Liu · Chenhao Lu · Zhuoguang Chen · Tao Jiang · Hang Zhao

· ICRA 2026 ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

DepthVLA adds a pretrained monocular-depth expert to a vision-language model and a continuous-action expert. Intermediate geometric features condition action generation, while an auxiliary depth loss helps preserve geometry during policy training. Reported averages improve from 58.8% to 74.8% on Simpler and from 93.6% to 94.9% on LIBERO relative to the reimplemented baseline. The real-robot result is a progress score of 78.5% versus 65.0%, not whole-task success probability.

I INTRODUCTION

Web-trained vision-language representations supply semantics but can miss precise geometry needed for grasping and collision avoidance. DepthVLA transfers a separate depth model’s prior into robot control, rather than relying exclusively on action-data pretraining. The resulting policy uses ordinary camera inputs and incurs a modest measured latency increase. Evidence concerns specific manipulation tasks; geometric representation is inferred from performance and depth predictions rather than directly established as general spatial reasoning.

DepthVLA overview, real and simulated inputs, expert architecture, and comparison charts.
Fig. 1: We propose DepthVLA, a vision-language-action (VLA) model that explicitly incorporates spatial reasoning through a pretrained depth expert. Quantitative comparisons across Simpler, LIBERO, and real-world benchmarks show that DepthVLA consistently outperforms baselines, particularly in tasks requiring fine-grained 3D perception.

II Related Work

Related work covers generalist robot policies, geometric enhancements to action models, and pretrained 3D perception.

II-A Generalist Robot Manipulation Policies

Generalist policies transfer visual-language knowledge into action tokens or continuous action chunks. Large robot datasets help, but do not automatically provide the fine geometric cues required by every manipulation task. DepthVLA adds a separate source of pretrained geometric supervision.

II-B VLAs with Spatial Awareness

Earlier approaches use depth sensors, pseudo-point clouds, future-state prediction, or autoregressively generated spatial tokens. DepthVLA instead gives the action stream access to intermediate depth features within the transformer. This avoids generating a long spatial-token sequence before each action, although it still adds model computation.

II-C 3D Geometry Perception

Monocular and multiview geometry foundation models learn useful priors from diverse 3D data. DepthVLA initializes its depth branch from Depth Anything V2 and further trains it for the architecture used in the policy.

III Method

The method specifies three expert streams and a joint action-and-depth training objective.

Refer to caption
Fig. 2: The proposed mixture-of-transformers (MoT) framework integrates three experts: a vision-language model (VLM) for semantic and language understanding, a depth expert for geometric reasoning, and an action expert for continuous control. Attention layers are shared across experts, while block-wise masking ensures pretrained modules retain their learned abilities. The action expert attends to features from both the VLM and depth expert at every layer to generate actions conditioned on language, visual, and spatial cues.

III-A Problem Formulation and Model Overview

The policy predicts an action chunk from current camera observations, language, and proprioception. Its mixture of transformers combines separately parameterized semantic, depth, and action streams. A block mask restricts semantic and depth tokens to their own streams while allowing action tokens to attend to both. Thus “fully shared attention” means a shared attention computation with explicit restrictions, not unrestricted cross-attention among all tokens.

Equation
𝑨𝒕=πθ​(ot,l,st).\boldsymbol{A_{t}}=\pi_{\theta}\left(o_{t},l,s_{t}\right).

III-B Depth Expert

A DINOv2-based encoder initialized from Depth Anything V2 feeds a transformer decoder and linear depth head. Before robot training, the depth branch learns on diverse 3D datasets with a scale-invariant logarithmic loss, using a scale coefficient of 0.5. The action expert attends to intermediate features rather than only the final depth map, retaining richer geometric information.

Equations
ℒsi​(d^,d)=1n​∑iy2−λ​(1n​∑iy)2,\mathcal{L}_{\text{si}}(\hat{d},d)=\sqrt{\frac{1}{n}\sum_{i}y^{2}-\lambda\left(\frac{1}{n}\sum_{i}{y}\right)^{2}},
where ​y=log⁡d^−log⁡d.\text{where }y=\log\hat{d}-\log d.
TABLE I: Success rates on the Simpler WidowX benchmark. Models are trained on BridgeData V2 and evaluated zero-shot in simulation. The ”Pretrained” column indicates whether the model is pretrained with additional robot action data. DepthVLA achieves the highest average performance.
Model Pretrained Put Spoon Put Carrot Stack Block Pick Eggplant Average
Diffusion Policy [45] ×\times 4.2% 0% 0% 0% 1.0%
Octo-Base [4] ✓ 12.5% 8.3% 0% 43.1% 16.0%
SpatialVLA [9] ✓ 16.7% 25.0% 29.2% 100.0% 34.4%
π0\pi_{0} (re-implemented) [3] ×\times 81.7% 64.2% 30.0% 59.2% 58.8%
DepthVLA (Ours) ×\times 75.8% 71.7% 62.5% 89.2% 74.8%

III-C DepthVLA Policy Training

Flow matching learns continuous action chunks from demonstrations. A retained depth-prediction loss jointly regularizes the geometric branch during policy training. The source equations specify noisy action interpolation, the target flow, and combined objective. Robot datasets without measured depth receive pseudo-labels from existing depth estimators, so geometric supervision depends partly on those models.

Equations
maxθ⁡𝔼p⁡(𝑨t,ot,l,st)​[log⁡πθ​(𝑨t∣ot,l,st)]\max_{\theta}\;\mathbb{E}_{p(\boldsymbol{A}_{t},{o}_{t},l,{s}_{t})}\left[\log\pi_{\theta}(\boldsymbol{A}_{t}\mid{o}_{t},l,{s}_{t})\right]
ℒflow​(θ)=𝔼p⁡(𝑨tτ∣ot,l,st)​[‖vθ​(𝑨tτ,τ,ot,l,st)−𝒖⁡(𝑨tτ∣𝑨t)‖2]\mathcal{L}_{\text{flow}}(\theta)=\mathbb{E}_{p(\boldsymbol{A}_{t}^{\tau}\mid{o}_{t},l,{s}_{t})}\left[\left\|v_{\theta}(\boldsymbol{A}_{t}^{\tau},\tau,{o}_{t},l,{s}_{t})-\boldsymbol{u}(\boldsymbol{A}_{t}^{\tau}\mid\boldsymbol{A}_{t})\right\|^{2}\right]
ℒ=ℒsi+ℒflow.\mathcal{L}=\mathcal{L_{\text{si}}}+\mathcal{L}_{\text{flow}}.

IV Experiments

Evaluation compares simulation success, real-robot subtask progress, and architectural ablations. These measure different outcomes and are kept separate.

IV-A Implementation Details

The implementation uses PaliGemma-3B and approximately 300M-parameter depth and action decoders. The depth encoder adds another roughly 300M parameters. Depth pretraining lasts 50,000 steps; policy training uses AdamW on 32 H100 GPUs. On an RTX 4090, memory rises from 6.7 to 8.0 GB and latency from 190 to 210 ms per chunk. Actions span about one second, with 16 steps on a 15 Hz platform. This amortizes the 20 ms increase but does not mean a new policy prediction arrives every control tick.

IV-B Simulation Benchmarking

BridgeData V2 training transfers zero-shot to Simpler WidowX, where DepthVLA averages 74.8% versus 58.8% for the matched baseline. Gains are largest on stacking and eggplant picking, while spoon placement decreases from 81.7% to 75.8%. On LIBERO, one model is trained jointly across all four suites and evaluated over 2,000 trials; the average reaches 94.9% versus 93.6%. Some compared literature models use different suite training and robot-pretraining resources. “No pretraining” in these tables means no additional robot action pretraining, not absence of pretrained vision, language, or depth knowledge.

TABLE II: Success rates on the LIBERO benchmark across four task suites. The ”Pretrained” column indicates whether the model is pretrained with additional robot action data. DepthVLA outperforms all baselines, showing that explicit depth reasoning improves generalization across diverse manipulation tasks.
Model Pretrained Spatial Object Goal Long Average
Octo-Base [4] ✓ 78.9% 85.7% 84.6% 51.1% 75.1%
OpenVLA [1] ✓ 84.7% 88.4% 79.2% 53.7% 76.5%
SpatialVLA [9] ✓ 88.2% 89.9% 78.6% 55.5% 78.1%
CoT-VLA [11] ✓ 81.5% 91.6% 87.6% 69.0% 83.9%
MolmoACT [17] ✓ 87.0% 95.4% 87.6% 77.2% 86.6%
DreamVLA [15] ✓ 97.5% 94.0% 89.5% 89.5% 92.6%
π0\pi_{0} (re-implemented) [3] ×\times 95.8% 96.4% 94.8% 87.4% 93.6%
π0\pi_{0} (reported) [3]* ✓ 96.8% 98.8% 95.8% 85.2% 94.2%
DepthVLA (Ours) ×\times 96.4% 98.0% 95.8% 89.2% 94.9%

* Reported in π0\pi_{0} official JAX implementation.

IV-C Real-World Benchmarking

Both matched policies first train on 100,000 Galaxea trajectories, then fine-tune on 100 demonstrations per real task, or 20 in the few-shot setting. Twenty runs per task score completed substeps in table bussing, microwave operation, and block stacking. The paper reports approximately 79% versus 65% progress with standard tuning and 63% versus 45% with few-shot tuning. Table bussing is comparable; benefits concentrate on stacking and microwave handling. The restored opening diagram labels the real-world baseline 63%, inconsistent with the abstract and experiment text’s 65%; this summary follows the explicit experiment account.

Refer to caption
Fig. 3: Real-robot experiment platform.
Refer to caption
Fig. 4: Performance of DepthVLA and baseline on three bimanual tasks with standard fine-tuning and few-shot adaptation. DepthVLA shows improvements in tasks requiring precise spatial reasoning and collision avoidance while maintaining comparable performance on simpler small-object manipulation.
Refer to caption
Fig. 5: Qualitative results of DepthVLA’s predicted depth maps across real-world and simulated environments. The predicted depth provides fine-grained geometric cues that guide accurate manipulation, collision avoidance, and precise object grasping. Even in cluttered or challenging scenes, DepthVLA captures the 3D layout robustly, highlighting the effectiveness of the pretrained depth expert in providing spatial awareness.

IV-D Ablation Studies

Removing depth pretraining, the auxiliary depth loss, or the attention separation reduces Simpler averages to 51.0%, 56.9%, and 55.6%, respectively, from 74.8%. Freezing the depth expert retains 71.9%, supporting useful transferable geometry even without continued depth adaptation. In a separate LIBERO ablation, direct ground-truth-depth input gives 93.3% versus 94.9% for predicted-depth features. This is an architectural comparison, not evidence that predicted depth is intrinsically more accurate than ground truth; the proposed modality-dominance explanation remains a hypothesis.

TABLE III: Ablation studies on different design of the depth expert.
Model Spoon Carrot Block Eggplant Average
(i) 60.0% 60.8% 43.3% 40.0% 51.0%
(ii) 69.2% 60% 28.3% 70.0% 56.9%
(iii) 65.8% 69.2% 74.2% 78.3% 71.9%
(iv) 66.7% 65.0% 2.5% 88.3% 55.6%
DepthVLA 75.8% 71.7% 62.5% 89.2% 74.8%
TABLE IV: Comparison between predicted and ground-truth depth inputs. Predicting depth yields stronger performance.
Model Spatial Object Goal Long Average
(v) 94.0% 97.6% 95.0% 86.4% 93.3%
DepthVLA 96.4% 98.0% 95.8% 89.2% 94.9%

IV-E Visualization of Depth Prediction

Depth visualizations retain all real and simulated examples, including clutter and object boundaries. They illustrate the branch’s outputs, but do not quantify depth accuracy or independently prove collision safety. Policy gains and ablations provide the stronger evidence for its practical value.

V Conclusion

Depth supervision and intermediate geometric features improve the tested manipulation policies, with extra memory and inference cost. Monocular estimation remains ill posed and can fail at small edges, reflective or transparent objects, and textureless surfaces. Such errors can affect action generation. Multiview depth or pointmap prediction is proposed as future work.

Concept demonstration

Open image ↗