Literature
Display
← All papers

DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Zachary Teed · Jia Deng

· Advances in Neural Information Processing Systems (NeurIPS) ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

DROID-SLAM jointly refines camera poses and dense inverse-depth maps using learned correspondence updates and a differentiable bundle-adjustment layer. A model trained only on synthetic monocular clips can use monocular, stereo, or RGB-D video at inference. The reported benchmarks show fewer tracking failures and lower trajectory error, with substantial GPU and memory requirements. Code is available.

1 Introduction

The central idea is to let a network revise image correspondences while geometry determines the pose and depth updates. DROID extends RAFT’s recurrent refinement from a pair of flow fields to a graph of many frames, enabling loop closure and global optimization. Its DBA layer directly optimizes per-pixel depth, rather than coefficients of a learned depth basis. [49], [47], [48].

Refer to caption
Figure 1: DROID-SLAM can operate on monocular, stereo, and RGB-D video. It builds a dense 3D map of the environment while simultaneously localizing the camera within the map.

2 Related Work

Classical indirect methods minimize geometric reprojection error after feature detection; direct methods optimize image appearance. DROID uses dense learned image information with a geometric objective. Earlier learned systems often operate on short windows or constrained depth representations; DROID maintains a longer-lived frame graph and global optimization. [31], [32], [5], [15], [16], [61], [9], [1], [47].

3 Approach

Each retained image has a camera pose in SE(3) and an inverse-depth value per pixel. Edges connect frames with overlapping views. The system updates these unknowns repeatedly and adds long-range edges when it recognizes previously mapped regions.

3.1 Feature Extraction and Correlation

Separate feature and context encoders produce maps at one-eighth image resolution. For each graph edge, all-pairs feature similarities form a four-dimensional correlation volume. A four-level pyramid and differentiable local lookup provide matching evidence near the correspondence predicted by the current geometry. [49].

Equation 1
Cu1​v1​u2​v2i​j=⟨gθ​(Ii)u1​v1,gθ​(Ij)u2​v2⟩C_{u_{1}v_{1}u_{2}v_{2}}^{ij}=\langle g_{\theta}(I_{i})_{u_{1}v_{1}},g_{\theta}(I_{j})_{u_{2}v_{2}}\rangle (1)

3.2 Update Operator

A convolutional GRU receives correlation lookups, induced optical flow, the preceding optimization residual, context features, and pooled global information. It predicts a correction to dense correspondences and confidence weights; additional pooled features predict positive depth damping and an upsampling mask.

DBA minimizes confidence-weighted geometric reprojection error against those corrected correspondences. A Gauss–Newton step jointly updates poses and inverse depths. The depth block is diagonal, so a Schur-complement solve eliminates it efficiently before recovering depth updates. Training differentiates through this optimization. Repeated updates are intended to converge to a consistent reconstruction; the paper does not prove global convergence.

Original figure
Figure 2: Illustration of the update operator. The operator acts on edges in the frame graph, predicting flow revisions which are mapped to depth and pose update through the (DBA) layer.
Equations 2–5
𝐆(k+1)=Exp⁡(Δ​𝝃(k))∘𝐆(k),𝐝(k+1)=Δ​𝐝(k)+𝐝(k).\mathbf{G}^{(k+1)}=\Exp(\Delta\boldsymbol{\xi}^{(k)})\circ\mathbf{G}^{(k)},\qquad\mathbf{d}^{(k+1)}=\Delta\mathbf{d}^{(k)}+\mathbf{d}^{(k)}. (2)
𝐩i​j=Πc​(𝐆i​j∘Πc−1​(𝐩i,𝐝i)),𝐩i​j∈ℝH×W×2𝐆i​j=𝐆j∘𝐆i−1.\mathbf{p}_{ij}=\Pi_{c}(\mathbf{G}_{ij}\circ\Pi_{c}^{-1}(\mathbf{p}_{i},\mathbf{d}_{i})),\qquad\mathbf{p}_{ij}\in\mathbb{R}^{H\times W\times 2}\qquad\mathbf{G}_{ij}=\mathbf{G}_{j}\circ\mathbf{G}_{i}^{-1}. (3)
𝐄⁡(𝐆′,𝐝′)=∑(i,j)∈ℰ‖𝐩i​j∗−Πc​(𝐆i​j′∘Πc−1​(𝐩i,𝐝i′))‖Σi​j2Σi​j=diag⁡𝐰i​j.\mathbf{E}(\mathbf{G}^{\prime},\mathbf{d}^{\prime})=\sum_{(i,j)\in\mathcal{E}}\left\lVert\mathbf{p}_{ij}^{*}-\Pi_{c}(\mathbf{G}^{\prime}_{ij}\circ\Pi_{c}^{-1}(\mathbf{p}_{i},\mathbf{d}^{\prime}_{i}))\right\rVert_{\Sigma_{ij}}^{2}\qquad\Sigma_{ij}=\diag\mathbf{w}_{ij}. (4)
[𝐁𝐄𝐄T𝐂]​[Δ​𝝃Δ​𝐝]=[𝐯𝐰]Δ​𝝃=[𝐁−𝐄𝐂−1​𝐄T]−1​(𝐯−𝐄𝐂−1​𝐰)Δ​𝐝=𝐂−1​(𝐰−𝐄T​Δ​𝝃)\begin{bmatrix}\mathbf{B}&\mathbf{E}\\ \mathbf{E}^{T}&\mathbf{C}\end{bmatrix}\begin{bmatrix}\Delta\boldsymbol{\xi}\\ \Delta\mathbf{d}\end{bmatrix}=\begin{bmatrix}\mathbf{v}\\ \mathbf{w}\end{bmatrix}\qquad\begin{array}[]{ll}&\Delta\boldsymbol{\xi}=[\mathbf{B}-\mathbf{E}\mathbf{C}^{-1}\mathbf{E}^{T}]^{-1}(\mathbf{v}-\mathbf{E}\mathbf{C}^{-1}\mathbf{w})\\ &\Delta\mathbf{d}=\mathbf{C}^{-1}(\mathbf{w}-\mathbf{E}^{T}\Delta\boldsymbol{\xi})\end{array} (5)

3.3 Training

Training fixes the first two poses to ground truth to remove coordinate and scale ambiguity. Seven-frame clips are sampled with at least 50% overlap and approximately 8–96 pixels of adjacent flow. Pose and geometry-induced flow losses supervise every iteration, with later iterations weighted more heavily. LieTorch supports differentiation of rigid transforms. [50].

3.4 SLAM System

The front end selects keyframes, estimates incoming motion, and performs local bundle adjustment. The asynchronous back end rebuilds a co-visibility graph and refines all keyframes, including loop connections. Initialization gathers twelve sufficiently separated frames and runs ten updates. Non-keyframe poses are recovered with motion-only optimization; evaluation uses the complete trajectory.

Memory-efficient correlation lookup and a sparse CUDA implementation support large graphs. RGB-D adds a penalty toward measured inverse depth while allowing noisy or missing measurements to be refined. Stereo introduces a second camera and fixes the calibrated relative transform. Neither change requires retraining the correspondence network.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 3: DROID-SLAM can generalize to new datasets. In order, we show results from Tanks & Temples [21], ScanNet [10], Sintel [3], and ETH-3D [42]; all using monocular video.

4 Experiments

One model is trained on TartanAir monocular video for 250,000 steps using seven-frame clips and fifteen recurrent updates; training takes about one week on four RTX 3090 GPUs. Evaluation emphasizes trajectory accuracy and failure rate, rather than quantitatively testing dense map quality.

On the TartanAir competition metric, monocular and stereo error are 0.129 and 0.047, about 62% and 60% below the best listed competing submissions. The separate hard-sequence monocular table reports mean ATE of 0.24 m. These are different metrics and test summaries. [55], [54], [48].

On EuRoC monocular video, mean ATE is 0.022 m across eleven sequences. The reported 43% improvement over ORB-SLAM3 compares only the ten sequences that baseline completes; the larger 82% reduction uses methods with no failures. On the nine TUM freiburg1 sequences, DROID completes every run with mean ATE of 0.038 m. The table includes heterogeneous baselines: DeepTAM receives depth, and TartanVO uses ground truth to scale relative motion. [2], [5], [44], [60], [54].

For RGB-D ETH3D, DROID successfully tracks 30 of 32 test sequences with image data and achieves test AUC of 207.79 versus 153.47 for BAD-SLAM. Completely dark sequences without image data are excluded. These rankings describe the paper’s evaluation, not a current leaderboard. [42].

Real-time experiments use two RTX 3090 GPUs, reduced resolution, and every-other-frame processing, averaging the incoming camera rates of 20 frames/s on EuRoC and 30 on TUM. TartanAir runs at 8 frames/s and is not real time. Some full-sequence evaluations require 24 GB of GPU memory; computational resources are the authors’ main stated limitation.

Monocular MH000 MH001 MH002 MH003 MH004 MH005 MH006 MH007 Avg
ORB-SLAM [31] 1.30 0.04 2.37 2.45 X X 21.47 2.73 -
DeepV2D [48] 6.15 2.12 4.54 3.89 2.71 11.55 5.53 3.76 5.03
TartanVO [54] 4.88 0.26 2.00 0.94 1.07 3.19 1.00 2.04 1.92
Ours 0.08 0.05 0.04 0.02 0.01 1.31 0.30 0.07 0.24
Table 1: Results on the TartanAir monocular benchmark.
Mono. Stereo
OV2SLAM [17] 0.510 0.182
VOLDOR [28] + COLMAP [41] 0.440 0.177
SuperGlue [39] + SuperPoint [13] + COLMAP [41] 0.340 0.119
Ours 0.129 0.047
Table 2: Results on the TartanAir test set, compared with the top 3 submission to the ECCV 2020 SLAM competition. The score is computed using normalized relative pose error for all possible sequences of length {5,10,15,…,40}\{5,10,15,...,40\} meters, see competition page for details.
MH01 MH02 MH03 MH04 MH05 V101 V102 V103 V201 V202 V203 Avg
Deep/Hyb. DeepFactors [9] 1.587 1.479 3.139 5.331 4.002 1.520 0.679 0.900 0.876 1.905 1.021 2.040
DeepV2D [48]† 0.739 1.144 0.752 1.492 1.567 0.981 0.801 1.570 0.290 2.202 2.743 1.298
DeepV2D (Tartan Air)† 1.614 1.492 1.635 1.775 1.013 0.717 0.695 1.483 0.839 1.052 0.591 1.173
TartanVO1 [54]† 0.639 0.325 0.550 1.153 1.021 0.447 0.389 0.622 0.433 0.749 1.152 0.680
D3VO + DSO [58]† - - 0.08 - 0.09 - - 0.11 - 0.05 0.19 -
Classical ORB-SLAM [31] 0.071 0.067 0.071 0.082 0.060 0.015 0.020 X 0.021 0.018 X -
DSO [15]† 0.046 0.046 0.172 3.810 0.110 0.089 0.107 0.903 0.044 0.132 1.152 0.601
SVO [18]† 0.100 0.120 0.410 0.430 0.300 0.070 0.210 X 0.110 0.110 1.080 -
DSM [61] 0.039 0.036 0.055 0.057 0.067 0.095 0.059 0.076 0.056 0.057 0.784 0.126
ORB-SLAM3 [5] 0.016 0.027 0.028 0.138 0.072 0.033 0.015 0.033 0.023 0.029 X -
Ours (odometry only)† 0.163 0.121 0.242 0.399 0.270 0.103 0.165 0.158 0.102 0.115 0.204 0.186
Ours 0.013 0.014 0.022 0.043 0.043 0.037 0.012 0.020 0.017 0.013 0.014 0.022
Table 3: Monocular SLAM on the EuRoC datasets, ATE[m]. † denotes visual odometry methods.
360 desk desk2 floor plant room rpy teddy xyz avg
ORB-SLAM2 [32] X 0.071 X 0.023 X X X X 0.010 -
ORB-SLAM3 [5] X 0.017 0.210 X 0.034 X X X 0.009 -
DeepTAM1 [60] 0.111 0.053 0.103 0.206 0.064 0.239 0.093 0.144 0.036 0.116
TartanVO2 [54] 0.178 0.125 0.122 0.349 0.297 0.333 0.049 0.339 0.062 0.206
DeepV2D [48] 0.243 0.166 0.379 1.653 0.203 0.246 0.105 0.316 0.064 0.375
DeepV2D (TartanAir) 0.182 0.652 0.633 0.579 0.582 0.776 0.053 0.602 0.150 0.468
DeepFactors [9] 0.159 0.170 0.253 0.169 0.305 0.364 0.043 0.601 0.035 0.233
Ours 0.111 0.018 0.042 0.021 0.016 0.049 0.026 0.048 0.012 0.038
Table 4: ATE on the TUM-RGBD benchmark. All methods are provided mono. video, 1except DeepTAM which uses RGB-D and 2TartanVO which uses ground truth to scale relative pose.
Method AUC (train) AUC (test)
BundleFusion [11] 84.10 33.84
ElasticFusion [57] 89.06 34.02
RFusion [56] 17.37 51.94
DVO-SLAM [20] 193.89 71.83
ORB-SLAM2 [32] 156.10 104.28
BAD-SLAM [42] 280.05 153.47
Ours 340.42 207.79
Original figure
Figure 4: Generalization results on the RGB-D ETH3D-SLAM benchmark. (Left) Our method, which is trained only on the synthetic TartanAir dataset, ranks 1st on both the train and test splits. (Right) Plot of the number successful trajectories as a function of ATE. Our method successfully tracks 30/32 of the datasets where image data is available.

5 Conclusion

DROID combines learned matching with explicit multiview geometry and global optimization. Its cross-dataset and cross-sensor results support this design, while compute demands, remaining tracking failures, and the absence of quantitative dense-reconstruction evaluation limit the conclusions. The work acknowledges National Science Foundation support.

Appendix A Additional Results

With stereo input on EuRoC, the same monocular-trained network obtains mean ATE of 0.024 m versus ORB-SLAM3’s 0.084 m, a reported 71% reduction. The fixed calibrated baseline provides metric scale; the preserved table gives all sequence results. [2], [5].

MH01 MH02 MH03 MH04 MH05 V101 V102 V103 V201 V202 V203 Avg
D3VO + DSO [58] - - 0.08 - 0.09 - - 0.11 - 0.05 - -
ORB-SLAM2 [32] 0.035 0.018 0.028 0.119 0.060 0.035 0.020 0.048 0.037 0.035 - -
VINS-Fusion [36] 0.540 0.460 0.330 0.780 0.500 0.550 0.230 - 0.230 0.200 - -
SVO [18] 0.040 0.070 0.270 0.170 0.120 0.040 0.040 0.070 0.050 0.090 0.790 0.159
ORB-SLAM3 [5] 0.029 0.019 0.024 0.085 0.052 0.035 0.025 0.061 0.041 0.028 0.521 0.084
Ours 0.015 0.013 0.035 0.048 0.040 0.037 0.011 0.020 0.018 0.015 0.017 0.024
Table 5: Stereo SLAM on the EuRoC datasets, ATE[m].

Appendix B Ablations

Validation ablations examine stereo observations, global optimization, local keyframe count, and pooled global context. The experiments use five local keyframes. Training the correspondence network with DBA is important: training on flow alone and adding bundle adjustment only at inference produces unstable tracking. These ablations use the TartanAir validation split.

Refer to caption
Figure 5: Visualizations of keyframe image, depth, flow and confidence estimates.
Original figure
Original figure
Figure 6: (Left) we show the performance of the system with different inputs (monocular vs. stereo) and whether global optimization is performed in addition to local BA (local vs. full). (Right) Tracking accuracy as a function of the number of keyframes. We use 5 keyframes (bold) in our experiments.
Original figure
Original figure
Figure 7: (Left) Impact of global context in the update operator. (Right) Impact of using the bundle adjustment layer during training vs training directly on optical flow, then applying BA at test time.

Appendix C Camera Model and Jacobians

The appendix derives homogeneous pinhole projection, inverse-depth backprojection, and the Jacobians needed for pose and depth optimization. Local rigid-motion perturbations and the adjoint transform relate derivatives in the two camera frames; the chain rule produces the full reprojection Jacobians. The original fourteen numbered equations, including these derivations, are retained in their respective sections.

Equations 6–14
𝐩′=Πc​(𝐆i​j⋅Π−1​(𝐩,𝐝))𝐆i​j=𝐆j∘𝐆i−1\mathbf{p}^{\prime}=\Pi_{c}(\mathbf{G}_{ij}\cdot\Pi^{-1}(\mathbf{p},\mathbf{d}))\qquad\mathbf{G}_{ij}=\mathbf{G}_{j}\circ\mathbf{G}_{i}^{-1} (6)
Πc​(𝐗)=(fx​XZ+cyfy​YZ+cy)Πc−1​(𝐩,d)=(px−cxfxpy−cyfy1d).\Pi_{c}(\mathbf{X})=\begin{pmatrix}f_{x}\frac{X}{Z}+c_{y}\\ f_{y}\frac{Y}{Z}+c_{y}\end{pmatrix}\qquad\Pi_{c}^{-1}(\mathbf{p},d)=\begin{pmatrix}\frac{p_{x}-c_{x}}{f_{x}}\\ \frac{p_{y}-c_{y}}{f_{y}}\\ 1\\ d\end{pmatrix}. (7)
∂Πc​(𝐗)∂𝐗=(fx​1Z0−fx​XZ200fy​1Z−fy​YZ20)∂Πc−1​(𝐩,d)∂d=(0001).\frac{\partial\Pi_{c}(\mathbf{X})}{\partial\mathbf{X}}=\begin{pmatrix}f_{x}\frac{1}{Z}&0&-f_{x}\frac{X}{Z^{2}}&0\\ 0&f_{y}\frac{1}{Z}&-f_{y}\frac{Y}{Z^{2}}&0\\ \end{pmatrix}\qquad\frac{\partial\Pi_{c}^{-1}(\mathbf{p},d)}{\partial d}=\begin{pmatrix}0\\ 0\\ 0\\ 1\end{pmatrix}. (8)
𝐗′=Exp⁡(ξj)⋅𝐆j⋅(Exp⁡(ξi)⋅𝐆i)−1⋅𝐗=Exp⁡(ξj)⋅𝐆j⋅𝐆i−1⋅Exp⁡(−ξi)⋅𝐗\mathbf{X}^{\prime}=\Exp(\xi_{j})\cdot\mathbf{G}_{j}\cdot(\Exp(\xi_{i})\cdot\mathbf{G}_{i})^{-1}\cdot\mathbf{X}=\Exp(\xi_{j})\cdot\mathbf{G}_{j}\cdot\mathbf{G}_{i}^{-1}\cdot\Exp(-\xi_{i})\cdot\mathbf{X} (9)
𝐗′=Exp⁡(ξj)⋅Exp⁡(−Adj𝐆j​𝐆i−1⁡ξi)⋅𝐆j⋅𝐆i−1⋅𝐗\mathbf{X}^{\prime}=\Exp(\xi_{j})\cdot\Exp(-\Adj_{\mathbf{G}_{j}\mathbf{G}_{i}^{-1}}\xi_{i})\cdot\mathbf{G}_{j}\cdot\mathbf{G}_{i}^{-1}\cdot\mathbf{X} (10)
∂𝐗′∂ξj=(W′000Z′−Y′0W′0−Z′0X′00W′Y′−X′0000000)\frac{\partial\mathbf{X}^{\prime}}{\partial\xi_{j}}=\begin{pNiceMatrix}\phantom{-}W^{\prime}\phantom{-}&0&0&0&Z^{\prime}&-Y^{\prime}\phantom{-}\\ 0&\phantom{-}W^{\prime}\phantom{-}&0&-Z^{\prime}\phantom{-}&0&X^{\prime}\\ 0&0&\phantom{-}W^{\prime}\phantom{-}&Y^{\prime}&-X^{\prime}\phantom{-}&0\\ 0&0&0&0&0&0\\ \end{pNiceMatrix}\phantom{\cdot\Adj_{\mathbf{G}_{j}\mathbf{G}_{i}^{-1}}} (11)
∂𝐗′∂ξi=−(W′000Z′−Y′0W′0−Z′0X′00W′Y′−X′0000000)⋅Adj𝐆j​𝐆i−1\frac{\partial\mathbf{X}^{\prime}}{\partial\xi_{i}}=-\begin{pNiceMatrix}\phantom{-}W^{\prime}\phantom{-}&0&0&0&Z^{\prime}&-Y^{\prime}\phantom{-}\\ 0&\phantom{-}W^{\prime}\phantom{-}&0&-Z^{\prime}\phantom{-}&0&X^{\prime}\\ 0&0&\phantom{-}W^{\prime}\phantom{-}&Y^{\prime}&-X^{\prime}\phantom{-}&0\\ 0&0&0&0&0&0\\ \end{pNiceMatrix}\cdot\Adj_{\mathbf{G}_{j}\mathbf{G}_{i}^{-1}} (12)
∂𝐩′∂ξj=∂Πc​(𝐗′)∂𝐗′​∂𝐗′∂ξj,∂𝐩′∂ξi=∂Πc​(𝐗′)∂𝐗′​∂𝐗′∂ξi\frac{\partial\mathbf{p}^{\prime}}{\partial\xi_{j}}=\frac{\partial\Pi_{c}(\mathbf{X}^{\prime})}{\partial\mathbf{X}^{\prime}}\frac{\partial\mathbf{X}^{\prime}}{\partial\xi_{j}},\qquad\frac{\partial\mathbf{p}^{\prime}}{\partial\xi_{i}}=\frac{\partial\Pi_{c}(\mathbf{X}^{\prime})}{\partial\mathbf{X}^{\prime}}\frac{\partial\mathbf{X}^{\prime}}{\partial\xi_{i}} (13)
∂𝐩′∂d=∂Πc​(𝐗′)∂𝐗′​∂𝐗′∂𝐗​∂Π−1​(𝐩,d)∂d=∂Πc​(𝐗′)∂𝐗′=∂Πc​(𝐗′)∂𝐗′​(txtytz1)\frac{\partial\mathbf{p}^{\prime}}{\partial d}=\frac{\partial\Pi_{c}(\mathbf{X}^{\prime})}{\partial\mathbf{X}^{\prime}}\frac{\partial\mathbf{X}^{\prime}}{\partial\mathbf{X}}\frac{\partial\Pi^{-1}(\mathbf{p},d)}{\partial d}=\frac{\partial\Pi_{c}(\mathbf{X}^{\prime})}{\partial\mathbf{X}^{\prime}}=\frac{\partial\Pi_{c}(\mathbf{X}^{\prime})}{\partial\mathbf{X}^{\prime}}\begin{pmatrix}t_{x}\\ t_{y}\\ t_{z}\\ 1\end{pmatrix} (14)

Appendix D Network Architecture

Both encoders use six residual blocks and output features at one-eighth resolution. The feature encoder uses instance normalization and 128 channels; the context encoder has no normalization and 256 channels. The update diagram shows how context, correlation, and flow features produce correspondence revisions and confidence weights.

Original figure
Figure 8: Architecture of the feature and context encoders. Both extract features at 1/8 the input image resolution using a set of 6 basic residual blocks. Instance normalization is used in the feature encoder; no normalization is used in the context encoder. The feature encoder outputs features with dimension D=128 which the context encoder outputs features with dimension D=256.
Original figure
Figure 9: Architecture of the update operator. During each iteration, context, correlation, and flow features get injected into the GRU. The revision (r) and confidence weights (w) are predicted from the updated hidden state.
Concept demonstration

Open image ↗