Literature
Display
← All papers

LONG3R: Long Sequence Streaming 3D Reconstruction

Zhuoguang Chen · Minghui Qin · Tianyuan Yuan · Zhe Liu · Hang Zhao

· ICCV 2025 ↗

Computer Vision9 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 12-page v1 source, retaining all seven numbered figures and their panels, six tables, nine numbered equations, the additional patch-distance formula and all 89 references.

At a glance

LONG3R reconstructs 3D pointmaps from image sequences using a recurrent network and bounded memory. Attention gating selects relevant history, a refined decoder alternates between that history and next-frame features, and voxel pruning limits spatial redundancy. Training progresses from five-frame to 10- and 32-frame sequences. At 224×224 resolution, it improves reconstruction over the tested streaming baselines on sequences of up to 200 sampled frames while running at roughly 21–22 frames per second on an RTX 3090. It still exhibits drift and viewpoint/dynamic-scene limitations.

1 Introduction

Pairwise pointmap models avoid much of the iterative geometry pipeline, but extending them to long sequences requires retaining useful context without accumulating redundant memory or errors. LONG3R builds on Spann3R’s recurrent reconstruction design [63] with selective memory access, repeated interaction with memory and adaptive 3D storage. “Long” means tens to hundreds of frames in this study, not indefinite error-free mapping.

The three proposed components are combined with a curriculum that expands training sequences. The decoder explicitly uses features from frame t+1 to predict frame t, so this streaming design includes one-frame look-ahead rather than strictly immediate prediction from present and past observations alone.

Refer to caption
Figure 1: (Top): Overview of our LONG3R framework, which integrates a 3D spatio-temporal memory module and memory gating to refine features through a dual-source refined decoder. Here, AM denotes All Memory and RM denotes Relevant Memory. (Bottom): Qualitative comparison of 3D reconstructions from Spann3R, our method LONG3R, and the ground truth. LONG3R can achieve more accurate point prediction.

2 Related Work

Figure 2 retains all three architecture panels, including the gating and alternating decoder blocks. The related work distinguishes geometric optimization, learned reconstruction and recurrent streaming models.

Refer to caption
(a) Main Architecture.
Original figure
(b) Attention-based Memory Gating.
Original figure
(c) Dual-Source Refined Decoder.
Figure 2: Method Overview. (a) Illustrates the overall architecture, where image features FtIF^{I}_{t} first interact with Ft−1IF^{I}_{t-1} in the Coarse Decoder to generate FtcF^{c}_{t}, after which a memory-gating module filters irrelevant entries from the spatio-temporal memory FmemF_{\text{mem}}. The Dual-Source Refined Decoder subsequently interacts with both the filtered memory and features from t+1t+1, ultimately generating the pointmap tt. (b) Details the attention-based memory gating module, which selects relevant information from the memory. (c) Illustrates the dual-source refined decoder, which alternately interacts with the next-frame features and relevant memory features through multiple self- and cross-attention layers to optimize memory information utilization and maintain alignment with the subsequent frame.

Traditional 3D Reconstruction.

Structure from motion and SLAM combine feature tracking, pose estimation and geometric optimization [49][61]. NeRFs and Gaussian splatting provide learned or optimized scene representations [37][27]. LONG3R instead regresses pointmaps using learned priors. This trades explicit global optimization for feedforward speed; it does not imply that every classical or optimized system requires the same calibration or has the same limitations.

Learning-Based 3D Reconstruction.

DUSt3R predicts geometry from image pairs, while MASt3R adds metric-scale geometry and matching [68][29]. Multi-view and recurrent models extend the representation to larger image sets [59][67]. LONG3R uses this pointmap foundation to improve how temporal and spatial history interact with each new view, rather than introducing a new renderer or globally optimized map representation.

Streaming Reconstruction.

Spann3R stores spatial features in memory, and CUT3R maintains a persistent recurrent state [63][67]. LONG3R reduces redundant spatial tokens and revisits relevant memory across decoder blocks. It has no explicit loop-closure or global post-optimization stage, so improved local/contextual consistency should not be confused with a guarantee against accumulated drift.

3 Method

Each frame is encoded and coarsely decoded, queries historical memory, and receives refined prediction using both relevant memory and the next frame. The pointmap supplies geometry for subsequent memory updates. All original mathematical definitions remain below.

3.1 Feature Encoding and Coarse Decoding

A Vision Transformer converts image patches into feature tokens. The coarse decoder applies self-attention, cross-attention and an MLP in each PairwiseBlock, interacting with the corresponding refined tokens from the preceding frame. Equation 2 specifies previous refined features; the architecture caption describes that input more loosely as previous image features.

Equations 1–2
𝑭tI=Encoder⁡(𝑰t).\bm{F}^{I}_{t}=\mathrm{Encoder}(\bm{I}_{t}). (1)
𝑭t,ic=PairwiseBlockic​(𝑭t,i−1c,𝑭t−1,i−1r),\bm{F}^{c}_{t,i}=\text{PairwiseBlock}^{c}_{i}\left(\bm{F}^{c}_{t,i-1},\bm{F}^{r}_{t-1,i-1}\right), (2)

3.2 Attention-based Memory Gating

Current coarse tokens attend to all stored keys and values, producing fused features and a P×S attention matrix. A memory token survives gating if any current patch assigns it attention above τ=5×10⁻⁴. The same mask selects keys and values for later refinement. This reduces downstream attention cost, but the initial gating operation still attends to the full memory bank. Relevance is measured by learned attention, not independently verified geometric usefulness.

Equations 3–6
Wt=Softmax⁡(𝑭tc​(𝑭memK)⊤C),W_{t}=\mathrm{Softmax}\!\left(\frac{\bm{F}^{c}_{t}\left(\bm{F}^{K}_{\text{mem}}\right)^{\top}}{\sqrt{C}}\right), (3)
𝑭tfuse=Wt​𝑭memV,\bm{F}_{t}^{\text{fuse}}=W_{t}\bm{F}^{V}_{\text{mem}}, (4)
δ⁡(s)={1,if ​maxp​Wt​(p,s)>τ,0,otherwise.\delta(s)=\begin{cases}1,&\text{if }\max_{p}W_{t}(p,s)>\tau,\\ 0,&\text{otherwise.}\end{cases} (5)
𝑭r_mem={𝑭mem​(s)∣δ⁡(s)=1}.\bm{F}_{\text{r\_mem}}=\{\,\bm{F}_{\text{mem}}(s)\mid\delta(s)=1\,\}. (6)

3.3 Dual-Source Refined Decoder

Odd refined-decoder blocks interact with coarse tokens from frame t+1; even blocks attend to selected memory. A DPT prediction head converts the refined representation into a pointmap. Alternating sources is intended to avoid directly mixing mismatched feature spaces. That explanation is a design interpretation: the experiment compares decoder variants, without directly measuring feature-space alignment.

Equation 7
𝑭t,ir={PairwiseBlock⁡(𝑭t,i−1r,𝑭t+1,i−1c),i​ odd,MemoryBlock⁡(𝑭t,i−1r,𝑭r_mem),i​ even,\bm{F}^{r}_{t,i}=\begin{cases}\mathrm{PairwiseBlock}(\bm{F}^{r}_{t,i-1},\bm{F}^{c}_{t+1,i-1}),&i\text{ odd},\\ \mathrm{MemoryBlock}(\bm{F}^{r}_{t,i-1},\bm{F}_{\text{r\_mem}}),&i\text{ even},\end{cases} (7)

3.4 3D Spatio-Temporal Memory

Short-term memory stores patch keys and values from a recent time window. Long-term memory stores older features grouped by their predicted 3D positions. The experiments use ten recent frames and a 3,000-token long-term budget. These bounds concern feature memory, not storage of every predicted pointmap in the final accumulated reconstruction.

Original figure
Figure 3: 3D Spatial Memory Pruning. Memory tokens are grouped into voxels based on 3D positions (illustrated here in a 2D simplified view), with only the token having the highest accumulated attention weight retained per voxel. Darker blue dots indicate higher attention weights.
Method 7Scenes NRGBD FPS
Acc↓\downarrow Comp↓\downarrow NC↑\uparrow Acc↓\downarrow Comp↓\downarrow NC↑\uparrow
Mean Med. Mean Med. Mean Med. Mean Med. Mean Med. Mean Med.
F-Recon [74] 12.43 7.62 5.54 2.31 61.89 68.85 28.55 20.59 15.05 6.31 65.47 75.77 ≪\ll1
Dust3R [68] 3.01 1.47 5.11 2.79 58.83 63.73 3.94 2.48 5.31 3.58 62.62 72.29 ≤\leq3
MASt3R [29] 2.82 1.56 5.26 3.24 58.22 62.46 3.85 2.54 5.50 3.62 60.92 68.67 ≤\leq3
MV-DUSt3R [59] 2.92 1.24 2.49 0.78 66.42 76.07 3.76 1.99 2.55 0.92 81.16 95.39 ∼\sim15
MV-DUSt3R+ [59] 2.93 1.07 8.63 0.95 66.38 76.18 3.47 1.60 3.69 0.85 84.33 97.27 ∼\sim3
CUT3R [67] 7.73 3.57 7.75 1.83 65.74 73.98 12.48 5.57 6.34 2.35 75.84 90.05 ∼\sim23
Spann3R[63] 3.42 1.48 2.41 0.85 66.35 76.25 6.91 3.15 2.91 1.10 77.75 93.71 ∼\sim22
Ours 2.57 1.14 2.08 0.73 66.55 76.43 6.66 2.54 3.11 1.21 77.56 93.08 ∼\sim22
Table 1: Quantitative results on 7Scenes [52] and NRGBD [3] datasets. All models are using 224×224224\times 224 image inputs.
Refer to caption
Figure 4: Qualitative comparisons. We present a comparison of reconstruction results on Office-06 and Office-09 from the 7Scenes [52] dataset with the Spann3R [63] and CUT3R [67] methods. In comparison with other online reconstruction methods, our approach achieves superior spatial consistency (e.g., the regions enclosed by the red bounding boxes) while preserving real-time performance.

Adaptive Voxel Size.

A weighted average of predicted point coordinates gives each patch token a 3D location. Its local spacing is the mean Euclidean distance to eight neighboring image patches; this additional original formula is retained as a display. The smallest token spacing sets the image voxel size, and Equation 8 averages previous image voxel sizes to obtain the scene voxel size.

This adapts the discretization to relative scene scale without choosing a universal metric voxel width. Because spacing comes from predicted geometry and uses a minimum, it is not an independent geometric calibration. Initial-frame handling, boundary neighborhoods and the interaction between adaptive voxels and the hard capacity limit are not fully detailed in the paper.

Equations 8di=0.125​∑j∈𝒩⁡(i)‖Pi−Pj‖2d_{i}=\mathrm{0.125}\sum_{j\in\mathcal{N}(i)}\|P_{i}-P_{j}\|_{2}
vimg=mini⁡diandvscene=1t−1​∑j=1t−1vimg,j,v_{\text{img}}=\min_{i}d_{i}\quad\text{and}\quad v_{\text{scene}}=\frac{1}{t-1}\sum_{j=1}^{t-1}v_{\text{img},j}, (8)

3D Spatial Memory Pruning.

Tokens in the same voxel compete by accumulated attention, and only the highest-weight token is retained. Figure 3 preserves the before/after example. Pruning controls spatial redundancy, but discarded history can no longer be revisited through that token; the representation is an approximation to the observed scene.

Method Replica100 Replica200 FPS
Acc↓\downarrow Comp↓\downarrow NC↑\uparrow Acc↓\downarrow Comp↓\downarrow NC↑\uparrow
Mean Med. Mean Med. Mean Med. Mean Med. Mean Med. Mean Med.
Dust3R [68] 6.34 3.99 6.44 3.68 61.67 69.27 4.99 2.76 4.63 2.59 62.26 70.76 ≤\leq3
MASt3R [29] 5.10 2.96 6.00 3.43 61.81 69.52 5.26 3.23 7.31 3.75 58.03 62.81 ≤\leq3
MV-DUSt3R [59] 10.41 6.48 4.34 1.22 73.76 88.36 17.02 11.70 5.10 1.36 66.74 78.24 ∼\sim7
MV-DUSt3R+ [59] 5.28 3.26 2.56 0.89 79.07 93.63 11.79 8.37 5.64 1.53 70.66 83.86 ∼\sim1
CUT3R [67] 20.44 14.64 5.67 2.32 69.63 84.31 28.3 20.68 6.61 1.88 63.95 73.85 ∼\sim23
Spann3R[63] 14.08 8.88 4.67 1.61 72.46 88.98 16.29 10.17 4.02 1.16 68.56 82.80 ∼\sim21
Ours 11.46 7.55 3.68 1.24 73.29 89.86 11.93 7.42 2.73 0.87 68.67 82.92 ∼\sim21
Table 2: Quantitative results on Replica [54] datasets. All models are using 224×224224\times 224 image inputs. Replica100 and Replica200 represent sequence lengths of 100 and 200 frames, respectively.
Refer to caption
Figure 5: Qualitative comparisons. We present a comparison of reconstruction results on Office-0 from the Replica [54] dataset with the Spann3R [63] and CUT3R [67] methods. During streaming observations within the same scene, existing methods suffer from increasing ambiguity due to error accumulation. In contrast, our approach effectively suppresses cumulative drift and maintains spatial consistency.

3.5 Training

Training combines geometric regression with a sequence-length curriculum. The paper studies memory and decoder variants but does not report an isolated ablation of the entire curriculum.

Loss Function.

The total objective adds confidence-aware 3D regression and a scale penalty, following Spann3R and DUSt3R [63][68]. The scale term discourages predicted point clouds whose average distance exceeds the ground truth. Only the combined objective is written here; the complete definitions of its two inherited terms are referred to prior work. This objective does not establish globally consistent metric scale across arbitrary streams.

Equation 9
ℒ=ℒconf+ℒscale.\mathcal{L}=\mathcal{L}_{\mathrm{conf}}+\mathcal{L}_{\mathrm{scale}}. (9)

Two-stage Curriculum Training.

The first stage samples five frames per sequence and trains the network. The second freezes the ViT encoder and fine-tunes remaining modules first on ten-frame and then 32-frame sequences. The longer context encourages use of memory while reducing the cost of backpropagating through the encoder. Training length remains shorter than the 100/200-frame evaluation sequences.

4 Experiments

The evaluation covers dense reconstruction, camera pose and component ablations. The original tables preserve all mean/median metrics and speed columns, including cases where LONG3R is not best.

4.1 Setup

Training mixes six datasets; reconstruction evaluation uses three other datasets. Comparisons share the reported image resolution and evaluation GPU, but models still differ in architecture, pretrained weights, optimization and training exposure.

Training Datasets.

The mixture is Habitat, ARKitScenes, BlendedMVS, ScanNet++, Co3D-v2 and ScanNet [48][4][80][82][45][10]. It includes real and synthetic examples with metric and normalized scales. Training scenes are static, which limits conclusions about large moving objects.

Baselines.

Spann3R and CUT3R are the main streaming baselines. DUSt3R and MASt3R include post-processing, while MV-DUSt3R and its plus variant operate offline. All listed tests use 224×224 images on one 24GB RTX 3090. This makes the reported timing setting concrete without equating offline throughput, streaming latency and all end-to-end application costs.

Implementation details.

The encoder is ViT-Large initialized from DUSt3R [14][68]. Stage one uses AdamW, learning rate 1.12×10⁻⁴, batch ten per GPU and 120 epochs. Stage two uses learning rate 10⁻⁵ for twelve epochs each at ten and 32 views. Training takes about 28 plus 20 hours on sixteen A100s, approximately 768 GPU-hours, excluding DUSt3R pretraining. Repeated-seed confidence intervals are not reported.

Method 7Scenes TUM ScanNet
ATE RPEt RPEr ATE RPEt RPEr ATE RPEt RPEr
Spann3R [63] 12.64 6.15 1.88 5.66 2.13 0.59 9.83 2.30 0.66
CUT3R [67] 12.40 7.65 2.34 6.25 2.55 0.69 14.27 3.58 0.92
Ours 8.72 5.03 1.67 5.40 2.36 0.60 6.44 2.14 0.61
Table 3: Evaluation of Camera Pose Estimation (cm/∘). All three models using 224 × 224 image inputs.

4.2 3D Reconstruction

Reconstruction is evaluated on 7Scenes, NRGBD and Replica with accuracy, completion and normal consistency [52][3][54]. Lower distance errors in centimeters and higher normal consistency are better. Replica samples 100 or 200 observations from the same scene; this changes observation density and processed frame count, not necessarily the physical path length or duration.

Quantitative comparisons.

On 7Scenes, LONG3R’s mean accuracy/completion are 2.57/2.08cm, compared with Spann3R’s 3.42/2.41cm and CUT3R’s 7.73/7.75cm, at approximately 22 FPS. On NRGBD, its mean accuracy 6.66cm improves on Spann3R’s 6.91cm, but completion 3.11cm is worse than 2.91cm and normal consistency is also slightly lower. Improvement is therefore not uniform across metrics.

On Replica, LONG3R’s mean accuracy changes from 11.46cm at 100 frames to 11.93cm at 200, whereas Spann3R worsens from 14.08 to 16.29cm and CUT3R from 20.44 to 28.30cm. LONG3R runs at roughly 21 FPS. At 200 frames, MASt3R’s mean accuracy is much better at 5.26cm, but its reported speed is at most 3 FPS. The result supports a speed/quality trade-off among these settings rather than elimination of drift or dominance over optimized methods.

Method 7Scenes NRGBD FPS
Acc↓\downarrow Comp↓\downarrow Acc↓\downarrow Comp↓\downarrow
Mean Med. Mean Med. Mean Med. Mean Med.
w/o Gating 2.53 1.12 2.12 0.74 6.72 3.14 2.91 1.20 18.0
w/ Gating 2.57 1.14 2.08 0.73 6.66 3.11 2.92 1.21 21.4
Table 4: Attention-based Memory Gating ablation study on 7Scenes [52] and NRGBD [3].
Original figure
Figure 6: Memory number comparison. This figure illustrates the number of memory tokens before (all memory) and after (relevant memory) passing through the memory gating module.

Qualitative comparisons.

Figures 4–5 preserve both office scenes and both Replica sampling densities for CUT3R, Spann3R, LONG3R and ground truth. Highlighted regions show selected reductions in duplicated or distorted surfaces. These qualitative examples complement aggregate metrics; they do not demonstrate every viewpoint, dynamic scene or revisitation pattern.

4.3 Camera Pose Estimation

Pose errors are reported as absolute translation error and relative translation/rotation errors, with translation in centimeters and rotation in degrees. LONG3R improves all three listed measures on 7Scenes and ScanNet: ATE is 8.72cm and 6.44cm, respectively. ScanNet is also one of the training dataset sources, so it is not evidence of an entirely unseen dataset distribution.

TUM Dynamics is mixed: LONG3R’s ATE 5.40cm improves on Spann3R’s 5.66cm and CUT3R’s 6.25cm, while its relative translation/rotation errors 2.36cm/0.60° trail Spann3R’s 2.13cm/0.59°. Static-only training can still produce competitive results on these sequences, but does not establish robust handling of large object motion.

4.4 Ablation and analysis

The ablations separately compare memory gating, interleaved decoding and spatial-memory designs. Their configurations differ where explicitly noted, so their numerical results should not all be treated as the main model’s identical training run.

Attention-based Memory Gating.

Gating removes about 27% of stored tokens from the refinement attention on 7Scenes and raises throughput from 18.0 to 21.4 FPS, an 18.9% increase, rounded to 20% in the paper. Accuracy changes are mixed: 7Scenes mean accuracy slightly worsens from 2.53 to 2.57cm while completion improves from 2.12 to 2.08cm. Table 4’s NRGBD median accuracy and mean completion differ from Table 1’s main-model values; the source does not explain that discrepancy.

Original figure
Figure 7: Dual-Source Decoder Comparison. This figure compares the interleaved and concatenated architectures of the Dual-Source Refined Decoder.

Dual-Source Refined Decoder.

The decoder comparison uses 24-frame fine-tuning because concatenation exceeds the stated memory constraints at 32 frames. At 200 evaluation frames, interleaving improves mean accuracy from 29.52 to 13.34cm and completion from 8.88 to 3.15cm. Figure 7 retains both architectures. These results support the interleaved design under a matched 24-frame comparison, not the same numerical setting as the main 32-frame model.

3D Spatio-Temporal Memory.

Without long-term spatial memory, Replica-200 mean accuracy degrades to 65.75cm, versus 11.93cm for the full design. Replacing it with Spann3R memory under the same network gives 12.41cm, a much smaller difference. Thus the major demonstrated effect is retaining long-term context; adaptive spatial pruning provides an additional gain in the reported comparison.

Method Replica100 Replica200
Acc↓\downarrow Comp↓\downarrow Acc↓\downarrow Comp↓\downarrow
Mean Med. Mean Med. Mean Med. Mean Med.
Concat. 14.83 10.26 4.59 1.81 29.52 21.04 8.88 4.05
Interleaved. 12.06 7.67 3.68 1.23 13.34 8.41 3.15 0.94
Table 5: Refined Decoder ablation study on Replica [54].
Method 7Scenes Replica200
Acc↓\downarrow Comp↓\downarrow Acc↓\downarrow Comp↓\downarrow
Mean Med. Mean Med. Mean Med. Mean Med.
w/o 3D Spa. Mem. 5.76 2.96 3.30 1.22 65.75 47.63 13.24 3.49
w/ Spann3R Mem. 2.64 1.16 2.10 0.74 12.41 7.87 3.07 0.88
LONG3R (ours) 2.57 1.14 2.08 0.73 11.93 7.42 2.74 0.87
Table 6: Memory frameworks ablation study on 7Scenes [52] and Replica [54]. w/o 3D Spa. Mem.: without 3D spatial memory, only with temporal memory.

5 Conclusion

LONG3R improves the efficiency and consistency of recurrent pointmap reconstruction through selective memory access, alternating attention and spatial pruning. It remains anchored to the first frame and can blur when viewpoints diverge substantially. Large object motions are difficult because training is static. The experiments do not establish unlimited operation, drift-free global mapping, broad outdoor transfer, or performance at higher image resolutions; those remain beyond the measured scope.

Acknowledgements

The authors acknowledge support from China’s National Key R&D Program (2022ZD0161700) and Tsinghua University’s Initiative Scientific Research Program.

Concept demonstration

Open image ↗