Literature
Display
← All papers

End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization

Guangming Wang · Yu Zheng · Yuxuan Wu · Yanfeng Guo · Zhe Liu · Yixiang Zhu · Wolfram Burgard · Hesheng Wang

· IEEE Transactions on Robotics ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

The complete article and network-hyperparameter appendix are condensed below. All numbered figures, scientific tables, displayed equations, and original author portraits are retained.

[Uncaptioned image] Guangming Wang received the B.S. degree from Department of Automation from Central South University, Changsha, China, in 2018, and Ph.D. degree in Control Science and Engineering from Shanghai Jiao Tong University, Shanghai, China. He visited Department of Computer Science of ETH Zurich from 2022 to 2023. He is currently a Research Associate with the Department of Engineering, University of Cambridge, UK. His current research interests include robot perception, localization, and mapping.
[Uncaptioned image] Yu Zheng received the B.Eng degree in Department of Artificial Intelligence, Shanghai Jiao Tong University, China, in 2023. He is currently pursuing the Ph.D. degree in Control Science and Engineering with Shanghai Jiao Tong University, China. His current research interests include SLAM and computer vision.
[Uncaptioned image] Yuxuan Wu received the B.Eng degree in School of Astronautics, Beihang University. He is currently pursuing the Ph.D. degree in Control Science and Engineering with Shanghai Jiao Tong University, China. Her latest research interests include SLAM and computer vision.
[Uncaptioned image] Yanfeng Guo received his B.Eng degree at Department of Automation from Shanghai Jiao Tong University, Shanghai, China, in 2022. He is currently pursuing the M.S degree in Electrical and Computer Engineering at University of California, Los Angeles, the United States. His current research interests include computer vision and autonomous driving system.
[Uncaptioned image] Zhe Liu received his B.S. degree in Automation from Tianjin University, Tianjin, China, in 2010, and Ph.D. degree in Control Technology and Control Engineering from Shanghai Jiao Tong University, Shanghai, China, in 2016. From 2017 to 2020, he was a Post-Doctoral Fellow with the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong. From 2020 to 2022, he was a Research Associate of Department of Computer Science and Technology, University of Cambridge, United Kingdom. He is currently a tenure-track associate professor of Institute of Artificial Intelligence, Shanghai Jiao Tong University. His research interests include scheduling and optimization of large-scale robotic systems, resilient navigation of industrial self-driving systems, localization of self-driving systems, coordination of multi-robot formations, and robot control.
[Uncaptioned image] Yixiang Zhu received his B.Eng degree at Department of Electrical Engineering from Shanghai Jiao Tong University, Shanghai, China, in 2023. He is currently pursuing the M.S degree in Computer Control and Automation at Nanyang Technological University, Singapore. His current research interests include SLAM and computer vision.
[Uncaptioned image] Wolfram Burgard is Vice President for Automated Driving Technology at the Toyota Research Institute in Los Altos, USA. He is on leave from a Professorship for Computer Science at the University of Freiburg, Germany where he heads the Laboratory for Autonomous Intelligent Systems. He received his Ph.D. degree in computer science from the University of Bonn in 1991. His areas of interest lie in robotics and artificial intelligence. In the past, Wolfram Burgard and his group developed several innovative probabilistic techniques for robot navigation and control. They cover different aspects including localization, mapping, path planning, and exploration. For his work, Wolfram Burgard received several best paper awards from outstanding national and international conferences. In 2009, Wolfram Burgard received the Gottfried Wilhelm Leibniz Prize, the most prestigious German research award. In 2010 he received the Advanced Grant of the European Research Council. Wolfram Burgard is the spokesperson of the Cluster of Excellence BrainLinks-BrainTools, President of the IEEE Robotics and Automation Society, and fellow of the AAAI, EurAi and IEEE.
[Uncaptioned image] Hesheng Wang received the B.Eng. degree in electrical engineering from the Harbin Institute of Technology, Harbin, China, in 2002, and the M.Phil. and Ph.D. degrees in automation and computer-aided engineering from The Chinese University of Hong Kong, Hong Kong, in 2004 and 2007, respectively. He is currently a Professor with the Department of Automation, Shanghai Jiao Tong University, Shanghai, China. His current research interests include visual servoing, service robot, computer vision, and autonomous driving. Dr. Wang is an Associate Editor of IEEE Transactions on Automation Science and Engineering, IEEE Robotics and Automation Letters, Assembly Automation and the International Journal of Humanoid Robotics, a Technical Editor of the IEEE/ASME Transactions on Mechatronics, an Editor of Conference Editorial Board (CEB) of IEEE Robotics and Automation Society. He served as an Associate Editor of the IEEE Transactions on Robotics from 2015 to 2019. He was the General Chair of IEEE ROBIO 2022 and IEEE RCAR 2016, and the Program Chair of the IEEE ROBIO 2014 and IEEE/ASME AIM 2019. He will be the General Chair of IEEE/RSJ IROS 2025.

At a glance

I2PNet learns to register a camera image directly with a raw LiDAR point cloud. Soft cross-modal correspondences, learned outlier weights, and iterative pose refinement are optimized together under pose supervision. Experiments cover large and small initial misalignments, transfer across driving and ground-robot datasets, and simulated camera–LiDAR miscalibration. The method needs known camera intrinsics and a relevant local point-cloud map; it does not solve city-scale place recognition.

I Introduction

A robot can use an economical monocular camera to localize against a map built previously with LiDAR. The difficult step is matching texture-rich image pixels to geometry-rich three-dimensional points. Earlier pipelines separately learn correspondences and solve pose, or turn points into a depth image that loses useful geometry under large initial misalignment. I2PNet instead retains raw point structure and connects feature extraction, association, outlier handling, and pose regression through a common supervised objective. Its normalized image-plane representation accommodates different supplied camera calibrations.

Refer to caption
Figure 1: The pipeline of monocular camera localization in LiDAR maps with image-to-point cloud registration. Image-to-point cloud registration associates the image and point cloud and estimates the spatial transformation between them. With the estimated transformation, the pose of the monocular camera in the LiDAR map is obtained and the robot is localized.

II Related Work

Prior work separates cross-modal localization from the related task of correcting a camera–LiDAR sensor transform.

II-A Image-to-Point Cloud Registration for Robot Localization

Feature matching followed by a perspective-n-point or inverse-projection solver can localize over large displacements, but its stages are not jointly optimized. Depth-image methods such as CMRNet allow end-to-end refinement yet lose projected points when the initial alignment is poor. I2PNet combines raw-point feature extraction with differentiable association and direct pose prediction, avoiding a separate iterative geometric solver.

II-B Camera-LiDAR Online Calibration

Calibration methods match contours or intensities, or learn alignment between camera and depth images. RegNet, CalibNet, and RGGNet emphasize different feature and loss designs. I2PNet applies the same raw-point/image registration architecture to both localization and synthetic extrinsic-calibration errors.

Refer to caption
Figure 2: The outline of I2PNet. I2PNet takes the RGB image and the raw LiDAR point cloud as input. Through the feature extraction pyramid, coarse registration, and fine registration, the network finally predicts relative pose between the image and point cloud. The detailed structure of coarse prior association is shown in Fig. 3.

III End-to-end 2D-3D Registration

III-A Network Architecture

Image and point-cloud pyramids provide multiscale features. Coarse association produces a pointwise cost volume; spatial context and learned outlier weights aggregate it into a rotation quaternion and translation. Fine registration warps points by the current estimate, recomputes more local correspondences, combines them with upsampled coarse information, and predicts a residual transform. Repeating this refinement trades inference time for accuracy.

Refer to caption
Figure 3: The detailed structure of the coarse prior association in Fig. 2.

III-B Feature Extraction

III-B 1 Image Feature Extraction

Three image-pyramid levels each contain five convolutional blocks with normalization, leaky rectification, and pooling. The network retains feature vectors and the corresponding pixel coordinates.

III-B 2 Point Cloud Feature Extraction

PointNet++-style aggregation builds four point-feature levels. For single rotating-LiDAR scans, spherical coordinates permit stride sampling and a restricted two-dimensional search window; three-dimensional nearest-neighbor selection then occurs within that window. This replaces expensive full-cloud sampling and search while preserving raw three-dimensional features. A distance threshold rejects remote neighbors. Accumulated maps used in the small-range experiment instead require ordinary PointNet++ sampling.

Equations 1–2
(usvs)=(12​⌊1−arctan⁡(y,x)⋅π−1⌋⋅W⌊1−(arcsin⁡(z/r)+fd​o​w​n)⋅f−1⌋⋅H),\left(\begin{matrix}u_{s}\\ v_{s}\end{matrix}\right)=\left(\begin{matrix}\frac{1}{2}\lfloor 1-\arctan(y,x)\cdot\pi^{-1}\rfloor\cdot W\\ \lfloor 1-(\arcsin(z/r)+f_{down})\cdot f^{-1}\rfloor\cdot H\end{matrix}\right), (1)
fil=M​a​x​P​o​o​l​(M​L​P​({fi,kl−1}k=1Kl)),f_{i}^{l}=MaxPool(MLP(\{f_{i,k}^{l-1}\}_{k=1}^{K^{l}})), (2)
Refer to caption
Figure 4: The process diagram of the 2D-3D cost volume module. The Implicit Correspondence (IC) generation module uses an all-to-all or KNN-based point-pixel mixture to match the points and pixels on the normalized plane of the pinhole camera model. Then, the similarities of each point-pixel pair are calculated and aggregated to generate the point-wise IC features. Based on the IC features, the Patch Spatial Transformation (PST) embedding module aggregates the IC features of the neighbors of each point to estimate the patch spatial transformation information and embeds it in the PST embedding features. The module finally outputs the PST embedding features as the 2D-3D cost volumes.
Refer to caption
Figure 5: The detailed pipeline of the 2D-3D cost volume module. Given the features of the image and point cloud, as well as their position information, the module estimates the PST embedding features as the 2D-3D cost volumes. The detailed structures of the point-pixel mixtures are shown in Fig. 6 and Fig. 7.

III-C 2D-3D Cost Volume Module

III-C 1 Overview

Points project to a normalized pinhole plane, and image pixels map to that plane using the inverse camera-intrinsic matrix. Learned soft matching yields implicit correspondence features; aggregation over neighboring points adds patch-transform information. Intrinsic-independent coordinates therefore do not mean that camera intrinsics are unknown or unnecessary.

III-C 2 Detailed Pipeline

Large initial errors use all image features as coarse matching candidates, then nearest pixels during refinement; small-error tasks use local candidates at both stages. Channelwise normalized point/image feature products measure similarity. The all-to-all version also includes an inverse-match feature to reduce ambiguous associations. Position-aware attention forms a soft weighted correspondence feature for each point. A second neighborhood aggregation incorporates absolute positions, offsets, and distances, producing a patch spatial transformation embedding used as the cost volume. These learned features encode evidence for alignment rather than explicit, guaranteed-correct point matches.

Equations 3–4
[xi¯,yi¯,1]T=1zi​[xi,yi,zi]T,{\left[{\begin{array}[]{*{20}{c}}{\overline{x_{i}}},{\overline{y_{i}}},{1}\end{array}}\right]}^{T}=\frac{1}{z_{i}}{\left[{\begin{array}[]{*{20}{c}}{{x}_{i}},{{y}_{i}},{z_{i}}\end{array}}\right]}^{T}, (3)
[u¯,v¯,1]T=Kc−1​[u,v,1]T,{\left[{\begin{array}[]{*{20}{c}}{\overline{u}},{\overline{v}},{1}\end{array}}\right]}^{T}={K_{c}}^{-1}{\left[{\begin{array}[]{*{20}{c}}{{u}},{{v}},{1}\end{array}}\right]}^{T}, (4)
Refer to caption
Figure 6: The all-to-all point-pixel mixture for the coarse registration. The all-to-all mixture treats all the pixels as the correspondence candidates of the point. In addition, the inverse similarity is adopted for robust correspondence generation.
Equations 5–8
sik=fi−μ⁡(fi)σ⁡(fi)⊙gik−μ⁡(gik)σ⁡(gik),{s}_{i}^{k}=\frac{{f}_{i}-\mu\left(f_{i}\right)}{\sigma\left(f_{i}\right)}\odot\frac{{g}_{i}^{k}-\mu\left(g_{i}^{k}\right)}{\sigma\left(g_{i}^{k}\right)}, (5)
h^ij=M​a​x​P​o​o​l​({fij,l⊙gij}l=1N).\hat{h}_{i}^{j}=MaxPool(\{f_{i}^{j,l}\odot g_{i}^{j}\}_{l=1}^{N}). (6)
hik\displaystyle{h}_{i}^{k} =M​L​P​(sik⊕h^ik⊕o¯ik⊕pi),\displaystyle=MLP({s}_{i}^{k}\oplus\hat{h}_{i}^{k}\oplus\overline{{o}}_{i}^{k}\oplus{p}_{i}), (7)
hik\displaystyle{h}_{i}^{k} =M​L​P​(sik⊕o¯ik⊕pi),\displaystyle=MLP({s}_{i}^{k}\oplus\overline{{o}}_{i}^{k}\oplus{p}_{i}), (8)
Original figure
Figure 7: The KNN-based point-pixel mixture. This type of point-pixel mixture treats the KK nearest pixel neighbors as the pixel correspondence candidates.
Equations 9–14
wik=S​o​f​t​m​a​x​(M​L​P​(hik⊕rik)),{w}_{i}^{k}=Softmax(MLP({h}_{i}^{k}\oplus r_{i}^{k})), (9)
i​ci=∑k=1K1(hik⊙wik).{ic}_{i}=\sum\limits_{k=1}^{K_{1}}({h}_{i}^{k}\odot{w}_{i}^{k}). (10)
uim=pi⊕pim⊕(pim−pi)⊕∥pim−pi∥,{u}_{i}^{m}={p}_{i}\oplus{p}_{i}^{m}\oplus\left({p}_{i}^{m}-{p}_{i}\right)\oplus\lVert{p}_{i}^{m}-{p}_{i}\rVert, (11)
bim=F​C​(uim).{b}_{i}^{m}=FC({u}_{i}^{m}). (12)
wim=S​o​f​t​m​a​x​(M​L​P​(fi⊕bim⊕i​cim)),{w}_{i}^{m}=Softmax(MLP({f}_{i}\oplus{b}_{i}^{m}\oplus{ic}_{i}^{m})), (13)
ei=∑m=1K2(i​cim⊙wim),{e}_{i}=\sum\limits_{m=1}^{K_{2}}({ic}_{i}^{m}\odot{w}_{i}^{m}), (14)

III-D Coarse Registration

III-D 1 Context Gathering Module

A further point-neighborhood aggregation pools coarse cost-volume context using the fourth point-pyramid level. This lets surrounding associations influence each point’s transformation evidence.

Equation 15
en​e​w,ic=M​a​x​P​o​o​l​(M​L​P​({ei,kc}k=1K4)),e^{c}_{new,i}=MaxPool(MLP(\{e^{c}_{i,k}\}^{K^{4}}_{k=1})), (15)

III-D 2 Outlier Mask Prediction Module

A shared network combines point appearance/geometry features with contextual cost volumes to predict outlier scores. Points outside the image view, or with unreliable correspondences, should receive less influence on pose.

Equation 16
mic=M​L​P​(en​e​w,ic⊕fi4).m_{i}^{c}=MLP({e}^{c}_{new,i}\oplus f_{i}^{4}). (16)

III-D 3 Pose Regression

Softmax-normalized outlier scores weight the cost-volume sum. Fully connected layers and training-time dropout decode this global feature into translation and a normalized quaternion. Learned weighting replaces a separate consensus-based outlier solver; it does not provide a geometric correctness certificate.

Equations 17–18
qc=F​C​(∑i=1N4(en​e​w,ic⊙m​wic))|F​C​(∑i=1N4(en​e​w,ic⊙m​wic))|,q_{c}=\frac{FC(\sum\limits_{i=1}^{N^{4}}({e}_{new,i}^{c}\odot{mw}_{i}^{c}))}{\lvert FC(\sum\limits_{i=1}^{N^{4}}({e}_{new,i}^{c}\odot{mw}_{i}^{c}))\rvert}, (17)
tc=F​C​(∑i=1N4(en​e​w,ic⊙m​wic)),t_{c}=FC(\sum\limits_{i=1}^{N^{4}}({e}_{new,i}^{c}\odot{mw}_{i}^{c})), (18)

III-D 4 Upsampling Layer

Two separate neighborhood-based upsampling modules transfer coarse cost volumes and outlier masks to the finer point set. Each combines relative coordinates, pooled coarse features, and fine point features.

Equations 19–20
u​ei=F​C​(fi3CLOSE\displaystyle ue_{i}=FC(f_{i}^{3} ⊕MaxPool(MLP({en​e​w,i,kc\displaystyle\oplus MaxPool(MLP(\{{e}_{new,i,k}^{c}
⊕(pi,k4−pi3)}k=1K))),\displaystyle\oplus(p_{i,k}^{4}-p_{i}^{3})\}_{k=1}^{K}))), (19)
u​mi=F​C​(fi3CLOSE\displaystyle um_{i}=FC(f_{i}^{3} ⊕MaxPool(MLP({mi,kc\displaystyle\oplus MaxPool(MLP(\{{m}_{i,k}^{c}
⊕(pi,k4−pi3)}k=1K))).\displaystyle\oplus(p_{i,k}^{4}-p_{i}^{3})\}_{k=1}^{K}))). (20)

III-E Fine Registration

III-E 1 Pose Warping

The current quaternion and translation transform the fine-level point coordinates. Nearest-pixel association on these warped coordinates estimates the remaining image–point misalignment.

Equation 21
(0,Pw​a​r​p​e​d3)=qfi​(0,P3)​(qfi)−1+(0,tfi),(0,{P}^{3}_{warped})={q}_{f}^{i}(0,{P}^{3})({q}_{f}^{i})^{-1}+(0,{t}_{f}^{i}), (21)

III-E 2 Optimization Module

Shared networks fuse fresh fine cost volumes with upsampled coarse volumes and point features. A parallel fusion predicts refined outlier masks from the optimized cost volumes, earlier masks, and point features.

Equations 22–23
o​ei=M​L​P​(eif⊕u​ei⊕fi3).oe_{i}=MLP(e^{f}_{i}\oplus ue_{i}\oplus f^{3}_{i}). (22)
mif=M​L​P​(o​ei⊕u​mi⊕fi3).m^{f}_{i}=MLP(oe_{i}\oplus{um}_{i}\oplus f^{3}_{i}). (23)

III-E 3 Pose Refinement

The weighted pose decoder estimates a residual rotation and translation. Quaternion composition and the corresponding translation update refine the current transform; the equations retain the composition order.

Equations 24–26
Δ​q=F​C​(∑i=1N3(o​ei⊙m​wif))|F​C​(∑i=1N3(o​ei⊙m​wif))|,\Delta q=\frac{FC(\sum\limits_{i=1}^{N^{3}}({oe}_{i}\odot{mw}_{i}^{f}))}{\lvert FC(\sum\limits_{i=1}^{N^{3}}({oe}_{i}\odot{mw}^{f}_{i}))\rvert}, (24)
Δ​t=F​C​(∑i=1N3(o​ei⊙m​wif)),\Delta t=FC(\sum\limits_{i=1}^{N^{3}}({oe}_{i}\odot{mw}_{i}^{f})), (25)
qfi+1=Δ​q⋅qfi,(0,tfi+1)=Δ​q​(0,tfi)​Δ​q−1+(0,Δ​t).{q}_{f}^{i+1}=\Delta{q}\cdot{q}_{f}^{i},(0,{t}_{f}^{i+1})=\Delta{q}(0,{t}_{f}^{i})\Delta{q}^{-1}+(0,\Delta{t}). (26)

III-F Loss Function

Supervised quaternion and translation errors train both coarse and fine predictions. Learned scale parameters balance their different units. Training uses one refinement iteration for efficiency, with a weighted sum of the two registration losses.

Equations 27–28
ℒ⁡(q,t,qg​t,tg​t)\displaystyle\mathcal{L}({q},{t},{q}_{gt},{t}_{gt}) =∥qg​t−q∥2⋅e−sq+sq\displaystyle={\lVert{q}_{gt}-{q}\rVert}_{2}\cdot{e}^{-{s}_{q}}+{s}_{q} (27)
+∥tg​t−t∥1⋅e−st+st,\displaystyle+{\lVert t_{gt}-{t}\rVert}_{1}\cdot{e}^{-{s}_{t}}+{s}_{t},
ℒ=αc⋅ℒ⁡(qc,tc,qg​t,tg​t)+αf⋅ℒ⁡(qf,tf,qg​t,tg​t),\mathcal{L}={\alpha_{c}}\cdot\mathcal{L}({q}_{c},{t}_{c},{q}_{gt},{t}_{gt})+{\alpha_{f}}\cdot\mathcal{L}({q}_{f},{t}_{f},{q}_{gt},{t}_{gt}), (28)

IV Experiments

Evaluation separates two different localization protocols and examines transfer, component ablations, runtime, and refinement depth.

IV-A Implementation Details

Experiments use an RTX 3090, Adam, batch size eight, initial learning rate 0.001 with 1% decay per epoch, and 0.5 pose-decoder dropout. Point features are extracted in the LiDAR frame. Comparisons use one fine iteration. Large-range perturbations cover full yaw and up to 10 m of ground-plane translation; small-range perturbations independently vary each rotation axis within ±10° and each translation axis within ±2 m. These settings are not equivalent global-localization problems.

Table I: Large-Range Localization Error on the KITTI Odometry and NuScenes Datasets
Method KITTI Odometry nuScenes
RRE (∘) ↓\downarrow RTE (m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m) ↓\downarrow
Grid. Cls. + EPnP [19] 6.48 ±\pm 1.66 1.07 ±\pm 0.61 7.20 ±\pm 1.65 2.35 ±\pm 1.12
DeepI2P (3D) [19] 6.26 ±\pm 2.29 1.27 ±\pm 0.80 7.18 ±\pm 1.92 2.00 ±\pm 1.08
DeepI2P (2D) [19] 4.27 ±\pm 2.29 1.46 ±\pm 0.96 3.54 ±\pm 2.51 2.19 ±\pm 1.16
CorrI2P [18] 2.07 ±\pm 1.64 0.74 ±\pm 0.65 2.65 ±\pm 1.93 1.83 ±\pm 1.06
Ours (I2PNet) 0.83 ±\pm 1.04 0.21 ±\pm 0.29 1.13 ±\pm 1.08 0.75 ±\pm 0.59
Refer to caption
Figure 8: The registration recall curves of the methods with different RTE and RRE thresholds on KITTI Odometry and nuScenes datasets. The y-axis (recall) of the recall curve presents the success rate that RREs or RTEs are less than the threshold in the x-axis.
Refer to caption
Figure 9: The histograms of RTE and RRE on the KITTI Odometry and nuScenes datasets. The x-axis is the RTE (m) or RRE (∘), and the y-axis is the percentage falling into the corresponding bin. The bin size of RRE and RTE is 0.5∘0.5^{\circ} and 0.1​m0.1m respectively.
Refer to caption
Figure 10: Visualization of large-range localization results on KITTI Odometry and nuScenes datasets. This figure presents the large-range localization results on the KITTI Odometry and nuScenes datasets through the visualization of image-to-point cloud registration. The color bar shows the depth of each LiDAR point.
Refer to caption
Figure 11: Visualization of coarse and fine registration results. The coarse and fine predictions are obtained from the coarse and fine registration respectively. The zoom-in views of the areas marked by the bounding boxes in each picture are to better present the difference among the coarse prediction, fine prediction, and ground truth. The color bar has the same meaning as Fig. 10.

IV-B Large-Range Localization

IV-B 1 Dataset and Data Pre-processing

KITTI sequences 0–8 train the large-range model and 9–10 test it; nuScenes uses 850/150 traversals. A same-frame KITTI scan, or a pose-aligned nearby nuScenes scan, is transformed synthetically to create each localization example. Cropped images resize to 160×512 or 160×640; all scan points are supplied. Initial features contain intensity plus estimated normals for KITTI, or intensity plus zero placeholders for nuScenes. Thus this protocol mainly tests registration of a perturbed scan, with known pairing to the image.

IV-B 2 Experimental Results and Visualization

Mean errors are reported only for samples below both 10° rotation error and 5 m translation error; recall curves expose failures omitted by those conditional means. I2PNet reports 0.83°/0.21 m on KITTI and 1.13°/0.75 m on nuScenes, versus CorrI2P’s 2.07°/0.74 m and 2.65°/1.83 m. Qualitative overlays and coarse/fine comparisons show improved alignment. These numbers should be read together with the recall curves and the synthetic perturbation protocol.

Equation 29
R​R​E=∑i=13|θi|,R​T​E=∥tp​r​e​d−tg​t∥2,RRE=\sum\limits_{i=1}^{3}\lvert\theta_{i}\rvert,RTE=\lVert t_{pred}-t_{gt}\rVert_{2}, (29)

IV-B 3 Validation on Non-Car-Like Platform

M2DGR evaluates a ground robot across outdoor, hall, elevator, and room scenes. When trained and tested on that dataset, I2PNet reports 0.89–1.70° rotation error and 0.24–0.36 m translation error across the four scene types, outperforming the listed baselines.

Table II: Large-Range Localization Evaluated on M2DGR Dataset. All Models Are Trained and Tested on M2DGR Dataset to Demonstrate the Performance on a Non-Car-Like Platform, a Ground Robot Platform
Method Outdoor Hall Lift Room
RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow
Grid. Cls. + EPnP [19] 6.49 ±\pm 2.59 2.50 ±\pm 1.22 6.71 ±\pm 2.29 2.61 ±\pm 1.33 5.33 ±\pm 2.90 3.12 ±\pm 1.09 4.34 ±\pm 1.79 2.59 ±\pm 0.75
DeepI2P (3D) [19] 5.96 ±\pm 0.84 1.10 ±\pm 1.01 6.92 ±\pm 2.00 1.62 ±\pm 1.23 6.20 ±\pm 2.90 1.51 ±\pm 0.62 5.56 ±\pm 1.62 0.42 ±\pm 0.09
DeepI2P (2D) [19] 5.57 ±\pm 2.54 2.18 ±\pm 1.16 4.70 ±\pm 2.83 2.20 ±\pm 1.20 5.06 ±\pm 2.70 1.82 ±\pm 0.99 5.42 ±\pm 2.23 1.82 ±\pm 0.52
CorrI2P [18] 5.12 ±\pm 2.30 2.17 ±\pm 1.05 5.10 ±\pm 2.40 1.53 ±\pm 0.93 5.81 ±\pm 2.27 2.17 ±\pm 1.05 6.98 ±\pm 1.84 1.18 ±\pm 0.88
Ours (I2PNet) 1.13 ±\pm 0.84 0.29 ±\pm 0.18 0.89 ±\pm 0.80 0.24 ±\pm 0.15 1.51 ±\pm 1.36 0.36 ±\pm 0.24 1.70 ±\pm 1.36 0.31 ±\pm 0.22
Table III: Generalization Experiments from KITTI Odometry Dataset to NuScenes and M2DGR Datasets. All Models Are Trained on KITTI Odometry Dataset and Tested on All the Other Datasets
Method nuScenes M2DGR (Outdoor) M2DGR (Hall) M2DGR (Lift) M2DGR (Room)
RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow RRE (∘) ↓\downarrow RTE (m)(m) ↓\downarrow
Grid. Cls. + EPnP [19] 4.94 ±\pm 2.84 3.22 ±\pm 1.20 6.04 ±\pm 3.15 2.21 ±\pm 1.23 4.46 ±\pm 2.83 3.53 ±\pm 1.25 4.43 ±\pm 2.65 3.43 ±\pm 1.11 failed failed
DeepI2P (3D) [19] failed failed failed failed failed failed failed failed failed failed
DeepI2P (2D) [19] failed failed failed failed failed failed failed failed failed failed
CorrI2P [18] failed failed failed failed failed failed failed failed failed failed
Ours (I2PNet) 2.76 ±\pm 2.13 2.39 ±\pm 1.14 2.71 ±\pm 2.29 1.70 ±\pm 1.00 3.91 ±\pm 2.58 1.73 ±\pm 1.04 3.78 ±\pm 2.72 1.89 ±\pm 1.11 4.91 ±\pm 2.66 3.40 ±\pm 1.06

IV-B 4 Generalization Analysis

A KITTI-trained model transfers better than the compared systems but loses substantial accuracy: nuScenes reaches 2.76°/2.39 m, and the M2DGR room reaches 4.91°/3.40 m. “Failed” means no test examples satisfy both error cutoffs. The experiment demonstrates relative cross-platform robustness within these tested datasets, not unchanged performance after domain shift.

IV-C Small-Range Localization

Small-range evaluation compares methods initialized near the correct local map region.

IV-C 1 Dataset and Data Pre-processing

KITTI sequences 03, 05–09 train the model and sequence 00 tests it. A LiDAR SLAM system builds an accumulated map, which is cropped around an initial pose and perturbed within the stated six-axis bounds. Images resize to 384×1280 and 8,192 map points are sampled. Because accumulated maps lack single-scan spherical structure, this setting uses ordinary PointNet++ sampling and neighborhood search.

Table IV: Small-Range Localization Error on KITTI Odometry Dataset. The Results in Parentheses Are Reproduced by Ourselves. The Reproduced Results of HyperMap Are Not Provided as HyperMap Does Not Release the Codes
Method Rot. (∘) ↓\downarrow Transl. (m) ↓\downarrow Median Rot. (∘) ↓\downarrow Median Transl. (m) ↓\downarrow
CASELITZ [14] 1.65 ±\pm 0.91 0.30 ±\pm 0.11 — —
CMRNet [26] — (1.98 ±\pm 1.30) — (0.62 ±\pm 0.43) 1.39 (1.68) 0.51 (0.51)
CMRNet++ [21] — (1.88 ±\pm 1.43) — (0.70 ±\pm 0.48) 1.46 (1.52) 0.55 (0.58)
HyperMap [27] — — 1.42 0.48
I2D-Loc [28] — (1.07 ±\pm 1.17) — (0.34 ±\pm 0.38) 0.70 (0.77) 0.18 (0.21)
Ours (I2PNet) 0.74 ±\pm 0.40 0.08 ±\pm 0.06 0.67 0.07
Table V: Generalization Experiments on KITTI Odometry, nuScenes, Argoverse, Waymo, and Lyft5 Datasets on Small-Range Localization Task. The Results of CMRNet, CMRNet++ and I2DLoc Are Reproduced by Ourselves for Extensive Generalization Testing
Training Dataset Method KITTI nuScenes Argoverse Waymo Lyft5
Rot. (∘) ↓\downarrow Transl. (mm) ↓\downarrow Rot. (∘) ↓\downarrow Transl. (mm) ↓\downarrow Rot. (∘) ↓\downarrow Transl. (mm) ↓\downarrow Rot. (∘) ↓\downarrow Transl. (mm) ↓\downarrow Rot. (∘) ↓\downarrow Transl. (mm) ↓\downarrow
KITTI CMRNet [26] 1.98±\pm1.30 0.62±\pm0.43 7.12±\pm3.20 1.67±\pm0.64 9.49±\pm3.09 1.65±\pm0.77 11.24±\pm3.30 2.01±\pm0.85 7.97±\pm2.96 1.83±\pm0.61
CMRNet++ [21] 1.88±\pm1.43 0.70±\pm0.48 7.70±\pm4.40 1.61±\pm0.78 3.92±\pm2.44 1.33±\pm0.69 8.84±\pm3.91 1.88±\pm0.83 7.96±\pm3.95 1.70±\pm0.77
I2DLoc [28] 1.07±\pm1.17 0.34±\pm0.38 6.60±\pm5.33 1.46±\pm0.81 3.09±\pm2.45 1.12±\pm0.70 7.17±\pm5.35 1.87±\pm0.82 7.95±\pm4.99 1.52±\pm0.73
Ours (I2PNet) 0.74±\pm0.40 0.08±\pm0.06 2.88±\pm0.93 1.25±\pm0.53 0.78±\pm0.15 0.68±\pm0.14 0.90±\pm0.55 0.62±\pm0.28 0.76±\pm0.10 0.51±\pm0.05
nuScenes CMRNet [26] 5.69±\pm2.93 1.74±\pm0.84 2.77±\pm1.87 1.02±\pm0.68 4.24±\pm2.25 1.98±\pm0.95 5.26±\pm3.00 2.10±\pm0.77 5.10±\pm2.04 1.70±\pm0.64
CMRNet++ [21] 5.56±\pm3.43 1.49±\pm0.76 3.88±\pm2.70 1.21±\pm0.67 4.86±\pm3.18 1.62±\pm0.81 6.22±\pm4.09 2.01±\pm0.95 5.69±\pm3.38 1.52±\pm0.70
I2DLoc [28] 4.46±\pm3.28 1.25±\pm0.72 2.70±\pm2.50 0.83±\pm0.60 3.89±\pm3.31 1.33±\pm0.74 6.36±\pm3.97 1.97±\pm0.76 6.16±\pm3.31 1.66±\pm0.72
Ours (I2PNet) 1.59±\pm0.27 1.14±\pm0.27 0.57±\pm0.75 0.58±\pm0.42 1.44±\pm0.40 0.72±\pm0.19 1.25±\pm0.67 0.72±\pm0.31 1.50±\pm0.39 0.69±\pm0.13

IV-C 2 Experiment Result and Visualization

Across ten test repetitions, I2PNet achieves mean rotation/translation errors of 0.74°/0.08 m and medians of 0.67°/0.07 m. The table distinguishes cited baseline results from the authors’ reproductions. I2D-Loc’s cited medians are 0.70°/0.18 m. Here rotation is the angle of the relative rotation matrix, unlike the summed absolute Euler-angle error used in large-range evaluation.

Equation 30
R​o​t.=arccos⁡t​r​(Re)−12,T​r​a​n​s​l.=∥te∥2,Rot.=\arccos\frac{tr\left(R_{e}\right)-1}{2},Transl.=\lVert t_{e}\rVert_{2}, (30)

IV-C 3 Generalization Comparison

Models trained on KITTI or nuScenes are tested on those datasets plus Argoverse, Waymo, and Lyft5. The compared CMRNet, CMRNet++, and I2DLoc results are reproduced under matched protocols. I2PNet leads the listed measures, though transfer errors remain much larger than same-dataset errors; for example, KITTI-to-nuScenes translation error is 1.25 m versus 0.08 m on KITTI.

IV-D Ablation Studies

Ablations use KITTI large-range localization.

IV-D 1 Effect of the Input Point Number

Increasing input points from 8,192 to 24,576 reduces error from 2.16°/0.57 m to 1.61°/0.36 m. Using all points with the efficient spherical-query implementation reaches 0.83°/0.21 m. This comparison changes both point count and sampling/search implementation.

Table VI: Ablation Study on the Number of Input Points
Method RRE (∘) ↓\downarrow RTE (m) ↓\downarrow
Ours (w/ 81928192 input points) 2.16 ±\pm 1.90 0.57 ±\pm 0.37
Ours (w/ 1638416384 input points) 2.05 ±\pm 1.85 0.48 ±\pm 0.36
Ours (w/ 2457624576 input points) 1.61 ±\pm 1.42 0.36 ±\pm 0.29
Ours (w/ all input points) 0.83 ±\pm 1.04 0.21 ±\pm 0.29
Table VII: Ablation Study on the Effectiveness of the Proposed Modules
Method RRE (∘) ↓\downarrow RTE (m) ↓\downarrow
Ours (w/o 2D-3D Cost Volume Module) 2.54 ±\pm 1.94 3.29 ±\pm 1.19
Ours (w/o Intrinsic-Independent Space) 1.85 ±\pm 1.58 0.71 ±\pm 0.56
Ours (w/o Outlier Mask) 1.12 ±\pm 1.28 0.28 ±\pm 0.27
Ours (w/o Fine Registration) 1.59 ±\pm 1.41 0.60 ±\pm 0.60
Ours (w/o Pose Warping) 1.63 ±\pm 1.41 0.53 ±\pm 0.54
Ours (w/o PST Embedding) 1.01 ±\pm 1.13 0.29 ±\pm 0.33
Ours (I2PNet) 0.83 ±\pm 1.04 0.21 ±\pm 0.29
Table VIII: Efficiency Comparison
Method Network size (MB) Inference (s)
Grid. Cls. + EPnP [19] 100.75 0.051
DeepI2P (3D) [19] 100.12 16.588
DeepI2P (2D) [19] 100.12 9.388
CorrI2P [18] 141.07 2.984
Ours (I2PNet) (8192 w/o projection) 3.38 0.069
Ours (I2PNet) (16384 w/o projection) 3.38 0.143
Ours (I2PNet) (24576 w/o projection) 3.38 0.271
Ours (I2PNet) (all points w/ projection) 3.38 0.048
Table IX: Large-Range Localization Error with Different Number of Iterations in the Fine Registration
Iter Num RRE (∘) ↓\downarrow RTE (m) ↓\downarrow Recall (%) ↑\uparrow Inference Time (s) ↓\downarrow
1 0.83 ±\pm 1.04 0.21 ±\pm 0.29 98.67 0.048
2 0.79 ±\pm 0.92 0.20 ±\pm 0.25 99.14 0.056
3 0.78 ±\pm 0.86 0.20 ±\pm 0.27 99.32 0.063
4 0.77 ±\pm 0.83 0.20 ±\pm 0.27 99.28 0.069
5 0.78 ±\pm 0.83 0.20 ±\pm 0.29 99.32 0.076
6 0.81 ±\pm 0.89 0.21 ±\pm 0.31 99.43 0.083

IV-D 2 Effectiveness of the Proposed Modules

Replacing the cost volume with simple attentive fusion raises translation error to 3.29 m. Removing normalized coordinates gives 1.85°/0.71 m; omitting outlier weighting gives 1.12°/0.28 m; omitting fine refinement gives 1.59°/0.60 m. Pose warping and patch-transform embedding also help. These controlled removals support each module’s contribution in this protocol without establishing that every component is necessary in all domains.

Table X: Online Calibration Results on the KITTI Raw Dataset
Method T1 T2a T2b T3
MSEE ↓\downarrow MRR ↑\uparrow MSEE ↓\downarrow MRR ↑\uparrow MSEE ↓\downarrow MRR ↑\uparrow MSEE ↓\downarrow MRR ↑\uparrow
β\beta-RegNet [44] 0.0480 53.23% 0.0440 37.08% 0.0460 34.14% 0.0920 -1.89%
TAYLOR [62] - - - - - - 0.0100 -
CalibNet [46] - - 0.0220 - 0.0220 - - -
RGGNet [47] 0.0210 78.40% 0.0140 75.61% 0.0170 72.64% 0.0100 83.22%
Ours (I2PNet) 0.00096 99.04% 0.00084 98.61% 0.00115 97.81% 0.00154 97.21%
Refer to caption
Figure 12: Visualization of online calibration results. This figure presents the online calibration results on the four test sets. The pictures are cropped for better presentation and the areas that can reflect the calibration quality are marked by the bounding boxes. The color bar has the same meaning as Fig. 10.

IV-E Efficiency Evaluation

All-point I2PNet takes 0.048 s per prediction with a 3.38 MB network, compared with 2.984 s for CorrI2P and 0.051 s for the fastest grid-classification baseline. Baseline times include network inference on the RTX 3090 and pose post-processing on a Xeon Gold 6346; I2PNet has no separate pose solver. Its approximately 20 Hz figure is this measured registration runtime, excluding global map retrieval and a complete deployed robot stack.

IV-F Iterations of the Fine Registration

Four fine iterations improve conditional error to 0.77°/0.20 m at 0.069 s, versus 0.83°/0.21 m at 0.048 s for one. More iterations offer little further accuracy improvement and can slightly worsen it; recall reaches 99.43% at six versus 98.67% at one. The default favors speed.

V Extension to Camera-LiDAR Online Calibration

The same registration model is also evaluated as an extrinsic-calibration correction method.

V-A Dataset and Data Pre-processing

KITTI raw scans are transformed by synthetic calibration errors. Training combines 24,000 joint-error examples within ±0.2 m/±15°, 3,000 translation-only examples within ±0.3 m, and 3,000 rotation-only examples within ±20°. Four 2,000-example tests vary drive splits and error ranges; T2a uses the training-drive pool with separately sampled perturbations, whereas T1/T2b use held-out drives. T3 uses another date/drive and ±0.3 m/±2° errors. I2PNet receives no extra T3 fine-tuning, unlike the cited RGGNet comparison. Images resize to 352×1216 and all scan points are used.

Equations 31–32
Hi​n​i​t​i​a​l=ϕ​Hg​t,H_{initial}=\phi H_{gt}, (31)
ϕg​t=ϕ−1.\phi_{gt}=\phi^{-1}. (32)

V-B Experiment Result and Visualization

Evaluation uses mean Lie-algebra pose error and mean recalibration rate, a normalized error-reduction measure. I2PNet’s recalibration rates are 99.04%, 98.61%, 97.81%, and 97.21% on T1, T2a, T2b, and T3. These are improvements in synthetic calibration-error metrics, not percentages of perfectly calibrated sensors. The retained overlays show results under all four perturbation regimes.

Equation 33
M​S​E​E=1n​∑i=1nEi,M​R​R=1n​∑i=1nηi−Eiηi,MSEE=\frac{1}{n}\sum_{i=1}^{n}E_{i},MRR=\frac{1}{n}\sum_{i=1}^{n}\frac{\eta_{i}-E_{i}}{\eta_{i}},\vskip-5.69054pt (33)

VI Conclusion

Joint raw-point/image association and pose regression improve the tested localization and calibration results, with relatively fast inference. The main unresolved issue is scale: fine-grained registration cannot directly search kilometer-scale maps, so the authors propose adding place recognition to select a local map. Further practical dependencies include an accurate map, supplied camera intrinsics, suitable initialization or map pairing, and learned generalization to new sensors and environments. Conditional error reporting and synthetic perturbations bound the interpretation of the reported accuracy.

-A Network Hyperparameters

The complete table specifies feature channels, neighbor counts, sampling strides, kernel sizes, and distance thresholds for every module. The first vertical point stride adapts to 32- or 64-row scans. Coarse correspondence uses all image pixels or 32 neighbors depending on the initial-error regime; fine matching uses 32. These task-dependent settings are part of the reported method.

Table XI: Network Hyperparameters of I2PNet
Module Layer Type KK (Sh,Sw)(S_{h},S_{w}) Kernel Size Distance Channel Dimensions
Layer 1 — (4,4) — — [16,16,16,16,32]
Layer 2 — (4,4) — — [32,32,32,32,64]
Image Feature Extraction Layer 3 — (2,2) — — [64,64,64,64,128]
Layer 1 32 (2 or 4,8) (9,15) 0.75 [16,16,32]
Layer 2 16 (2,2) (9,15) 3.00 [32,32,64]
Layer 3 16 (2,2) (5,9) 6.00 [64,64,128]
Layer 4 16 (1,2) (5,9) 12.0 [128,128,256]
Point Cloud Feature Extraction Context Gathering for EcE^{c} 16 (1,2) (5,9) 12.0 [128,64,64]
Layer 4 for EcE^{c} M3M^{3} or 32, 4 — (3,5) 4.50 [128,64,64], [128,64], [64]
2D-3D Cost Volume Layer 3 for EfE^{f} 32, 4 — (3,5) 4.50 [128,64,64], [128,64], [64]
Upsampling Layer for U​EUE 8 (1,1/2) (5,9) 9.00 [128,64],[64]
Upsampling Layer for U​MUM 8 (1,1/2) (5,9) 9.00 [128,64],[64]
Upsampling Layer Cost Volume Optimization for O​EOE — — — — [128,64]
Outlier Mask Prediction for McM^{c} — — — — [128,64]
Outlier Mask Prediction for MfM^{f} — — — — [128,64]
Cost Volume Optimization and Outlier Mask Prediction FC for Middle Feature — — — — [256]
FC for qcq_{c}, FC for tct_{c} — — — — [4], [3]
FC for qfi+1q_{f}^{i+1}, FC for tfi+1t_{f}^{i+1} — — — — [4], [3]
Pose Regression
Concept demonstration

Open image ↗