Literature
Display
← All papers

Acquisition of Localization Confidence for Accurate Object Detection

Borui Jiang · Ruixuan Luo · Jiayuan Mao · Tete Xiao · Yuning Jiang

· ECCV 2018 ↗

Computer Vision5 min text · CondensedOriginal paper ↗
Contents

Overview

This edition retains the complete paper’s seven numbered figures, four tables, two algorithms, and displayed equations. Subfigures remain paired with their original captions.

At a glance

IoU-Net predicts how well a detected bounding box overlaps its target, separating localization confidence from class confidence. The predicted overlap guides duplicate suppression and provides a differentiable objective for box refinement. On the reported MS-COCO validation split, a joint ResNet-101-FPN system improves average precision from 38.5 to 40.6. Gains concentrate at stricter overlap thresholds; low-overlap estimation error and additional computation remain limitations.

1 Introduction

Detection needs both the right label and an accurately placed box, but class scores do not directly measure localization. Ranking by class confidence can suppress a better box, and repeated regression can eventually worsen coordinates. IoU-Net adds a localization estimate that can rank boxes and guide gradient-based refinement. The contribution is compatible with the evaluated region-based detectors, rather than a new complete detection pipeline.

Refer to caption
(a) Demonstrative cases of the misalignment between classification confidence and localization accuracy. The yellow bounding boxes denote the ground-truth, while the red and green bounding boxes are both detection results yielded by FPN [16]. Localization confidence is computed by the proposed IoU-Net. Using classification confidence as the ranking metric will cause accurately localized bounding boxes (in green) being incorrectly eliminated in the traditional NMS procedure. Quantitative analysis is provided in Section 2.1
Refer to caption
(b) Demonstrative cases of the non-monotonic localization in iterative bounding box regression. Quantitative analysis is provided in Section 2.2.
Figure 1: Visualization on two drawbacks brought by the absence of localization confidence. Examples are selected from MS-COCO minival [17].

2 Delving into object localization

The diagnostic uses a standard feature-pyramid detector trained on COCO trainval35k and evaluated on minival. It separates errors caused by suppression from errors caused by repeated box regression.

2.1 Misaligned classification and localization accuracy

Class scores distinguish foreground from background under an overlap threshold; they are not trained to order every positive box by localization quality. For boxes above 0.5 overlap, the plotted correlation with class confidence is 0.217, compared with 0.617 for the learned localization estimate. Ordinary suppression removes many highly accurate boxes when a less accurate neighbor has a higher class score. The no-suppression reference measures available candidate recall, not a usable final detector with controlled duplicates.

Refer to caption
(a) IoU vs. Classification Confidence
Refer to caption
(b) IoU vs. Localization Confidence
Figure 2: The correlation between the IoU of bounding boxes with the matched ground-truth and the classification/localization confidence. Considering detected bounding boxes having an IoU (>0.5>0.5) with the corresponding ground-truth, the Pearson correlation coefficients are: (a) 0.217, and (b) 0.617.
(a) The classification confidence indicates the category of a bounding box, but cannot be interpreted as the localization accuracy.
(b) To resolve the issue, we propose IoU-Net to predict the localization confidence for each detected bounding box, i.e., its IoU with corresponding ground-truth.
Refer to caption
Figure 3: The number of positive bounding boxes after the NMS, grouped by their IoU with the matched ground-truth. In traditional NMS (blue bar), a significant portion of accurately localized bounding boxes get mistakenly suppressed due to the misalignment of classification confidence and localization accuracy, while IoU-guided NMS (yellow bar) preserves more accurately localized bounding boxes.

2.2 Non-monotonic bounding box regression

Bounding-box regression learns a transformation from proposals to targets. Feeding its output back repeatedly changes the input distribution and need not improve localization monotonically. Experiments with feature-pyramid and cascade detectors show precision dropping after sufficient repeated regression. A learned localization objective offers a stopping signal and a direction for refinement, although its quality still depends on prediction error.

Refer to caption
(a) FPN
Refer to caption
(b) Cascade R-CNN
Figure 4: Optimization-based v.s. Regression-based BBox refinement. (a) Comparison in FPN. When applying the regression iteratively, the AP of detection results firstly get improved but drops quickly in later iterations. (b) Camparison in Cascade R-CNN. Iteration 0, 1 and 2 represents the 1st, 2nd and 3rd regression stages in Cascade R-CNN. For iteration i≥3i\geq 3, we refine the bounding boxes with the regressor of the third stage. After multiple iteration, AP slightly drops, while the optimization-based method further improves the AP by 0.8%.

3 IoU-Net

IoU-Net first learns an overlap predictor, then uses it for suppression and optimization-based refinement. The final experiment jointly trains this predictor alongside the detector.

3.1 Learning to predict IoU

Training boxes are generated by randomly transforming ground-truth boxes rather than using only region proposals. Boxes below 0.5 overlap are discarded, and the remaining examples are sampled uniformly across overlap values. A class-aware two-layer head predicts overlap from pooled feature-pyramid features. This sampling is intended to reduce dependence on one detector’s proposal distribution; broad robustness beyond the tested detectors is not established.

Original figure
Figure 5: Full architecture of the proposed IoU-Net described in Section 3.4. Input images are first fed into an FPN backbone. The IoU predictor takes the output features from the FPN backbone. We replace the RoI Pooling layer with a PrRoI Pooling layer described in Section 3.3. The IoU predictor shares a similar structure with the R-CNN branch. The modules marked within the dashed box form a standalone IoU-Net.

3.2 IoU-guided NMS

IoU-guided suppression selects the remaining box with the highest predicted overlap, then suppresses sufficiently overlapping neighbors. The selected box inherits the maximum classification score among the boxes it suppresses. This keeps geometric ranking distinct from label confidence. Algorithm 1 preserves the complete procedure; inference uses predicted overlap, not ground-truth overlap.

Algorithm 1 IoU-guided NMS. Classification confidence and localization confidence are disentangled in the algorithm. We use the localization confidence (the predicted IoU) to rank all detected bounding boxes, and update the classification confidence based on a clustering-like rule.
1: ℬ={b1,…,bn},𝒮,ℐ,Ωnms\mathcal{B}=\{b_{1},...,b_{n}\},~\mathcal{S},~\mathcal{I},~\Omega_{\text{nms}}
ℬ\mathcal{B} is a set of detected bounding boxes.
𝒮\mathcal{S} and ℐ\mathcal{I} are functions (neural networks) mapping bounding boxes to their classification confidence and IoU estimation (localization confidence) respectively.
Ωnms\Omega_{\text{nms}} is the NMS threshold.
2: 𝒟\mathcal{D}, the set of detected bounding boxes with classification scores.
3: 𝒟←∅\mathcal{D}\leftarrow{}\varnothing
4: while ℬ≠∅\mathcal{B}\neq\varnothing do
5:   bm←arg⁡max⁡ℐ⁡(bj)b_{m}\leftarrow{}\arg\max{\mathcal{I}(b_{j})}
6:   ℬ←ℬ∖{bm}\mathcal{B}\leftarrow{}\mathcal{B}\setminus\{b_{m}\}
7:   s←𝒮⁡(bm)s\leftarrow\mathcal{S}(b_{m})
8:   for bj∈ℬb_{j}\in\mathcal{B} do
9:    if IoU⁡(bm,bj)>Ωnms\mathrm{IoU(b_{m},b_{j})>\Omega_{\text{nms}}} then
10:      s←m​a​x​(s,𝒮⁡(bj))s\leftarrow{max(s,\mathcal{S}(b_{j}))}
11:      ℬ←ℬ∖{bj}\mathcal{B}\leftarrow{}\mathcal{B}\setminus\{b_{j}\}
12:    end if
13:   end for
14:   𝒟←𝒟∪{⟨bm,s⟩}\mathcal{D}\leftarrow{\mathcal{D}\cup\{\langle b_{m},s\rangle\}}
15: end while
16: return 𝒟\mathcal{D}

3.3 Bounding box refinement as an optimization procedure

Refinement treats predicted IoU as an objective and performs gradient ascent on box coordinates, initialized by one ordinary regression step. Gradients are scaled by box dimensions, and stopping rules use predicted improvement and degeneration tolerance. Algorithm 2 and all equations are retained. Precise RoI Pooling integrates a bilinearly interpolated continuous feature map over each pooling bin, enabling continuous coordinate derivatives without quantization. The empirically monotonic improvements in the tested plots are not a general guarantee that optimizing estimated IoU always improves true overlap.

Algorithm 2 Optimization-based bounding box refinement
1: ℬ={b1,…,bn},ℱ,T,λ,Ω1,Ω2\mathcal{B}=\{b_{1},...,b_{n}\},~\mathcal{F},~T,~\lambda,~\Omega_{1},~\Omega_{2}
ℬ\mathcal{B} is a set of detected bounding boxes, in the form of (x0,y0,x1,y1)(x_{0},y_{0},x_{1},y_{1}).
ℱ\mathcal{F} is the feature map of the input image.
TT is number of steps. λ\lambda is the step size, and Ω1\Omega_{1} is an early-stop threshold and Ω2<0\Omega_{2}<0 is an localization degeneration tolerance.
Function PrPool\mathrm{PrPool} extracts the feature representation for a given bounding box and function IoU\mathrm{IoU} denotes the estimation of IoU by the IoU-Net.
2: The set of final detection bounding boxes.
3: 𝒜←∅\mathcal{A}\leftarrow\varnothing
4: for i=1​t​o​Ti=1~to~T do
5:   for bj∈ℬb_{j}\in\mathcal{B} and bj∉𝒜b_{j}\notin\mathcal{A} do
6:    𝒈​𝒓​𝒂​𝒅←∇b​jIoU​(PrPool⁡(ℱ,bj))\bm{grad}\leftarrow\nabla_{bj}\mathrm{IoU(PrPool}(\mathcal{F},b_{j}))
7:    P​r​e​v​S​c​o​r​e←IoU⁡(PrPool⁡(ℱ,bj))PrevScore\leftarrow{\mathrm{IoU(PrPool}(\mathcal{F},b_{j}))}
8:    bj←bj+λ∗s​c​a​l​e​(𝒈​𝒓​𝒂​𝒅,bj)b_{j}\leftarrow b_{j}+\lambda*scale(\bm{grad},b_{j})
9:    N​e​w​S​c​o​r​e←IoU⁡(PrPool⁡(ℱ,bj))NewScore\leftarrow{\mathrm{IoU(PrPool}(\mathcal{F},b_{j}))}
10:    if |P​r​e​v​S​c​o​r​e−N​e​w​S​c​o​r​e|<Ω1|PrevScore-NewScore|<\Omega_{1} or N​e​w​S​c​o​r​e−P​r​e​v​S​c​o​r​e<Ω2NewScore-PrevScore<\Omega_{2} then
11:      𝒜←𝒜∪{bj}\mathcal{A}\leftarrow\mathcal{A}\cup\{b_{j}\}
12:    end if
13:   end for
14: end for
15: return ℬ\mathcal{B}
Equation 1
c∗=arg⁡minc​c​r​i​t​(transform​(b​o​xdet,c),b​o​xgt),c^{*}=\arg\min_{c}crit{\large(}\textbf{transform}(box_{\text{det}},c),box_{\text{gt}}\large{)}\,, (1)
Refer to caption
Figure 6: Illustration of RoI Pooling, RoI Align and PrRoI Pooling.
Equations 2–4
f⁡(x,y)=∑i,jI​C​(x,y,i,j)×wi,j,f(x,y)=\sum_{i,j}IC(x,y,i,j)\times w_{i,j}, (2)
PrPool⁡(b​i​n,ℱ)=∫y​1y​2∫x​1x​2f⁡(x,y)​𝑑x​𝑑y(x2−x1)×(y2−y1).\mathrm{PrPool}(bin,\mathcal{F})=\frac{\displaystyle\int_{y1}^{y2}\hskip-5.69046pt\int_{x1}^{x2}\!f(x,y)\,dxdy}{(x_{2}-x_{1})\times(y_{2}-y_{1})}. (3)
∂PrPool⁡(b​i​n,ℱ)∂x1\displaystyle\frac{\partial\mathrm{PrPool}(bin,\mathcal{F})}{\partial x_{1}} =PrPool⁡(b​i​n,ℱ)x2−x1−∫y​1y​2f⁡(x1,y)​𝑑y(x2−x1)×(y2−y1).\displaystyle=\ \ \ \frac{\mathrm{PrPool}(bin,\mathcal{F})}{x_{2}-x_{1}}-\frac{\int_{y1}^{y2}f(x_{1},y)\,dy}{(x_{2}-x_{1})\times(y_{2}-y_{1})}. (4)

3.4 Joint training

The overlap head shares the ResNet-FPN backbone with classification and regression, with Precise RoI Pooling replacing ordinary pooling. It is trained with smooth-L1 loss on normalized overlap targets and separately generated boxes. Training uses 16-image batches for 160,000 iterations, with warmup and a learning-rate decrease after 120,000 steps. At inference, suppression runs first, followed by refinement of the 100 highest-class-confidence boxes. The reported refinement uses at most five iterations, step size 0.5, early-stop threshold 0.001, and degeneration tolerance -0.01.

4 Experiments

Experiments use the 80-category COCO detection benchmark: approximately 115,000 training images and 5,000 validation images. Standalone-predictor comparisons isolate suppression and refinement; the final experiments evaluate joint training. The preserved tables report both overall average precision and increasingly strict overlap thresholds.

4.1 IoU-guided NMS

IoU-guided suppression particularly improves strict-localization metrics. At matching overlap 0.9, recall is 18.7% after ordinary suppression and 28.9% after IoU-guided suppression, versus 39.7% before suppression. Overall gains are smaller and are not uniformly larger than Soft-NMS for every detector; for Cascade R-CNN, Table 1 gives 40.7 overall AP for IoU-NMS and 40.9 for Soft-NMS, while IoU-NMS has the higher AP90.

Method   +Soft-NMS   +IoU-NMS   AP\mathrm{AP} AP50\mathrm{AP_{50}} AP60\mathrm{AP_{60}} AP70\mathrm{AP_{70}} AP80\mathrm{AP_{80}} AP90\mathrm{AP_{90}}
FPN 36.4 58.0 53.1 44.9 31.2 9.8
✓\checkmark 36.8 57.5 53.1 45.7 32.3 10.3
✓\checkmark 37.3 56.0 52.2 45.6 33.9 13.3
Cascade R-CNN 40.6 59.3 55.2 49.1 38.7 16.7
✓\checkmark 40.9 58.2 54.7 49.4 39.9 17.8
✓\checkmark 40.7 58.0 54.7 49.2 38.8 18.9
Mask-RCNN 37.5 58.6 53.9 46.3 33.2 10.9
✓\checkmark 37.9 58.2 53.9 47.1 34.4 11.5
✓\checkmark 38.1 56.4 52.7 46.7 35.1 14.6
Table 1: Comparison of IoU-guided NMS with other NMS methods. By preserving bounding boxes with accurate localization, IoU-guided NMS shows significant improvement in AP with high matching IoU threshold (e.g., AP90{}_{\text{90}}).
Figure 7: Recall curves of different NMS methods at different IoU threshold for matching detected bounding boxes with the ground-truth. No-NMS (no box is suppressed) is provided as the upper bound of the recall. The proposed IoU-NMS has a higher recall and effectively narrows the gap to the upper-bound at high IoU threshold (e.g., 0.8).
Refer to caption

4.2 Optimization-based bounding box refinement

Adding optimization-based refinement improves the tested feature-pyramid, cascade, and mask detectors. For Cascade R-CNN, overall AP rises from 40.6 to 41.4 and AP90 from 16.7 to 19.5. These are increases of 0.8 and 2.8 percentage points, not relative percent gains.

Method +Refinement AP\mathrm{AP} AP50\mathrm{AP_{50}} AP60\mathrm{AP_{60}} AP70\mathrm{AP_{70}} AP80\mathrm{AP_{80}} AP90\mathrm{AP_{90}}
FPN 36.4 58.0 53.1 44.9 31.2 9.8
✓\checkmark 38.0 57.7 53.1 46.1 34.3 14.6
Cascade R-CNN 40.6 59.3 55.2 49.1 38.7 16.7
✓\checkmark 41.4 59.3 55.3 49.6 39.4 19.5
Mask-RCNN 37.5 58.6 53.9 46.3 33.2 10.9
✓\checkmark 39.2 57.9 53.6 47.4 36.5 16.4
Table 2: The optimization-based bounding box refinement further improves the performance of several CNN-based object detectors.

4.3 Joint training

Joint training plus suppression and refinement raises ResNet-50-FPN from 36.4 to 38.1 AP and ResNet-101-FPN from 38.5 to 40.6. On a TITAN X at 1200-by-800 input resolution, the listed ResNet-50 IoU-Net takes 0.305 seconds per image versus 0.255 for the FPN baseline. AP50 can decline because low-overlap boxes have more variable appearance and less accurate IoU estimates. Sampling more low-overlap training boxes is suggested as a partial remedy.

Backbone Method +IoU-NMS  +Refine   AP\mathrm{AP} AP50\mathrm{AP_{50}} AP60\mathrm{AP_{60}} AP70\mathrm{AP_{70}} AP80\mathrm{AP_{80}} AP90\mathrm{AP_{90}}
ResNet-50 FPN 36.4 58.0 53.1 44.9 31.2 9.8
IoU-Net 37.0 58.3 53.8 45.7 31.9 10.7
✓\checkmark 37.6 56.2 52.4 46.0 34.1 14.0
✓\checkmark ✓\checkmark 38.1 56.3 52.4 46.3 35.1 15.5
ResNet-101 FPN 38.5 60.3 55.5 47.6 33.8 11.3
IoU-Net 38.9 60.2 55.5 47.8 34.6 12.0
✓\checkmark 40.0 59.0 55.1 48.6 37.0 15.5
✓\checkmark ✓\checkmark 40.6 59.0 55.2 49.0 38.0 17.1
Table 3: Final experiment results on MS-COCO. IoU-Net denotes ResNet-FPN embedded with IoU predictor. We improve the FPN baseline by ≈2%\approx 2\% in AP.
Method     FPN  Mask-RCNN Cascade R-CNN     IoU-Net
Speed (sec./image) 0.255 0.267 0.384 0.305
Table 4: Inference speed of multiple object detectors on a single TITAN X GPU. The models share the same backbone network ResNet50-FPN. The input resolution is 1200x800. All hyper-parameters are set to be the same.

5 Conclusion

Localization confidence provides useful information that a class score alone omits. The experiments demonstrate improved high-overlap detection through separate geometric ranking and differentiable refinement. Benefits depend on the accuracy of the learned overlap predictor and add inference cost; the reported evidence is limited to the specified COCO split and detector families.

Concept demonstration

Open image ↗