Literature
Display
← All papers

Multi-Scale Context Aggregation by Dilated Convolutions

Fisher Yu · Vladlen Koltun

Published · ICLR 2016 ↗

Computer Vision4 min text · CondensedOriginal paper ↗
Contents

At a glance

Dense prediction needs both precise pixel locations and broad context. Dilated convolutions enlarge the receptive field while preserving feature-map resolution. A small context module improves several segmentation systems; simplifying an inherited image-classification backbone also improves accuracy. The appendix extends the design to urban-scene datasets.

1 Introduction

Pooling-heavy classification networks discard spatial detail that segmentation must later recover. Rather than repeatedly downsample and upsample or process multiple image scales, the proposed context module keeps resolution constant and expands filter spacing. Controlled experiments separate the benefits of this module from structured prediction and extra training data.

2 Dilated Convolutions

A dilation factor spaces a convolution’s sampling positions without adding filter parameters. Stacking 3 × 3 filters with exponentially increasing dilation gives exponentially larger receptive fields while keeping dense output coverage. This operator has roots in wavelet methods; it is a component of the à trous algorithm, not the algorithm itself.

Equations 1–3
(F∗k)​(𝐩)=∑𝐬+𝐭=𝐩F⁡(𝐬)​k​(𝐭).(F\ast k)(\mathbf{p})=\sum_{\mathbf{s}+\mathbf{t}=\mathbf{p}}F(\mathbf{s})\,k(\mathbf{t}). (1)
(F∗lk)​(𝐩)=∑𝐬+l​𝐭=𝐩F⁡(𝐬)​k​(𝐭).(F\ast_{l}k)(\mathbf{p})=\sum_{\mathbf{s}+l\mathbf{t}=\mathbf{p}}F(\mathbf{s})\,k(\mathbf{t}). (2)
Fi+1=Fi∗2ikifor ​i=0,1,…,n−2.\displaystyle F_{i+1}=F_{i}\ast_{2^{i}}k_{i}\quad\textup{for\ \ }i=0,1,\ldots,n-2. (3)
Original research figure
(a)
Original research figure
(b)
Original research figure
(c)
Figure 1: Systematic dilation supports exponential expansion of the receptive field without loss of resolution or coverage. (a) F1F_{1} is produced from F0F_{0} by a 1-dilated convolution; each element in F1F_{1} has a receptive field of 3×33\mathbin{\!\times\!}3. (b) F2F_{2} is produced from F1F_{1} by a 2-dilated convolution; each element in F2F_{2} has a receptive field of 7×77\mathbin{\!\times\!}7. (c) F3F_{3} is produced from F2F_{2} by a 4-dilated convolution; each element in F3F_{3} has a receptive field of 15×1515\mathbin{\!\times\!}15. The number of parameters associated with each layer is identical. The receptive field grows exponentially while the number of parameters grows linearly.

3 Multi-Scale Context Aggregation

The basic context module uses seven 3 × 3 layers with dilations 1, 1, 2, 4, 8, 16, and 1, followed by a 1 × 1 output layer. It keeps one feature map per class and reaches a 67 × 67 receptive field. Identity initialization lets it start by passing predictions through; ordinary random initialization was ineffective here. A larger version adds channels and uses a generalized identity initialization with small noise to break symmetry.

Equation 4
kb(𝐭,a)=1[𝐭=0]1[a=b],k^{b}(\mathbf{t},a)=1_{[\mathbf{t}=0]}1_{[a=b]}, (4)
Layer 1 2 3 4 5 6 7 8
Convolution 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 3×33\mathbin{\!\times\!}3 1×11\mathbin{\!\times\!}1
Dilation 1 1 2 4 8 16 1 1
Truncation Yes Yes Yes Yes Yes Yes Yes No
Receptive field 3×33\mathbin{\!\times\!}3 5×55\mathbin{\!\times\!}5 9×99\mathbin{\!\times\!}9 17×1717\mathbin{\!\times\!}17 33×3333\mathbin{\!\times\!}33 65×6565\mathbin{\!\times\!}65 67×6767\mathbin{\!\times\!}67 67×6767\mathbin{\!\times\!}67
Output channels
Basic CC CC CC CC CC CC CC CC
Large 2​C2C 2​C2C 4​C4C 8​C8C 16​C16C 32​C32C 32​C32C CC
Table 1: Context network architecture. The network processes CC feature maps by aggregating contextual information at progressively increasing scales without losing resolution.
Equation 5
kb​(𝐭,a)={Cci+1𝐭=0​ and ​⌊a​Cci⌋=⌊b​Cci+1⌋εotherwisek^{b}(\mathbf{t},a)=\begin{cases}\displaystyle{C\over c_{i+1}}\quad&\displaystyle\mathbf{t}=0\textup{\ \ and\ \ }\left\lfloor{aC\over c_{i}}\right\rfloor=\left\lfloor{bC\over c_{i+1}}\right\rfloor\\ \quad\!\varepsilon\quad&\displaystyle\textup{otherwise}\end{cases} (5)
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(a) Image
Refer to caption
(b) FCN-8s
Refer to caption
(c) DeepLab
Refer to caption
(d) Our front end
Refer to caption
(e) Ground truth
Figure 2: Semantic segmentations produced by different adaptations of the VGG-16 classification network. From left to right: (a) input image, (b) prediction by FCN-8s (Long et al., 2015), (c) prediction by DeepLab (Chen et al., 2015a), (d) prediction by our simplified front-end module, (e) ground truth.

aero

bike

bird

boat

bottle

bus

car

cat

chair

cow

table

dog

horse

mbike

person

plant

sheep

sofa

train

tv

mean IoU

FCN-8s 76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 62.5 46.8 71.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 55.1 62.2
DeepLab 72 31 71.2 53.7 60.5 77 71.9 73.1 25.2 62.6 49.1 68.7 63.3 73.9 73.6 50.8 72.3 42.1 67.9 52.6 62.1
DeepLab-Msc 74.9 34.1 72.6 52.9 61.0 77.9 73.0 73.7 26.4 62.2 49.3 68.4 64.1 74.0 75.0 51.7 72.7 42.5 67.2 55.7 62.9
Our front end 82.2 37.4 72.7 57.1 62.7 82.8 77.8 78.9 28 70 51.6 73.1 72.8 81.5 79.1 56.6 77.1 49.9 75.3 60.9 67.6
Table 2: Our front-end prediction module is simpler and more accurate than prior models. This table reports accuracy on the VOC-2012 test set.

4 Front End

The front end adapts VGG-16 by removing the last two pooling/striding operations and dilating later filters, preserving pretrained weights while increasing output resolution. It also removes intermediate padding and uses reflection padding around the input image. Trained on the standard augmented PASCAL split, this simpler front end reaches 67.6% test mean intersection over union, more than five points above the compared FCN-8s and DeepLab versions.

5 Experiments

A stronger front end first trains on PASCAL plus matching COCO categories, then fine-tunes on PASCAL alone. It reaches 69.8% validation and 71.3% test mean intersection over union before context or structured prediction. This distinction matters: comparisons must account for the extra training images.

Controlled evaluation of context aggregation.

Both context-module sizes improve the front end alone, the front end with a dense conditional random field, and the recurrent conditional-random-field system. The larger module helps more. Its benefits complement structured prediction; joint front-end/context training does not significantly help in these PASCAL experiments.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(a) Image
Refer to caption
(b) Front end
Refer to caption
(c)  + Context
Refer to caption
(d)  + CRF-RNN
Refer to caption
(e) Ground truth
Figure 3: Semantic segmentations produced by different models. From left to right: (a) input image, (b) prediction by the front-end module, (c) prediction by the large context network plugged into the front end, (d) prediction by the front end + context module + CRF-RNN, (e) ground truth.

aero

bike

bird

boat

bottle

bus

car

cat

chair

cow

table

dog

horse

mbike

person

plant

sheep

sofa

train

tv

mean IoU

Front end 86.3 38.2 76.8 66.8 63.2 87.3 78.7 82 33.7 76.7 53.5 73.7 76 76.6 83 51.9 77.8 44 79.9 66.3 69.8
Front + Basic 86.4 37.6 78.5 66.3 64.1 89.9 79.9 84.9 36.1 79.4 55.8 77.6 81.6 79 83.1 51.2 81.3 43.7 82.3 65.7 71.3
Front + Large 87.3 39.2 80.3 65.6 66.4 90.2 82.6 85.8 34.8 81.9 51.7 79 84.1 80.9 83.2 51.2 83.2 44.7 83.4 65.6 72.1
Front end + CRF 89.2 38.8 80 69.8 63.2 88.8 80 85.2 33.8 80.6 55.5 77.1 80.8 77.3 84.3 53.1 80.4 45 80.7 67.9 71.6
Front + Basic + CRF 89.1 38.7 81.4 67.4 65 91 81 86.7 37.5 81 57 79.6 83.6 79.9 84.6 52.7 83.3 44.3 82.6 67.2 72.7
Front + Large + CRF 89.6 39.9 82.7 66.7 67.5 91.1 83.3 87.4 36 83.3 52.5 80.7 85.7 81.8 84.4 52.6 84.4 45.3 83.7 66.7 73.3
Front end + RNN 88.8 38.1 80.8 69.1 65.6 89.9 79.6 85.7 36.3 83.6 57.3 77.9 83.2 77 84.6 54.7 82.1 46.9 80.9 66.7 72.5
Front + Basic + RNN 89 38.4 82.3 67.9 65.2 91.5 80.4 87.2 38.4 82.1 57.7 79.9 85 79.6 84.5 53.5 84 45 82.8 66.2 73.1
Front + Large + RNN 89.3 39.2 83.6 67.2 69 92.1 83.1 88 38.4 84.8 55.3 81.2 86.7 81.3 84.3 53.6 84.4 45.8 83.8 67 73.9
Table 3: Controlled evaluation of the effect of the context module on the accuracy of three different architectures for semantic segmentation. Experiments performed on the VOC-2012 validation set. Validation images were not used for training. Top: adding the context module to a semantic segmentation front end with no structured prediction (Long et al., 2015). The basic context module increases accuracy, the large module increases it by a larger margin. Middle: the context module increases accuracy when plugged into a front-end + dense CRF configuration (Chen et al., 2015a). Bottom: the context module increases accuracy when plugged into a front-end + CRF-RNN configuration (Zheng et al., 2015).

Evaluation on the test set.

Held-out test submissions confirm the gains. Context alone beats the compared DeepLab configuration; context plus the dense conditional random field matches the cited recurrent system, and combining context with that recurrent system improves further. The original classwise table preserves the exact comparison settings and scores.

aero

bike

bird

boat

bottle

bus

car

cat

chair

cow

table

dog

horse

mbike

person

plant

sheep

sofa

train

tv

mean IoU

DeepLab++ 89.1 38.3 88.1 63.3 69.7 87.1 83.1 85 29.3 76.5 56.5 79.8 77.9 85.8 82.4 57.4 84.3 54.9 80.5 64.1 72.7
DeepLab-MSc++ 89.2 46.7 88.5 63.5 68.4 87.0 81.2 86.3 32.6 80.7 62.4 81.0 81.3 84.3 82.1 56.2 84.6 58.3 76.2 67.2 73.9
CRF-RNN 90.4 55.3 88.7 68.4 69.8 88.3 82.4 85.1 32.6 78.5 64.4 79.6 81.9 86.4 81.8 58.6 82.4 53.5 77.4 70.1 74.7
Front end 86.6 37.3 84.9 62.4 67.3 86.2 81.2 82.1 32.6 77.4 58.3 75.9 81 83.6 82.3 54.2 81.5 50.1 77.5 63 71.3
Context 89.1 39.1 86.8 62.6 68.9 88.2 82.6 87.7 33.8 81.2 59.2 81.8 87.2 83.3 83.6 53.6 84.9 53.7 80.5 62.9 73.5
Context + CRF 91.3 39.9 88.9 64.3 69.8 88.9 82.6 89.7 34.7 82.7 59.5 83 88.4 84.2 85 55.3 86.7 54.4 81.9 63.6 74.7
Context + CRF-RNN 91.7 39.6 87.8 63.1 71.8 89.7 82.9 89.8 37.2 84 63 83.3 89 83.8 85.1 56.8 87.6 56 80.2 64.7 75.3
Table 4: Evaluation on the VOC-2012 test set. ‘DeepLab++’ stands for DeepLab-CRF-COCO-LargeFOV and ‘DeepLab-MSc++’ stands for DeepLab-MSc-CRF-LargeFOV-COCO-CrossJoint (Chen et al., 2015a). ‘CRF-RNN’ is the system of Zheng et al. (2015). ‘Context’ refers to the large context module plugged into our front end. The context network yields very high accuracy, ourperforming the DeepLab++ architecture without performing structured prediction. Combining the context network with the CRF-RNN structured prediction module increases the accuracy of the CRF-RNN system.

6 Conclusion

Dilated context aggregation offers a dedicated architecture for dense prediction. Its success also shows that components inherited from classification can hurt segmentation. Failure examples remain: the method does not eliminate semantic confusion or fine-boundary errors. End-to-end dense training without classification pretraining is proposed as future work.

Acknowledgements

The authors thank Vibhav Vineet for experimental help and discussion, and Jonathan Long and the Caffe team for feedback and integration of dilated convolution.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(a) Image
Refer to caption
(b) Our result
Refer to caption
(c) Ground truth
Refer to caption
(d) Image
Refer to caption
(e) Our result
Refer to caption
(f) Ground truth
Figure 4: Failure cases from the VOC-2012 validation set. The most accurate architecture we trained (Context + CRF-RNN) performs poorly on these images.

Appendix A Urban Scene Understanding

Urban-scene experiments use the same front-end/context pattern, with context depth matched to image resolution. All reported results here omit structured prediction and train only on the training split. Front-end training uses random reflected-padding crops, SGD, and momentum; dataset-specific joint fine-tuning follows.

A.1 CamVid

CamVid uses 367 training, 100 validation, and 233 test images at 640 × 480 resolution with 11 classes. Dilation8 trains the front end for 20,000 iterations and then jointly fine-tunes front end and context. It improves on the compared systems; the original classwise results are retained.

Building

Tree

Sky

Car

Sign

Road

Pedestrian

Fence

Pole

Sidewalk

Bicyclist

mean IoU

ALE 73.4 70.2 91.1 64.2 24.4 91.1 29.1 31.0 13.6 72.4 28.6 53.6
SuperParsing 70.4 54.8 83.5 43.3 25.4 83.4 11.6 18.3 5.2 57.4 8.9 42.0
Liu and He 66.8 66.6 90.1 62.9 21.4 85.8 28.0 17.8 8.3 63.5 8.5 47.2
SegNet 68.7 52.0 87.0 58.5 13.4 86.2 25.3 17.9 16.0 60.5 24.8 46.4
DeepLab-LFOV 81.5 74.6 89.0 82.2 42.3 92.2 48.4 27.2 14.3 75.4 50.1 61.6
Dilation8 82.6 76.2 89.9 84.0 46.9 92.2 56.3 35.8 23.4 75.3 55.5 65.3
Table 5: Semantic segmentation results on the CamVid dataset. Our model (Dilation8) is compared to ALE (Ladicky et al., 2009), SuperParsing (Tighe & Lazebnik, 2013), Liu and He (Liu & He, 2015), SegNet (Badrinarayanan et al., 2015), and the DeepLab-LargeFOV model (Chen et al., 2015a). Our model outperforms the prior work.

A.2 KITTI

KITTI uses 100 training and 46 test images at 1,226 × 370 resolution. The smaller vertical extent motivates removing one context layer, producing Dilation7. After front-end training, 20,000 joint-training iterations improve on the compared baselines.

Building

Tree

Sky

Car

Sign

Road

Pedestrian

Fence

Pole

Sidewalk

Bicyclist

mean IoU

Ros et al. 71.8 69.5 84.4 51.2 4.2 72.4 1.7 32.4 2.6 45.3 3.2 39.9
DeepLab-LFOV 82.8 78.6 82.4 78.0 28.8 91.3 0.0 39.4 29.9 72.4 12.9 54.2
Dilation7 84.6 81.1 83 81.4 41.8 92.9 4.6 47.1 35.2 73.1 26.4 59.2
Table 6: Semantic segmentation results on the KITTI dataset. We compare our results to Ros et al. (2015) and to the DeepLab-LargeFOV model (Chen et al., 2015a). Our network (Dilation7) yields higher accuracy than the prior work.

A.3 Cityscapes

Cityscapes’ 2,048 × 1,024 images require a larger context: Dilation10 adds dilation factors 32 and 64. Training proceeds through the front end, the context module, then joint fine-tuning on image halves. The reported model improves over the contemporary compared systems; the figure and two tables preserve stagewise examples and class/category results.

Refer to caption
Figure 5: Results produced by the Dilation10 model after different training stages. (a) Input image. (b) Ground truth segmentation. (c) Segmentation produced by the model after the first stage of training (front-end only). (d) Segmentation produced after the second stage, which trains the context module. (e) Segmentation produced after the third stage, in which both modules are trained jointly.

Road

Sidewalk

Building

Wall

Fence

Pole

Light

Sign

Vegetation

Terrain

Sky

Person

Rider

Car

Truck

Bus

Train

Motorcycle

Bicycle

mean IoU

Validation set
97.2 79.5 90.4 44.9 52.4 55.1 56.7 69 91 58.7 92.6 75.7 50 92.2 56.2 72.6 54.3 46.2 70.1 68.7
Test set
97.6 79.2 89.9 37.3 47.6 53.2 58.6 65.2 91.8 69.4 93.7 78.9 55 93.3 45.5 53.4 47.7 52.2 66 67.1
Table 7: Per-class and mean class-level IoU achieved by our model (Dilation10) on the Cityscapes dataset.
Flat Nature Object Sky Construction Human Vehicle  mean IoU
Validation set
98.2 91.4 62.3 92.6 90.7 77.6 91 86.3
Test set
98.3 91.4 60.5 93.7 90.2 79.8 91.8 86.5
Table 8: Per-category and mean category-level IoU on the Cityscapes dataset.
Concept demonstration

Open image ↗