Literature
Display
← All papers

Identity Mappings in Deep Residual Networks

Kaiming He · Xiangyu Zhang · Shaoqing Ren · Jian Sun

Published · ECCV 2016 ↗

Computer Vision4 min text · CondensedOriginal paper ↗
Contents

At a glance

Identity shortcuts work best when the addition itself is also followed by an identity mapping. Moving batch normalization and ReLU before the residual weights creates this clean path. Controlled ablations explain why it eases optimization; pre-activation networks also generalize better, including a 1,001-layer CIFAR model and a 200-layer ImageNet model.

1 Introduction

The original ResNet applies ReLU after combining the shortcut and residual branch. This paper separates two questions: whether the shortcut should remain an identity, and whether the after-addition activation should also be an identity. Both theory and ablations favor an uninterrupted additive path. The residual branch retains nonlinear processing; the whole network does not become linear. See the original formulation in [1].

Equation
𝐲l=h⁡(𝐱l)+ℱ⁡(𝐱l,𝒲l),\displaystyle\mathbf{y}_{l}=h(\mathbf{x}_{l})+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}),
𝐱l+1=f⁡(𝐲l),\displaystyle\mathbf{x}_{l+1}=f(\mathbf{y}_{l}),
Original research figure
Figure 1: Left: (a) original Residual Unit in [1]; (b) proposed Residual Unit. The grey arrows indicate the easiest paths for the information to propagate, corresponding to the additive term “𝐱l\mathbf{x}_{l}” in Eqn.(4) (forward propagation) and the additive term “1” in Eqn.(5) (backward propagation). Right: training curves on CIFAR-10 of 1001-layer ResNets. Solid lines denote test error (y-axis on the right), and dashed lines denote training loss (y-axis on the left). The proposed unit makes ResNet-1001 easier to train.

2 Analysis of Deep Residual Networks

When both relevant mappings are identities, a deeper representation equals a shallower representation plus the sum of intervening residual outputs. Backpropagation likewise contains a direct additive gradient term alongside the residual derivatives. The argument applies within constant-dimensional stages; the few projection/downsampling transitions are exceptions. It suggests a reliable information path, not an unconditional guarantee against every optimization failure.

Equations 1–5
𝐲l=h⁡(𝐱l)+ℱ⁡(𝐱l,𝒲l),\displaystyle\mathbf{y}_{l}=h(\mathbf{x}_{l})+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}), (1)
𝐱l+1=f⁡(𝐲l).\displaystyle\mathbf{x}_{l+1}=f(\mathbf{y}_{l}). (2)
𝐱l+1=𝐱l+ℱ⁡(𝐱l,𝒲l).\mathbf{x}_{l+1}=\mathbf{x}_{l}+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}). (3)
𝐱L=𝐱l+∑i=lL−1ℱ⁡(𝐱i,𝒲i),\mathbf{x}_{L}=\mathbf{x}_{l}+\sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i}), (4)
∂ℰ∂𝐱l=∂ℰ∂𝐱L​∂𝐱L∂𝐱l=∂ℰ∂𝐱L​(1+∂∂𝐱l​∑i=lL−1ℱ⁡(𝐱i,𝒲i)).\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{l}}}=\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\frac{\partial{\mathbf{x}_{L}}}{\partial{\mathbf{x}_{l}}}=\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\left(1+\frac{\partial}{\partial{\mathbf{x}_{l}}}\sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i})\right). (5)

Discussions

The direct forward and backward terms require both a clean skip connection and identity after-addition mapping. The following experiments change these components separately to test the analysis.

3 On the Importance of Identity Skip Connections

Scaling each shortcut multiplies the direct gradient path by the product of all intervening scale factors. Repeated factors below one can attenuate it; factors above one can amplify it. Gates and learned shortcut transforms introduce analogous products of derivatives. The extra representational flexibility can therefore make optimization harder.

Equations 6–8
𝐱l+1=λl​𝐱l+ℱ⁡(𝐱l,𝒲l),\displaystyle\mathbf{x}_{l+1}=\lambda_{l}\mathbf{x}_{l}+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}), (6)
𝐱L=(∏i=lL−1λi)​𝐱l+∑i=lL−1ℱ^​(𝐱i,𝒲i),\displaystyle\mathbf{x}_{L}=(\prod_{i=l}^{L-1}\lambda_{i})\mathbf{x}_{l}+\sum_{i=l}^{L-1}\mathcal{\hat{F}}(\mathbf{x}_{i},\mathcal{W}_{i}), (7)
∂ℰ∂𝐱l=∂ℰ∂𝐱L​((∏i=lL−1λi)+∂∂𝐱l​∑i=lL−1ℱ^​(𝐱i,𝒲i)).\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{l}}}=\frac{\partial{\mathcal{E}}}{\partial{\mathbf{x}_{L}}}\left((\prod_{i=l}^{L-1}\lambda_{i})+\frac{\partial}{\partial{\mathbf{x}_{l}}}\sum_{i=l}^{L-1}\mathcal{\hat{F}}(\mathbf{x}_{i},\mathcal{W}_{i})\right). (8)
Original research figure
Figure 2: Various types of shortcut connections used in Table 1. The grey arrows indicate the easiest paths for the information to propagate. The shortcut connections in (b-f) are impeded by different components. For simplifying illustrations we do not display the BN layers, which are adopted right after the weight layers for all units here.
Table 1: Classification error on the CIFAR-10 test set using ResNet-110 [1], with different types of shortcut connections applied to all Residual Units. We report “fail” when the test error is higher than 20%.
case Fig. on shortcut on ℱ\mathcal{F} error (%) remark
original [1] Fig. 2(a) 1 1 6.61
constant scaling Fig. 2(b) 0 1 fail This is a plain net
0.5 1 fail
0.5 0.5 12.35 frozen gating
exclusive gating Fig. 2(c) 1−g⁡(𝐱)1-g(\mathbf{x}) g⁡(𝐱)g(\mathbf{x}) fail init bgb_{g}=0 to −5-5
1−g⁡(𝐱)1-g(\mathbf{x}) g⁡(𝐱)g(\mathbf{x}) 8.70 init bgb_{g}=-6
1−g⁡(𝐱)1-g(\mathbf{x}) g⁡(𝐱)g(\mathbf{x}) 9.81 init bgb_{g}=-7
shortcut-only gating Fig. 2(d) 1−g⁡(𝐱)1-g(\mathbf{x}) 1 12.86 init bgb_{g}=0
1−g⁡(𝐱)1-g(\mathbf{x}) 1 6.91 init bgb_{g}=-6
1×\times1 conv shortcut Fig. 2(e) 1×\times1 conv 1 12.22
dropout shortcut Fig. 2(f) dropout 0.5 1 fail

3.1 Experiments on Skip Connections

On CIFAR-10, the original 110-layer network reaches 6.61% error. Constant scaling, dropout on shortcuts, and many gated variants fail or train poorly. Even learned 1 × 1 shortcuts reach 12.22%. Shortcut-only gates approach the baseline only when initialized close to identity. Results report median accuracy over five runs; these shortcut experiments still use the original post-addition ReLU.

Original research figure
Figure 3: Training curves on CIFAR-10 of various shortcuts. Solid lines denote test error (y-axis on the right), and dashed lines denote training loss (y-axis on the left).

3.2 Discussions

Several modified shortcuts can represent identity in principle yet have higher training error. Their weakness is consequently an optimization effect, not simply insufficient expressiveness. A direct information path is valuable even when a more complicated alternative includes it in its hypothesis class.

4 On the Usage of Activation Functions

Placing normalization after addition disrupts the shortcut. Moving only ReLU before addition restricts the residual output to nonnegative values. Full pre-activation instead places normalization and ReLU before each weight layer, allowing signed residual corrections and identity behavior after addition.

Table 2: Classification error (%) on the CIFAR-10 test set using different activation functions.
case Fig. ResNet-110 ResNet-164
original Residual Unit [1] Fig. 4(a) 6.61 5.93
BN after addition Fig. 4(b) 8.17 6.50
ReLU before addition Fig. 4(c) 7.84 6.14
ReLU-only pre-activation Fig. 4(d) 6.71 5.91
full pre-activation Fig. 4(e) 6.37 5.46
Original research figure
Figure 4: Various usages of activation in Table 2. All these units consist of the same components — only the orders are different.

4.1 Experiments on Activation

Full pre-activation improves CIFAR-10 error from 6.61% to 6.37% for ResNet-110 and from 5.93% to 5.46% for the bottleneck ResNet-164. It also helps one-layer skips and CIFAR-100 models. ReLU-only pre-activation gives little benefit; combining it with normalization matters. The 1,001-layer model improves particularly strongly.

Original research figure
Figure 5: Using asymmetric after-addition activation is equivalent to constructing a pre-activation Residual Unit.
Equation 9
𝐱l+1=𝐱l+ℱ⁡(f^​(𝐱l),𝒲l),.\mathbf{x}_{l+1}=\mathbf{x}_{l}+\mathcal{F}(\hat{f}(\mathbf{x}_{l}),\mathcal{W}_{l}),. (9)
Table 3: Classification error (%) on the CIFAR-10/100 test set using the original Residual Units and our pre-activation Residual Units.
dataset network baseline unit pre-activation unit
CIFAR-10 ResNet-110 (1layer skip) 9.90 8.91
ResNet-110 6.61 6.37
ResNet-164 5.93 5.46
ResNet-1001 7.61 4.92
CIFAR-100 ResNet-164 25.16 24.33
ResNet-1001 27.82 22.71
Original research figure
Figure 6: Training curves on CIFAR-10. Left: BN after addition (Fig. 4(b)) using ResNet-110. Right: pre-activation unit (Fig. 4(e)) on ResNet-164. Solid lines denote test error, and dashed lines denote training loss.

4.2 Analysis

Two effects appear in the learning curves. Deep pre-activation models reduce training loss more readily because addition does not truncate the direct path. Shallower models sometimes finish with slightly higher training loss but lower test error, consistent with normalization regularizing every weight layer’s input. The authors distinguish this interpretation from the stronger optimization evidence at 1,001 layers.

5 Results

With standard flip/translation augmentation, ResNet-1001 reaches 4.92% CIFAR-10 error at batch size 128 and 4.62% at batch size 64; CIFAR-100 error is 22.71%. On ImageNet, the original 200-layer model overfits relative to 152 layers, while pre-activation reduces 320-pixel single-crop error to 20.7% top-one and 5.3% top-five. Stronger scale/aspect augmentation reaches 20.1% and 4.8%. Training cost still grows roughly linearly with depth.

Table 4: Comparisons with state-of-the-art methods on CIFAR-10 and CIFAR-100 using “moderate data augmentation” (flip/translation), except for ELU [12] with no augmentation. Better results of [13, 14] have been reported using stronger data augmentation and ensembling. For the ResNets we also report the number of parameters. Our results are the median of 5 runs with mean±\pmstd in the brackets. All ResNets results are obtained with a mini-batch size of 128 except † with a mini-batch size of 64 (code available at https://github.com/KaimingHe/resnet-1k-layers).
CIFAR-10 error (%)
NIN [15] 8.81
DSN [16] 8.22
FitNet [17] 8.39
Highway [7] 7.72
All-CNN [14] 7.25
ELU [12] 6.55
FitResNet, LSUV [18] 5.84
ResNet-110 [1] (1.7M) 6.61
ResNet-1202 [1] (19.4M) 7.93
ResNet-164 [ours] (1.7M) 5.46
ResNet-1001 [ours] (10.2M) 4.92 (4.89±\pm0.14)
ResNet-1001 [ours] (10.2M)† 4.62 (4.69±\pm0.20)
CIFAR-100 error (%)
NIN [15] 35.68
DSN [16] 34.57
FitNet [17] 35.04
Highway [7] 32.39
All-CNN [14] 33.71
ELU [12] 24.28
FitNet, LSUV [18] 27.66
ResNet-164 [1] (1.7M) 25.16
ResNet-1001 [1] (10.2M) 27.82
ResNet-164 [ours] (1.7M) 24.33
ResNet-1001 [ours] (10.2M) 22.71 (22.68±\pm0.22)
Table 5: Comparisons of single-crop error on the ILSVRC 2012 validation set. All ResNets are trained using the same hyper-parameters and implementations as [1]). Our Residual Units are the full pre-activation version (Fig. 4(e)). †: code/model available at https://github.com/facebook/fb.resnet.torch/tree/master/pretrained, using scale and aspect ratio augmentation in [20].
method augmentation train crop test crop top-1 top-5
ResNet-152, original Residual Unit [1] scale 224×\times224 224×\times224 23.0 6.7
ResNet-152, original Residual Unit [1] scale 224×\times224 320×\times320 21.3 5.5
ResNet-152, pre-act Residual Unit scale 224×\times224 320×\times320 21.1 5.5
ResNet-200, original Residual Unit [1] scale 224×\times224 320×\times320 21.8 6.0
ResNet-200, pre-act Residual Unit scale 224×\times224 320×\times320 20.7 5.3
ResNet-200, pre-act Residual Unit scale+asp ratio 224×\times224 320×\times320 20.1† 4.8†
Inception v3 [19] scale+asp ratio 299×\times299 299×\times299 21.2 5.6

6 Conclusions

Clean identity paths provide a useful architectural condition for deep residual learning. The derivations and ablations support pre-activation as an effective design, while the results remain specific to the investigated datasets, training schedules, and architectures.

Appendix: Implementation Details

CIFAR training uses flips and translations, momentum 0.9, weight decay 0.0001, and learning-rate drops at 32,000 and 48,000 iterations. A short 0.01 warm-up precedes 0.1 training; it is not necessary for the new unit. ImageNet uses batch size 256 and drops at 30 and 60 epochs. The first unit activates before splitting its paths; the final unit adds an activation before pooling. Bottleneck stage transitions use pre-activated projection shortcuts.

Concept demonstration

Open image ↗