At a glance
Identity shortcuts work best when the addition itself is also followed by an identity mapping. Moving batch normalization and ReLU before the residual weights creates this clean path. Controlled ablations explain why it eases optimization; pre-activation networks also generalize better, including a 1,001-layer CIFAR model and a 200-layer ImageNet model.
1 Introduction
The original ResNet applies ReLU after combining the shortcut and residual branch. This paper separates two questions: whether the shortcut should remain an identity, and whether the after-addition activation should also be an identity. Both theory and ablations favor an uninterrupted additive path. The residual branch retains nonlinear processing; the whole network does not become linear. See the original formulation in [1].
Equation
2 Analysis of Deep Residual Networks
When both relevant mappings are identities, a deeper representation equals a shallower representation plus the sum of intervening residual outputs. Backpropagation likewise contains a direct additive gradient term alongside the residual derivatives. The argument applies within constant-dimensional stages; the few projection/downsampling transitions are exceptions. It suggests a reliable information path, not an unconditional guarantee against every optimization failure.
Equations 1–5
| (1) | |||
| (2) |
| (3) |
| (4) |
| (5) |
Discussions
The direct forward and backward terms require both a clean skip connection and identity after-addition mapping. The following experiments change these components separately to test the analysis.
3 On the Importance of Identity Skip Connections
Scaling each shortcut multiplies the direct gradient path by the product of all intervening scale factors. Repeated factors below one can attenuate it; factors above one can amplify it. Gates and learned shortcut transforms introduce analogous products of derivatives. The extra representational flexibility can therefore make optimization harder.
Equations 6–8
| (6) |
| (7) |
| (8) |
| case | Fig. | on shortcut | on | error (%) | remark |
| original [1] | Fig. 2(a) | 1 | 1 | 6.61 | |
| constant scaling | Fig. 2(b) | 0 | 1 | fail | This is a plain net |
| 0.5 | 1 | fail | |||
| 0.5 | 0.5 | 12.35 | frozen gating | ||
| exclusive gating | Fig. 2(c) | fail | init =0 to | ||
| 8.70 | init =-6 | ||||
| 9.81 | init =-7 | ||||
| shortcut-only gating | Fig. 2(d) | 1 | 12.86 | init =0 | |
| 1 | 6.91 | init =-6 | |||
| 11 conv shortcut | Fig. 2(e) | 11 conv | 1 | 12.22 | |
| dropout shortcut | Fig. 2(f) | dropout 0.5 | 1 | fail |
3.1 Experiments on Skip Connections
On CIFAR-10, the original 110-layer network reaches 6.61% error. Constant scaling, dropout on shortcuts, and many gated variants fail or train poorly. Even learned 1 × 1 shortcuts reach 12.22%. Shortcut-only gates approach the baseline only when initialized close to identity. Results report median accuracy over five runs; these shortcut experiments still use the original post-addition ReLU.
3.2 Discussions
Several modified shortcuts can represent identity in principle yet have higher training error. Their weakness is consequently an optimization effect, not simply insufficient expressiveness. A direct information path is valuable even when a more complicated alternative includes it in its hypothesis class.
4 On the Usage of Activation Functions
Placing normalization after addition disrupts the shortcut. Moving only ReLU before addition restricts the residual output to nonnegative values. Full pre-activation instead places normalization and ReLU before each weight layer, allowing signed residual corrections and identity behavior after addition.
| case | Fig. | ResNet-110 | ResNet-164 |
|---|---|---|---|
| original Residual Unit [1] | Fig. 4(a) | 6.61 | 5.93 |
| BN after addition | Fig. 4(b) | 8.17 | 6.50 |
| ReLU before addition | Fig. 4(c) | 7.84 | 6.14 |
| ReLU-only pre-activation | Fig. 4(d) | 6.71 | 5.91 |
| full pre-activation | Fig. 4(e) | 6.37 | 5.46 |
4.1 Experiments on Activation
Full pre-activation improves CIFAR-10 error from 6.61% to 6.37% for ResNet-110 and from 5.93% to 5.46% for the bottleneck ResNet-164. It also helps one-layer skips and CIFAR-100 models. ReLU-only pre-activation gives little benefit; combining it with normalization matters. The 1,001-layer model improves particularly strongly.
Equation 9
| (9) |
| dataset | network | baseline unit | pre-activation unit |
|---|---|---|---|
| CIFAR-10 | ResNet-110 (1layer skip) | 9.90 | 8.91 |
| ResNet-110 | 6.61 | 6.37 | |
| ResNet-164 | 5.93 | 5.46 | |
| ResNet-1001 | 7.61 | 4.92 | |
| CIFAR-100 | ResNet-164 | 25.16 | 24.33 |
| ResNet-1001 | 27.82 | 22.71 |
4.2 Analysis
Two effects appear in the learning curves. Deep pre-activation models reduce training loss more readily because addition does not truncate the direct path. Shallower models sometimes finish with slightly higher training loss but lower test error, consistent with normalization regularizing every weight layer’s input. The authors distinguish this interpretation from the stronger optimization evidence at 1,001 layers.
5 Results
With standard flip/translation augmentation, ResNet-1001 reaches 4.92% CIFAR-10 error at batch size 128 and 4.62% at batch size 64; CIFAR-100 error is 22.71%. On ImageNet, the original 200-layer model overfits relative to 152 layers, while pre-activation reduces 320-pixel single-crop error to 20.7% top-one and 5.3% top-five. Stronger scale/aspect augmentation reaches 20.1% and 4.8%. Training cost still grows roughly linearly with depth.
| CIFAR-10 | error (%) |
|---|---|
| NIN [15] | 8.81 |
| DSN [16] | 8.22 |
| FitNet [17] | 8.39 |
| Highway [7] | 7.72 |
| All-CNN [14] | 7.25 |
| ELU [12] | 6.55 |
| FitResNet, LSUV [18] | 5.84 |
| ResNet-110 [1] (1.7M) | 6.61 |
| ResNet-1202 [1] (19.4M) | 7.93 |
| ResNet-164 [ours] (1.7M) | 5.46 |
| ResNet-1001 [ours] (10.2M) | 4.92 (4.890.14) |
| ResNet-1001 [ours] (10.2M)† | 4.62 (4.690.20) |
| method | augmentation | train crop | test crop | top-1 | top-5 |
|---|---|---|---|---|---|
| ResNet-152, original Residual Unit [1] | scale | 224224 | 224224 | 23.0 | 6.7 |
| ResNet-152, original Residual Unit [1] | scale | 224224 | 320320 | 21.3 | 5.5 |
| ResNet-152, pre-act Residual Unit | scale | 224224 | 320320 | 21.1 | 5.5 |
| ResNet-200, original Residual Unit [1] | scale | 224224 | 320320 | 21.8 | 6.0 |
| ResNet-200, pre-act Residual Unit | scale | 224224 | 320320 | 20.7 | 5.3 |
| ResNet-200, pre-act Residual Unit | scale+asp ratio | 224224 | 320320 | 20.1† | 4.8† |
| Inception v3 [19] | scale+asp ratio | 299299 | 299299 | 21.2 | 5.6 |
6 Conclusions
Clean identity paths provide a useful architectural condition for deep residual learning. The derivations and ablations support pre-activation as an effective design, while the results remain specific to the investigated datasets, training schedules, and architectures.
Appendix: Implementation Details
CIFAR training uses flips and translations, momentum 0.9, weight decay 0.0001, and learning-rate drops at 32,000 and 48,000 iterations. A short 0.01 warm-up precedes 0.1 training; it is not necessary for the new unit. ImageNet uses batch size 256 and drops at 30 and 60 epochs. The first unit activates before splitting its paths; the final unit adds an activation before pooling. Bottleneck stage transitions use pre-activated projection shortcuts.