Literature
Display
← All papers

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky · Ilya Sutskever · Geoffrey E. Hinton

Published · NeurIPS 2012 ↗

Computer Vision4 min text · CondensedOriginal paper ↗
Contents

At a glance

A large convolutional network trained on labeled ImageNet images substantially improves object recognition. Five convolutional layers and three fully connected layers combine ReLU activation, two-accelerator training, data augmentation, and dropout. The 2012 winning result is a seven-model ensemble, partly pretrained on more ImageNet categories: 15.3% top-five test error versus 26.2% for the runner-up.

1 Introduction

Large image datasets make high-capacity supervised models practical, while convolution supplies useful assumptions about local structure and shared visual patterns. Efficient accelerator implementation makes training feasible. The paper’s achievement combines architecture, compute, and regularization; no single ingredient alone explains the result.

2 The dataset

The competition subset has about 1.2 million training images, 50,000 validation images, and 150,000 test images across 1,000 categories. Images are resized and center-cropped to 256 × 256, then mean-centered. Top-one error checks the highest-ranked class; top-five error checks whether the correct class appears anywhere among five predictions.

3 The architecture

The network has eight learned layers, approximately 60 million parameters, and 650,000 neurons. The following components improve training speed or generalization within this specific design.

3.1 ReLU nonlinearity

Rectified linear units use the positive part of their input and avoid saturation on that branch. In a four-layer CIFAR-10 comparison, they reach 25% training error about six times faster than tanh. Faster optimization enables practical experiments with the much larger model. See the earlier rectified-unit work in [20].

Figure 1. ReLU versus tanh training convergence on CIFAR-10; the solid ReLU curve reaches 25% training error about six times faster.
Figure 1. ReLU versus tanh training convergence on CIFAR-10; the solid ReLU curve reaches 25% training error about six times faster.

3.2 Training on multiple accelerators

The model is split across two accelerators because one has only three gigabytes of memory. Most groups communicate locally, with cross-device connections in selected layers. The larger two-device architecture improves accuracy while controlling communication cost; the comparison is not a pure test of parallel execution at fixed architecture.

3.3 Local response normalization

Local response normalization divides each activation by a term depending on nearby channel activations at the same spatial location. It encourages competition between large responses and improves this model’s reported errors. The hyperparameters are selected on validation data; the preserved equation specifies the operation.

Equation Local response normalization
Local response normalizationLocal response normalization

3.4 Overlapping pooling

Overlapping max pooling uses 3 × 3 windows at stride two. Compared with nonoverlapping pooling of equal output dimensions, it slightly improves top-one and top-five error and appears to reduce overfitting.

3.5 Overall architecture

The convolutional widths are 96, 256, 384, 384, and 256 filters, followed by two 4,096-unit hidden layers and a 1,000-way softmax. ReLU follows learned hidden layers. Normalization follows the first two convolutions, and pooling follows those normalized outputs and the fifth convolution. The original architecture figure preserves spatial sizes and the two-device connectivity.

Figure 2. The complete eight-layer architecture, with the two accelerator partitions and cross-partition connections.
Figure 2. The complete eight-layer architecture, with the two accelerator partitions and cross-partition connections.

4 Reducing overfitting

Even 1.2 million labeled images do not eliminate overfitting in a 60-million-parameter model. Image augmentation and dropout are the main countermeasures.

4.1 Data augmentation

Training samples random 224 × 224 crops and horizontal reflections. Evaluation averages ten crop predictions. Color augmentation adds randomly scaled principal components of training-set color variation, with one perturbation shared across each image. These transformations preserve labels while exposing variation cheaply during training.

Equation Principal-component color perturbation
Principal-component color perturbationPrincipal-component color perturbation

4.2 Dropout

Dropout independently suppresses half the hidden activations in the first two fully connected layers during training. At evaluation all units are used with scaled outputs. It reduces co-adaptation and substantial overfitting, at roughly twice the iterations needed to converge. The technique is introduced in [10].

5 Details of learning

Training uses SGD with batch size 128, momentum 0.9, and weight decay 0.0005. Weights begin as small Gaussian samples; selected biases start at one to activate rectified units. The learning rate begins at 0.01 and is manually reduced when validation error stops improving. About 90 epochs take five to six days on two contemporary accelerators.

Figure 3. All 96 learned first-layer 11 × 11 × 3 filters; the upper and lower halves come from different accelerators.
Figure 3. All 96 learned first-layer 11 × 11 × 3 filters; the upper and lower halves come from different accelerators.
Equation Momentum and weight-decay update
Momentum and weight-decay updateMomentum and weight-decay update

6 Results

On ImageNet 2010, the network achieves 37.5% top-one and 17.0% top-five test error with crop averaging. For 2012, one model has 18.2% top-five validation error; five-model averaging reaches 16.4%. Combining these with two models pretrained on the larger 2011 ImageNet release gives the winning 15.3% test error. The tables preserve the distinctions between single models, ensembles, pretraining, validation, and test.

Table 1. ImageNet 2010 top-one and top-five test error; the convolutional network reaches 37.5% and 17.0%.
Table 1. ImageNet 2010 top-one and top-five test error; the convolutional network reaches 37.5% and 17.0%.
Table 2. ImageNet 2012 validation and test errors. Asterisks indicate pretraining on the larger ImageNet Fall 2011 release.
Table 2. ImageNet 2012 validation and test errors. Asterisks indicate pretraining on the larger ImageNet Fall 2011 release.

6.1 Qualitative evaluations

First-layer filters learn oriented patterns and color-sensitive features, with different specializations across the two device groups. Nearest neighbors in the last hidden layer share semantic content despite different pixel layouts. Prediction examples also reveal ambiguity about which object a photograph is intended to label; these illustrations are qualitative, not a separate robustness benchmark.

Figure 4. Left: eight test images and predicted labels. Right: five test queries and their six nearest training images in the final hidden representation.
Figure 4. Left: eight test images and predicted labels. Right: five test queries and their six nearest training images in the final hidden representation.

7 Discussion

Removing any convolutional layer hurts performance, supporting the importance of depth within this architecture. The work shows that supervised learning at sufficient scale can strongly outperform the contemporary alternatives. Larger models, more data, and video are proposed directions; the experiment does not establish unrestricted generalization beyond ImageNet.

Supplement: learned feature maps

The original supplement supplies feature-map sheets for five convolutional layers and three pooling layers. The leftmost column contains random test images; the remaining columns show a random subset of feature activation maps in the named layer. All eight supplied sheets are retained.

Supplement: learned feature maps: conv1.
Supplement: learned feature maps: conv1.
Supplement: learned feature maps: conv2.
Supplement: learned feature maps: conv2.
Supplement: learned feature maps: conv3.
Supplement: learned feature maps: conv3.
Supplement: learned feature maps: conv4.
Supplement: learned feature maps: conv4.
Supplement: learned feature maps: conv5.
Supplement: learned feature maps: conv5.
Supplement: learned feature maps: pool1.
Supplement: learned feature maps: pool1.
Supplement: learned feature maps: pool2.
Supplement: learned feature maps: pool2.
Supplement: learned feature maps: pool3.
Supplement: learned feature maps: pool3.

Supplement: classification predictions

All eleven original prediction sheets are retained. Each image is accompanied by the model’s five most probable labels; a red bar marks the correct category when it appears among those five. The gallery includes successful, ambiguous, and failed examples.

Supplement: classification predictions: 1.
Supplement: classification predictions: 1.
Supplement: classification predictions: 10.
Supplement: classification predictions: 10.
Supplement: classification predictions: 11.
Supplement: classification predictions: 11.
Supplement: classification predictions: 2.
Supplement: classification predictions: 2.
Supplement: classification predictions: 3.
Supplement: classification predictions: 3.
Supplement: classification predictions: 4.
Supplement: classification predictions: 4.
Supplement: classification predictions: 5.
Supplement: classification predictions: 5.
Supplement: classification predictions: 6.
Supplement: classification predictions: 6.
Supplement: classification predictions: 7.
Supplement: classification predictions: 7.
Supplement: classification predictions: 8.
Supplement: classification predictions: 8.
Supplement: classification predictions: 9.
Supplement: classification predictions: 9.

Supplement: nearest-neighbor retrieval

All eight original retrieval sheets are retained. Each first-column test image is followed by the closest training examples under Euclidean distance in the final hidden representation. This displays semantic structure learned from classification labels, without claiming a separately evaluated retrieval system.

Supplement: nearest-neighbor retrieval: 1.
Supplement: nearest-neighbor retrieval: 1.
Supplement: nearest-neighbor retrieval: 2.
Supplement: nearest-neighbor retrieval: 2.
Supplement: nearest-neighbor retrieval: 3.
Supplement: nearest-neighbor retrieval: 3.
Supplement: nearest-neighbor retrieval: 4.
Supplement: nearest-neighbor retrieval: 4.
Supplement: nearest-neighbor retrieval: 5.
Supplement: nearest-neighbor retrieval: 5.
Supplement: nearest-neighbor retrieval: 6.
Supplement: nearest-neighbor retrieval: 6.
Supplement: nearest-neighbor retrieval: 7.
Supplement: nearest-neighbor retrieval: 7.
Supplement: nearest-neighbor retrieval: 8.
Supplement: nearest-neighbor retrieval: 8.
Concept demonstration

Open image ↗