Overview
At a glance
Dropout can regularize an LSTM without repeatedly corrupting its memory: apply it to connections between layers and to outputs, while leaving recurrent connections intact. The paper reports improved language modeling, speech-frame classification, translation, and image captioning. Benefits depend on the task and evaluation setting.
1 Introduction
Large recurrent networks overfit, but applying independent dropout noise throughout a recurrence can undermine stored information. This work restricts where dropout acts, allowing larger LSTMs to generalize. The authors provide their language-model implementation.
2 Related work
Earlier work studied deterministic approximations to dropout and warned that recurrent dynamics amplify noise. The authors acknowledge that Pham and colleagues independently proposed the same non-recurrent dropout scheme for handwriting recognition. Their contribution is its evaluation across several domains; claims should not be read as establishing that every form of recurrent dropout fails.
3 Regularizing RNNs with LSTM cells
A stacked LSTM propagates information both through time within a layer and upward between layers. The notation below distinguishes these two routes: subscripts index time, superscripts index layers, and the top layer predicts the next output.
3.1 Long-short term memory units
An LSTM maintains a cell state alongside its hidden output. Input, forget, and output gates determine what to write, retain, and expose; additive cell updates provide a route for long-range memory. Figure 1 and the equations retain the exact cell variant used here.
Equations

3.2 Regularization with Dropout
Randomly zero only non-recurrent activations. A signal may cross many time steps without encountering additional dropout; along the illustrated path, corruption occurs once per vertical transition, for L + 1 applications rather than growing with elapsed time. This provides regularization while protecting temporal storage, as Figures 2–3 illustrate.
Equation


4 Experiments
The experiments cover four tasks, with different metrics and training configurations; the result tables preserve those distinctions.
4.1 Language modeling
On the Penn Treebank, two-layer regularized LSTMs with 650 or 1,500 units outperform a tuned, smaller unregularized model: test perplexity falls from 114.5 to 82.7 or 78.4. Ensembles improve further, reaching 68.7 for 38 large models. Training carries hidden states between sequential minibatches, uses gradient clipping, and drops 50% or 65% of non-recurrent activations. Figure 4 preserves generated samples, which illustrate local fluency rather than coherent reasoning.
| Model | Validation set | Test set |
| A single model | ||
| Pascanu et al. 2013 | 107.5 | |
| Cheng et al. | 100.0 | |
| non-regularized LSTM | 120.7 | 114.5 |
| Medium regularized LSTM | 86.2 | 82.7 |
| Large regularized LSTM | 82.2 | 78.4 |
| Model averaging | ||
| Mikolov 2012 | 83.5 | |
| Cheng et al. | 80.6 | |
| 2 non-regularized LSTMs | 100.4 | 96.1 |
| 5 non-regularized LSTMs | 87.9 | 84.1 |
| 10 non-regularized LSTMs | 83.5 | 80.0 |
| 2 medium regularized LSTMs | 80.6 | 77.0 |
| 5 medium regularized LSTMs | 76.7 | 73.3 |
| 10 medium regularized LSTMs | 75.2 | 72.0 |
| 2 large regularized LSTMs | 76.9 | 73.6 |
| 10 large regularized LSTMs | 72.8 | 69.5 |
| 38 large regularized LSTMs | 71.9 | 68.7 |
| Model averaging with dynamic RNNs and n-gram models | ||
| Mikolov & Zweig 2012 | 72.9 | |
the meaning of life is that only if an end would be of the whole supplier. widespread rules are regarded as the companies of refuses to deliver. in balance of the nation ’s information and loan growth associated with the carrier thrifts are in the process of slowing the seed and commercial paper.
the meaning of life is nearly in the first several months before the government was addressing such a move as president and chief executive of the nation past from a national commitment to curb grounds. meanwhile the government invests overcapacity that criticism and in the outer reversal of small-town america.
4.2 Speech recognition
On an internal Google Icelandic speech dataset containing 93,000 utterances, dropout raises validation frame accuracy from 68.9% to 70.5% while lowering training accuracy. The experiment evaluates acoustic-state classification, not word error rate; a decoding and language-model evaluation would be needed to establish the latter.
4.3 Machine translation
A four-layer English-to-French LSTM with 1,000 units per layer uses a 12-hypothesis beam and 20% dropout. Test perplexity improves from 5.8 to 5.0 and BLEU from 25.9 to 29.03 on the selected WMT 2014 data. The comparison phrase-based system still scores higher, at 33.30 BLEU.
4.4 Image Caption Generation
For image captioning on MSCOCO, a frozen visual encoder feeds a single LSTM. Dropout improves a single model from 23.5 to 24.3 BLEU, nearly matching a ten-model unregularized ensemble at 24.4. Here its main practical benefit is reducing the need for an ensemble; the visual network itself is not trained with this dataset.
| Model | Test perplexity | Test BLEU score |
|---|---|---|
| Non-regularized model | 8.47 | 23.5 |
| Regularized model | 7.99 | 24.3 |
| 10 non-regularized models | 7.5 | 24.4 |
5 Conclusion
Separating spatial regularization from temporal state propagation makes dropout useful in these LSTM experiments. The evidence supports this particular recipe across the reported tasks, with the largest gains in language modeling and task-specific limits in the other evaluations.
6 Acknowledgments
The authors thank Tomas Mikolov for comments on the initial manuscript.