Literature
Display
← All papers

Recurrent Neural Network Regularization

Wojciech Zaremba · Ilya Sutskever · Oriol Vinyals

Preprint posted · arXiv (preprint) ↗

Contents

Overview

At a glance

Dropout can regularize an LSTM without repeatedly corrupting its memory: apply it to connections between layers and to outputs, while leaving recurrent connections intact. The paper reports improved language modeling, speech-frame classification, translation, and image captioning. Benefits depend on the task and evaluation setting.

1 Introduction

Large recurrent networks overfit, but applying independent dropout noise throughout a recurrence can undermine stored information. This work restricts where dropout acts, allowing larger LSTMs to generalize. The authors provide their language-model implementation.

2 Related work

Earlier work studied deterministic approximations to dropout and warned that recurrent dynamics amplify noise. The authors acknowledge that Pham and colleagues independently proposed the same non-recurrent dropout scheme for handwriting recognition. Their contribution is its evaluation across several domains; claims should not be read as establishing that every form of recurrent dropout fails.

3 Regularizing RNNs with LSTM cells

A stacked LSTM propagates information both through time within a layer and upward between layers. The notation below distinguishes these two routes: subscripts index time, superscripts index layers, and the top layer predicts the next output.

3.1 Long-short term memory units

An LSTM maintains a cell state alongside its hidden output. Input, forget, and output gates determine what to write, retain, and expose; additive cell updates provide a route for long-range memory. Figure 1 and the equations retain the exact cell variant used here.

Equations
RNN:htl−1,ht−1l→htl\displaystyle\text{RNN}:h^{l-1}_{t},h^{l}_{t-1}\rightarrow h^{l}_{t}
htl=f⁡(Tn,n​htl−1+Tn,n​ht−1l)​, wheref∈{sigm,tanh}\displaystyle h^{l}_{t}=f(T_{n,n}h^{l-1}_{t}+T_{n,n}h^{l}_{t-1})\text{, where $f\in\{\mathrm{sigm},\tanh\}$ }
LSTM:htl−1,ht−1l,ct−1l→htl,ctl\displaystyle\text{LSTM}:h^{l-1}_{t},h^{l}_{t-1},c^{l}_{t-1}\rightarrow h^{l}_{t},c^{l}_{t}
(ifog)=(sigmsigmsigmtanh)​T2​n,4​n​(htl−1ht−1l)\displaystyle\begin{pmatrix}i\\ f\\ o\\ g\end{pmatrix}=\begin{pmatrix}\mathrm{sigm}\\ \mathrm{sigm}\\ \mathrm{sigm}\\ \tanh\end{pmatrix}T_{2n,4n}\begin{pmatrix}h^{l-1}_{t}\\ h^{l}_{t-1}\end{pmatrix}
ctl=f⊙ct−1l+i⊙g\displaystyle c^{l}_{t}=f\odot c^{l}_{t-1}+i\odot g
htl=o⊙tanh⁡(ctl)\displaystyle h^{l}_{t}=o\odot\tanh(c^{l}_{t})
Figure 1: A graphical representation of LSTM memory cells used in this paper (there are minor differences in comparison to Graves 2013 ).
Figure 1:A graphical representation of LSTM memory cells used in this paper (there are minor differences in comparison toGraves 2013).

3.2 Regularization with Dropout

Randomly zero only non-recurrent activations. A signal may cross many time steps without encountering additional dropout; along the illustrated path, corruption occurs once per vertical transition, for L + 1 applications rather than growing with elapsed time. This provides regularization while protecting temporal storage, as Figures 2–3 illustrate.

Equation
(ifog)=(sigmsigmsigmtanh)​T2​n,4​n​(𝐃⁡(htl−1)ht−1l)\displaystyle\begin{pmatrix}i\\ f\\ o\\ g\end{pmatrix}=\begin{pmatrix}\mathrm{sigm}\\ \mathrm{sigm}\\ \mathrm{sigm}\\ \tanh\end{pmatrix}T_{2n,4n}\begin{pmatrix}{\bf D}(h^{l-1}_{t})\\ h^{l}_{t-1}\end{pmatrix}
ctl=f⊙ct−1l+i⊙g\displaystyle c^{l}_{t}=f\odot c^{l}_{t-1}+i\odot g
htl=o⊙tanh⁡(ctl)\displaystyle h^{l}_{t}=o\odot\tanh(c^{l}_{t})
Figure 2: Regularized multilayer RNN. The dashed arrows indicate connections where dropout is applied, and
the solid lines indicate connections where dropout is not applied.
Figure 2:Regularized multilayer RNN. The dashed arrows indicate connections where dropout is applied, and the solid lines indicate connections where dropout is not applied.
Figure 3: The thick line shows a typical path of information flow in the LSTM. The
information is affected by dropout L + 1 L+1 times, where L L is
depth of network.
Figure 3:The thick line shows a typical path of information flow in the LSTM. The information is affected by dropoutL+1L+1 times, whereLL is depth of network.

4 Experiments

The experiments cover four tasks, with different metrics and training configurations; the result tables preserve those distinctions.

4.1 Language modeling

On the Penn Treebank, two-layer regularized LSTMs with 650 or 1,500 units outperform a tuned, smaller unregularized model: test perplexity falls from 114.5 to 82.7 or 78.4. Ensembles improve further, reaching 68.7 for 38 large models. Training carries hidden states between sequential minibatches, uses gradient clipping, and drops 50% or 65% of non-recurrent activations. Figure 4 preserves generated samples, which illustrate local fluency rather than coherent reasoning.

Model Validation set Test set
A single model
Pascanu et al. 2013 107.5
Cheng et al. 100.0
non-regularized LSTM 120.7 114.5
Medium regularized LSTM 86.2 82.7
Large regularized LSTM 82.2 78.4
Model averaging
Mikolov 2012 83.5
Cheng et al. 80.6
2 non-regularized LSTMs 100.4 96.1
5 non-regularized LSTMs 87.9 84.1
10 non-regularized LSTMs 83.5 80.0
2 medium regularized LSTMs 80.6 77.0
5 medium regularized LSTMs 76.7 73.3
10 medium regularized LSTMs 75.2 72.0
2 large regularized LSTMs 76.9 73.6
10 large regularized LSTMs 72.8 69.5
38 large regularized LSTMs 71.9 68.7
Model averaging with dynamic RNNs and n-gram models
Mikolov & Zweig 2012 72.9
Table 1:Word-level perplexity on the Penn Tree Bank dataset.


the meaning of life is that only if an end would be of the whole supplier. widespread rules are regarded as the companies of refuses to deliver. in balance of the nation ’s information and loan growth associated with the carrier thrifts are in the process of slowing the seed and commercial paper.


the meaning of life is nearly in the first several months before the government was addressing such a move as president and chief executive of the nation past from a national commitment to curb grounds. meanwhile the government invests overcapacity that criticism and in the outer reversal of small-town america.


Figure 4:Some interesting samples drawn from a large regularized model conditioned on “The meaning of life is”. We have removed “unk”, “N”, “$” from the set of permissible words.

4.2 Speech recognition

On an internal Google Icelandic speech dataset containing 93,000 utterances, dropout raises validation frame accuracy from 68.9% to 70.5% while lowering training accuracy. The experiment evaluates acoustic-state classification, not word error rate; a decoding and language-model evaluation would be needed to establish the latter.

Model Training set Validation set
Non-regularized LSTM 71.6 68.9
Regularized LSTM 69.4 70.5
Table 2:Frame-level accuracy on the Icelandic Speech Dataset. The training set has 93k utterances.

4.3 Machine translation

A four-layer English-to-French LSTM with 1,000 units per layer uses a 12-hypothesis beam and 20% dropout. Test perplexity improves from 5.8 to 5.0 and BLEU from 25.9 to 29.03 on the selected WMT 2014 data. The comparison phrase-based system still scores higher, at 33.30 BLEU.

Model Test perplexity Test BLEU score
Non-regularized LSTM 5.8 25.9
Regularized LSTM 5.0 29.03
LIUM system 33.30
Table 3:Results on the English to French translation task.

4.4 Image Caption Generation

For image captioning on MSCOCO, a frozen visual encoder feeds a single LSTM. Dropout improves a single model from 23.5 to 24.3 BLEU, nearly matching a ten-model unregularized ensemble at 24.4. Here its main practical benefit is reducing the need for an ensemble; the visual network itself is not trained with this dataset.

Model Test perplexity Test BLEU score
Non-regularized model 8.47 23.5
Regularized model 7.99 24.3
10 non-regularized models 7.5 24.4
Table 4:Results on the image caption generation task.

5 Conclusion

Separating spatial regularization from temporal state propagation makes dropout useful in these LSTM experiments. The evidence supports this particular recipe across the reported tasks, with the largest gains in language modeling and task-specific limits in the other evaluations.

6 Acknowledgments

The authors thank Tomas Mikolov for comments on the initial manuscript.

Concept demonstration

Open image ↗