Literature
Display
← All papers

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau · Kyunghyun Cho · Yoshua Bengio

· ICLR 2015 ↗

Contents

Overview

At a glance

A fixed vector can bottleneck translation of a long sentence. This model instead keeps a representation for every source position and learns a soft alignment for each output word. Jointly trained attention improves English–French translation over a fixed-context encoder–decoder, especially on long sentences; rare words remain a major limitation.

1 Introduction

The encoder no longer has to compress the whole sentence into one unchanging context. A decoder searches source annotations anew at every step, selecting information relevant to its next prediction. Translation and alignment are learned together from sentence pairs without supplied word alignments.

2 Background: Neural Machine Translation

Neural translation models learn the conditional probability of a target sentence given a source and then approximately search for its most probable translation. Earlier neural systems either supplied features to phrase-based pipelines or used a fixed-vector encoder–decoder.

2.1 RNN Encoder–Decoder

The baseline reads source words recurrently, summarizes them in one context vector, and predicts target words autoregressively. A probability chain rule combines these predictions into a sequence likelihood. The proposed model changes access to source information while retaining this decoder framework.

Equations 1–3
ht=f⁡(xt,ht−1)\displaystyle h_{t}=f\left(x_{t},h_{t-1}\right) (1)
c=q⁡({h1,⋯,hTx}),\displaystyle c=q\left(\left\{h_{1},\cdots,h_{T_{x}}\right\}\right),
p⁡(𝐲)=∏t=1Tp⁡(yt∣{y1,⋯,yt−1},c),\displaystyle p(\mathbf{y})=\prod_{t=1}^{T}p(y_{t}\mid\left\{y_{1},\cdots,y_{t-1}\right\},c), (2)
p⁡(yt∣{y1,⋯,yt−1},c)=g⁡(yt−1,st,c),\displaystyle p(y_{t}\mid\left\{y_{1},\cdots,y_{t-1}\right\},c)=g(y_{t-1},s_{t},c), (3)

3 Learning to Align and Translate

A bidirectional encoder constructs source annotations; an attention-equipped recurrent decoder retrieves a different context at each output step.

3.1 Decoder: General Description

For each target word, a small alignment network scores each source annotation against the previous decoder state. Softmax weights form a weighted sum of annotations, used by the decoder and output predictor. This differentiable soft alignment supports backpropagation through the whole system rather than selecting a discrete latent alignment.

Original figure
Figure 1: The graphical illustration of the proposed model trying to generate thett-th target wordyty_{t} given a source sentence(x1,x2,…,xT)(x_{1},x_{2},\dots,x_{T}).
Equations 4–6
p⁡(yi|y1,…,yi−1,𝐱)=g⁡(yi−1,si,ci),\displaystyle p(y_{i}|y_{1},\ldots,y_{i-1},\mathbf{x})=g(y_{i-1},s_{i},c_{i}), (4)
si=f⁡(si−1,yi−1,ci).s_{i}=f(s_{i-1},y_{i-1},c_{i}).
ci=∑j=1Txαi​j​hj.\displaystyle c_{i}=\sum_{j=1}^{T_{x}}\alpha_{ij}h_{j}. (5)
αi​j=exp⁡(ei​j)∑k=1Txexp⁡(ei​k),\displaystyle\alpha_{ij}=\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{T_{x}}\exp\left(e_{ik}\right)}, (6)
ei​j=a⁡(si−1,hj)e_{ij}=a(s_{i-1},h_{j})

3.2 Encoder: Bidirectional RNN for Annotating Sequences

Forward and backward RNN states are concatenated at each source position. Every annotation therefore includes context from both directions while emphasizing its local word. Attention can selectively access this sequence of representations instead of a single terminal state.

4 Experiment Settings

The controlled comparison trains fixed-context and attention models on the same selected English–French parallel corpus. Figure 2 breaks performance down by sentence length, the bottleneck the new architecture is designed to address.

Original figure
Figure 2: The BLEU scores of the generated translations on the test set with respect to the lengths of the sentences. The results are on the full test set which includes sentences having unknown words to the models.

4.1 Dataset

The selected training corpus contains 348 million words, drawn from the WMT 2014 bilingual data. Each language keeps only its 30,000 most frequent words; others map to an unknown token. Evaluation uses 3,003 newstest2014 sentences, with no added monolingual training data. Figure 3 preserves example learned alignments.

Refer to caption
Refer to caption

(a)

(b)

Refer to caption
Refer to caption

(c)

(d)

Figure 3: Four sample alignments found by RNNsearch-50. The x-axis and y-axis of each plot correspond to the words in the source sentence (English) and the generated translation (French), respectively. Each pixel shows the weightαi​j\alpha_{ij} of the annotation of thejj-th source word for the ii-th target word (see Eq. (6)), in grayscale (00: black,11: white). (a) an arbitrary sentence. (b–d) three randomly selected samples among the sentences without any unknown words and of length between 10 and 20 words from the test set.

4.2 Models

Both systems use gated recurrent units, a maxout output network, Adadelta, and minibatches of 80 sentences. Models train on sentences of at most 30 or 50 words; the attention encoder has 1,000 units in each direction and the decoder 1,000 units. Beam search approximates the best translation.

5 Results

5.1 Quantitative Results

Attention improves BLEU from 17.82 to 26.75 for the 50-word models, reaching 28.45 with longer training. Moses remains better on the full test set at 33.30. On the subset without unknown words, the longer-trained attention model reaches 36.15 versus Moses’s 35.63; this restricted comparison does not establish superiority on unrestricted translation. Attention also reduces degradation with sentence length.

Model All No UNK∘
RNNencdec-30 13.93 24.19
RNNsearch-30 21.50 31.44
RNNencdec-50 17.82 26.71
RNNsearch-50 26.75 34.16
RNNsearch-50⋆ 28.45 36.15
Moses 33.30 35.63
Table 1:BLEU scores of the trained models computed on the test set. The second and third columns show respectively the scores on all the sentences and, on the sentences without any unknown word in themselves and in the reference translations. Note that RNNsearch-50⋆ was trained much longer until the performance on the development set stopped improving. (∘\circ) We disallowed the models to generate [UNK] tokens when only the sentences having no unknown words were evaluated (last column).

5.2 Qualitative Analysis

5.2.1 Alignment

The selected alignment maps show plausible word correspondences and reordering, including one-to-many relationships. Their flexibility distinguishes soft attention from a monotonic alignment rule. These are qualitative examples rather than supervised alignment-accuracy measurements.

5.2.2 Long Sentences

Selected long-sentence examples show the fixed-context decoder losing meaning after roughly thirty words, while the attention model retains more of the source. Appendix C preserves the complete source, reference, and output comparisons. The examples support the quantitative length analysis but are not a separate representative test.

6 Related Work

6.1 Learning to Align

Earlier handwriting-generation attention moved monotonically along its input. Translation requires flexible reordering, so this model scores every source position for every target word. That flexibility incurs a cost proportional to the product of source and target lengths.

6.2 Neural Networks for Machine Translation

Neural language and phrase models had already improved statistical translation pipelines. This work instead directly generates translations with a single jointly trained neural system; the distinction concerns the overall modeling pipeline, not the first use of neural networks in translation.

7 Conclusion

Soft alignment relieves the fixed-context bottleneck and makes long-sentence translation more robust in this experiment. The remaining vocabulary restriction and gap to phrase-based translation on the full test set motivate better treatment of rare and unknown words.

Acknowledgments

The authors acknowledge Theano, research and computing support, and collaborators who provided feedback.

Appendix A Model Architecture

A.1 Architectural Choices

The general attention framework permits different recurrent cells and scoring functions. The following specifications describe the particular gated recurrent implementation used in the experiments.

A.1.1 Recurrent Neural Network

Update gates interpolate between stored and proposed states; reset gates control how the old state contributes to a proposal. Context enters the decoder’s gated transitions. A maxout hidden layer and softmax produce the next-word distribution. The full update equations remain available below.

Equations
si=f⁡(si−1,yi−1,ci)=(1−zi)∘si−1+zi∘s~i,\displaystyle s_{i}=f(s_{i-1},y_{i-1},c_{i})=(1-z_{i})\circ s_{i-1}+z_{i}\circ\tilde{s}_{i},
s~i=tanh⁡(W​e​(yi−1)+U⁡[ri∘si−1]+C​ci),\displaystyle\tilde{s}_{i}=\tanh\left(We(y_{i-1})+U\left[r_{i}\circ s_{i-1}\right]+Cc_{i}\right),
zi\displaystyle z_{i} =σ⁡(Wz​e​(yi−1)+Uz​si−1+Cz​ci),\displaystyle=\sigma\left(W_{z}e(y_{i-1})+U_{z}s_{i-1}+C_{z}c_{i}\right),
ri\displaystyle r_{i} =σ⁡(Wr​e​(yi−1)+Ur​si−1+Cr​ci),\displaystyle=\sigma\left(W_{r}e(y_{i-1})+U_{r}s_{i-1}+C_{r}c_{i}\right),

A.1.2 Alignment Model

A single-hidden-layer alignment network combines the previous decoder state with each source annotation. The annotation-dependent projection can be computed once per input sentence, reducing repeated work across target positions.

Equation
a⁡(si−1,hj)=va⊤​tanh⁡(Wa​si−1+Ua​hj),\displaystyle a(s_{i-1},h_{j})=v_{a}^{\top}\tanh\left(W_{a}s_{i-1}+U_{a}h_{j}\right),

A.2 Detailed Description of the Model

A.2.1 Encoder

The appendix specifies the forward and backward gated recurrences, their dimensions, and shared source embeddings. The two directions have separate recurrent parameters; their concatenated states are the annotations supplied to attention.

Equations
𝐱=(x1,…,xTx),xi∈ℝKx\mathbf{x}=(x_{1},\ldots,x_{T_{x}}),\mbox{ }x_{i}\in\mathbb{R}^{K_{x}}
𝐲=(y1,…,yTy),yi∈ℝKy,\mathbf{y}=(y_{1},\ldots,y_{T_{y}}),\mbox{ }y_{i}\in\mathbb{R}^{K_{y}},
h→i=\displaystyle\overrightarrow{h}_{i}= {(1−z→i)∘h→i−1+z→i∘h¯→i, if​i>00, if​i=0\displaystyle\begin{cases}(1-\overrightarrow{z}_{i})\circ\overrightarrow{h}_{i-1}+\overrightarrow{z}_{i}\circ\overrightarrow{\underline{h}}_{i}&\mbox{, if }i>0\\ 0&\mbox{, if }i=0\end{cases}
h¯→i=\displaystyle\overrightarrow{\underline{h}}_{i}= tanh⁡(W→​E¯​xi+U→​[r→i∘h→i−1])\displaystyle\tanh\left(\overrightarrow{W}\overline{E}x_{i}+\overrightarrow{U}\left[\overrightarrow{r}_{i}\circ\overrightarrow{h}_{i-1}\right]\right)
z→i=\displaystyle\overrightarrow{z}_{i}= σ⁡(W→z​E¯​xi+U→z​h→i−1)\displaystyle\sigma\left(\overrightarrow{W}_{z}\overline{E}x_{i}+\overrightarrow{U}_{z}\overrightarrow{h}_{i-1}\right)
r→i=\displaystyle\overrightarrow{r}_{i}= σ⁡(W→r​E¯​xi+U→r​h→i−1).\displaystyle\sigma\left(\overrightarrow{W}_{r}\overline{E}x_{i}+\overrightarrow{U}_{r}\overrightarrow{h}_{i-1}\right).
hi=[h→ih←i]\displaystyle h_{i}=\left[\begin{array}[]{c}\overrightarrow{h}_{i}\\ \overleftarrow{h}_{i}\end{array}\right]

A.2.2 Decoder

The decoder is initialized from the backward encoder state and updates using the prior word and current attention context. The appendix retains all gate, alignment, and maxout-output formulas. Replacing every context with the terminal forward encoder state recovers a fixed-context encoder–decoder.

Equations
si=\displaystyle s_{i}= (1−zi)∘si−1+zi∘s~i,\displaystyle(1-z_{i})\circ s_{i-1}+z_{i}\circ\tilde{s}_{i},
s~i=\displaystyle\tilde{s}_{i}= tanh⁡(W​E​yi−1+U⁡[ri∘si−1]+C​ci)\displaystyle\tanh\left(WEy_{i-1}+U\left[r_{i}\circ s_{i-1}\right]+Cc_{i}\right)
zi=\displaystyle z_{i}= σ⁡(Wz​E​yi−1+Uz​si−1+Cz​ci)\displaystyle\sigma\left(W_{z}Ey_{i-1}+U_{z}s_{i-1}+C_{z}c_{i}\right)
ri=\displaystyle r_{i}= σ⁡(Wr​E​yi−1+Ur​si−1+Cr​ci)\displaystyle\sigma\left(W_{r}Ey_{i-1}+U_{r}s_{i-1}+C_{r}c_{i}\right)
ci=\displaystyle c_{i}= ∑j=1Txαi​j​hj,\displaystyle\sum_{j=1}^{T_{x}}\alpha_{ij}h_{j},
αi​j=\displaystyle\alpha_{ij}= exp⁡(ei​j)∑k=1Txexp⁡(ei​k)\displaystyle\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{T_{x}}\exp\left(e_{ik}\right)}
ei​j=\displaystyle e_{ij}= va⊤​tanh⁡(Wa​si−1+Ua​hj),\displaystyle v_{a}^{\top}\tanh\left(W_{a}s_{i-1}+U_{a}h_{j}\right),
p⁡(yi|si,yi−1,ci)∝\displaystyle p(y_{i}|s_{i},y_{i-1},c_{i})\propto exp⁡(yi⊤​Wo​ti),\displaystyle\exp\left(y_{i}^{\top}W_{o}t_{i}\right),
ti=\displaystyle t_{i}= [max⁡{t~i,2​j−1,t~i,2​j}]j=1,…,l⊤\displaystyle\left[\max\left\{\tilde{t}_{i,2j-1},\tilde{t}_{i,2j}\right\}\right]_{j=1,\ldots,l}^{\top}
t~i=\displaystyle\tilde{t}_{i}= Uo​si−1+Vo​E​yi−1+Co​ci.\displaystyle U_{o}s_{i-1}+V_{o}Ey_{i-1}+C_{o}c_{i}.

A.2.3 Model Size

Experiments use hidden dimension 1,000, embeddings of dimension 620, maxout dimension 500, and 1,000 alignment hidden units. Table 2 records updates, epochs, hardware, wall-clock time, and training/development likelihood for each model; sentence-length differences affect these likelihood averages.

Model Updates (×105\times 10^{5}) Epochs Hours GPU Train NLL Dev. NLL
RNNenc-30 8.46 6.4 109 TITAN BLACK 28.1 53.0
RNNenc-50 6.00 4.5 108 Quadro K-6000 44.0 43.6
RNNsearch-30 4.71 3.6 113 TITAN BLACK 26.7 47.2
RNNsearch-50 2.88 2.2 111 Quadro K-6000 40.7 38.1
RNNsearch-50⋆ 6.67 5.0 252 Quadro K-6000 36.7 35.2
Table 2:Learning statistics and relevant information. Each update corresponds to updating the parameters once using a single minibatch. One epoch is one pass through the training set. NLL is the average conditional log-probabilities of the sentences in either the training set or the development set. Note that the lengths of the sentences differ.

Appendix B Training Procedure

B.1 Parameter Initialization

Recurrent matrices start orthogonal, most other weights use small Gaussian initializations, and biases start at zero. The alignment parameters use their separately specified initialization scales.

B.2 Training

Adadelta adapts learning rates, gradients are clipped to norm one, and minibatches contain 80 pairs. Sorting blocks of 1,600 sentence pairs by length before splitting them into batches reduces wasted padding computation.

Appendix C Translations of Long Sentences

The original comparison table preserves three long English sentences, reference French translations, both neural outputs, and Google Translate outputs collected on 27 August 2014. These examples illustrate differing omissions and wording; they are not an evaluation of present-day translation products.

Source An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
Reference Le privilège d’admission est le droit d’un médecin, en vertu de son statut de membre soignant d’un hôpital, d’admettre un patient dans un hôpital ou un centre médical afin d’y délivrer un diagnostic ou un traitement.
RNNenc-50 Un privilège d’admission est le droit d’un médecin de reconnaître un patient à l’hôpital ou un centre médical d’un diagnostic ou de prendre un diagnostic en fonction de son état de santé.
RNNsearch-50 Un privilège d’admission est le droit d’un médecin d’admettre un patient à un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, selon son statut de travailleur des soins de santé à l’hôpital.
GoogleTranslate Un privilège admettre est le droit d’un médecin d’admettre un patient dans un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, fondée sur sa situation en tant que travailleur de soins de santé dans un hôpital.
Source This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added.
Reference Ce type d’expérience entre dans le cadre des efforts de Disney pour ”étendre la durée de vie de ses séries et construire de nouvelles relations avec son public grâce à des plateformes numériques qui sont de plus en plus importantes”, a-t-il ajouté.
RNNenc-50 Ce type d’expérience fait partie des initiatives du Disney pour ”prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes.
RNNsearch-50 Ce genre d’expérience fait partie des efforts de Disney pour ”prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes”, a-t-il ajouté.
GoogleTranslate Ce genre d’expérience fait partie des efforts de Disney à “étendre la durée de vie de sa série et construire de nouvelles relations avec le public par le biais des plates-formes numériques qui deviennent de plus en plus important”, at-il ajouté.
Source In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might constitute a ”reasonable motive” that could lead to criminal charges being brought against the mayor.
Reference En conférence de presse, jeudi, M. Blair a affirmé qu’il n’y avait rien dans cette vidéo qui puisse constituer des ”motifs raisonnables” pouvant mener au dépôt d’une accusation criminelle contre le maire.
RNNenc-50 Lors de la conférence de presse de jeudi, M. Blair a dit qu’il n’y avait rien dans cette vidéo qui pourrait constituer une ”motivation raisonnable” pouvant entraîner des accusations criminelles portées contre le maire.
RNNsearch-50 Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vidéo qui pourrait constituer un ”motif raisonnable” qui pourrait conduire à des accusations criminelles contre le maire.
GoogleTranslate Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vido qui pourrait constituer un ”motif raisonnable” qui pourrait mener à des accusations criminelles portes contre le maire.
Table 3:The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences (30 words or more) selected from the test set. For each source sentence, we also show the gold-standard translation. The translations by Google Translate were made on 27 August 2014.
Concept demonstration

Open image ↗