Overview
At a glance
A fixed vector can bottleneck translation of a long sentence. This model instead keeps a representation for every source position and learns a soft alignment for each output word. Jointly trained attention improves English–French translation over a fixed-context encoder–decoder, especially on long sentences; rare words remain a major limitation.
1 Introduction
The encoder no longer has to compress the whole sentence into one unchanging context. A decoder searches source annotations anew at every step, selecting information relevant to its next prediction. Translation and alignment are learned together from sentence pairs without supplied word alignments.
2 Background: Neural Machine Translation
Neural translation models learn the conditional probability of a target sentence given a source and then approximately search for its most probable translation. Earlier neural systems either supplied features to phrase-based pipelines or used a fixed-vector encoder–decoder.
2.1 RNN Encoder–Decoder
The baseline reads source words recurrently, summarizes them in one context vector, and predicts target words autoregressively. A probability chain rule combines these predictions into a sequence likelihood. The proposed model changes access to source information while retaining this decoder framework.
Equations 1–3
| (1) |
| (2) |
| (3) |
3 Learning to Align and Translate
A bidirectional encoder constructs source annotations; an attention-equipped recurrent decoder retrieves a different context at each output step.
3.1 Decoder: General Description
For each target word, a small alignment network scores each source annotation against the previous decoder state. Softmax weights form a weighted sum of annotations, used by the decoder and output predictor. This differentiable soft alignment supports backpropagation through the whole system rather than selecting a discrete latent alignment.
Equations 4–6
| (4) |
| (5) |
| (6) |
3.2 Encoder: Bidirectional RNN for Annotating Sequences
Forward and backward RNN states are concatenated at each source position. Every annotation therefore includes context from both directions while emphasizing its local word. Attention can selectively access this sequence of representations instead of a single terminal state.
4 Experiment Settings
The controlled comparison trains fixed-context and attention models on the same selected English–French parallel corpus. Figure 2 breaks performance down by sentence length, the bottleneck the new architecture is designed to address.
4.1 Dataset
The selected training corpus contains 348 million words, drawn from the WMT 2014 bilingual data. Each language keeps only its 30,000 most frequent words; others map to an unknown token. Evaluation uses 3,003 newstest2014 sentences, with no added monolingual training data. Figure 3 preserves example learned alignments.
(a)
(b)
(c)
(d)
4.2 Models
Both systems use gated recurrent units, a maxout output network, Adadelta, and minibatches of 80 sentences. Models train on sentences of at most 30 or 50 words; the attention encoder has 1,000 units in each direction and the decoder 1,000 units. Beam search approximates the best translation.
5 Results
5.1 Quantitative Results
Attention improves BLEU from 17.82 to 26.75 for the 50-word models, reaching 28.45 with longer training. Moses remains better on the full test set at 33.30. On the subset without unknown words, the longer-trained attention model reaches 36.15 versus Moses’s 35.63; this restricted comparison does not establish superiority on unrestricted translation. Attention also reduces degradation with sentence length.
| Model | All | No UNK∘ |
|---|---|---|
| RNNencdec-30 | 13.93 | 24.19 |
| RNNsearch-30 | 21.50 | 31.44 |
| RNNencdec-50 | 17.82 | 26.71 |
| RNNsearch-50 | 26.75 | 34.16 |
| RNNsearch-50⋆ | 28.45 | 36.15 |
| Moses | 33.30 | 35.63 |
5.2 Qualitative Analysis
5.2.1 Alignment
The selected alignment maps show plausible word correspondences and reordering, including one-to-many relationships. Their flexibility distinguishes soft attention from a monotonic alignment rule. These are qualitative examples rather than supervised alignment-accuracy measurements.
5.2.2 Long Sentences
Selected long-sentence examples show the fixed-context decoder losing meaning after roughly thirty words, while the attention model retains more of the source. Appendix C preserves the complete source, reference, and output comparisons. The examples support the quantitative length analysis but are not a separate representative test.
6 Related Work
6.1 Learning to Align
Earlier handwriting-generation attention moved monotonically along its input. Translation requires flexible reordering, so this model scores every source position for every target word. That flexibility incurs a cost proportional to the product of source and target lengths.
6.2 Neural Networks for Machine Translation
Neural language and phrase models had already improved statistical translation pipelines. This work instead directly generates translations with a single jointly trained neural system; the distinction concerns the overall modeling pipeline, not the first use of neural networks in translation.
7 Conclusion
Soft alignment relieves the fixed-context bottleneck and makes long-sentence translation more robust in this experiment. The remaining vocabulary restriction and gap to phrase-based translation on the full test set motivate better treatment of rare and unknown words.
Acknowledgments
The authors acknowledge Theano, research and computing support, and collaborators who provided feedback.
Appendix A Model Architecture
A.1 Architectural Choices
The general attention framework permits different recurrent cells and scoring functions. The following specifications describe the particular gated recurrent implementation used in the experiments.
A.1.1 Recurrent Neural Network
Update gates interpolate between stored and proposed states; reset gates control how the old state contributes to a proposal. Context enters the decoder’s gated transitions. A maxout hidden layer and softmax produce the next-word distribution. The full update equations remain available below.
Equations
A.1.2 Alignment Model
A single-hidden-layer alignment network combines the previous decoder state with each source annotation. The annotation-dependent projection can be computed once per input sentence, reducing repeated work across target positions.
Equation
A.2 Detailed Description of the Model
A.2.1 Encoder
The appendix specifies the forward and backward gated recurrences, their dimensions, and shared source embeddings. The two directions have separate recurrent parameters; their concatenated states are the annotations supplied to attention.
Equations
A.2.2 Decoder
The decoder is initialized from the backward encoder state and updates using the prior word and current attention context. The appendix retains all gate, alignment, and maxout-output formulas. Replacing every context with the terminal forward encoder state recovers a fixed-context encoder–decoder.
Equations
A.2.3 Model Size
Experiments use hidden dimension 1,000, embeddings of dimension 620, maxout dimension 500, and 1,000 alignment hidden units. Table 2 records updates, epochs, hardware, wall-clock time, and training/development likelihood for each model; sentence-length differences affect these likelihood averages.
| Model | Updates () | Epochs | Hours | GPU | Train NLL | Dev. NLL |
| RNNenc-30 | 8.46 | 6.4 | 109 | TITAN BLACK | 28.1 | 53.0 |
| RNNenc-50 | 6.00 | 4.5 | 108 | Quadro K-6000 | 44.0 | 43.6 |
| RNNsearch-30 | 4.71 | 3.6 | 113 | TITAN BLACK | 26.7 | 47.2 |
| RNNsearch-50 | 2.88 | 2.2 | 111 | Quadro K-6000 | 40.7 | 38.1 |
| RNNsearch-50⋆ | 6.67 | 5.0 | 252 | Quadro K-6000 | 36.7 | 35.2 |
Appendix B Training Procedure
B.1 Parameter Initialization
Recurrent matrices start orthogonal, most other weights use small Gaussian initializations, and biases start at zero. The alignment parameters use their separately specified initialization scales.
B.2 Training
Adadelta adapts learning rates, gradients are clipped to norm one, and minibatches contain 80 pairs. Sorting blocks of 1,600 sentence pairs by length before splitting them into batches reduces wasted padding computation.
Appendix C Translations of Long Sentences
The original comparison table preserves three long English sentences, reference French translations, both neural outputs, and Google Translate outputs collected on 27 August 2014. These examples illustrate differing omissions and wording; they are not an evaluation of present-day translation products.
| Source | An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital. |
| Reference | Le privilège d’admission est le droit d’un médecin, en vertu de son statut de membre soignant d’un hôpital, d’admettre un patient dans un hôpital ou un centre médical afin d’y délivrer un diagnostic ou un traitement. |
| RNNenc-50 | Un privilège d’admission est le droit d’un médecin de reconnaître un patient à l’hôpital ou un centre médical d’un diagnostic ou de prendre un diagnostic en fonction de son état de santé. |
| RNNsearch-50 | Un privilège d’admission est le droit d’un médecin d’admettre un patient à un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, selon son statut de travailleur des soins de santé à l’hôpital. |
| GoogleTranslate | Un privilège admettre est le droit d’un médecin d’admettre un patient dans un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, fondée sur sa situation en tant que travailleur de soins de santé dans un hôpital. |
| Source | This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added. |
| Reference | Ce type d’expérience entre dans le cadre des efforts de Disney pour ”étendre la durée de vie de ses séries et construire de nouvelles relations avec son public grâce à des plateformes numériques qui sont de plus en plus importantes”, a-t-il ajouté. |
| RNNenc-50 | Ce type d’expérience fait partie des initiatives du Disney pour ”prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes. |
| RNNsearch-50 | Ce genre d’expérience fait partie des efforts de Disney pour ”prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes”, a-t-il ajouté. |
| GoogleTranslate | Ce genre d’expérience fait partie des efforts de Disney à “étendre la durée de vie de sa série et construire de nouvelles relations avec le public par le biais des plates-formes numériques qui deviennent de plus en plus important”, at-il ajouté. |
| Source | In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might constitute a ”reasonable motive” that could lead to criminal charges being brought against the mayor. |
| Reference | En conférence de presse, jeudi, M. Blair a affirmé qu’il n’y avait rien dans cette vidéo qui puisse constituer des ”motifs raisonnables” pouvant mener au dépôt d’une accusation criminelle contre le maire. |
| RNNenc-50 | Lors de la conférence de presse de jeudi, M. Blair a dit qu’il n’y avait rien dans cette vidéo qui pourrait constituer une ”motivation raisonnable” pouvant entraîner des accusations criminelles portées contre le maire. |
| RNNsearch-50 | Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vidéo qui pourrait constituer un ”motif raisonnable” qui pourrait conduire à des accusations criminelles contre le maire. |
| GoogleTranslate | Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vido qui pourrait constituer un ”motif raisonnable” qui pourrait mener à des accusations criminelles portes contre le maire. |