At a glance
Neural Phrase-to-Phrase Machine Translation (NP2MT) attends to source segments and generates target segments. Training sums over possible target segmentations with a dynamic program. At decoding time, source phrases containing unknown words can be translated through an external bilingual dictionary, with the neural decoder selecting among candidate translations.
On the tested German-English and English-Vietnamese tasks, ordinary translation quality is comparable to the Transformer baseline. The clearest advantage is dictionary-assisted translation under artificially restricted vocabularies or a shift from talks to news. The study uses word-level vocabularies and greedy dictionary decoding; subword systems and dictionary-enhanced beam search are left for future work.
1 Introduction
Phrase-based statistical translation explicitly models groups of words and their alignments, while standard neural sequence models usually generate one token at a time. NP2MT combines phrase-level source attention with segment-level target generation. Unlike NPMT's monotonic input-output alignment, attention allows source phrases to be revisited in a different order. [13] [15]
Phrase alignment also provides an interface to bilingual dictionaries. This is especially relevant when a fixed training vocabulary lacks words encountered at test time. The contribution is an inductive bias and a decoding mechanism, rather than a claim that phrase modeling universally improves in-domain neural translation.
2 Neural phrase-to-phrase machine translation
The model combines phrase representations, a segmentation-marginalizing training objective, and an optional dictionary rule at inference.
2.1 NP 2 MT
A bidirectional sentence encoder contextualizes source words, then a phrase encoder represents candidate source spans up to a maximum length L. The target encoder summarizes the concatenated output prefix, and phrase attention conditions a Transformer segment decoder. The same output prefix induces the same attention state regardless of how earlier output was segmented. [18] [29]
The decoder emits tokens until an end-of-segment symbol or end-of-sentence symbol. Concatenating generated segments gives the translation. Several segmentations can yield that same sentence, so its likelihood sums their probabilities. The original generative procedure and equations below retain the precise formulation.
Equations 1–2
| (1) |
| (2) |
- 1.
For segment index
- (a)
Update the attention state given all previous segments,
Equation 3
(3) where is the total length of all previous segments. Note that different collections of previous segments can lead to the same attention state as long as their concatenation is the same.
- (b)
Given the current state , sample words from segment decoder until we reach the end of segment symbol or end of sentence . This gives us a segment .
- (c)
If we see in , stop.
- (a)
- 2.
Concatenate to obtain the output sentence .
Equation 4
| (4) |
2.2 Training
Training minimizes negative log-likelihood after marginalizing over target segmentations. A forward variable accumulates the probability of every target prefix by considering possible preceding segment boundaries. Prefix-dependent attention allows this dynamic program to reuse state across alternative segmentations.
The paper gives O(T′L) complexity for this segmentation dynamic program, compared with O(T × T′ × L) for NPMT, where T and T′ are source and target lengths. This comparison concerns the recurrence, not the total cost of encoding all source spans, computing attention, or neural decoding. The source's recurrence is retained below. [13]
Equations 5
| (5) |
2.3 Decoding
Ordinary decoding uses the word-level beam search retained in Appendix A. Dictionary integration is implemented separately for greedy decoding.
Dictionary integration in decoding.
At each segment boundary, choose the most-attended source phrase. Use the dictionary only if that phrase contains an unknown word and has a dictionary entry. When several translations are available, score them with the segment decoder and select the best; otherwise generate a segment neurally. Figure 2 and Algorithm 1 show the alignment and decision rule.
The dictionary trigger is rule-based rather than a joint probabilistic mixture. The authors do not integrate it into beam search because neural and dictionary alternatives lack a common score under this rule. Dictionary quality, coverage, and candidate ranking therefore directly constrain the method.
3 Experiments
Experiments compare ordinary translation, simulated open-vocabulary translation, and cross-domain transfer. All systems here use word-level vocabularies. The paper explicitly leaves byte-pair or other subword representations to future work, which matters when interpreting the unknown-word advantage.
3.1 IWSLT Machine Translation Tasks
IWSLT14 German-English contains about 153,000 training pairs; IWSLT15 English-Vietnamese uses about 133,000. Both derive from TED talks. NP2MT uses six-layer source and target recurrent encoders and a six-layer Transformer segment decoder. Adam, 256-dimensional embeddings, dropout, and a warmup/plateau/linear-decay schedule train the system. The Transformer baseline receives separately tuned hyperparameters. [4] [5]
German-English beam-search BLEU is 31.70 for NP2MT versus 32.30 for Transformer; English-Vietnamese is 30.60 versus 30.74. Greedy results are also close. Thus ordinary benchmark results support comparability, not consistent superiority. The paper reports about two hours of NP2MT training versus 24 hours for NPMT on a V100 GPU; this is an implementation-specific timing comparison, not the full dynamic-program asymptotic bound.
| Model | BLEU | |
|---|---|---|
| Greedy | Beam Search | |
| BSO (Wiseman & Rush, 2016) | 23.83 | 25.48 |
| Seq2Seq with attention | 26.17 | 27.61 |
| Actor-Critic (Bahdanau et al., 2016) | 27.49 | 28.53 |
| Transformer (Vaswani et al., 2017) | 31.27 | 32.30 |
| NPMT (Huang et al., 2018) | 28.57 | 29.92 |
| NP2MT | 30.99 | 31.70 |
| Model | BLEU | |
|---|---|---|
| Greedy | Beam Search | |
| Luong & Manning (2015) | - | 23.30 |
| Seq2Seq with attention | 25.50 | 26.10 |
| Transformer (Vaswani et al., 2017) | 29.72 | 30.74 |
| NPMT (Huang et al., 2018) | 26.91 | 27.69 |
| NP2MT | 29.93 | 30.60 |
3.2 Translation with Dictionary
The open-vocabulary experiment replaces words below frequency thresholds of 3, 5, 10, 20, 50, or 100 with unknown tokens in the model vocabulary, while retaining their text for dictionary lookup. This deliberately creates distributional gaps; it is not an unrestricted evaluation of naturally occurring unseen vocabulary.
Moses extracts in-domain phrase dictionaries from the training pairs. German-English additionally uses a much larger dictionary from WMT14 news. The Transformer comparison translates attended unknown words with a simpler lookup rule. As the unknown-word rate increases, both systems benefit more from dictionaries, and NP2MT's phrase lookup often adds a larger gain. At the largest threshold, the German-English in-domain dictionary improves NP2MT from 23.08 to 26.61 BLEU; English-Vietnamese improves from 23.63 to 26.31.
Dictionary coverage and domain match both matter: the large news dictionary helps more at lower unknown-word rates, while the smaller in-domain dictionary can be better at high rates. Tables 3–5 retain every setting and example; the results do not show a universal win at every threshold.
| Threshold | 3 | 5 | 10 | 20 | 50 | 100 | |
| Vocab Size | German | 35,479 | 23,327 | 13,679 | 8,055 | 3,831 | 2,152 |
| English | 24,743 | 17,807 | 11,559 | 7,359 | 3,807 | 2,224 | |
| Test Data OOV Rate | German | 3.8% | 4.7% | 6.2% | 8.1% | 11.6% | 14.8% |
| English | 1.5% | 2.1% | 3.0% | 4.4% | 7.2% | 10.2% | |
| Transformer | 31.27 | 30.92 | 30.35 | 28.88 | 26.27 | 23.73 | |
| Transformer + | 31.27 | 30.97 | 30.65 | 29.34 | 27.78 | 26.01 | |
| Transformer + | 31.67 | 31.40 | 31.04 | 29.43 | 27.42 | 25.20 | |
| NP2MT | 30.99 | 30.92 | 29.86 | 28.29 | 25.81 | 23.08 | |
| NP2MT + | 30.99 | 31.01 | 30.33 | 29.52 | 28.02 | 26.61 | |
| NP2MT + | 31.48 | 31.75 | 31.06 | 30.04 | 27.86 | 25.85 | |
| Threshold | 3 | 5 | 10 | 20 | 50 | 100 | |
|---|---|---|---|---|---|---|---|
| Vocab Size | English | 24,415 | 17,191 | 10,919 | 6,799 | 3,431 | 1,943 |
| Vietnamese | 10,663 | 7,711 | 5,343 | 3,863 | 2,575 | 1,943 | |
| Test Data OOV Rate | English | 2.2% | 2.8% | 3.9% | 5.6% | 8.9% | 12.1% |
| Vietnamese | 0.8% | 1.0% | 1.4% | 2.0% | 3.3% | 4.6% | |
| Transformer | 29.72 | 29.71 | 28.97 | 28.40 | 25.68 | 23.00 | |
| Transformer + | 29.72 | 29.72 | 29.01 | 28.71 | 26.92 | 25.02 | |
| NP2MT | 29.93 | 29.78 | 29.27 | 27.87 | 25.89 | 23.63 | |
| NP2MT+ | 29.94 | 29.79 | 29.36 | 28.38 | 27.65 | 26.31 | |
| Source | |
|---|---|
| Greedy decoding | |
| Target ground truth | this is the machine below geneva . |
| Source | |
| Greedy decoding | |
| Target ground truth | this is a modest little app . |
| Source | |
| Greedy decoding | |
| Target ground truth | our magazines are read by millions . |
3.3 Cross-Domain Translation
Models trained and validated on IWSLT14 talks are tested on WMT14 news. Despite the broadest training vocabulary used here, test unknown-word rates are 12.8% in German and 6.7% in English. A dictionary extracted from the news domain supplies additional target-domain information.
NP2MT rises from 14.86 to 16.11 BLEU with the dictionary; Transformer rises from 14.69 to 15.60. The final difference is 0.51 BLEU, and the dictionary-induced gains are 1.25 and 0.91 respectively. This demonstrates dictionary-assisted adaptation under the stated setup, rather than transfer without target-domain resources.
4 Related Work
NPMT and segment-to-segment transduction marginalize monotonic alignments; NP2MT uses phrase attention instead. Other approaches store phrase tables in neural memory, impose lexical constraints during beam search, or add phrase-aware source representations. NP2MT's distinction is to combine attended source phrases, generated target segments, and the explicit unknown-phrase dictionary rule. [10] [13] [24] [35]
5 Conclusions
Phrase-level attention supplies a useful interface between a neural translator and an external dictionary. The experiments show larger dictionary gains under severe vocabulary restriction and a modest advantage in one talks-to-news transfer test. Remaining questions include subword vocabularies, robust dictionary scoring within beam search, and generalization beyond these two language pairs and resource settings.
Acknowledgements
The authors thank Dani Yogatama and Chris Dyer for comments and discussions.
Appendix A Beam Search
Appendix A retains the full word-level beam-search procedure. It maintains unfinished and completed candidates, expands the next token, handles segment and sentence terminators, and prunes unfinished hypotheses to the top k scores. The highest-scoring completed sentence is returned. This algorithm is the ordinary neural decoder; it does not include the rule-based dictionary extension.
The pseudocode is preserved as published, including its score initialization and additive updates, rather than presented as a new executable implementation.