Literature
Display
← All papers

Neural Phrase-to-Phrase Machine Translation

Jiangtao Feng · Lingpeng Kong · Po-Sen Huang · Chong Wang · Da Huang · Jiayuan Mao · Kan Qiao · Dengyong Zhou

Preprint posted · arXiv (preprint) ↗

Language models6 min text · CondensedOriginal paper ↗
Contents

At a glance

Neural Phrase-to-Phrase Machine Translation (NP2MT) attends to source segments and generates target segments. Training sums over possible target segmentations with a dynamic program. At decoding time, source phrases containing unknown words can be translated through an external bilingual dictionary, with the neural decoder selecting among candidate translations.

On the tested German-English and English-Vietnamese tasks, ordinary translation quality is comparable to the Transformer baseline. The clearest advantage is dictionary-assisted translation under artificially restricted vocabularies or a shift from talks to news. The study uses word-level vocabularies and greedy dictionary decoding; subword systems and dictionary-enhanced beam search are left for future work.

1 Introduction

Phrase-based statistical translation explicitly models groups of words and their alignments, while standard neural sequence models usually generate one token at a time. NP2MT combines phrase-level source attention with segment-level target generation. Unlike NPMT's monotonic input-output alignment, attention allows source phrases to be revisited in a different order. [13] [15]

Phrase alignment also provides an interface to bilingual dictionaries. This is especially relevant when a fixed training vocabulary lacks words encountered at test time. The contribution is an inductive bias and a decoding mechanism, rather than a claim that phrase modeling universally improves in-domain neural translation.

2 Neural phrase-to-phrase machine translation

The model combines phrase representations, a segmentation-marginalizing training objective, and an optional dictionary rule at inference.

2.1 NP 2 MT

A bidirectional sentence encoder contextualizes source words, then a phrase encoder represents candidate source spans up to a maximum length L. The target encoder summarizes the concatenated output prefix, and phrase attention conditions a Transformer segment decoder. The same output prefix induces the same attention state regardless of how earlier output was segmented. [18] [29]

The decoder emits tokens until an end-of-segment symbol or end-of-sentence symbol. Concatenating generated segments gives the translation. Several segmentations can yield that same sentence, so its likelihood sums their probabilities. The original generative procedure and equations below retain the precise formulation.

Original figure
Figure 1: The architecture of NP2MT model. The example shows how the target phrase y3:4y_{3:4} is translated conditioning directly on the source phrase x2:3x_{2:3}, using the phrase-level attention. Note that for brevity, in the phrase-level encoder, we only show one possible segmentation of the source sentence. We use “⋯\cdots” to indicate all the possible segments xi:jx_{i:j} in Eq. 2.
Equations 1–2
𝐬1:T=fsentsrc(x1:T).\displaystyle\mathbf{s}_{1:T}=f_{\text{sent}}^{\text{src}}(x_{1:T}). (1)
𝐡[i,j]=fphrasesrc(𝐬i:j),∀(i,j)∈{(i,j):1≤i≤j≤i+L−1≤T}.\displaystyle\mathbf{h}_{[i,j]}=f_{\text{phrase}}^{\text{src}}(\mathbf{s}_{i:j}),\forall(i,j)\in\{(i,j):1\leq i\leq j\leq i+L-1\leq T\}. (2)
  1. 1.

    For segment index n=1,…,∞n=1,...,\infty

    1. (a)

      Update the attention state given all previous segments,

      Equation 3
      𝐚j=attn(𝐡,π(z1:n−1)),\displaystyle\mathbf{a}_{j}={\rm attn}(\mathbf{h},\pi(z_{1:n-1})), (3)

      where j=∑m=1n−1|zm|j=\sum_{m=1}^{n-1}|z_{m}| is the total length of all previous segments. Note that different collections of previous segments can lead to the same attention state as long as their concatenation is the same.

    2. (b)

      Given the current state 𝐚j\mathbf{a}_{j}, sample words from segment decoder gg until we reach the end of segment symbol $\$ or end of sentence e​o​seos. This gives us a segment znz_{n}.

    3. (c)

      If we see e​o​seos in znz_{n}, stop.

  2. 2.

    Concatenate {z1,…,zN}\{z_{1},\ldots,z_{N}\} to obtain the output sentence π(z1:N)=y1:T′\pi(z_{1:N})=y_{1:T^{\prime}}.

Equation 4
p(y1:T′|x1:T)=∑𝐳:π(𝐳)=y1:T′∏n=1|𝐳|p(zn|𝐚j),\displaystyle p(y_{1:T^{\prime}}|x_{1:T})=\sum_{\mathbf{z}:\pi(\mathbf{z})=y_{1:T^{\prime}}}\prod_{n=1}^{|\mathbf{z}|}p(z_{n}|\mathbf{a}_{j}), (4)

2.2 Training

Training minimizes negative log-likelihood after marginalizing over target segmentations. A forward variable accumulates the probability of every target prefix by considering possible preceding segment boundaries. Prefix-dependent attention allows this dynamic program to reuse state across alternative segmentations.

The paper gives O(T′L) complexity for this segmentation dynamic program, compared with O(T × T′ × L) for NPMT, where T and T′ are source and target lengths. This comparison concerns the recurrence, not the total cost of encoding all source spans, computing attention, or neural decoding. The source's recurrence is retained below. [13]

Equations 5
ℒ(θ)=−logp(y1:T′|x1:T).\mathcal{L}(\mathbf{\theta})=-\log p(y_{1:T^{\prime}}|x_{1:T}). (5)
α⁡(j)\displaystyle\alpha(j) =∑j′<jα(j′)p(yj′+1:j|𝐚j′),\displaystyle=\sum_{j^{\prime}<j}{\alpha(j^{\prime})p(y_{j^{\prime}+1:j}|\mathbf{a}_{j^{\prime}})},
p(yj′+1:j|𝐚j′)\displaystyle p(y_{j^{\prime}+1:j}|\mathbf{a}_{j^{\prime}}) =p($|yj′+1:j,𝐚j′)p(yj′|𝐚j′)∏j′<k≤jp(yk|yj′:k−1,𝐚j′),\displaystyle=p(\$|y_{j^{\prime}+1:j},\mathbf{a}_{j^{\prime}})p(y_{j^{\prime}}|\mathbf{a}_{j^{\prime}})\prod_{j^{\prime}<k\leq j}{p(y_{k}|y_{j^{\prime}:k-1},\mathbf{a}_{j^{\prime}})},

2.3 Decoding

Ordinary decoding uses the word-level beam search retained in Appendix A. Dictionary integration is implemented separately for greedy decoding.

Dictionary integration in decoding.

At each segment boundary, choose the most-attended source phrase. Use the dictionary only if that phrase contains an unknown word and has a dictionary entry. When several translations are available, score them with the segment decoder and select the best; otherwise generate a segment neurally. Figure 2 and Algorithm 1 show the alignment and decision rule.

The dictionary trigger is rule-based rather than a joint probabilistic mixture. The authors do not integrate it into beam search because neural and dictionary alternatives lack a common score under this rule. Dictionary quality, coverage, and candidate ranking therefore directly constrain the method.

Original figure
Figure 2: The decoding process of NP2MT model with dictionary extension. The example shows how a German sentence “sie sehen in der mitte das kolosseum , den fluß tiber .” is translated into the English sentence “you see the colosseum in the middle , the river tiber .” Different colors are used to represent the aligned phrases. The underlined words (“kolosseum”, “fluß”, “tiber”) are OOV words, and the italic phrases (“sehen”, “das kolosseum”, “tiber”) can be found in the dictionary. The solid lines are used to denote the translation by the neural model, and the dashed ones are by the dictionary.
Data: a source sentence: x1:Tx_{1:T}, a dictionary: DD
Result: a target sentence: yy
Compute hidden representation 𝐡\mathbf{h} for the source sentence;
Set initial output segment ztgt=n​u​l​lz^{\text{tgt}}=null;
Set output sentence y=Øy=\O;
while eos ∉\notin ztgtz^{\text{tgt}} do
   Compute attention state 𝐚j\mathbf{a}_{j} and the attention distribution;
   Select the attended source phrase zsrcz^{\text{src}} with the maximum attention score;
   if UNK ∈\in zsrcz^{\text{src}} && zsrcz^{\text{src}} ∈\in DD then
      Retrieve all target translations D⁡(zsrc)D(z^{\text{src}}) from the dictionary;
      Select the best candidate ztgtz^{\text{tgt}} in D⁡(zsrc)D(z^{\text{src}}) by using the neural segment decoder gg as the scoring function
   else
      Generate the next output segment ztgtz^{\text{tgt}} by using the neural segment decoder g⁡(𝐚j)g(\mathbf{a}_{j}));
   end if
   Append ztgtz^{\text{tgt}} to output yy;
end while
Algorithm 1 NP2MT greedy decoding with dictionaries.

3 Experiments

Experiments compare ordinary translation, simulated open-vocabulary translation, and cross-domain transfer. All systems here use word-level vocabularies. The paper explicitly leaves byte-pair or other subword representations to future work, which matters when interpreting the unknown-word advantage.

3.1 IWSLT Machine Translation Tasks

IWSLT14 German-English contains about 153,000 training pairs; IWSLT15 English-Vietnamese uses about 133,000. Both derive from TED talks. NP2MT uses six-layer source and target recurrent encoders and a six-layer Transformer segment decoder. Adam, 256-dimensional embeddings, dropout, and a warmup/plateau/linear-decay schedule train the system. The Transformer baseline receives separately tuned hyperparameters. [4] [5]

German-English beam-search BLEU is 31.70 for NP2MT versus 32.30 for Transformer; English-Vietnamese is 30.60 versus 30.74. Greedy results are also close. Thus ordinary benchmark results support comparability, not consistent superiority. The paper reports about two hours of NP2MT training versus 24 hours for NPMT on a V100 GPU; this is an implementation-specific timing comparison, not the full dynamic-program asymptotic bound.

Model BLEU
Greedy Beam Search
BSO (Wiseman & Rush, 2016) 23.83 25.48
Seq2Seq with attention 26.17 27.61
Actor-Critic (Bahdanau et al., 2016) 27.49 28.53
Transformer (Vaswani et al., 2017) 31.27 32.30
NPMT (Huang et al., 2018) 28.57 29.92
NP2MT 30.99 31.70
Table 1: Performance on IWSLT14 German-English test set.
Model BLEU
Greedy Beam Search
Luong & Manning (2015) - 23.30
Seq2Seq with attention 25.50 26.10
Transformer (Vaswani et al., 2017) 29.72 30.74
NPMT (Huang et al., 2018) 26.91 27.69
NP2MT 29.93 30.60
Table 2: Performance on IWSLT15 English-Vietnamese test set.

3.2 Translation with Dictionary

The open-vocabulary experiment replaces words below frequency thresholds of 3, 5, 10, 20, 50, or 100 with unknown tokens in the model vocabulary, while retaining their text for dictionary lookup. This deliberately creates distributional gaps; it is not an unrestricted evaluation of naturally occurring unseen vocabulary.

Moses extracts in-domain phrase dictionaries from the training pairs. German-English additionally uses a much larger dictionary from WMT14 news. The Transformer comparison translates attended unknown words with a simpler lookup rule. As the unknown-word rate increases, both systems benefit more from dictionaries, and NP2MT's phrase lookup often adds a larger gain. At the largest threshold, the German-English in-domain dictionary improves NP2MT from 23.08 to 26.61 BLEU; English-Vietnamese improves from 23.63 to 26.31.

Dictionary coverage and domain match both matter: the large news dictionary helps more at lower unknown-word rates, while the smaller in-domain dictionary can be better at high rates. Tables 3–5 retain every setting and example; the results do not show a universal win at every threshold.

Threshold 3 5 10 20 50 100
Vocab Size German 35,479 23,327 13,679 8,055 3,831 2,152
English 24,743 17,807 11,559 7,359 3,807 2,224
Test Data OOV Rate German 3.8% 4.7% 6.2% 8.1% 11.6% 14.8%
English 1.5% 2.1% 3.0% 4.4% 7.2% 10.2%
Transformer 31.27 30.92 30.35 28.88 26.27 23.73
Transformer + DID_{I} 31.27 30.97 30.65 29.34 27.78 26.01
Transformer + DOD_{O} 31.67 31.40 31.04 29.43 27.42 25.20
NP2MT 30.99 30.92 29.86 28.29 25.81 23.08
NP2MT + DID_{I} 30.99 31.01 30.33 29.52 28.02 26.61
NP2MT + DOD_{O} 31.48 31.75 31.06 30.04 27.86 25.85
Table 3: IWSLT14 German-English Translation with in-domain and out-of-domain dictionaries.
Threshold 3 5 10 20 50 100
Vocab Size English 24,415 17,191 10,919 6,799 3,431 1,943
Vietnamese 10,663 7,711 5,343 3,863 2,575 1,943
Test Data OOV Rate English 2.2% 2.8% 3.9% 5.6% 8.9% 12.1%
Vietnamese 0.8% 1.0% 1.4% 2.0% 3.3% 4.6%
Transformer 29.72 29.71 28.97 28.40 25.68 23.00
Transformer + DID_{I} 29.72 29.72 29.01 28.71 26.92 25.02
NP2MT 29.93 29.78 29.27 27.87 25.89 23.63
NP2MT+DID_{I} 29.94 29.79 29.36 28.38 27.65 26.31
Table 4: IWSLT15 English-Vietnamese Translation with an in-domain dictionary.
Original figure
(a) IWSLT14 DE-EN
Original figure
(b) IWSLT15 EN-VI
Figure 3: Comparison on BLEU score improvement and lookup phrase ratio under different source language OOV rates both under in-domain dictionaries.
Figure 4: Comparison on BLEU score improvement and lookup phrase ratio for IWSLT14 DE-EN under in-domain and out-of-domain dictionaries.
Original figure
Source [dies]1​ist​[die maschine]2​[unterhalb]3​(von genf)4​[.]5\text{[dies]}_{1}\;\text{ist}\;\text{[die maschine]}_{2}\;\text{[unterhalb]}_{3}\;\text{(von \lx@text@underline{genf})}_{4}\;\text{[.]}_{5}
Greedy decoding [this is]1​[the machine]2​[below]3​(geneva)4​[.]5\text{[this is]}_{1}\;\text{[the machine]}_{2}\;\text{[below]}_{3}\;\text{(geneva)}_{4}\;\text{[.]}_{5}
Target ground truth this is the machine below geneva .
Source [dies]1​ist​(eine bescheidene)2​[kleine]3​(app)4​[.]5\text{[dies]}_{1}\;\text{ist}\;\text{(eine \lx@text@underline{bescheidene})}_{2}\;\text{[kleine]}_{3}\;\text{(\lx@text@underline{app})}_{4}\;\text{[.]}_{5}
Greedy decoding [this is]1​(a modest)2​[little]3​(app)4​[.]5\text{[this is]}_{1}\;\text{(a modest)}_{2}\;\text{[little]}_{3}\;\text{(app)}_{4}\;\text{[.]}_{5}
Target ground truth this is a modest little app .
Source [unsere]1​(zeitschriften)2​[werden von]3​[millionen]4​[gelesen]5​[.]6\text{[unsere]}_{1}\;\text{(\lx@text@underline{zeitschriften})}_{2}\;\text{[werden von]}_{3}\;\text{[millionen]}_{4}\;\text{[gelesen]}_{5}\;\text{[.]}_{6}
Greedy decoding [our]1​(journals)2​[are]3​[read by]5​[millions]4​[.]6\text{[our]}_{1}\;\text{(journals)}_{2}\;\text{[are]}_{3}\;\text{[read by]}_{5}\;\text{[millions]}_{4}\;\text{[.]}_{6}
Target ground truth our magazines are read by millions .
Table 5: DE-EN translation examples, where “[⋅\cdot]” represents the phrase boundary, “(⋅\cdot)” represents the phrase is looked up by the external dictionary, and “  ⋅\cdot  ” represents the frequency of the word is under the vocabulary threshold and hence the word is replaced by the “UNK” token in the network vocabulary. The subscript represents the corresponding phrases found by NP2MT.

3.3 Cross-Domain Translation

Models trained and validated on IWSLT14 talks are tested on WMT14 news. Despite the broadest training vocabulary used here, test unknown-word rates are 12.8% in German and 6.7% in English. A dictionary extracted from the news domain supplies additional target-domain information.

NP2MT rises from 14.86 to 16.11 BLEU with the dictionary; Transformer rises from 14.69 to 15.60. The final difference is 0.51 BLEU, and the dictionary-induced gains are 1.25 and 0.91 respectively. This demonstrates dictionary-assisted adaptation under the stated setup, rather than transfer without target-domain resources.

Model BLEU
Transformer 14.69
Transformer + DWMTD_{\text{WMT}} 15.60
NP2MT 14.86
NP2MT+DWMTD_{\text{WMT}} 16.11
Table 6: Results on cross-domain translation by training the model on IWSLT14 and testing on WMT14, both with German-English data. In the test data from WMT14, OOV rates are 12.812.8% for German and 6.76.7% for English.

4 Related Work

NPMT and segment-to-segment transduction marginalize monotonic alignments; NP2MT uses phrase attention instead. Other approaches store phrase tables in neural memory, impose lexical constraints during beam search, or add phrase-aware source representations. NP2MT's distinction is to combine attended source phrases, generated target segments, and the explicit unknown-phrase dictionary rule. [10] [13] [24] [35]

5 Conclusions

Phrase-level attention supplies a useful interface between a neural translator and an external dictionary. The experiments show larger dictionary gains under severe vocabulary restriction and a modest advantage in one talks-to-news transfer test. Remaining questions include subword vocabularies, robust dictionary scoring within beam search, and generalization beyond these two language pairs and resource settings.

Acknowledgements

The authors thank Dani Yogatama and Chris Dyer for comments and discussions.

Appendix A Beam Search

Appendix A retains the full word-level beam-search procedure. It maintains unfinished and completed candidates, expands the next token, handles segment and sentence terminators, and prunes unfinished hypotheses to the top k scores. The highest-scoring completed sentence is returned. This algorithm is the ordinary neural decoder; it does not include the rule-based dictionary extension.

The pseudocode is preserved as published, including its score initialization and additive updates, rather than presented as a new executable implementation.

Data: a source sentence: x1:Tx_{1:T}, beam size kk
Result: a target sentence: yy
compute the hidden representation 𝐡\mathbf{h} for the source sentence;
prev_sent ←\leftarrow end-of-sentence token eos;
prev_segment ←\leftarrow end-of-segment token $\$;
init_score ←\leftarrow 11;
// Z is the appendable candidates;
// Y is the finished candidates;
Z={(prev_sent,prev_segment,init_score)}Z=\{(\text{prev\_sent},\text{prev\_segment},\text{init\_score})\};
Y=ØY=\O;
while Z does not reach maximum length do
   // Z~\tilde{Z} is the next appendable candidates;
   Z~=Ø\tilde{Z}=\O;
   for prev_sent, prev_segment, score in ZZ do
      Generate kk next_tokens with probability pkp_{k};
      for next_token in next_tokens candidate set do
         if next_token is eos then
            prev_segment ←\leftarrow prev_segment + next_token;
            prev_sent ←\leftarrow prev_sent + prev_segment;
            score ←\leftarrow score + pkp_{k};
            Append (prev_sent, score) to YY;
         end if
         if next_token is $\$ then
            prev_sent ←\leftarrow prev_sent + prev_segment;
            score ←\leftarrow score + pkp_{k};
            Append (prev_sent, $\$, score) to Z~\tilde{Z};
         end if
         if next_token is not $\$ or eos then
            prev_segment ←\leftarrow prev_segment + next_token;
            prev_sent ←\leftarrow prev_sent + prev_segment;
            score ←\leftarrow score + pkp_{k};
            Append (prev_sent, prev_segment, score) to Z~\tilde{Z};
         end if
      end for
   end for
   Keep only top kk candidates in Z~\tilde{Z};
   Z←Z~Z\leftarrow\tilde{Z}
end while
yy ←\leftarrow the best candidates with highest score in YY
Algorithm 2 NP2MT Beam Search.
Concept demonstration

Open image ↗