1 Introduction↩︎

Neural machine translation (NMT) models with encoder-decoder framework [1], [2] significantly outperform conventional statistical machine translation models [3], [4]. Despite their success, the state-of-the-art NMT models usually suffer from the slow inference speed, which has become a bottleneck to apply NMT in real-world translation systems. The slow inference speed of NMT models is due to their autoregressive property, i.e., decoding the target sentence word-by-word according to the translation history.

Recently, [5] introduced non-autoregressive NMT (NAT) which can simultaneously decode all target words to break the bottleneck of the autoregressive NMT (AT) models. To this end, NAT models [5]–[8] usually directly copy the source word representations to the input of the decoder, instead of using previous predicted target word representations. Hence, the inference of different target words are independent, which enables parallel computation of the decoder in NAT models. NAT models could achieve \(10\)-\(15\) times speedup compared to AT models while maintaining considerable translation quality.

However, NAT models still suffer from the multimodality problem [5]: it discards the dependencies among the target words, and therefore the target words may be chosen from multiple feasible translations, resulting in duplicate, missing or even wrong words. For example, the German phrase “Vielen Dank” can be translated as both ”

” and ”

“. Unfortunately, as each target word is generated independently,”

” and ”

” may also be assigned high probabilities, resulting in inferior translation quality. In this work, we argue reordering information is essential for NAT models and helpful for alleviating the multimodality problem.

To this end, we propose a novel NAT framework named ReorderNAT in this work, which explicitly models the reordering information to guide the decoding of NAT. To be specific, as shown in Figure 1, ReorderNAT first reorders the source sentence into a pseudo-translation formed by source words but in the target language word order, and then translates the source sentence conditioned on it. We further introduce two guiding decoding strategies which utilize the reordering information (i.e. pseudo-translation) to guide the word selection in decoding. The first one is deterministic guiding decoding which first generates a most likely pseudo-translation and then generates the target sentence based on it. The second one is non-deterministic guiding decoding which utilizes the conditional distribution of the pseudo-translation as a latent variable to guide the decoding of target sentences.

Ideally, the pseudo-translation can be viewed as a final translation written in source language. Guiding decoding with it could help to model the conditional dependencies of the target words and encourage the decoder to choose words belonging to the same translation, which naturally reduces the multimodality problem. Moreover, the decoding space of generating pseudo-translation is limited to the permutation of words in the source sentence, which can be well modeled by a small model. Therefore, ReorderNAT could effectively alleviate the multimodality problem by introducing the reordering information in NAT.

Experimental results on several widely-used benchmarks show that our proposed ReorderNAT model achieves significant and consistent improvements compared to existing NAT models by explicitly modeling the reordering information to guide the decoding. Moreover, by introducing a simple but effective AT module to model reordering information, our ReorderNAT immensely narrows the translation quality gap between AT and NAT models, while maintaining considerable speedup (nearly six times faster). The source codes are available at https://github.com/ranqiu92/ReorderNAT.

2 Background↩︎

Non-autoregressive neural machine translation (NAT) is first proposed by [5] to alleviate the slow decoding issue of autoregressive neural machine translation (AT) models, which could simultaneously generate target words by removing their dependencies. Formally, given a source sentence \(\mathbf{X}=\{x_1, \cdots, x_n\}\) and a target sentence \(\mathbf{Y} = \{y_1, \cdots, y_m\}\), NAT models the translation probability from \(\mathbf{X}\) to \(\mathbf{Y}\) as a product of conditionally independent target word probability: \[\label{eq:nat}P(\mathbf{Y}|\mathbf{X}) = \prod_{i=1}^m P(y_i|\mathbf{X}). \tag{1}\]

Instead of utilizing the translation history, NAT models usually copy source word representations as the input of the decoder. Hence, when translating a sentence, NAT models could predict all target words with their maximum likelihood individually by breaking the dependency among them, and therefore the decoding procedure of NAT models is in parallel and has very low translation latency.

However, since NAT models discard the sequential dependencies among words in the target sentence, they suffer from the potential performance degradation due to the multimodality problem. To be specific, a source sentence may have multiple translations. During decoding, NAT models may choose the target words from different translations, resulting in duplicate, missing or even wrong words. Consequently, NAT models cannot effectively learn the intricate translation patterns from source sentences to target sentences, leading to inferior translation quality.

3 Methodology↩︎

In this section, we introduce a novel NAT model named ReorderNAT, which aims to alleviate the multimodality problem in NAT models.

3.1 ReorderNAT↩︎

As shown in Figure 1, ReorderNAT employs a reordering module to explicitly model the reordering information in the decoding². Formally, ReorderNAT first employs the reordering module to translate the source sentence \(\mathbf{X}\) into a pseudo-translation \(\mathbf{Z} = \{z_1, \cdots, z_m\}\) which reorganizes source sentence structure into the target language, and then uses the decoder module to generate target translation \(\mathbf{Y}\) based on the pseudo-translation. ReorderNAT models the overall translation probability as: \[\begin{align} P(\mathbf{Y}|\mathbf{X}) &=& \sum_{\mathbf{Z}}P(\mathbf{Y}|\mathbf{Z},\mathbf{X})P(\mathbf{Z}|\mathbf{X}),\end{align}\] where \(P(\mathbf{Z}|\mathbf{X})\) is modeled by the reordering module and \(P(\mathbf{Y}|\mathbf{Z},\mathbf{X})\) is modeled by the decoder module. Next, we will introduce the reordering and decoder modules in detail³.

3.1.1 Reordering Module↩︎

The reordering module determines the source-side information of each target word by learning to translate the source sentence into a pseudo-translation. We propose two feasible implementations of the reordering module:

(1) NAT Reordering Module: Intuitively, the pseudo-translation probability can also be modeled as NAT: \[\label{eq:word-reorder} P(\mathbf{Z}|\mathbf{X}) = \prod_{i=1}^m P(z_i|\mathbf{X}),\qquad{(1)}\] where \(P(z_i|\mathbf{X})\) is calculated by a single-layer Transformer. During inference, the NAT reordering module needs to determine the length of the pseudo-translation. To this end, we use a length predictor and copy the embeddings of the source sentence as the input of the reordering module similar to existing NAT models.

(2) AT Reordering Module: We find that AT models are more suitable for modeling the reordering information compared to NAT models, and even a light AT model with similar decoding speed to a large NAT model could achieve better performance in modeling reordering information. Hence, we also introduce a light AT model to model the pseudo-translation probability as: \[\label{eq:word-reorder1} P(\mathbf{Z}|\mathbf{X}) = \prod_{i=1}^m P(z_i|\mathbf{z}_{<i},\mathbf{X}), \tag{2}\] where \(\mathbf{z}_{<i} = \{z_1, \cdots, z_{i-1}\}\) indicates the pseudo-translation history, and \(P(z_i|\mathbf{z}_{<i},\mathbf{X})\) is calculated by a single-layer recurrent neural network.

3.1.2 Decoder Module↩︎

The decoder module translates the source sentence into the target translation with the guiding of pseudo-translation, which regards the translation of each word as NAT: \[\label{eq:word-translation} P(\mathbf{Y}|\mathbf{Z},\mathbf{X}) = \prod_{i=1}^m P(y_i|\mathbf{Z},\mathbf{X}). \tag{3}\]

As shown in Figure 1, the encoder module and the decoder module can be viewed as a seq-to-seq model which translates the source sentence to target sentence. Different from original NAT, the input of our decoder module is the embeddings of pseudo-translation instead of copied embeddings of source sentence, which is used to guide the word selection. Note that the encoder outputs are also fed into the decoder attention module, which can help alleviate the reordering errors.

To make the model parameter number comparable with the baseline model, we use \(K\) and \(N-K\) decoder blocks for the reordering and decoder modules respectively ⁴.

3.2 Guiding Decoding Strategy↩︎

ReorderNAT explicitly models reordering information of NAT and aims to utilize it to alleviate the multimodality problem. Now the remaining problem is how to perform decoding with the guide of reordering information. We propose to utilize the pseudo-translation as a bridge to guide the decoding of the target sentence, which can be formulated as: \[\begin{align} \label{eq:split} \mathbf{Y}^*&=& \mathop{\arg\max}_{\mathbf{Y}}P(\mathbf{Y}|\mathbf{X}) \nonumber\\ &=& \mathop{\arg\max}_{\mathbf{Y}}\sum_{\mathbf{Z}} P(\mathbf{Y}|\mathbf{Z},\mathbf{X})P(\mathbf{Z}|\mathbf{X}). \end{align}\tag{4}\]

It is intractable to obtain an exact solution for maximizing Eq. 4 due to the high time complexity. Inspired by the pre-ordering works in statistical machine translation, we propose a deterministic guiding decoding (DGD) strategy and a non-deterministic guiding decoding (NDGD) strategy to solve this problem.

The DGD strategy first generates the most probable pseudo-translation of the source sentence, and then translates the source sentence conditioned on it: \[\begin{align} \mathbf{Z}^* &=& \mathop{\arg\max}_{\mathbf{Z}}P(\mathbf{Z}|\mathbf{X}), \\ \mathbf{Y}^* &=& \mathop{\arg\max}_{\mathbf{Y}} P(\mathbf{Y}|\mathbf{Z}^*,\mathbf{X}).\label{eq:split1}\end{align}\tag{5}\]

The DGD strategy is simple and effective, but the hard approximation may bring in some noises. Different from the DGD strategy which utilizes a deterministic pseudo-translation, the NDGD strategy regards the probability distribution \(\mathbf{\mathcal{Q}}\) of words to generate the most probable pseudo-translations as a latent variable, and models the translation as generating the target sentence according to \(\mathbf{\mathcal{Q}}\), i.e., Eq. 5 is re-formulated as: \[\begin{align} \label{eq:ndgd} \mathbf{Y}^* &=& \mathop{\arg\max}_{\mathbf{Y}}P(\mathbf{Y}|\mathbf{\mathcal{Q}}, \mathbf{X}),\end{align}\tag{6}\] where \(\mathbf{\mathcal{Q}}\) is defined as: \[\label{eq:q} \mathbf{\mathcal{Q}}(z_i) = P(z_i|\mathbf{z}^*_{<i}, \mathbf{X}) = \frac{\exp\big(s(z_i)/T\big)}{\sum_{z_i'}\exp\big(s(z_i')/T\big)}, \tag{7}\] where \(s(\cdot)\) is a score function of pseudo-translation (the input of softmax layer in the decoder) and \(T\) is a temperature coefficient. Since \(\mathbf{\mathcal{Q}}\) can be viewed as a non-deterministic form of the pseudo-translation, the translation with the NDGD strategy is also guided by the pseudo-translation.

To be specific, as shown in Figure 1, the major difference between DGD and NDGD strategy is the inputs of decoder module (No. 2 dashed arrow), where the DGD strategy directly utilizes the word embeddings of generated pseudo-translation and the NDGD strategy utilizes the word embeddings weighted by the word probability of pseudo-translation. The detailed architecture of ReorderNAT model is introduced in Appendix.

3.3 Discussion↩︎

In ReorderNAT, the decoding space of generating pseudo-translation with reordering module is much smaller than that of the whole translation in NAT since the decoding vocabulary is limited to the words in the source sentence. The reordering module is more likely to be guided to one pseudo-translation among multiple alternatives. Therefore, ReorderNAT could easily capture the reordering information compared to the original NAT by explicitly modeling with pseudo-translation as internal supervision. Besides, the candidates of the \(i\)-th word of the final translation can be narrowed to the translations of \(z_i\) to some extent since \(z_i\) is the \(i\)-th word in the pseudo-translation which indicates the corresponding source information of \(y_i\). In other words, pseudo-translations could be viewed as a translation written in source language which helps the decoder to capture the dependencies among target words and choose words belonging to the same translation.

3.4 Training↩︎

In the training process, for each training sentence pair \((\mathbf{X}, \mathbf{Y}) \in D\), where \(D\) is the training set, we first generate its corresponding pseudo-translation \(\hat{\mathbf{Z}}\)⁵. And then ReorderNAT is optimized by maximizing a joint loss: \[\mathcal{L} = \mathcal{L}_{R} + \mathcal{L}_{T},\] where \(\mathcal{L}_{R}\) and \(\mathcal{L}_{T}\) indicate the reordering and translation losses respectively. Formally, for both DGD and NDGD, the reordering loss \(\mathcal{L}_{R}\) is defined as⁶ : \[\mathcal{L}_{R} = \sum_{(\mathbf{X}, \hat{\mathbf{Z}}, \mathbf{Y})\in D} \log P(\hat{\mathbf{Z}}|\mathbf{X}).\]

For the DGD approach, the translation loss is defined as an overall maximum likelihood of translating pseudo-translation into the target sentence: \[\mathcal{L}_{T} = \sum_{(\mathbf{X}, \hat{\mathbf{Z}}, \mathbf{Y})\in D} \log P(\mathbf{Y}|\hat{\mathbf{Z}},\mathbf{X}),\]

For the NDGD approach, the translation loss is defined as an overall maximum likelihood of decoding the target sentence from the conditional probability of pseudo-translation: \[\mathcal{L}_{T} = \sum_{(\mathbf{X}, \hat{\mathbf{Z}}, \mathbf{Y})\in D} \log P(\mathbf{Y}|\mathbf{\mathcal{Q}},\mathbf{X}).\]

In particular, we use the trained model for the DGD approach to initialize the model for the NDGD approach since if \(\mathbf{\mathcal{Q}}\) is not well trained, \(\mathcal{L}_{T}\) will converge very slowly.

4 Experiments↩︎

4.1 Datasets↩︎

The main experiments are conducted on three widely-used machine translation tasks: WMT14 En-De (\(4.5\)M pairs), WMT16 En-Ro (\(610\)k pairs) and IWSLT16 En-De (\(196\)k pairs)⁷. For WMT14 En-De task, we take newstest-2013 and newstest-2014 as validation and test sets respectively. For WMT16 En-Ro task, we employ newsdev-2016 and newstest-2016 as validation and test sets respectively. For IWSLT16 En-De task, we use test2013 for validation.

We also conduct our experiments on Chinese-English translation which differs more in word order. The training set consists of \(1.25\)M sentence pairs extracted from the LDC corpora. We choose NIST 2002 (MT02) dataset as validation set, and NIST 2003 (MT03), 2004 (MT04), 2005 (MT05), 2006 (MT06) and 2008 (MT08) datasets as test sets.

4.2 Experimental Settings↩︎

We follow most of the model hyperparameter settings in [5], [6], [10] for fair comparison. For IWSLT16 En-De, we use a \(5\)-layer Transformer model (\(d_{model}=278\), \(d_{hidden}=507\), \(n_{head}=2\), \(p_{dropout}=0.1\)) and anneal the learning rate linearly (from \(3 \times 10^{-4}\) to \(10^{-5}\)) as in [10]. For WMT14 En-De, WMT16 En-Ro and Chinese-English translation, we use a \(6\)-layer Transformer model (\(d_{model}=512\), \(d_{hidden}=512\), \(n_{head}=8\), \(p_{dropout}=0.1\)) and adopt the warm-up learning rate schedule [9] with \(t_{warmup}=4000\). For the GRU reordering module, we set it to have the same hidden size with the Transformer model in each dataset. We employ label smoothing of value \(\epsilon_{ls}=0.15\) and utilize the sequence-level knowledge distillation [11] for all tasks. For each dataset, we also select the optimal guiding decoding strategy according to the model performance on validation sets⁸. We also set \(T\) in Eq. 7 to \(0.2\) according to a grid search on the validation set (see more details in Appendix). We measure the model inference speedup on the validation set of IWSLT16 En-De task with a NVIDIA P40 GPU and set batch size to \(1\).

4.3 Baselines↩︎

In the experiments, we compare ReorderNAT(NAT) and ReorderNAT(AT) which utilize an NAT and an AT reordering modules respectively with several baselines⁹.

We select three models as our autoregressive baselines: (1) Transformer\(_{full}\) [9], which is the teacher model used in the knowledge distillation and of which the hyperparameters are described in experimental settings. (2) Transformer\(_{one}\), a lighter version of Transformer, of which the decoder layer number is \(1\). (3) Transformer\(_{gru}\), which replaces the decoder of Transformer\(_{full}\) with a single-layer GRU [12]. And we set the beam size to \(4\) in the experiments.

Besides, we compare with several typical NAT models, which also have the ability to alleviate the multimodality problem and are highly relevant to our work: (1) NAT-IR [10], which adopts an iterative refinement mechanism for better translation quality;(2) NAT-FS [13], which introduces the autoregressive property to the top decoder layer of NAT; (3) FlowSeq-base [14], which uses generative flow to help model dependencies within target sentences. For fair comparison, We use the “base” version as it has comparable model size with our model; (4) imitate-NAT [6], which imitates the behavior of an AT model. (5) CMLM-small [15], which is built on a conditional masked language model and also iteratively refines the translation. We use the “small” version for fair comparison; (6) NART-DCRF [16], which uses CRF to capture the word dependencies; (7) LevT [17], which models the sequence generation as multi-step insertion and deletion operations.

3pt

4.4 Overall Results↩︎

We compare ReorderNAT(NAT) and ReorderNAT(AT) that utilize an NAT reordering module and an AT reordering module respectively with all baseline models. All the results are shown in Table 1. From the table, we can find that:

(1) Excluding six better BLEU scores with significant low speedup, ReorderNAT(AT) achieves the best performance on most of the benchmark datasets, which is even close to the AT model with smaller than \(1\) BLEU gap (\(26.49\) vs. \(27.17\) on WMT14 En\(\to\)De task, \(31.99\) vs. \(32.60\) on WMT16 Ro\(\to\)En task, \(30.26\) vs. \(31.18\) on IWSLT16 En\(\to\)De task). It is also worth mentioning that although ReorderNAT utilizes a small AT model to better capture reordering information, it could still maintain low translation latency (about \(16\times\) speedup for ReorderNAT(NAT) and \(6\times\) speedup for ReorderNAT(AT)). Compared to Transformer\(_{one}\) and Transformer\(_{gru}\), ReorderNAT(AT) uses a much smaller vocabulary in the AT reordering module, which is limited to the words in the source sentence and makes it faster. Besides, unlike NAT-IR, CMLM-small and LevT, our model can decode all target words in parallel without multiple iterations, which helps maintain the efficiency.

(2) ReorderNAT(NAT) and ReorderNAT(NAT)+LPD also gain significant improvements compared to most existing NAT models. It verifies the reordering information modeled by ReorderNAT could effectively guide the translation.

(3) A small AT model with close latency to large NAT models could perform much better in modeling reordering information¹⁰. On all benchmark datasets, ReorderNAT(AT) with small AT GRU reordering module achieves much better translation quality than that with large NAT model (\(2\)-\(5\) BLEU scores). Moreover, we find that the AT model Transformer\(_{one}\) and Transformer\(_{gru}\) with a single-layer AT Transformer or GRU for decoding could also outperform most existing NAT models while maintaining acceptable latency (\(2.42\times\) and \(3.10\times\) speedup respectively). The reason is that a major potential performance degradation of NAT models compared to AT models comes from the difficulty of modeling the reordering information, which is neglected by most existing NAT models but can be well modeled by the small AT module¹¹.

4.5 Results on Chinese-English Translation↩︎

To show the effectiveness of modeling reordering information in NAT, we compare ReorderNAT with baselines on Chinese-English translation since the word order between Chinese and English is more different than that between German and English (En-De). From Table 2, we can find that in Chinese-English translation, ReorderNAT(AT) achieves much more improvements (\(6\)-\(7\) BLEU points) compared to ReorderNAT(NAT) and imitate-NAT. The reason is that more different word order in Chinese-English translation makes the decoding search space more complicated, which could be effectively alleviated by ReorderNAT.

4.6 Translation Quality v.s. Sentence Lengths↩︎

Figure 2 shows the BLEU scores of translations generated by the Transformer (AT model), the NAT model (ReorderNAT without the reordering module), ReorderNAT(NAT) and ReorderNAT(AT) on the IWSLT16 validation set with respect to input sentence lengths¹². We can observe that:

(1) ReorderNAT(NAT) and ReorderNAT(AT) achieve significant improvement compared to the NAT model for most lengths and ReorderNAT(AT) achieves nearly comparable performance to Transformer. It verifies the reordering information modeled by ReorderNAT could effectively help word selection and improve the translation quality.

(2) ReorderNAT(AT) achieves much better translation performance than the NAT model for sentences longer than \(20\) words, of which word order tends to be more different. Together with the results on Chinese-English translation (Table 2), we can conclude that NAT is weak on word reordering and our model is more effective especially when word order is more different.

4.7 Multimodality Related Error Reduction↩︎

In this section, we investigate how our reordering module reduces the multimodality errors in NAT. Specially, we evaluate the RIBES [18] score and the reduction of duplicate and missing words (two most typical multimodality related errors). The results are shown in Table 3, where “Dup” and “Mis” denote the relative increment of duplicate and missing token ratios compared with the Transformer\(_{full}\) model respectively¹³ , and NAT is ReorderNAT without the reordering module. From the table, we can observe that:

(1) Our three ReorderNAT models achieve higher RIBES scores than the NAT model, validating our reordering module can help capture the word order difference between source and target language. Moreover, the ReorderNAT(AT) model performs the best in RIBES, indicating the AT reordering module can model reordering information more effectively than the NAT reordering module.

(2) Compared with the NAT model, both Dup and Mis are significantly better for the three ReorderNAT models, indicating ReorderNAT is effective for alleviating the multimodality problem.

4.8 Case Study↩︎

Table 4 shows example translations of the NAT model (ReorderNAT without the reordering module), ReorderNAT(NAT) and ReorderNAT(AT). We find that the problem of missing and duplicate words are severe in the translation (both \(5\) occurrences) of the NAT model, while this problem is effectively alleviated by ReorderNAT. Moreover, we find that most of the missing, duplicate or wrong words in the translation of ReorderNAT(NAT) and ReorderNAT(AT) come from the errors in the pseudo-translation, demonstrating that NAT models could well translate the pseudo-translation which is in the the target language word order, and the remaining problem of NAT lies on modeling reordering information.

5 Related Work↩︎

5.1 Non-Autoregressive Neural Machine Translation↩︎

[5] first proposed the non-autoregressive neural machine translation (NAT), which enables parallel decoding for neural machine translation (NMT) and significantly accelerates the inference of NMT. However, its performance degrades greatly since it discards the sequential dependencies among words in the target sentence. Recently, a variety of works have been investigated to improve the performance of NAT in various aspects including [8], [20] which enhance the representation of decoder with source information; [13], [21]–[23] which optimize models with respect to sequence-level loss functions; [24] which incorporates monolingual data for training; [7], [25] which alleviate the multimodality problem by making the decoder representation more informative via regularization or speical decoding strategies; [6], [26]–[29] which guide the learning of NAT models with AT models; [30]–[32] which introduce latent variables to guide the decoding process of NAT models; and [10], [15], [17], [19], [33]–[38] which extend one-step NAT to multi-step NAT and generate translations iteratively. Different from existing works, we propose to explicitly model reordering information in NAT models, which serves as a proxy in capturing the dependencies of the target words and encourages the decoder to choose words belonging to the same translation to alleviate the multimodality problem. This work intends to enhance the translation quality of one-step NAT models and has the potential to improve the performance of each iteration of the multi-step NAT methods without loss of efficiency.

5.2 Modelling Reordering Information in Machine Translation↩︎

Re-ordering model is a key component in statistical machine translation (SMT), which handles the word order differences between source and target language. There has been a large number of works focusing on word reordering in SMT, including deterministic reordering approaches [39]–[42], which find a single optimal reordering of the source sentence; non-deterministic reordering approaches [43], [44] which encode multiple alternative reorderings into a word lattice and remain the choosing strategy of best path in the decoder; and target word reordering approaches [45] which first select target lexicals and then reorder them to form final sentence. In neural machine translation (NMT), it has been shown that the attention mechanism [2] could implicitly capture the reordering information to some extent. [46] presented three distortion models to further incorporate reordering knowledge into attention-based NMT models. [47] introduced a reordering mechanism for NMT models to learn the reordering embedding of a word based on its contextual information. Except for incorporating reordering knowledge in attention mechanism, researchers also proposed to learn to reorder the source-side word orders according to the sentence structure in target language with neural networks [48]–[50]. This work empirically justifies that reordering information is essential for NAT.

6 Conclusion and Future Work↩︎

In this work, to address the multimodality problem in NAT, we propose a novel NAT framework named ReorderNAT which explicitly models the reordering information in the decoding procedure. We further introduce deterministic and non-deterministic guiding decoding strategies to utilize the reordering information to encourage the decoder to choose words belonging to the same translation. Experimental results on several widely-used benchmarks show that our ReorderNAT model achieves better performance than most existing NAT models, and even achieves comparable translation quality as AT model with a significant speedup. We believe that to well model the reordering information is a potential way towards better NAT.

7 Model Architecture↩︎

In this section, we will introduce the details of the encoder, reordering and decoder modules in Figure 3 respectively.

7.1 Encoder and Decoder Blocks↩︎

We first introduce the encoder and decoder blocks used in the three modules.

In our proposed ReorderNAT model, we employ the Transformer encoder block [9] as our encoder block, which is composed by a multi-head self-attention layer and two linear transformation layers. It could be formulated as: \[E(\mathbf{H}) = \texttt{FFN}(\texttt{Self-Att}(\mathbf{H})), \label{enc-block}\tag{8}\] where \(\mathbf{H}\) is the input embeddings of the encoder block, \(\texttt{FFN}(\cdot)\) denotes the two linear transformation layers and \(\texttt{Self-Att}(\cdot)\) is the multi-head self-attention layer.

We employ two types of decoder blocks:

(1) Transformer decoder block [9], which is composed by a multi-head self-attention layer, a multi-head inter-attention layer and two linear transformation layers. It is formulated as: \[\begin{align} D(\mathbf{H}, \mathbf{S}) &=& \texttt{FFN}(\texttt{Inter-Att}(\mathbf{S},\nonumber\\ &&(\texttt{Self-Att}(\mathbf{H}))),\end{align}\] where \(\mathbf{H}\) is the input embeddings of the decoder block, \(\mathbf{S}\) is the hidden representation of the source sentence, and \(\texttt{Inter-Att}(\cdot)\) is the multi-head inter-attention layer.

(2) GRU decoder block, which consists of a multi-head inter-attention layer and a GRU layer. Its hidden state is computed as \[\mathbf{D}_i=\texttt{GRU}(\mathbf{D}_{i-1}, [\boldsymbol{C}_{i-1}; \mathbf{H}_{i-1}]),\] where \(\mathbf{H}_{i-1}\) is the embedding of the previous word in the target sentence, \(\boldsymbol{C}_{i-1}\) is calculated by the inter-attention to the encoder representation, and \(\texttt{GRU}(\cdot)\) is a gated recurrent unit (GRU) [12].

7.2 Encoder Module↩︎

In the ReorderNAT model, we feed the input sentence embedding \(\texttt{Emb}(\mathbf{X})\) into an encoder \(E^N(\cdot)\) consisting of \(N\) encoder blocks \(E(\cdot)\) (see Eq. 8 ) to obtain the hidden representation \(\mathbf{S}\) of the source sentence \(\mathbf{X}\): \[\mathbf{S} = E^N(\texttt{Emb}(\mathbf{X})).\]

7.3 Reordering Module↩︎

In the reordering module, we employ an NAT model with Transformer decoder block or an AT model with GRU decoder block as the pseudo-translation decoder. Formally, for the NAT Reordering Module, we utilize the uniform copied embeddings as [5] for the input of the decoder, and then obtain the hidden representation \(\mathbf{R}\) in reordering module: \[\mathbf{R} = D(\texttt{Uniform-Copy}(\texttt{Emb}(\mathbf{X})), \mathbf{S}).\]

After obtaining the decoder hidden state \(\mathbf{R}\), the conditional probability \(P(z_i|\mathbf{X})\) of the \(i\)-th word in the pseudo-translation is computed as: \[\label{eq:z} P(z_i|\mathbf{X}) = \texttt{Softmax}(\mathbf{R}_i),\tag{9}\] where \(\mathbf{R}_i\) is the \(i\)-th column of \(\mathbf{R}\).

For the AT Reordering Module, we use a greedy search algorithm to decode the pseudo-translation word-by-word. When decoding the \(i\)-th word \(z_i\) in the pseudo-translation, we obtain its hidden representation as: \[\mathbf{R}_i=\texttt{GRU}(\boldsymbol{R}_{i-1}, [\boldsymbol{C}_{i-1}; Emb(z^*_{i-1})]),\] where \(z*_{i-1}\) is the previous word in the decoded pseudo-translation and \(\boldsymbol{C}_{i-1}\) is calculated by the inter-attention to the encoder representation. After obtaining the hidden representation, the conditional probability \(P(z_i|\mathbf{X})\) is also calculated as Eq. 9 .

7.4 Decoder Module↩︎

In the decoder module, we employ an NAT decoder with \(N\)-\(1\) layers of Transformer decoder block. We utilize two types of input representation for deterministic guiding decoding (DGD) strategy and non-deterministic guiding decoding (NDGD) strategy. Formally, for DGD strategy, we use the word embeddings of the predicted pseudo-translation \(\mathbf{Z}^*\) as the input of the decoder, and then obtain the final translation probability: \[P(\mathbf{Y}|\mathbf{Z}^*, \mathbf{X}) = \texttt{Softmax}( D^{N-1}(\texttt{Emb}(\mathbf{Z}^*), \mathbf{S})),\] where \(D^{N-1}(\cdot)\) indicates stacking \(N-1\) layers of Transformer decoder block \(D(\cdot)\).

For NDGD strategy, we use the word embeddings weighted by the conditional probability \(\mathbf{\mathcal{Q}}\) (i.e. \(\mathbf{\mathcal{Q}}(\mathbf{Z}) = P(\mathbf{Z}|\mathbf{X})\)) of pseudo-translation as the input of the decoder, and then obtain the final translation probability: \[P(\mathbf{Y}|\mathbf{\mathcal{Q}}, \mathbf{X}) = \texttt{Softmax}( D^{N-1}(\mathbf{\mathcal{Q}}^T\texttt{Emb}(\mathbf{X}), \mathbf{S})).\]

8 Pseudo-Translation Construction↩︎

We use the fast_align tool¹⁴ to generate the pseudo-translation used for training in our experiments. Specifically, we use fast_align to obtain word alignments between source and target sentences¹⁵. The maximum number of source words aligned to each target word is set to 1. For each target word, the corresponding word in the pseudo-translation is set as the source word aligned to it (We add a special symbol if the target word does not align to any source word.).

9 Other Experimental Results↩︎

9.1 Overall Performance↩︎

In Table 5, we compare our ReorderNAT model with more NAT baselines: NAT-FT [5], NAT-IR [10], NAT-FS [13], FlowSeq-base [14], imitate-NAT [6], NAT-REG [7], LV NAR [51], NART [26], TCL-NAT [28], CMLM-small [15], NAT-EM [29], FCL-NAT [27], NART-DCRF [16] and LevT [17].

The performance on the validation sets are shown in Table 6 ¹⁶.

3pt

9.2 All Results on Chinese-English Translation↩︎

We show all results on all datasets including MT02, MT03, MT04, MT05, MT06 and MT08 in Table 7.

9.3 Translation Quality v.s. Sentence Lengths↩︎

Figure 2 shows the BLEU scores of translations generated by the Transformer (AT model), the NAT model (ReorderNAT without the reordering module), ReorderNAT(NAT) and ReorderNAT(AT) on the WMT14 and WMT16 test sets with respect to input sentence lengths.

9.4 Multimodality Related Error Reduction↩︎

We show an quantitative investigation of the reduction of the two most typical multimodality related errors, i.e., duplicate and missing words, on WMT14 En-De and WMT16 En-Ro tasks in Table 3.

9.5 Effect of Guiding Decoding Strategy↩︎

We show the results of ReorderNAT(NAT) and ReorderNAT(AT) using DGD and NDGD on all datasets in Table 9.

9.6 Effect of Layer Number of Decoder Module↩︎

To compare the NAT and AT baselines with the same model complexity, we utilize \(N\)-\(1\) layers of Transformer decoder blocks for the decoder module in the experiments. However, we argue that the decoding space of decoder module is quite small and may be modeled by smaller architecture. To verify our assumption, we compare the translation quality of ReorderNAT model with different Transformer layer numbers in decoder module on the IWSLT16 validation set. Figure 5 shows the results, from which we can see that our ReorderNAT model also performs well with smaller decoder module, which demonstrates that our model can be further speedup.

9.7 Effect of Temperature Coefficient \(T\)↩︎

The temperature coefficient \(T\) controls the smoothness of the \(\mathbf{\mathcal{Q}}\) distribution (see Eq. 10 in the main text). As shown in Figure 6, we find that \(T\) affects the BLEU scores on the IWSLT16 validation set to some extent. While \(T=0.1\) deceases BLEU scores, \(T=0.2\) improves translation quality significantly and consistently. However, increasing \(T\) further to \(0.5\) or \(1\) results in worse translation quality compared to \(T=0.2\) after training \(150\)k steps. Hence, we set \(T=0.2\) for the NDGD strategy in our experiments.

9.8 Effect of Alignment Quality↩︎

To investigate the effect of the alignment quality for training the pseudo-translation, we evaluate the performance of our ReorderNAT model on IWSLT16 validation set when varying the amount of the training data used in fast_align. The results are shown in Table 10. From the table, we can observe that the BLEU scores are similar when using different proportions of training data for fast_align, which indicates that our ReorderNAT model is not sensitive to the alignment quality. Moreover, our ReorderNAT model achieves higher BLEU scores on most proportions when using the NDGD strategy than using the DGD strategy, probably because the NDGD strategy could help mitigate the noise of pseudo-translation in decoding translations.

9.9 Speedup on CPU↩︎

The speedup of our ReorderNAT model and AT baselines is shown in Table 11. Mathematical operations such matrix multiplication on CPUs are not as effective as on GPUs. Therefore, the speedup on CPU is smaller than that on GPU. One exception is that for our ReorderNAT(AT) model. We believe the reason is that the AT reordering module is small such that it can be also computed effectively on CPU. Taking results in Table 5 into consideration, we conclude that ReorderNAT(AT) should be a better choice for CPU as it achieves a better trade-off between speedup and translation quality.

9.10 Parameter Number↩︎

The parameter numbers of our models are shown in Table 12, which are comparable with those of the baselines.

References↩︎