1 Introduction↩︎

The field of machine translation (MT) is making amazing progress, thanks to the advent of neural models and deep learning. While just few years ago research in MT was struggling to achieve useful translations for the most requested and high-resourced languages, the level of translation quality reached today has raised the demand and interest for less-resourced languages and the solution of more subtle and interesting translation tasks [1]. If the goal of machine translation is to help worldwide communication, then the time has come to also cope with dialects or more generally language varieties¹. Remarkably, up to now, even standard national language varieties, such as Brazilian and European Portuguese, or Canadian and European French, which are used by relatively large populations have been quite neglected both by research and industry. Prominent online commercial MT services, such as Google Translate and Bing, are currently not offering any variety of Portuguese and French. Even worse, systems offering such languages tend to produce inconsistent outputs, like mixing lexical items from different Portuguese (see for instance the translations shown in Table ¿tbl:table:examples?). Clearly, in the perspective of delivering high-quality MT to professional post-editors and final users, this problem urges to be fixed.

While machine translation from many to one varieties is intuitively simpler to approach², it is the opposite direction that presents the most relevant problems. First, languages varieties such as dialects might significantly overlap thus making differences among their texts quite subtle (e.g., particular grammatical constructs or lexical divergences like the ones reported in the example). Second, parallel data are not always labeled at the level of language variety, making it hard to develop specific NMT engines. Finally, training data might be very unbalanced among different varieties, due to the population sizes of their respective speakers or for other reasons. This clearly makes it harder to model the lower-resourced varieties [2].

In this work we present our initial effort to systematically investigate ways to approach NMT from English into four pairs of language varieties: Portuguese European - Portuguese Brazilian, European French - Canadian French, Serbian - Croatian, and Indonesian - Malay³. For each couple of varieties, we assume to have both parallel text labeled with the corresponding couple member, and parallel text without such information. Moreover, the considered target pairs, while all being mutually intelligible, present different levels of linguistic similarity and also different proportions of available training data. For our tasks we rely on the WIT\(^3\) TED Talks collection⁴, used for the International Workshop of Spoken Language Translation, and OpenSubtitles2018, a corpus of subtitles available from the OPUS collection⁵.

After presenting related work (Section 2) on NLP and MT of dialects and related languages, we introduce (in Section 3) baseline NMT systems, either language/dialect specific or generic, and multilingual NMT systems, either trained with fully supervised (or labeled) data or with partially supervised data. In Section 4, we introduce our datasets, NMT set-ups based on the Transformer architecture, and then present the results for each evaluated system. We conclude the paper with a discussion and conclusion in Sections 5 and 6.

2 Related work↩︎

2.1 Machine Translation of Varieties↩︎

Most of the works on translation between and from/to written language varieties involve rule-based transformations, e.g., for European and Brazilian Portuguese [3], Indonesian and Malay [4], Turkish and Crimean Tatar [5]; or phrase-based statistical MT (SMT) systems, e.g., for Croatian, Serbian, and Slovenian [6], Hindi and Urdu [7], or Arabic dialects [8]. Notably, build an unsupervised deciphering model to translate between closely related languages without parallel data. handle mixed Arabic dialect input in MT by using a sentence-level classifier to select the most suitable model from an ensemble of multiple SMT systems. In NMT, however, there have been fewer studies addressing language varieties. It is reported that an RNN model outperforms SMT when translating from Catalan to Spanish [9] and from European to Brazilian Portuguese [10]. propose a technique to augment training data for under-resourced dialects via projecting word embeddings from a resource-rich related language, thus enabling training of dialect-specific NMT systems. The authors generate spoken Levantine-English data from larger Arabic-English corpora and report improvement in BLEU scores compared to a low-resourced NMT model.

2.2 Dialect Identification↩︎

A large body of research in dialect identification stems from the DSL shared tasks [11]–[14]. Currently, the best-performing methods include linear machine learning algorithms such as SVM, naïve Bayes, or logistic regression, which use character and word \(n\)-grams as features and are usually combined into ensembles [15]. present the idea of leveraging parallel corpora for language identification: content comparability allows capturing subtle linguistic differences between dialects while avoiding content-related biases. The problem of ambiguous sentences, i.e., those for which it is impossible to decide upon the dialect tag, has been demonstrated for Portuguese by through inspection of disagreement between human annotators.

2.3 Multilingual NMT↩︎

In a one-to-many multilingual translation scenario, proposed a multi-task learning approach that utilizes a single encoder for source languages and separate attention mechanisms and decoders for every target language. used distinct encoder and decoder networks for modeling language pairs in a many-to-many setting. introduced a way to share the attention mechanism across multiple languages. A simplified and efficient multilingual NMT approach is proposed by and by prepending language tokens to the input string. This approach has greatly simplified multi-lingual NMT, by eliminating the need of having separate encoder/decoder networks and attention mechanism for every new language pair. In this work we follow a similar strategy, by incorporating an artificial token as a unique variety flag.

3 NMT into Language Varieties↩︎

Our assumption is to translate from language \(E\) (English) into each of two varieties \(A\) and \(B\). We assume to have parallel training data \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\) for each variety as well as unlabeled data \(D_{E\rightarrow A\cup B}\). For the sake of experimentation we consider three application scenarios in which a fixed amount of parallel training data \(E\)-\(A\) and \(E\)-\(B\) is partitioned in different ways:

• Supervised: all sentence pairs are respectively put in \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\), leaving \(D_{E\rightarrow A\cup B}\) empty;

• Unsupervised: all sentence pairs are jointly put in \(D_{E\rightarrow A\cup B}\), leaving \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\) empty;

• Semi-supervised: two-third of \(E\)-\(A\) and \(E\)-\(B\) are, respectively, put in \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\), and the remaining sentence pairs are put in \(D_{E\rightarrow A\cup B}\).

Supervised and Unsupervised Baselines. For each translation direction we compare three baseline NMT systems. The first system is an unsupervised generic (Gen) system trained on the union of the language varieties training data. Notice that Gen makes no distinction between \(A\) and \(B\) and uses all data in an unsupervised way. The second is a supervised variety-specific system (Spec) trained on the corresponding language variety training set. The third system (Ada) is obtained by adapting the Gen system to a specific variety. ⁶ Adaptation is carried out by simply restarting the training process from the generic model using all the available variety specific training data.

Supervised Multilingual NMT. We build on the idea of multilingual NMT (Mul), where one single NMT system is trained on the union of \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\). Each source sentence both at training and inference time is prepended with the corresponding target language variety label (\(A\) or \(B\)). Notice that the multilingual architecture leverages the target forcing symbol both as input to the encoder to build its states, and as initial input to the decoder to trigger the first target word.

Semi-Supervised Multilingual NMT. We consider here multilingual NMT models that make also use of unlabeled data \(D_{E\rightarrow A\cup B}\). The first model we propose, named M-U, uses the available data \(D_{E\rightarrow A}\), \(D_{E\rightarrow B}\) and \(D_{E\rightarrow A\cup B}\) as they are, by not specifying any label at training time for entries from \(D_{E\rightarrow A\cup B}\). The second model, named M-C2, works similarly to Mul, but relying on a language variety identification module (trained on the target data of \(D_{E\rightarrow A}\) and \(D_{E\rightarrow B}\)) that maps each unlabeled data point either to \(A\) or \(B\). The third model, named M-C3, can be seen as an enhancement of M-U, as the unlabeled data is automatically classified into one of three classes: \(A\), \(B\), or \(A\cup B\). For the third class, like with M-U, no label is applied on the source sentence.

4 Experimental Set-up↩︎

4.1 Dataset and Preprocessing↩︎

The experimental setting consists of eight target varieties and English as source. We use publicly available datasets from the WIT\(^3\) TED corpus [16]. The summary of the partitioned training, dev, and test sets are given in Table 1, where Tr. 2/3 is the labeled portion of the training set used to train the semi-supervised models, while the other 1/3 are either held out as unlabeled (M-U) or classified automatically (M-C2, M-C3). In the preprocessing stages, we tokenize the corpora and remove lines longer than 70 tokens. The Serbian corpus written in Cyrillic is transliterated into Latin script with CyrTranslit⁷. In addition, to also run a large-data experiment, we expand the English\(-\)European/Brazilian Portuguese data with the corresponding OpenSubtitles2018 datasets from the OPUS corpus. Table 1 summarizes the augmented training data, while keeping the same dev and test sets.

4.2 Experimental Settings↩︎

We trained all systems using the Transformer model⁸ [17]. We use the Adam optimizer [18] with an initial learning rate of \(0.2\) and a dropout also set to \(0.2\). A shared source and target vocabulary of size 16k is generated via sub-word segmentation [19]. The choice for the vocabulary size follows the recommendations in regarding training of NMT systems on TED Talks data. Overall we use a uniform setting for all our models, with a \(512\) embedding dimension and hidden units, and 6 layers of self-attention encoder-decoder network. The training batch size is of \(6144\) sub-word tokens and the max length after segmentation is set to \(70\). Following and for a fair comparison, experiments are run for 100k training steps, i.e., in the low-resource settings all models are observed to converge within these steps. Adaptation experiments are run to convergence, which requires roughly half of the steps (i.e., 50k) required to train the generic low-resource model. On the other hand, large-data systems are trained for up to 800k steps, which also showed to be a convergence point. For the final evaluation we take the best performing checkpoint on the dev set. All models are trained using Tesla V100-pcie-16gb on a single GPU.

4.3 Language Variety Identification↩︎

To automatically identify the language variety of unlabeled target sentences, we train a fastText model [20], a simple yet efficient linear bag of words classifier. We use both word- and character-level \(n\)-grams as features. In the low-resource condition, we train the classifier on the 2/3 portion of the labeled training data. For the large-data experiment, instead, we used a relatively smaller and independent corpus consisting of 3.3 million pt-BR\(-\)pt-EU parallel sentences extracted from OpenSubtitles2018 after filtering out identical sentences pairs and sentences occurring (in any of the two varieties) in the NMT training data. Additionally, low-resource training sentences (fr-CA and ms) are randomly oversampled to mitigate class imbalance.

For each pair of varieties, we train five base classifiers differing in random initialization. In the M-C2 experiments, prediction is determined based on soft fusion voting, i.e., the final label is the argmax of the sum of class probabilities. Due to class skewness in the evaluation set, we report binary classification performance in terms of ROC AUC [21] instead of accuracy in Table 2. For M-C3 models, we handle ambiguous examples using the majority voting scheme: in order for a label to be assigned, its softmax probability should be strictly higher than fifty percents according to the majority of the base classifiers, otherwise no tag is applied. On average, this resulted in \(<\)​1% of unlabeled sentences for the small data condition, and about 2% of unlabeled sentences for the large data condition.

5 Results and Discussion↩︎

We run experiments with all the systems introduced in Section 3, on four pairs of languages varieties. Results are reported in Table 3 for the low-resource setting and in Table 4 for the large data setting.

5.1 Low-resource setting↩︎

Among the supervised models, which are using all the available training data, the multilingual NMT model Mul outperforms the variety-specific models on all considered directions. Remarkably, the Mul model also outperforms the adapted Ada model on the available translation directions. The unsupervised generic model Gen, that mixes together all the available data, as expected tends to perform better than the supervised specific models of the less resourced varieties. Particularly, this improvement is observed for Malay (ms) and Canadian French (fr-CA), which respectively represent the 3.3% and 10% of the overall training data used by their corresponding (Gen) systems. On the contrary, a degradation is observed for European Portuguese (pt-Eu) and Serbian (sr), which represent 42% and 45% of their respective training sets. Even though very low-resourced varieties can benefit from the mix, it is also evident that the Gen model can easily get biased because of the imbalance between the datasets.

In the semi-supervised scenario, we report results with three multilingual systems that integrate the 1/3 of unlabeled data to the training corpus in three different ways: (i) without labels (M-U), (ii) with automatic labels forcing one of two possible classes (M-C2), (iii) with automatic labels of one of the two options or no label in case of low confidence of the classifier (M-C3).

Results show that on average automatic tagging of the unlabeled data is better than leaving them unlabeled, although M-U still remains a better choice than using specialized and generic systems. The best between M-C2 and M-C3 performs on average from very close to better than the best supervised method.

If we look at the single language variety, the obtained figures are not showing a coherent picture. In particular, in the Croatian-Serbian and Indonesian-Malay pairs the best resourced language seems to benefit more from keeping the data unlabeled (M-U). Interestingly, even the worst semi-supervised model performs very close or even better than the best supervised model, which suggests the importance of taking advantage of all available data even if they are not labeled.

Focusing on the statistically significant improvements, the best supervised (Mul) is better than the unsupervised (Gen), whereas the best semi-supervised (M-C2 or M-C3) is either comparable or better than the best supervised.

5.2 High-resource setting↩︎

Unlike what observed in the low-resource setting, where Mul outperforms Spec in the supervised scenario, in the large data condition, variety specific models apparently seem the best choice. Notice, however, that the supervised multilingual system Mul provides just a slightly lower level of performance with a simpler architecture (one network in place of two). The unsupervised generic model Gen, trained with the mix of the two varieties datasets, performs significantly worse than the other two supervised approaches, this is particularly visible for the pt-EU direction. Very likely, in addition to the ambiguities that arise from naively mixing the data of the two different dialects, there is also a bias effect towards pt-BR which is due to the very unbalanced proportions of data between the two dialects (almost 1:2).

Hence, in the considered high-resource setting, the Spec and Mul models result as best possible solutions against which comparing our semi-supervised approaches.

In the semi-supervised scenario, the obtained results confirm that our approach of automatically classifying the unlabeled data \(D_{E\rightarrow A \cup B}\) improves over using the data as they are (M-U). Nevertheless, M-U still confirms to perform better than the fully unlabeled Gen model. In both translation directions, M-C2 and M-C3 get quite close to the performance of the supervised Spec model. In particular, M-C3 shows to outperform the M-C2 model, and even outperforms on average the supervised Mul model. In other words, the semi-supervised model leveraging three-class automatic labels (of \(D_{E\rightarrow A \cup B}\)) seems to perform better than the supervised model with two dialect labels. Besides the comparable BLEU scores, the supervised (Spec and Mul) perform in statistically insignificant way against the best semi-supervised (M-C3), although outperforming the unsupervised (Gen) model.

This result raises the question if relabeling all the training data can be a better option than using a combination of manual and automatic labels. This issue is investigated in the next subsection.

Unsupervised Multilingual Models↩︎

As discussed in Section 4.3, the language classifier for the large-data condition is trained on dialect-to-dialect parallel data that does not overlap with the NMT training data. This condition permits hence to investigate a fully unsupervised training condition. In particular, we assume that all the available training data is unlabeled and create automatic language labels for all 47.2M sentences of pt-BR and 25.5M sentences of pt-EU (see Table 1). In a similar way as in Table 4, we keep the experimental setting of M-C2 and M-C3 models.

Table 5 reports the results of the multilingual models trained under the above described unsupervised condition. In comparison with the semi-supervised condition, both M-C2 and M-C3 show a slight performance improvement. In particular, the three-label M-C3 performs on average slightly better than the two-label M-C2 model. Actually, the little difference is justified by the fact that the classifier used the “third” label only for 6% of the data. Remarkably, despite the relatively low performance of the classifier, average score of the best unsupervised model M-C2 is almost on par with the supervised model Mul.

5.3 Translation Examples↩︎

Finally, in Table ¿tbl:table:examples95more?, we show an additional translation example produced by our semi-supervised multilingual models (both under low and high resource conditions) translating into the Portuguese varieties. For comparison we also include output from Google Translate which offers only a generic English-Portuguese direction. In particular, the examples contain the word refrigerator that has specific dialect variants. All our variety-specific systems show to generate consistent translations of this term, while Google Translate prefers to use the Brazilian translation variants for these sentences.

6 Conclusions↩︎

We presented initial work on neural machine translation from English into dialects and related languages. We discussed both situations where parallel data is supplied or not supplied with target language/dialect labels. We introduced and compared different neural MT models that can be trained under unsupervised, supervised, and semi-supervised training data regimes. We reported experimental results on the translation from English to four pairs of language varieties with systems trained under low-resource conditions. We show that in the supervised regime, best performance is achieved by training a multilingual NMT system. For the semi-supervised regime, we compared different automatic labeling strategies that permit to train multilingual neural MT systems with performance comparable to the best supervised NMT system. Our findings were also confirmed by large scale experiments performed on English to Brazilian and European Portuguese. In this scenario, we have also shown that multilingual NMT fully trained on automatic labels can perform very similarly to its supervised version.

In future work, we plan to extend our approach to language varieties in the source side, as well as investigate the possibility of applying transfer-learning [23], [24] for language varieties by expanding our Ada adaptation approach.

Acknowledgments↩︎

This work has been partially supported by the EC-funded project ModernMT (H2020 grant agreement no. 645487). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. Moreover, we thank the Erasmus Mundus European Program in Language and Communication Technology.

References↩︎