1 Introduction↩︎

Machine translation (MT) is currently one of the hot application areas of deep learning, with neural machine translation (NMT) achieving outstanding performance where large amounts of parallel data are available [1]–[4]. In low-resource settings, transfer learning methods have proven useful for improving system performance [4]–[6]. In this work, we focus on studying NMT, including in the low-resource scenario. In particular, we focus our attention on investigating the effect of language relatedness on the transfer process. We define relatedness of a pair of languages based on belonging to the same language family. That is, by ‘related’ we mean ‘within the same language family’ whereas by ‘unrelated’ we mean ‘belong to two different language families’. For example, we call English and Swedish related since they belong to the Germanic language family but English and French not related since the latter belongs to the Romance language family. As an analogy to human learning, we would like to ask: if there are two translators (pre-trained models), for example one Catalan \(\rightarrow\) Spanish translator and one Catalan \(\rightarrow\) English translator, will they (after extra training, i.e., fine-tuning/transfer learning process) have different abilities to translate from Catalan into Occitan? If the Catalan-Spanish translator proves to perform better Catalan \(\rightarrow\) Occitan, we may attribute this to Spanish and Occitan being members of the Romance language family while English being a member of the, Germanic, different family.

Of particular interest to us are two sets of languages belonging to two different language families, one set to Romance and the other set to Germanic. For the former set, we take Catalan (ca), Italian (it), Occitan (oc), Romanian (ro), and Spanish (es); and we take English (en) for the latter set. We note that both Romance and Germanic are two branches of the larger Indo-European language family, and hence there are some level of relatedness between all the languages we study in this work. Nevertheless, languages in Romance and Germanic differ in some syntactic structures. For example, the position of attributive adjectives in Germanic languages is before the noun while it is after the noun for Romance languages [7]. Despite differences, the writing system of all languages in this work is the Latin script. This can be beneficial to transfer learning because these languages can potentially share common lexical items or morphemes, which may facilitate the transfer learning process.

As mentioned, we adopt transfer learning since it has been shown to improve translation quality for low-resource language pairs. For example, [5] sequentially build a parent model and a child model where each is trained, respectively, on high-resource and low-resource language pairs with the child model retaining parameters from the parent model. In addition, [4] successfully transfer knowledge from a parent model to a child model where both models are trained on low-resource but related language pairs. [6] also adopt a similar approach to these previous works, but based their work on the Transformer architecture [8] instead of using a recurrent encoder-decoder network with attention  [9], [10].

Our work builds on these studies. Namely, we empirically apply transfer learning under different conditions to members of various language families. The outcomes of our work are similar to those of [5], [6]. That is, while we find relatedness to be beneficial, a positive transfer between an unrelated language pair can still be possible (although with a potentially diminished performance).

The rest of this paper is organized as follows: In Section 2, we overview related work. We introduce our datasets and experiments in Section 3. In Section 4, we present and analyze our results. We conclude in Section 5.

2 Background↩︎

2.1 Transfer learning↩︎

Transfer learning is a machine learning approach that aims at transferring the knowledge of one task to another. As an analogy to human learning, one who masters the skills to ride a bicycle may transfer the knowledge to riding a motorcycle because these two tasks share common abilities such as maintaining balance on a two-wheel moving vehicle [11], [12]. We employ transfer learning to port knowledge from a model trained on one pair of languages to another. We now discuss transfer learning in NMT.

2.2 Transfer learning in Machine Translation↩︎

 [5] design a framework where a parent model is trained on a high-resource language pair while retaining model parameters for the child model to start fine-tuning with. Using this method, [5] improve system performance by an average of \(5.6\) BLEU points. The improvement is realized by transferring what is learnt in the high-resource language pair to the low-resource language pair. The Uzbek-English model obtains \(10.7\) BLEU score without the parent model and improves to \(15.0\) with the French-English parent model. The Spanish-English model has \(16.4\) BLEU score without the parent model and \(31.0\) with the French-English parent model. These results show that applying transfer learning contributes \(4.3\) and \(14.6\) BLEU points gain. Based on results from  [5], the closer the two source languages, the more performance gain acquired. Due to the relatedness between Spanish and French (both are members of the Roman language family), performance gain is higher for this pair.

Following previous work, [4] design a paradigm similar to that of [5] but maintain one major difference. In particular, [4] try to make use of relatedness between the parent and child models at the vocabulary level: instead of randomly mapping tokens in the parent and child vocabulary, they retain the parent tokens for the child model if these tokens exist in child language pair. This approach is based on two assumptions - (i) the lexicons of the parent and child language pair have at least some partial overlap and (ii) these identical tokens have similar meaning. Instead of the word-level tokenization in [5],  [4] use Byte Pair Encoding (BPE) [13], [14] to obtain subword tokens which may increase the number of overlapped tokens between the parent and child models. Improvement of \(0.8\) and \(4.3\) in BLEU score were obtained for the Turkish-English and Uyghur-English child models as transferred from an Uzbek-English parent model.

Following the previous two works,  [6] take a similar approach but use the Transformer architecture. They obtain an improvement of \(3.38\) BLEU for an English-Estonian child model transferred from an English-Czech parent model. Similarly, [15] add a second language related to the added low-resource language to avoid overfitting when fine-tuning. This mechanism has shown to be effective. Other works have investigated NMT approaches to similar languages by pre-training new language models on the low-resource languages [16] or without necessarily applying transfer learning [17]–[20], and there are several works on low resource languages [21]. We now introduce our experimental settings.

3 Experimental Settings↩︎

3.1 Languages & Settings↩︎

We carry out experiments on three language pairs: ca-it, ca-oc, and ca-ro. The number of parallel sentences of each dataset is shown in Table 4. Training data are from OPUS [22], particularly version 1 of the WikiMatrix datasets [23]. They are data of child language pair and are used to fine-tune pre-trained model. Development and test data are provided by organizers of the Multilingual Low-Resource Translation for Indo-European Languages shared task. The shared task is hosted in EMNLP 2021 Sixth Conference on Machine Translation (WMT21).

We build two systems: a Primary system and a Contrastive system. The primary system fine-tunes pre-trained ca-es and ca-it models, while the contrastive system fine-tunes a pre-trained ca-en model as shown in Table 1. The primary and contrastive systems serve as context for studying the role of language relatedness in transfer learning in NMT. We submitted predictions of the two systems to the WMT2021 shared task, and evaluation was based on blind test sets with a number of metrics as run by shared task organizers. An exception is the ca-it language pair fine-tuned on top of the ca-es pre-trained model–we train the model of this pair post shared task formal evaluation.

3.2 Model Architecture↩︎

We leverage publicly accessible pre-trained models on Huggingface [24] from Helsinki-NLP [25]. The pre-trained MT models released by Helsinki-NLP are trained on OPUS. These models are Transformer-based implemented in the Marian-NMT framework [26]. Each model has six self-attention layers in both the encoder and decoder sides, and each layer has eight attention heads. The tokenization method is SentencePiece [27] which produces vocabulary of size \(49,621\), \(21,528\) and \(55,255\) for ca-es, ca-it, and ca-en models, respectively.

3.3 Approach↩︎

The pre-trained models are chosen based on the degree of relatedness of the original target language on which the model is trained and the new target language on which the model is fine-tuned. Primary system takes related languages while contrastive system takes unrelated languages. Since Catalan, Italian, Occitan, Romanian, and Spanish are all members of the Roman language family, we take ca-es as our pre-trained MT model for transfer learning. As English is a member of the Germanic language family, we use a ca-en pre-trained model for our transfer learning. Our model choices are summarized in Table 1.

Without modifying the architecture of the MT pre-trained models, all architecture-related hyperparameters are identical to the original edition. As for hyperparameters related to fine-tuning, the number of beams for beam search is modified from four for pre-training to six for fine-tuning. The batch size is set to be \(25\). Pre-trained models are further fine-tuned for \(30,000\) steps on OPUS bitext. The checkpoint with the lowest validation loss is then selected as our best model for prediction.

Similar to [4]–[6], to achieve transfer learning, we retain the parameters of the parent model when fine-tuning the child model. Besides, parent and child models share a common vocabulary. That is, we do not build distinct vocabularies for the parent model and child models. A shared vocabulary can contribute to better transfer learning since all our language pairs employ the same Latin writing system. We suspect a shared vocabulary is more influential when the two languages are related to each other since the languages may have common morphemes, lexical items, or syntactical structure. For unrelated languages, a shared vocabulary may not hurt since the token embeddings are not frozen throughout the fine-tuning process. That is, token embeddings can still be updated to attain better representations during training.

3.4 Baseline Models↩︎

To demonstrate the effectiveness of our transfer learning approach, we provide a baseline model for each language pair that is simply a parent model (a pre-trained model) without any fine-tuning on data of the child language pair.

3.5 Evaluation↩︎

The adopted metrics are BLEU [28], chrF [29], TER [30], COMET [31], and BERTScore [32]. BLEU, chrF and TER are measured with the implementation of Sacrebleu [33]¹.

4 Results and Analysis↩︎

4.1 Primary and Contrastive Systems↩︎

As can be seen in the rightmost five columns in Table 2 and Table 3, primary system outperforms contrastive system across all metrics. We believe that ca-es pre-trained MT model performs better transfer learning because Spanish is closer to Italian, Occitan, and Romanian than English is to these languages. These results, as such, indicate that transfer learning between related language pairs can produce better performance than between unrelated language pairs.

4.2 Baseline and Fine-tuned Models↩︎

Our results in Table 2 and Table 3 show the effectiveness of transfer learning for both related and unrelated language pairs. This is the case since both systems experience a performance gain after fine-tuning.

As an interesting observation, it seems counter-intuitive to have the unrelated language pairs experience slightly higher performance gain. For example, regarding ca-oc language pair, the transfer learning provides \(47.37\) BLEU score improvement transferring from ca-es parent model but \(49.3\) BLEU score improvement transferring from ca-en parent model. We suspect this is because in our work, when fine-tuning, we fix source language and alter the target language.

Unlike multilingual MT models which requires target language label to be prepended at the beginning of a source sentence [34] or notifying the model what target language is for this forward propagation [35], the pre-trained models we use in this work are bilingual models which lack a mechanism to provide the model any information about current target language. Therefore, the ca-en pre-trained model does not know it should now be translating Catalan to Occitan instead of English. Due to producing prediction in an incorrect target language, the metrics will be very poor. After fine-tuning the parent models on data of the child language pairs, the models are likely abler to produce prediction in the correct target language. Due to baseline metrics being too low, the difference in metric values between non-fine-tuned (baseline) and fine-tuned models are large and that is why the performance gain can be higher in contrastive system than in primary system.

5 Conclusion↩︎

In this work, we confirm previous works showing that transfer learning benefits NMT. Besides, an empirical comparison between transferring from related and unrelated languages shows that relatedness is not strictly required for knowledge transfer, but it does result in higher performance than transferring with unrelated languages.

Acknowledgements↩︎

We appreciate the support from the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, Canadian Foundation for Innovation, Compute Canada (www.computecanada.ca), and UBC ARC–Sockeye (https://doi.org/10.14288/SOCKEYE).

References↩︎