1 Introduction↩︎

Transfer learning with pretrained Language Models (LMs) such as BERT [1] and RoBERTa [2] offers unmatched performance in many NLP tasks [3], [4]. However, despite the wealth of semantic knowledge stored in the pretrained LMs [5], [6], they do not produce coherent and effective sentence representations when used off-the-shelf [7]: to this effect, further specialization for sentence-level semantics – not unlike the standard task fine-tuning – is needed [8]–[10]. LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives [11], [12], in supervised (i.e., leveraging labeled external data such as NLI or sentence similarity annotations) [8], [13], [14] or, more recently, fully unsupervised fine-tuning [7], [15] setups.

Following the procedures from monolingual setups, another line of research has been transforming multilingual LMs into multilingual SEs [16], [17], which enable effective sentence matching and ranking in multiple languages as well as cross-lingually [18]. The transformation is typically done by coupling 1) LM objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as Translation Language Modeling (TLM) [19] and/or cross-lingual contrastive ranking [20]. Such multilingual SEs consume a large number of parallel sentences for the latter objectives. Consequently, they outperform multilingual off-the-shelf LMs in cross-lingual sentence similarity and ranking applications [18], [21]. However, as we show in this work, such multilingual SEs may still lag behind traditional static cross-lingual word embeddings (CLWEs) when encoding sub-sentence lexical items (e.g., words or phrases) [7] for cross-lingual lexical tasks (e.g., BLI).

In this work, we probe multilingual SEs for cross-lingual lexical knowledge. We demonstrate that, due to their fine-tuning on multilingual and parallel data, they indeed store a wealth of such knowledge, much more than what ‘meets the eye’ when they are used ‘off the shelf’. However, this lexical knowledge needs to be exposed from the original multilingual SEs, (again) through additional fine-tuning. In other words, we show that multilingual SEs can be ‘rewired’ into effective cross-lingual lexical encoders, as illustrated in Figure 1. This rewiring is again done via a quick and inexpensive contrastive learning procedure: with merely 1k-5k word translation pairs, we successfully convert multilingual SEs into state-of-the-art bilingual lexical encoders for any language pair.¹

We probe the original LMs and SEs as well as demonstrate the usefulness of the proposed contrastive procedure for ‘exposing’ cross-lingual lexical knowledge on three standard lexical cross-lingual tasks using standard evaluation data and protocols: BLI, cross-lingual lexical semantic similarity (XLSIM), and cross-lingual entity linking (XL-EL). We show that the ‘exposure’ procedure is highly effective for both vanilla multilingual LMs (mBERT and XLM-R) and multilingual SEs (LaBSE and xMPNET): e.g., we observe \(\approx\)+10 Precision@1 points gains on standard BLI benchmarks [25]. Multilingual SEs offer substantially better cross-lingual lexical performance than vanilla LMs, both before and after being subjected to contrastive cross-lingual lexical fine-tuning (see Figure 1). This indicates that multilingual SEs have more cross-lingual lexical knowledge than their vanilla LM counterparts, likely owing to their additional exposure to parallel data.

Finally, inspired by , we validate that word vectors produced by cross-lingual lexical encoders (i.e., after contrastive cross-lingual lexical ‘exposure’) can be effectively interpolated with static CLWEs [26] and offer even stronger performance in cross-lingual lexical tasks. Encouragingly, our cross-lingual lexical specialization of multilingual SEs (as well as the further interpolation with static CLWEs), yields particularly massive performance gains for pairs of low-resource languages, as demonstrated on the low-resource BLI benchmark [27].

2 From Multilingual Sentence Encoders to Cross-Lingual Lexical Encoders↩︎

Motivation. The motivation for this work largely stems from the research on probing and analyzing pretrained LMs for various types of knowledge they (implicitly) store in their parameters [5], [28], [29]. In this paper, we focus on a particular knowledge type: cross-lingual lexical knowledge, and its extraction from multilingual LMs and SEs.

Previous work tried to prompt multilingual LMs for word translations via masked natural language templates [30] and extract type-level word embeddings from LMs (i) directly without context [31], [32] or (ii) by averaging contextual embeddings over a large auxiliary corpus in the target language [18], [33]. This existing body of work 1) demonstrated that even sophisticated templates and extraction strategies cannot outperform cross-lingual word embedding spaces (e.g., induced from monolingual fastText vectors) in cross-lingual lexical tasks such as BLI [6] and 2) did not attempt to expose cross-lingual lexical knowledge from multilingual SEs and compare it against the (same type of) knowledge extracted from vanilla multilingual LMs.

Multilingual Sentence Encoders. Off-the-shelf LMs contextualize (sub)word representations, but are unable to encode the precise meaning of input text out of the box. SEs – LMs fine-tuned via sentence-level objectives – in contrast, directly produce a precise semantic encoding of input text. A large body of work focuses on inducing multilingual encoders that capture sentence meaning across languages [16], [20], [34].

The most popular approach obtains multilingual SEs [17] by distilling the knowledge from the monolingual English SE teacher (trained on English semantic similarity and NLI data) into multilingual LM student (e.g., mBERT), using parallel sentences to guide the distillation process. SEs, being specialized for sentence similarity, encode sentence meaning more accurately and are useful in various (unsupervised) text similarity and ranking tasks, monolingually and across languages.

While SEs’ primary purpose is sentence encoding, they can, in principle, be applied to sub-sentential text: words and phrases. In this work, we show that multilingual SEs can be turned into effective cross-lingual lexical encoders. We achieve this through additional cross-lingual lexical fine-tuning [32], requiring as supervision only a small set of word translation pairs.

2.1 Cross-Lingual Lexical Fine-Tuning↩︎

For a given language pair \(L_s\)-\(L_t\), our contrastive cross-lingual lexical fine-tuning of multilingual encoders (LMs and SEs alike) requires a dictionary spanning \(N\) (typically \(N\) \(\leq\) 5,000) word translation pairs, \(\mathcal{D}=\{(w_{i,s},v_{i,t})\}_{i = 1}^N\).² We consider the translation pairs from \(\mathcal{D}\) to be positive examples for the contrastive fine-tuning procedure. For each of the \(N\) source language words in the dictionary (\(w_{i,s}\)), we precompute a set of \(N_n\) hard negative samples: these are the \(L_t\) words that are the closest to \(w_{i,s}\) in the representation space of the multilingual encoder, but not its direct translation \(v_{i,t}\). Let \(\textit{f}_{\theta}(\cdot)\), be the encoding function of the multilingual LM/SE, with \(\theta\) as parameters, and let \(S(\cdot, \cdot)\) be a function of similarity between two vectors. For a source word \(w_{i,s}\), we select as hard negatives words \(v_t\) from \(L_t\) that have the highest \(S(\textit{f}_{\theta_0}(w_{i,s}), \textit{f}_{\theta_0}(v_t))\) score, with \(\theta_0\) as the original encoder’s parameters, before fine-tuning.

We encode all training words – those from the seed dictionary \(\mathcal{D}\) and (at most) \(N\cdot N_n\) precomputed hard \(L_t\) negatives – independently and in isolation. Concretely, for an input \(w\) with \(M\) subword tokens \([sw_1]\ldots[sw_M]\), we feed the sequence \([SPEC1] [sw_1]\ldots[sw_M] [SPEC2]\) into the multilingual encoder (with \([SPEC1]\) and \([SPEC2]\) as encoder’s sequence start and end tokens, resp.), and take the average of the transformed representation (from the last Transformer layer) of the \(w\)’s subword tokens as the \(w\)’s encoding \(\textit{f}_{\theta}(w)\).³

Following common practice in contrastive learning [32], [39], we instantiate \(S\) as the scaled cosine similarity: \(S(\textit{f}_{\theta}(w_i), \textit{f}_{\theta}(w_j)) = C\cdot cos(\textit{f}_{\theta}(w_i), \textit{f}_{\theta}(w_j))\), with \(C\) as the scaling constant. We then train in batches of \(B\) translation pairs, with the variant of the widely used multiple negatives ranking loss (mneg) [39]–[41] as the fine-tuning objective:

\[\begin{align} \mathcal{L} = &-\sum_{i=1}^B S(\textit{f}_{\theta}(w_i),\textit{f}_{\theta}(v_i))\text{(positives)} \notag \\ &+ \sum_{i=1}^B \log\sum_{j=1,j\neq i}^{B} e^{S(\textit{f}_{\theta}(w_i),\textit{f}_{\theta}(v_j))}\text{(in-batch\,negatives)} \notag \\ &+ \sum_{i=1}^B \log \sum_{k=1}^{N_n} e^{S(\textit{f}_{\theta}(w_i),\textit{f}_{\theta}(v_{k,i}))}\text{(hard\,negatives)} \notag \label{eq:mneg} \end{align}\tag{1}\] where \(v_{k,i}\) denotes the \(k\)-th hard negative from the language \(L_t\) for the \(L_s\) word \(w_i\). mneg combines the \(N_n\) hard negatives per each positive example with \(B\)-1 in-batch negatives (i.e., for a source language word \(w_{i, k}\), each target language word \(v_{j,t}\), \(j \neq i\) from \(B\) is used as an in-batch negative of \(w_{i, k}\)). mneg aims to reshape the representation space of the encoder by simultaneously (a) maximising the similarity for positive pairs – i.e., bringing closer together (‘attracting’) the words from the positive pairs and (b) minimising the similarity for (both in-batch and hard) negative pairs – i.e., pushing (‘repelling’) the words from negative pairs further away from each other).⁴

2.2 Interpolation with Static CLWEs↩︎

[38] recently showed that further performance gains in the BLI task might be achieved by combining the type-level output of the encoding function \(\textit{f}\) with static CLWEs, but they experimented only with multilingual LMs. Static CLWEs and multilingual encoder-based representations of the same set of words can be seen as two different views of the same data point. Following [38], we learn an additional linear orthogonal mapping from the static cross-lingual WE space – e.g., a CLWE space induced from monolingual fastText embeddings [42] using VecMap[26] – into the cross-lingual space spanned by the multilingual encoder. The mapping transforms \(\ell_2\)-normed \(d_1\)-dimensional static CLWEs into \(d_2\)-dimensional cross-lingual WEs obtained through the multilingual encoder (fine-tuned \(\textit{f}_{\theta}\) or original \(\textit{f}_{\theta_0}\)).

Learning the linear map \(\boldsymbol{W}\) \(\in\) \(\mathbb{R}^{d_{1}\times d_{2}}\), when \(d_1 < d_2\),⁵ is formulated as a Generalized Procrustes problem [43], [44]. It operates on all (i.e., both \(L_s\) and \(L_t\)) words from the seed translation dictionary \(\mathcal{D}\).⁶ Unless noted otherwise, a final representation of an input word \(w\) is then computed as follows:

\[\begin{align} (1-\lambda)\frac{\mathit{clwe}(w)\boldsymbol{W}}{\left\|\mathit{clwe}(w)\boldsymbol{W}\right\|_{2}} + \lambda \frac{f_{\theta}(w)}{\left\|f_{\theta}(w)\right\|_{2}}, \end{align}\]

where \(\lambda\) is a tunable interpolation hyper-parameter, \(\mathit{clwe}(w)\) denotes the static CLWE of \(w\), and \(f_{\theta}(w)\) the representation of \(w\) obtained with the (contrastively fine-tuned or original) multilingual LM/SE. This simple procedure yields an ‘interpolated’ shared cross-lingual WE space.

3 Experimental Setup↩︎

Multilingual Sentence Encoders. We probe two widely used multilingual SEs: 1) Language-agnostic BERT Sentence Embedding (LaBSE) [16] which adapts pretrained multilingual BERT (mBERT) [1] into a multilingual SE; 2) Multilingual xMPNET is a distillation-based adaptation [17] of XLM-R [45] as the student model into a multilingual SE, based on the monolingual English MPNet encoder [46] as the teacher model. LaBSE is the current state-of-the-art multilingual SE and supports 109 languages, while xMPNET is the best-performing multilingual SE in the Sentence-BERT repository [8].⁷ Along with LaBSE and xMPNET as SEs, we experiment with the original multilingual LMs – mBERT and XLM-R – using the the same training and evaluation protocols (see Figure 1 and §2), aiming to quantify: (i) the extent to which cross-lingual lexical knowledge can be exposed from LMs that have not been specialized for sentence-level semantics, as well as (ii) the increase in quality of lexical knowledge brought about with sentence-level specialisation (i.e., when multilingual LMs get transformed into multilingual SEs).

Evaluation Tasks. We evaluate on the standard and diverse cross-lingual lexical semantic tasks:

Task 1: Bilingual Lexicon Induction (BLI), a standard task to assess the “semantic quality” of static cross-lingual word embeddings (CLWEs) [35], allows us to 1) directly assess the extent to which cross-lingual word translation knowledge can be exposed from multilingual LMs and SEs and 2) immediately test the ability to transform multilingual sentence encoders into bilingual lexical encoders; We run a series of BLI evaluations on two standard BLI evaluation benchmarks. First, GT-BLI [25], constructed semi-automatically from Google Translate, comprises 28 language pairs with a good balance of typologically similar and distant languages (Croatian: hr, English: en, Finnish: fi, French: fr, German: de, Italian: it, Russian: ru, Turkish: tr). Second, PanLex-BLI [27] focuses on BLI evaluation for lower-resource languages, deriving training and test data from PanLex [47]. We evaluate on 10 pairs comprising the following five typologically and etymologically diverse languages: Bulgarian (bg), Catalan (ca), Estonian (et), Hebrew (he), and Georgian (ka).

Standard BLI setups and data are adopted: 5k training word pairs are used as seed dictionary \(\mathcal{D}\), and another 2k pairs as test data. Note that \(\mathcal{D}\) is used to (i) contrastively fine-tune multilingual encoders (§2.1), (ii) learn the (baseline) static VecMap CLWE space, as well as to (iii) learn the projection between the static CLWE space and the representation spaces of multilingual encoders required to obtain the interpolated representations (§2.2). The evaluation metric is standard Precision@1 (P@1).⁸ For PanLex-BLI, we also run additional experiments using smaller bilingual dictionaries \(\mathcal{D}\), spanning 1k translation pairs.

Task 2: Cross-Lingual Lexical Semantic Similarity (XLSIM) is another cross-lingual lexical task: it tests the extent to which lexical representations can capture the (human perception of) fine-grained semantic similarity of words across languages. We use the recent comprehensive XLSIM benchmark Multi-SimLex [31], which comprises cross-lingual datasets of 2k-4k scored word pairs over 66 language pairs. We evaluate on a subset of language pairs shared with the GT-BLI dataset: en, fi, ru, fr.⁹

Task 3: Cross-Lingual Entity Linking (XEL) is a standard task in knowledge base (KB) construction [48], where the goal is to link an entity mention in any language to a corresponding entity in an English KB or in a language-agnostic KB.¹⁰ We evaluate on the cross-lingual biomedical entity linking (XL-BEL) benchmark of [21]: it requires the model to link an entity mention to entries in UMLS [49], a language-agnostic medical knowledge base. We largely follow the XL-BEL experimental setup of [21] and probe the encoders first without any additional task-specific fine-tuning on UMLS data, and then with subsequent UMLS fine-tuning (i) only on the en UMLS data, (ii) on all the UMLS data in 10 languages of the XL-BEL dataset.¹¹ Due to a large number of experiments, we again focus on the subset of languages in XL-BEL shared with GT-BLI: en, de, fi, ru, tr.

Static CLWEs and Word Vocabularies. As monolingual static WEs, we select CommonCrawl fastText vectors [42] of the top 200k most frequent word types in the training data, following prior work on learning static CLWEs [23], [26], [50].¹² Static CLWEs are then induced via the standard and popular supervised mapping-based VecMap method [26], leveraging the seed dictionary \(\mathcal{D}\). These CLWEs are used for interpolation with encoder-based WEs (see §2.2) but also as the baseline approaches for BLI and XLSIM tasks. We compute the type-level WEs from multilingual LMs and SEs for the same 200K most frequent words of each language.

Technical Details and Hyperparameters. The implementation is based on the SBERT framework [8], using the suggested settings: AdamW [51]; learning rate of \(2e\)-\(5\); weight decay rate of \(0.01\). We run contrastive fine-tuning for 5 epochs with all the models, with the batch size of \(B=128\) positives for mneg. The number of hard negatives per each positive is set to \(N_n=10\) (see §2.1).¹³ Since standard BLI and XLSIM datasets lack a validation portion [35], we follow prior work [25] and tune hyperparameters on a single, randomly selected language pair from each dataset, and use those values in all other runs.

Model Configurations. They are labelled as follows: ENC-{noCL,+CL} (\(\lambda\)), where (i) ENC denotes the input multilingual Transformer, which can be a multilingual LM (mBERT, XLM-R), or a multilingual SE (LaBSE, xMPNET), (ii) ‘noCL’ refers to using the input model ‘off-the-shelf’ without any contrastive lexical fine-tuning, while ‘+CL’ variants apply the contrastive fine-tuning, and (iii) \(\lambda\) is the factor that defines the interpolation with the static CLWE space, obtained with VecMap(see Figure 1 and §2.2). Note that \(\lambda = 1.0\) implies no interpolation with static CLWE space, i.e., WEs come purely from the multilingual LM/SE.

Important Disclaimer. We note that the main purpose of the chosen evaluation tasks and experimental protocols is not necessarily achieving state-of-the-art performance, but rather probing different model variants in different cross-lingual lexical tasks, and offering fair and insightful comparisons

4 Results and Discussion↩︎

Bilingual Lexicon Induction (BLI). Table 1 displays our main BLI results, aggregated over all 28 language pairs of GT-BLI, for two multilingual LMs (mBERT and XLM-R) and their SE counterparts (LaBSE and xMPNET). Two trends hold across the board. First, multilingual SEs, LaBSE and xMPNET, largely outperform their multilingual LM counterparts, mBERT and XLM-R. The gains are visible in all four experimental configurations (with/without contrastive cross-lingual lexical specialisation \(\times\) with/without interpolation with the VecMap CLWE space). This confirms our intuition that multilingual SEs, having been (additionally) trained on parallel data [16], [17], should better reflect the cross-lingual alignments at the lexical level than off-the-shelf multilingual LMs, which have not been exposed to any cross-lingual signal in pretraining. The poor cross-lingual lexical alignment in the representation spaces of mBERT and XLM-R also reflects in the fact that with those encoders, we only surpass the baseline VecMap performance by a small margin (+1.1 for XLM-R, +1.6 for mBERT) after subjecting them to contrastive lexical fine-tuning and interpolating their word encodings with VecMap WEs.

The behavior of SEs, on the other hand, is much more favorable. LaBSE, for example, surpasses baseline VecMap performance with interpolation alone, even without the contrastive lexical fine-tuning. When contrastively fine-tuned (and then interpolated with VecMap) both LaBSE and xMPNET surpass the baseline VecMap performance by a much wider margin (+6.4 and +5.2, respectively). This implies that our contrastive fine-tuning exposes more of the high quality cross-lingual lexical knowledge from multilingual SEs.

Both (i) contrastive cross-lingual lexical learning (+CL) and (ii) interpolation with VecMap consistently improve the performance for all four encoders: we reach peak scores by combining contrastive fine-tuning and interpolation with static CLWEs (+CL (\(\lambda\))). Contrastive fine-tuning crucially contributes to the overall performance: compared to interpolation alone (noCL (\(\lambda\))), +CL (\(\lambda\)) brings an average gain of over 6 BLI points.

Table 2 shows the BLI results on 10 low(er)-resource language pairs from PanLex-BLI. While overall relative trends are similar to those observed for high(er)-resource languages from GT-BLI (Table 1), the gains stemming from cross-lingual contrastive lexical fine-tuning are substantially larger in this case. The best-performing configuration – contrastive fine-tuning and interpolation (+CL (0.4)) applied on LaBSE – surpasses VecMap by 11 BLI points on average (compared to 6 points on GT-BLI), with gains for some language pairs (e.g., he-ka, et-he) approaching the impressive margin of 20 BLI points. This finding indicates that cross-lingual lexical knowledge stored in multilingual SEs is even more crucial when dealing with lower-resource languages.

In Table 3 we compare the results of LaBSE (as the best-performing multilingual SE) against VecMap on PanLex-BLI in a scenario with less external bilingual supervision: \(|\mathcal{D}| = 1k\). Interestingly, in this setup LaBSE already substantially outperforms VecMap out of the box (noCL (1.0)); contrastive lexical fine-tuning (+CL (\(1.0\))) and interpolation with VecMap embeddings (+CL (0.5)) again bring further substantial gains, and we again observe a strong synergistic effect of the two components: +CL (1.0) yields gains over noCL (1.0) for 9/10 language pairs, and +CL (0.5) results in further boosts for all 10 pairs. Further, the contrastively fine-tuned LaBSE seems to be much more resilient to training data scarcity than VecMap: reduction of the training dictionary size from 5k to 1k reduces the performance of LaBSE+CL (\(\lambda\)) by 27% (from 34.7 to 25.2 P@1 points) compared to a massive performance drop of almost 70% for VecMap(from 23.5 to mere 7.4 P@1).

Cross-Lingual Lexical Semantic Similarity (XLSIM). The average XLSIM results are summarized in Table 1. They again corroborate one of the main findings from BLI experiments: multilingual SEs store more cross-lingual lexical knowledge than multilingual LMs. This is validated by substantial gains of SEs over corresponding LMs across all configurations in Table 1. Interestingly, due to their contrastive learning objectives on sentence-level parallel data [16], LaBSE and xMPNET provide very strong XLSIM results when used off-the-shelf (noCL (1.0)), outperforming the CLWE VecMap embeddings. Contrastive lexical fine-tuning with 5k word translation pairs (+CL (1.0)) in this case does not bring any gains. However, the opposite is true for multilingual LMs: contrastive cross-lingual lexical fine-tuning on only 5k word translation pairs brings large benefits in the XLSIM task (e.g., compare noCL (1.0) and +CL (1.0) configurations for mBERT and XLM-R), and turns them into more effective lexical encoders. This result corroborates a similar finding from prior work in monolingual setups [32]. Finally, interpolation with static CLWEs benefits the final XLSIM performance of all four underlying multilingual encoders: interpolated vectors (\(\lambda=0.5\)) yield highest scores across the board, substantially surpassing gains both VecMap and WEs from fine-tuned encoders (\(\lambda=1.0\)).

Interpolation with Static CLWEs. A more detailed analysis over different \(\lambda\) values for BLI and XLSIM, summarized in Figure 2, reveals that interpolation can bring large performance gains, especially for \(\lambda\) in the \([0.3,0.5]\) interval. The optimal \(\lambda\) value is, however, task- and even dataset-dependent. For instance, for low-resource BLI on PanLex-BLI more knowledge comes from the multilingual encoders as VecMap CLWEs are of lower quality for such languages: in consequence, the optimal \(\lambda\) value ‘moves away’ from the static CLWEs towards encodings obtained by fine-tuned multilingual SE. We also note that larger benefits from interpolation are observed when VecMap CLWEs are combined with contrastively fine-tuned multilingual SEs: cf., the large gains in Figure 2 (b) and in Table 3 for the LaBSE+CL model variant.

Cross-Lingual Entity Linking (XEL). Experiments on XL-BEL [21], summarized in Table 4, demonstrate that additional contrastive tuning with word or phrase pairs can greatly boost performance of multilingual LMs: even fine-tuning with 5k word translation pairs without any domain-specific knowledge yields strong benefits for XLM-R. As expected, using a much larger and domain-specific external database UMLS yields much higher scores and is more crucial for performance. In fact, contrastively fine-tuning on UMLS generally improves XEL performance with all four underlying models. Again, we observe that SE-based (xMPNET) configurations outperform the respective LM-based (XLM-R) configurations across the board. This finding again indicates that multilingual SEs store more cross-lingual lexical knowledge than multilingual LMs: this difference is particularly salient when the models are used off-the-shelf without any additional contrastive fine-tuning, and xMPNET retains the edge over XLM-R even after task-specific fine-tuning with the UMLS-based domain-specific knowledge.

What is more, for fi, ru, and tr, the multilingual xMPNET-based variants match or surpass the performance of respective XEL models trained on top of monolingual LMs (e.g., for fi, a model based on the Finnish BERT) reported by [21]. This further validates our hypothesis that multilingual SEs store rich multilingual lexical knowledge, which is then also exposed in domain-specific (multilingual) UMLS fine-tuning, yielding performance gains. Contrastive fine-tuning on UMLS synonyms (+CL+UMLS variants) expectedly outpeforms fine-tuning on (5k) general-domain word translations (+CL), indicating that in specialized domains, if available, in-domain cross-lingual lexical signal should be exploited.

5 Conclusion and Future Work↩︎

In this work, we investigated strategies to expose cross-lingual lexical knowledge from pretrained models, including multilingual language models (LMs) and multilingual sentence encoders (SEs). Based on an extensive evaluation on a suite of cross-lingual lexical tasks, we verified that multilingual SEs (e.g., LaBSE and xMPNET) are superior to multilingual LMs (mBERT and XLM-R) in terms of stored cross-lingual lexical knowledge. Moreover, we proposed new methods to further fine-tune their representations based on contrastive learning to ‘rewire’ the models’ parameters and transform them from LMs and SEs into more effective cross-lingual lexical encoders. We also empirically demonstrated that for some lexical tasks performance can be further boosted through interpolation of type-level lexical encodings with static cross-lingual word embeddings. These yield gains for all underlying models, and are especially significant for resource-poor languages and in low-data learning regimes. While our work has focused on two widely used state-of-the-art multilingual SEs, the contrastive fine-tuning framework is versatile and model-independent and can be directly applied on top of other multilingual SEs in future work. We will also investigate other more sophisticated contrastive learning strategies, look into ensembling of knowledge extracted from different SEs, and expand our evaluation to more tasks and languages.

Limitations↩︎

This work focuses on lexical specialization of multilingual encoders, off-the-shelf LMs (experiments with mBERT and XLM-R Base) and, in particular multilingual encoders specialized for sentence-level semantics (experiments with LaBSE and xMPNET). While these are all widely used models, they are arguably among the smaller pretrained multilingual encoders. Due to computational constraints, we have not evaluated the effectiveness of the proposed cross-lingual lexical specialisation for larger multilingual LMs, e.g., XLM-R-Large [45] or mT5 (Large, XL, and XXL) [52]. It is possible that these larger multilingual LMs would close (some of) the performance gap w.r.t. multilingual SEs. Such large LMs, however, are effectively available to fewer researchers and practitioners. Our work includes less resource-demanding LMs and SEs, making their lexically specialized variants that we offer, more widely accessible.

Lexical input (i.e., words or phrases) are provided to each multilingual encoder fully “in isolation” (see §2), without any surrounding context. However, the alternative of using external corpora and averaging-over-context [18], which we have not evaluated in this work for clarity and space constraints, might yield slightly improved task performance. Nonetheless, the ‘in isolation’ approach has been verified in previous work [18], [32], [38] as very competitive, and is more lightweight: 1) it disposes of any external text corpora and is not impacted by the external data; 2) it encodes words more efficiently due to the absence of context. Moreover, it allows us to directly study the richness of cross-lingual information stored in the encoders’ parameters, and its interaction with additional cross-lingual signal from bilingual lexicons.

The contrastive cross-lingual lexical fine-tuning we proposed in this is work is bilingual. It leverages a small bilingual dictionary \(\mathcal{D}\) for each language pair and specializes the multilingual encoders (LMs and SEs) independently for each language pair. Assuming interest in cross-lingual lexical tasks between all pairs of \(N_L\) languages, this entails \(\frac{N_L\cdot (N_L - 1)}{2}\) fine-tuning procedures and as many resulting bilingual models. Although our contrastive fine-tuning is relatively fast and lightweight, given that it leverages at most 5k translation pairs, for large \(N_L\) it could easily exceed the computational and time budget for most users.

Intuitively, for each bilingually fine-tuned model, we evaluate the performance for that respective language pair. Currently, we do not investigate the spillover effects that a bilingual lexical fine-tuning of multilingual encoders could have on lexical representations of other languages. Such an analysis, planned for future work, would be particularly interesting in the context of low-resource languages, unseen from the point of view of cross-lingual lexical fine-tuning, and in particular closely related low-resource languages. For instance, if we are doing cross-lingual lexical fine-tuning for language pairs involving Turkish, are there spillover benefits for low(er)-resource Turkic languages such as Uyghur or Kazakh?

Finally, we acknowledge that our choice of lexical tasks as probing tasks is non-exhaustive: we put focus on standard tasks from previous work on (multilingual) lexical semantics that are especially convenient as cross-lingual lexical probes: such tasks directly test and compare the quality of cross-lingual lexical representations obtained via different methods.

Acknowledegments↩︎

We would like to thank Yaoyiran Li for interesting insights and discussions. This work has been supported by the ERC PoC Grant MultiConvAI (no. 957356) and a Huawei research donation to the University of Cambridge. Fangyu Liu is supported by Grace & Thomas C.H. Chan Cambridge Scholarship. The work of Goran Glavaš has been supported by the Multi2ConvAI project of MWK Baden-Württemberg. The work of Edoardo Maria Ponti’s work has been supported by the Facebook CIFAR AI Chair program during his affiliation with Mila – Quebec AI Institute. Nigel Collier kindly acknowledges grant-in-aid funding from ESRC (grant number ES/T012277/1).

6 List of Languages↩︎

The list of languages used in this work, along with their ISO 639-1 codes, is available in Table 5.

7 BLI Results across Individual Language Pairs↩︎

Additional experiments and analyses over individual language pairs and other \(\lambda\) values, which further support the main claims of the paper, have been relegated to the appendix for clarity and compactness of the presentation in the main paper:

Table 6. It provides results over all 28 language pairs in GT-BLI with 2 multilingual LMs and 2 multilingual SEs in the noCL variant without contrastive fine-tuning.

Table 7. It provides results over all 28 language pairs in GT-BLI with 2 multilingual LMs and 2 multilingual SEs in the +CL variant with contrastive fine-tuning.

Table 8. It provides results over all 28 language pairs in GT-BLI and across different \(\lambda\) values with the LaBSE+noCL variant.

Table 9. It provides results over all 28 language pairs in GT-BLI and across different \(\lambda\) values with the LaBSE+CL variant.

8 XLSIM Results across Individual Language Pairs↩︎

Table 10. It provides results over selected 6 language pairs from Multi-SimLex with 2 multilingual LMs and 2 multilingual SEs in the noCL variant without contrastive fine-tuning.

Table 11. It provides results over selected 6 language pairs from Multi-SimLex with 2 multilingual LMs and 2 multilingual SEs in the +CL variant with contrastive fine-tuning.

Table 12. It provides results over selected 6 language pairs from Multi-SimLex and across different \(\lambda\) values with the xMPNET+noCL variant.

Table 13. It provides results over selected 6 language pairs from Multi-SimLex and across different \(\lambda\) values with the xMPNET+CL variant.

9 Models and Evaluation Data↩︎

URLs to the models used in this paper are provided in Table 14. Training and test data for all three tasks (BLI, XLSIM, XEL) is available online:

• GT-BLI is available here: https://github.com/codogogo/xling-eval

• PanLex-BLI: https://github.com/cambridgeltl/panlex-bli

• Multi-SimLex [XLSIM]: https://multisimlex.com/

• XL-BEL [XEL]: https://github.com/cambridgeltl/sapbert

Our code is based on PyTorch, and relies on the following two widely used repositories:

References↩︎