1 Introduction↩︎

Pretrained language models (LMs) [1]–[3] can be finetuned to a variety of natural language processing (NLP) tasks and generally yield high performance. Increasingly, these models and their generative variants are used to solve tasks by simple text generation, without any finetuning [4]. This motivated research on how much knowledge is contained in LMs: [5] used models pretrained with masked language to answer fill-in-the-blank templates such as “Paris is the capital of [MASK].”

This research so far has been exclusively on English. In this paper, we focus on using multilingual pretrained LMs as knowledge bases. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? To answer these questions, we translate English datasets and analyze mBERT for 53 languages. (iii) A multilingual model is trained on more text, e.g., BERT’s training data contains the English Wikipedia, but mBERT is trained on 104 Wikipedias. Can mBERT leverage this fact? Indeed, we show that pooling across languages helps performance.

In summary our contributions are:

Code and data are available online (https://github.com/norakassner/mlama).

2 Data↩︎

2.1 LAMA↩︎

We follow the LAMA setup introduced by [5]. More specifically, we use data from TREx [6] and GoogleRE.² Both consist of triples of the form (object, relation, subject). The underlying idea of LAMA is to query knowledge from pretrained LMs using templates without any finetuning: the triple (Paris, capital-of, France) is queried with the template “Paris is the capital of [MASK].” In LAMA, TREx has 34,039 triples across 41 relations, GoogleRE 5528 triples and 3 relations. Templates for each relation have been manually created by [5]. We call all triples from TREx and GoogleRE together LAMA.

LAMA has been found to contain many “easy-to-guess” triples; e.g., it is easy to guess that a person with an Italian sounding name is born in Italy. LAMA-UHN is a subset of triples that are hard to guess introduced by .

2.2 Translation↩︎

We translate both entities and templates. We use Google Translate to translate templates in the form “[X] is the capital of [Y]”. After translation, all templates were checked for validity (i.e., whether they contain “[X]”, “[Y]” exactly once) and corrected if necessary. In addition, German, Hindi and Japanese templates were checked by native speakers to assess translation quality (see Table [tab:comp_manual]). To translate the entity names, we used Wikidata and Google knowledge graphs.

mBERT covers 104 languages. Google Translate covers 77 of these. Wikidata and Google Knowledge Graph do not provide entity translations for all languages and not all entities are contained in the knowledge graphs. For English we can find a total of 37,498 triples which we use from now on. On average, 34% of triples could be translated (macro average over languages). We only consider languages with a coverage above 20%, resulting in the final number of languages we include in our study: 53. The macro average of translated triples in these 53 languages is 43%. Figure [fig:langs] gives statistics. We call the translated dataset mLAMA.

3 Experiments↩︎

3.1 Model↩︎

We work with mBERT [3], a model pretrained on the 104 largest Wikipedias. We denote mBERT queried in language x as mBERT[x]. As comparison we use the English BERT-Base model and refer to it as BERT. In initial experiments with XLM-R [7] we observed worse performance, similar to . Thus, for simplicity we only report results on mBERT.

3.2 Typed and Untyped Querying↩︎

[5] use templates like “Paris is the capital of [MASK]” and give \(\arg\max_{w \in \mathcal{V}}p(w|t)\) as answer where \(V\) is the vocabulary of the LM and \(p(w|t)\) is the (log-)probability that word \(w\) gets predicted in the template \(t\). Thus the object of a triple must be contained in the vocabulary of the language model. This has two drawbacks: it reduces the number of triples that can be considered drastically and hinders performance comparisons across LMs with different vocabularies. We refer to this procedure as UnTyQ.

We propose to use typed querying, TyQ: for each relation a candidate set \(\mathcal{C}\) is created and the prediction becomes \(\arg\max_{c\in \mathcal{C}}p(c|t)\). For templates like “[X] was born in [MASK]”, we know which entity type to expect, in this case cities. We observed that (English-only) BERT-base predicts city names for MASK whereas mBERT predicts years for the same template. TyQ prevents this.

We choose as \(\mathcal{C}\) the set of objects across all triples for a single relation. The candidate set could also be obtained from an entity typing system (e.g., [8]), but this is beyond the scope of this paper. Variants of TyQ have been used before [9].

3.3 Singletoken vs. Multitoken Objects↩︎

Assuming that objects are in the vocabulary [5] is a restrictive assumption, even more in the multilingual case as e.g., “Hamburg” is in the mBERT vocabulary, but French “Hambourg” is tokenized to [“Ham",”##bourg”]. We consider multitoken objects by including multiple [MASK] tokens in the templates. For both TyQ and UnTyQ we compute the score that a multitoken object is predicted by taking the average of the log probabilities for its individual tokens.

Given a template \(t\) (e.g., “[X] was born in [Y].”) let \(t_1\) be the template with one mask token, (i.e., “[X] was born in [MASK].”) and \(t_k\) be the template with \(k\) mask tokens (i.e., “[X] was born in [MASK] [MASK] …[MASK].”). We denote the log probability that the token \(w \in V\) is predicted at \(i\)th mask token as \(p(m_i = w | t_k)\), where \(V\) is the vocabulary of the LM. To compute \(p(e|t)\) for an entity \(e\) that is tokenized into \(l\) tokens \(\epsilon_1, \epsilon_2, \dots, \epsilon_l\) we simply average the log probabilities across tokens: \[p(e|t) = \frac{1}{l} \sum_{i=1}^{l} p(m_i = \epsilon_i | t_l).\] If \(k\) is the maximum number of tokens of any entity \(e\in \mathcal{C}\) gets split into, we consider all templates \(t_1\), …, \(t_k\), with \(C\) being the candidate set. The prediction is then the word with the highest average log probability across all templates \(t_1\), …, \(t_k\).

Note that for UnTyQ the space of possible predictions is \(V\times V \times \dots \times V\) whereas for TyQ it is the candidate set \(\mathcal{C}\).

3.4 Evaluation↩︎

We compute precision at one for each relation, i.e., \(1/|T| \sum_{t \in T}\mathbb{1}\{\hat{t}_{object} = t_{object}\}\) where \(T\) is the set of all triples and \(\hat{t}_{object}\) is the object predicted by TyQ or UnTyQ. Note that \(T\) is different for each language. Our final measure (p1) is then the precision at one averaged over relations (i.e., macro average). Results for multiple languages are the macro average p1 across languages.

4 Results and Discussion↩︎

We first investigate TyQ and UnTyQ and find that TyQ is better suited for investigating knowledge in LMs. After exploring the translation quality, we use TyQ on mLAMA and observe rather stable performance for 21 and poor performance for 32 languages. When investigating the languages more closely, we find that prediction results highly depend on the language. Finally, we validate our initial hypothesis that mBERT can leverage its multilinguality by pooling predictions: pooling indeed performs better.

4.1 UnTyQ vs. TyQ↩︎

Figure [fig:robustness] shows the distribution of p1 scores for single and multitoken objects. As expected, TyQ works better, both for single and multitoken objects. With UnTyQ, performance not only depends on the model’s knowledge, but on at least three extraneous factors: (i) Does the model understand the type constraints of the template (e.g., in “X is the capital of Y”, Y must be a country)? (ii) How “fluent” a substitution is an object under linguistic constraints (e.g., morphology) that can be viewed as orthogonal to knowledge? Many English templates cannot be translated into a single template in many languages, e.g., “in X” (with X a country) has different translations in French: “à Chypre”, “au Mexique”, “en Inde”. But the LAMA setup requires a single template. By enforcing the type, we reduce the number of errors that are due to surface fluency. (iii) The inadequacy of the original LAMA setup for multitoken answers. Figure [fig:robustness] (right) shows that the original UnTyQ struggles with multitokens (mean p1 .03 vs. for TyQ).

Overall, TyQ allows us to focus the evaluation on the core question: what knowledge is contained in LMs? From now on, we report numbers in the TyQ setting.

Manual template tuning or automatic template mining [10] has been investigated in the literature to approach the typing problem. We had native speakers check templates for German, Hindi and Japanese, correct mistakes in the automatic translation and paraphrase the template to obtain predictions with the correct type. Table [tab:comp_manual] shows that corrections do not yield strong improvements. We conclude that template modifications are not an effective solution for the typing problem.

4.2 Translation Quality↩︎

Contemporaneous work by provides manual translations of LAMA templates for 23 languages respecting grammatical gender and inflection constraints. We evaluate our machine translated templates by comparing performance on a common subset of 14 languages using TyQ querying on the TREx subset. Surprisingly, we find a performance difference of 1 percentage points (0.23 vs., p1 averaged over languages) in favor of the machine translated templates. This indicates that the machine translated templates in combination with TyQ exhibit comparable performance but come with the benefit of larger language coverage (53 vs. 23 languages).

4.3 Multilingual Performance↩︎

In mLAMA, not all triples are available in all languages. Thus absolute numbers are not comparable across languages and we adopt a relative performance comparison: we report p1 of a model-language combination divided by p1 of mBERT’s performance in English (mBERT[en]) on the exact same set of triples and call this rel-p1. A rel-p1 score of 0.5 for mBERT[fi] means that p1 of mBERT on Finnish is half of mBERT[en]’s performance on the same triples. rel-p1 of English BERT is usually greater than 1 as monolingual BERT tends to outperform mBERT[en].

Figure [fig:langauge_plot] shows that mBERT performs reasonably well for 21 languages, but for 32 languages rel-p1 is less than 0.6 (i.e., their p1 is 60% of English’s p1). We conclude that mBERT does not exhibit a stable performance across languages. The variable performance (from 20% to almost 100% rel-p1) indicates that mBERT has no common representation for, say, “Paris” across languages, i.e., mBERT representations are language-dependent.

4.4 Bias↩︎

If mBERT captured knowledge independent of language, we should get similar answers across languages for the same relation. However, Table [tab:bias] shows that mBERT exhibits language-specific biases; e.g., when queried in Italian, it tends to predict Italy as the country of origin. This effect occurs for several relations: Table [tab:biastable] in the supplementary presents data for ten relations and four languages.

4.5 Pooling↩︎

We investigate pooling of predictions across languages by picking the object predicted by the majority of languages. Table [tab:pooling] shows that pooled mBERT outperforms mBERT[en] by 6 percentage points on LAMA, presumably in part because the language-specific bias is eliminated. mBERT[pooled] even outperforms BERT by 3 percentage points on LAMA-UHN. This indicates that mBERT can leverage the fact that it is trained on 104 Wikipedias vs.just one and even outperforms the much stronger model BERT.

5 Related Work↩︎

[5] first asked the question: can pretrained LMs function as knowledge bases? Subsequent analyses focused on different aspects, such as negation [11], easy to guess names [12], integrating adapters [13] or finding alternatives to a “fill-in-the-blank” approach with single-token answers [10], [14], [15]. Other work combines pretrained LM with information retrieval [16]–[20]. None of this work addresses languages other than English.

Multilingual models like mBERT [3] and XLM-R [7] perform well for zero-shot crosslingual transfer [21]. However, we are not aware of any prior work that analyzed to what degree pretrained multilingual models can be used as knowledge bases. There are many multilingual question answering datasets such as XQuAD [22], TiDy [23], MKQA [24] and MLQA [25]. Usually, multilingual models are finetuned to solve such tasks. Our goal is not to improve question answering or create an alternative multilingual question answering dataset, but instead to investigate which knowledge is contained in pretrained multilingual LMs without any kind of supervised finetuning.

There is a range of alternative multilingual knowledge bases that could be used for evaluation. Those include ConceptNet [26] or BabelNet [27]. We decided to provide a translated versions of TREx and GoogleRE for the sake of comparability across languages. By translating manually created templates and entities we can ensure comparability across languages. This is not possible for crowd-sourced databases like ConceptNet.

In contemporaneous work, create and investigate a multilingual version of LAMA. They provide human template translations for 23 languages, propose several methods for multitoken decoding and code-switching, and experiment with a number of PLMs. In contrast to their work, we investigate typed querying, focus on comparabiliy and pooling across languages, and explore language biases.

6 Conclusion↩︎

We presented mLAMA, a dataset to investigate knowledge in language models (LMs) in a multilingual setting covering 53 languages. While our results suggest that correct entities can be retrieved for many languages, there is a clear performance gap between English and, e.g., Japanese and Thai. This suggests that mBERT is not storing entity knowledge in a language-independent way. Experiments investigating language bias confirm this finding. We hope that this paper and the dataset we publish will stimulate research on investigating knowledge in LMs multilingually rather than just in English.

Acknowledgements↩︎

This work was supported by the European Research Council (# 740516) and the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibility for its content. The second author was supported by the Bavarian research institute for digital transformation (bidt) through their fellowship program. We thank Yannick Couzinié and Karan Tiwana for correcting the Japanese and Hindi templates. We thank the anonymous reviewers for valuable comments.

7 Language Bias↩︎

Table [tab:biastable] shows the language bias for 10 relations. For each relation we aggregated the predictions across all triples and show the most common two predicted entities together with its count (in brackets). The querying language clearly affects results. The effect is drastic for relations that ask for a country (e.g., P495 or P1001). P39 yields very different results without exhibiting a clear pattern. Other relations such as P463 or P178 are rather stable.

8 Data Samples↩︎

Table [tab:examples1] and Table [tab:examples2] show randomly sampled entries from the data.

9 Pretraining Data↩︎

We investigate whether performance across languages is correlated with the amount of pretraining data for each language. To this end we investigate the number of articles per language as of January 2021³ and p1 for TyQ in Figure [fig:sizes]. We do not have access to the original pretraining data of mBERT. Thus, the number of articles we consider in the analysis might be different to the actual data used to train mBERT.

References↩︎