1 Introduction↩︎

Multilingual large language models have significantly advanced natural language processing by enabling a single model to represent multiple languages within a shared parameter space [1], [2]. A central component of these models is the tokenization process, which constructs a subword vocabulary used to encode textual input across languages. Tokenization plays a critical role in balancing computational efficiency with linguistic expressiveness, particularly in multilingual settings [3].

Because vocabulary size is limited, selecting an appropriate tokenizer configuration becomes a key architectural decision [4]. Multilingual contexts introduce unique design considerations, such as how vocabulary capacity should be distributed across languages, whether to train a shared tokenizer or aggregate language-specific ones, and how increasing the number of languages affects cross-lingual transfer performance.

In a shared vocabulary, an important challenge emerges when training tokenizers on multiple related languages: the presence of shared tokens. Related languages often contain identical surface forms that are assigned a single subword unit in the learned vocabulary. An example is the word barn, shared between English and Swedish. These shared tokens receive a single embedding representation that is jointly trained across languages during pre-training [5]. However, this shared representation may not equally capture the meanings and usage patterns of all contributing languages. Such imbalance can arise due to differences in corpus size, token frequency, or contextual usage patterns across languages.

We investigate the shared-token phenomenon through a specific class of words: cross-lingual homographs, words that share identical surface forms across different languages. While many such words preserve similar meanings across languages (or denote identical entities), they may appear in distinct contextual distributions. Moreover, some homographs correspond to false friends, carrying different meanings despite their identical surface forms. For example, the form bank in English can refer to a financial institution or the side of a river, as in In German, however, bank commonly means a bench, as in (I hit my head on the bench). False friends introduce additional complexity to multilingual tokenization: how can a single shared token represent language-dependent meanings and usage patterns?

We begin by showing that mainstream tokenizer algorithms, such as Byte-Pair Encoding [6] and Unigram Language Modeling [7], often treat homographs in a language-agnostic manner during vocabulary construction. Because the tokenizer is trained on a concatenated multilingual corpus without explicit language conditioning, identical surface forms are typically assigned a single shared subword unit. As a result, the downstream embedding representation aggregates signals from multiple languages, potentially reinforcing the imbalance we described. In contrast, the SaGe tokenizer [8], which incorporates contextual information during vocabulary construction, creates vocabularies that segment shared surface forms differently across languages more often.

To improve homograph handling even more, we introduce language cues into the tokenizer training process. These cues are language-specific Unicode characters injected into the training corpus. For each shared homograph, the first character is mapped to a language-unique cue, allowing the tokenizer to observe minimal but systematic language distinctions during vocabulary learning. The multilingual tokenizer is thus encouraged to distinguish homographs across languages, potentially allocating distinct subword units per language and enabling more balanced embeddings.

To examine whether this approach is useful beyond vocabulary construction itself, we evaluate the proposed language cues in a multilingual machine translation setting. We compare baseline and cue-based English-to-X translation models across several target languages, using both BPE and ULM tokenization and evaluating on general as well as homograph-focused test sets. While the results do not suggest a uniform improvement in every setting, they indicate that language cues can lead to small and encouraging gains, particularly in some evaluation conditions involving homographs and on more challenging benchmarks. These findings provide preliminary evidence that introducing lightweight language signals during tokenization may help mitigate problematic shared-token behavior in multilingual downstream tasks.

2 Homographs in Tokenization↩︎

Over the years, a range of approaches has been proposed to improve multilingual language models. Studies have looked into specialized vocabulary allocation schemes, composition of multilingual tokenizers from monolingual ones, increasing multilingual vocabularies, alternative encoding methods, vocabulary extension, and data-balancing or adaptation strategies for low-resource languages. Together, these methods address important challenges such as vocabulary bottlenecks, unfair allocation across languages, and weak performance on underrepresented languages. At the same time, they mostly treat multilingual tokenization as a problem of global vocabulary design, corpus balance, or model adaptation.

One common assumption in multilingual language modeling is that some degree of vocabulary overlap across languages is useful. [9] show that multilingual BERT [10] can transfer knowledge between languages even without explicit cross-lingual training, which suggests that shared structure can be useful in multilingual models. [3] show that vocabulary overlap can be beneficial in some settings, particularly for sentence-level cross-lingual tasks, even if it is not equally helpful for all tasks. [11] refine this picture by asking what kind of overlap is actually helpful. They show that overlap is usually better than having completely separate vocabularies, but mostly for words with similar meanings. Overlap between words with different meanings is much less useful , highlighting the concern for the cases of homographs and false friends.

Recently, [12] and [13] showed that models often struggle to disambiguate cross-lingual homographs and may rely too heavily on orthographic similarity rather than semantic interpretation, adding to the motivation of the present work. We do not treat token sharing only as a question of vocabulary size, clustering, or vocabulary allocation. Instead, we focus on cases where identical surface forms may be tokenized in the same way, even when their meanings or usage differ across languages. Our goal is not to eliminate overlap altogether, but to provide the tokenizer with a signal so that potentially misleading shared forms can be distinguished earlier in the pipeline. In that sense, the proposed language cues operate at a different level from approaches such as [14]’s, where language embeddings are introduced on the model side during pretraining. Here, the language signal is introduced directly into the input data, at the point where shared subword units are first created.

2.1 Analysis of Homograph Tokenization↩︎

To further motivate the proposed approach, we first examine how cross-lingual homographs are tokenized by existing subword tokenizers in monolingual and bilingual settings. If multilingual tokenizers tend to segment homographs in the same way as their monolingual counterparts, even when meanings differ across languages, this would suggest that the same written form is being segmented similarly regardless of language. Such a pattern would support the need for an explicit language signal during tokenizer training.

We use a false friends dataset [15], [16] to focus on the clearest case in which identical surface forms should not necessarily be tokenized in the same way across languages. 1 compares monolingual and multilingual tokenizers on this subset under two settings: BPE,¹ and SaGe with a BPE-initialized vocabulary. All tokenizers are trained on monolingual and concatenated multilingual training corpora [18], [19]. The token sequence triplets resulting from each setting (English, L2 \(\in\{\)German, French\(\}\), bilingual) are grouped into several cases: same_splits denotes cases where the English monolingual, L2 monolingual, and multilingual tokenizers all produce the same segmentation; en_t=mult_t and L2_t=mult_t denote cases where the multilingual tokenizer matches only one of the monolingual tokenizers; en_t=L2_t denotes cases where the two monolingual tokenizers agree with each other but differ from the multilingual tokenizer; and different_splits denotes cases where all three tokenizations differ.

The comparison shows a clear contrast between tokenizer families. Standard BPE often produces the same segmentation in monolingual and multilingual settings, while SaGe diverges more strongly. To determine whether this pattern extends beyond the false-friend subset, we next examine the broader set of homographs. We build a homograph dataset using a dictionary repository [20], adding four more languages to our set, and use Earth mover’s distance (EMD) to compare the tokenization distributions produced by the BPE and SaGe tokenizer families. Unlike exact category counts, EMD provides a single measure of how different the overall tokenization patterns are. Our EMD function is detailed in 7.

2 shows that the broader homograph set follows the same general pattern observed for false friends. Across all language pairs, BPE produces many more cases in the same_splits category, indicating that monolingual and multilingual tokenizers often preserve the same segmentation. In contrast, the SaGe variant yields substantially more cases in the different_splits category, suggesting greater divergence between monolingual and multilingual tokenization. UnigramLM (2, right) follows a similar pattern to BPE, with monolingual and multilingual tokenizers often producing the same segmentation for homographs, and SaGe reversing the trend. Taken together, these results suggest that standard BPE and ULM tend to treat cross-lingual homographs in a more language-agnostic way.

3 Language Cues for Homographs↩︎

We introduce lightweight language-specific cues during tokenizer training, at the stage where subword units are learned from text. First, we construct language-pair-specific homograph sets for English and each target language. Using these sets, we modify the training corpus by inserting language-specific Unicode cues into selected words, replacing the first character of each marked homograph with a language-unique cue. We then train tokenizers on the modified corpus and use them in a downstream machine translation setting, where we compare cue-based systems with standard baselines across several English-to-L2 language pairs.

3.1 Homograph Set Construction↩︎

To construct the homograph sets used in this work, we extracted candidate words from the all-words-in-all-languages repository [20]. For each English–L2 language pair, we found the intersection between the English word list and the corresponding target-language word list, yielding a set of shared surface forms. We then applied two filtering steps: first, we removed words shorter than three characters, in order to avoid irregular forms such as Roman numerals (e.g., ii). Then, we filtered out words that appeared less than 5 times in either language-specific training corpus. This excluded words that are either very rare overall or strongly imbalanced across the two languages, and are therefore less likely to provide reliable evidence during tokenizer training. The resulting homograph sets are diverse and include names, entities, false friends, and medical terms, among other forms.

3.2 Language Cues↩︎

Our method introduces language-specific cues directly into the tokenizer training data. 6 contains the full mapping from lowercase English letters to the language-specific Unicode cue characters we used and explains our selection. For each language, we define a unique set of 26 Unicode characters, where each cue is mapped to one of the 26 lowercase English letters.² This creates a simple one-to-one correspondence between the first letter of a homograph and a language-specific replacement character. When a word from the homograph set is encountered in the corpus, its first character is replaced with the cue corresponding to that language and letter.

The rationale behind our cue protocol is to provide a lightweight language signal without forcing the two forms into fully separate representations. Recoverability of the original text is maintained, as the cue replaces only the first character in a systematic one-to-one manner. We verified that the selected characters are extremely rare in the corpora we used for all tasks, and never exist in beginnings of words where all other characters are standard. The cue serves as an additional indicator of the linguistic context in which the homograph appears, while the remaining characters can still be shared across languages when the tokenizer finds this useful. For example, if the English and German forms of bank receive different initial cues, the tokenizer may still learn shared subword units such as ank or nk. This allows the method to distinguish language-specific uses of a homograph while preserving the possibility of partial cross-lingual sharing.

4 Evaluation↩︎

Before evaluating the proposed method in downstream settings, we examine its static vocabulary features. We compare multilingual tokenizers trained with language cues against their standard baseline counterparts, with the goal of identifying how cue injection changes basic tokenization behavior. To characterize these differences, we use several tokenizer-level measures that capture whether cue injection changes the compactness of the learned vocabulary and the granularity of word segmentation. 3 presents statistics for token length at the vocabulary level and for Rényi efficiency [21] over a held-out development set for English–German and English–French. Cue-based multilingual tokenizers differ only modestly from their baseline counterparts, but the differences are fairly consistent.³ In both language pairs, over both tokenizer variants, cued tokens are slightly shorter and efficiency is slightly higher. The former is expected: as alphabet size increases, more tokens are necessary to encode sequences. The latter is encouraging: higher Rényi efficiency signifies a more balanced distribution of token types across a corpus.

Figures 3–4 compare the fertility, presented as tokens per word distributions, between baselines and cue-based multilingual tokenizers. Cued tokenizers produce fewer single-token words and more words split into three to five subwords, suggesting that cue injection encourages somewhat finer segmentation of the training data.

Although these intrinsic changes are modest, they indicate that cue injection shifts tokenizer behavior in a consistent direction before any downstream training is performed.

4.1 Language Modeling↩︎

For each evaluated setup, we train a small GPT language model [22] to evaluate the effect of language cues through perplexity. Model architecture details are in 9. The models are trained on multilingual text from the OPUS-100 [23], [24] parallel training data, split into 90% training and 10% evaluation. For each tokenizer type, we train one model on the baseline-tokenized data and one model on the cue-tokenized data, and compare their perplexity on the evaluation split.

4 and 5 report token-level and word-level perplexity for the baseline and cue-based models. Token-level perplexity is lower for the cue-based models across all language pairs under both BPE and UnigramLM. However, word-level perplexity is generally higher for the cue-based models, with the only exception being EN–IT under UnigramLM, where it improves slightly. Since token-level perplexity depends on segmentation, word-level perplexity provides the more stable comparison in the present setting. Under this normalization, the cue-based models do not show a consistent intrinsic language-modeling advantage. The token-level perplexity results should therefore be interpreted cautiously.

4.2 Machine Translation↩︎

To further test downstream evaluation, we apply the cue-based tokenization setup to machine translation from English to L2. For each language pair, we compare two main systems: a baseline model trained on the original parallel data, and a cue-based model trained on a modified version of the same data in which homographs from the constructed set are marked with language-specific cues. Aside from the cue injection step, the baseline and cue-based systems follow the same training setup. Models use a shared vocabulary for both languages, and a shared input–output embedding matrix structure, providing a direct assessment of the cue protocol in its intended setting. We used vocabulary size 8,000. Further model architecture and implementation details are in 10.

4.2.0.1 Data

We evaluate the translation models on both standard parallel training data and additional evaluation sets in order to assess the proposed method under different conditions. As an in-domain benchmark, we use data from OPUS-100 [23], [24], a large multilingual parallel corpus covering many language pairs. In addition, we source FLORES+ [25], [26], a higher-quality multilingual dataset designed to support more reliable cross-lingual evaluation. We use it to examine whether the effects of the proposed method remain visible under stricter evaluation conditions. Finally, from each dataset we derive a homograph-focused subset: this includes, for each language pair, all and only the sentences that contain at least one homograph from the constructed homograph set on the source side.

4.2.0.2 Results

Tables 6 and 7 present the BLEU scores for baseline and cued systems across the general and homograph-focused evaluation sets. Overall, the effect of cue injection is mixed, but several encouraging patterns emerge. Under BPE, the cued system improves over the baseline in many settings. These improvements are usually small, but they appear across both the full test sets and the homograph-focused subsets. At the same time, there are still cases in which the baseline remains stronger, such as EN–DE on OPUS-100 and EN–SV on FLORES+. A second pattern is that, for BPE, the gains are often more visible on FLORES+ than on OPUS-100. This is especially noticeable for EN–IT, EN–RO, and EN–DE. The homograph-focused subsets generally follow the same direction as the full test sets, although they do not consistently show larger gains. This suggests that the cues may influence translation behavior beyond the selected homograph cases, but not in a uniformly targeted way.

The UnigramLM results are less stable. Some language pairs show gains, such as EN–DE and EN–FR on OPUS-100 and EN–ES and EN–RO on FLORES+, but in other settings the baseline remains stronger, particularly for EN–SV. Compared with BPE, UnigramLM shows a less consistent response to cue injection across languages and evaluation sets, possibly because its probabilistic vocabulary selection makes it less sensitive to inserted cues.

Taken together, these results suggest that our cueing method is promising, but its effect is clearly tokenizer- and language-dependent. BPE appears to benefit more consistently from the cues, while UnigramLM produces a more mixed picture. Although the gains are modest, their repeated appearance across several settings supports the idea that lightweight language signals at the tokenizer level can have a meaningful downstream effect.

4.2.0.3 Statistical Robustness

For statistical robustness, we apply paired bootstrap resampling with 1,000 iterations for each evaluation set. In each case, we compute the bootstrap distribution of the BLEU difference, defined as cue BLEU minus baseline BLEU, and report its mean, standard deviation, and 95% confidence interval. This allows the observed BLEU differences to be examined beyond a single point estimate and helps assess whether the direction of the effect remains stable under repeated resampling. The full results are in 11. On BPE, two confidence intervals out of 24 do not cross zero, favoring the cued model: English–Italian on both FLORES+ and its homograph subset (mean diff 0.71 and 0.75, resp.). In UnigramLM, three confidence intervals favor the base model: English–Romanian on OPUS-100 (\(-\)​0.82), and English–Swedish on FLORES+ and its homograph subset (\(-\)​2.28 and \(-\)​2.17, resp.). The aggregate means for the bootstrap setup are \(+\)​0.1724 for BPE, and \(-\)​0.2495 for UnigramLM, greatly dominated by the two Swedish FLORES+ datasets (without which the difference drops to 0.0699).

The BPE results seem to suggest that the cue-based system often moves in a favorable direction, echoing the relatively stronger results (compared to UnigramLM) on Rényi efficiency and perplexity. UnigramLM’s results do not replicate the overall success cues had on intrinsic and language modeling benchmarks. While the differences are still small in magnitude, the number of language pairs and setups considered allow us to speculate as to the reason these two algorithms behave differently: BPE is bottom-up, merging tokens based on their components’ co-occurrence statistics, which focuses the cues’ role as alphabet items that follow similar patterns as the characters they replaced; tokens with cues will form at a slower rate than tokens with the original characters but at a similar order, which will eventually lend consistency to the resulting vocabulary. UnigramLM, on the other hand, is a top-down algorithm prioritizing candidate token likelihood, which causes it to be less sensitive to low-frequency tokens like the ones cues appear in, making its behavior on them more erratic and denying them consistency. The cues’ differentiating role may be enough to improve intrinsic statistics and LM prediction, but the semantics-heavy task of translation requires more.

4.2.0.4 Instance Analysis

It is difficult to find examples in well-established translation datasets where false friends occur in contexts that can throw English-to-L2 translations off. However, we found several instances where homograph words were translated into native words in the cued models, where the baseline kept the source form, as borrowed-from-English variants. One example is the word tour in the English sentence the rock band was due to tour the united states and canada until september 16. The baseline Spanish model translated this to la banda de rock se debió al tour a los estados unidos y a canadá hasta el 16 de septiembre., while the cued model produced la banda de rock se debía a la gira por los estados unidos y canadá hasta el 16 de septiembre. In another example, the English italy’s national football, along with german national football team is the second most successful team in the world and were the fifa world cup champions in 2006. was translated by the baseline Italian model to il calcio nazionale dell’italia, insieme alla nazionale di calcio tedesca, è il secondo team di successo nel mondo ed è stato il campionato della fifa nel 2006., with the cued model preferring squadra.

5 Conclusion↩︎

Multilingual tokenization relies on shared vocabularies in order to represent many languages within a limited number of subword units. While this sharing is often beneficial, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ. We explored this problem through cross-lingual homographs and proposed a simple way to introduce language information earlier in the tokenization process.

The results presented in this work are modest, but generally encouraging. The proposed language cues do not lead to consistent gains across all settings, and their effect depends on the tokenizer, the language pair, and the evaluation set. At the same time, the repeated appearance of small improvements suggests that this type of intervention may be capturing a limitation of standard multilingual tokenization. This leaves several promising directions for future work, such as exploring whether language cues can be developed into a more general tokenization method rather than remaining a targeted intervention, or extending them to multi-script settings.

Limitations↩︎

One limitation of this work is its experimental scope. The downstream evaluation is restricted to English-to-L2 machine translation task for a specific set of languages, most of which are fairly related and use Latin-based scripts. This means that the findings should be interpreted with caution, since it is not yet clear how well the same behavior would carry over to more distant language pairs, other writing systems, or broader multilingual settings.

The proposed cueing method is also intentionally narrow: it applies only to a selected set of cross-lingual homographs, and therefore does not address other forms of potentially problematic token sharing. Other forms of cross-lingual ambiguity may not be captured by this design. At the same time, the empirical gains are generally modest and not fully consistent across tokenizers, datasets, and language pairs. Taken together, these limitations suggest that the current results are best viewed as encouraging initial evidence rather than as a complete solution. Future work will be needed to test whether similar ideas remain useful in larger multilingual models, on other tasks, and in more general language-aware tokenization schemes.

Acknowledgements↩︎

This research was supported by the Israel Science Foundation (grant No. 1166/23).

6 Cue Replacement Mappings↩︎

¿tbl:tab:cue95table? presents the full mapping between English ASCII characters and those for other languages.

The language cues were selected from the Extended Latin Unicode block. This choice was made for practical reasons related to SentencePiece training. First, these characters remain compatible with SentencePiece normalization. Second, they make it possible to keep the split_by_unicode_script argument set to true, which is preferred in order to avoid mixing characters from unrelated scripts within the same token. At the same time, because the cues belong to a Latin-based block, they can still participate in merge operations during tokenization training.

7 Earth Mover’s Distance↩︎

Earth mover’s distance (EMD) is used for comparing two distributions where the between the points on the supports are meaningful. For geographic histograms, this is the most natural: euclidean distance dictates the amount of effort needed to from one location to another; EMD then minimizes the total effort needed given the complete distributions. For less-trivial cases like ours, custom distance functions can be defined. The function we implement for \(d(x,y)\) is given in 8, intended to distance bins that have different counts of identical tokenizations and bring closer the bins where the multilingual vocabulary behaves regularly.

8 Intrinsic Results on All Language Pairs↩︎

Figures 5, 6, 7, and 8 present the fertility comparisons for English and Spanish, Italian, Swedish, and Romanian, respectively.

9 GPT Experimental Setup↩︎

The parameters used for the language model experiments are presented in 9.

10 NMT Experimental Setup↩︎

All machine translation experiments were conducted using the fairseq-based [27] implementation described by [28]. The only modification made to fairseq was in the post-processing of the validation and test outputs, so that BLEU would be computed on the correctly reconstructed text. Final BLEU scores were computed with SacreBLEU after detokenization, matching the post-processing pipeline used for the generated outputs. Apart from this step, the training pipeline followed the standard fairseq workflow, with parameters provided in 10. The tokenizer used in all experiments was SentencePiece with the standard settings. The translation architecture used in all experiments was transformer_iwslt_de_en. As a result, each model used a single tokenizer with a shared vocabulary of size 8,000, trained either with BPE or with UnigramLM. Training took between 2 and 4 hours for all models on a given setup (baseline / cues \(\times\) BPE / Unigram) on an NVidia GeForce RTX 4090 GPU server.

11 Bootstrap MT↩︎

Tables 11 and 12 present full results for the bootstrap tests on BPE and UnigramLM, respectively.

4pt

4pt

References↩︎