1 Introduction↩︎

Language Identification (LID) systems are an important component of today’s natural language processing (NLP) landscape. They are particularly important for the training and evaluation of large multilingual language models, where they are used to: (i) construct labeled corpora with adequate coverage of the world’s languages, (ii) analyze the composition of large-scale training data, and (iii) evaluate models’ cross-lingual performance. As a result, reliable and efficient LID systems have become a vital tool for multilingual language modeling pipelines.

Simple and efficient algorithms have worked well for LID [1]–[3]; many regard it as a “solved” task since performance on high- and medium-resource languages is often near perfect [4], [5]. However, LID systems often still perform poorly on low-resource languages, closely related language pairs, or fine-grained dialectal distinctions [6]–[8]. Even state-of-the-art language models fail to consistently identify less common languages [9]. Collectively, these findings suggest that LID remains far from solved.

In this work, we propose a simple and computationally efficient approach to LID that combines the generative modeling framework underlying the tokenization algorithm with the classical Bayes decision rule. Concretely, assumes that a string is generated as a sequence of latent subword units drawn independently from a unigram distribution. Rather than learning a single global distribution, we estimate a separate unigram distribution per language. Given these language-specific distributions, we approximate the likelihood of an input string in each language using the probability of its most likely subword segmentation under the corresponding unigram model—intuitively, this allows segmentation itself to be treated as a language-dependent latent variable. Finally, we apply Bayes’ rule to the resulting language-conditional likelihoods to obtain a posterior distribution over languages, and we assign the language label with the highest posterior probability.

The proposed method offers several advantages over prior LID approaches. While related to earlier generative LID models [1], [10], it departs in a key respect by treating segmentation as a language-specific phenomenon rather than enforcing a fixed tokenization across languages. This design choice encourages token boundaries that align with meaningful morphological structure rather than arbitrary statistical artifacts, an approach that is both linguistically motivated and empirically supported in prior NLP work [11], [12]. The per-language distribution formulation further enables incremental extension to new languages or dialects without retraining existing models, and empirically, only a few samples per language are needed to learn language-specific parameters that achieve strong performance. Finally, inference is computationally efficient: using clever dynamic programming tricks, inference can be done in an amount of time comparable to the inference step of the classic tokenization algorithm.

We compare against widely used LID systems: , , and . Across a diverse set of benchmarks, achieves performance competitive with these baselines while requiring substantially fewer labeled examples. The method is particularly effective in regimes where LID remains challenging. In low-resource settings, attains \(\sim\)​70% accuracy with as few as five samples per language and \(\sim\)​90% accuracy with fewer than 50 samples. On fine-grained dialect identification [7], improves macro F1 from 0.53 to 0.72 relative to a baseline. From a practical perspective, exhibits substantially shorter training times than , while achieving comparable inference throughput, despite using a non-optimized research implementation. Taken together, these results suggest that treating segmentation as language-specific, rather than as a fixed preprocessing step, enables more effective LID systems.¹

2 Language Identification↩︎

Let \(\mathbf{s}=\langle s_1, \dots, s_T\rangle\) be a string, i.e., a sequence of characters² from an alphabet \(\Sigma\), and \(\Lambda\) a set of language labels. Closed-set LID is the task of assigning a language \(\ell\in\Lambda\) to \(\mathbf{s}\). This problem is often framed probabilistically, i.e., \[\label{eq:lang-id} f_\mathtt{lid}(\mathbf{s}) = \mathop{\mathrm{\mathrm{argmax}}}_{\ell\in\Lambda} \,\,p(\ell\mid\mathbf{s})\tag{1}\] and the task then becomes approximating \(p\) with a learned model \(p_{\theta}\). Two modeling paradigms are typically employed for solving this task. Discriminative approaches model this distribution using a score \(f_\theta(\mathbf{s},\ell)\in\mathbb{R}\), which is normalized using a softmax function: \[\label{eq:discriminative} p_{\theta}(\ell\mid \mathbf{s}) = \frac{\exp\{f_\theta(\mathbf{s},\ell)\}}{\sum_{\ell'\in\Lambda}\exp\{f_\theta(\mathbf{s},\ell')\}}\tag{2}\] Generative approaches model this distribution via language-conditional likelihoods \(p_{\theta}(\mathbf{s}\mid \ell)\); the desired distribution can then be computed via the Bayes rule:³ \[\label{eq:bayes} p_{\theta}(\ell\mid \mathbf{s}) = \frac{p_{\theta}(\mathbf{s}\mid \ell)}{\sum_{\ell'\in\Lambda} p_{\theta}(\mathbf{s}\mid \ell')}\tag{3}\] Prior approaches to LID proposed different methods for parameterizing these distributions, as we discuss next.

2.1 Prior Approaches↩︎

2.1.0.1 \(n\)-gram Models.

Early work on LID established the effectiveness of using plain character-level \(n\)-gram statistics with the generative modeling approach of 3. Explicitly, they estimated language-conditional probabilities with character-level \(n\)-gram language models—often augmented with standard backoff or smoothing techniques [13]. These models set a long-standing baseline for robust LID, particularly on short or noisy texts [14], [15].

2.1.0.2 Discriminative Frameworks.

In supervised settings, discriminative frameworks often showed improvements over earlier generative approaches, particularly when sufficient training data was available and languages were well represented [5]. The standard approach is to represent input strings as feature vectors, encoding statistics of character \(n\)-grams such as term-frequency or TF-IDF weights. Formally, let \(\gamma(\mathbf{s}) \in \mathbb{R}^d\) denote such a feature representation; this representation is then scored using a linear discriminative model, instantiating the framework of 2 , with a class-specific linear function \(f_\theta(\mathbf{s}, \ell) = \theta_{\ell}^{\top} \gamma(\mathbf{s})\), where \(\theta_\ell\) are language-specific parameters learned by minimizing the empirical cross-entropy over a labeled training set. A canonical example of this approach is [2]. This architecture represents an input string as a bag of character \(n\)-grams, mapping each to a dense vector. These vectors are averaged to form a single hidden representation, which is then passed to a linear classifier to predict the language label. Despite its architectural simplicity, this approach provides a strong accuracy–efficiency trade-off and has become a standard reference point in LID research. Indeed, recent systems such as and combine this architecture with large-scale, carefully curated training data to achieve state-of-the-art coverage and reliability [9], [16]. Subsequently, [17] applies supervised contrastive learning to the same architecture, encouraging representations of examples from the same language to be close and different languages to be separated.

2.1.0.3 Neural Approaches.

In more recent years, neural approaches have been applied to LID. Most are based on character- and byte-level neural sequence models, which arguably provide a more flexible alternative to bag-of-\(n\)-gram representations such as . Character-level CNN and bidirectional RNN baselines learn feature representations directly from raw text, eliminating the need for explicit feature engineering. These systems have shown competitive performance, including for streaming or short text LID [18]–[20]. Google’s Compact Language Detector v3 ( ) is based on a neural classifier: input strings are first transformed into normalized character \(n\)-gram frequency features and then a shallow feed-forward network uses these features to predict a posterior over \(\Lambda\). While multilingual Transformer encoders (e.g., mBERT, XLM-R) have shown high accuracy in LID with task-specific fine-tuning, the substantial computational overhead and memory footprint of these architectures often render them impractical for high-throughput or resource-constrained environments.

2.2 Open Problems↩︎

2.2.0.1 Varietal and dialectal discrimination.

Closely related languages or dialects (e.g., Bosnian, Croatian, and Serbian; Hindi and Urdu; or Maghrebi and Levantine Arabic) have highly similar statistical properties, with a strong overlap in subword-structure and orthography. As a result, even state-of-the-art models struggle with distinguishing between them. Further, there is typically insufficient labeled training data to be able to learn these more fine-grained distinctions [7], [21].

2.2.0.2 Low-resource performance.

As with many NLP datasets, there exists much more data for a core set of high-resource languages than for the long tail of low-resource ones. Further, the data that does exist for these low-resource languages is often noisy, e.g., contaminated with HTML code or snippets of text from high-resource languages [22]. Parameter estimates for these languages are often poor and, consequently, so is LID performance [6], [9]. Reweighting, data augmentation, and contrastive objectives have shown promising improvements but have not yet eliminated performance gaps across different resource levels.[16], [17], [23]

2.2.0.3 Domain shift and orthographic noise.

Models trained on curated text collections often generalize poorly to informal or domain-mismatched inputs such as social media or transliterated scripts [8], [24]. Many systems are also not robust to perturbations in orthography (e.g., diacritic omission in languages like French or Vietnamese), as these substantially alter character \(n\)-gram statistics, leading to pronounced drops in both accuracy and calibration [6].

2.2.0.4 This work.

UniLID directly targets the first two challenges. As we show in (6), it substantially improves dialect discrimination on DSL-ML 2024 (macro F1 0.53 \(\rightarrow\) 0.72) and yields large sample-efficiency gains in the low-resource regime (\(\sim\)​70% accuracy with five samples per language). On out-of-domain inputs it gives partial improvements; on orthographic noise we find UniLID and the strongest baseline to be roughly tied (6).

3 UnigramLM↩︎

Tokenization is the process of segmenting strings into sequences of subword units, i.e., character spans referred to as tokens in this context. Formally, let \(\Sigma\) be the alphabet of characters \(s\), and \(\mathcal{V}\) the vocabulary of tokens \(v\). Then, tokenization is the mapping of sequences of characters \(\mathbf{s}=\langle s_1,s_2,\dots\rangle \in\Sigma^*\) to sequences of tokens \(\mathbf{v}=\langle v_1,v_2,\dots\rangle\in\mathcal{V}^*\), which we denote as \(\tau\colon \Sigma^* \rightarrow \mathcal{V}^*\). This mapping is typically injective and thus reversible: the original string can be obtained from its tokenized form through a deterministic detokenization function \(\rotatebox[origin=c]{180}{\tau}\colon \mathcal{V}^* \rightarrow \Sigma^*\), i.e., \(\mathbf{s}=\rotatebox[origin=c]{180}{\tau}(\tau(\mathbf{s}))\). As each token (typically) represents a character span, the definition of \(\rotatebox[origin=c]{180}{\tau}\) is often simple, reducing to concatenating the characters within each token together, i.e.: \(\rotatebox[origin=c]{180}{\tau}(\langle v_1, v_2, \ldots\rangle) \mathrel{\stackrel{\textrm{\tiny def}}{=}}v_1 \circ v_2 \circ \cdots\). For example, \(\rotatebox[origin=c]{180}{\tau}(\langle\text{`he'},\text{`llo'}\rangle) = \text{\rmfamily\textit{``hello''}}\). Notably, the detokenization function is not necessarily 1-to-1: different token sequences may yield the same string. Reusing the example above, we observe that \(\rotatebox[origin=c]{180}{\tau}(\langle\text{`hell'},\text{`o'}\rangle)\) still detokenizes to \(\text{\rmfamily\textit{``hello''}}\). The non-injectivity of \(\rotatebox[origin=c]{180}{\tau}\) is an aspect of tokenization that we will later take advantage of in our LID approach. Going forward, we refer to \(\mathbf{v}\) as a segmentation of \(\mathbf{s}\) if \(\rotatebox[origin=c]{180}{\tau}(\mathbf{v}) = \mathbf{s}\) and define \(\mathcal{T}_{\mathcal{V}}(\mathbf{s}) = \{\mathbf{v}: \rotatebox[origin=c]{180}{\tau}(\mathbf{v}) = \mathbf{s}\}\) as the set of all valid segmentations of \(\mathbf{s}\) under vocabulary \(\mathcal{V}\).

Numerous tokenization algorithms exist. These algorithms define a mapping function \(\tau\) and the method for learning the parameters of this mapping function from data. For example, the Byte-Pair Encoding [25] algorithm defines a \(\tau\) parametrized by a list of merges, and a method to learn this list. In this work, we focus on the algorithm [26], which will form the basis of our LID approach.

3.1 Generative Model↩︎

In broad strokes, casts tokenization as the uncovering of a latent segmentation: it assumes strings are simply the surface form of a latent sequence of tokens \(\mathbf{v}\), where each token is drawn independently from a categorical (unigram) distribution.

Let \(\boldsymbol{\phi}\in \Delta^{|\mathcal{V}|-1}\) denote a probability distribution over a vocabulary \(\mathcal{V}\), where \(\boldsymbol{\phi}[v]\) gives the probability of token \(v\). By construction, \(\boldsymbol{\phi}[v]\geq 0\) and \(\sum_{v\in\mathcal{V}}\boldsymbol{\phi}[v]= 1\). Under the framework, a token sequence of fixed length \(M\) is generated by independently sampling each token from this distribution, and the likelihood of a token sequence \(\mathbf{v}= \langle v_1, ..., v_M\rangle\) is the joint probability of those tokens:⁴ \[\label{eq:pZ95single} \underset{\text{for}\,\,m= 1, \ldots, M}{v_{m} \stackrel{\text{i.i.d.}}{\sim} \mathrm{Categorical}(\boldsymbol{\phi})} \, \implies p_{\boldsymbol{\phi}}(\mathbf{v}) \mathrel{\stackrel{\textrm{\tiny def}}{=}}\prod_{m=1}^{|\mathbf{v}|} \boldsymbol{\phi}[v_{m}]\tag{4}\] A string \(\mathbf{s}\) is then deterministically generated by detokenization. Consequently, the probability of \(\mathbf{s}\) is obtained by marginalizing over all valid segmentations: \[p_{\boldsymbol{\phi}}(\mathbf{s}) = \sum_{\mathbf{v}\in\mathcal{T}_{\mathcal{V}}(\mathbf{s})}p_{\boldsymbol{\phi}}(\mathbf{v})\] which formalizes the data-generating process of a string \(\mathbf{s}\) under ’s assumptions. The posterior distribution over segmentations then follows naturally: \[\label{eq:posterior} p_{\boldsymbol{\phi}}(\mathbf{v}\mid\mathbf{s}) = \frac{p_{\boldsymbol{\phi}}(\mathbf{v})}{p_{\boldsymbol{\phi}}(\mathbf{s})} \quad \text{for \mathbf{v}\in \mathcal{T}_{\mathcal{V}}(\mathbf{s}) and 0 otherwise}\tag{5}\]

Crucially, both \(\boldsymbol{\phi}\) and \(\mathcal{V}\) are unknown; we only observe the strings resulting from the generative process. The algorithm estimates these parameters from text data by combining the Expectation–Maximization (EM) algorithm with an iterative vocabulary pruning procedure.

3.2 Learning Model Parameters↩︎

The aim of the procedure proposed by the algorithm is to find the values of \(\boldsymbol{\phi}\) and \(\mathcal{V}\) that maximize data likelihood. Under the assumptions about the generative process of strings, the “complete” data consists of \((\mathbf{s},\mathbf{v})\) pairs, i.e., strings and the sequence of tokens that produced them. Since we only observe strings, and not their underlying segmentations, we cannot directly optimize for complete-data log-likelihood; we must instead optimize for the observed-data log-likelihood, where the observed data is our text corpus \(\mathcal{C}=\{\mathbf{s}_k\}_{k=1}^K\): \[\label{eq:corpus-likelihood} \mathcal{L}(\mathcal{C}; \boldsymbol{\phi}) \mathrel{\stackrel{\textrm{\tiny def}}{=}}\sum_{k=1}^K\,\log\!\!\sum_{\mathbf{v}\in \mathcal{T}_{\mathcal{V}}(\mathbf{s}_k)} p_{\boldsymbol{\phi}}(\mathbf{v})\tag{6}\] The quantity in 6 is difficult to optimize directly due to the log-sum structure. The EM algorithm establishes a relationship between this quantity and the expected value of the complete-data log-likelihood, which allows us to solve an easier problem in its stead. In short, the EM algorithm iteratively optimizes parameters with respect to the expected value of the complete data log-likelihood given our observed data \(\mathcal{C}\) and the distributions induced by our current belief of model parameter values. In the context of , this boils down to performing the following two steps iteratively until our estimates \(\boldsymbol{\phi}^{(n)}\) converge:⁵

1. E-Step. Let \(c_{v}(\mathbf{v}) \mathrel{\stackrel{\textrm{\tiny def}}{=}}\sum_{m=1}^{|\mathbf{v}|}\mathbb{1}\{v_m= v\}\), i.e., the number of times token \(v\in\mathcal{V}\) appears in a segmentation \(\mathbf{v}\). Given current parameters \(\boldsymbol{\phi}^{(n)}\) and fixed \(\mathcal{V}\), the E-step computes expected counts in our corpus as \[\widehat{c}_{v}(\mathcal{C};\boldsymbol{\phi}^{(n)}) \mathrel{\stackrel{\textrm{\tiny def}}{=}}\sum_{k=1}^K \sum_{\mathbf{v}\in\mathcal{T}_{\mathcal{V}}(\mathbf{s}_k)} p_{\boldsymbol{\phi}^{(n)}}(\mathbf{v}\mid \mathbf{s}_k)\; c_{v}(\mathbf{v})\] This computation is performed using dynamic programming, specifically via the forward-backward algorithm.

2. M-Step. The M-step then computes the new parameter estimates as simply the normalized expected counts of each token: \[\label{eq:mstep} \boldsymbol{\phi}^{(n+1)}[v] \;=\; \frac{\widehat{c}_{v}(\mathcal{C};\boldsymbol{\phi}^{(n)})}{\sum_{v'\in{\mathcal{V}}}\widehat{c}_{v'}(\mathcal{C};\boldsymbol{\phi}^{(n)})}.\tag{7}\]

For a proof of why this is a valid approximation of MLE parameters (for a fixed \(\mathcal{V}\)), we refer the reader to the exposition on in [28].

Intuitively, this procedure resembles maximum likelihood estimation for a categorical distribution, but with a key difference: we cannot compute observed (“hard”) counts of token occurrences, since the segmentation of each string into tokens is latent. We instead compute expected (“soft”) token counts under the current model by marginalizing over all possible segmentations. In this way, the EM procedure integrates out segmentation uncertainty when estimating \(\boldsymbol{\phi}\), yielding parameter estimates that reflect the full distribution over tokenizations rather than a single, fixed segmentation.

3.2.0.1 Learning \(\mathcal{V}\).

The EM procedure above operates over a fixed vocabulary \(\mathcal{V}\). Indeed, the validity of the approximation relies on the fact that \(\mathcal{V}\) is fixed (and known). In the case of our language identification algorithm, we can pre-specify it as, e.g., the vocabulary of a chosen language model. If one wishes to also learn the vocabulary, the algorithm proposes a heuristic approach which works as follows. Initially, \(\mathcal{V}\) is defined as an oversized set consisting of the most frequently occurring substrings in the corpus; this set can be found efficiently using Enhanced Suffix Array-based algorithms [29]. The vocabulary is then “learned” by iterating between: (i) holding it fixed and estimating \(\boldsymbol{\phi}^{(n)}\) via EM; (ii) a pruning step, where the tokens whose removal hurts corpus log-likelihood the least—under current parameter beliefs \(\boldsymbol{\phi}^{(n)}\)—are removed. Via this pruning step, the algorithm ultimately trims the vocabulary down to the user’s desired size.

3.3 Tokenizing Text↩︎

At inference time (i.e., when turning text into tokens), the tokenization algorithm finds the most likely segmentation of \(\mathbf{s}\) under the learned parameters \(\widehat{\boldsymbol{\phi}}= \boldsymbol{\phi}^{(N)}\) as:⁶\[\tag{8} \begin{align} \tau_{\widehat{\boldsymbol{\phi}}}(\mathbf{s}) &\stackrel{\phantom{\text{(\ref{eq:posterior})}}}{=} \mathop{\mathrm{\mathrm{argmax}}}_{\mathbf{v}\in \mathcal{V}^*} p_{\widehat{\boldsymbol{\phi}}}(\mathbf{v}\mid\mathbf{s}) \\ &\stackrel{\text{(\ref{eq:posterior})}}{=} \mathop{\mathrm{\mathrm{argmax}}}_{\mathbf{v}\in \mathcal{T}_{\mathcal{V}}(\mathbf{s})} p_{\widehat{\boldsymbol{\phi}}}(\mathbf{v})\tag{9}\\ &\stackrel{\text{(\ref{eq:pZ95single})}}{=} \mathop{\mathrm{\mathrm{argmax}}}_{\mathbf{v}\in \mathcal{T}_{\mathcal{V}}(\mathbf{s})} \prod_{m=1}^{|\mathbf{v}|}\widehat{\boldsymbol{\phi}}[v_m] \tag{10} \end{align}\] where we dropped \(p_{\widehat{\boldsymbol{\phi}}}(\mathbf{s})\) in 9 since it does not depend on \(\mathbf{v}\). The exact solution to 10 can be found efficiently using a Viterbi-style algorithm, which runs in \(\mathcal{O}(N\cdot T_{\mathrm{max}})\) time, where \(N\) is the length of the input string \(\mathbf{s}\) and \(T_{\mathrm{max}}\) is the length of the longest token \(v\in\mathcal{V}\).

4 ↩︎

We propose a generative modeling approach for the LID task using the framework. In short, we define language-conditional distributions \(p_{\theta}^{\ell}\) by learning unigram distribution parameters for each language \(\ell\), albeit with a shared vocabulary across languages. We then compute language label probabilities, as in 3 , applying the Bayes’ decision rule to ultimately assign the language label.

4.0.0.1 Language-Conditional Distribution Estimation.

Let \(\mathcal{V}\) be the vocabulary of a base tokenizer.⁷ We fix \(\mathcal{V}\) as our shared-across-languages vocabulary, but assume there exists, for each language \(\ell\), a distinct unigram distribution \(\widehat{\boldsymbol{\phi}}_{\ell}\) over \(\mathcal{V}\). Given a corpus \(\mathcal{C}^\ell\) consisting of strings from language \(\ell\), we then learn each of these distributions’ parameters \(\widehat{\boldsymbol{\phi}}_{\ell}\) using the EM procedure described in 3.2. In other terms, similarly to as in standard , we compute an approximate maximum likelihood estimate by maximizing the observed-data log-likelihood, albeit on only a subset of the data: \[\widehat{\boldsymbol{\phi}}_{\ell}\mathrel{\stackrel{\textrm{\tiny def}}{\approx}}\mathop{\mathrm{\mathrm{argmax}}}_{\boldsymbol{\phi}} \,\,\mathcal{L}(\mathcal{C}^\ell; \boldsymbol{\phi})\] As in the standard formulation, this procedure integrates over segmentation uncertainty when estimating parameters. Because data from each language is processed separately, each language gets its own token frequency profile—and consequently its own set of optimal segmentations—despite sharing a common vocabulary across languages. We discuss several design choices we make in our practical implementation of (e.g., initialization of \(\widehat{\boldsymbol{\phi}}_{\ell}\) for the EM loop and floor values for token probabilities) in 5.

4.0.0.2 Inference.

At inference time, we use the inference procedure of discussed in 3.3 to find the most probable segmentation of a string under each language’s \(\widehat{\boldsymbol{\phi}}_{\ell}\). Importantly, this means that the most probable segmentation \(\tau_{\widehat{\boldsymbol{\phi}}_{\ell}}(\mathbf{s})\) can vary across languages. This yields a flexible model where the probability of a string reflects language-specific token frequencies, despite sharing a common vocabulary. We then approximate the probability of a string under language \(\ell\) as the probability of its most probable segmentation in that language: \(p_{\theta}(\mathbf{s}\mid\ell) = p_{\widehat{\boldsymbol{\phi}}_{\ell}}(\tau_{\widehat{\boldsymbol{\phi}}_{\ell}}(\mathbf{s}))\). Finally, we assign the language label that optimizes the posterior probability, i.e., 1 . We write the instantiation of 1 here explicitly for the reader’s convenience:\[\begin{align} f_\mathtt{lid}(\mathbf{s}) = \mathop{\mathrm{\mathrm{argmax}}}_{\ell\in\Lambda} p(\ell\mid \mathbf{s}) =\mathop{\mathrm{\mathrm{argmax}}}_{\ell\in\Lambda}p_{\widehat{\boldsymbol{\phi}}_{\ell}}(\tau_{\widehat{\boldsymbol{\phi}}_{\ell}}(\mathbf{s})) \nonumber \end{align}\]

4.0.0.3 Computational Complexity.

At training time, estimates a separate unigram distribution for each language by running the EM procedure over the corresponding language-specific corpus. For a corpus \(\mathcal{C}^\ell\) of total character length \(N_\ell\) (e.g., number of Unicode code points or bytes), each EM iteration requires computing expected token counts via the forward–backward algorithm, which runs in \(\mathcal{O}(N_\ell\cdot T_{\mathrm{max}})\), where again \(T_{\mathrm{max}}\) denotes the maximum token length. Parameter updates are linear in the vocabulary size \(|\mathcal{V}|\). Since per-language distribution parameters are estimated independently, training parallelizes trivially across languages. At inference time, uses a similar Viterbi-style dynamic program as in : first building a lattice that enumerates all valid segmentations of a string \(\mathcal{T}_\mathcal{V}(\mathbf{s})\) and then finding the maximum probability option among them. The lattice needs only to be constructed once per string, regardless of the number of language-conditional distributions. Finding the optimal segmentation of a string and its probability for each language then takes \(\mathcal{O}(|\Lambda| \cdot E)\) time, where \(E\) is the number of edges in the lattice. Finally, we need a \(\mathcal{O}(|\Lambda|)\) pass to evaluate the Bayes decision rule. In practice, this additional cost is modest relative to other standard components of modern NLP pipelines, e.g., a single forward pass through a neural encoder, which typically dominates end-to-end runtime. Further, the number of edges \(E\) in a lattice will be \(N\cdot b\), where \(b\) is a branching factor of the lattice, which tells us on average how many tokens in the vocabulary start at a certain string position and are part of one of its valid segmentations; typically this is on the order of 1-5 tokens, making the optimal-segmentation step much cheaper than lattice construction. In the pathological case, a lattice could have \(b = T_{\mathrm{max}}\). In practice, \(b \ll T_{\mathrm{max}}\) and so inference takes \(\mathcal{O}(|\Lambda|\cdot N\cdot b + N \cdot T_{\mathrm{max}})\).

For reference, and share a similar input-processing cost linear in \(N\): extracts character n-grams over a sliding window (\(\mathcal{O}(N \cdot k)\) for window size \(k\)). It then averages n-gram embeddings (\(\mathcal{O}(N \cdot d))\) and scores all languages with a single matrix multiplication (\(\mathcal{O}(d \cdot|\Lambda|)\)); ’s scoring is asymptotically equivalent. The structural difference is that the language-scoring step in and is independent of input length, whereas UniLID’s scales with \(N\). In practice this is offset by the fact that the segmentation lattice is constructed once and reused across languages. We report empirical latency in 6.

5 Experimental Setup↩︎

We evaluate the performance of across several prominent LID benchmarks. We note that the majority of our training protocol and evaluation paradigms follow prior work [9], [17]. Our experiments compare against widely-used baselines: , , and . We perform analyses to show data efficiency, robustness to input length, and the impact of using different tokenizer vocabularies.

5.1 Datasets↩︎

We choose benchmarks that cover a spectrum of LID regimes, ranging from broad linguistic coverage and low-resource settings to fine-grained dialect identification: GlotLID-C [9] for large-scale coverage of the long tail of nearly 2,000 language–script labels; UDHR [14] and FLORES-200 [30] for content-controlled cross-lingual evaluation via parallel text across hundreds of languages; DSL-ML 2024 [7] for fine-grained dialect identification across 14 regional varieties with high lexical overlap; Tatoeba [31] for robustness on short, noisy, community-contributed text; and WiLI-2018 [32] for controlled ablations, as its balanced 500/500 train/test split across 235 languages supports controlled comparisons across different dataset stratifications at manageable scale. We use official train/test splits where available; full per-dataset descriptions, including sample counts and split construction, are deferred to 9.

5.2 Baselines↩︎

We compare against the following systems. For a fair comparison, we hold the training dataset constant across systems that we train ourselves. For systems that did not release reproduction code, or for which we could not reproduce the associated paper’s reported performance, we use the released checkpoints.

5.2.0.1 .

We train supervised [2] classifiers using the author’s Python package⁸ and following their recommended hyperparameter settings. Models were trained for 100 epochs.⁹

5.2.0.2 (pre-trained).

Google’s Compact Language Detector v3 ( ) is a lightweight neural network that supports 107 languages and is optimized for short text segments and low-latency inference. We use the publicly available pre-trained model from their Python package.¹⁰ Its language coverage does not fully overlap with that of our test sets. Thus, for a fairer comparison, we provide systems’ results computed over only the subset of languages covered by (right side of 1).

5.2.0.3 (pre-trained).

is the model provided with the GlotLID-C benchmark dataset. Architecturally, it utilizes the framework but differentiates itself through its training data curation. We use their openly available checkpoint, hosted on the same platform as the dataset. The exact training splits for were not released but the authors state that UDHR and FLORES-200 are not part of the training set,¹¹ so we restrict GlotLID-M evaluations to these two benchmarks.

5pt

5.3 Evaluation Metrics↩︎

We report standard evaluation metrics for language identification, namely the macro-averaged F1 score and the macro-averaged false positive rate (FPR). Macro-averaging ensures that each language or dialect contributes equally to the overall score, regardless of class frequency. For analyses of data efficiency and robustness on the WiLI dataset, we additionally report classification accuracy.

5.4 Training↩︎

For each benchmark, we learn language-conditional unigram distributions using the EM procedure of the tokenization algorithm, described in 3.2. In all experiments, we initialize the unigram distribution as the uniform distribution and run EM for \(20\) rounds.¹² To avoid zero-probability segmentations when a token’s expected counts are estimated to be 0 under a given language (most common in the few-sample regime), we floor token probabilities at \(1e-12\). We did not find further smoothing measures to help. We consider two variants of : one that uses a dedicated base tokenizer trained on the LID training set, and one that uses the vocabulary of a pretrained language model; more details below.

5.4.0.1 .

The base version of trains a tokenizer on the training split of each benchmark dataset. We train a byte-level tokenizer using the standard tokenization algorithm. Our implementation closely follows the design choices of the SentencePiece library [33], which provides the original implementation of . Unless otherwise specified, we use a vocabulary size of \(100\text{k}\); ablation experiments indicate that performance is relatively insensitive to this choice (see [sec:ablations]).

5.4.0.2 -X.

This variant uses the vocabulary of a pretrained base tokenizer and learns only the per-language unigram distributions. This setting evaluates ’s ability to operate with pretrained, task-agnostic vocabularies. We report results using the ¹³ tokenizer [34], which we refer to as -Mistral-Nemo. Results obtained with other base tokenizers, which yield comparable performance, are reported in [sec:ablations].

4pt

7pt

6pt

6 Results↩︎

6.0.0.1 Base Results.

We first evaluate on standard LID benchmarks: GlotLID-C, UDHR, and FLORES-200. Although is not always the top-performing system in terms of F1, 1 shows that both and -Mistral-Nemo achieve competitive macro-averaged F1 scores while consistently maintaining low FPR across both large-scale settings and low-resource datasets. Notably, on the largest evaluation set (full GlotLID-C test), reduces FPR by roughly 25% compared to (2.03e-5 vs 2.71e-5)—a property that can be more critical than F1 in many LID applications. For example, in web-scale crawling, a poor FPR for a low-resource language can lead to a training corpus dominated by noise (common languages misclassified as rare ones). Overall, these results demonstrate ’s efficacy as a general-purpose LID system. In subsequent sections, we compare primarily against . It is the only of these systems for which we can control training data ( is an instantiation of , just with a particular dataset); we cannot control for test-set contamination or sample efficiency with the other systems.

6.0.0.2 Dialect Differentiation.

2 reports results on the DSL-ML 2024 benchmark. demonstrates strong performance in comparison to participant submissions in the DSL-ML 2024 shared task. On French dialects, achieves a macro-F1 of 0.534 in comparison to 0.385 for the top shared-task system. provides smaller but consistent gains on the other dialect groups as well: Spanish (0.850 vs 0.823), Portuguese (0.770 vs 0.752), and BCMS (0.769 vs 0.762). It also substantially outperforms on macro F1, improving from 0.53 to 0.72. We note that part of this result is driven by the 0.00 F1 achieved by on several of the Slavic languages. This poor performance may in part be due to the small amount of training data for those languages (see 9 for the dataset breakdown), a regime that 1 likewise evinces that struggles with. There are several potential reasons why performs particularly well in this regime: First, UniLID’s vocabulary \(\mathcal{V}\) provides a structural inductive bias; only per-language token frequencies must be estimated from the data. This stands in contrast to setups like , where high-dimensional embeddings for millions of \(n\)-gram spans must be adjusted to give signal for the language. Second, generative classifiers are known to be more sample-efficient than discriminative ones in low-data regimes [35], and UniLID is generative while fastText is discriminative.

6.0.0.3 Low-resource Regime.

We evaluate sample efficiency on WiLI by varying the training dataset size. Concretely, we train each model on a subset of the WiLI training set, created by taking \(K\) points from each language set. E.g., at \(K=5\), there are a total of \(5\times |\Lambda|\) data points in the entire training set; training subsets are the same for each model at a given value of \(K\). 1 shows system accuracy as a function of \(K\). Results show that dramatically outperforms ¹⁴ in the low-resource regime. With as few as 5 samples per language, achieves \(\sim\)​70% accuracy, while fails to generalize. Performance remains strong even with fewer than 50 samples per language, highlighting the strength of in scenarios in which labeled data is scarce or expensive to obtain. We provide the values displayed in 1 in 6 in 10.

6.0.0.4 Out-of-domain Performance.

3 evaluates generalization by training on Wikipedia (WiLI) and testing on short, noisy community-contributed text (Tatoeba) and formal legal documents (UDHR). demonstrates superior domain robustness, more than doubling the Macro F1 of on Tatoeba (0.414 vs. 0.160). These results evince the generalization abilities of the approach, particularly for short, informal inputs, which is a domain in which current SOTA systems struggle.

6.0.0.5 Robustness Analysis.

4 reports accuracy as a function of input length on WiLI. consistently outperforms across all length buckets, with the largest gains observed for shorter inputs, which is a regime in which most LID systems struggle. This highlights the robustness of ’s language-specific segmentation assumptions for low-context settings, which are common in practical LID applications, such as determining the language of social media posts. We additionally evaluate robustness to character-level corruption on WiLI by stochastically replacing non-whitespace characters at rates of 5% and 10%. Full setup details and results can be found in 10.2. In short, and still perform well in the presence of light noise and degrade at very similar rates. While shows slightly better performance in the \(<10\%\) noise setting, neither system is meaningfully more robust than the other.

6.0.0.6 Vocabulary Sensitivity.

1 includes three UniLID-X variants that reuse off-the-shelf LLM tokenizer vocabularies (Mistral-Nemo, DeepSeek-3.2, Qwen-3); all are within 2.5 F1 of base UniLID on GlotLID-C, with similar behavior for other LLM tokenizers (10). The gap in F1 is not negligible though. Clearly, training the base tokenizer on broad multilingual data, such that it attains a comprehensive multilingual vocabulary, does lead to better performance—both in-distribution (e.g., GlotLID-C train \(\rightarrow\) GlotLID-C test) and out-of-distribution (GlotLID-C train \(\rightarrow\) UDHR, FLORES-200, neither of which appears in the training set; see 1). LLM tokenizers, on the other hand, are optimized for general-purpose language modeling on data distributions that under-represent the low-resource tail of GlotLID-C. We accordingly recommend, for production deployment, training the base tokenizer once on a broad, web-representative multilingual corpus, and then estimating per-language unigram distributions incrementally as data for new languages arrives. -X is intended as a convenience mode for practitioners who wish to reuse an existing LLM tokenizer or to integrate LID into an existing LLM tokenization pipeline at no additional vocabulary cost. We additionally explore the effect of vocabulary size in 10. Performance improves modestly with increasing vocabulary size before plateauing, suggesting that UniLID is relatively robust to this choice and does not require extremely large vocabularies to perform well.

6.0.0.7 Efficiency.

9 reports inference latency and wall-clock training time for , , and measured on identical hardware. Training UniLID on the full GlotLID-C corpus (1,940 language–script classes) takes 17.8k seconds end-to-end versus \(\sim\)​163k seconds for the 100-epoch configuration, an approximately \(9\times\) reduction despite our unoptimized research implementation. Inference latency varies with the label set size: on GlotLID-C (1,940 labels), is \(1.65\times\) slower than (0.274 vs 0.166 ms/sample) and \(1.46\times\) slower than (0.187 ms/sample); on WiLI (235 labels), UniLID is the second-fastest of the three (0.158 ms/sample, vs 0.113 for and 0.427 for ). The inference time dependency on label-set size can easily be mitigated in practice: for deployments with restricted target label sets, the generative formulation allows us to remove excluded languages trivially without changing the results. We discuss this next.

7 Discussion↩︎

7.0.0.1 Limitations.

A main limitation of is its reliance on a unigram assumption, which ignores dependencies between neighboring tokens. Natural language is inherently contextual, and modeling token interactions is a critical part of most NLP approaches. In this respect, trades expressive power for computational and statistical efficiency. However, this design choice is conceptually aligned with widely adopted baselines such as , which treats character \(n\)-grams as an orderless bag of features and thus likewise ignores neighboring token dependencies. The strong empirical performance of both methods suggests that a large fraction of the signal required for practical LID is captured by local subword statistics rather than long-range dependencies. Further, while is efficient at inference, its storage and memory requirements scale linearly with the number of modeled languages. This could prove potentially challenging for scenarios involving thousands of languages under strict latency constraints. Notably, the modularity of our formulation means: (i) the per-language step parallelizes trivially across language partitions; (ii) the generative formulation allows the label set to be subsetted to a deployment’s actual target languages without any retraining. Using these characteristics can drastically mitigate any inference and memory bottlenecks.

7.0.0.2 Extensions and Future Work.

can be extended to text classification tasks outside of LID. Indeed, nothing in the formulation is specific to this task: estimating a separate unigram distribution per class over a shared vocabulary and scoring by Viterbi-segmentation applies to any partition of text with distinct subword statistics, such as domain, register, or source identification. Several other natural extensions and alterations to follow directly from the aforementioned limitations. First, the unigram assumption of could be relaxed so as to incorporate token dependencies. For example, one could define language-conditional \(n\)-gram models over token sequences—either baking this directly into the generative model formulation or learning \(n\)-gram models over texts segmented with the unigram generative model formulation. This approach would substantially increase computational complexity, especially under the first approach where segmentation would need to account for token-pair interactions. To mitigate the scaling of inference costs and memory with the size of the language set, several modifications could be explored: (i) hierarchical decoding: first selecting a script or language family, then discriminating within it would reduce the effective \(|\Lambda|\). This could be done with something as simple as a script detection filter; (ii) sparse per-language distribution representations: in practice the per-language distributions are sparse after EM convergence for any given language (most directly, tokens in scripts a language does not use) so sparse representations could reduce the deployed footprint substantially.

8 Conclusion↩︎

We introduced , a simple and efficient approach for language identification. In short, to predict a string’s language label, we ask: under which language’s unigram distribution is this string most likely? More concretely, we adapt the generative framework of the tokenization algorithm, creating language-conditional probability distributions that enable segmentation to be treated as a language-specific phenomenon. A simple application of Bayes’ rule to per-language string likelihoods then provides a posterior over language labels. Empirically, demonstrates performance competitive with widely used LID systems. In particular, it exhibits stronger data efficiency and greater robustness in low-resource and fine-grained identification regimes, suggesting it will be a valuable asset for curating high-quality, diverse datasets for next-generation multilingual language models.

Impact Statement↩︎

This work aims to advance language identification in low-resource and fine-grained dialectal settings. As LID is often an initial step in multilingual data curation pipelines, the performance of these systems can strongly influence which languages and language varieties are supported by large-scale language models. Consequently, a potential positive impact of this work is to support more inclusive and representative multilingual datasets. By improving performance in low-resource and short-text regimes, the proposed method may facilitate the inclusion of languages and language varieties that are underrepresented in current large-scale corpora. The low false positive rate of our method is of particular importance, as data contamination in these settings can have a severe negative impact. Thus, we believe this method can contribute towards efforts to reduce the disparity in language model performance on high-resource vs.low-resource languages.

As with many language technologies, better LID capabilities raise dual-use concerns. While more accurate identification can facilitate inclusion, it can also be repurposed to filter, restrict, or suppress content associated with specific linguistic communities. In addition, fine-grained dialect identification could be misused for profiling or monitoring individuals or groups based on linguistic characteristics. These risks are not unique to the proposed method but are common to LID and related text classification technologies more broadly. We encourage practitioners to apply LID systems in ways that are consistent with ethical data collection and responsible use of language technologies.

9 Additional Dataset Details↩︎

9.0.0.1 GlotLID-C.

GlotLID-C [9] is a massive open-source benchmark designed to address the long-tail of LID. It covers \(1940\) different language–script classes,¹⁵ significantly expanding the label space compared to traditional benchmarks. The dataset places a strong emphasis on low-resource and closely related language varieties. Train/test splits were not released by the dataset authors; we create our own splits following the pipeline of [17]: for each language, we split the data into 85%/15% for training/testing. We then subset the training set for each language to be at most \(100k\) samples. Finally, we check explicitly for contamination and remove all occurrences of data from our eval only sets (described next) that we find in the training set. The resulting training set contains approximately \(60M\) samples and test set contains approximately \(45M\) samples of 1940 language–script combinations.

9.0.0.2 UDHR [14].

The Universal Declaration of Human Rights (UDHR)¹⁶ dataset contains parallel text across hundreds of languages. Because the semantic content is identical across all samples, UDHR is often used as a control dataset in LID; the parallel texts allow us to control for content difficulty when assessing per language performance. In LID, UDHR is only used as a test set, and its domain differs from most LID training sets, making it well-suited for testing generalization robustness.

9.0.0.3 FLORES-200 [30].

FLORES-200 is another benchmark consisting of parallel texts. It covers 204 languages. Unlike crowdsourced datasets, FLORES-200 consists of professional translations of sentences extracted from Wikimedia projects (WikiNews, WikiVoyage, and WikiBooks). It thus provides a robust measuring stick against which we can assess low-resource language performance. We do not train on the FLORES-200 dataset; we evaluate on the devtest portion.

9.0.0.4 DSL-ML 2024 [7].

To evaluate performance on closely related languages, we use the DSL-ML 2024 shared task dataset. This benchmark focuses on fine-grained dialect identification with 14 labels covering regional variants of five major languages (English, French, Portuguese, Spanish, and South Slavic). It explicitly evaluates a domain in which lexical overlap is high and differentiating factors between dialects are nuanced. We use the train and dev splits released by the shared task, training on the train set and evaluating on the dev set.

9.0.0.5 WiLI-2018 [32].

The Wikipedia Language Identification benchmark contains 235,000 paragraphs balanced equally across 235 languages, all derived from Wikipedia. We use WiLI-2018 to study data efficiency, base tokenizer choice, and the impact of input sequence length on classification accuracy, as it provides sufficient depth (1,000 samples per language, 500 train/500 test) to support statistically significant ablation studies. We use the official train/test splits.

9.0.0.6 Tatoeba (eval only) [31].

Tatoeba is a large, community-curated multilingual corpus of short, user-contributed sentences covering hundreds of languages and language varieties. The dataset is characterized by wide linguistic diversity, informal style, and substantial variation in sentence length and orthography, making it well suited for evaluating LID systems under realistic, noisy conditions. We use Tatoeba to assess robustness in low-resource and cross-lingual settings, where training data are limited and closely related languages frequently co-occur. As we use this dataset for testing out-of-domain robustness, we do not train on any portion of it. Models are evaluated on the entire dataset for the subset of languages that are in their label set. For evaluations with models trained on WiLI, this results in an evaluation on approximately \(~12M\) samples across 201 languages.

10 Additional Ablation Results↩︎

6pt

6pt

8pt

8pt

6pt

4pt

10.1 Viterbi vs. Marginalization at Inference↩︎

uses Viterbi decoding at inference (3.3), scoring each language by its most likely segmentation. An alternative is to compute the exact marginal likelihood \(p(s \mid \ell)\) by summing over all segmentations using the forward algorithm. 9 compares the two strategies on GlotLID-C. The marginalized variant yields a negligible improvement (within noise) at approximately \(2\times\) the inference cost. Given the empirical near-tie, we retain Viterbi as the default inference method.

10.2 Robustness to Orthographic Noise↩︎

To evaluate robustness under realistic input corruption, we apply stochastic character-level perturbations to the WiLI test set: with probability \(p\), each non-whitespace character is independently replaced with another character drawn uniformly from the inventory of characters observed in the WiLI training set. We evaluate at \(p \in \{0\%, 5\%, 10\%, 25\%, 50\%\}\). Example inputs at \(p=5\%\) and \(p=10\%\) are shown below; 10 reports accuracy, macro F1, and macro FPR for and on the perturbed test sets.

10.2.0.1 Example perturbations.

Original

Anton (or Antonius) Maria Schyrleus (also Schyrl, Schyrle) of Rheita (1604–1660) was an astronomer and optician. He developed several inverting and erecting eyepieces…

\(p=5\%\)

Anton (or Antonius) Maria Schyrlezs \(\ldots\) Antonín \(\ldots\) astronmmer and optijian…

\(p=10\%\)

Anton (or Aston,us) MDria SchyrKeus \(\ldots\) of iheith…

At low noise (\(p < 10\%\)), maintains a small edge in F1 and accuracy. At moderate-to-high noise (\(p \ge 25\%\)), degrades more gracefully than : the character n-gram representations underlying appear to absorb localized character corruptions more robustly than ’s segmentation-based scoring, where corrupted characters can fragment otherwise-high-probability tokens. This points to a possible avenue for future work — explicit noise-aware token scoring or character-level smoothing within the framework.

11 Hyperparameter Sensitivity on Dialect Identification↩︎

To validate the configuration used in our dialect experiments (6), we ran a hyperparameter sweep over the number of training epochs and the minimum word-count threshold (\(MC\)) on the DSL-ML 2024 train/dev splits. 11 reports per-dialect F1 along with the macro-averaged F1 across all 14 labels. Performance improves monotonically with more epochs, with the 100-epoch + min-count=1000 configuration achieving the best macro F1. We use this best-performing configuration as our baseline throughout the paper. Crucially, the three South Slavic dialects (HR, BS, ME) remain at 0.000 F1 across every configuration in the sweep, indicating that fails on these closely related dialects regardless of training budget.

12 Inference Latency and Training Time↩︎

[tab:latency_glotlid,tab:latency_wili] reports end-to-end inference latency and throughput for , , and measured on the same hardware. On GlotLID-C (\(|\Lambda|=1940\)), is approximately \(1.65\times\) slower than per sample; on WiLI (\(|\Lambda|=235\)), the gap closes to \(1.4\times\) and is over \(2.7\times\) faster than . 14 compares wall-clock training time for the GlotLID-C training set.

8pt

8pt

8pt

13 Performance Breakdown↩︎

In order to understand its strengths and weaknesses, we analyze ’s performance along three axes: writing script, training-data quantity, and segmentation length under predicted vs. true language.

13.1 Script↩︎

15 partitions GlotLID-C test performance by Unicode script. UniLID performs almost identically to on Latin-script languages (1,700 of the 1,940 labels; F1 0.940 vs. 0.946), but underperforms on every non-Latin script. The gaps are largest on Greek (\(\Delta\) F1 = -0.248), Hebrew (-0.227), Devanagari (-0.121), and Bengali (-0.100). We attribute this to the base vocabulary (\(\mathcal{V}\)) of the tokenizer: \(\mathcal{V}\) is itself learned by on the GlotLID-C training corpus, in which Latin-script data dominates. A Latin-biased \(\mathcal{V}\) leaves languages in other scripts with fewer informative subword units, which in turn limits how much signal per-language unigram estimation can recover. Larger vocabularies help modestly (10) but do not close the gap. The most direct solution would be to train \(\mathcal{V}\) on a script-balanced or script-upsampled corpus so that non-Latin substrings receive proportional coverage in \(\mathcal{V}\). We leave this exploration to future work.

13.2 Resource Tier↩︎

16 stratifies results by per-language training-sample count. is slightly exceeds for languages with 500\(+\) training samples. The differences arise in the very low resource settings: lags slightly on languages with fewer than 500 training samples. Many of the low-resource languages in GlotLID-C are from non-Latin scripts. Because our experiments on low-resource languages of the same script (6) provide evidence that performs better than in this setting, we hypothesize that the aforementioned script-bias issue (a property of the vocabulary) is a confounder for these sets of experiments.

13.3 Length Bias↩︎

We lastly look potential length biases. The unigram likelihood lacks an explicit length prior, so a language whose \(\widehat{\boldsymbol{\phi}}_{\ell}\) puts more mass on longer tokens accumulates fewer multiplicative factors than one favoring shorter tokens. We checked whether this manifests as a systematic bias in errors and find that on misclassifications, the Viterbi segmentation under the predicted language is on average 0.17 tokens shorter than under the true language, with the gap growing for longer inputs. Length-normalizing the per-language log-likelihood by segmentation length led to worse performace in preliminary experiments; we report results in 17 18.

References↩︎

---

1. We release code for UniLID at https://github.com/Ahmetcanyvz/UNILID.↩︎

2. The term “characters” denotes the atomic symbols of the input alphabet, encompassing both Unicode code points and bytes.↩︎

3. We assume a uniform prior over languages in \(\Lambda\) and so drop the explicit \(p(\ell)\) term to reduce clutter.↩︎

4. We would need either an eos token (absorbing state) or a length prior in order to make this a valid probability distribution over \(\mathcal{V}^*\). This detail is ignored in the standard implementation, and so we disregard it here as well for faithfulness to the original algorithm.↩︎

5. \(\boldsymbol{\phi}^{(0)}\) can be initialized as e.g., the uniform distribution. See [27] for a discussion of this design choice.↩︎

6. In the original algorithm, the most-probable segmentation is chosen rather than the one that has the highest marginal probability. We compared both strategies at inference and found no statistically significant accuracy difference, but marginalizing took roughly twice as much time (see 10.1).↩︎

7. This does not need to be a tokenizer. Any tokenizer that uses a fixed vocabulary can be used.↩︎

8. .↩︎

9. While the authors’ base setup uses fewer epochs, models trained for fewer epochs seriously underperformed on benchmarks. We tuned extensively. 100 epochs gave the best validation performance in every setting; see 11 for results on our hyperparameter sweep on DSL-ML 2024.↩︎

10. .↩︎

11. Their contamination check finds that for 57 languages, \(\leq 10\%\) of UDHR sentences appear in training; see §3 of [9].↩︎

12. We observed empirically that unigram distributions always converged within 20 rounds (i.e., the total variation distance between subsequent rounds fell below \(10^{-6}\)).↩︎

13. .↩︎

14. We tuned extensively for this setting but could not improve performance; the best configuration is reported here, with a subset of our sweep of hyperparameters in 11.↩︎

15. This is the number of unique labels in the officially maintained corpus on HuggingFace, which is what we employed: datasets/cis-lmu/glotlid-corpus.↩︎

16. https://www.un.org/en/about-us/universal-declaration-of-human-rights↩︎