1 Introduction↩︎

Language Identification (LID), the task of determining the language of a given text, is a foundational step in curating multilingual corpora from web crawls [1], [2]. LID errors propagate to downstream stages such as tokenization [3], filtering [4], [5], and data scheduling for multilingual pretraining [6]–[8]. Crucially, LID systems determine not only how reliably each language is predicted but also the scope of identifiable languages. If a language is out of scope, its text is either dropped or misattributed to an in-scope language, distorting corpus composition and downstream evaluation [9], [10].

Although LID is often treated as largely solved [11], [12], recent evaluations show that performance remains uneven across languages and domains [13], [14]. LID for African languages remains especially challenging. Existing benchmarks cover only a small fraction of the continent’s languages, and even within this limited coverage, performance often lags behind that of higher-resource languages. At the same time, major web-crawled corpora for African languages suffer from systematic quality issues [15], including noisy or unusable text, misattributed documents [16], and heavy concentration in religious or translated material that does not adequately reflect everyday language use [17]. These artifacts degrade downstream performance and inflate apparent coverage—a form of representation washing [18] that reinforces disparities in language technology [19]. Recent LID systems [2], [17], including African-focused models [10], have made important progress. However, two central gaps remain: scope, i.e., the set of African languages that can be reliably identified, and granularity, i.e., the ability to distinguish closely related languages and varieties. We address these gaps with AfroScope, a unified framework for African LID. As shown in Figure [fig:main95fig], AfroScope consists of three main contributions:

(i) Coverage-oriented data and models. We introduce AfroScope-Data, a large-scale multilingual dataset curated to expand both language and domain coverage for African LID. AfroScope-Data spans \(640\) languages across multiple orthographies and domains (§3), enabling a more comprehensive assessment of African LID performance, including fine-grained analysis across domains. Using AfroScope-Data, we train AfroScope-Models, a family of LID models that outperform prior African LID baselines across our evaluation setting (§4).

(ii) Hierarchical disambiguation of closely related languages. We show that genetically related and geographically proximate languages are a major source of false positives, especially when labels correspond to closely related varieties or macrolanguage members. To address this, we introduce AfroScope-Mirror, a lightweight contrastive embedding model specialized for frequently confused language groups, and use it in a hierarchical inference procedure that performs targeted disambiguation only when fine-grained separation is needed (§6.1.1). This design improves discrimination among closely related languages while preserving broad-coverage LID.

(iii) Transfer and robustness analysis. Leveraging AfroScope-Data, we analyze LID performance by coverage and domain (§5), and study multilingual transfer effects—including positive transfer and negative interference—as a function of language family structure and script overlap (§6.2). These analyses provide practical guidance for building, evaluating, and curating robust African LID systems.

2 Related Works↩︎

2.0.0.1 Linguistic diversity in Africa.

Africa is among the most linguistically diverse regions, spanning many language families and typological profiles [20], [21]. For NLP systems, this diversity manifests in phenomena that directly stress corpus curation and LID, including rich morphology, orthographic variation, and pervasive multilingual practices such as code-switching [13], [22]. In addition, language varieties with fluid boundaries complicate labeling and evaluation [23], [24], particularly for diglossic African languages like Arabic. Recent work has responded with new resources, benchmarks, and African-focused models, which we discuss—along with related challenges—in Appendix 8.1.

2.0.0.2 Data authenticity and corpus quality.

Large multilingual corpora frequently contain misattributed text, ambiguous language codes, and other quality issues that disproportionately affect low-resource settings [15]. Prior studies of widely used multilingual resources and pipelines document systematic noise and labeling errors [25]–[27], and emphasize the role of LID quality and preprocessing in mitigating such artifacts [28]. Improving authenticity is therefore central to building reliable and culturally representative language technologies [29]–[31], motivating dataset construction that explicitly controls for coverage, domain diversity, and contamination.

2.0.0.3 Progress in African language identification.

Recent evaluations show that LID coverage for African languages remains narrow and performance lags substantially [14]. Despite progress from FastText-based systems [17], [18], [32] to African-focused transformer-based models [10], [33], [34], as well as methodological advances through contrastive learning [2] and hierarchical approaches [28], targeted efforts for African LID remain necessary to address problems specific to African languages, such as domain variation and closely related varieties [35].

3 Addressing Coverage and Domain Skew: AfroScope-Data↩︎

Building robust LID systems for African languages requires data that is broad across two axes: language coverage and domain coverage. Existing resources cover only a small subset of African languages, leaving many underrepresented or entirely absent and making it difficult to measure true progress. Even among covered languages, sentence counts are highly concentrated in a small number of relatively high-resource languages (Figure 1 (a)). Domain coverage is similarly skewed: religious texts, predominantly Bible translations, account for \(73.10\)% and \(94.70\)% of sentences in GlotLID (Figure 1 (a)) and AfroLID (Figure 1 (b)), respectively. This narrow distribution limits generalization to heterogeneous digital text, including web text, administrative documents, and transcribed speech.

To address these challenges, we curate AfroScope-Data (Figure 1 (c)) using a strategy guided by two objectives: (i) maximizing language coverage to reduce out-of-model cousin errors [15], [40], where text from an unsupported language is misattributed to the closest supported relative; and (ii) increasing domain diversity to mitigate the narrow domain concentration in available African language data.

AfroScope-Data (Table 1) spans \(640\) African languages across eight language families, seven scripts, and eight domains. To the best of our knowledge, AfroScope-Data provides the broadest publicly described coverage for African LID in terms of the joint number of African language labels and domain categories.

3.1 Data Curation and Sampling↩︎

We compile AfroScope-Data from publicly described sources, prioritizing datasets that provide sufficient provenance metadata for assigning coarse domain labels. To maximize coverage, we use GlotLID-C [17], AfroLID [10], and SimbaText [36] as training sources, given their breadth and metadata availability (Table 1).

To mitigate language imbalance, we sample a fixed-size training corpus using temperature smoothing, following prior multilingual practice [9], [18], [41]. For a language \(l\) with corpus fraction \(p_l\), we sample proportional to \(p_l^{\alpha}\), where \(\alpha=0.3\). We apply the same smoothing hierarchically at the domain level: within each language’s budget, a domain \(d\) with intra-language fraction \(p_d\) is sampled proportional to \(p_d^{\alpha}\). This reduces the dominance of high-frequency domains, such as religious texts, and increases the representation of lower-frequency ones.

Through this sampling strategy, AfroScope-Data mitigates imbalance along both axes (Figure 1 (c)). The top-10 languages account for only \(9\)% of sentences, down from \(29\)% in GlotLID-C. Bible-derived content is reduced to \(53.5\)%, compared to \(73.1\)% and \(94.7\)% in GlotLID-C and AfroLID, respectively. We provide more details regarding data curation and sampling in Appendix 9.

3.2 Family, Script, and Domain Coverage↩︎

3.2.0.1 Language family.

AfroScope-Data spans eight high-level groupings: Afro-Asiatic, Austronesian, Creole, Indo-European, Khoe-Kwadi, Mixed language, Niger-Congo, and Nilo-Saharan. This diversity reduces reliance on cues from dominant families such as Niger-Congo and supports evaluation of cross-family generalization. We use this hierarchy in our transfer analyses (§6.2).

3.2.0.2 Script.

We include seven writing systems: Latin (Latn), Arabic (Arab), Ge’ez (Ethi), N’Ko (Nkoo), Tifinagh (Tfng), Coptic (Copt), and Vai (Vaii). We explicitly label scripts for each language (language_script), as individual languages may employ multiple writing systems (e.g., gof, ttq), enabling script-aware evaluation and analysis of orthographic variation.

3.2.0.3 Domain.

To analyze domain effects, we first adopt the source-domain categorization from GlotLID [17], which groups sources into coarse categories such as bible, news, web, government, and social text. We extend this mapping to sources absent from the original GlotLID taxonomy, using the origin metadata each dataset provides. This yields a compact set of domains: Speech, Human Rights, Crowdsourcing, News, Bible, Web, Government, and Other. We assign each sentence a domain label through this source-level mapping and use the labels for controlled evaluation by domain; we discuss domain effects in detail in §5.

4 Experimental Setup↩︎

4.0.0.1 Evaluation data.

We evaluate on six external Evaluation sets spanning diverse domains and language coverage: FLORES+ [37], UDHR [17], MAFAND [38], SmolSent [39], MCS-350 [28], and CommonLID [14]. We also evaluate on a held-out AfroScope-Data test split. Together, these benchmarks allow us to assess broad-coverage African LID and robustness across domains.

4.0.0.2 Contamination analysis.

Because several African LID resources are derived from overlapping public sources, we explicitly measure residual overlap between Train and Evaluation data. We use a strict 4-gram contamination criterion: a test sentence is marked as contaminated if all of its 4-grams appear within a single training sentence. Table 3 reports contamination rates across three training corpora and the six external evaluation sets. AfroScope-Data exhibits no detected contamination on any of the Evaluation sets under this criterion, whereas the AfroSimba and GlotLID-C corpora show non-trivial overlap, particularly on MCS-350 (\(14.15\)% with GlotLID-C) and MAFAND (\(8.13\)% with GlotLID-C). We return to the relationship between contamination and downstream model performance in §5.

4.1 Evaluation Protocol and Metrics↩︎

Following [17], we evaluate each model on the full benchmark rather than filtering examples to the model’s supported language set. This setting reflects more realistic LID deployment conditions, where systems encounter text in languages outside their training label space and must avoid incorrectly assigning such text to an in-scope language. It also makes coverage differences visible: models with narrower label spaces are penalized when they force unsupported languages into related or superficially similar supported labels.

We report macro-F₁, which averages performance across languages, together with False Positive Rate (FPR), defined as: \(\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}\), where FP is the number of false positives and TN is the number of true negatives. Macro-F₁ measures average per-language identification quality, while FPR captures how often a model falsely attributes text to a language it does not belong to. This distinction is important for African LID, where unsupported or closely related languages can otherwise be misclassified as higher-resource relatives.

4.2 Baselines↩︎

We compare against a diverse set of LID systems, ranging from FastText-based classifiers to transformer-based African language models.

4.2.0.1 FastText models.

We evaluate three existing FastText-based LID systems: GlotLID-M [17], OpenLID [18] and ConLID [2]. ConLID incorporates supervised contrastive learning to improve robustness on out-of-domain data.

4.2.0.2 Neural models.

We evaluate three transformer-based models developed for African languages: AfroLID [10], Serengeti [33], and Cheetah [34]. To isolate the effect of training data, we fine-tune these models on each Train source where applicable, and refer to the resulting models collectively as AfroScope-Models. Because SimbaText is substantially smaller than the other training sources, we merge it with AfroLID to form a combined training source, AfroSimba. Fine-tuning hyperparameters are provided in Appendix 10.

5 Results↩︎

Table 2 reports macro-F₁ and FPR across all external Evaluation sets for the full set of baselines and AfroScope-Modelsvariants. AfroScope-Modelsachieve the strongest overall performance, leading on five of seven benchmarks, with the largest gains appearing on benchmarks with broader language coverage. The strongest improvements occur on the two broadest evaluation sets, AfroScope-DataTest (\(640\) languages) and UDHR (\(117\) languages), where the best Serengeti_{AF\(+\) Mirror} variant obtains \(97.87\) and \(77.91\) macro-F_\(1\), respectively, with FPRs of \(0.0000\) and \(0.0006\). Datasets with narrower language coverage, such as FLORES+, CommonLID, and SmolSent, show the same general pattern but with smaller margins. GlotLID-M and ConLID, both trained on the GlotLID-C corpus, achieve the strongest results only on MAFAND (\(90.87\) and \(90.65\) macro-F₁) and MCS-350 (\(77.56\) macro-F₁). These are also the evaluation datasets for which the GlotLID-C corpus exhibits the highest contamination, suggesting that evaluation overlap may contribute to their advantage on these benchmarks rather than reflecting broader generalization.

5.0.0.1 Performance and coverage.

Coverage is a major driver of performance differences across evaluation sets. Languages absent from a model’s training label space are vulnerable to forced misattributions, where the model assigns them to a related or superficially similar supported language, thereby inflating FPR. Across the seven evaluation sets, the GlotLID-C corpus label space lacks coverage for \(171\) evaluation-language entries while AfroScope-Datacovers all but \(31\). On datasets where both corpora cover nearly all evaluation languages, such as FLORES+, MAFAND, and SmolSent, model performance clusters within a relatively narrow macro-F₁ range. In contrast, on broader evaluation sets with larger coverage gaps, such as UDHR and MCS-350, AfroScope-Modelsvariants achieve an average macro-F₁ of \(75.25\) compared to \(67.25\) for non-AfroScope-Databaselines, while average FPR decreases from \(0.00101\) to \(0.00067\).

5.0.0.2 Performance by domain.

Figure 2 shows that model performance is strongly shaped by the amount and distribution of domain-specific training data. High-resource domains such as Bible, News, Crowdsourcing, and Government generally achieve strong per-language macro-F₁ scores, with most languages clustered near the top of the distribution. In contrast, lower-resource or more heterogeneous domains exhibit weaker and less stable behavior. Human Rights contains only a few training examples across \(60\) languages, and its per-language results include a severe lower tail, with several languages receiving near-zero macro-F₁. This pattern helps explain the relatively low performance on UDHR, a human-rights benchmark. Similarly, the Web domain has moderate training coverage but noticeably weaker lower-tail performance, suggesting that domain diversity and text heterogeneity also affect robustness beyond raw example count alone. Overall, these results show that LID performance depends on both language coverage and domain coverage: domains that are well represented in training generalize more reliably, while sparse or unevenly distributed domains produce brittle performance for some languages.

5.0.0.3 Comparison with frontier LLMs.

Given that LLMs now set the state of the art across many NLP tasks, we evaluate three frontier LLMs on AfroScope-Datatest: Gemini-Pro, Claude-4.7-OPUS, and GPT-5.4-mini. Prompt templates, candidate-label formatting, and other evaluation details are provided in Appendix 11. As shown in Table 4, all three LLMs lag substantially behind dedicated LID systems: the strongest, Claude-4.7-OPUS, reaches only \(47.40\) F₁ (FPR \(0.0041\)), compared to \(69.08\) for GlotLID-M and \(97.71\) for AfroScope-Models. This gap suggests that despite their broad multilingual capabilities, general-purpose LLMs are not a substitute for purpose-built LID models on African languages.

6 Analysis↩︎

In this section, we analyze the patterns behind the aggregate results in Table 2. We focus primarily on AfroScope-Data test because it has the broadest language coverage, while using external evaluation sets to test whether the same patterns recur across domains and benchmarks. Our analysis addresses two questions: (i) how does confusability among closely related languages affect performance? and (ii) which languages synergize or interfere most with one another, and how do language family relationships and script compatibility shape these effects?

6.1 Confusability in related languages↩︎

We find that label confusability is a primary failure mode: the repeated false-positive assignment between a focus language and a small set of related or geographically proximate labels. Table 5 previews representative confusion patterns identified across the evaluation sets. Such cases are especially common across ISO macrolanguage groupings and closely related languages or varieties that share substantial lexical and orthographic overlap. We find that these confusions recur in at least four of six evaluation sets, indicating the errors are stable across datasets rather than isolated benchmark artifacts. Among macrolanguage varieties, Arabic (ara: aeb, ary, arz) and Fulah (ful: fub, fuf, fuv) are representative. High-confusion pairs are often genetically related and geographically proximate, such as Lamnso (lns) and Tikar (tik) in Cameroon, and Anii (blo) and Lukpa (dop) in Togo. These groupings account for a disproportionate share of false positives, suggesting that models struggle to discriminate fine-grained varieties. Together, these patterns indicate that many errors reflect genuine linguistic similarity and label granularity rather than random noise. We provide further examples in Appendix 12.1.

6.1.1 AfroScope-Mirror Disambiguation↩︎

To address these systematic confusions, we introduce AfroScope-Mirror, a lightweight contrastive embedding model used inside a hierarchical inference procedure. The base LID model first produces a broad-coverage prediction. When the prediction falls within a predefined confusable group and the model’s confidence is below a threshold \(\tau\), AfroScope-Mirrorperforms a group-specific disambiguation step using specialized embeddings. We build AfroScope-Mirroron top of AfroLID_AF and Serengeti_AF, our strongest base models on the external evaluation sets, and train it with Mirror-BERT [42], an unsupervised contrastive learning objective that pulls semantically similar representations together while pushing unrelated ones apart. Detailed training procedures and hyperparameters are in Appendix 11.

Figure 3 compares the embedding spaces produced by Serengeti_AF and Serengeti_{AF \(+\) Mirror}, visually illustrating clearer separation among confusable labels and tighter within-label clustering after specialization. We evaluate this strategy on the \(30\) confusable languages in Table 12, including focus languages and their top confusion partners, across confidence thresholds. On the full set, the average macro-F₁ gain is +\(1.57\) points. As shown in Table 2, adding AfroScope-Mirror yields consistent gains across the external benchmarks: Serengeti_{AF \(+\) Mirror} improves over its base model on all seven evaluation sets (e.g., +\(2.40\) on FLORES+, +\(3.28\) on UDHR, +\(2.47\) on Mafand), and AfroLID_{AF \(+\) Mirror} improves on five of seven datasets. At the language level (Table 6), we observe improvements for closely related varieties within macro-language groupings, such as Arabic (ara) varieties aeb, arq, and ary, which improve by +\(0.63\), +\(0.79\), and +\(0.31\), respectively, as well as Swahili (swa) variety swc (+\(0.55\)). We also observe improvements for confusable regional pairs, including Kinyarwanda (kin) and run (+\(0.71\)). However, a small number of languages decline, including niq (\(-0.10\)) and kbp (\(-0.14\)), suggesting that hierarchical routing can introduce unnecessary complexity when the base model is already reliable for a given label.

6.2 Transfer Synergy and Interference↩︎

Given limited and skewed resources, effective African LID depends on cross-lingual transfer. We investigate which languages synergize or interfere most, and how language family and script compatibility shape these effects. To quantify this, we adapt the Bilingual Transfer Score (BTS) [43] to language identification. We compute BTS using macro-F₁, and cross-entropy loss across \(30\) representative African languages spanning five family clusters.

Figure 7 shows that cross-lingual transfer in African LID is governed by shared script and fine-grained language relatedness rather than coarse family membership. Co-training yields the largest F₁ gains for low-resource targets with weak monolingual baselines (e.g., fuv, dyu, bam; F₁ BTS up to +\(0.22\)), consistent with baseline rescue rather than family-specific transfer. Critically, the closest relationships are not uniformly beneficial: mutually intelligible varieties such as the Arabic cluster (arz, ary, aeb) and Mande pair bam–dyu improve F₁ but degrade probability calibration, with loss-based BTS as low as -\(0.24\). A regression across all pairs confirms that script and this confusable-cousin relationship—not language family—are the dominant predictors of transfer; family membership alone is not significant. This accuracy–calibration trade-off motivates hierarchical disambiguation. We report full per-family analysis in Appendix 12.2.

7 Conclusion↩︎

We introduce AfroScope, a unified framework for African LID that combines broad-coverage data, strong LID models, targeted disambiguation, and linguistic analysis. We present AfroScope-Data, a large-scale dataset spanning \(640\) language labels, which we use to train AfroScope-Models, a family of African LID models that outperform prior African-focused baselines across internal and external evaluations.

To address persistent confusions among closely related languages and varieties, we propose a hierarchical inference approach based on AfroScope-Mirror, a specialized embedding model for frequently confused language groups. On the full set of identified confusion groups, this approach improves macro-F₁ by \(+1.57\) points on average. Finally, our transfer analysis shows that African LID transfer is shaped by script, fine-grained family relatedness and confusability than by coarse family membership. These relationships can improve low-resource performance while degrading calibration, motivating targeted disambiguation with AfroScope-Mirror. We hope AfroScope, AfroScope-Data, and AfroScope-Modelssupport reliable African NLP and future work on fine-grained varieties, domain shifts, and mixed-language text.

Limitations↩︎

We note several limitations of the current study.

1. Mixed-language and code-switched text. Our formulation treats each instance as belonging to a single language label. This does not fully capture important phenomena in Africa’s linguistic landscape, including code-switching, mixed-language documents, translanguaging practices, and contact varieties such as pidgins and creoles. Extending AfroScopeto multi-label, document-level, or span-level language identification is an important direction for future work. Relatedly, the granularity at which languages are annotated is uneven across our data sources, and in some cases closely related varieties or distinct languages may be conflated under a single label. A more careful audit of these labeling decisions, ensuring that language distinctions are drawn consistently and appropriately, is needed but lies beyond the scope of this work.

2. Language metadata and classification choices. AfroScope-Datarelies on external catalogs, primarily Ethnologue, to assign language identifiers, genealogical groupings, and script metadata. Alternative resources, such as Glottolog, may differ in classification, naming, macrolanguage treatment, and family hierarchy. These choices may affect analyses that depend on genealogical proximity or script compatibility. Future work should evaluate sensitivity to alternative metadata sources and provide mappings across catalog standards.

3. Confidence-based routing and calibration. Our hierarchical disambiguation method relies on model confidence to decide when to invoke the group-specific refinement step. While this improves performance on many confusable labels, gains are not uniform across languages or thresholds, and some cases exhibit degradation. Improving probability calibration and learning a routing policy, rather than relying on fixed confidence thresholds, may further increase robustness.

4. Domain and source imbalance. Although AfroScope-Dataimproves domain diversity relative to prior African LID resources, some domains remain substantially smaller or more heterogeneous than others. As a result, performance may still be brittle for domains with limited training data or high internal variation, such as web text, human-rights documents, or informal user-generated content. Future work should expand naturally occurring data in underrepresented domains and evaluate robustness under domain shift.

The following appendices provide comprehensive supplementary material supporting the main findings of this work. We include an expanded literature review, detailed descriptions of the datasets and models, a more in-depth discussion of the experimental setup, and additional analyses and discussions.

• §8: More in depth literature review

• §9: Data Curation & Preprocessing

• §10: Baseline Models

• §12: Discussions

8 Literature Review↩︎

8.1 Linguistic Diversity of Africa.↩︎

Many African languages are agglutinative (e.g., Swahili, Zulu), complicating tokenization and parsing [22], [23], while everyday communication often involves code-switching with English and other languages [44].

Datasets like AfroCS-xs demonstrate that small, high-quality resources can boost LLM performance on code-switched tasks [45]. Models such as Afrolid, SERENGETI and Cheetah show the feasibility of massively multilingual modeling for African languages [10], [33], [34]. Benchmarks including Sahara [46], SimbaBench [36], IrokoBench [47], and AfroBench [29].

8.2 Data Authenticity.↩︎

Studies of multilingual corpora such as ParaCrawl [25], WikiMatrix [26], and mC4 [27] reveal widespread errors, mislabeled data, and ambiguous language codes [15]. For instance, [16] showed that many FastText embeddings were heavily mislabeled with words from other languages. Poor data authenticity directly undermines cultural representation, as inaccurate or mislabeled data fails to capture the nuanced linguistic and cultural contexts inherent in low-resource languages [30]. Addressing these quality issues is essential for building reliable language technologies  [29], [31].

9 Data Curation and Sampling↩︎

9.1 Data Curation↩︎

We deduplicate sentences across all sources to ensure that each example appears at most once in AfroScope-Data. We then partition the data into train, dev, and test splits, performing deduplication prior to splitting so that no sentence is shared across splits. We ensure that each sentence is written in the correct script, based on the writing system databases of Ethnologue [48]. When temperature sampling we set the maximum sentences to \(10\)M sentences. The resulting splits contain \(5{,}952{,}573\) sentences for train, \(463{,}875\) for dev, and \(232{,}563\) for test. The full list of languages supported by AfroScope, together with the number of sentences per language, is provided in Tables 15–18.

Below, we describe the sources used to construct AfroScope-Data.

9.1.0.1 Train Corpus

9.1.0.2 GlotLID-C.

We extract 523 African languages from GlotLID-C, a collection spanning 2,099 languages globally [17].

9.1.0.3 AfroLID.

A manually curated multi-domain web dataset covering 513 African languages [10].

9.1.0.4 SimbaText.

Speech-derived text data spanning 103 African languages, originally collected for speech and language identification [36].

9.1.0.5 Evaluation Datasets

9.1.0.6 FLORES\(+\).

Professionally translated sentences across \(200\) languages, providing a clean and standardized benchmark, particularly for low-resource languages [37].

9.1.0.7 MAFAND.

Manually audited data from news, Wikipedia, and religious texts across 55 African languages [38].

9.1.0.8 SMOL.

Multilingual benchmark dataset consisting of professionally translated sentence- and document-level data for 115 LRLs [39].

9.1.0.9 MCS-350.

Parallel children’s stories across 151 African languages, drawn from a multilingual collection of 50k texts in over 350 languages [28].

9.1.0.10 UDHR.

The Universal Declaration of Human Rights, translated into a wide range of languages [17].

9.1.0.11 CommonLID.

Professionally translated parallel data for 115 under-represented languages [14].

9.2 Domain classification.↩︎

We assign domains by matching keywords found in the metadata associated with each sentence, following the categorization scheme from [17]. Table 8 lists the specific keyword mappings.

3pt

10 Training Hyperparameters for Baseline Models↩︎

To isolate the effect of training data, we train or fine-tune comparable model architectures on each Train source where applicable. This allows us to evaluate whether the gains come from the model architecture, the training corpus, or their interaction. Because SimbaText is substantially smaller than the other training sources, we merge it with AfroLID to form a combined training source, AfroSimba. AfroLID and Serengeti are XLM-RoBERTa variants, and Cheetah is a T5-based model.

10.1 Neural Models↩︎

We use the hyperparameters in Table 9 for AfroLID [10] and Serengeti [33] (both XLM-R variants), and Table 10 for Cheetah [34].

11 Evaluating Open-Srouce LLMs↩︎

11.1 Prompts for Open-Source Models↩︎

Figure 4 shows the prompt template used to elicit language predictions from the LLMs. We provide the candidate ISO 639-3 labels in the prompt and instruct the model to return only the language code. To reduce inference cost we downsample each language examples to 100 examples per language.

12 Discussions↩︎

12.1 Confusability in related languages↩︎

To investigate languages with high FPR, we isolate languages with scores below F₁ 90 across all evaluation datasets—and identify the top three most frequent misclassifications for each to form confusion groups. This analysis yields \(13\) distinct groups comprising \(30\) languages in total. We find that these confusions primarily stem from either macrolanguage structures (e.g., ful vs. fub) or geographic proximity (e.g., bsq vs. bas). Table 12 details the composition of all confusion groups, and we report the corresponding performance improvements for each individual language.

12.1.0.1 Mirror-BERT training procedure

We follow the procedures from the official github repository for Mirror-BERT². We use Serengeti_AF for this experiment as it being our best performing model. Table 11 shows the hyperparamerters we use to train AfroScope-Mirror. We see in Figure 5 that the label spaces of Serengeti_{AF + Mirror}  shows much better separation in the embedding spaces than Serengeti_AF.

12.2 Transfer Synergy and Interference↩︎

We adapt the Bilingual Transfer Score (BTS) [43] to language identification. For a source \(s\) and target \(t\), we compare a monolingual model trained on target data \(\mathcal{D}_t\) against a bilingual model trained on \(\mathcal{D}_t \cup \mathcal{D}s\), evaluated on a held-out target test set. For cross-entropy loss, where lower is better, we define: \[\mathrm{BTS}^{\mathcal{L}}_{s \rightarrow t} = \frac{ \mathcal{L}_t(\mathcal{D}_t) - \mathcal{L}_t(\mathcal{D}_t \cup \mathcal{D}_s) }{ \mathcal{L}_t(\mathcal{D}_t) + \epsilon }. \label{eq:bts95loss}\tag{1}\] so that positive values indicate improvement. For macro-F₁, where higher is better, we define: \[\mathrm{BTS}^{F1}_{s \rightarrow t} = \frac{ F1_t(\mathcal{D}_t \cup \mathcal{D}_s) - F1_t(\mathcal{D}_t) }{ F1_t(\mathcal{D}_t) + \epsilon }. \label{eq:bts95f1}\tag{2}\]

12.2.1 Language selection↩︎

We select \(30\) languages spanning five clusters. To populate a range of genealogical distances, we group eligible languages (those with at least \(10{,}000\) sentences) by their shared family hierarchy and retain clusters that contain documented confusable pairs, drawn from our false-positive analysis (Appendix 12.1). This yields clusters of mutually intelligible varieties—the Arabic dialects (arz, ary, aeb), Fula varieties (fub, fuv, fuh, fuq, fue), and the Mande pair (bam, dyu)—alongside more distant same-family languages and unrelated outgroups (plt, afr, sag). Figure 6 shows the fine-grained relatedness of the selected languages across the family hierarchy.

12.2.2 Experimental setup↩︎

Each language contributes \(1{,}000\) target sentences and a held-out test set; bilingual models receive \(200\) background sentences from every other language so all models share one label space. We fine-tune Serengeti_AF for the \(30\) monolingual baselines (median over three seeds) and the \(435\) language pairs (one seed), yielding a \(30\) X \(30\) directional transfer matrix per metric.

12.2.2.1 F₁ gains track headroom.

The largest F₁-BTS values occur for the weakest baselines—fuv (\(0.62\)), fuq (\(0.53\)), dyu (\(0.69\)), bam (\(0.73\))—which improve from nearly any source (bam\(\rightarrow\)dyu \(=+0.22\), fub\(\rightarrow\)fuv \(=+0.16\) ; Table 13). F₁-BTS correlates with target headroom (\(r=0.60\)), indicating low-resource rescue rather than family-specific transfer; strong baselines (kin, xho, tso, tsn, all \(>0.98\)) show near-zero BTS.

12.2.2.2 Cousins degrade calibration.

The clearest signal is in loss-BTS. The Arabic varieties form a block of negative loss-BTS (aeb\(\rightarrow\)ary \(=-0.24\), aeb\(\rightarrow\)arz \(=-0.18\), ary\(\rightarrow\)aeb \(=-0.15\)), and the Mande pair replicates this (dyu\(\rightarrow\)bam \(=-0.17\)). All show positive F₁-BTS: co-training mutually intelligible varieties picks the right label but spreads probability mass across the cousin, raising cross-entropy. This trade-off is visible in Figure 7 (a), where cousin pairs sit in the mutual-gain region under F₁ but the mutual-loss region under cross-entropy. Bambara (bam) and Jula (dyu) are adjacent Manding varieties spoken across Mali, Côte d’Ivoire, and Burkina Faso [48], so this interference is expected.

12.2.2.3 Script gates transfer.

Substantial transfer is confined to same-script pairs: \(78\%\) of F₁ changes with \(|\mathrm{BTS}|\geq0.05\) and all calibration changes with \(|\mathrm{BTS}|\geq0.10\) occur within a script (Table 14). Controlling for headroom and cousin status, shared script independently predicts larger F₁-transfer magnitude (\(\beta=+0.005\)). The exception is low-resource rescue, where weak Fula targets improve from any source. Distinctive scripts with strong baselines, like the Ethiopic pair amh–tir (\(>0.99\)), show negligible transfer (mean \(|\mathrm{BTS}^{\mathcal{L}}|=0.006\)), as orthographic distinctiveness leaves no confusion to resolve.

12.2.2.4 Distance gates the effect.

Different-family pairs show loss-BTS near zero (distance-\(5\) mean \(=+0.002\), \(n=472\)). Regressing loss-BTS on genealogical distance \(d\) and \(d^2\) yields a significant inverted-U (\(\beta_d=+0.015\), \(\beta_{d^2}=-0.002\)): distance-\(0\) cousins are uniquely negative (mean \(=-0.029\)), all greater distances near zero. F₁-BTS shows no distance pattern, masked by ceiling saturation.

3.0pt

3.0pt

3.0pt

3.0pt

3.0pt

References↩︎

---

1. UBC-NLP/AfroScope↩︎

2. https://github.com/cambridgeltl/mirror-bert↩︎