CHALIS: A Challenge Dataset for Language Identification
in Difficult Scenarios
June 04, 2026
We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.
Language identification (LangID), the task of determining the language of a given text, is a foundational component of many NLP pipelines [1]. Although some researchers have characterized it as a solved problem [2], growing demand for large-scale web-crawled multilingual corpora [3], [4] has exposed significant weaknesses in current systems.
[5] audited 205 languages across five datasets and found that over 8% of samples were mislabeled with the wrong language, and a further 10% were not linguistic in character at all. The most error-prone cases involve languages that are closely related to a higher-resource neighbor, and texts that have been transliterated into a different script.
These findings motivate our targeted evaluation benchmark. Existing datasets used to assess LangID models, such as FLORES-200 [6], originally designed for multilingual machine translation, do not specifically probe these hard cases.
We introduce CHALIS to fill this gap. The dataset covers both cousin language pairs and transliteration and orthographic variation, with a human-annotated subset for cousin language pairs where annotators validated whether sentences are grammatically correct in each language. We evaluate four state-of-the-art LangID systems on CHALIS, revealing systematic failure modes in both scenarios.
Language identification is a classic text classification task [1]. Modern systems such as FastText [7], OpenLID [8], GlotLID glotlid?, and GCLD3 achieve strong results on standard benchmarks. However, as [9] shows, performance degrades significantly in realistic web-crawled settings, where mislabeled data is common.
Defining a “language” is a notoriously difficult exercise, nicely illustrated by the Weinreich witticism: “a language is a dialect with an army and navy” [10]. Language communities that coexist geographically develop varieties that mutually influence each other, often remaining mutually intelligible [11]. Examples include Norwegian and Danish, or Czech and Slovak.
We refer to such pairs as language cousins. They pose a dual challenge for LangID: they are not equally represented in training data (one is typically less prevalent), and there exist sentences that are grammatically correct in both languages. For example, “Je to obrovský problém” (“It is a huge problem”) is valid in both Czech and Slovak. Both [5] and [9] note that mistaking a lower-resource language for its more prominent cousin is among the most frequent errors in web-crawled corpora.
Transliteration, i.e., converting text from one script to another, often approximating pronunciation, is another major source of LangID errors. Transliterated text does not match the script profile that LangID models expect, yet it retains the language’s lexical and syntactic patterns. For instance, Russian Cyrillic transliterated to Latin characters is meaningless to a model that associates Russian exclusively with the Cyrillic script. Moreover, different languages have different transliteration standards.
The most frequent orthography issues in writing are probably mistakes and typos. These naturally occur in most digital texts and presumably also in datasets used for LangID training. However, there are more systematic issues that might affect the LangID performance. These issues include removing diacritics and only using ASCII characters in Latin, historically driven by software and hardware limitations. Homoglyph attacks use the fact that some graphemes look similar across scripts (e.g., o and o in Latin and Cyrillic) and are often used to confuse automatic systems such as spam detectors.
We collect a challenge set that addresses the issues discussed in the previous section. The dataset consists of three parts: Sentences that are valid in two cousin languages (and multiclass classifiers need to decide for one of them only), sentences that are only valid in one of the cousin-language pairs, but consist of words that are common in both languages, and the third part focuses on the orthography phenomena, and unlike the first two, it was created fully automatically.
Basic statistics of the dataset are presented in Table 1. The dataset is available at Huggingface Hub1 under the CC-BY-NC 4.0 license, which is consistent with the underlying data. More technical details are in Appendix 8.
| Subset / Language | Lines | Avg.Words | |
|---|---|---|---|
| Czech & Slovak | 655 | 5.9 | |
| Spanish & Catalan | 642 | 5.8 | |
| Danish & Norwegian | 459 | 7.0 | |
| Portuguese & Galician | 1,143 | 7.9 | |
| Czech | 491 | 6.5 | |
| Danish | 361 | 8.7 | |
| Norwegian | 327 | 7.9 | |
| Spanish | 546 | 6.5 | |
| Cyrilics to English | 50k | 15.4 | |
| Cyrilics to Czech | 10k | 13.9 | |
| Arabic (Kazakh) | 10k | 12.0 | |
| Diacritics | 30k | 17.2 | |
| Homoglyps | 30k | 14.5 | |
| Antspeak | 10k | 11.8 | |
| Leetspeak | 10k | 14.3 |
We collected data for four cousin pairs, all Indo-European languages from Europe. Czech/Slovak (Slavic), Spanish/Catalan (Romance), Portuguese/Galician (Romance), Danish/Norwegian (Germanic). There is a different level of resource disparities between the languages. Czech and Slovak are both official languages of the countries where they are spoken, with roughly 10M and 5M speakers respectively. There is a similar situation with Danish and Norwegian, both of which are spoken by roughly 5M people. The situation is slightly different with the Romance languages. Due to the colonial history, Spanish is spoken by 500M speakers, whereas Catalan is spoken by only 10M people in Europe. Similarly, Portuguese is spoken by 250M people in Europe and South America, whereas Galician is only spoken by around 3M people in Spain.
For each language in a pair, we tokenized text using Moses tokenizer [12] and built a frequency dictionary from the WMT Monolingual News Crawl [13], supplemented by Wikipedia-sourced data from the Leipzig Corpora Collection [14] for lower-resource languages. We extracted the 10,000 most frequent words in each language and computed their intersection to obtain a shared vocabulary. Table 2 shows the resulting shared vocabulary sizes.
| Count | ||
|---|---|---|
| Czech | Slovak | 2,339 |
| Danish | Norwegian | 4,266 |
| Spanish | Catalan | 1,623 |
| Portuguese | Galician | 3,491 |
We then retrieved all sentences from the source corpora whose tokens all appear in the shared vocabulary, yielding candidate sentences that could belong to either language.
Candidates were scored using 4-gram KenLM language models [15] with Kneser–Ney smoothing [16], trained on the source corpora. Each sentence received a probability under each language’s model, normalized for length. We ranked sentences by the harmonic mean of the two normalized log-likelihoods, and selected the top 1,200 per pair as most plausibly correct in both languages.
Inspection of high-scoring sentences revealed that many consisted largely of proper names (e.g., sports results). We applied the GLiNER named-entity recognizer [17] to remove sentences in which more than 25% of tokens were classified as named entities (persons, locations, organizations, etc.). Near-duplicate sentences were also removed using an edit-distance threshold of 10% of sentence length.
The top 1,200 filtered sentences per pair were presented to human annotators, who were native speakers or advanced students of the target languages. Annotators labeled each sentence as valid or invalid in their language. Where two annotators were available (Czech, Slovak, Danish), only sentences with unanimous agreement were kept. More details on the annotation process are in Appendix 6.
The resulting items fall into two subsets:
Single: sentences valid in only one language but structurally similar to the other—a particularly hard case for LangID. We disregard language, for which only a few sentences remained after filtering.
Joint: sentences valid in both languages of the pair.
We evaluate the ‘Single’ portion of the dataset with the F1 score. For the ‘Joint’ part of the dataset, we report the distribution of how often the model chose the respective language in language pairs vs. other languages.
We transliterated Bulgarian, Macedonian, Russian, Serbian, and Ukrainian text (sourced from the Leipzig Wikipedia corpus) into Latin script using two target typographies: English (via CyrTranslit; [18]). We further transliterated Russian into Czech using Czech orthography (via conversion scripts from the Charles translator project; [19]).
Kazakh has historically been written in Arabic script (before Soviet-era Cyrillisation), and standards for Arabic-script Kazakh are well established. We transliterated Cyrillic Kazakh text into Arabic using scripts from the MC2 dataset [20].
In the Latin text, letters are randomly replaced by visually identical Cyrillic or Greek characters in Czech, Slovak, and English.
We also include several related phenomena: (1) diacritics removal for Czech, Slovak, and Vietnamese; (2) antspeak (s p a c e s b
e t w e e n l e t t e r s) and leetspeak (character substitution,
e.g.n00b) for English, using pyleetspeak [21].
We evaluate four widely used LangID systems: FastText [7], OpenLID [8], GlotLID glotlid?, and GCLD3. All four are multiclass classifiers that always output exactly one language label. More details on the models are in Table 4 in Appendix 7.
5pt
l c| rr rr @ |@ rrr| rrr| rrr| rrr & & &
(lr8pt)3-6(l-4ptr)7-18 (lr)7-9(lr)10-12(lr)13-15(lr)16-18 & & ces & spa & nob & dan & ces & slk & oth & spa & cat & oth & por & glg & oth & nob & dan & oth
FastText & & & & & & & & & & & & & & & & &
OpenLID & & & & & & & & & & & & & & & & &
GlotLID & & & & & & & & & & & & & & & & &
GCLD3 & & & & & & & & & & & & & & & & &
2pt
| Transliteration to | Orthography noise | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (lr8pt)2-7(l-4ptr)8-15 | eng | ces | arb | Diacritics | Homoglyphs | Internet slang | ||||||||
| 2-5(lr)6-6(lr)7-7 (lr)8-10(lr)11-13(lr)14-15 | bul | mkd | rus | srp | rus | kaz | ces | slk | vie | eng | ces | slk | Ant | Leet |
| FastText | .99\(\rangle\) .00 | .99\(\rangle\) .00 | .99\(\rangle\) .00 | .99\(\rangle\) .00 | .99\(\rangle\) .00 | .99\(\rangle\) .00 | 1\(\rangle\) .89 | 1\(\rangle\) .92 | 1\(\rangle\) .90 | .99\(\rangle\) .92 | 1\(\rangle\) .98 | 1\(\rangle\) .97 | .99\(\rangle\) .12 | .99\(\rangle\) .98 |
| OpenLID | .99\(\rangle\) .00 | .99\(\rangle\) .00 | .95\(\rangle\) .00 | .99\(\rangle\) .01 | .95\(\rangle\) .00 | .99\(\rangle\) .00 | .99\(\rangle\) .81 | .99\(\rangle\) .88 | .99\(\rangle\) .94 | .99\(\rangle\) .54 | .95\(\rangle\) .96 | .99\(\rangle\) .94 | .95\(\rangle\) .00 | .95\(\rangle\) .91 |
| GlotLID | 1\(\rangle\) .00 | 1\(\rangle\) .00 | 1\(\rangle\) .00 | 1\(\rangle\) .00 | 1\(\rangle\) .00 | 1\(\rangle\) .00 | 1\(\rangle\) .85 | 1\(\rangle\) .91 | 1\(\rangle\) .89 | .99\(\rangle\) .77 | 1\(\rangle\) .97 | 1\(\rangle\) .97 | .99\(\rangle\) .00 | .99\(\rangle\) .92 |
| GCLD3 | .97\(\rangle\) .68 | .97\(\rangle\) .00 | .98\(\rangle\) .76 | .97\(\rangle\) .00 | .98\(\rangle\) .48 | .99\(\rangle\) .00 | .97\(\rangle\) .62 | .97\(\rangle\) .74 | .98\(\rangle\) .13 | .93\(\rangle\) .35 | .97\(\rangle\) .80 | .97\(\rangle\) .72 | .93\(\rangle\) .00 | .93\(\rangle\) .72 |
’
Table [tab:langid] shows the results both for the Single-language subsect (evaluation using F1 score) and the language distributions for the Joint subset. In the ‘Single’ subset, the scores are relatively low, given that they are high-resource languages. They lag behind the models’ overall performance, indicating they are easy to confuse with the cousin languages.
The ‘Joint’ subset (sentences valid in both languages) reveals strong systematic bias. For Czech/Slovak, OpenLID and GlotLID assign over 90% of jointly valid sentences to Czech, effectively ignoring Slovak. Surprisingly, FastText prefers lower-resourced Slovak. A similar pattern holds for Portuguese/Galician and Spanish/Catalan. GCLD3 shows less extreme bias but achieves lower overall F1, suggesting it trades accuracy for a more balanced, if still imprecise, distribution.
Table 2 presents the results on the orography subset of the dataset. The F1 scores of all systems before the orthography modification are almost perfect, showing that the work with examples that are otherwise easy for LangID systems.
Transliteration is catastrophic for nearly all systems. FastText, OpenLID, and GlotLID score 0 across all Cyrillic-to-Latin conditions. They fail to recognize the language at all once the expected script is removed. GCLD3 shows partial robustness for Russian and Bulgarian transliteration into English (F1 scores of 0.76 and 0.68), though it, too, fails on Macedonian and Serbian, and, while not entirely, on Czech-orthography Russian. All systems score 0 on Arabic-script Kazakh.
In contrast, diacritics removal and homoglyph substitution have a more moderate, but still strong impact. Most systems retain F1 above 0.80 for diacritics-stripped Czech, Slovak, and Vietnamese, and above 0.9 for Czech and Slovak homoglyph attacks. OpenLID is a partial exception, dropping to 0.54 on English homoglyphs.
Antspeak proves most disruptive: FastText scores 0.12, while all other systems score 0, suggesting that inter-character spacing breaks n-gram statistics entirely. Leetspeak is handled more gracefully, with all systems achieving F1 above 0.70.
We presented CHALIS, a challenge benchmark targeting two systematic failure modes in language identification: cousin-language discrimination and non-standard orthography. Evaluating four widely used LangID systems, we found a consistent asymmetry: models reliably identify higher-resource languages while largely failing on lower-resource counterparts, with several models assigning nearly all ambiguous sentences to the dominant language in a pair. Transliteration proves catastrophic across the board, while antspeak similarly defeats current approaches, and diacritics removal and homoglyph substitution are handled more gracefully. These findings suggest that current LangID systems overfit to surface-level script and frequency cues. We hope CHALIS will motivate the development of stronger identification systems for lower-resource languages whose speakers deserve reliable NLP tooling.
CHALIS covers only four cousin language pairs, all Indo-European languages from Europe. While these pairs illustrate systematic LangID failure modes, the benchmark does not capture the full diversity of closely related language families, such as Arabic dialects, Turkic languages, or South and Southeast Asian language groups, where resource disparities and mutual intelligibility raise similar concerns.
The human annotation phase, which validates the ‘Single’ and ‘Joint’ subsets, was conducted by a small number of annotators. The orthography subset was constructed fully automatically and does not reflect the natural distribution of transliteration or slang usage found in real web-crawled data. Transliteration standards vary across communities and over time, and our choices (e.g., CyrTranslit for Cyrillic-to-English, Czech orthography for Russian) represent only a subset of conventions in actual use. Similarly, antspeak and leetspeak were generated programmatically rather than sampled from genuine user-generated content.
Finally, CHALIS is an evaluation-only benchmark. It does not include a training signal, so it cannot be used directly to improve LangID systems; it can only be used to diagnose them. Addressing the failure modes we identify will require targeted data collection and model development efforts beyond the scope of this work.
We used GitHub Copilot when writing code. We used Grammarly for grammar checking, Claude for proofreading and for text shortening.
| Model | Architecture | Model size | License |
|---|---|---|---|
| FastText (lid218e) | FastText | 1.2 GB | CC BY-NC 4.0 |
| OpenLID | FastText | 1.3 GB | GPL 3.0 |
| GlotLID (v3) | FastText | 1.7 GB | Apache 2.0 |
| GCLD3 | Char-n-gram-based NN | 15MB | Apache 2.0 |
Annotators were university students recruited via internal social media. We explained to them what data we want to annotate for what purpose. The annotators were asked to decide whether a sentence is in their native language. We estimated that processing 1,200 sentences would take approximately one hour and offered the students a compensation of 500 CZK (3.7\(\times\) the minimum wage and 2\(\times\) the median wage in 2025 in Czechia).
Table 4 summarizes the licenses of the four evaluated LangID systems. All evaluated models are freely available for research use: FastText (lid218e) is released under CC BY-NC 4.0, restricting commercial use; OpenLID under GPL 3.0; and both GlotLID and GCLD3 under the permissive Apache License 2.0. The training corpora used to construct CHALIS are similarly open: WMT News Crawl data and the Leipzig Corpora Collection are freely available for research purposes, and Wikipedia-derived data is released under CC BY-SA 4.0. The KenLM language modeling toolkit is distributed under the LGPL 3.0 license. The GLiNER named-entity recognizer used for filtering is released under the Apache License 2.0. Transliteration tools include CyrTranslit (MIT license), the Charles Translator conversion scripts, and the MC2 dataset scripts; pyleetspeak is also MIT-licensed. The Moses tokenizer is distributed under the LGPL 2.1 license. CHALIS itself will be released under CC BY 4.0 to maximize reusability.
CHALIS is distributed as a single JSON file structured as a dictionary with the following top-level keys: single, joint, cyrilic_to_czech, cyrilic_to_english, antspeak,
arabic, diacritics, homoglyphs, and leet. Each key maps to a list of items, where every item is a dictionary with the following fields:
text: the string to be classified;
label: a list of language labels in BCP-47-style format extended with script information (e.g., [’ces_Latn’, ’slk_Latn’]), following the OpenLID convention. For the joint subset, the list contains two labels;
for all other subsets, it contains exactly one;
original: the source text prior to rule-based modification. This field is present in all subsets except single and joint, where the texts were collected directly rather than derived from another text.