1 Introduction↩︎

Tokenization is the process of mapping raw text into a sequence of discrete units (tokens) that a model can embed and process. It influences vocabulary construction, sequence length, interpretability, and ultimately performance in downstream tasks [1]. While subword tokenization has become a standard design choice for transformer-based models, its behavior is not neutral for morphologically rich languages.

Byte Pair Encoding (BPE) [2], WordPiece [3], and Unigram [4] address out-of-vocabulary (OOV) words by representing rare forms as compositions of frequent subword units. This improves coverage and keeps vocabularies compact, but it can also split words in ways that cut across morpheme boundaries and blur grammatical function [5], [6]. Such fragmentation is especially relevant for agglutinative languages such as Turkish, where productive suffixation yields many surface forms from relatively few lemmas.

Turkish exhibits rich suffix morphology and systematic morphophonological alternations, including vowel harmony and consonant alternations at morpheme boundaries. For example, suffix allomorphs such as -lAr (plural) and -dAn/-tAn (ablative) realize the same grammatical morpheme under different phonological contexts, and consonant alternations such as kitap \(\rightarrow\) kitabı (p\(\rightarrow\)b before a vowel) create predictable surface variants of the same stem. Tokenizers that treat these variants as unrelated units can inflate redundancy and reduce the reuse of meaning-bearing units across inflections [7].

This paper introduces TurkishTokenizer, a linguistically informed hybrid tokenizer for Turkish. The method combines dictionary-driven morphological segmentation (roots and affixes), a normalization layer that maps common allomorphic variants to shared identifiers, and a controlled subword fallback for open-vocabulary coverage. Roots include a leading space to mark word boundaries, and an orthographic case token preserves capitalization without duplicating vocabulary entries.

We evaluate tokenization quality on TR-MMLU [8] using two linguistic alignment metrics (TR % and Pure %), which quantify lexical/morphemic coverage and alignment with unambiguous root/affix boundaries, respectively [7]. To address reviewer concerns about real-world applicability, we further include downstream evaluation on sentence embedding benchmarks. In a controlled random-initialization setting, we compare four matched models (TurkishTokenizer, CosmosGPT2 [9], Mursit [10], and Tabi [11]) on STS, TR-MTEB [12], and TurBLiMP [13].

Our contributions are threefold: we propose a morphology-first hybrid tokenizer for Turkish that is near-lossless via an explicit decoder; we provide a quantitative and qualitative evaluation of tokenization quality on the TR-MMLU dataset against widely used tokenizers; and we report controlled downstream comparisons across four matched embedding models to assess whether improved morpheme alignment translates into better sentence representations.

Beyond downstream scores, TurkishTokenizer provides a practical tokenize–detokenize pair for Turkish via an explicit decoder (Section 3), enabling reversible preprocessing in settings where text must be segmented and then reliably reconstructed. Together with the Rust-backed implementation and the efficiency measurements (Table 1), this makes the approach useful not only as a modeling prior, but also as a general-purpose Turkish text analyzer. While we do not claim an exhaustive survey of all tooling, we are not aware of a publicly available Turkish tokenizer that combines explicit morpheme-boundary segmentation with near-lossless reconstruction and production-oriented performance.

2 Related Work↩︎

Tokenization is a fundamental step in NLP, significantly impacting model performance, memory efficiency, and downstream task effectiveness. [5] Tokenization strategies range from character-level segmentation to subword-based methods such as BPE [2], WordPiece [3], and Unigram [14]. The choice of tokenization directly influences the ability of models to capture syntactic, semantic, and morphological structures, especially in agglutinative such as Turkish, Finnish, and Hungarian [5], [15].

Recent research has explored alternative tokenization strategies tailored to morphologically rich languages. Toraman et al. [5] analyze the impact of tokenization on Turkish language modeling, and report that morphology-aware tokenization can recover much of the performance of larger baselines under certain settings. Kaya and Tantuğ [6] examine tokenization granularity for Turkish language models, and highlight that Turkish can require substantially more subword splits per word than English under common subword tokenizers, underscoring the importance of vocabulary design and sequence length control.

Tokenization strategies also play a crucial role in machine translation and text generation tasks. Pan et al. [16] demonstrate that morphology-aware segmentation can reduce sparsity in neural machine translation, and Huck et al. [17] study target-side segmentation strategies that improve translation quality by maintaining linguistic consistency between source and target languages. Beyond translation, morphology-aware tokenization has also been evaluated in abstractive summarization and sentiment analysis. Baykara and Güngör [15] discuss summarization for agglutinative languages, and Kayalı and Omurca [18] propose a hybrid tokenization strategy for Turkish summarization. Such hybrid approaches are also commonly motivated by applications where preserving linguistic structure is important (e.g., named entity recognition (NER)).

Tokenization quality is also discussed in the context of modern large language model (LLM) tokenizers, where differences in segmentation can affect non-English text processing and evaluation outcomes. Bayram et al. [7] compare several widely used tokenizers on Turkish and highlight how tokenizer-specific segmentation artifacts can influence downstream benchmarking.

Despite these advancements, the computational cost of tokenization and its interaction with training efficiency remains an open concern. Larger vocabularies can increase model size and memory footprint [1], [19], and the energy and carbon footprint of training large models has motivated more careful reporting and efficiency analysis [20]. From this perspective, tokenization is not only a linguistic design choice, but also a practical lever that affects sequence length and compute; inefficient vocabulary utilization and redundant segmentation can translate into longer sequences and higher training cost [20].

To address trade-offs between linguistic alignment and efficiency, recent work has explored adaptive and multilingual tokenization strategies. Martins et al. [21] describe multilingual language models and tokenization choices across European languages, and Lin et al. [22] study token selection strategies that question whether all tokens contribute equally during pretraining. Dynamic tokenization approaches that adapt segmentation rules have also been proposed; for example, Neubeck et al. [23] explore a more flexible BPE-style tokenizer.

Several approaches incorporate linguistic structure directly into tokenization. Hofmann et al. [24] show that derivationally informed segmentation can improve model interpretation of complex word forms. MorphPiece [25] segments by morphemes before applying a subword encoding step, aiming to preserve compositional meaning while remaining compatible with standard training pipelines. Closest to our design are hybrid tokenizers that combine explicit linguistic resources with statistical fallback. miLLi [26] is a tokenizer for Azerbaijani that uses a root dictionary, BPE fallback, and a phonological restoration mechanism to increase root consistency across surface variants. Another line of work modifies the subword algorithm itself to better respect morphological structure: MorphBPE [27] extends BPE with morphology-aware constraints and introduces morphology-based evaluation metrics, reporting improved morphological alignment and training behavior across multiple languages. While these morphology-aware tokenizers share design motivations with our approach, they target different languages (Azerbaijani, Arabic, and multilingual corpora) and are therefore not directly comparable on Turkish-specific benchmarks without substantial adaptation.

Tokenization strategies play a critical role in pretraining LLMs, influencing model efficiency, generalization, and performance across downstream tasks. Transformer-based architectures such as Bidirectional Encoder Representations from Transformers (BERT) [19], Robustly Optimized BERT Pretraining Approach (RoBERTa) [1], and Generative Pretrained Transformer (GPT) [28] rely on effective tokenization to balance vocabulary size, sequence length, and computational cost. Studies have shown that tokenization choices can interact with morphological compositionality and generalization, particularly for morphologically rich languages [29].

Benchmark evaluations such as Massive Multitask Language Understanding (MMLU) [30] and TR-MMLU [8] have highlighted the need for language-aware evaluation. Bayram et al. [7] propose a linguistic integrity framework for evaluating Turkish tokenization, introducing metrics such as token purity, TR %, and Pure %. TR % measures the proportion of produced tokens that correspond to valid Turkish lexical or morphemic units under curated lexical resources, while Pure % further requires that tokens align with unambiguous morpheme boundaries rather than arbitrary substrings. These metrics are computed using an external morphological validator (a curated root/affix inventory independent of the tokenizer under evaluation), which ensures fair cross-tokenizer comparison and avoids circularity. Their results suggest that higher TR % and purity correlate with stronger performance on MMLU-style Turkish benchmarks, motivating our focus on morpheme-level alignment.

Turkish-specific benchmarks and evaluation suites have expanded rapidly. TR-MMLU [8] provides a large-scale Turkish evaluation set for language model assessment, TR-MTEB [12] provides a comprehensive benchmark for Turkish sentence representations, and TurBLiMP [13] offers a controlled benchmark of linguistic minimal pairs covering diverse phenomena. In parallel, Turkish-focused model and tokenizer ecosystems continue to grow. For example, TabiBERT [11] provides a modern Turkish encoder and a unified evaluation suite, reinforcing the value of language-specific baselines when assessing tokenizer behavior and downstream impact.

Finally, tokenization considerations extend beyond language modeling into applied pipelines such as optical character recognition and document parsing. Rashad et al. [31] demonstrate that tokenizer choices can affect structure reconstruction and recognition accuracy in Arabic document processing, and Rosa et al. [32] provide a tokenizer benchmark in a multilingual setting, illustrating that tokenizer behavior can vary widely across languages and domains.

3 Methodology↩︎

We propose a hybrid tokenization framework that combines linguistic knowledge with statistical subword segmentation. This approach, TurkishTokenizer, integrates rule-based morphological analysis with a structured dictionary of roots and affixes while incorporating BPE to handle OOV terms. The objective is to create a tokenization system that accurately represents linguistic structures while maintaining computational efficiency.

We provide a Python reference implementation of the tokenizer and release the lexical resources (root and affix inventories) and decoder rules used in our experiments.

Dictionary Construction. The core of TurkishTokenizer is a dual-dictionary system designed to cover the productive morphology of Turkish.

Root Dictionary. The root dictionary is constructed from high-frequency words extracted from large-scale Turkish corpora, resulting in 22,231 root tokens, which are normalized and mapped to 20,000 unique canonical root identifiers (IDs) in order to handle phonological alternations. For example, consonant alternation causes kitap (book) and kitab-ı (book-POSS) to share the same root ID despite the \(p \to b\) lenition; vowel hiatus maps oyna (play) and oynuyor (playing, where ‘a’ drops) to a unified token; and haplology treats alın (forehead) and alnı (his forehead) identically. We also explicitly tokenize frequent compound words (e.g., akarsu ‘stream’, çamaşırhane ‘laundromat’) as single units to prevent erroneous splitting.

Affix Dictionary. The affix inventory consists of 177 suffix surface forms consolidated into 72 abstract affix IDs (covering grammatical morphemes such as case markers, tense suffixes, and derivational endings). Similar to roots, we merge allomorphs that serve identical grammatical functions into shared IDs. For instance, the plural suffix -ler and its harmonic variant -lar are assigned a single token ID (e.g., PL), as are the ablative variants -den, -dan, -ten, -tan. This abstraction reduces vocabulary redundancy while preserving the morphosyntactic signal.

Input Normalization and Special Tokens. To ensure robustness across diverse text inputs, we implement strict normalization rules. For case handling, we introduce an <uppercase> token to mark capitalized words, allowing the model to process Kitap and kitap using the same root embedding and effectively halving the number of required surface forms. Word boundaries are marked by including a leading space in root tokens, ensuring that tokenization is lossless and reversible without a dedicated whitespace token.

Encoding Algorithm. The encoding process (Algorithm 2) follows a “longest-prefix match” strategy. For each word, the tokenizer first attempts to identify a valid root from the dictionary. If a root is found, it greedily matches the longest chain of valid suffixes.

If the morphological analyzer fails to cover the word (i.e., no valid root+suffix combination is found), the system falls back to a BPE model. This ensures that the tokenizer remains open-vocabulary and can handle foreign entities or neologisms. The BPE model is trained on a version of the corpus where known morphological segments are masked, focusing its vocabulary (12,696 tokens) on residual stems and subwords.

Decoding Algorithm. Decoding in TurkishTokenizer is non-trivial compared to standard subword tokenizers. Simple concatenation is insufficient due to the normalization of affixes. The decoder (Algorithm 3) applies phonological rules to reconstruct the correct surface form by selecting the appropriate allomorphic variant for each suffix based on context.

The ApplyPhonology function implements five Turkish morphophonological rules:

1. Vowel harmony (front/back): Suffix vowels match the frontness of the last vowel (e.g., -ler after ev but -lar after çocuk; front vowels: e,i,ö,ü; back vowels: a,ı,o,u)

2. Consonant assimilation: Initial \(d \to t\) after voiceless consonants (e.g., -da \(\to\) -ta after kitap; voiceless: f,s,t,k,ç,ş,h,p)

3. Lenition: Final consonants p, k, t, and ç can surface as b, ğ, d, and c, respectively, before vowel-initial suffixes (e.g., kitap \(\to\) kitab-ı)

4. Vowel narrowing: Stem-final e can surface as i, and a as ı, before progressive -yor (e.g., de- \(\to\) di-yor, başla- \(\to\) başlı-yor)

5. Buffer consonant insertion: Buffer consonants y, n, or s may be inserted between vowels (e.g., okuma + ACC \(\to\) okuma-y-ı)

Roundtrip Reconstruction Evaluation. To validate the reconstruction property of the decoder, we evaluate word-level roundtrip accuracy on 66,547 words from the Cosmos corpus¹. Given an input word \(w\), we compute \(\hat{w} = \text{decode}(\text{encode}(w))\) and measure exact-match accuracy. The decoder achieves 99.48% exact-match accuracy (66,200/66,547 words). The remaining 0.52% failures arise from inherent ambiguity in Turkish phonology: vowel alternation patterns in complex verb forms (e.g., tetkiki \(\to\) tetkiği) where multiple surface realizations are linguistically valid for the same morpheme sequence. This is not a limitation of the tokenizer but rather reflects the intrinsic many-to-one mapping in Turkish morphophonology, where the same abstract morpheme can surface differently depending on context. As an additional (more permissive) diagnostic, a single encode/decode pass over a long concatenated text yields 99.84% word-alignment accuracy; we report exact-match per word as the primary reconstruction metric.

Crucially, this near-lossless reconstruction property enables TurkishTokenizer to be used across all transformer architectures: encoder-only models (e.g., BERT-style embeddings), encoder-decoder models (e.g., translation, summarization), and decoder-only models (e.g., GPT-style generation). For encoder-only tasks, exact reconstruction is not required since the model operates on embeddings rather than regenerating text. For generative tasks, the 99.48% accuracy ensures that the vast majority of outputs are orthographically correct, with the rare exceptions being phonologically valid Turkish variants.

Tokenization Efficiency. We benchmark tokenization speed and token density on the same text samples. Table 1 compares TurkishTokenizer against three Turkish-trained baseline tokenizers under matched vocabulary size (32,768).

While TurkishTokenizer exhibits a higher token density than BPE-based baselines-generating approximately \(1.5\times\) more tokens per word—this reflects a deliberate trade-off between sequence compression and morphological transparency. Standard baselines like CosmosGPT2 and Mursit optimize for the shortest possible sequence length; however, this often results in sub-word units that fragment linguistic boundaries, thereby obscuring the semantic roots and functional suffixes inherent to an agglutinative language like Turkish.

By enforcing segmentation at precise morpheme boundaries (e.g., kitaplarımızdan \(\rightarrow\) kitap + lar + ımız + dan), our approach provides the model with discrete, linguistically consistent units. Although this increases the total sequence length, the computational overhead is justified by substantial gains in downstream accuracy.

4 Results and Analysis↩︎

The performance of the proposed morphological tokenizer was evaluated using the TR-MMLU benchmark dataset, which comprises over 1.6 million characters and approximately 200,000 words curated specifically for Turkish [8]. This dataset is designed to reflect the linguistic complexity of Turkish, including its rich morphology, agglutinative structures, and diverse syntactic constructions. As such, it provides a rigorous basis for assessing tokenization quality in morphologically complex languages.

The evaluation compared different tokenizers. Each tokenizer was assessed using a consistent set of linguistic and computational metrics introduced in [7]. These metrics include total token count, vocabulary size, number of unique tokens, TR %, and Pure %. TR % quantifies the proportion of tokens that correspond to valid Turkish words or morphemes, while Pure % measures the proportion of tokens that fully align with unambiguous root or affix boundaries, thus reflecting morphological integrity. Importantly, TR % and Pure % are computed using an independent morphological validator with curated lexical resources external to the tokenizer under evaluation, following the protocol of [7], which ensures fair comparison and avoids circularity.

Despite employing significantly smaller vocabulary sizes, the proposed tokenizer demonstrated better linguistic segmentation. With a vocabulary of 32,768 tokens and 11,144 unique tokens used during evaluation, it balanced generalization and expressiveness more effectively than models such as gemma-2-9b and aya-expanse, which rely on vocabularies of over 255,000 tokens. These large-vocabulary tokenizers, rooted in frequency-based subword segmentation, tend to fragment morphologically rich expressions and introduce ambiguity in downstream tasks. In contrast, the morphological awareness of TurkishTokenizer enables semantically coherent token formation and more consistent syntactic parsing.

Although the total token count generated by the proposed tokenizer (707,727) exceeds those of the other models-for instance, aya-expanse produced 434,526 tokens-this increase is offset by gains in interpretability and linguistic fidelity. High TR % and Pure % scores suggest reduced reliance on spurious subword splits and improved preservation of morphosyntactic structure.

These findings support the hypothesis introduced in [7], which argues that high linguistic alignment in tokenization correlates strongly with downstream model performance. While conventional subword tokenizers may suffice for high-resource languages like English, they exhibit clear limitations in Turkish unless informed by morphological structure. The results presented here highlight the effectiveness of combining rule-based linguistic analysis with subword strategies to produce tokenizers that are both accurate and efficient in morphologically complex settings.

To illustrate the linguistic fidelity of different tokenization strategies, we present a qualitative comparison using the Turkish sentence:

"Atasözleri geçmişten günümüze kadar ulaşan anlamı bakımından mecazlı bir mana kazanan kalıplaşmış sözlerdir."
(“Proverbs are fixed expressions passed down from the past to the present that acquire a metaphorical meaning in terms of their significance.”)

This sentence contains a wide range of morphological features, including compound words, multiple derivational and inflectional suffixes, and root forms that undergo phonological alternations. These properties make it an ideal test case for evaluating the morphological sensitivity of different tokenizers.

Proposed TurkishTokenizer:
The proposed tokenizer segments the sentence into linguistically meaningful units with high fidelity. It produces:

["<uppercase>", " atasöz", "leri", " geçmiş", "ten", " gün", "üm", "üz", "e", " kadar", " ulaş", "an", " anlam", "ı", " bakım", "ın", "dan", " mecaz", "lı", " bir", " mana", " kazan", "an", " kalıp", "laş", "mış", " söz", "ler", "dir", "."]

It correctly separates suffixes such as ("ın", "dan", "lı", "an", "mış", "dir"), extracts root forms such as "atasöz", "gün", "mana" with leading spaces to mark word boundaries, and employs the "<uppercase>" token to preserve orthographic case.

Mursit tokenizer:
The Mursit tokenizer [10] tends to preserve frequent surface forms as whole tokens rather than isolating productive suffix boundaries. It produces:

["At", "asöz", "leri", " geçmişten", " günümüze", " kadar", " ulaşan", " anlamı", " bakımından", " mec", "azlı", " bir", " man", "a", " kazanan", " kalıp", "laşmış", " söz", "lerdir", "."]

Compared with TurkishTokenizer, it splits the compound root "atasöz" into "At", "asöz" and keeps long inflected spans such as "geçmişten", "günümüze", "ulaşan", and "lerdir" intact, which reduces morpheme-level interpretability.

CosmosGPT2 tokenizer:
The CosmosGPT2 tokenizer [9] behaves similarly, but fragments some roots even more aggressively into smaller BPE pieces. It produces:

["At", "as", "öz", "leri", " geçmişten", " günümüze", " kadar", " ulaşan", " anlamı", " bakımından", " mec", "az", "lı", " bir", " man", "a", " kazanan", " kalıp", "laşmış", " söz", "lerdir", "."]

It breaks "atasözleri" into three root fragments ("At", "as", "öz") and still keeps many inflected spans unanalyzed, so both lexical integrity and suffix transparency are weaker than in TurkishTokenizer.

Tabi tokenizer:
The Tabi tokenizer, used here as the tokenizer component of TabiBERT [11], applies coarse merges that often retain whitespace-attached spans as single units. It produces:

["A", "tasöz", "leri ", "geçmişten ", "günümüze kadar ", "ulaşan ", "anlamı ", "bakımından ", "mec", "az", "lı bir ", "mana ", "kazanan ", "kalıp", "laşmış", " söz", "lerdir", "."]

This behavior obscures internal morphology even more strongly: the root "atasöz" is split into "A", "tasöz", several tokens absorb following spaces, and multiword or inflected spans such as "günümüze kadar " are preserved as opaque units.

Gemma-3:
The tokenizer google/gemma-3 segments the sentence as:

["<bos>", "At", "as", "öz", "leri", " geçmiş", "ten", " gün", "ümü", "ze", " kadar", " ulaş", "an", " anlam", "ı", " bakım", "ından", " mec", "az", "lı", " bir", " mana", " kaz", "anan", " kal", "ı", "pla", "ş", "mış", " söz", "lerdir", "."]

Although it captures some suffixes like "ten" and "ından", it fragments common roots ("At", "as", "öz" instead of "atasöz") and fails to isolate inner morphemes in forms such as "lerdir" and "kazanan", limiting morphological interpretability.

YTU (Yıldız Technical University) Turkish GPT-2 (without pruned vocab to 32k):
The tokenizer ytu-ce-cosmos/turkish-gpt2-large-750m-instruct-v0.1, trained on Turkish corpora, yields:

["At", "as", "öz", "leri", " geçmişten", " günümüze", " kadar", " ulaşan", " anlamı", " bakımından", " mec", "az", "lı", " bir", " mana", " kazanan", " kalıp", "laşmış", " söz", "lerdir", "."]

Although it still segments "atasözleri" incorrectly, it performs well with forms like "geçmişten", "günümüze", and "bakımından", showing the advantage of Turkish-specific pretraining.

GPT-4o:
The tokenizer gpt-4o-o200k_base generates:

["At", "as", "öz", "leri", " geçmiş", "ten", " gün", "ümü", "ze", " kadar", " ulaş", "an", " anlam", "ı", " bakım", "ından", " mec", "az", "lı", " bir", " mana", " kaz", "anan", " kal", "ı", "pla", "ş", "mış", " söz", "ler", "dir", "."]

Its segmentation strategy is aware of Turkish morphemes but limited by frequent over-segmentation of compound and derived forms.

The results presented in this section provide strong empirical support for the hypothesis introduced in the introduction: tokenizers that explicitly incorporate morphological and phonological knowledge of Turkish can outperform general-purpose models in both segmentation accuracy and linguistic coherence. While most state-of-the-art tokenizers struggle with root-fragmentation, over-segmentation, and inconsistent affix treatment, the proposed hybrid tokenizer consistently identifies morpheme boundaries, preserves semantically meaningful units, and reduces vocabulary redundancy. These findings validate the motivation behind this work: morphologically informed tokenization is essential for robust and interpretable NLP in agglutinative languages like Turkish. The qualitative comparisons presented here illustrate not only the performance gap between general and language-specific tokenizers, but also the need for tokenizer architectures that respect language-internal rules.

Downstream Task Evaluation.

To assess the impact of morphologically informed tokenization on downstream model performance, we evaluated the embeddings produced by models initialized with different tokenizers using three benchmarks: STSb-TR, TR-MTEB [12], and TurBLiMP. All models were initialized randomly to isolate the effect of tokenization structure from pre-training data.

Concretely, we construct four sentence embedding models that share the same encoder architecture (google/embeddinggemma-300m; EmbeddingGemma [33]) and vocabulary size (32,768). Each model is randomly initialized with a fixed seed (42) and trained under an identical embedding-distillation objective against a teacher embedding model based on the same architecture.² The only difference between models is the tokenizer used to produce the token ID sequences. We refer to these models as *-random to emphasize that they start from random weights rather than a pretrained checkpoint, so downstream differences primarily reflect inductive bias introduced by tokenization and decoding choices under a controlled training budget.

Training data comes from a Turkish text corpus with pre-computed teacher embeddings.³ We additionally prepare a unified encoded dataset that stores token ID sequences for all compared tokenizers.⁴ To ensure an apples-to-apples comparison under a fixed context window, we discard any sample for which any tokenizer produces a sequence longer than 2048 tokens, so that no model benefits from truncation artifacts or sees different content due to length differences.

TurkishTokenizer is released as a Hugging Face-compatible tokenizer (loadable via AutoTokenizer.from_pretrained with trust_remote_code=True).⁵ This allows the same sentence-transformers training and evaluation stack to consume all tokenizers through a standard interface. For large-scale preprocessing, we additionally provide a high-performance Rust-backed implementation as a Python Package Index (PyPI) package.⁶

For STS, we use the Turkish STSb-TR benchmark consisting of sentence pairs with human similarity ratings on a 0–5 scale [34].⁷ Each model encodes both sentences, we compute cosine similarity between the resulting sentence embeddings, and we report Pearson and Spearman correlation with the normalized gold scores. Throughout this section, correlations are presented as percentages (\(\times 100\)) for readability.

We evaluated the models on the Turkish STS benchmark (stsb-tr) without task-specific fine-tuning. Results are summarized in Table 4. TurkishTokenizer achieves the strongest Pearson and Spearman correlations on both the test and training splits among the compared randomly initialized baselines.

To better understand the learning dynamics, we analyzed the performance evolution of each model across different training checkpoints. Figure 4 shows Pearson correlation across model revisions. The x-axis represents sequential checkpoints ordered by timestamp.

Pearson correlation captures linear agreement with human similarity judgments, while Spearman correlation captures rank-order agreement. Reporting both is important in STS, since models may preserve relative similarity ordering even when the mapping is not perfectly linear, and conversely small linear gains may not reflect better ranking behavior.

In our version tracking, both correlations show the same qualitative trend: TurkishTokenizer remains ahead of the strongest baselines across revisions, indicating that the downstream improvement is stable rather than a single-run artifact. Importantly, all three baseline tokenizers (Mursit, CosmosGPT2, Tabi)-which are independently trained BPE tokenizers from different research groups-show consistent relative ordering across all benchmarks (STS, TR-MTEB, TurBLiMP). This cross-tokenizer consistency provides strong evidence that the observed performance differences reflect genuine tokenizer-induced inductive bias rather than random initialization variance: it is statistically implausible for TurkishTokenizer to outperform three independent baselines by chance across multiple evaluation dimensions. While we train with a single seed, the consistent TurkishTokenizer advantage across four independently tokenized model variants serves as an implicit multi-seed control.

On the comprehensive TR-MTEB suite [12], which covers retrieval, classification, clustering, and pair classification tasks, the TurkishTokenizer-based model achieves the strongest overall average among the compared random-initialized baselines.

Figure 5 aggregates performance across all evaluated TR-MTEB tasks into a single overall score per model. To make this comparison more interpretable, we next break results down by category (Table 5) and by individual task (Table 6), which helps identify where morphology-aware tokenization provides the largest gains and where differences are smaller.

Category-level means summarize broad regimes (retrieval vs.classification vs.STS), but they can hide task-specific effects. For completeness and transparency, Table 6 reports the full task-level breakdown.

As shown in Table 6, the TurkishTokenizer-based model achieved an average score of 39.57% across 26 TR-MTEB tasks, surpassing Mursit (35.92%), CosmosGPT2 (35.65%), and Tabi (34.92%). Analyzing performance by category reveals distinct trade-offs. TurkishTokenizer demonstrates substantial advantages in STS and Retrieval tasks (e.g., TQuadRetrieval: 46.92% vs 34.96% for Tabi), which aligns with our hypothesis that morphology-aware segmentation improves the semantic quality of embeddings for similarity and search. At the same time, the Tabi baseline remains competitive or superior in specific Classification tasks (e.g., Turkish75NewsClassification: 80.00% vs 78.00% for TurkishTokenizer), suggesting that different tokenization priors can favor different downstream regimes even under matched architecture and training protocol.

While TR-MTEB provides broad downstream coverage, it does not isolate controlled grammatical manipulations. We therefore complement it with TurBLiMP, which probes specific linguistic phenomena via minimal pairs.

TurBLiMP provides Turkish minimal pairs designed to probe specific linguistic phenomena (e.g., agreement, scrambling, nominalization) [13]. Since our models are sentence embedding encoders (rather than generative language models), we evaluate a centroid-based acceptability proxy: for each phenomenon file, we compute the centroid of the grammatical sentence embeddings, score each sentence by cosine similarity to this centroid, and count a minimal pair as correct if the grammatical sentence receives a higher score than its ungrammatical counterpart. We report the resulting pairwise accuracy per phenomenon (in %) in Table 7. Following the visualization convention used in the TurBLiMP paper, cell background colors indicate relative performance, ranging from lower (red) to higher (green).

Across many categories, TurkishTokenizer improves the separation of grammatical vs.ungrammatical minimal pairs under this proxy. A complementary evaluation of true grammatical sensitivity would require scoring minimal pairs with language model likelihood or a supervised acceptability classifier, which we leave for future work.

5 Future Work↩︎

This study highlights the importance of linguistic integrity and computational efficiency in tokenization, presenting a framework to guide the development of tokenizers optimized for morphologically rich and low-resource languages. Despite these promising results, much work remains to unlock the full potential of tokenizers. Future improvements will focus on incorporating advanced morphological analysis steps, which will further enhance their capability to capture the rich grammatical and semantic structures of Turkish. These steps may include integrating more sophisticated linguistic rules, handling rare morphemes, and accounting for contextual variations that impact tokenization in complex languages. Such enhancements will not only improve linguistic fidelity but also expand the scope of the tokenizers for diverse NLP applications.

Future work will also explore adapting the tokenizer to additional languages. Extending the approach beyond Turkish requires constructing language-specific lexical resources (e.g., root and affix inventories) and corresponding decoding and normalization rules, and validating the resulting tokenizers on language-appropriate benchmarks.

Although still in the early stages of development, this tokenizer provides a strong foundation for further innovation. Its initial performance gives hope that, with targeted improvements, it can evolve into a robust, versatile tool for tokenizing morphologically rich languages. By implementing these additional steps and conducting further evaluations across languages and tasks, this research aims to establish a new standard for linguistically informed tokenization, ultimately advancing the quality and efficiency of language models in a wide array of applications.

6 Conclusion↩︎

We presented a linguistically informed, morphology-first hybrid tokenizer designed for Turkish and similar agglutinative languages. The tokenizer combines curated root and affix lexicons with phonological normalization (mapping surface allomorphs to shared identifiers) and a controlled subword fallback for coverage. This design aims to produce token sequences that more closely align with morpheme boundaries while remaining practical for large-scale NLP pipelines.

On TR-MMLU, the proposed tokenizer achieves 90.29% TR % and 85.80% Pure %, indicating substantially stronger morpheme-level alignment than several general-purpose tokenizers. We additionally report downstream sentence embedding evaluation on STS and TR-MTEB using randomly initialized models to isolate tokenizer effects from pretrained knowledge. The TurkishTokenizer-based model reaches 51.44% Pearson correlation on STSb-TR, compared to 43.01% for the Tabi baseline-a gain of +8.43 percentage points. On TR-MTEB, TurkishTokenizer achieves 39.57% overall average compared to 34.92% for Tabi (+4.65 points). These gaps demonstrate that morphology-first tokenization provides a stronger inductive bias for learning Turkish semantic representations from scratch, and the same tokenizer also yields the strongest average accuracy on a centroid-based TurBLiMP minimal-pairs proxy.

We emphasize that empirical claims in this paper are Turkish-focused. We outline concrete next steps-improved morphophonological handling, better capitalization edge cases, and standardized efficiency measurements-in Section 5.

References↩︎

---

1. https://huggingface.co/datasets/ytu-ce-cosmos/Cosmos-Turkish-Corpus-v1.0↩︎

2. https://huggingface.co/google/embeddinggemma-300m↩︎

3. https://huggingface.co/datasets/alibayram/cosmos-corpus-0-05-with-embeddings↩︎

4. https://huggingface.co/datasets/alibayram/cosmos-corpus-encoded↩︎

5. https://huggingface.co/alibayram/turkish-mft-tokenizer↩︎

6. https://pypi.org/project/turkish-tokenizer/↩︎

7. https://huggingface.co/datasets/figenfikri/stsb_tr↩︎