1 Introduction↩︎

Tokenizers chosen at the start of pre-training typically allocate vocabulary in proportion to the distribution of the pre-training corpus, which reflects deployment priorities at that time. As those priorities shift, languages that were underrepresented originally, or became relevant only later, are split into many more tokens per word than the languages the tokenizer was trained for [1], [2]. Because generation invokes the decoder once per output token, this fragmentation raises decode latency, compute, and energy consumption for users of those languages. For large cloud models, the embedding and LM-head matrices are a negligible fraction of total parameters, so broader script coverage can be bought with a larger vocabulary. On-device models face tighter constraints: the output LM-head projection touches the full vocabulary dimension on every decode step, and at batch size 1 decode is generally memory-bandwidth-bound [3], so a larger vocabulary noticeably increases both the on-device footprint (the embedding/LM-head matrix that must be held in memory) and the per-token decode cost. Vocabulary size is therefore a first-order design constraint, and edge models typically ship with compact vocabularies covering a chosen set of priority languages and domains, accepting fragmentation outside that set.

Open model families resolve the trade-off between vocabulary size and language coverage at different points, from compact tokenizers to much larger ones [4]–[7]. LFM2 [8] sits at the compact end, as expected for an on-device model. Its tokenizer is a 65K-vocabulary byte-level BPE tokenizer optimized for English, code, and a fixed set of additional languages. It allocates little of that budget to several languages that it encodes inefficiently, such as Hindi and Vietnamese, which it therefore splits into several times as many tokens as comparable English text. Two standard remedies are unattractive. Training a new tokenizer and pre-training from scratch discards the compute already invested in the checkpoint. Replacing the tokenizer post-hoc with a zero-shot transfer method [9] solves a harder problem than necessary: it must infer the source-to-target relationship for an arbitrary target tokenizer.

We present an approach (Figure 1) for in-place tokenizer expansion, targeting a narrower but common production setting in which the model producer controls the tokenizer upgrade. We initialize the merge table from the pre-existing tokenizer, hold the existing merges fixed, and append new merges learned from a multilingual corpus. This continued-BPE construction maps most source tokens one-to-one into the expanded vocabulary and gives every new token an exact decomposition into source tokens, neither of which is available to methods that target an independently trained third-party tokenizer. The model is initialized to match: rows for the carried-over tokens are copied unchanged, and new rows take the mean of their source-token decomposition. We then adapt the model in two stages: embedding-only training with the body frozen, followed by full-model continued pre-training on a balanced multilingual corpus. We refer to this procedure as tokenizer expansion, a form of cross-lingual vocabulary adaptation (CVA) [10], [11] in which the tokenizer’s BPE merges are extended rather than vocabulary entries merely appended.

We apply the recipe during continued pre-training of an LFM2-8B-A1B checkpoint. The adapted model then goes through the standard mid-training and post-training phases to produce LFM2.5-8B-A1B [8], [12]. The token-count reduction is large: the expanded tokenizer encodes Hindi in roughly \(2.4\times\) fewer tokens and Vietnamese in \(2.6\times\) fewer than the source tokenizer for the same text, with the largest reduction on Thai at \(4.0\times\) (Figure 2). Combined with the measured per-token cost of the larger vocabulary, we estimate this compression yields a \(2.2\)–\(3.7\times\) per-character decode speedup on these languages across our reference devices (Section 7).

Our contribution is on the model and deployment side. To our knowledge, ours is the first account of the model-side recipe for in-place, continued-BPE tokenizer expansion at production scale (an 8B-parameter on-device MoE). We present an embedding-initialization and two-stage adaptation recipe that applies the vocabulary upgrade to an existing checkpoint without sacrificing accumulated quality (Section 6), an on-device latency characterization that separates tokenizer compression from the per-token cost of a larger vocabulary (Section 7), and the negative findings that shaped the recipe (Section 8). The continued-BPE construction itself is shared with independent, concurrent work [13], which studies its tokenizer-side properties. We build the model-side recipe on top of it and release the model weights and the expanded tokenizer.

2 Tokenizer Construction↩︎

The expanded tokenizer is a byte-level BPE tokenizer [14], [15] built by continuing the merge training of the LFM2 tokenizer on a multilingual corpus weighted toward the under-tokenized languages.

2.0.0.1 Source tokenizer.

The LFM2 tokenizer has a 65K-token byte-level BPE vocabulary trained on a corpus weighted toward English, code, and a fixed set of additional languages (Arabic, Chinese, French, German, Japanese, Korean, Spanish) [8]. Its vocabulary predates the broader multilingual deployment requirements that motivate this work.

2.0.0.2 Extended training and mapping properties.

We build the expanded 128K tokenizer by initializing the merge table from the LFM2 tokenizer and continuing BPE training on a multilingual corpus that adds coverage of under-represented languages such as Hindi and Thai, while retaining English and code to preserve compression on existing workloads. Existing merges are kept and new merges are appended, so every new token decomposes, by construction, into a sequence of one or more source tokens. Although the expanded vocabulary reassigns token IDs, 63,151 of the 64,400 source tokens still appear as single tokens in it (Table 1), so their embedding rows transfer directly. Every other expanded token splits into two or more source tokens. Section 3 uses this decomposition for embedding initialization. Further construction details, including special-token handling and a comparison against other production tokenizers, are given in Appendix 12.

2.0.0.3 Why continued-BPE extension rather than vocabulary union.

An alternative, used by early language-specific Llama derivatives such as Chinese-LLaMA [16], trains a separate tokenizer on the target language and takes the union of its vocabulary with the source vocabulary. Because the two tokenizers are trained independently, a new token need not decompose into source tokens, so there is no canonical source-side decomposition from which to initialize its embedding. Continued-BPE extension keeps a single merge table, so every new token decomposes deterministically into source tokens, which is exactly what the embedding initialization in Section 3 requires.

2.0.0.4 Language coverage and fertility.

We prioritized languages by how poorly the source tokenizer encodes them. Token fertility is the average number of tokens a tokenizer uses per word, and the source tokenizer has high fertility precisely on the languages that received little vocabulary budget, splitting their words into many tokens. The under-tokenized languages span non-Latin scripts (for example Hindi, Thai, and Bengali) and Latin-script languages poorly served by the source tokenizer. Figure 2 reports, per language, the ratio of source-tokenizer to expanded-tokenizer token count on the same held-out text (non-English web text drawn from FineWeb 2 [17], plus held-out English, code, and JSON samples), so 1.0\(\times\) is parity and higher means a shorter encoding. The under-tokenized languages improve substantially (for example Hindi \(2.4\times\) and Thai \(4.0\times\)), while English and code stay at parity by design. Because batch-1 decode cost is dominated by per-token weight reads, this compression largely carries over into decode-time speedup, as discussed in Section 7.

3 Embedding Initialization↩︎

Given the pre-trained input embedding matrix \(E_{\text{src}} \in \mathbb{R}^{|V_{\text{src}}| \times d}\) for the 65K source vocabulary, we build the embedding matrix \(E_{\text{new}} \in \mathbb{R}^{|V_{\text{new}}| \times d}\) for the 128K expanded vocabulary (\(|V_{\text{new}}| = 125{,}017\) active entries, Table 1) with a single rule. Each new token \(t\) decomposes into a sequence \((t'_1, \ldots, t'_N)\) of source tokens (Section 2), and we set its row to the mean of the corresponding source rows: \[E_{\text{new}}[t] = \frac{1}{N} \sum_{i=1}^{N} E_{\text{src}}[t'_i].\] For the 63,151 tokens that decompose to a single source token (\(N=1\)) this copies the source row unchanged, and for the rest it averages two or more rows.

Because LFM2-8B-A1B and LFM2.5-8B-A1B tie the input embedding and the output LM head, the same matrix \(E_{\text{new}}\) provides both, so initializing a row sets both the token’s input representation and its output classifier vector. The same construction applies to untied models, which initialize the two matrices separately.

3.0.0.1 Why the mean.

We average rather than sum so that new rows keep a norm comparable to existing rows rather than growing with \(N\). Because every new token decomposes into known source tokens, this centroid is always defined and no row needs random initialization. We do not claim the centroid is the model’s internal representation of the token, only that it is a sensible starting point from which Stage 1 training converges cleanly (Section 6.1). Initializing new rows by averaging existing embeddings is an established baseline [11], [18]. More elaborate weighted or learned initializers were developed mainly for tokenizer replacement, where the new tokenizer is trained independently and new tokens have no canonical decomposition into source tokens (Section 5). Here every new token has such a decomposition and Stage 1 trains the new rows, so we did not expect a more elaborate initializer to change the outcome and did not pursue it.

4 Training Recipe↩︎

The source model for the expansion pipeline is an internal continuation of LFM2-8B-A1B (8.3B total parameters, 1.5B active per token), continued pre-trained from its released 12T-token base [8] to 35T tokens, still using the original 65K-vocabulary tokenizer. We call this checkpoint the source model throughout. Our primary evidence that the recipe preserves pre-expansion quality is the comparison of the expanded model to this source model.

After embedding initialization, the model is adapted to the expanded tokenizer in two stages, illustrated in Figure 1 (lower row of the pipeline) and summarized in Table 2, after which mid-training and post-training follow. The staged structure is empirical: training all parameters at once degraded the parts of the model that already worked. Restricting the first stage to the new embedding rows isolates the change, and the subsequent full-model continued pre-training re-balances the model on data tokenized with the expanded tokenizer. As an MoE model, it requires healthy expert routing. We monitored expert load balancing and router health throughout the adaptation, with the router frozen in Stage 1 and trainable in Stage 2 and mid-training. These internal diagnostics, not reported here, showed no degradation from the vocabulary expansion.

4.1 Stage 1: Embedding-only Training↩︎

In Stage 1 we train the new rows of the tied embedding/LM-head matrix \(E_{\text{new}}\) over 600B tokens while keeping all other parameters frozen, including the rows copied directly from the source (the \(N=1\) case of Section 3). Because the input embeddings and output head are tied, updating a new row changes both the input representation for that token and its output-logit vector. The training data is a multilingual mixture weighted toward the under-tokenized languages whose new rows are being trained, with the standard next-token prediction objective. These 600B tokens train only the embedding and we exclude them from the model’s pre-training-token total. We train at a 4,096-token sequence length, matching the source model’s pre-training context length. Context-length extension to 32K and then 128K is deferred to mid-training (Section 4.3). We reinitialize the optimizer state, warm up the learning rate to a relatively high peak, and then hold it constant. The high peak suits training a small set of freshly initialized parameters.

This embedding-only step is closest to ReTok [19], which also trains only the embedding after a vocabulary change (full comparison in Section 5). We differ by additionally freezing the copied rows: they already reproduce the source model’s lookup and logits for those tokens, and we find they drift harmfully when the rest of the body cannot follow (Section 8).

Stage 1 alone recovers most of the dip from the zero-shot vocabulary swap. On the per-stage trajectory in Figure 3, it closes 4.8 of the 5.8 aggregate points lost at the swap, with English and code largely recovered. The under-tokenized languages also recover from the zero-shot dip, but their largest gains come in Stage 2. Stage 1 alone is not sufficient: the full-model continued pre-training of Stage 2 is needed to close the remaining gap and integrate the new vocabulary into the body of the model (Section 6).

4.2 Stage 2: Continued Pre-training↩︎

In Stage 2 we unfreeze all parameters and continue pre-training for 400B tokens, at the same sequence length as Stage 1, on a balanced multilingual corpus tokenized with the expanded tokenizer. As in Stage 1, we reinitialize the optimizer state and warm up the learning rate before holding it constant, here at a lower peak because the full model starts near a working configuration.

The Stage 2 data mixture is the main factor in whether the unfrozen training preserves what Stage 1 recovered. We studied its effect directly as an ablation (Section 8), where a predominantly English mixture left multiple-choice accuracy intact but caused a broad regression on generative benchmarks. A mixture with substantially more multilingual content, including coverage of the under-tokenized languages, removed the regression and yielded a checkpoint that matched or exceeded the source model on aggregate benchmarks (Section 6).

4.3 Mid-training and Post-training↩︎

Mid-training and post-training follow the LFM2.5 pipeline [12], an update of the earlier LFM2 procedure [8]. Tokenizer expansion requires no change to this pipeline: the Stage 2 checkpoint is a drop-in replacement for the base model the pipeline normally consumes. Mid-training proceeds in two context-length phases, a 32K-context phase over 2T tokens followed by a 128K-context extension over 400B tokens, both drawing on higher-quality and naturally long-context sources, with the learning rate annealed. The chat template, special tokens, and tool-use formats follow the released LFM2.5 configuration.

5 Related Work↩︎

Tokenizer expansion sits between cross-lingual vocabulary adaptation (CVA), which adds language coverage to a pre-trained model by expanding the vocabulary and continuing pre-training, and tokenizer transfer, which replaces a model’s tokenizer outright and retrains or transfers its embeddings. Prior work establishes the deployment cost of poor tokenization [1], [20] and studies CVA’s inference efficiency and embedding-initialization choices [10], [11]. We position our recipe against the most directly comparable works by the component each one shares with it.

5.0.0.1 Tokenizer construction.

The continued-BPE construction we use, initializing the merge table from the source tokenizer and continuing BPE merge training on the target corpus, is also introduced in independent, concurrent work by [13], who study its tokenizer-side properties (compression, avoidance of unreachable tokens, leaf-based pruning) across several base models at smaller scale and release a toolkit. Our contribution is on the model side: we apply the construction at 8B MoE scale and develop the embedding initialization and two-stage adaptation that recover quality, together with the on-device latency analysis and the accompanying negative findings. [21] show that swapping or substantially altering a pre-trained model’s tokenizer requires extensive continued pre-training to avoid performance degradation, which supports the full-model continued pre-training in our Stage 2.

5.0.0.2 Adaptation recipe.

Two model-side works are each closest to a single stage of our recipe. ReTok [19] trains only the embedding after a tokenizer change with the body frozen, as in our Stage 1, but it replaces the tokenizer and initializes new rows by heuristic segmentation under the old tokenizer. Our continued-BPE construction makes that decomposition exact, and we additionally unfreeze the body in Stage 2, which we find necessary at scale (Section 8). EEVE [22] progressively unfreezes the model across seven stages, similar to our staged structure, but appends merges from a separately trained tokenizer rather than continuing the source merges.

5.0.0.3 Embedding initialization.

The mean-of-subwords centroid we use is a simple, established baseline [11], [18], and it suffices here because the continued-BPE construction makes the source-side decomposition exact. A large body of work develops more elaborate initializers, using alignment, overlap statistics, weighted source combinations, hypernetworks, or distillation, for the harder setting where the new tokenizer is trained independently of the source and the source-target relationship must be recovered [9], [23]–[31].

5.0.0.4 Vocabulary extension with continued pre-training.

Vocabulary extension followed by continued pre-training is well established for language-specific Llama derivatives [16], [32] and for multilingual continued training of Southeast Asian [33], [34], low-resource [35], [36], instruction-tuned [37], and translation-oriented [38] models. [39] show competitive target-language quality from as little as \(\sim\)​0.01 GB of continued pre-training text. We target a less-explored point in this space: an in-place upgrade of a single already-deployed on-device model, where the central challenge is preserving the model’s existing quality while improving its tokenization.

6 Evaluation↩︎

We evaluate along two axes. First, quality recovery across the expansion stages (Section 6.1) tracks an aggregate of pre-training-style benchmarks from the source through the zero-shot swap to Stages 1 and 2. Because these benchmarks are dominated by English, code, and high-resource languages, this axis is primarily evidence that expansion preserves the source checkpoint’s existing capability, rather than a measure of the newly added languages. Second, per-language detail on Global-MMLU (Section 6.2) covers both sides of the goal: it confirms that previously supported languages do not regress and quantifies the gains on the under-tokenized languages that motivated the expansion. Beyond these two axes, the released LFM2.5-8B-A1B also improves over LFM2-8B-A1B on instruction following, tool use, math, and agentic tasks. Those gains reflect the full pipeline of additional pre-training and reasoning-oriented post-training rather than tokenizer expansion alone, and are reported in the LFM2.5 release materials [12] rather than here.

6.1 Quality Recovery Across Stages↩︎

Figure 3 tracks an aggregate quality score, the unweighted mean of eight public benchmarks (Appendix 10), across four controlled stages: Source \(\to\) Zero-shot \(\to\) Stage 1 \(\to\) Stage 2. These share one training objective and differ only in the vocabulary change and its recovery training, so the trajectory isolates the tokenizer-expansion pipeline. Mid-training and post-training are excluded, since they would confound the tokenizer change with later capability gains.

Because the source is a base pre-training checkpoint with no instruction tuning, we restrict the aggregate to benchmarks a base model can be scored on without instruction following. These are log-probability benchmarks, which rank fixed answer options by the probability the model assigns them, answer-extraction benchmarks, which read a short final answer out of the model’s free-form output, and execution-based code benchmarks. We exclude chat-dependent instruction-following benchmarks (e.g.IFEval), which would understate the base checkpoint’s quality because it has not been trained for chat-template adherence.

The recovery is monotonic (Figure 3) after the zero-shot checkpoint. The zero-shot checkpoint loses 5.8 aggregate points, reflecting a body not yet adapted to the new vocabulary. Stage 1 (embedding-only, body frozen) recovers 4.8 of them, and Stage 2 (full continued pre-training) closes the gap and surpasses the source by 3.6 points. Since Stage 2 adds 400B tokens of full-model continued pre-training, we read this surplus as potentially the effect of that additional training rather than of the vocabulary change in isolation. The headline result is therefore preservation: the Stage 2 checkpoint matches or exceeds the source on pre-training-style benchmarks while now using the 128K tokenizer. The under-tokenized-language gains that motivated the expansion are reported separately in Section 6.2. By category, Stage 2 exceeds the source on Knowledge (\(+4.5\)), Math (\(+2.8\)), Code (\(+6.3\)), and Multilingual (\(+1.6\)) (Table 3), though a few individual benchmarks dip slightly, mainly in math and multilingual generation. The Multilingual category here is MMMLU and MGSM, which cover mostly high-resource languages rather than the under-tokenized targets. These dips may reflect the Stage 2 mixture’s rebalancing toward the under-tokenized languages rather than a property of the recipe, but we did not isolate the cause. Per-benchmark detail across stages is in Appendix 10, Table 8.

8pt

6.2 Multilingual Benchmark Detail↩︎

Per-language results confirm the headline pattern: previously supported languages hold their ground while the under-tokenized languages gain, so the expansion adds coverage without regressing what already worked.

6.2.0.1 Global-MMLU (broad multilingual MCQ).

Global-MMLU [40] renders the full MMLU test set (about \(14{,}000\) four-choice questions per language, chance \(25\%\)) into 42 languages, using a combination of professional, community-verified, and machine translations. We report the 39 languages on which at least one checkpoint exceeds 30%. The other 3 stay at the floor throughout (Appendix Table 9). The expanded model recovers and exceeds the source by Stage 2 (mean over these 39 languages: 41.4, 37.0, 40.6, 43.7 across Source, Zero-shot, Stage 1, Stage 2, net \(+2.3\)). The zero-shot vocabulary swap causes a transient dip (mean 37.0, Arabic falling to 32.9) that Stages 1 and 2 recover. By Stage 2 the previously supported languages are preserved, with per-language changes ranging from \(-0.3\) (Spanish) to \(+1.8\) (Arabic), at or near the sampling noise, while the under-tokenized languages gain substantially (Vietnamese \(+11.6\), Indonesian \(+9.1\), Hindi \(+7.6\), Malay \(+6.8\)) (Table 4).

6pt

6.2.0.2 Under-tokenized languages.

Among the under-tokenized languages, Global-MMLU covers Hindi, Bengali, Indonesian, Malay, and Vietnamese, all of which improve at Stage 2. Hindi, Indonesian, Malay, and Vietnamese appear in Table 4, and Bengali improves by \(+4.8\) (Appendix Table 9). Thai, another under-tokenized language, is not part of Global-MMLU and is not separately evaluated here.

7 On-device Latency↩︎

The primary motivation for the tokenizer-expansion effort is on-device decoding efficiency. The user-facing metric is wall-clock time per character of generated text rather than raw tokens per second, so the expansion is a net win on a given language only if its compression gain there outweighs the added per-token cost of the larger vocabulary. In brief, we estimate that on all three reference devices the expanded tokenizer decodes the heavily under-tokenized languages (Thai, Bengali, Vietnamese, Hindi) 2.2\(\times\) to 3.7\(\times\) faster per character than the source tokenizer, at a per-character cost of up to about 9% on the least-compressed languages (English and code). The two components behind these figures, per-language compression and per-device decode cost, are measured separately in the rest of this section.

7.0.0.1 Why we report a synthesis rather than a direct end-to-end measurement.

Both LFM2-8B-A1B and LFM2.5-8B-A1B are released, so one direct option is to decode the same Hindi text on the same device with each release and compare. We do not treat that comparison as our primary evidence, because it does not isolate the tokenizer: the two releases differ along many axes beyond it, and LFM2.5-8B-A1B is a reasoning model that emits variable-length thinking traces, so the same prompt yields different output lengths and token distributions. Restricting the change to the tokenizer alone, by swapping it into a single post-trained checkpoint, is not feasible without re-running the full adaptation pipeline, which would introduce changes of its own. Either route therefore confounds the tokenizer change with other simultaneous changes.

Instead, we decompose decode time per character of output into two independently measurable components. The decoding speedup of the expanded tokenizer over the source tokenizer for the same model body factors cleanly: \[\underbrace{\frac{\text{chars/sec}_{\text{new}}}{\text{chars/sec}_{\text{old}}}}_{\text{decode speedup per character}} \;=\; \underbrace{\frac{\text{chars/token}_{\text{new}}}{\text{chars/token}_{\text{old}}}}_{\text{compression (per-language)}} \;\times\; \underbrace{\frac{\text{decode t/s}_{\text{new}}}{\text{decode t/s}_{\text{old}}}}_{\text{vocabulary-size cost (per-device)}}\] The compression term is a property of the tokenizers alone, measured on multilingual text without running the model (Figure 2, Section 2). The vocabulary-size cost term is a property of the model on a given device, measured on the same LFM2-8B-A1B body with only the embedding/LM-head matrix size varied (Section 7.1). The factorization is an identity. What makes the synthesized speedup an estimate is that we measure the vocabulary-size cost once and reuse it as a constant across languages and output lengths. That is a good approximation here: at batch size 1, decode is memory-bandwidth-bound and each step is dominated by reading the model weights and the LM-head matrix, a near-constant number of bytes per token regardless of which tokens are generated. The main thing it leaves out is the KV cache, which grows with context length and is identical for both vocabularies, so on long sequences it takes a larger share of each step and the per-token vocabulary penalty shrinks.

7.1 Per-token decode cost of vocabulary scaling↩︎

To isolate the vocabulary-size cost from the tokenization compression, we hold the model body fixed and vary only the embedding and LM-head matrix size. We benchmark the same LFM2-8B-A1B weights with that matrix resized from the source tokenizer’s 65K entries to the expanded tokenizer’s 128K entries, feeding identical token sequences in both configurations so that the measurement isolates the per-token decode cost of the larger matrix from any tokenization-induced change in token count. The benchmark uses llama-bench [41] with a fixed decode length of 128 tokens and Q4_0 quantization throughout. Prefill is measured with a 512-token prompt on the M4 Max and a 256-token prompt on the Snapdragon 8 Elite Gen 5 phone (shorter to limit thermal throttling), but our synthesis depends only on the decode throughput.

Prefill and decode respond differently to the larger matrix. Prefill is compute-bound and amortizes the LM-head reads across the input, so it barely moves with vocabulary size. Decode is memory-bandwidth-bound and pays the larger LM-head read on every step, dropping 7–10% per token. Full per-device numbers and disk sizes are in Appendix 13, Table 12.

7.2 Per-character decode speedup (synthesis)↩︎

Combining the per-language compression from Figure 2 with the per-device decode cost ratio from Table 12 gives the estimated decode speedup per character of output, Table 5.

8pt

Two patterns stand out in Table 5. On Thai, Bengali, Vietnamese, and Hindi, where compression is largest, the expanded tokenizer produces 2.2\(\times\) to 3.7\(\times\) as many characters per unit time across all three reference devices, the primary benefit. Because the per-token cost of the larger LM-head is the same for every language, the change nets out as a gain wherever compression covers it (those four, plus previously supported languages like Arabic and Korean) and as a regression of up to about 9% per character where compression is near parity (English, code, the major European languages). This is the deliberate trade-off of the design: large gains on under-served scripts, paid for by a small uniform per-token cost. We discuss it further in Section 9.1.

7.3 Vocabulary-size scaling beyond 128K↩︎

We considered, but did not adopt, larger vocabulary sizes. Figure 5 extends the experiment in Figure 4 to 192K and 256K matrix sizes on the same three devices, plotted as decode-throughput ratio relative to the 65K baseline on each device.

Decode cost rises with vocabulary size on every device, fastest on the Snapdragon 8 Elite Gen 5, where doubling to 128K costs about 9% of decode throughput, tripling to 192K costs 26%, and quadrupling to 256K costs 37%. A larger vocabulary is worth it on a language only if its added compression outweighs this added device cost. Take 256K on the Snapdragon: the decode-cost ratio is 0.63, so a language needs compression above \(1/0.63 \approx 1.59\times\) over the source just to break even. Only Thai, Bengali, Vietnamese, and Hindi reach that, and they already clear it at 128K, where the device cost is far lower. The incremental compression from doubling the vocabulary again (128K to 256K) would have to be large to overcome the further throughput loss, and BPE compression has diminishing returns as the vocabulary grows. We therefore stop at 128K on these devices.

8 Ablations and Negative Findings↩︎

Two findings shaped the final recipe. We report them here both for transparency about what did not work and as practical guidance for practitioners attempting similar tokenizer-expansion efforts.

8.0.0.1 Training the full embedding matrix in Stage 1 degrades performance.

To decide how much of the embedding to train in Stage 1, we ran the alternative configuration: training all rows of \(E_{\text{new}}\), including the rows copied from the source model, rather than only the newly added rows, with the rest of the model frozen in both cases. The hypothesis was that letting the existing rows adjust would help the model reconcile the embedding space with the new vocabulary. Instead it degraded the model, with the damage concentrated on generation: MATH500, HumanEval+, and MGSM all fell while the model began to repeat itself, even though its multiple-choice scores (MMLU and MMMLU) were unchanged or higher. Because the input and output embeddings are tied, training \(E_{\text{new}}\) also moves the unembedding that produces the output logits. We did not isolate the mechanism directly, but the most likely explanation is that the copied rows are already matched to the frozen body, and letting them drift while the body cannot adapt breaks that match between the body and the logit layer. The reason the damage shows up in generation while sparing multiple-choice ranking is the same asymmetry we analyze under the Stage 2 mixture ablation below. Restricting Stage 1 to the new rows avoids the problem. Table 6 reports the comparison.

6pt

8.0.0.2 Stage 2 data mixture is critical.

To measure how much the Stage 2 mixture matters, we ran it with a deliberately unbalanced, predominantly English continued pre-training mix, testing the hypothesis that multilingual coverage was already established in Stage 1 and that English data would preserve general capability. The result was not a graceful loss of multilingual quality but a broad generative collapse. The clearest evidence is MATH500, an English-language benchmark, which fell from 61.8 to 27.0. MGSM fell from 61.2 to 28.4, and the model exhibited markedly more degenerate repetition. All of this happened while the model improved on log-probability-scored MCQ benchmarks (MMLU and MMMLU). Constructing a more balanced multilingual mixture eliminated the regression and produced the checkpoint reported in Section 6. The comparison is in Table 7.

The MCQ-versus-generation asymmetry is the key diagnostic. Multiple-choice benchmarks scored by log-probability over A/B/C/D options measure whether the model assigns higher probability to the correct option than to the distractors. They do not require the model to generate coherent text. The bad-mixture model retains, and slightly improves, this discriminative ability, while losing the generative ability needed for free-form math and multilingual reasoning. The MMMLU improvement under the bad mixture fits this story: the model still reads the multilingual prompt correctly and ranks options sensibly, while having lost the ability to produce coherent multilingual output. MMLU-Pro is the partial exception: although multiple-choice in format, it drops with the generative benchmarks (35.2 to 20.6) rather than holding like MMLU and MMMLU.

6pt

A per-language breakdown of the MGSM collapse, and a tentative script-mediated reading of why Latin-script European languages collapse most sharply, is given in Appendix 11.

Taken together, the two ablations point to a practical lesson about evaluation. Two different interventions, drifting the tied embedding against a frozen body and continued pre-training on an unbalanced mixture, produced the same signature, and a practitioner relying on MMLU-style accuracy alone would have judged both broken checkpoints healthy. Tokenizer expansion, and continued pre-training more generally, can degrade a model in ways that multiple-choice accuracy does not register. This is why the evaluation in Section 6 includes generative benchmarks rather than relying on multiple-choice accuracy alone.

9 Conclusion↩︎

We present tokenizer expansion, an in-place recipe for extending the tokenizer of an already pre-trained LLM in the case where the model producer controls the design of the new tokenizer. By constructing the new tokenizer as a continuation of the source tokenizer’s BPE merges (with existing merges held fixed), the embedding initialization problem reduces to direct row copies for the tokens that carry over and the mean of the source sub-token embeddings for the new tokens. A two-stage adaptation (embedding-only training, then unfrozen continued pre-training), followed by the mid-training and post-training phases, preserves source-checkpoint quality on the evaluated suite and substantially reduces token counts on under-tokenized languages. Per-language results show that performance and efficiency on previously supported languages are preserved, while the under-tokenized languages see large tokenizer-efficiency improvements (Section 7), together with multilingual quality gains on the evaluated languages from the Stage 2 data (Section 6.2).

We used the recipe to help produce LFM2.5-8B-A1B, an updated version of the LFM2-8B-A1B on-device MoE model with a 128K tokenizer. The model is released with open weights. On languages that the source tokenizer fragmented heavily (for example Thai, Hindi, and Vietnamese), the expanded tokenizer needs fewer tokens for the same text, which our synthesis estimates as a substantial per-character decode speedup once the larger vocabulary’s per-token cost is accounted for (Section 7). We document two negative findings that shaped the recipe: training the full embedding matrix in Stage 1 degrades performance, and the Stage 2 data mixture must be balanced multilingually to avoid regression at the unfreezing step.

The recipe is narrow by design, suited to the setting where a model producer wishes to extend coverage of an existing checkpoint to additional languages or scripts without restarting training. For that setting, this tokenizer expansion recipe offers a practical reference point alongside the broader literature on cross-lingual vocabulary adaptation and zero-shot tokenizer transfer.

9.1 Limitations and Scope↩︎

Several constraints bound where the recipe applies and what it costs.

9.1.0.1 Restricted to continued-BPE constructions.

The principal restriction is that the new tokenizer must be obtained by continuing the source tokenizer’s BPE merges on a new corpus (rather than being trained independently). This requires the ability to retrain BPE on a new corpus while keeping the existing merge table fixed, which in turn requires access to the original tokenizer’s merge rules and special-token configuration. The recipe does not apply when the new tokenizer is an off-the-shelf tokenizer designed by a third party. For that setting, the zero-shot transfer methods of [9], [30] are the appropriate tool.

9.1.0.2 Single model family.

We validate the recipe on one model family (LFM2), the setting in which we control the tokenizer and can apply the upgrade end to end. Beyond the continued-BPE requirement above, the recipe is not specific to this architecture: it assumes only a checkpoint whose embeddings can be extended and trained further. We therefore expect it to transfer to other model families whose producers have the same control over the tokenizer.

9.1.0.3 Continued pre-training compute.

Stage 2 is a continued pre-training stage and carries a corresponding compute cost. In our runs, this was a small fraction of the original pre-training budget (LFM2-8B-A1B was pre-trained on 12 T tokens [8], and the LFM2.5 release adds further pre-training to 38 T total [12]), but it is not free. We did not determine the minimum continued pre-training budget needed to recover source-checkpoint quality, so this cost reflects our setup rather than a requirement of the method. The recipe is most attractive when the original pre-training cost is large enough that even a fractional re-investment in continued pre-training is preferable to training from scratch.

9.1.0.4 Decode regression on previously supported languages.

On languages already encoded efficiently by the source tokenizer (English, code, and the major European languages), the larger vocabulary’s per-token cost is not offset by a compression gain and shows up as a per-character decode regression of up to about 9% (Section 7.2, Table 5). The trade-off is favorable for our deployment targets, but practitioners whose workloads are dominated by these languages should weigh it when applying the recipe.

9.2 Availability↩︎

LFM2.5-8B-A1B is released with open weights at huggingface.co/LiquidAI/LFM2.5-8B-A1B under the LFM Open License v1.0 along with the expanded 128K tokenizer and deployment guides for llama.cpp, MLX, vLLM, and SGLang.

Acknowledgments↩︎

We thank the broader Liquid team for the base model and the surrounding infrastructure on which this work depends [8].

10 Evaluation Benchmark Details↩︎

The per-stage recovery aggregate in Section 6.1 (Figure 3, Table 3) is computed over a fixed suite of eight public benchmarks chosen to isolate the tokenizer-expansion contribution rather than downstream mid-training gains. The suite is:

• Knowledge (1 benchmark). MMLU-Pro [42] (we report the answer-ranking variant scored by log-probability over the option set, which is more stable across base and post-trained checkpoints than free-form answer extraction).

• Math (3 benchmarks). GSM8K [43], MATH500 (the 500-problem test subset of MATH [44] selected by [45]), and GSM-Plus [46].

• Code (2 benchmarks). HumanEval+ [47] and LiveCodeBench v5 [48].

• Multilingual (2 benchmarks). MMMLU [49] (the multilingual counterpart of MMLU, scored by log-probability over multiple-choice options, reported on the language subset configured for this evaluation) and MGSM [50] (multilingual GSM8K, generation-style).

We exclude chat-template-dependent benchmarks (IFEval [51], Multi-IF [52], and other tool-use and instruction-following benchmarks) from the Section 6.1 aggregate, for the reason given there: the source pre-training checkpoint has not been trained for chat-template adherence or tool-use formatting, so these benchmarks would understate it. The released LFM2.5-8B-A1B incorporates the corresponding capabilities through post-training (Section 6).

6pt

6pt

11 Ablation Details↩︎

This appendix gives the per-language breakdown behind the Stage 2 data-mixture ablation (Section 8, Table 7).

The per-language breakdown of MGSM (Table 10) sharpens the diagnosis. The drop relative to the good mixture is not uniform across languages. The largest are on Latin-script European languages: French falls from 71.6 to 10.4 (\(-\)​61 points), German from 66.4 to 18.0 (\(-\)​48), and Spanish from 56.4 to 12.4 (\(-\)​44). CJK and Cyrillic languages hold up substantially better: Japanese drops only 7 points, Russian 14, and Chinese 22. We read this as further evidence that the failure mode under the bad mixture is generation behavior rather than lost capability. The observed pattern is consistent with, though does not establish, a script-mediated mechanism: when the model is steered toward English-style output by the predominantly English mixture, languages whose Latin script and shared vocabulary make the boundary with English porous may be more easily shifted toward English-like output, while script-distinct languages (CJK, Cyrillic) retain their language-specific generation behavior more readily. A controlled study with more languages per script group, statistical testing, and a defined linguistic-distance metric would be needed to make this stronger than an observation.

6pt

12 Tokenizer Construction Implementation Notes↩︎

The expanded tokenizer is constructed by initializing the BPE merge table with the source tokenizer’s merges and continuing the merge training on the multilingual corpus. The special-token configuration (chat template special tokens, function-call delimiters, BOS/EOS) is preserved unchanged from the source tokenizer. Byte-level pre-tokenization rules are inherited. The vocabulary is capped at 128K and trimmed to the most frequent merges up to that limit. The target vocabulary holds 125,017 active tokens (Table 1), with the remaining slots reserved for future expansion.

When the source tokenizer is implemented in tiktoken or another non-SentencePiece BPE framework, the merge-extension procedure differs in implementation details from the SentencePiece case. We refer the reader to [13] for a careful discussion of the implementation differences between byte-level and SentencePiece-based BPE.

12.0.0.1 Comparison to other production tokenizers.

Table 11 compares the expanded 128K tokenizer to four widely used production tokenizers on a fixed multilingual evaluation corpus. Two observations frame the rest of the table. First, on the total corpus the expanded tokenizer produces the lowest token count of any tokenizer measured (11.3M tokens), despite having a smaller vocabulary than all four (gpt-4o 200K, Mistral Small 3.1 131K, Qwen3 151K, gemma-3 262K). Second, on the under-tokenized languages the expanded tokenizer is competitive with all peers and far better than the source, though it does not displace tokenizers that simply allocate more vocabulary slots to those scripts. Gemma-3 (262K vocabulary, more than 2\(\times\) the expanded tokenizer’s) wins on Hindi (by roughly 2\(\times\)), Bengali (by roughly 2.3\(\times\)), Thai, Indonesian, Vietnamese, and Japanese, an unsurprising consequence of having twice the vocabulary budget to spend. With only 128K entries, the expanded tokenizer still produces the best total compression on a multilingually balanced corpus, wins on Arabic, code, and JSON, and comes close to the best peers on the remaining languages, all while remaining a far smaller embedding matrix to serve on-device.

4pt

13 On-device Latency Details↩︎

Table 12 reports the full per-device throughput and disk-size numbers behind Figure 4, comparing the 65K and 128K embedding/LM-head matrix on an otherwise-identical LFM2-8B-A1B body.

6pt

References↩︎