1 Introduction↩︎

Autoregressive decoding from LLMs requires \(K\) serial forward passes (each reloading the model weights into memory) to generate a sequence of \(K\) tokens. Speculative decoding is a popular technique to accelerate inference by drafting a sequence of tokens with a lightweight draft model and verifying the drafted tokens in parallel [1], [2], which can drastically reduce the total number of forward passes with the target model (also known as the verifier).

However, the effectiveness of speculative decoding depends on the similarity between the verifier model and draft model’s next-token probability distributions, often measured via the acceptance rate: how often a drafted token is accepted by the verifier model. A poor acceptance rate results in a minimal speed-up, or even worse generation speed if the cost of drafting tokens outweighs the benefit.

This raises a concern for languages other than English. It is well known that model capacity is correlated with multilingual capabilities [3], [6], suggesting that smaller draft models may differ significantly from the verifier model when generating text in a less common language. [4], [5] provide empirical support for this claim, observing significantly worse acceptance rates for non-English languages. The issue is exacerbated by tokenization biases, where less common languages can require far more tokens on average [7]. As a result, non-English users of LLMs may suffer far slower generation speed, as yet another instance of disparities in language technologies [8].

We test a standard speculative decoding setup with multilingual verifier and draft models (Qwen 3.5) on translation from English into eleven different languages, observing poor acceptance rates. We compare three different approaches for training draft models for specific languages: (1) task-specific distillation, (2) distillation on general domain monolingual corpora, and (3) n-gram modeling. We test whether these approaches generalize to a new domain—story generation—without specifically training for that domain. We find resounding evidence in favor of n-gram models, which achieve large speed-ups due to inexpensive forward passes. Meanwhile, distillation is effective in-domain, but generalizes to new tasks poorly. We will release our code on GitHub (anonymized).

2 Background and Related Work↩︎

[1], [2], [9] proposed speculative decoding as a technique for accelerating generation. Specifically, the draft model autoregressively generates \(\gamma\) new tokens with conditional probabilities \(p(x_1) \dots p(x_\gamma)\), and the verifier model runs a single forward pass over the full drafted sequence, assigning distinct probabilities \(q(x_1) \dots q(x_\gamma)\). For each token \(i\), if \(p_i\leq q_i\), the token is accepted. If \(p_i > q_i\), we randomly choose between: a) accepting the token with probability \(\frac{q_i}{p_i}\); or b) rejecting the token and all subsequent draft tokens and then sampling a new token from the adjusted distribution \(p'=q(x)-p(x)\). This results in a sampling distribution equivalent to the verifier model’s distribution.

To improve the probability of accepting a draft token, a popular technique is performing knowledge distillation from the verifier model to the draft model [10], where the draft model is finetuned to minimize its KL divergence with the teacher’s distribution. [4] applies this technique to a multilingual setting, distilling models for translation from another language into English, but they do not study the more difficult case of generating text in the target language.¹

Our work is most similar to [5], which observes acceptance rate disparities across tasks and languages. They propose methods to mitigate these issues by balancing the training dataset or scaling per-task gradients during distillation. However, their approach requires a representative dataset for each of the desired tasks, which is infeasible to acquire for many languages. Instead, we focus on testing whether methods can generalize to new tasks without task-specific training data.

3 Methodology↩︎

We evaluate on two tasks: machine translation (MT) and open-ended story generation. MT serves as our in-domain task, as we perform distillation (see §3.3) on task-specific translation examples. Story generation serves as an out-of-domain task (where the draft model is not trained on story generation data) in order to measure the robustness of our methods to different types of tasks.

3.1 Datasets↩︎

For machine translation, we use parallel data from a variety of sources—limited to 5,200 examples per language—with dataset counts described in 1 and sources in 8. Our main evaluation for speculative decoding efficiency uses the test split only, and the train split is reserved for distillation. We also collect monolingual, unlabeled data for distillation and training, also in 1. We use datasets for eleven different languages, covering a range of resourcedness, typological properties, and geographic regions.

For story generation, we create topics by pairing nouns from [11] and adjectives from the Brown Corpus [12] by similarity according to Glove word embeddings [13]. During evaluation we prompt the model to generate a story in the target language about each topic. A few examples of generated stories appear in 11. We do not compute any metrics over these stories, but we do verify that they are consistently generated in the correct language and do not get stuck in endless repetition loops. As generated stories tend to be longer than translations, we only use 200 test examples.

3.2 Models↩︎

We use the Qwen 3.5 family of models [14]. Our main results are presented using the 9b parameter model as the verifier and the 0.8b parameter model as the draft model; however, we include results with 2b and 4b parameter draft models in §9.3. We use sampling-based inference using a top-k of 100, top-p of 0.9, and a maximum 128 generated tokens. We use KV caching for both the draft and verifier models [15]. For each setting, we sweep the \(\gamma\) value over \([2,4]\) and report the best result.

3.3 Experimental Settings↩︎

Our baseline setting is an unmodified smaller model of the same family (0.8b Qwen). We then perform soft-target distillation (details in §7) from the verifier model to the 0.8b model on two types of corpora. First, for our distilled (task) setting, we use parallel data for distillation on translation prompts. Second, we distill on monolingual, general domain text for the distilled (general) setting.

Finally, we train simple n-gram draft models on the same monolingual corpora, using the Qwen tokenizer to segment tokens. We test various \(n\) values and find that bigrams (2-grams) produce the highest acceptance rates for all languages. During inference, we compute a simple logit distribution for an \((n-1)\) length prefix from the conditional distribution observed during training.

3.4 Metrics↩︎

3.4.0.1 Acceptance Rate (\(\alpha\)).

Following [1], Definition 3.1, we measure the probability that a draft token is accepted by the target model, given all prior tokens have been confirmed. We estimate this probability for a single example \(i\) with the Monte Carlo estimator \(\hat{\alpha_i} = \frac{m_i}{d_i}\) where \(m_i\) is the number of accepted tokens and \(d_i\) is the number proposed, and we take the mean over examples.

3.4.0.2 Speed-up Factor (\(f\)).

Using \(\alpha\), we compute the speed-up factor, which quantifies the expected improvement under speculative decoding: \[f = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}\] where \(c = t_{\text{draft}} / t_{\text{target}}\) is the cost ratio of a single draft forward pass to a single target forward pass (measured via CUDA timing). Speed-up values are reported as multiplicative factors (e.g., \(1.2\times\)).

4 Findings↩︎

4.1 Translation↩︎

3 reports the average acceptance rates and speed-up factors per setting for the translation task.

4.1.0.1 Standard speculative decoding is ineffective due to low acceptance rates.

In the baseline setting using unspecialized draft models, acceptance rates are fairly low, ranging from \(0.30\) (ber) to \(0.54\) (amh) with an average over languages of \(\bar{\alpha} =0.40\). This results in most languages having speed-up factors close to 1 (\(\bar{f} = 1.02\times\)), in which case speculative decoding is not beneficial. We find no correlation between translation quality (chrF score) and acceptance rates, discussed in §[sec:chrf95acceptance].

4.1.0.2 Task-specific distillation improves acceptance rates, while general domain distillation is less effective.

As expected, distillation on translation prompts clearly improves acceptance rates (\(\bar{\alpha} = 0.60\)) and therefore speed-up factors (\(\bar{f} = 1.28 \times\)). Meanwhile, the general domain distillation setting (\(\bar{\alpha} = 0.39, \bar{f} =1.03\times\)) only beats the baseline in five of eleven languages. Thus, distillation is an effective option only for specific, predetermined domains. In this case, the distillation loss provides a lower bound on acceptance rate (see §9.5).

4.2 Story Generation↩︎

4 and 6 report speed-up factors and acceptance rates, respectively, for the story generation prompts. We see similar trends as the prior setting for the baseline (\(\bar{\alpha} = 0.46, \bar{f} =1.09\times\)), n-gram model (\(\bar{\alpha} = 0.30, \bar{f} =1.39\times\)), and general domain distilled model (\(\bar{\alpha} = 0.47, \bar{f} =1.10\times\)).

4.2.0.1 Task-specific distillation does not generalize to new domains well.

The task-specific distilled models show a large deterioration on the new task (\(\bar{\alpha} = 0.43, \bar{f} =1.03\times\)), underperforming the baseline and the general distillation approach. 5 shows this trade-off: distillation improves acceptance on the training task at the expense of the other, unseen task.

4.2.0.2 N-gram models are highly effective due to favorable draft-to-verifier cost ratios.

While the n-gram models tend to have inferior acceptance rates (\(\bar{\alpha}=0.24\)), they are much faster to run, with a mean forward pass of \(0.001\)s compared to \(0.033\)s for the 0.8b draft model (2). This results in highly competitive speed-up factors (\(\bar{f}=1.30\times\)), beating the baseline on all but one language (oci), and suggesting that n-gram models are a viable option if a monolingual training corpus is available.

5 Conclusion↩︎

Our results confirm that standard speculative decoding may not be useful for non-English languages, and can even be detrimental. We identify distillation as an effective way to improve per-language performance, but improvements are limited to the training domain and fail to generalize to new tasks. Meanwhile, we demonstrate that simple n-gram models can provide superior speed-up factors (across domains) thanks to the low cost of inference. In real-world usage, then, it would be beneficial to dynamically switch the draft model based on the language being generated, and in many cases an n-gram model may be the best choice.

Limitations↩︎

This work studied a single family of models (Qwen 3.5), using the 9b model as a verifier and primarily using the 0.8b model as the draft. Absolute results may vary across other size combinations and model families, but we expect the general trends to hold. Our work also studies a small set of eleven languages, and results may differ for languages dissimilar to those eleven. Finally, our work studies two realistic tasks in a rare language, and our results may not hold for tasks like mathematical reasoning (as in [5]).

Ethical Considerations↩︎

This work suggests approaches to mitigate user experience disparities for disadvantaged languages. However, we recognize that work done without the involvement of language stakeholders risks poor alignment with the actual needs of speakers. In particular, we are aware that the generations in our evaluation suites were very poor quality, largely unusable for the desired tasks. This was not a primary concern as our main focus was generation speed, but there remains a large gap in usability for many languages. Our work used data sourced from existing open access datasets, and we defer to their ethical policies regarding data collection and processing. Finally, we recognize the severe environmental cost of training large-scale models, and have striven to use our resources efficiently. We used AI assistants for code review and to help modify plotting code.

6 Implementation↩︎

We use the Hugging Face libraries for model implementations and tokenizers, Weights and Biases for tracking, Sacrebleu for evaluation, and NLTK for story generation.

7 Distillation Details↩︎

We train draft models via distillation using the truncated cross-entropy loss on the top twenty logits of the teacher. We use the train splits from each dataset, and we select 5% of these examples to use as a validation split. For translation, we use greedy decoding to generate translations using the teacher model, and perform soft-target distillation on the generated outputs using the top 20 logits per-token [16]. We ignore the tail of the logit distribution, which is known to incur bias [17]. For the general domain distillation, we do not generate outputs, but rather compute teacher logits on the raw texts. All languages and models use the hyperparameters in 2 and bfloat16 precision, and we sweep the learning rate over \(\{1e-4, 5e-5, 2e-5\}\) and select the best model by validation loss. For the general domain distillation approach, we truncate sequences to 128 tokens. For translation, we allow up to 128 generated tokens (excluding the prompt).

8 Dataset Sources↩︎

We list the various data sources in 3, as well as the per-language counts in 4 and 5.

9 Additional Results↩︎

9.1 Story Generation Acceptance Rates↩︎

Acceptance rates for story generation are provided in 6, corresponding to the speed-up factors in 4.

9.2 Throughput↩︎

We report throughput (tokens/second) metrics for both tasks in 7. Raw decoding throughput is an implementation-agnostic measure of generation speed. We measure this as total generated tokens over total decode wall clock time, excluding the prefill:

\[\text{TPS} = \frac{\sum_{i=1}^{N} g_i}{\sum_{i=1}^{N} t_i}\]

where \(g_i\) is the number of output tokens for sentence \(i\) and \(t_i\) is its decode time.

9.3 Draft Model Size↩︎

In addition to the 0.8b parameter draft model, we also test the baseline setting (no distillation) with the 2b and 4b parameter Qwen 3.5 models. From the 0.8b results, we select the language with the highest acceptance rate (amh), the lowest acceptance rate (ber), and a random language from the middle (grn). We report speed-up factors and acceptance rates in 8. We observe that larger draft models improve both the acceptance rate and speed-up factor most of the time, although in one case (amh) the largest model shows a drop in speed-up.

9.4 Translation Quality↩︎

In 9, we visualize the relationship between the model’s translation capability in a language (measured via the chrF score of generated translations) and acceptance rate. We had hypothesized that acceptance rates would be higher for languages the model is stronger on (as a proxy for the prevalence of a given language in the training data). However, we observe a very weak correlation (\(r=0.170\)), failing to provide evidence for our hypothesis.

9.5 Distillation and Acceptance Rate↩︎

When performing distillation on the same domain as the test data, we can compute a lower bound on the acceptance rate from the KL divergence between the teacher and student model. First, we use Theorem 3.5 from [1] to get: \[\alpha = \sum_x \min(p(x), q(x))\] Let \(A = \{x: p(x) \geq q(x)\}\) and \(B = \{x: q(x) > p(x)\}\). Then, we get: \[\alpha = \sum_{x\in A} q(x) + \sum_{x\in B} p(x)\] \[= \sum_{x\in A} p(x) - (p(x) - q(x)) + \sum_{x\in B} p(x)\] \[= \sum_{x\in A} p(x) + \sum_{x\in B} p(x) - \sum_{x \in A} (p(x) - q(x))\] \[= 1 - \sum_{x \in A} (p(x) - q(x)) \label{eq:acc}\tag{1}\] Next, we use the definition of total variation distance \(\delta\): \[\delta(P,Q) = \frac{1}{2}\sum_x \vert p(x)-q(x) \rvert\] Using the same definitions of \(A\) and \(B\), \[\delta(P,Q) = \frac{1}{2} \left( \sum_{x \in A} p(x) - q(x) + \sum_{x \in B} q(x) - p(x) \right)\] We show these two sums are equal: \[\sum_{x \in A} p(x) - q(x) - \sum_{x \in B} q(x) - p(x)\] \[= \sum_{x \in A} p(x) - q(x) + \sum_{x \in B} p(x) - q(x)\] \[= \sum_x p(x) - \sum_x q(x) = 0\] \[\therefore \sum_{x \in A} p(x) - q(x) = \sum_{x \in B} q(x) - p(x) = \delta(P,Q)\] Substituting into 1 , we get \[\alpha = 1 - \delta(P,Q)\] By Pinsker’s inequality, we have an upper bound on the total variation distance: \[\delta(P,Q) \leq \sqrt {\frac{1}{2} D_{KL}(P \vert\vert Q)}\] where \(D_{KL}\) is the KL-divergence. Finally, this gives the lower bound on acceptance rate: \[\alpha \geq 1-\sqrt {\frac{1}{2} D_{KL}(P \vert\vert Q)}\]

We validate this bound against the task-specific distillation runs in 10. We observe that all of the models well out-perform the bound, and there is not necessarily the negative correlation we would expect. However, this insight makes it possible to predict a lower bound on acceptance rate solely based on the distillation training run.

10 Generated Story Examples↩︎

References↩︎