1 Introduction↩︎

Large language models (LLMs) have enabled a wide range of powerful applications, including conversational agents [1], [2] and automated programming tools. Despite their success, LLM inference remains dominated by a sequential, token-by-token generation process, where each output depends on the full preceding context. This inherent seriality creates a major performance bottleneck: inference is slow, memory-bound, and fails to fully utilize modern GPUs. With the recent emergence of long Chain-of-Thought (CoT) reasoning models [2], [3], this bottleneck has become increasingly critical, as prolonged inference times now dominate the generation process.

Speculative decoding [4]–[8] has emerged as a primary solution to this bottleneck. This paradigm employs a lightweight draft model to speculate a sequence of future tokens, which are then verified in parallel by the large target model. While this approach achieves lossless acceleration and has been widely integrated into production frameworks, state-of-the-art methods like EAGLE-3 [7] still rely on autoregressive drafting. This serial drafting process is not only inherently inefficient but also susceptible to error accumulation, which effectively caps achievable speedups at approximately 2\(-\)​3\(\times\).

Recently, Diffusion LLMs (dLLMs) [9] offer a promising alternative to autoregressive LLMs by enabling parallel text generation and bidirectional context modeling. Block diffusion models [10]–[12] can denoise a block of masked tokens simultaneously. However, current open-source dLLMs typically underperform their autoregressive counterparts in terms of generation quality. Furthermore, maintaining acceptable output quality often necessitates a high number of denoising steps, which significantly diminishes their raw inference speed [13].

This landscape reveals a critical trade-off: autoregressive models deliver superior performance but suffer from sequential latency, while diffusion models allow for fast, parallel generation but often at the cost of accuracy. A natural research question follows: Can we combine the strengths of both paradigms while mitigating their respective weaknesses? A compelling solution lies in leveraging diffusion models for high-speed, parallel drafting, while relying on high-quality autoregressive models for verification to ensure the final output remains lossless.

However, utilizing diffusion for drafting is non-trivial, and existing methods are either impractical or offer limited speedups. Methods such as DiffuSpec [14] and SpecDiff-2 [15] utilize massive (e.g.., 7B parameter) draft models. This significant memory footprint is often prohibitively expensive for real-world serving. Furthermore, while these large drafters offer relatively high quality draft tokens and acceptance lengths, the high drafting latency limits their practical speedups to a modest 3\(-\)​4\(\times\). In contrast, PARD [16] trains small autoregressive models to mimic diffusion-style parallel generation, and then perform speculative decoding for target LLMs. However, the resulting small models lack the modeling capacity of the target LLMs, leading to limited acceptance lengths and a speedup ceiling of approximately 3\(\times\).

Is there truly “no free lunch”? Can we build a diffusion drafter that is both lightweight and highly accurate?

In this paper, we introduce DFlash, a speculative decoding framework that uses a lightweight block diffusion model to achieve both fast and high-quality drafting. Our key insight is simple: the target knows best. As observed by [17], large autoregressive LLMs’ hidden features implicitly contain information about multiple future tokens. DFlash utilizes these hidden features as context, conditioning the draft model to predict future blocks of tokens in parallel. In effect, the draft model becomes a diffusion adapter that efficiently leverages the deep context features modeled by the large target model. Instead of requiring a tiny draft model to reason from scratch, DFlash fuses the reasoning capabilities of the target model with the parallel generation speed of a small diffusion drafter.

We evaluate DFlash across a wide range of models and benchmarks, and demonstrate its practical benefits under realistic serving setups using SGLang [18]. As shown in 1, DFlash achieves up to a 6.1\(\boldsymbol{\times}\) speedup on Qwen3-8B [1], and is nearly 2.5\(\boldsymbol{\times}\) faster than the state-of-the-art EAGLE-3 across most benchmarks. We believe DFlash represents a significant step forward in accelerating LLM inference and democratizing high-performance AI.

1.0.0.1 Conflict of Interest Disclosure.

The authors declare no financial conflicts of interest related to this work. Research support and compute resources are acknowledged in the Acknowledgements section.

2 Related Work↩︎

2.1 Speculative Decoding↩︎

Speculative decoding accelerates LLM inference by mitigating the sequential bottleneck of autoregressive generation. Early methods [4] employ a smaller draft model to propose token sequences that are verified in parallel by a larger target model. Medusa [8] eliminates the external draft model by augmenting the base LLM with multiple prediction heads and using tree attention for parallel verification. The EAGLE series [5]–[7] further improves speculative decoding by exploiting feature-level context from the frozen target model. EAGLE-1 predicts future hidden-state distributions to boost acceptance rates, EAGLE-2 introduces adaptive drafting trees, and EAGLE-3 refines training objectives to scale speedups.

Despite these advances, most existing methods rely on autoregressive drafting, which remains inherently sequential, limiting their speedups.

2.2 Diffusion Language Models↩︎

Diffusion large language models (dLLMs) offer an alternative to autoregressive generation by predicting masked tokens in parallel. LLaDA [9] was the first to scale dLLMs to billions of parameters, achieving performance comparable to LLaMA-3.1-8B [19]. However, fully parallel diffusion models suffer from fixed-length generation and lack efficient KV cache support. Block diffusion models [10] address these issues by denoising sequences block-by-block, blending parallelism with autoregressive structure. Building on this idea, Fast-dLLM v2 [12] and SDAR [11] adapt pre-trained autoregressive LLMs into block-diffusion variants, enabling parallel generation while preserving generation quality on specific tasks. Nevertheless, existing dLLMs generally underperform state-of-the-art autoregressive models and often require many denoising steps, which limits their practical inference speed.

2.3 Diffusion-based Speculative Decoding↩︎

Recent work explores using diffusion models as drafters within speculative decoding. TiDAR [20] jointly trains diffusion and autoregressive objectives, enabling parallel “thinking” via diffusion and sequential “talking” via autoregressive decoding, though final generation quality is not yet lossless.

Other approaches repurpose autoregressive models for diffusion-style drafting. [17] observe that autoregressive LLMs implicitly encode future-token information and train a LoRA adapter to enable parallel drafting, while retaining the base model for verification.

DiffuSpec [14] and SpecDiff-2 [15] employ large pre-trained dLLMs as speculative drafters, with inference-time search or train–test alignment to improve acceptance. However, these approaches rely on massive drafters (e.g.., 7B parameters), incurring substantial memory and latency overhead. While they achieve long acceptance lengths, the high drafting cost often offsets the practical speedups in real-world serving scenarios.

3 Preliminaries↩︎

This section formalizes the speedup mechanism of speculative decoding and clarifies the efficiency trade-offs between autoregressive and diffusion-based drafting. Our analysis highlights why diffusion drafters are uniquely positioned to achieve both low drafting latency and high acceptance rates.

3.1 Speculative Decoding Speedup↩︎

Speculative decoding accelerates inference of a target model \(\mathcal{M}_t\) using a smaller draft model \(\mathcal{M}_d\). In each decoding cycle, the draft model proposes \(\gamma\) tokens, which are verified in parallel by the target model.

Following [21], the average per-token latency is \[L = \frac{T_{\text{draft}} + T_{\text{verify}}}{\tau},\] where \(T_{\text{draft}}\) is the time spent generating draft tokens, \(T_{\text{verify}}\) is the cost of verification, and \(\tau \in [1, \gamma + 1]\) is the expected number of accepted tokens per cycle, including the bonus token produced by the target model. Let \(L_{\text{target}}\) denote the autoregressive per-token latency of \(\mathcal{M}_t\); the resulting speedup is \(\eta = L_{\text{target}} / L\).

This expression makes the trade-off explicit: speedup improves either by increasing the expected acceptance length \(\tau\) or by reducing the drafting overhead \(T_{\text{draft}}\).

3.2 Autoregressive vs.. Diffusion Drafting↩︎

Autoregressive drafters generate tokens sequentially, incurring a drafting cost \[T_{\text{draft}} = \gamma \cdot t_{\text{step}},\] where \(t_{\text{step}}\) is the latency of a single forward pass. Drafting costs therefore grow linearly with the speculation budget \(\gamma\).

To keep latency manageable, autoregressive drafters are constrained to very shallow architectures (e.g.., a single transformer layer in EAGLE-3). This severely limits the draft quality: while increasing \(\gamma\) increases drafting cost, acceptance length \(\tau\) quickly saturates due to limited model capacity. In practice, this imbalance restricts achievable speedups.

Diffusion drafters generate all \(\gamma\) tokens in parallel within a single forward pass, yielding \[T_{\text{draft}} = t_{\text{parallel}},\] where \(t_{\text{parallel}}\) denotes the latency of block generation. Modern GPUs execute such parallel operations far more efficiently than multiple sequential passes, making \(t_{\text{parallel}} \ll \gamma \cdot t_{\text{step}}\) for models of comparable size. For moderate block sizes, \(T_{\text{draft}}\) is therefore largely insensitive to \(\gamma\).

This parallelism fundamentally changes the design space. Because drafting cost no longer scales with the number of generated tokens, diffusion drafters can afford deeper, more expressive architectures without sacrificing latency. This increased capacity substantially improves draft quality and acceptance length. Empirically, a five-layer DFlash draft model generating 16 tokens achieves both lower latency (3) and higher acceptance length than EAGLE-3 generating 8 tokens, placing DFlash on a more favorable Pareto frontier between draft quality and drafting cost.

4 Method↩︎

4.1 Inference↩︎

The system design of DFlash is illustrated in 2. In this section, we explain the key design choices that allow DFlash to achieve high draft acceptance length using a very small and efficient draft model.

Context features from the target model. Prior work like [16] naively applied a small diffusion model as a speculative drafter, which leads to poor acceptance length and limited speedups. To validate this, we train a five-layer block diffusion draft model without any conditioning from the target model and evaluate it on several math benchmarks. As the results shown in the 7, the resulting speedups are modest, typically around \(2\)–\(3\times\). This limitation stems from the lack of rich contextual guidance: without access to the internal representations of the target model, the diffusion drafter must effectively predict future tokens from scratch.

In contrast, the hidden representations of large autoregressive target models encode substantially more information than token-level logits. These features capture long-range dependencies and task-specific semantics, and—crucially—implicitly encode information about future token predictions, as also observed by [17].

In DFlash, given an input prompt, the target model first performs a standard prefill pass to generate the first token. During this pass, we extract hidden representations from a fixed set of layers uniformly sampled from shallow to deep. These hidden states are concatenated and passed through a lightweight projection layer to fuse cross-layer information into a compact target context feature, which is then used to condition the draft model.

Conditioning via KV injection enables acceptance scaling. Existing methods such as EAGLE-3 also leverage hidden features from the target model, but they fuse these features with the draft model’s token embeddings and feed them only as inputs to the draft model. As the draft model depth increases, the information from target model becomes more and more diluted, resulting in diminishing gains in acceptance length when adding more draft layers.

DFlash adopts a fundamentally different strategy. We treat the fused target context feature as persistent contextual information and directly inject it into the Key and Value projections of every draft model layer. The projected features are stored in the draft model’s KV cache and reused across drafting iterations. We provide more details about the KV injection mechanism in 7.3. This design provides strong and consistent conditioning throughout the draft model, enabling acceptance length to scale effectively with the number of draft layers. We analyze this behavior in more detail in 5.5.2.

Parallel diffusion drafting. Another key contributor to DFlash’s speed is its low drafting latency. Autoregressive draft models must perform multiple sequential forward passes to generate draft tokens or trees, which limits parallelism and leads to inefficient GPU utilization. In contrast, DFlash predicts the next token block using a block-level diffusion process. All masked positions within a block are decoded in parallel in a single forward pass. Compared to autoregressive drafting, this block-wise parallel generation substantially reduces drafting latency and achieves significantly higher hardware utilization, even when using deeper draft models.

Overall, DFlash combines diffusion-based parallel drafting with tightly coupled conditioning from the target model, enabling high-quality drafting with substantially reduced drafting latency.

4.2 Training↩︎

DFlash draft models are trained to align block-level diffusion predictions with the outputs of a frozen autoregressive target model. Rather than directly adopting standard block diffusion training [10], we introduce several key modifications that improve training efficiency, scalability, and alignment with the inference-time speculative decoding behavior.

KV injection. Following the inference pipeline, given a sequence consisting of a prompt and its response, we first pass the entire clean sequence through the target model to extract and fuse the hidden features for all tokens. The hidden features are then injected into the draft model as Key and Value projections, as illustrated in 4.

Random sampling of masked blocks. In standard block diffusion training, the response is uniformly divided into blocks and random positions within each block are masked, with the model trained to denoise the masked tokens.

DFlash instead tailors block construction to the speculative decoding setting. We randomly sample anchor tokens from the response, use each anchor as the first position of a block, and mask the remaining positions. The draft model is trained to predict the next \(\text{block\_size} - 1\) tokens in parallel. This directly matches inference-time behavior, where the draft model always conditions on a clean token produced by the target model (i.e.., the bonus token from the previous verification step). Randomizing anchor positions also exposes the draft model to more diverse target context features, improving data efficiency and coverage. As shown in 10, this strategy substantially improves both acceptance length and speedup.

During training, all blocks are concatenated into a single sequence and processed jointly using a sparse attention mask as shown in 4. Tokens attend bidirectionally within the same block and to the corresponding injected target context features, while attention across different blocks is disallowed. This design enables multiple draft blocks to be trained efficiently within a single forward and backward pass using Flex Attention [22].

Efficient long-context training. Training speculative draft models on long contexts is challenging for methods such as EAGLE-3 due to their costly training-time test. DFlash achieves efficient long-context training by fixing the number of masked blocks per sequence and randomly sampling anchor positions for each sequence at every epoch. This strategy provides effective data augmentation while keeping training cost bounded.

Loss weighting for faster convergence. In speculative decoding, not all tokens are equal. Errors at early positions within a draft block invalidate all subsequent tokens. This makes early predictions disproportionately important for acceptance length. We reflect this asymmetry by weighting the cross-entropy loss to emphasize earlier token positions during training.

Specifically, for a token at position \(k\) within a block, we apply an exponentially decaying weight \[\label{eq:loss-decay} w_k = \exp\!\left(-\frac{k-1}{\gamma}\right),\tag{1}\] where \(\gamma\) controls the decay rate. This weighting prioritizes early positions, accelerating convergence and yielding a higher acceptance length than uniform weighting (5).

Shared embedding and LM head. To improve training efficiency, the draft model shares the token embedding layer and language modeling head with the target model and keeps them frozen during training. Only the draft Transformer layers are updated. This design reduces the number of trainable parameters and encourages the draft model to function as a lightweight diffusion adapter tightly aligned with the target model’s representation space.

5 Experiments↩︎

Models and Evaluations. We conduct experiments on LLaMA-3.1 Instruct (8B) and Qwen3 (4B, 8B, Coder-30B-A3B-Instruct) pre-trained models. We evaluate tasks in three categories: Math: GSM8K [23], MATH [24], and AIME25 [25]; Code: HumanEval [26], MBPP [27], and LiveCodeBench [28]; Chat: MT-Bench [29] and Alpaca [30]. For each task, we assess the performance of the draft models using average acceptance length (\(\tau\)) and end-to-end decoding speedup over the autoregressive baseline. We conduct all experiments on NVIDIA H200 GPUs unless otherwise specified.

Datasets. To provide a diverse set of training data, we collect a mixture of around 800K samples from NVIDIA Nemotron Post-Training Dataset V2 [31] and CodeAlpaca [32]. Instead of directly using the original dataset, we construct our training set with the responses generated by the target model for better target alignment.

Implementation. For DFlash draft models, we set the number of layers to 5 (8 for Qwen3 Coder) and use a block size of 16 (10 for LLaMA 3.1). The target hidden features are extracted from 5 layers uniformly selected between the second layer and the third-to-last layer of the target model. More training details are presented in 7.1.

Baselines. We compare DFlash with the vanilla autoregressive decoding (baseline) and state-of-the-art speculative decoding method EAGLE-3 [7]. We did not include comparisons with other dLLM-based speculative decoding methods [14], [15], [17], [20] due to lack of open-source implementation. For comparisons with EAGLE-3 on Qwen3 models (5.1), we use the checkpoints released by AngelSlim [33]; for LLaMA-3.1-Instruct (5.5.1), we use the official checkpoint released by EAGLE-3 team.

5.1 Instruct Models↩︎

In this section, we evaluate DFlash against EAGLE-3 on Qwen3 models with thinking mode disabled, using the Transformers backend. For EAGLE-3, we consider two settings: a tree size of 16, which matches DFlash with block size 16 for a fair drafting-budget comparison, and a tree size of 60, as used in the EAGLE-3 paper to maximize acceptance length with higher verification cost. In both cases, the draft steps and top-\(k\) are set to 7 and 10, respectively.

As shown in ¿tbl:tab:main-results?, DFlash consistently outperforms EAGLE-3 across all tasks and settings. Under greedy decoding (temperature = 0), DFlash achieves an average speedup of 4.9\(\times\) over the autoregressive baseline, corresponding to a 2.4\(\times\) improvement over EAGLE-3 (16). Under non-greedy sampling (temperature = 1), DFlash maintains a 4.1\(\times\) speedup over baseline and a 2.2\(\times\) improvement over EAGLE-3. Notably, DFlash also surpasses EAGLE-3 with tree size 60, achieving higher acceptance length while incurring substantially lower verification overhead. These results demonstrate the effectiveness and efficiency of diffusion-based drafting in DFlash.

5.2 Reasoning Models↩︎

In this section, we evaluate DFlash for Qwen3 models with thinking mode enabled using Transformers. The draft models are trained on target-model outputs with reasoning traces.

As shown in 1, DFlash maintains the high acceptance length, achieving speedups of roughly 4.5\(\times\) and 3.9\(\times\) over the baseline. This efficiency gain is particularly valuable for the practical deployment of reasoning models, given their prolonged generation time.

5.3 Performance on Serving Frameworks↩︎

In this section, we evaluate the performance of DFlash on the popular open-source inference framework SGLang using Qwen3-4B, Qwen3-8B, and Qwen3-Coder-30B-A3B-Instruct. All experiments are conducted on a single B200 GPU with the FlashAttention-4 (FA4) backend. We enable Spec-v2 scheduling overlap to maximize achievable throughput.

As shown in ¿tbl:tab:sglang-all?, DFlash consistently provides speedups across all three models over concurrency levels ranging from 1 to 32, achieving up to a 5.1\(\times\) speedup on Qwen3-8B. These results demonstrate the practical value of DFlash in real-world serving scenarios, where it can substantially reduce serving cost.

We report additional DFlash speedup results for more models and for vLLM [34] in 7.4.

5.4 Long Context Adaptation↩︎

In this section, we show that the base DFLash draft models trained on 4K context can adapt to longer context with minimal fine-tuning. We fine-tune the base Qwen3.5-27B draft model with 1.6K samples from LongAlign-10K [35] for 3 epochs and test the performance on several datasets from LongBench [36].

As shown in 2, the base DFlash draft model degrades as context length grows beyond 4K, while the fine-tuned model maintains or even improves the acceptance length. This demonstrates that the extracted target features remain representative at long contexts and the draft model can learn the longer-range patterns with lightweight adaptation.

5.5 Ablation Study↩︎

In this section, we ablate the impact of training data and several key design choices of the DFlash draft model. Unless otherwise specified, all ablation models are trained on 100K samples randomly drawn from the full data mixture. All experiments are conducted on a single H200 GPU with greedy decoding, except those evaluated on SGLang.

5.5.1 Training Data↩︎

We compare DFlash against EAGLE-3 on LLaMA-3.1-8B-Instruct. DFlash is trained on UltraChat [37] and ShareGPT, using the exactly same training data as EAGLE-3, and is evaluated against the official EAGLE-3 checkpoints. The DFlash draft model uses a block size of 10, with other configurations matching those of the DFlash Qwen3-8B draft model. All experiments are conducted using SGLang with Spec-v1 (without scheduling overlap), as Spec-v2 does not support tree-based drafting for EAGLE-3. Evaluations are performed on a single B200 GPU.

As shown in 3, DFlash consistently outperforms EAGLE-3 across all tasks, concurrency levels, and EAGLE-3 tree-size configurations. This performance gap holds for math, code, and chat benchmarks, demonstrating the robustness and efficiency advantages of DFlash over autoregressive tree-based speculative decoding.

5.5.2 Number of Draft Layers↩︎

3pt

One advantage of DFlash is that acceptance length scales effectively with the depth of the draft model. However, this comes with a trade-off between drafting cost and draft quality. Deeper draft models are more expressive and achieve higher acceptance lengths, but they also incur higher drafting latency. As a result, the optimal number of layers depends on the deployment setting. As shown in 4, while the 8-layer draft model achieves longer acceptance lengths, the 5-layer model attains higher overall speedup due to a better balance between drafting cost and quality.

5.5.3 Number of Target Hidden Features↩︎

3pt

The number of target hidden features affects both acceptance length and end-to-end speedup. Extracting features from more target layers provides richer semantic and future-token information, improving draft quality. As shown in 5, conditioning on five hidden features consistently outperforms using three. However, this benefit comes at higher training cost: in offline training, the storage required to cache target hidden states increases linearly with the number of extracted features.

5.5.4 Training-Inference Time Block Size↩︎

3pt

Block size is a critical design choice for the DFlash draft model. An equally important question is whether a pretrained DFlash model can generalize from its training-time block size to different block sizes during inference. To study this, we train two draft models with block sizes 8 and 16 on the same data and evaluate their inference-time scaling behavior, as shown in ¿tbl:tab:ablation95block95size?.

When training and inference block sizes match (8\(\rightarrow\)​8 and 16\(\rightarrow\)​16), the block-size-16 model achieves substantially higher acceptance lengths on math and coding tasks. Acceptance histograms on Math500 reveal that the block-8 model frequently fully accepts entire blocks (35.7%), suggesting that block size 8 is often underutilized. In contrast, the block-16 model exhibits a more spread-out acceptance distribution with higher average acceptance length, indicating more effective use of larger blocks.

We further examine cross-block-size generalization at inference time and observe a clear asymmetry. A model trained with a larger block size generalizes well to smaller inference-time block sizes: using block size 8 with a model trained at block size 16 yields acceptance lengths close to those of a model trained and evaluated at block size 8. However, the reverse does not hold.

Overall, DFlash models trained with larger block sizes generalize well to smaller inference-time block sizes. This property enables dynamic block-size scheduling during inference to improve end-to-end efficiency. In practical serving scenarios, large blocks can increase verification cost under compute-bound settings (e.g.., large batch sizes); reducing the block size in such cases can therefore yield better overall speedup. We leave adaptive block-size scheduling to future work.

5.5.5 KV Injection vs. Input Fusion↩︎

This ablation studies whether target features should be injected only once at the input layer, as in EAGLE-3 style input fusion, or injected into every draft layer as KV entries. We compare these two conditioning strategies under both autoregressive drafting and block-diffusion drafting. Results are shown in 6.

3pt

The results show that KV injection is more effective than input fusion. In autoregressive drafting, DFlash-AR achieves higher acceptance length than EAGLE-3-5L on all tasks. In block-diffusion drafting, DFlash with KV injection also improves acceptance length over DFlash with input fusion on all tasks. This suggests that exposing every draft layer to target features through KV entries is more effective than injecting target features only at the input layer.

DFlash achieves acceptance length comparable to EAGLE-3-5L, but obtains much higher speedup because block diffusion drafts multiple tokens in parallel. Therefore, DFlash benefits from both stronger conditioning through KV injection and faster parallel drafting.

6 Conclusion↩︎

In this paper, we present DFlash, a diffusion-based speculative decoding framework that rethinks the role of diffusion language models in accelerating autoregressive LLM inference. By confining diffusion models to the drafting stage, DFlash exploits their inherent parallelism while avoiding the quality degradation that has limited their standalone use. Conditioning the diffusion drafter on rich target-model context enables high acceptance rates, allowing DFlash to significantly push inference speed beyond prior speculative decoding methods.

Beyond empirical improvements, DFlash suggests a new development paradigm for diffusion LLMs. Rather than competing with autoregressive models in end-to-end generation quality, diffusion models can serve as lightweight, specialized drafters optimized for fast and accurate block prediction. This reframing permits aggressive reduction in denoising steps to maximize parallelism, while speculative verification provides a principled guarantee of output quality. We hope DFlash establishes diffusion-based drafting as a practical and scalable paradigm for speculative decoding, advancing more efficient and accessible LLM deployment.

Acknowledgements↩︎

The authors would like to express their sincere gratitude to David Wang for leading the fast and high-quality SGLang integration for DFlash, and to Richard Gong and other members of the Modal team for their strong engineering support. Their efforts were truly instrumental in enabling the practical, production-grade deployment of DFlash.

We thank Qualcomm and Amazon for their support of this research. We also acknowledge Modal, Yotta Labs, Eigen AI, and InnoMatrix for providing the compute resources that made this work possible.

Impact Statement↩︎

This paper presents work whose goal is to advance the efficiency of LLM inference through improved speculative decoding. The proposed method is primarily a system- and algorithm-level optimization that reduces LLM inference and serving costs, without altering model capabilities or intended use cases. We do not foresee significant new ethical risks beyond those already associated with large language models in general. Potential societal impacts are therefore consistent with existing deployments of LLMs, including both their benefits and known limitations.

7 Appendix↩︎

7.1 Training Implementation↩︎

The draft models are optimized for 6 epochs using AdamW with a learning rate of \(6 \times 10^{-4}\), a gradient clipping threshold of 1.0, and a cosine schedule with a warmup ratio of 0.04. We train on our training data mixture with a maximum sequence length of 3072 tokens (4096 for Qwen3-Coder); for each sequence, 512 anchor positions are randomly sampled. The hyperparameter \(\gamma\) for the loss decay in 1 is set to 7 for block size 16, 5 for block size 10, and 4 for block size 8 models.

Training can be performed either online or offline. In online training, target hidden features are computed on the fly during each training step. In offline training, target hidden features are precomputed and cached, then loaded during draft model optimization to reduce computational overhead.

7.2 Diffusion Drafter without Target Feature↩︎

4pt

7.3 KV Injection Mechanism and Memory Overhead↩︎

DFlash uses KV injection to condition the diffusion drafter on target-model features. We first concatenate hidden states from selected target layers and project them once into the draft hidden dimension: \[\mathbf{H}_{t} = \mathrm{RMSNorm} \left( W_c[\mathbf{H}^{(l_1)};\ldots;\mathbf{H}^{(l_5)}] \right).\] The projected target features are shared by all draft layers. At layer \(i\), draft tokens produce queries, while both target features and draft tokens are projected into keys and values: \[\begin{align} \mathbf{Q}_i &= W_i^Q \mathbf{H}_d, \\ \mathbf{K}_i &= [W_i^K \mathbf{H}_t;\, W_i^K \mathbf{H}_d]_{\mathrm{seq}}, \\ \mathbf{V}_i &= [W_i^V \mathbf{H}_t;\, W_i^V \mathbf{H}_d]_{\mathrm{seq}}. \end{align}\] Thus, target features only serve as additional KV entries for the masked-block draft tokens. They bypass the draft model’s \(Q\) projection, output projection, self-attention update, and FFN.

The memory overhead is small. The only extra parameterized component is the shared projection \(W_c \in \mathbb{R}^{D \times 5D}\). For Qwen3.5-35B-A3B with \(D=2048\) and BF16 weights, this adds \[5 \times 2048 \times 2048 \times 2 \approx 42\text{ MB},\] which is negligible compared with the roughly 70 GB target model. The activation overhead is also modest: for batch size 1 and sequence length 2048, the projection input and output require about 40 MB and 8 MB, respectively. During decoding with block size 16, the temporary activation is below 400 KB.

7.4 Results on More Models and vLLM↩︎

We further evaluate DFlash across more target models and inference frameworks. 8 reports SGLang results on one B200 with concurrency 8. Each entry shows acceptance length / speedup over autoregressive decoding. DFlash consistently improves over native MTP when both are available, and also scales to larger Qwen3.5, Qwen3-Coder, and GPT-OSS [38] models.

3pt

We also evaluate DFlash in vLLM on Qwen3.5-9B. 9 reports throughput and speedup over autoregressive decoding under different concurrency levels. DFlash achieves strong speedup at low and medium concurrency, while still maintaining throughput gains at high concurrency.

5pt

7.5 Further Ablations↩︎

7.5.1 Loss Decay↩︎

We ablate the position-dependent loss decay introduced in 4.2. Specifically, we compare the default setting with exponentially decaying token weights against a variant trained with uniform token weighting within each draft block. This study isolates the effect of emphasizing early-token accuracy during training. Results in 5 show that applying loss decay leads faster and better convergency.

7.5.2 Random Sampling of Masked Blocks↩︎

3pt

References↩︎