1 Introduction↩︎

Speculative decoding accelerates generation by letting a lightweight draft model propose future tokens that the target model verifies in parallel [1], [2]. The attainable speedup is determined not only by whether the first drafted token is correct, but also by how many consecutive draft tokens can be accepted. This makes long-range draft accuracy especially important.

DeepSeek-V4 includes a native Multi-Token Prediction (MTP) module [3], [4], which provides a natural drafter for speculative decoding. However, due to its sequential training and inference formulation, MTP maintains high acceptance only at the first one or two draft positions; the acceptance rate drops sharply at later positions as the module extrapolates from unverified draft tokens. As illustrated in Figure 1, this positional degradation limits the accepted prefix length and therefore the practical acceleration potential of native MTP.

To address this limitation, we adopt DFlash [5] as a draft model. Unlike autoregressive MTP-style drafting, DFlash predicts an entire draft block from an accepted anchor token and target hidden-state context, avoiding the accumulation of errors along unverified draft tokens. However, directly transferring DFlash to DeepSeek-V4 is non-trivial because DeepSeek-V4 uses a multi-hyper-connection (MHC) residual stream [4], [6]. In this architecture, each token is represented by multiple parallel residual paths before prediction, rather than by a single hidden state. These paths are collapsed by the target model’s own hc_head module immediately before the LM head. Therefore, a generic DFlash adaptation faces two sources of mismatch: using intermediate-layer features moves the conditioning signal away from the final prediction state, while reducing the multi-path residual with a dense learned projection ignores the target model’s native path-aggregation mechanism.

HyperDFlash instead reuses the final pre-collapse residual already stored for MTP, aligns the drafting signal with the target MHC prediction pathway via a lightweight inherited gated reducer, and further enhances draft token quality with a targeted KL distillation objective. Focused experiments validate that our MHC-consistent designs substantially improve accepted token length and generation speed over vanilla DFlash and native MTP across math, code, and conversational benchmarks.

2 Method↩︎

Figure 2 illustrates the overall pipeline of HyperDFlash. We first establish consistent conditioning by leveraging the target model’s pre-collapse MHC residual states (Section 2.1). Then, we introduce a inherited gated reducer to achieve parameter-efficient and model-aligned path collapsing (Section 2.2). Finally, we adopt a targeted LM-head distillation strategy to refine early-position draft predictions during training (Section 2.3).

2.1 Pre-Collapse Residual Conditioning for MHC Consistency↩︎

A core challenge in adapting speculative drafting to DeepSeek-V4’s MHC architecture lies in bridging the contextual gap between the drafter and the target prediction pipeline. Conventional drafters rely on aggregated intermediate layer features, which incur redundant computation and break structural alignment with the MHC multi-path residual stream. Instead, we use the target’s final pre-collapse MHC residual states as the target-side conditioning feature for block drafting, while the drafter still follows the standard anchor-plus-mask block input format. This preserves the native multi-path semantics and improves consistency with the target’s prediction pathway.

2.2 Inherited HC-Gate Reducer for Efficient Path Collapsing↩︎

DeepSeek-V4’s unique MHC paradigm maintains multiple parallel residual paths per token, which cannot be properly aggregated by conventional static linear projections. A naive dense collapse introduces severe structural mismatch and enormous parameter overhead, breaking consistency between drafting and target inference. To tackle this issue, we design an Inherited HC-Gate Reducer that follows the functional form of the target’s native HC-head aggregation mechanism.. By initializing the gate parameters from the pretrained target hc_head weights, our method achieves structurally aligned residual collapsing with three orders of magnitude fewer parameters.

A generic DFlash reducer uses a static \(\texttt{fc}:\mathbb{R}^{md}\!\rightarrow\!\mathbb{R}^{d}\) projection (\(\sim\)​67M parameters for DeepSeek-V4-Flash). This treats the flattened MHC residual as an ordinary long vector and learns a dense reduction from scratch. However, the target model itself does not collapse its residual paths with a static linear map. Instead, the target’s own hc_head collapse is input-dependent: for token \(t\), it first normalizes the flattened multi-path residual and then computes one scalar gate per path,

\[\begin{align} \tilde{\mathbf{x}}_t &= \mathrm{RMSNorm}\!\big(\mathrm{vec}(\mathbf{H}_t)\big),\\ \boldsymbol{\alpha}_t &= \sigma\!\big(W_f\,\tilde{\mathbf{x}}_t + b\big),\\ \mathbf{y}_t &= \sum_{j=1}^{m} \alpha_{t,j}\,\mathbf{H}_{t,j}, \end{align} \label{eq:hchead}\tag{1}\] where \(W_f\in\mathbb{R}^{m\times md}\) and each \(\alpha_{t,j}\) is a scalar sigmoid gate applied to the entire residual path \(\mathbf{H}_{t,j}\). The gates are independent per-path sigmoids rather than a softmax distribution over paths. In implementation, the target module uses its original inverse-RMS scaling, scale factor, and small numerical offset; Eq. 1 abstracts this implementation into its input-dependent path-gating form. A static linear map cannot express this input-dependent gating, so it is misaligned with the target from the start.

We instantiate a drafter-side reducer using the same input-dependent path-gating form as Eq. 1 , with an additional lightweight per-path RMSNorm for the drafter input, and initialize its gate parameters from the target’s hc_head weights. The draft model thus starts from a collapse already aligned with the target’s prediction pathway; the large parameter reduction is a by-product of this alignment, not the goal (Table 1).

The reducer is specialized: it assumes the pre_hc_head source and matching target and draft hidden widths. When either assumption fails, the generic fc projection remains the fallback.

2.3 LM-Head KL Distillation for Early-Stage Draft Refinement↩︎

Standard cross-entropy (CE) loss performs one-hot token supervision based on discrete label signals and fails to exploit the full probabilistic distribution modeled by the target language model. To enhance the regularization of block-parallel drafting, we incorporate an auxiliary KL distillation objective. This scheme produces soft supervisory signals from the target’s LM head on cached hidden states without additional forward propagation overhead. In block-wise drafting, later prediction steps suffer from inconsistent contextual information due to masked intermediate tokens. Accordingly, we restrict distillation supervision to early block steps. This targeted regularization stabilizes model training and effectively improves the prediction quality of critical early draft tokens.

Since the target hidden state \(\mathbf{h}_p\) at position \(p\) encodes the context \([0{:}p]\), \(\mathrm{LMHead}(\mathbf{h}_p)\) predicts the token at \(p{+}1\). For a block anchored at position \(a\), draft position \(k\) (which predicts the token at \(a{+}k\)) is therefore aligned as \[\text{teacher}_k = \mathrm{LMHead}\big(\mathbf{h}_{a+k-1}\big),\quad \text{student}_k = \mathbf{z}_k,\] where \(\mathbf{z}_k\) denotes the draft logits. With distillation temperature \(T_{\mathrm{KD}}\) and the first \(P\) block positions supervised, \[\begin{align} \mathcal{L}_{\mathrm{KL}} &= \frac{T_{\mathrm{KD}}^2}{P}\sum_{k=1}^{P} \mathbb{E}\Big[\mathrm{KL}\big(p^{T_{\mathrm{KD}}}_k \,\|\, q^{T_{\mathrm{KD}}}_k\big)\Big],\\ \mathcal{L} &= \mathcal{L}_{\mathrm{CE}} + \alpha\,\mathcal{L}_{\mathrm{KL}}, \label{eq:loss} \end{align}\tag{2}\] where \(p^{T_{\mathrm{KD}}}_k\) and \(q^{T_{\mathrm{KD}}}_k\) are the softmax distributions of the teacher and draft logits scaled by \(1/T_{\mathrm{KD}}\). The \(T_{\mathrm{KD}}^2\) factor follows standard distillation practice [7].

This teacher is most reliable at \(k{=}1\): both teacher and student condition on the same context \([0{:}a]\), so the teacher simply provides the full target distribution that the CE one-hot label omits. For \(k{\geq}2\), the teacher also conditions on the ground-truth intermediate tokens \(a{+}1,\dots,a{+}k{-}1\), whereas the drafter sees only mask tokens at those slots and should instead model the marginal distribution over them; matching the teacher’s sharp, per-example conditionals then yields high-variance, conflicting gradients. We therefore apply KL only to the first two positions with a conservative \(\alpha\) (typically \(0.1\)–\(0.2\)). In the current implementation, the KL teacher mean-pools the MHC paths rather than using the gated hc_head collapse, which gives us a further reason to keep the weight of this auxiliary term small.

2pt

2pt

3 Experiments↩︎

3.1 Datasets and Hyperparameters↩︎

We train the drafter in two stages. Stage 1 trains a general-purpose drafter on about 300K examples from public instruction, dialogue, and code data, with the public portion mainly based on EagleChat [8].

Stage 2 continues training from the Stage 1 checkpoint on about 150K task-oriented examples, including code and instruction data such as Evol-CodeAlpaca [9]. This second stage adapts the drafter to the math, code, and chat-style generation scenarios used in our evaluation. Both stages are trained for 5 epochs on 8 NVIDIA H20 GPUs with a per-GPU batch size of 4. The learning rate is \(8{\times}10^{-4}\) for Stage 1 and \(1{\times}10^{-4}\) for Stage 2. The final checkpoint used for evaluation is selected after the second-stage adaptation.

3.2 Benchmarks↩︎

We evaluate a single DeepSeek-V4-Flash target [10] on a benchmark suite covering math, code, and chat generation. The math group contains GSM8K [11], MATH-500 [12], and AIME25; the code group contains HumanEval [13], MBPP [14], and LiveCodeBench (LCB) [15]; and the chat group contains MT-Bench [16]. This grouping separates short-form arithmetic, competition-style math, program synthesis, benchmark-style code completion, live coding tasks, and open-ended conversational generation.

All methods are evaluated under the same decoding and evaluation configuration, and all throughput and speedup measurements are conducted on the vLLM [17] inference stack. We report results for both Non-think and Think-high modes, with temperature set to 0 and 1.

3.3 Decoding Configurations↩︎

We evaluate four drafting settings under the same target model and verification pipeline, varying only the drafting mechanism and the number of drafted steps per verification round. MTP (3) is the native MTP deployment point with three drafted steps, while MTP (6) keeps the native MTP mechanism but matches the six-step draft budget used by the DFlash variants. Vanilla DFlash (6) is the generic six-step DFlash adaptation. HyperDFlash (6) enables the DeepSeek-specific conditioning source, inherited path reducer, and KL regularization described in Section 2. Parenthesized values indicate the number of drafted steps per verification round.

3.4 Metrics↩︎

We report decoding speedup over target-only autoregressive decoding and the mean accepted length \(\tau\). Speedup measures the relative generation throughput of each speculative decoding configuration against the same target-only baseline. The accepted length \(\tau\) is the average number of draft tokens accepted per verification round and is the direct acceptance-quality signal used throughout the report.

3.5 Main Results↩︎

Tables ¿tbl:tab:main-results-nonthink? and ¿tbl:tab:main-results-thinkhigh? report the main decoding results across benchmark groups, reasoning modes, and sampling temperatures. HyperDFlash consistently improves over the native MTP deployment setting. In non-thinking mode at temperature 0, it increases the average accepted length from 2.93 to 3.69 over MTP (3), with the average decoding speedup improving from 2.25\(\times\) to 2.80\(\times\).

The gain remains clear under the same six-step draft budget. Compared with MTP (6), HyperDFlash improves the average accepted length from 3.08 to 3.69 in non-thinking mode at temperature 0, while also increasing the average speedup from 1.76\(\times\) to 2.80\(\times\). This shows that the improvement does not simply come from drafting more tokens.

HyperDFlash also substantially outperforms Vanilla DFlash (6), whose average accepted length is only 2.14 under the same non-thinking temperature-0 setting. This gap indicates that a direct DFlash adaptation is insufficient for DeepSeek-V4, and that MHC-aware conditioning and path reduction are important for effective block-parallel speculative decoding.

4 Related Work↩︎

Speculative decoding accelerates autoregressive generation by using a cheaper proposal mechanism to draft future tokens and then verifying those tokens with the target model in parallel [1], [2]. Subsequent work extends this draft-and-verify template with token-tree verification, retrieval drafting, lookahead candidates, or self-speculative early exits [18]–[22]. These methods reduce target-model decoding steps, but they differ in how draft tokens are proposed and what serving or training infrastructure is required.

Learned drafters train auxiliary prediction modules to improve acceptance. Blockwise parallel decoding predicts multiple tokens per pass [23], Medusa and Hydra attach multi-token draft heads [24], [25], and EAGLE trains feature-level speculative drafters with dynamic draft trees [26]–[28]. DFlash is closest to our setting because it drafts an entire block from target hidden-state context [5]. HyperDFlash keeps this block-parallel interface but adapts it to DeepSeek-V4’s MHC residual stream [4], [6] and native multi-token prediction setting [3], [4]. Our auxiliary KL term follows standard knowledge distillation [7], but is applied conservatively because later draft positions condition on a different information set from the teacher.

5 Conclusion↩︎

This work presents HyperDFlash, an MHC-aligned block-parallel speculative decoding framework tailored for DeepSeek-V4’s unique multi-hyper-connection architecture. We systematically resolve the structural mismatch between conventional block drafting and MHC residual streams via three compact, coordinated designs: pre-collapse residual conditioning for strict target alignment, inherited gated reduction for ultra-efficient path aggregation, and early-stage KL distillation for high-quality draft refinement. Evaluations across math reasoning, code synthesis and conversational tasks verify that HyperDFlash surpasses both native MTP and vanilla DFlash in accepted token length and decoding speed. Our architecture-aware drafting paradigm offers an effective, lightweight solution for accelerating modern MHC-based large language models.

Limitations↩︎

Although we report decoding speedup, a more complete serving study should further separate drafter cost, verification cost, batching effects, and end-to-end latency under production traffic; we leave a full production-level latency comparison against MTP to future work. The KL objective is motivated by the position-wise analysis in Section 2.3 but is not ablated in isolation; moreover, the current teacher mean-pools the MHC paths instead of applying the gated hc_head collapse, a bias we mitigate by keeping its weight small. The inherited reducer assumes the pre_hc_head source and matching target and draft hidden widths, and falls back to a generic projection otherwise. Finally, all results are obtained on a single target model and include an internal workload that is not publicly reproducible; validating the recipe on other MHC-based targets is left to future work.

References↩︎