1 Introduction↩︎

Large Language Models (LLMs) have demonstrated unprecedented capabilities across diverse tasks [1]–[3]. To capture multilingual and domain-specific semantics [4], [5], their vocabulary sizes frequently exceed 100k tokens (e.g., 128k for Llama-3, 152k for Qwen-2 and Qwen-3). While beneficial for generation quality, these expansive vocabularies pose significant challenges for efficient inference.

Speculative decoding (SD) [6]–[8] has emerged as a premier technique to accelerate inference without compromising quality, employing a fast draft mechanism to generate tentative sequences for parallel verification by the target LLM. The efficacy of SD relies fundamentally on the draft mechanism being significantly faster than the target. However, recent work exposes a critical bottleneck when scaling SD to large-vocabulary models: the computational cost of the draft model’s final linear projection (the LM head) becomes prohibitive [9]–[11]. Crucially, this bottleneck affects diverse draft architectures, including separate small models and integrated heads like EAGLE or Medusa [12], [13] as they all rely on the LM head to map the hidden states to massive vocabulary spaces. For example, this single projection step can consume over 60% of the total draft inference time on Llama-3-8B and Qwen-2.5-7B [10], severely limiting achievable speedups.

A direct remedy is to prune the draft model’s output vocabulary. As draft tokens are verified by the target model whose vocabulary remains untouched, pruning does not compromise final output quality. Existing approaches typically adopt context-agnostic or coarse-grained strategies. Methods like FR-Spec [9] and VocabTrim [14] statically select high-frequency tokens as the pruned vocabulary based on corpus statistics, while others like CORAL [10] and DynaSpec [11] employ trained auxiliary routers to select from predefined sub-vocabulary clusters [10], [11]. However, a fundamental limitation of these approaches is their inability to adapt fine-grainedly to immediate contexts. To maintain reasonable draft quality and capture necessary long-tail tokens, they are forced to operate with relatively large active vocabularies (e.g., \(\sim\)​30k tokens as shown in Table 1), missing the opportunity for substantial acceleration.

In this work, we challenge this paradigm by seeking the minimal sufficient vocabulary required for each specific generation step. Our core insight is that language generation exhibits strong temporal locality [15]–[17]: the next token is overwhelmingly likely to be present in the immediate context or be a close extension of it. Driven by this, we propose MicroSpec, a training-free approach that dynamically constructs a minimalist active vocabulary (e.g., 2k-3k tokens) per step solely from token history and recent high-probability candidates, eschewing learned routers completely. While theoretically compelling, realizing the benefits of a highly dynamic, ultra-small vocabulary presents its own system-level challenge: the resulting sparse memory access patterns for gathering LM head weights are notoriously inefficient on modern GPUs, potentially negating computational savings. To overcome this, MicroSpec integrates system-algorithm co-design through asynchronous gathering and GPU-resident state management, effectively mitigating the latency overhead of dynamic sparse computation.

As summarized in Table 1, MicroSpec achieves superior state-of-the-art generation speeds (392.7 tokens/s on Llama-3-8B) using an average dynamic vocabulary of \(<\!3\)k tokens, an order of magnitude smaller than existing approaches (27k-32k), without requiring any additional training or auxiliary parameters. By leveraging the inherent contextual nature of language supported by optimized system design, we break the long-standing trade-off between vocabulary size and draft acceptance rate, unlocking a new regime of efficient speculative decoding.

In summary, our work makes the following contributions:

• We provide empirical analysis quantifying the untapped potential of dynamic vocabulary pruning and the strong temporal locality in LLM generation.

• We propose a simple, training-free dynamic pruning method achieving high coverage with minimal vocabulary size (\(<\)​3k), co-designed with system-level techniques to resolve sparse memory access overheads.

• We demonstrate that our method serves as a plug-and-play module, achieving \(1.12-1.32\times\) complementary speedup over the state-of-the-art EAGLE-2 and surpassing complex trained baselines across diverse benchmarks.

2 Motivation↩︎

In this section, we first quantify the untapped performance potential in existing speculative decoding systems due to suboptimal vocabulary usage. Then, we validate our core insight, “temporal locality”, demonstrating that a small, dynamically constructed vocabulary can unlock this potential.

2.1 Rethinking Draft Vocabulary Pruning: Beyond the Accuracy-Efficiency Trade-off↩︎

Validated by recent studies [9]–[11], the output projection (LM head) of the draft model has emerged as a dominant bottleneck in speculative decoding, especially as model vocabularies inflate to over 100k text tokens (e.g., 128k of Llama-3, 152k of Qwen-2) [4]. To mitigate this, pruning the draft vocabulary is a compelling strategy.

However, existing state-of-the-art approaches operate under a static paradigm: either selecting a fixed subset of high-frequency tokens (e.g., FR-Spec [9], VocabTrim [14]) or training a router to select from a fixed set of pre-clustered vocabularies (e.g., CORAL [10], DynaSpec [11]). These methods seem trapped in an inevitable dilemma: a smaller vocabulary size is necessary for compute savings, but blindly restricting it severely degrades the draft’s acceptance rate due to failure of recalling long-tail tokens, often neutralizing any speed gains. We argue that this perceived trade-off stems from the inherent limitations of their static or coarsely-grained nature, which forces them to ignore fine-grained, instance-specific context. To quantify this trade-off and the resulting untapped potential, we define two scenarios:

• Oracle Scenario (Theoretical Limit): we hypothesize an ideal state where the draft model’s acceptance rate remains unaffected by vocabulary reduction. This isolates and highlights the pure computational gain achievable by shrinking the LM head’s computation.

• Actual Scenario: we evaluate the real-world performance using FR-Spec [9] as a representative static pruning method, where acceptance rate drops with the pruned vocabulary shrinks.

We vary the pruned vocabulary sizes from 32k down to 0.5k to explore the potential speedup, where 32k is the optimal setting empirically reported in FR-Spec. Figure 1 (a) illustrates this analysis on Llama-3-8B-Instruct with a case on multi-turn conversation tasks.

The oracle line (gray) shows that theoretically, shrinking the vocabulary from 32k down to roughly 2k should yield continuous speed gains. However, the actual line (green) reveals a stark reality: static pruning yields rapidly diminishing returns below a critical threshold (\(\sim\)​16k) and suffers catastrophic performance drops at smaller sizes (\(<\)​2k). This sharp decline occurs because overly aggressive, context-agnostic pruning fails to capture long-tail tokens crucial for the specific current context, drastically reducing the draft acceptance rate, as shown in the blue bars of Figure 1 (a). The substantial shaded area between the actual and theoretical curves represents a massive performance gap, the untapped potential of speculative decoding currently masked by inefficient, static vocabulary management. This gap is the direct target of our work.

2.2 The Bridge: Leveraging Strong Temporal Locality↩︎

Our approach is rooted in a key insight designed to bridge this gap: the behavior of LLM generation is governed by strong temporal locality [15], [16]. We hypothesize that the tokens LLMs generate next are overwhelmingly likely to be present in the recent context history or are closely related extensions of it. Based on this hypothesis, we argue that the active vocabulary of the draft model can be dynamically pruned per step to a much smaller, context-aware subset while capturing sufficient correct semantics.

To validate this empirically, we analyze the ground truth coverage: the probability that the correct token \(y_t^*\) selected by the target LLM at each step \(t\) is present in a pruned vocabulary. We compare two strategies:

• FR-Spec [9]: Static vocabulary with top-\(K\) high-frequency tokens based on corpus statistics.

• MicroSpec: Dynamic vocabulary based on the current context and generation trajectory (detailed in Section 3).

Figure 1 (b) compares these coverage rates across varied domains (detailed in Section 4.1) using Llama-3-8B-Instruct. The static methods (dashed lines) require prohibitively large vocabularies (often \(>16\)k or even \(>32\)k) to achieve acceptable coverage (\(>85\%\)). This perfectly explains the performance sharp drop observed in Figure 1 (a): below a certain size, static vocabularies simply miss too many correct tokens. In stark contrast, MicroSpec (stars) consistently achieves high coverage (\(73\%\)-\(97\%\) depending on the task) while maintaining an exceedingly small average vocabulary size of a maximum of 3k tokens.

This analysis provides strong motivating evidence: a minimalist, dynamically constructed vocabulary based on explicit context is sufficient to capture the vast majority of generated tokens. This pivotal insight allows us to operate in the high-speed (\(<3\)k size) regime that was previously unattainable by static methods, effectively enabling us to recover the untapped potential identified in Figure 1 (a).

3 Methodology↩︎

In this section, we first formulate the critical computational bottleneck in large-vocabulary speculative decoding. Then we introduce our training-free algorithm for constructing context-aware dynamic vocabulary and detail the accompanying system implementation designed to realize the theoretical speedups of this dynamic approach on modern GPUs.

3.1 Preliminaries and Problem Formulation↩︎

Consider a standard auto-regressive large language model \(M_\theta\). At decoding step \(t\), given context \(x_{<t}\), the model predicts the next token distribution via a final linear projection layer, conventionally termed the LM head. Let \(W_{head} \in \mathbb{R}^{V \times d}\) represent the weight of the LM head, where \(V\) is the vocabulary size originally defined during pretraining, and \(d\) is the hidden dimension. The logits \(z_t \in \mathbb{R}^V\) are computed from the final hidden state \(h_t \in \mathbb{R}^d\) as: \[z_t = W_{head} h_t. \label{eq:dense95gemm}\tag{1}\]

Speculative decoding [6]–[8], [12] employs a smaller, faster draft model \(M_{draft}\) to generate a draft of \(\gamma\) speculative tokens, which are subsequently verified by the target LLM \(M_{target}\) in parallel. While effective, the theoretical speedup is heavily bounded by the latency of the draft model itself. As vocabulary sizes \(V\) in modern LLMs scale massively (e.g., \(V \approx 128\)k for Llama-3, \(V \approx 152\)k for Qwen-2.5), the matrix-vector multiplication in Eq. 1 becomes surprisingly expensive, reaching a complexity of \(O(V \cdot d)\) per token. For efficient draft models where the backbone consists of few layers (e.g., Medusa heads or EAGLE layers [12], [13]), this final linear projection often dominates total inference latency, severely capping the achievable speedup of existing approaches [9], [10].

Our objective is to accelerate this operation by replacing the draft model’s full vocabulary set indices \(\mathcal{I}_{full} = \{1, \dots, V\}\) with a substantially smaller, dynamically determined subset of indices \(\mathcal{I}_{t} \subset \mathcal{I}_{full}\) at each decoding step \(t\), such that \(|\mathcal{I}_{t}| \ll V\). This allows replacing the dense computation in Eq. 1 with a sparse operation involving only a sub-matrix \(W_{head}[\mathcal{I}_{t}, :]\). Crucially, as we only optimize the draft model and do not alter the target model’s vocabulary, the generation quality is guaranteed to be lossless (i.e., identical to standard sampling [8]).

3.2 Algorithm: Context-Aware Dynamic Vocabulary Construction↩︎

As outlined in Section 2, we propose a deterministic mechanism to construct a context-aware \(\mathcal{I}_{t}\) by leveraging the strong temporal locality inherent in LLM generation. Our core insight is that the optimal draft vocabulary at any moment is heavily skewed towards tokens present in the immediate context history and high-probability candidates recently considered by the target model.

Figure 2 illustrates how MicroSpec updates its dynamic vocabulary. We formalize this by defining an evolving candidate token stream and deriving \(\mathcal{I}_{t}\) via a fixed-size sliding window over this stream.

Candidate Stream Initialization (Prefill Phase). Given an input prompt sequence \(x_{1:L}\), the target model processes the prompt to generate a sequence of logits \(Z_{1:L} = (z_1, \dots, z_L)\). We define a Top-K operator \(\mathcal{T}_K(z)\) that returns the set of indices corresponding to the \(K\) largest values in a logit vector \(z\). The candidate stream \(\mathbf{S}\) is initialized as the concatenation of the exact prompt tokens and the union of top-\(K_{pre}\) candidates from each position in the prompt: \[\mathbf{S}_{init} = (x_1, \dots, x_L) \oplus \text{tuple}\left( \bigcup_{i=1}^{L} \mathcal{T}_{K_{pre}}(z_i) \right),\] where \(\oplus\) denotes sequence concatenation, and \(K_{pre}\) is a hyper-parameter controlling the exploration breadth during prefill.

Dynamic Stream Update (Decoding Phase). At any decoding step \(t\), let the draft model generate a speculative token tree, and let the target model’s verification process on the validated branch yield final logits \(z_{verify}\). We collect two sets of new candidates to append to the stream:

1. Draft Candidates (\(\mathcal{C}_{draft}\)): The set of all unique token indices present anywhere in the generated draft tree at step \(t\), regardless of acceptance.

2. Verify Candidates (\(\mathcal{C}_{ver}\)): The top-\(K_{ver}\) high-probability tokens according to the target model’s distribution at this step, i.e., \(\mathcal{C}_{ver} = \mathcal{T}_{K_{ver}}(z_{verify})\). \(K_{ver}\) is a hyper-parameter controlling per-step exploration.

The candidate stream is updated by appending the sequence of these new indices: \[\mathbf{S}_{new} = \mathbf{S}_{old} \oplus \text{tuple}(\mathcal{C}_{draft}) \oplus \text{tuple}(\mathcal{C}_{ver}).\]

Vocabulary Construction via Sliding Window. To ensure bounded memory usage and computational efficiency while retaining the most relevant history, we impose a strict size limit \(W_{max}\) on the active vocabulary. The dynamic vocabulary index set \(\mathcal{I}_{t}\) for the next step is formally defined as the unique set of the most recent \(W_{max}\) tokens in the stream: \[\mathcal{I}_{t+1} = \text{Unique}(\text{Suffix}(\mathbf{S}_{new}, W_{max})),\] where \(\text{Suffix}(\cdot, W_{max})\) returns the last \(W_{max}\) elements of the sequence, and \(\text{Unique}(\cdot)\) extracts unique elements. This construction naturally implements a First-In-First-Out (FIFO) eviction policy based on observational recency, requiring zero parameter training.

3.3 System: Hardware-Accelerated Dynamic Vocabulary Realization↩︎

While the dynamic vocabulary theoretically reduces FLOPs by a factor of roughly \(V/|\mathcal{I}_t|\), realizing this speedup on modern GPUs is non-trivial. Implementing MicroSpec naively requires matrix computation on non-contiguous rows from \(W_{head}\) corresponding to the dynamic indices \(\mathcal{I}_t\). This random, sparse memory access pattern breaks GPU memory coalescing, resulting in severe bandwidth underutilization that often negates theoretical gains. To address this, we propose a hardware-aware implementation tailored to the specific requirements of our context-aware algorithm.

Avoiding Sparse Computation via Asynchronous Gathering. Instead of performing inefficient sparse computations, our strategy is to efficiently transform the sparse data access into a dense computation: we maintain a pre-allocated buffer in GPU memory, dynamically gather the weights of new active tokens from \(W_{head}\) and pack them contiguously into the active buffer at each step. This approach enables computing \(W_{head}[\mathcal{I}_{t}, :]\) using a dense matrix.

To eliminate blocking caused by \(M_{draft}\) waiting for packed weights, we further pipeline copy stream with execution of model backbone by compute stream. Specifically, as the indices \(\mathcal{I}_{t+1}\) required for the next draft step are only known after the target model completes verification at step \(t\) (generating \(\mathcal{C}_{ver}\)), we overlap the memory-intensive weight gathering for step \(t+1\) with the compute-intensive draft execution for step \(t+1\). We utilize dual CUDA streams and fine-grained event synchronization:

• Compute Stream: executes the main neural network layers (backbone) of both \(M_{draft}\) and \(M_{target}\).

• Copy Stream: executes specialized kernels that gather required weights from global memory into a pre-allocated, contiguous dense buffer.

As soon as \(\mathcal{I}_{t+1}\) is determined via the GPU-resident state update (described below), we launch the draft model’s backbone computation on the compute stream and simultaneously launch the weight gather kernel on the copy stream. A CUDA event is recorded after the gather operation. The compute stream is made to wait on this event only immediately before executing the draft’s LM head projection.

This pipeline design ensures that the gathering latency is effectively hidden behind the draft backbone computation. By the time the draft backbone finishes, the requisite weights (\(W_{head}[\mathcal{I}_{t+1}, :]\)) are continuously packed in the buffer, allowing performing LM head projection with a highly efficient dense matrix multiplication.

Eliminating Overhead via GPU-Resident State Management. The sliding window mechanism necessitates high-frequency updates to the active token set \(\mathcal{I}_t\) at every generation step. Managing this dynamic state on the CPU involves prohibitive host-device synchronization overhead, which would introduce pipeline bubbles.

To eliminate this overhead, MicroSpec implements fully GPU-resident state management. The active sliding window states are maintained entirely in VRAM using bitmaps. We employ highly parallel, lock-free CUDA kernels to identify unique new candidates from draft trees and verification results, pushing them into the stream and updating the active indices set directly on the device. This near-zero-overhead management ensures that the dynamic logic of our algorithm does not become a new system bottleneck.

4 Evaluation↩︎

In this section, we first introduce the experimental setup, including benchmarks, baselines and models. Then, we present the comparison of end-to-end speedup, acceptance lengths and draft time across different scenarios. Finally, we conduct an ablation study to provide measured insights into the source of our performance gains. Hyper-parameter sensitivity analysis is detailed in Appendix 7.3.

4.1 Experimental Setup↩︎

Benchmarks. We employ the SpecBench benchmark [18], which is specifically designed to evaluate speculative decoding methods across a diverse set of six tasks: machine translation (MT.) [19], multi-turn conversation (Conv.) [20], RAG and QA (RAG, QA) [21], mathematical reasoning (Math) [22], summarization (Summ.) [23]. We further augment it with HumanEval [24] for code generation (Code) tasks. The maximum generation length is set to 1024 tokens and the greedy sampling is adopted for all tasks.

Baselines. We compare MicroSpec against auto-regressive decoding and several strong baselines:

• EAGLE [12]: Standard state-of-the-art speculative decoding method EAGLE-2 without vocabulary pruning, computing the full LM head.

• FR-Spec [9]: A static pruning method that selects a high-frequency token subset based on corpus statistics for LM head computation.

• CORAL [10]: A trained method that uses a feed-forward network (FFN) as router to decide sub-vocabularies for each draft token. The sub-vocabularies are partitioned offline and kept static during inference.

• DynaSpec [11]: A trained extension of FR-Spec that utilizes KMeans to cluster vocabularies offline and trains a router to select active clusters.

Implementation. Our system is implemented on top of the open-source FR-Spec framework, with custom modifications to the Python and CUDA kernels to realize our proposed techniques.

Models, Hyper-Parameters and Hardware. Our evaluation includes three popular LLMs: Llama-3-8B-Instruct, Llama-3.2-1B-Instruct and Qwen-2-7B-Instruct. All baselines use the EAGLE-2 [12] and its official released checkpoints as the draft model for speculative decoding. The hyper-parameters of EAGLE-2 are identically set among all baselines. Specifically, we use draft tree depth of 5 and maximum draft tokens of 60. The pruned vocabulary size of FR-Spec is set to 32k, which is the optimal setting reported in its paper. For hyper-parameters of MicroSpec, we set \(K_{pre}\), \(K_{ver}\) to 3 and the window scale \(W_{max}\) to 3072. All experiments are conducted on a single NVIDIA H20Z GPU.

4.2 End-to-End Speedup↩︎

We first evaluate the end-to-end generation throughput (including both prefill and decoding, measured in tokens/s) across diverse tasks and models. The results are summarized in Table [tab:cross95model95speed95detailed] and Figure 3. Results of Qwen2 are detailed in Appendix 7.2.

On the Llama-3-8B-Instruct model, MicroSpec consistently achieves the highest throughput across nearly all evaluated tasks, culminating in an average speed of 392.7 tokens/s. This corresponds to a substantial 2.25\(\times\) speedup over the auto-regressive baseline (172.6 tokens/s), surpassing the strongest pruning methods, including the trained DynaSpec (2.17\(\times\)) and static FR-Spec (2.12\(\times\)). The efficiency advantage of our approach is even more pronounced on the smaller, faster Llama-3.2-1B-Instruct model. This is because on such fast base models, the drafting overhead becomes the primary bottleneck. Standard EAGLE-2, hindered by full-vocabulary computation, offers negligible gains than AR (1.05\(\times\)). In sharp contrast, benefiting from our minimized drafting overhead and maintenance of high acceptance rates (discussed in Section 4.3), MicroSpec unlocks a significant 1.35\(\times\) average speedup. We substantially outperform both static pruning (FR-Spec: 1.17\(\times\)) and complex trained methods (CORAL: 1.12\(\times\); DynaSpec: 1.20\(\times\)).

These results demonstrate that MicroSpec achieves the state-of-the-art trade-off between drafting accuracy and computational efficiency, delivering consistent and superior performance gains over existing frameworks without requiring any additional training.

4.3 Average Acceptance Length↩︎

To investigate the quality of the speculative drafts generated by different pruning strategies, we analyze the average acceptance length, which quantifies the mean number of draft tokens verified as correct by the target model per step. This metric serves as a crucial assessment of drafting accuracy without the influence of hardware.

As shown in the Table [tab:cross95model95speed95detailed], full-vocabulary methods like EAGLE achieve a high baseline acceptance length of 3.80 and 3.16. Static pruning methods, such as FR-Spec, suffer a noticeable drop in drafting quality, decreasing the average length to 3.62 and 2.70 due to the exclusion of necessary task-specific tokens from their fixed top-k selection. Notably, MicroSpec achieves an average acceptance length of 3.59 and 3.11, which is highly competitive with the full-vocabulary EAGLE baseline (3.80 and 3.16) and significantly outperforms the static FR-Spec on 1B model. This indicates that our training-free dynamic mechanism effectively identifies and includes vital tokens that static methods miss, maintaining high drafting quality without the architectural complexity or extra training required by router-based methods. This high acceptance rate provides a strong foundation for realizing end-to-end speedups.

4.4 Draft Time Analysis↩︎

For a deeper understanding of the efficiency gains, we investigate the computational overhead during the drafting phase. Figure 4 presents a comparison of the total drafting time across various tasks on Llama-3-8B-Instruct.

As observed, EAGLE suffers from significant latency due to the necessity of computing the hidden states against the full vocabulary at every drafting step. While FR-Spec mitigates this by statically pruning the vocabulary to top-\(K\) high-frequency tokens, it still suffers from noticeable overhead as it requires a large \(K\) (e.g., 32k) to capture long-tail tokens. MicroSpec achieves the lowest overhead, demonstrating the effectiveness of both our pruning algorithm and optimized system implementation. Specifically, MicroSpec reduces drafting time by an average of approximately 51.6% compared to EAGLE and 20.3% compared to the already pruned FR-Spec baseline, directly translating to the substantial end-to-end speedups reported earlier.

4.5 Ablation Study↩︎

To examine the individual contributions of our proposed components, we conduct an ablation study on the Llama-3-8B-Instruct model. We isolate the effects of our dynamic vocabulary strategy and asynchronous optimization by comparing four experimental configurations:

• Ctx. Only: The active vocabulary \(\mathcal{I}\) is restricted to tokens present in the input prompt and high-probability candidates from the prefill phase (\(\mathbf{S}_{init}\)). No new candidates are added during decoding steps.

• Ext. Only: The vocabulary is composed solely of tokens from the draft tree and top candidates from the target model’s latest verification (\(\mathcal{C}_{draft} \cup \mathcal{C}_{ver}\)), ignoring the prompt context.

• Ctx. + Ext. (\(\times\) Async.): Our full dynamic vocabulary strategy is used while the system relies on a naive Indexed GEMM kernel for sparse computation, without our asynchronous gathering optimization.

• Ctx. + Ext. (\(✔\) Async.): Our complete method, employing both the full dynamic vocabulary and the system-level optimization.

The results summarized in Table 2 demonstrate that relying solely on prompt-derived tokens (Ctx. Only) yields a very low speed of 313.3 tokens/s. Switching to Ext. Only significantly improves speed to 351.4 tokens/s, showing the importance of dynamic vocabulary. Combining both sources (Ctx. + Ext.) provides the best result (364.1 tokens/s), confirming that capturing both global logic from the prompt and local continuity for drafting is essential. The results also show that the naive implementation for dynamic vocabulary (\(\times\)Async.) is limited by slow, non-contiguous memory access in indexed GEMM. Applying our asynchronous gathering (\(✔\)Async.) transforms sparse access into efficient dense computation and hides data transfer latency, boosting speed from 364.1 to 392.7 tokens/s. This highlights the necessity of hardware-aware system design to realize the theoretical benefits of dynamic vocabulary on GPUs.

5 Related Work↩︎

5.0.0.1 Speculative Decoding.

Speculative decoding [6]–[8] has emerged as a prominent technique for accelerating auto-regressive LLM inference. It employs a smaller, faster draft model to tentatively generate draft sequences, which are then verified in parallel by the larger target model. EAGLE [12] uses single Transformer layer as the draft model, while Gumiho [25] explores combining Transformer layer and MLP layer. RSD [26] invokes verification dynamically through a reward model. Recent work [27] also proposes augmenting the vocabulary of target model for better performance. Our optimization is complementary with these work as they all rely on LM head to compute logits and suffer from inefficiencies with the LLM vocabulary scaling.

5.0.0.2 Vocabulary Pruning.

Beyond speculative decoding, vocabulary pruning is employed in other LLM domains for different objectives, such as early-exiting [28], [29] and reinforcement learning [30], [31] to improve efficiency. Other work proposes pruning for inference to reduce memory consumption [32], [33]. A critical distinction is that these methods apply pruning that directly affects the final output or training objective, often incurring a trade-off in accuracy. As our work applies pruning solely to the draft model, the speculative decoding framework ensures the final output is rigorously verified by the target model, guaranteeing lossless outputs identical to standard auto-regressive decoding.

6 Conclusion↩︎

In this paper, we challenged the prevailing paradigm of using static or complex trained routers for vocabulary pruning in speculative decoding, proposing MicroSpec, a training-free dynamic method with system-level co-design to break the long-standing trade-off between vocabulary size and acceptance rate. The empirical evaluation demonstrates that MicroSpec achieves state-of-the-art end-to-end speedups with an unprecedentedly small vocabulary of \(<\)​3k tokens.

Impact Statement↩︎

This work aims to improve the inference efficiency of large language models, particularly focusing on reducing the bottleneck caused by large vocabularies. The primary potential benefits of our approach lie in improving computational resource utilization and promoting accessibility. By achieving significant speedups without requiring additional training or complex auxiliary models, our method lowers the barrier toward deploying capable LLMs in resource-constrained environments. Furthermore, increased inference efficiency can contribute to reduced energy consumption per generated token, supporting more environmentally sustainable AI practices. While faster text generation inherently applies to both beneficial and potentially harmful use cases of LLMs, the focus of this contribution is strictly on infrastructural efficiency optimization.

7 Appendix↩︎

7.1 Pseudo-Code of MicroSpec↩︎

Algorithm 5 provides the complete pseudo-code for MicroSpec, illustrating the control logic of speculative decoding and the asynchronous streams for computation and memory gathering.

7.2 Robustness Analysis on LLM with Larger Vocabulary↩︎

To investigate the robustness of MicroSpec when producing text from a larger vocabulary space, we analyze its performance on the Qwen-2-7B-Instruct model, which possesses a vocabulary size of approximately 152k (compared to 128k for Llama-3). The hyper-parameters of MicroSpec are the same with the settings in Section 4.1. The results of DynaSpec and CORAL are from data reported in their paper, as their implementations or trained router models are not available. As CORAL does not report acceptance lengths for individual tasks, the relevant data is left blank.

Table 3 reports the average acceptance length across diverse tasks, which directly reflects the quality of the draft model’s generation and the coverage capability of the pruned vocabulary. The results demonstrate that despite Qwen-2’s massive vocabulary, MicroSpec, operating with a highly constrained maximum dynamic range of only \(W_{max}=3072\) tokens, maintains a high acceptance length (3.48), comparable to baselines that utilize much larger static vocabularies (i.e., 3.44 of FR-Spec) or full vocabulary spaces (i.e., 3.65 of EAGLE). This provides strong empirical evidence for the robustness of our context-driven, training-free approach in handling large-scale vocabulary models.

7.3 Hyper-parameter Sensitivity Analysis↩︎

The maximum size of the dynamic vocabulary (\(W_{max}\)) is a critical hyper-parameter in MicroSpec. It governs a fundamental trade-off between draft quality (vocabulary coverage, measured by acceptance length) and compute overhead (the latency associated with gathering weights and performing GEMM operations for a larger vocabulary). To define the operational boundaries of our approach, we conduct a sensitivity analysis by varying \(W_{max}\) from 1024 to 8192 on SpecBench using Llama3-8B-Instruct. The results are visualized in Figure 6.

Analysis of Draft Quality. The dashed blue line in Figure 6 illustrates the impact of \(W_{max}\) on the average acceptance length. We observe a sharp increase in acceptance length as the window size grows from 1024 to 2048. However, beyond \(W_{max} \approx 2048\), the acceptance length rapidly saturates and plateaus around a value of 3.6. This trend provides strong empirical evidence for our core insight regarding temporal locality: a relatively small, historically relevant context window is sufficient to capture the vast majority of tokens required for efficient speculative generation. Increasing the window size further yields diminishing returns in terms of vocabulary coverage.

Analysis of System Performance. The solid green line indicates the end-to-end generation speed. Initially, the speed increases alongside accepted length, peaking within the shaded optimal region (\(2048 \le W_{max} \le 4096\)). Crucially, as \(W_{max}\) increases further beyond this region (e.g., up to 8192), the generation speed begins to decline, despite the acceptance length remaining stable. This decline highlights the diminished benefits of increasing vocabulary size under a length-specific context: as most generation tasks include an amount of total tokens that is less than 4096, larger sliding window has been hardly filled while the system overhead of computing LM Head continuously increases.

Consequently, an optimal operating region exists where the vocabulary is sufficiently large to maximize acceptance rate while staying small enough to minimize system overhead. We thus select \(W_{max} = 3072\) (marked with stars in Figure 6) as our default robust setting, striking an effective balance between high algorithmic efficiency and minimal system latency.

References↩︎