0.0.1 Introduction↩︎

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive large language models [1]–[3]. By adopting bidirectional attention, dLLMs enable any-order parallel decoding, overcoming the sequential generation bottleneck of autoregressive language models. Research has demonstrated that diffusion large language models achieve performance competitive with autoregressive LLMs while offering improvements in inference efficiency [4]–[6]. Despite the capability of parallel decoding, dLLMs still require a large number of denoising steps to maintain generation quality, considerably limiting their inference efficiency. Recently, many studies have sought to accelerate dLLMs by leveraging Key-Value (KV) Cache to reduce the computational cost per step or by designing approaches to reduce the number of denoising steps [7]–[9]. These methods have been reported to achieve substantial inference speedups without degrading generation quality, sometimes even with performance gains.

However, existing studies report inconsistent evaluation results even under seemingly identical evaluation settings, hindering reliable evaluation of decoding methods [1], [10], [11]. For instance, the reported accuracy on GSM8K for LLaDA-8B-Instruct exhibits a discrepancy of up to 9% across different studies, despite using the same decoding method and generation configuration [12]–[15]. To investigate this evaluation inconsistency, we conduct a rigorous evaluation of current decoding methods across various benchmarks and evaluation settings. We reveal an evaluation pitfall in dLLMs: the ranking of decoding methods can vary significantly across prompt templates, undermining the reliability of evaluations under a single prompt template. Through comprehensive experiments, we surprisingly find that current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off.

In this work, we identify a key factor behind this evaluation inconsistency as the high sensitivity of decoding methods to minor variations in prompt templates. As illustrated in Figure 1, minor changes in the prompt template can lead to significant performance variation—even under identical random seeds and evaluation settings. In particular, parallel decoding methods exhibit greater performance variability, as decoding multiple tokens simultaneously results in dependency conflicts [16]. Through further analysis, we find that prompt templates can play a dominant role in determining evaluation results. An effective template can yield high performance scores even with fewer denoising steps, substantially exceeding the marginal gains from increasing denoising steps.

Beyond prompt templates, our experiments indicate that overlooked evaluation settings (e.g., generation length, few-shot prompting, and hardware) can also affect the assessment of decoding methods for dLLMs. Our empirical analysis shows that increasing generation length does not consistently yield performance gains, and can even degrade performance under certain prompt templates. In addition, parallel decoding methods benefit more from increasing few-shot examples than the single-token decoding method, which may overestimate their performance gains in few-shot evaluation settings. Moreover, we find that evaluating decoding methods across different hardware platforms can yield inconsistent results, undermining fair comparisons between decoding methods. Based on these findings, we propose practical guidelines for reliable evaluation of decoding methods in dLLMs, emphasizing the importance of multi-prompt evaluation and transparent reporting of evaluation settings.

We summarize our main contributions as follows:

• We reveal a key evaluation pitfall in the assessment of dLLM decoding methods: the rankings of decoding methods can vary significantly across prompt templates, undermining the reliability of evaluations.

• Our comprehensive experiments show that current decoding methods fail to overcome the trade-off between speed and quality. We further identify this evaluation inconsistency as the high sensitivity of decoding methods to minor variations in prompt templates.

• We provide a systematic analysis of dLLM decoding methods across various benchmarks and evaluation settings. Our analysis shows that, beyond prompt templates, overlooked evaluation settings can affect evaluation results.

0.0.2 Preliminary↩︎

0.0.2.1 Diffusion Large Language Models

Diffusion large language models formulate text generation as a forward process and a reverse process. The forward process gradually adds noise to clean text by masking tokens, while the reverse process recovers masked sequences by iteratively predicting masked tokens.

0.0.2.1.0.1 Forward Process

Let \(x_0\) denote a clean sequence consisting of \(N\) tokens. We denote by \(x_{t}\) the partially masked sequences, where each token is independently masked with probability \(t \in [0,1]\). The forward process progressively masks tokens independently in \(x_{0}\), resulting in a corrupted \(x_t\). At each time \(t\), the forward process is defined by the transition probability: \[q_{t\mid 0}(x_t \mid x_0; t)=\prod_{i=1}^{N} q_{t\mid 0}(x_t^i \mid x_0^i), \quad \text{where} \quad q_{t\mid 0}(x_t^i \mid x_0^i; t)= \begin{cases} t, & \text{if } x_t^i=[\texttt{MASK}],\\ 1-t, & \text{if } x_t^i=x_0^i,\\ 0, & \text{otherwise.} \end{cases}\] where \(q_{t\mid 0}(x_t^i \mid x_0^i; t)\) denotes the transition probability for the \(i\)-th token, where \(x^i_t\) is independently replaced by [MASK] with probability \(t\). Consequently, the sequence-level transition probability \(q_{t|0}(x_t \mid x_0; t)\) can be factorized as the product of token-level transition probabilities at each position.

0.0.2.1.0.2 Reverse Process

The reverse process seeks to recover masked sequences by predicting masked tokens. Specifically, the mask predictor is parameterized by a neural network \(p_\theta\) that takes \(x_{t}\) as input and predicts all masked tokens simultaneously. Formally, we define the reverse process as follows:

\[p_\theta\!\left(x_0 \mid x_{t};t \right) = \prod_{i\in \mathcal{M}_t} p_\theta\!\left(x_0^{i} \mid x_t; t\right),\]

where \(\mathcal{M}_t:=\{i\mid x_t^i =[\texttt{MASK}]\}\) represents masked token set. The core of dLLMs is to learn a mask predictor that recovers masked tokens from corrupted text. Consequently, given the noisy sequence, the model is trained to denoise \(x_{t}\) by maximizing a variational lower bound on the log-likelihood: \[\log p_\theta(x_0) \ge \mathbb{E}_{t, x_t} \left[ \frac{1}{t} \sum_{i=1}^{N} \mathbf{1}[x_t^i=\texttt{MASK}] \log p_{\mathrm{\theta}}(x_{0}^{i}\mid x_{t}) \right],\] where \(t\) is sampled from the uniform distribution \(t \sim U(0,1)\), and the \(x_t\) is the corrupted text sampled from the distribution \(q_{t\mid 0}(x_t \mid x_0; t)\).

0.0.2.1.0.3 Generation

At generation time, given a prompt \(q\) with a fully masked response, the generation process progressively denoises the masked response into plain text over \(T\) steps. To maintain generation quality, the mask predictor decodes only a subset of the masked tokens at each step, rather than decoding all masked tokens simultaneously. During generation, the model progressively decodes the masked response through iterative refinement over multiple steps. As a result, the masked response is gradually unmasked through iterative refinement, resulting in a coherent and fluent output.

0.0.3 Evaluation Inconsistency in Diffusion Large Language Models↩︎

0.0.3.1 Problem Setup

Accelerating decoding in dLLMs typically degrades generation quality. As a result, recent work has sought to develop decoding algorithms that improve inference efficiency without sacrificing generation quality. To assess the performance of decoding methods, researchers commonly evaluate them on benchmark datasets, where higher accuracy indicates better generation quality.

Formally, we denote by \(\mathcal{A} = \{A_1, \dots, A_n\}\) a set of decoding algorithms for dLLMs and \(\mathcal{D}\) a benchmark dataset. Given a decoding algorithm \(A_i\), we query the model for each benchmark question \(q\) with a prompt template \(p\), and evaluate the response generated to obtain the performance score \(s(A_i, p; \mathcal{D})\). The performance scores of all decoding algorithms induce a ranking over \(\mathcal{A}\): \[s(A_{(1)},p;\mathcal{D}) \ge s(A_{(2)},p;\mathcal{D}) \ge \cdots \ge s(A_{(n)},p;\mathcal{D}),\] where \(A_{(r)}\) denotes the decoding algorithm with the \(r\)-th highest score. A higher score often indicates that the decoding algorithm produces better performance on the benchmark dataset. This evaluation typically assumes that the relative ranking of decoding algorithms remains unchanged across different prompt templates.

Definition 1 (Evaluation Inconsistency). Let \(\mathcal{P}\) denote the set of prompt templates. For each prompt template \(p \in \mathcal{P}\), let \(s(A_i, p)\) denote the performance score of method \(A_i\) on a benchmark under prompt \(p\). The evaluation procedure exhibits evaluation inconsistency if there exist \(p, p' \in \mathcal{P}\) and \(A_i, A_j \in \mathcal{A}\) such that \[s(A_i, p) > s(A_j, p) \quad \text{and} \quad s(A_i, p') < s(A_j, p').\] While performance scores may vary across prompt templates, a reliable evaluation should yield a consistent ranking of methods on the same benchmark. Evaluation inconsistency implies that assessing decoding methods under a single prompt template may yield unreliable comparisons, leading to evaluation illusions.

0.0.3.2 Evaluation Inconsistency Undermines Performance Metric Reliability

0.0.3.2.0.1 Experimental setup

We conduct experiments using recent decoding methods for dLLMs, including vanilla baseline (Low-confidence [1]), Fast-dLLM [7], AdaBlock-dLLM [8], AdaBlock-Fast [8], dKV-Cache-Decode [9], Elastic-Cache [17], and WINO [18], on commonly used math and code benchmarks. In our experiments, we set the generation length to 128 and adopt the semi-autoregressive decoding strategy. To eliminate the effects of hardware and system configuration on evaluation results, our experiments are conducted on a single machine with a single GPU and a batch size of 1. Further details on the hyperparameter configurations of each decoding method are provided in Appendix 1.0.3.1. To examine the sensitivity of existing decoding methods to different evaluation settings, we evaluate decoding methods across multiple prompt templates, as presented in Appendix 1.0.5.1. In our experiments, Vanilla denotes the low-confidence decoding baseline, which decodes a single masked token with the highest confidence at each step. Parallel decoding methods (e.g., Fast-dLLM) decode multiple masked tokens at each step according to a predefined parallel decoding strategy.

0.0.3.2.0.2 dLLM decoding methods exhibit pervasive evaluation inconsistency

Table 1 and Table 2 report the accuracy of various decoding methods on math datasets across various prompt templates under LLaDA-8B-Instruct [1] and LLaDA-1.5 [19], respectively. The experimental results reveal that the performance of decoding methods is highly sensitive to prompt template choices, exhibiting substantial instability across different prompt templates. For instance, in Table 1, Elastic-Cache achieves better performance than Vanilla under Template 3, but Vanilla substantially outperforms Elastic-Cache under Template 2. Figure 2 (a) and Figure 2 (b) show the pairwise Kendall’s tau correlations (See Definition in Appendix 1.0.3.3) among method rankings obtained under different prompt templates for LLaDA-8B-Instruct and LLaDA-1.5, respectively. Higher values indicate more consistent relative ranking of decoding methods, whereas lower values reflect greater disagreement in method ranking across prompt templates. The average pairwise Kendall’s tau across prompt templates is only 0.539 for LLaDA-8B-Instruct and 0.495 for LLaDA-1.5, indicating pervasive evaluation inconsistency across dLLM decoding methods. These findings suggest that evaluating decoding methods under a single prompt template may undermine the reliability of evaluations.

0.0.3.2.0.3 Current parallel decoding methods cannot overcome the speed-quality trade-off

A fundamental challenge in dLLM decoding methods is the dilemma between inference efficiency and generation quality. To address this, recent studies have explored parallel decoding methods that reduce the number of denoising steps to accelerate generation without sacrificing generation quality. Table 1 and Table 2 compare the accuracy of the vanilla baseline and current parallel decoding methods across various prompt templates. Although some decoding methods occasionally produce marginal improvements over the vanilla baseline on certain prompt templates, degradation is widespread across the majority of prompt templates. In particular, the average accuracy across prompt templates further suggests that current parallel decoding methods consistently underperform the vanilla baseline, failing to overcome the speed-quality trade-off. Additionally, we find that methods without KV caching (e.g., AdaBlock-dLLM) outperform methods employing KV caching (e.g., Fast-dLLM ), suggesting that KV caching often results in performance degradation. In Appendix 1.0.4.1, the evaluation results of code benchmarks are presented in Table 5 and Table 6, further demonstrating that most parallel decoding methods fail to achieve acceleration without performance degradation. These results suggest that relying on a single prompt template risks evaluation illusion that current decoding methods can simultaneously accelerate generation and maintain generation quality.

0.0.4 Understanding Performance Dynamics of dLLM Decoding Methods↩︎

0.0.4.1 Investigating Sensitivity of Decoding Methods to Minor Prompt Variations

0.0.4.1.0.1 Experimental setup

Based on the previous observation that decoding methods are highly sensitive to prompt template choices, we are motivated to systematically examine how minor prompt variations affect performance metrics in evaluation. Specifically, we construct a set of near-identical prompt templates that differ only in individual words or characters, as presented in Appendix 1.0.5.2. We evaluate three decoding strategies on GSM8K under LLaDA-8B-Instruct: single-token decoding (Vanilla), Top-\(k\) decoding, and threshold-based confidence decoding. We provide the details of implementation in Appendix 1.0.3.2. Vanilla decoding employs low-confidence remasking, committing only the highest-confidence masked token at each step, and generally achieves superior performance. Parallel decoding accelerates generation by decoding multiple tokens per step, reducing denoising steps but often degrading task performance. The Top-\(k\) decoding method selects the \(k\) masked tokens with the highest confidence scores, while the threshold-based confidence decoding method unmasks all tokens whose confidence exceeds a predefined threshold.

0.0.4.1.0.2 Parallel decoding is more susceptible to minor prompt variations

Table 3 presents the accuracy of the single-token decoding method and different parallel decoding strategies across various near-identical prompt templates on GSM8K under LLaDA-8B-Instruct. We quantify the sensitivity of decoding methods to prompt variation using the standard deviation and range of accuracy across prompt templates. Surprisingly, even minor variations in prompt templates can substantially affect the performance of the decoding methods evaluated. Although the Vanilla baseline exhibits the lowest standard deviation and range among all methods, it still shows noticeable performance differences across near-identical prompt templates. In contrast, parallel decoding methods exhibit substantially greater sensitivity to minor variations of the prompt template. Top-\(k\) methods exhibit considerably higher standard deviation and range than threshold-based decoding methods, suggesting that they are more susceptible to prompt template variations. This stems from Top-\(k\) decoding potentially sampling low-confidence tokens, which disrupts token dependencies under the conditional independence assumption, thereby amplifying sensitivity to minor prompt variations. Our results further show that decoding more tokens per step leads to greater performance variability, as more concurrently decoded tokens are more prone to inter-token dependency conflicts. These findings indicate that parallel decoding methods, owing to their substantial sensitivity to prompt template variation, are particularly susceptible to the illusion of evaluation.

0.0.4.2 Understanding the Impact of Denoising Steps and Prompt Templates on Evaluation

0.0.4.2.0.1 Experimental setup

Diffusion large language models generate text through iterative denoising, progressively unmasking tokens across multiple denoising steps. While increasing the number of denoising steps is generally expected to improve generation quality, a natural question arises: Do more denoising steps necessarily lead to better performance on downstream tasks? In this subsection, we empirically investigate how the number of denoising steps and the choice of prompt template jointly affect evaluation performance. In our experiments, we construct a set of semantically equivalent prompt templates A–H with lexical and stylistic variations but identical semantics and output format instructions, as presented in the Appendix 1.0.5.3. We conduct experiments with threshold-based parallel decoding across varying confidence thresholds \(\gamma \in \{0.6, 0.7, 0.8, 0.9 \}\), obtaining varying numbers of denoising steps per generation. The detailed results of accuracy and numbers of denoising steps of each decoding method are presented in Appendix 1.0.4.2.

0.0.4.2.0.2 Increasing denoising steps improves accuracy within each prompt template

Figure 3 (a) reports the accuracy across various prompt templates on GSM8K under LLaDA-8B-Instruct using the threshold-based parallel decoding method, where different confidence thresholds \(\gamma \in \{0.6, 0.7, 0.8, 0.9 \}\) produce varying numbers of denoising steps. The results show that increasing the number of denoising steps consistently improves accuracy within each prompt template, but the marginal gains diminish as the number of steps increases.

0.0.4.2.0.3 More denoising steps do not necessarily lead to better performance across prompts

Figure 3 (b) shows the accuracy of threshold-based decoding (\(\gamma\)=0.9) across different prompt templates (A-H), where each point corresponds to a prompt template, with the x-axis indicating the average number of denoising steps and the y-axis indicating the accuracy. The results show that accuracy varies substantially across prompt templates even when the number of denoising steps is comparable. In particular, template H achieves 78.47% accuracy with 48 steps, while template G achieves only 67.85% accuracy despite consuming 57 steps, a significant degradation of 10.62% accuracy. This reveals that performance differences cannot be attributed solely to denoising steps, as a well-designed prompt template can achieve superior performance with fewer steps.

0.0.4.2.0.4 Prompt templates dominate evaluation results

Figure 4 shows the accuracy of threshold-based decoding under varying confidence thresholds (\(\gamma \in \{0.6, 0.7, 0.8, 0.9\}\)), and the Vanilla baseline across four semantically equivalent prompt templates on GSM8K under LLaDA-8B-Instruct. The experimental results show that prompt template choice has a dominant effect on evaluation metrics. Surprisingly, while increasing the confidence threshold yields moderate performance gains within a given template, the performance gap across prompt templates is substantially larger. For instance, parallel decoding under prompt template H with a low acceptance threshold of 0.6 achieves 74% accuracy within only 25 steps, substantially outperforming vanilla decoding under prompt template G, which demands 128 denoising steps while yielding a remarkably lower accuracy of 68.16%. An effective prompt template consistently achieves superior performance across decoding configurations. These findings indicate that performance gains of decoding methods may stem from the choice of prompt template rather than the algorithmic effectiveness.

0.0.4.3 Potential Factors Contributing to Variation in Evaluation Metric

0.0.4.3.0.1 How does few-shot prompting affect the evaluation of decoding methods?

To investigate the effect of few-shot prompting on the performance of decoding methods, we conduct experiments with current decoding methods under different few-shot prompting settings. Table 4 reports the accuracy of vanilla and parallel decoding methods across varying few-shot settings on GSM8K under LLaDA-8B-Instruct. Few-shot prompting consistently improves accuracy across all methods, yet the performance gain differs substantially between single-token decoding and parallel decoding methods. The Vanilla baseline achieves superior performance with 2-shot examples, followed by a decline. In contrast, parallel decoding methods achieve their best performance with more few-shot examples, such as in the 4-shot setting. As a result, evaluating decoding methods exclusively in the 4-shot setting may create the evaluation illusion that parallel decoding methods can outperform the Vanilla baseline. In fact, the results show that they consistently underperform the Vanilla baseline across most few-shot settings. This suggests that few-shot prompting evaluation cannot ensure robust evaluation of dLLM decoding methods.

0.0.4.3.0.2 How does generation length affect evaluation results?

dLLMs generate text by iteratively denoising over a fixed generation length, and increasing the generation length is generally expected to improve generation quality. Our experiments reveal that the benefit of increasing generation length varies across prompts: it does not consistently yield performance gains and may even degrade performance. Specifically, we construct a set of zero-shot CoT prompt templates with identical task instructions and output format requirements, as presented in Appendix 1.0.5.4. These prompt templates differ only in their reasoning instructions, ranging from no explicit reasoning instruction (Prompt A) to increasingly detailed reasoning instructions (Prompts B–E). Figure 5 shows the accuracy of the threshold-based parallel decoding method (\(\gamma = 0.8\)) with varying generation length on GSM8K. The results show that the accuracy improves as the generation length increases from 64 to 128 across all prompt templates, suggesting that a short generation length may restrict the model’s reasoning. For prompt templates B–E, accuracy consistently improves with increasing generation length, stabilizing beyond 512. In contrast, for prompt template A, accuracy degrades when the generation length exceeds 128, as a larger generation length may mismatch the predefined generation length with the task-required response length. We find that under Prompt A, the model tends to skip intermediate reasoning steps and directly output final answers when generation length exceeds 128, degrading generation quality. Our findings show that different prompt templates exhibit inconsistent preferences for generation length, with increasing generation length sometimes degrading performance for certain prompts.

0.0.4.3.0.3 Conducting evaluations across GPU types risks unreliable evaluation results.

Table 8 in Appendix [label:float] reports the accuracy discrepancies across different GPU types, including L40, Pro6000, A100, and RTX 4090, under identical decoding method and evaluation settings. We observe that non-trivial accuracy discrepancies arise across GPU types even under identical experimental conditions, indicating that GPU types are a non-negligible factor in evaluation. This performance difference across GPU types stems from the non-associative nature of floating-point arithmetic under limited numerical precision [20]. Our analysis shows that evaluating decoding methods across different hardware platforms risks the illusion of evaluation, ultimately leading to questionable conclusions. Furthermore, we also conduct experiments across GPU types under FP32 precision. Table 9 in Appendix [label:float] reports the accuracy discrepancies across GPU types under identical decoding methods and prompt templates, evaluated in FP32 precision. The results show that the evaluation results on L40 and A100 are nearly identical, suggesting that adopting FP32 precision can substantially mitigate the evaluation variance introduced by hardware.

0.0.5 Discussion↩︎

Prior work regarding decoding methods for dLLMs has made substantial efforts to advance our understanding of this field. Although this work reveals evaluation concerns in our community, our goal is to establish more reliable evaluation practices that advance future research. Based on our findings, we propose the following practical suggestions for evaluating decoding methods in diffusion large language models.

• Report performance metrics across multiple prompt templates. Evaluating dLLM decoding methods with a single prompt template may unintentionally introduce an evaluation illusion and yield biased evaluation conclusions.

• Conduct evaluations on identical GPU platforms. When conducting evaluations across different GPU platforms, we recommend adopting FP32 precision or other determinism measures to mitigate evaluation variance introduced by hardware platforms.

• Provide detailed evaluation settings. We recommend thoroughly reporting the evaluation setup, including the employed prompt template, algorithmic parameters, generation configurations (e.g., generation length), and hardware specifications (e.g., GPU type) to ensure reproducibility.

0.0.6 Conclusion↩︎

In this work, we conduct a rigorous evaluation of current decoding methods for dLLMs across evaluation settings. Through analysis, we reveal a key evaluation pitfall in dLLMs: the ranking of decoding methods can vary significantly across prompt templates, undermining the reliability of evaluations with a single prompt template. By conducting comprehensive experiments, we find that the current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off. We further attribute the key factor of evaluation inconsistency to the pronounced sensitivity of parallel decoding methods to minor variations in prompt templates. We find that prompt templates dominate evaluation results, suggesting that the performance gains of decoding methods may stem from the choice of prompt template rather than algorithmic effectiveness. Additionally, our experiments also indicate that other overlooked evaluation settings (e.g., hardware) can also notably affect the assessment of decoding methods. Ultimately, we propose practical guidelines for the evaluation of decoding methods in dLLMs, highlighting the importance of evaluating and reporting the results across multiple prompt templates. We hope that our study deepens the understanding of evaluation challenges in dLLMs and advances future research.

1 Appendix↩︎

1.0.1 Impact Statement and Limitation↩︎

1.0.1.0.0.1 Impact Statement

This work aims to improve the reliability and reproducibility of evaluations for diffusion large language models. All experiments are conducted on public benchmarks and involve no human subjects, private data, or personal identification information. We identify no direct societal risks beyond those broadly associated with language model research. Our work conducts evaluation using existing open-source models and releases no new models, posing no additional safety risks. This work aims to shed light on evaluation illusions in dLLMs and guide future research toward more reliable assessment practices.

1.0.1.0.0.2 Limitations

Our work reveals evaluation concerns in dLLMs: the ranking of decoding methods can vary significantly across prompt templates, undermining the reliability of evaluations under a single prompt template. Although we provide practical guidelines for reliable evaluation of decoding methods in dLLMs, a more systematic evaluation framework remains an important direction for future work.

1.0.2 Related Work↩︎

1.0.2.0.0.1 Decoding methods for dLLMs

Recent efforts to accelerate dLLM inference have focused on two primary challenges: reducing the computational cost per denoising step and decreasing the number of denoising steps [18], [21]. The bidirectional attention of dLLMs precludes the use of standard autoregressive KV caching, motivating several works to propose approximate caching strategies. dKV-Cache proposes delayed and conditioned KV caching, while Fast-dLLM introduces block-wise approximate KV caching tailored for bidirectional diffusion models [7], [9]. Other methods further reduce computational overhead per step by exploiting the stability of the token representation, applying selective cache refresh, or introducing sparse computation patterns [17], [22]–[25]. Additionally, some approaches aim to reduce the number of denoising steps by unmasking multiple tokens per step [8], [21], [26]–[30]. EB-Sampler uses an entropy-bounded unmasking strategy to control the number of decoded tokens per step [31]. WINO proposes a draft-and-verify decoding strategy that allows aggressively drafted tokens to be re-masked upon verification failure [18]. These methods have been reported to achieve substantial inference speedups without degradation in generation quality. Our work reveals an evaluation illusion in dLLMs: the ranking of decoding methods can vary significantly across prompt templates, undermining the reliability of evaluations with a single prompt template.

1.0.2.0.0.2 Evaluation robustness in LLMs

LLM evaluation has been shown to be highly sensitive to seemingly minor design choices, such as prompt wording, formatting, and demonstration selection and ordering [32]–[37]. Prior work on in-context learning has shown that prompt format and demonstration order can cause substantial variance in few-shot accuracy [36], [37]. Recent work also demonstrates that LLM performance is sensitive to formatting changes [38] and broader prompt formulation variations [39]. Adversarial perturbations at the character, word, sentence, and semantic levels further amplify this sensitivity [40]. To achieve reliable estimates, some work advocates evaluating models across multiple instruction templates and reporting performance scores aggregated over these templates [41]–[43]. Recent analysis further shows that some observed prompt sensitivity may stem from evaluation artifacts such as rigid answer matching or likelihood-based scoring [44]. While these studies reveal the instability of LLM evaluation, they primarily focus on evaluating the performance of autoregressive LLMs. In our work, we investigate how prompt templates affect the evaluation of dLLM decoding methods, showing that parallel decoding methods are highly sensitive to the choice of prompt template.

1.0.3 Experimental Details↩︎

1.0.3.1 Decoding methods for dLLMs

Here, we summarize the decoding methods for diffusion language models examined in this work. For the main experiments, we set the generation length to 128.

1.0.3.1.0.1 Low-confidence

The low-confidence remasking strategy is a widely used decoding method for dLLMs [1]. During generation, the model predicts all masked positions but commits only the most confident token, leaving the remaining masked for subsequent refinement. The single-token decoding method achieves strong performance but requires a number of denoising steps equal to the generation length. In experiments, we use this method as the Vanilla baseline with generation length \(L=128\) and block size \(B=32\).

1.0.3.1.0.2 Fast-dLLM

Fast-dLLM combines block-wise approximate KV caching with confidence-aware parallel decoding to accelerate inference in dLLMs [7]. The caching component reuses approximate KV states across denoising steps within a block-wise decoding process, while the decoding component commits all predictions exceeding a confidence threshold and keeps the remaining positions masked. Thus, Fast-dLLM reduces both redundant per-step computation and the number of denoising steps. Following the official implementation of Fast-dLLM [7], we evaluate Fast-dLLM with confidence threshold \(\tau= 0.9\), and block size \(B=32\).

1.0.3.1.0.3 AdaBlock-dLLM

AdaBlock-dLLM addresses the fixed block size limitation of semi-autoregressive dLLM decoding by adaptively determining the block size during generation [8]. At the start of each block, the model inspects a local context window over the predicted sequence and examines the associated confidence scores. If a semantic delimiter is identified with high confidence above the threshold, the block boundary is placed at that position. Otherwise, the predefined default block size is adopted. Following the official implementation of AdaBlock-dLLM [8], we evaluate AdaBlock-dLLM with predefined default block size \(B=32\), delimiter set \(\mathcal{D}=[198]\), confidence threshold \(\tau=0.9\) and delimiter threshold \(\tau_D=0.3\).

1.0.3.1.0.4 AdaBlock-Fast

AdaBlock-Fast [8] integrates the adaptive block size determination mechanism of AdaBlock-dLLM into the Fast-dLLM decoding framework. Following the official implementation of AdaBlock-dLLM [8], we evaluate AdaBlock-dLLM with predefined default block size \(B=32\), delimiter set \(\mathcal{D}=[198]\), confidence threshold \(\tau=0.9\) and delimiter threshold \(\tau_D=0.3\).

1.0.3.1.0.5 dKV-Cache-Decode

dKV-Cache adapts KV caching to diffusion language model decoding through a delayed and conditioned caching strategy [9]. The method exploits the observation that token representations stabilize at different rates during the denoising process. Instead of immediately reusing all KV states, it delays cache reuse and conditions caching decisions on the current token states. Following the official implementation of dKV-Cache-Decode [9], we evaluate dKV-Cache-Decode with a cache refresh step \(step=8\), and block size \(B=\texttt{32}\). To achieve parallel decoding, we commit the two tokens with the highest predicted confidence at each denoising step.

1.0.3.1.0.6 Elastic-Cache

Elastic-Cache is a training-free KV caching method for diffusion LLMs that adaptively decides when to refresh cached states and from which layer recomputation should resume [17]. By using attention-aware drift to invalidate caches, Elastic-Cache reuses stable shallow-layer and off-window MASK caches instead of recomputing all cache states at every denoising step. Following the official implementation of Elastic-Cache [9], we evaluate Elastic-Cache with a refresh threshold of 0.9, a window length of 32, a confidence threshold of 0.9, and a tracked-token count of 1.

1.0.3.1.0.7 WINO

WINO achieves revocable parallel decoding via a wide-in narrow-out draft-and-verify mechanism [18]. During the draft stage, multiple candidate tokens are unmasked in parallel. The verification stage evaluates each drafted token under full bidirectional context, identifying unreliable predictions. Tokens that fail verification are re-masked and refined in subsequent denoising steps, while reliable tokens remain committed. This mechanism allows for aggressive parallel decoding without permanently committing to erroneous predictions. Following the official implementation of WINO [9], we evaluate WINO with a forward confidence threshold of 0.6, a backward verification threshold of 0.9, and an acceptance window of 30 tokens. Additionally, consistent with the original implementation, we set the block size to half of the generation length.

1.0.3.2 Implementation details

In the main experiments, we evaluate the performance of current decoding methods on math and code benchmarks. To eliminate the effects of hardware and system configuration on evaluation results, our experiments are conducted on a single machine with a single GPU and a batch size of 1. All experiments are conducted on 8 L40 GPUs unless otherwise stated. Experiments across hardware platforms are conducted on L40, Pro6000, A100, and RTX 4090 GPUs. The prompt templates used in our evaluation are sourced from lm-evaluation-harness and a set of manually crafted prompts, as presented in the Appendix 1.0.5. For mathematical datasets, we employ Math-Verify to assess the accuracy of generated answers. For code datasets, following prior work [8], we employ the Evaluate module from Hugging Face Evaluate to assess the correctness of generated solutions. Unless otherwise specified, for the main experiments, we set the generation length to 128 and adopt the semi-autoregressive decoding strategy with a block size of 32.

1.0.3.3 The definition of Kendall’s tau

Kendall’s tau quantifies the agreement between two rankings. Given two rankings \(r\) and \(r'\) over the same set of \(n\) items, let \(N_c\) and \(N_d\) be the numbers of concordant and discordant item pairs, respectively. We define Kendall’s tau as: \[\tau(r, r') = \frac{N_c - N_d}{\binom{n}{2}}.\] Higher values indicate stronger agreement between the two rankings, whereas lower values indicate greater disagreement. In our work, we use Kendall’s tau to quantify ranking consistency across prompt templates, which serves as a metric for measuring evaluation inconsistency in comparison of decoding methods.

1.0.4 Experimental Results↩︎

1.0.4.1 Experimental results on code benchmarks

Table 5 and Table 6 report the accuracy of various decoding methods on code datasets across various prompt templates under LLaDA-8B-Instruct and LLaDA-1.5, respectively. The results show that current decoding methods achieve occasional marginal improvements over the vanilla baseline on certain prompt templates. However, degradation is widespread across the majority of prompts. These findings show that current decoding methods fail to overcome the generation speed-quality trade-off.

1.0.4.2 Experimental results across different prompt templates

Table 7 reports the accuracy and decoding steps of single-token decoding and confidence-threshold decoding strategies on GSM8K under LLaDA-8B-Instruct across a set of semantically equivalent prompt templates. The results show that prompt templates can play a dominant role in determining evaluation results. An effective template can yield high performance scores even with fewer denoising steps, substantially exceeding the marginal gains from increasing denoising steps.

2mm

1.0.4.3 Evaluation results with FP32 Precision

Table 8 show the accuracy discrepancies across GPU types under identical decoding methods and prompt templates, evaluated in BF16 precision. Table 9 presents the accuracy discrepancies between L40 and A100 under the threshold-based decoding method (\(\gamma=0.8\)) across various prompt templates, evaluated in float32 precision. The results show that the evaluation results remain consistent between the GPU types at the precision of two decimal places, demonstrating that float32 precision can substantially reduce evaluation variance across different hardware platforms.

1.0.5 Examples of Prompt Templates↩︎

1.0.5.1 Prompt templates for mathematical and code benchmarks

Example [math_prompts] presents the prompt templates employed in our mathematical benchmark evaluation. Specifically, Templates 1–6 are designated for GSM8K, and Templates 7–8 for MATH-500, and the results in Table 1 and Table 2 are obtained using these templates. All templates are sourced from lm-evaluation-harness, except for Template 6, which is newly designed in this work. Example [code_prompts] presents the prompt templates employed in code benchmark evaluation. Specifically, Templates 1–3 are designated for HumanEval, and Templates 4–6 for MBPP, and the results in Table 5 and Table 6 are obtained using these templates. All templates are adapted from lm-evaluation-harness, with minor modifications to enhance template diversity.

1.0.5.2 Near-identical prompt templates

Example [prompt_subtle] presents a set of near-identical prompt templates constructed for the GSM8K benchmark evaluation, where each template differs from the others only in individual words or characters. For clarity, characters that differ from the first prompt are highlighted in red. Table 3 reports the accuracy of different parallel decoding strategies across this set of prompt templates.

1.0.5.3 Semantically equivalent prompt templates

Example [prompts_large] presents a set of semantically equivalent prompt templates constructed for the GSM8K benchmark evaluation, where each template maintains identical semantics while varying in lexical and stylistic expression. Table 7 reports the accuracy and number of decoding steps of each parallel decoding strategy across this set of prompt templates.

1.0.5.4 Chain of thought prompt templates

Example [prompts_cot] presents a set of prompt templates for the GSM8K benchmark, where the prompts have identical task instructions and output format requirements, differing only in the chain-of-thought elicitation phrase, from no explicit reasoning instruction to elaborate step-by-step guidance increasingly.

References↩︎

---

1. Corresponding author↩︎