1 Introduction↩︎

The deployment of Large Language Models (LLMs) has shifted focus from model architecture to efficient serving systems [1]. In the autoregressive (AR) paradigm, continuous batching [2], [3] has become the industry standard, scheduling at the granularity of a single token to maintain high GPU utilization. However, the emergence of Diffusion LLMs (dLLMs) [4], [5] introduces a fundamental paradigm shift. Unlike AR models that generate tokens sequentially, dLLMs employ a parallel denoising process, generating entire blocks of tokens iteratively. While recent works like Fast-dLLM [6] have successfully optimized dLLMs for low-latency single-request inference, efficient batched serving for these models remains challenging when requests converge at different rates across block boundaries.

We term this phenomenon convergence heterogeneity. In autoregressive serving, output length variation is the primary source of scheduling heterogeneity, which token-level continuous batching handles by preempting at every generation step [7], [8]. dLLMs, however, generate in discrete blocks, and requests reach completion at different block boundaries even under the same generation budget. As a result, some requests can finish early while others continue denoising, creating mixed completion states within a single batched serving iteration. In current dLLM batching, the entire batch is gated by the slowest request (the “straggler”), forcing already-completed requests to remain idle while occupying resources in the active batch. The resulting “compute bubbles”—idle time where GPU resources are underutilized—lead to severe long-tail latency, calling for a dedicated block-level batch scheduling framework.

To address these challenges, we propose BlockServe, a specialized serving framework that enables high-throughput continuous batching for dLLMs without altering the underlying model architecture. BlockServe integrates a block-grained scheduler with a mixed-state memory manager, preventing stragglers from stalling the batch while enabling the execution of requests at different block indices within a unified dense tensor. A compute-aware admission controller under a token budget further adapts concurrency to workload geometry. These components form a block-centric execution loop: completed requests are reclaimed at block boundaries, heterogeneous in-flight requests remain executable in a shared dense batch, and the recovered capacity is then converted into additional admissions. Our contributions are summarized as follows:

• We introduce a block-grained scheduler that operates at block granularity, immediately evicting completed requests via block-wise completion checks to reduce straggler-induced compute bubbles in batched diffusion inference.

• We design a mixed-state memory manager that materializes unified dense tensors with explicit position alignment, extending dual cache and parallel decoding to heterogeneous batches without requiring custom kernels.

• We propose a compute-aware admission controller that replaces fixed batch sizes with a token budget, dynamically adapting concurrency to workload geometry and enabling a 2–4\(\times\) increase in effective batch capacity.

• On Dream and LLaDA across five benchmarks, BlockServe achieves 1.9–10.6\(\times\) throughput over Fast-dLLM with comparable generation quality.

2 Method↩︎

2.1 Problem formulation↩︎

We consider the concurrent serving of a diffusion LLM processing a batch of user requests. Unlike autoregressive generation, dLLMs produce the sequence via an iterative denoising process. For a specific request \(r_i\) with prompt length \(p_i\) and target generation length \(g_i\), the generation workload is decomposed into \(K_i = \lceil g_i / L \rceil\) fixed-length execution blocks of length \(L\) along the generated sequence. Each block is iteratively denoised over \(S\) diffusion steps. The state of request \(r_i\) is characterized by its current block index \(b_i \in \{0, \dots, K_i-1\}\). The core objective of BlockServe is to maximize system throughput by maintaining an active set \(\mathcal{A}\) of mixed-state requests—where \(r_i, r_j \in \mathcal{A}\) may be at distinct block indices \(b_i \neq b_j\)—while respecting hardware memory constraints.

2.2 System overview↩︎

BlockServe operates as a continuous iteration loop centered on block-grained scheduling and mixed-state execution. In each iteration, the Block-Grained Scheduler (Section 2.3) advances all active requests by exactly one block-cycle, regardless of their individual lifecycle stages, while the Mixed-State Memory Manager (Section 2.5) maps these heterogeneous requests into a unified dense tensor layout for vectorized execution. To further expand effective batch capacity, the Compute-Aware Admission Controller (Section 2.4) selects candidates from the pending queue \(\mathcal{P}\) to populate the active set \(\mathcal{A}\) under a length-aware token budget.

Figure 1 illustrates how these stages interact. In Example 1, when request 3 finishes its last block, the scheduler evicts it at the block boundary and the admission controller refills the vacant slot with request 4 from the sorted queue. Example 2 demonstrates how the batch can expand: after request 1 completes and is evicted, its large row width no longer constrains the bounding box, so previously required padding (purple) is reclaimed, freeing budget for two additional admissions and expanding the batch from three to four.

2.3 Block-grained scheduling↩︎

BlockServe’s core innovation is the shift from request-level locking to block-level preemption. Each scheduling quantum corresponds to one block-cycle of \(S\) denoising steps. Algorithm 2 outlines the protocol. In each iteration, the admission controller populates the active set \(\mathcal{A}\) from the pending queue \(\mathcal{P}\) under the token budget constraint (Section 2.4). The scheduler then executes one block-cycle for the batch in parallel (Section 2.5); this execution is block-agnostic, as requests at different generation stages are advanced within the same block-cycle. After execution, the scheduler performs block-boundary completion checks: any request that has reached its block limit \(K_i\) or produced an end-of-sequence token is immediately evicted from \(\mathcal{A}\). This fine-grained preemption mitigates the straggler effect by reclaiming resources at each block boundary rather than waiting for the entire batch to synchronize.

2.4 Compute-aware admission control↩︎

To maximize hardware utilization without causing Out-Of-Memory (OOM) errors, we abandon fixed batch sizes in favor of a token budget policy. Since standard GPU kernels operate on regular dense tensors, serving unequal-length requests within the same batch requires padding, which leads to memory waste. We define the memory cost of an active batch \(\mathcal{A}\) based on the bounding box of the dense tensor that must be materialized at each iteration. Since BlockServe only allocates space for prompt tokens and blocks that have been completed or are currently being denoised, the effective row width of request \(r_i\) at block index \(b_i\) is \(p_i + (b_i + 1) \times L\). The batch cost is: \[\label{eq:cost} \text{Cost}(\mathcal{A}) = |\mathcal{A}| \times \max_{r_i \in \mathcal{A}}\!\bigl(p_i + (b_i + 1) \times L\bigr)\tag{1}\] Prompt length and block progress determine the bounding box. When the request defining the current maximum completes and is evicted, the bounding box shrinks—freeing budget for new admissions without explicit memory reclamation. Minimizing padding waste requires reducing the gap between each request’s row width and the batch-wide maximum.

To approximate this, we implement a length-aware greedy batching strategy. The pending queue \(\mathcal{P}\) is maintained in sorted order by prompt length. During the refill phase, the scheduler searches for candidates \(r_{\text{cand}}\) whose initial row width \(p_{\text{cand}} + L\) (starting at block 0) fits within the current bounding box without expanding it (“sand filling”). A candidate is admitted only if \(\text{Cost}(\mathcal{A} \cup \{r_{\text{cand}}\}) \le \beta_{\text{budget}}\), where \(\beta_{\text{budget}}\) is determined by offline profiling of peak GPU memory utilization. This enables BlockServe to safely surpass static batch size limits, adapting concurrency to workload geometry.

2.5 Mixed-state memory management↩︎

Efficient mixed-state execution requires reconciling request raggedness with the physical density required by GPU kernels. BlockServe achieves this through two coordinated mechanisms: Logical Positional Alignment and Vectorized Mixed-Schedule Execution.

2.5.0.1 Logical positional alignment

We materialize the bounding box (Eq. 1 ) as a dense tensor of \(|\mathcal{A}|\) rows, each of length \(\max_{r_i \in \mathcal{A}}(p_i + (b_i + 1) \times L)\). To preserve positional semantics, we apply left-padding so that each prompt’s tokens occupy contiguous positions ending immediately before its generation region. We then compute explicit position IDs via cumulative indexing over the attention mask, ensuring that the first non-padding token always starts at logical position zero. This preserves the logical relative distances required by rotary embeddings (RoPE) [9] while keeping positional encoding, attention masking, and prefix-cache alignment consistent for each request within the batched execution.

2.5.0.2 Vectorized mixed-schedule execution

While existing dLLM engines optimize single-request latency via dual cache and parallel decoding [6], applying these to mixed-state continuous batching is non-trivial because requests in a batch may be processing blocks at different sequence positions. BlockServe addresses this through gather-scatter indexing that selects each request’s active block region within the shared batch tensor.

Standard dual cache mechanisms rely on static physical positions, which break under the dynamic realignments of continuous batching. We introduce a gather-scatter update mechanism that decouples logical token positions from physical memory addresses. By computing a sparse mask of active block regions, BlockServe performs in-place KV updates only at the physical offsets corresponding to the current execution block of each request, preserving the integrity of the frozen prefix cache across batch reconfigurations. Standard parallel decoding requires all requests to be synchronized at the same block index. BlockServe removes this constraint by decoupling each request’s generation trajectory. At the start of each block-cycle, the per-step transfer quota is computed by uniformly distributing the block’s masked tokens across \(S\) diffusion steps; since every block begins fully masked, this computation is block-agnostic. The same gather-based indexing enables a unified block-cycle to advance all active requests regardless of their individual block progress.

Gather-scatter indexing is the implementation primitive that makes mixed-state batching practical: gather extracts each request’s active block region into a compact tensor for denoising, and scatter writes results back to the corresponding positions. This mechanism is not an independent algorithmic contribution but the execution substrate that enables block-grained scheduling over heterogeneous request states.

3 Experiments↩︎

We evaluate BlockServe by answering the following research questions:

1. RQ1: How does BlockServe compare to state-of-the-art baselines in terms of end-to-end throughput, total wall-clock time, and generation quality?

2. RQ2: How does per-sample latency relate to output length under batching?

3. RQ3: How does the system scale with increasing batch size?

4. RQ4: What is the impact of token-budget admission control on batch capacity?

5. RQ5: How do block length and prompt-length sorting affect performance?

3.1 Experimental setup↩︎

We implement BlockServe and all baselines on a single NVIDIA H200 GPU using the LLaDA-8B-Instruct and Dream-v0-Instruct-7B models in bfloat16. The default block length is \(L=32\) tokens with \(S=32\) denoising steps per block.

We compare against two baselines: Vanilla, the default LLaDA and Dream configurations without serving optimizations, and Fast-dLLM [6], a dLLM inference accelerator with Dual Cache and Parallel Decoding enabled. For a fair comparison, BlockServe also incorporates these optimizations, isolating gains to our scheduling and memory management. We evaluate two BlockServe variants: BlockServe (Fixed), which replaces the token-budget admission rule with a fixed batch-size cap, and BlockServe (Budget), which uses the token budget policy to determine effective concurrency dynamically (Section 2.4).

We report throughput (tokens/sec), total wall-clock time, and task-specific accuracy across five widely used standard benchmarks—GSM8K [10], HumanEval [11], MBPP [12], MATH [13], and TruthfulQA-Gen [14]—covering mathematical reasoning, code generation, and factual question answering. Implementation details and benchmark statistics, including the gap between generation limits and actual output lengths, are provided in Section 3.1.1.

3.1.1 Detailed experimental setup↩︎

3.1.1.1 Implementation details

We implement BlockServe and all baselines on a single NVIDIA H200 GPU (141GB HBM3e) using the official model implementations of LLaDA-8B-Instruct and Dream-v0-Instruct-7B, both loaded in bfloat16 precision. The default block length is \(L=32\) tokens with \(S=32\) denoising steps per block. To ensure deterministic evaluation, all evaluations are conducted with temperature=0.0 (greedy decoding) under fixed seeds. For BlockServe (Budget), the token budget is set based on offline profiling on the target H200 GPU. We use 5-shot prompting for GSM8K, 3-shot for MBPP, 4-shot for MATH, and 0-shot for HumanEval and TruthfulQA-Gen. All benchmarks use the test split.

3.1.1.2 Benchmarks

We evaluate our system using five widely used standard benchmarks that together cover a diverse range of reasoning and generation tasks:

• GSM8K: A dataset of grade-school math word problems [10].

• HumanEval: A code-generation benchmark of 164 handwritten Python function-synthesis problems evaluated with unit tests [11].

• MBPP: A benchmark of programming tasks for synthesizing short Python programs from natural language descriptions [12].

• MATH: A dataset of 12,500 challenging competition mathematics problems with step-by-step solutions [13].

• TruthfulQA-Gen: The generation setting of TruthfulQA, a benchmark to measure whether a language model is truthful in generating answers to questions [14]. We adopt the reference-guided LLM-as-judge evaluation protocol of Badshah and Sajjad [15] for automated accuracy assessment.

3.1.1.3 Baselines

We compare BlockServe against the following baselines:

• Vanilla: The default LLaDA and Dream configurations, used as plain reference baselines in our evaluation without serving optimizations.

• Fast-dLLM (Static) [6]: An inference acceleration method for dLLMs. Fast-dLLM does not include a batching mechanism; in our evaluation, requests are processed in fixed groups where all must finish before the next batch begins. To ensure a strong baseline, we enable Dual Cache and Parallel Decoding.

• BlockServe (Ours): Our proposed system equipped with block-grained mixed-state scheduling. We evaluate two admission policies: BlockServe (Fixed) with a fixed batch size limit, and BlockServe (Budget) with the token budget policy. Both variants employ the length-aware sorting strategy (Section 2.4) to minimize padding. For fair comparison, BlockServe also incorporates Dual Cache and Parallel Decoding, ensuring that performance gains are attributed to our scheduling and memory management rather than kernel-level optimizations.

3.1.1.4 Metrics

We evaluate performance using the following metrics:

• Accuracy: Task-specific accuracy to assess whether generation quality remains comparable across BlockServe and the baselines.

• Throughput: Defined as the total number of valid generated tokens (excluding padding and masked tokens) divided by the total wall-clock time (in seconds). This measures the effective computation rate of the serving system.

• Total Time: The wall-clock time required to complete the entire benchmark dataset.

3.1.1.5 Benchmark statistics

We provide a comprehensive breakdown of the benchmark datasets used in our evaluation in Table 1. The substantial variance between the predefined generation limit and the actual generated sequence length—most notably observed in the TruthfulQA benchmark—underscores the inefficiency of static batching mechanisms. Such systems are compelled to reserve memory resources based on the worst-case generation limit, leading to significant underutilization when actual generations are short.

3.2 End-to-end performance (RQ1)↩︎

We evaluate the end-to-end performance in terms of tokens per second (TPS), accuracy, and total wall-clock time. Table 2 summarizes the results for LLaDA; Dream results, which exhibit consistent trends, are provided in Section 3.2.1.

3.2.0.1 Throughput improvement

BlockServe consistently achieves higher throughput than the Fast-dLLM baseline across all benchmarks. For example, on HumanEval with LLaDA (Table 2, Gen Len 256), BlockServe increases throughput from 231.5 TPS (Fast-dLLM, best-performing batch size) to 552.6 TPS (BlockServe, Fixed, batch size (BS)=16), and further reaches 597.8 TPS with token-budget admission control (Budget, reported as the peak throughput across batch sizes), corresponding to a 2.6\(\times\) speedup over the best-performing Fast-dLLM configuration we evaluated. On LLaDA, speedups range from 2.6\(\times\) to 7.7\(\times\) across benchmarks (Table 2); on Dream, the range extends to 1.9–10.6\(\times\) (Section 3.2.1).

3.2.0.2 Accuracy maintenance

Although all evaluated methods exhibit minor accuracy variations under different batch size settings, BlockServe achieves accuracy comparable to the baselines across the benchmarks considered. As shown in Table 2, BlockServe maintains generation quality broadly comparable to the baselines. For instance, on the MATH benchmark, BlockServe matches the baseline accuracy of \(\sim\)​36%, suggesting that our throughput gains are achieved while maintaining comparable generation accuracy.

3.2.1 Dream model results (RQ1)↩︎

Table 3 reports the end-to-end results for the Dream model across all five benchmarks. The trends mirror those observed for LLaDA in Section 3: BlockServe achieves 1.9–10.6\(\times\) throughput improvement over Fast-dLLM across benchmarks while maintaining comparable generation accuracy. The largest speedup (10.6\(\times\)) occurs on TruthfulQA at generation length 1024, where short output lengths amplify the straggler effect in static batching.

3.3 Latency–length relationship (RQ2)↩︎

To understand the source of our speedup, we analyze per-sample service-time latency—the wall-clock time from scheduling to completion—as a function of output length. Figure 3 presents the distribution for Dream on HumanEval at batch sizes 4 and 16.

3.3.0.1 Latency analysis

Figure 3 highlights the distributional advantage of BlockServe. BlockServe maintains a consistent, low-variance latency across output-length buckets. In contrast, Fast-dLLM exhibits pronounced “straggler effects” characterized by a heavy tail and multiple modes. Even for short generations, the baseline incurs high latency because the entire batch remains blocked by the longest-running request.

3.3.0.2 Effective concurrency analysis

Figure 4 quantifies the effective concurrency gap between methods. We report the Effective Compute Ratio \(R\) (Eq. 2 ), measuring the average number of active requests per scheduling step, normalized by the baseline batch size: \[\label{eq:effective95compute95ratio} R = \frac{1}{TB}\sum_{t=1}^{T} a_t,\tag{2}\] where \(a_t\) is the number of active requests at block-cycle \(t\), \(B\) is the baseline batch size, and \(T\) is the total number of block-cycles. For Fast-dLLM, the effective ratio drops sharply because finished requests occupy slots while the batch completes. On TruthfulQA (Figure 4 (a)), which has short generations, the ratio at BS=16 drops to 0.07. In contrast, BlockServe (Fixed) maintains a ratio near 1.0, indicating that block-grained preemption effectively reduces compute bubbles. BlockServe (Budget) can exceed 2.0 (e.g., \(\sim\)​2.4 at BS=4 on HumanEval, Figure 4 (b)), reflecting efficient oversubscription under token-budget control.

3.4 Batch-size scaling of straggler effects (RQ3)↩︎

Figure 5 demonstrates the critical scalability advantage of BlockServe. At small batch sizes (BS=1, 2), the latency gap between Fast-dLLM (Blue) and BlockServe (Orange) is minimal. However, as batch size increases to 16, the baseline latency grows disproportionately because the probability of encountering a long straggler request within the batch increases, leading to extreme tail latency, particularly on HumanEval where generation lengths vary significantly. BlockServe effectively decouples request latency from batch composition: for compute-intensive tasks like HumanEval, latency grows steadily with batch size; for shorter tasks like TruthfulQA, latency remains nearly constant as batch size increases. BlockServe’s latency distribution remains compact even at BS=16, confirming that block-grained scheduling ensures predictable latency even under heavy contention.

3.5 Admission control and batch capacity (RQ4)↩︎

We compare the maximum safe batch size, defined as the largest number of concurrent requests that completes without triggering OOM under a fixed generation-length limit. Table 4 reports the per-benchmark results. Under static batching, Fast-dLLM must reserve all memory slots for the entire batch lifetime, enforcing a conservative batch size. BlockServe reclaims slots at block granularity and admits new requests under the token budget, progressively tightening the memory bounding box as high-progress requests complete (Section 2.4). For example, on LLaDA MBPP, the maximum batch size increases from 16 (Fast-dLLM) to 64 (BlockServe), a \(4.0\times\) improvement. Across benchmarks, this yields a \(2.0\times\)–\(4.0\times\) increase in effective batch capacity under the same hardware constraints.

Table 4 compares the maximum safe batch size—the largest number of concurrent requests that completes without OOM under a fixed generation length of 1024 tokens. BlockServe’s token-budget admission control achieves \(2.0\times\)–\(4.0\times\) larger batch capacity than Fast-dLLM’s fixed limit across both models and the evaluated benchmarks, confirming that block-grained slot reclamation effectively relaxes the conservative memory constraints of static batching.

3.6 Ablation studies (RQ5)↩︎

We ablate two design choices affecting throughput–quality trade-offs: block length \(L\), which controls preemption granularity, and prompt-length sorting, which reduces padding waste.

3.6.0.1 Impact of block generation length

We evaluate block length \(L \in \{16, 32, 64\}\) on GSM8K and HumanEval. The results are presented in Table ¿tbl:tab:ablation95block95size95gsm8k? and Table [tbl:tab:ablation95block95size95humaneval].

At \(L=16\), throughput is consistently lower than at \(L=32\), indicating overhead at fine granularity. Increasing to \(L=64\) can degrade accuracy—Dream on GSM8K suffers a \(\sim\)​9% accuracy drop—because larger blocks denoise more tokens simultaneously, which can amplify the approximation effects inherent in parallel decoding [6]. We adopt \(L=32\), balancing scheduling overhead against generation stability.

3.6.0.2 Impact of prompt-length sorting

The admission controller sorts the pending queue by prompt length to reduce padding overhead. To evaluate this effect, we compare sorted and unsorted queues with all other parameters held constant.

Table 5 reports results at BS=16 across both models. Length-aware sorting consistently improves throughput by 4.7%–9.2%. This aligns with the analysis in Section 2.4: grouping requests of similar prompt length ensures that newly admitted requests have comparable initial row widths, resulting in a tighter bounding box with less padding waste.

4 Related Work↩︎

4.1 Diffusion Language Models↩︎

Diffusion models have emerged as a compelling alternative to autoregressive (AR) language models, producing text by iteratively denoising a corrupted sequence rather than generating tokens sequentially. Early discrete diffusion foundations—structured transition matrices [16], score entropy [17], and masked diffusion with semi-autoregressive samplers [18]—progressively narrowed the quality gap with AR models. Building on these advances, LLaDA [4] scaled masked diffusion to 8B parameters with semi-autoregressive block generation, achieving performance competitive with LLaMA3 8B. Ye et al. [19] demonstrated that discrete diffusion models are particularly effective for complex reasoning and planning tasks. Dream [5] further advanced this line by training a 7B-parameter diffusion LLM with AR-based initialization and context-adaptive noise rescheduling, establishing a new state of the art among open diffusion language models. BlockServe is evaluated on both LLaDA and Dream.

4.2 LLM Serving Systems↩︎

The AR serving ecosystem has converged on iteration-level scheduling [2], paged KV cache management [3], chunked prefills with stall-free batching [7], [8], and prefill-decode disaggregation [20]. These techniques are tailored to the AR execution model, where each iteration produces exactly one token per request. In contrast, dLLMs exhibit a fundamentally different execution pattern: each request undergoes multiple denoising iterations over a fixed-length block. This mismatch renders token-level continuous batching inapplicable to dLLM serving, motivating the block-grained scheduling introduced in BlockServe.

4.3 Diffusion LLM Inference Optimization↩︎

Fast-dLLM [6] accelerates single-request dLLM inference via a block-wise approximate KV cache enabling computation reuse across denoising steps despite bidirectional attention, and a confidence-aware parallel decoding scheme that selectively unmasks high-confidence tokens to reduce denoising iterations. Concurrently, dLLM-Serve [21] targets the memory footprint challenge in production dLLM serving, introducing logit-aware activation budgeting to cap peak memory from output projections, a phase-multiplexed scheduler that interleaves compute-bound and bandwidth-bound denoising phases across requests, and head-centric sparse attention.

While Fast-dLLM optimizes single-request latency and dLLM-Serve addresses phase-level resource heterogeneity in continuous dLLM serving, neither explicitly targets the convergence heterogeneity that arises when batched requests complete at different block boundaries. BlockServe fills this gap with block-grained preemption—immediately evicting completed requests to eliminate the straggler effect—mixed-state memory alignment for concurrent requests at different denoising stages, and compute-aware admission control that dynamically adapts batch concurrency to workload geometry. The single-request optimizations of Fast-dLLM are complementary to BlockServe and are integrated as drop-in accelerators within our scheduling framework without modification.

5 Conclusion↩︎

BlockServe demonstrates that block-grained scheduling substantially alleviates the straggler bottleneck in batched diffusion LLM serving, achieving 1.9–10.6\(\times\) throughput improvement across two models compared with the best-performing baseline we evaluated, while maintaining comparable generation quality. The core insight is that convergence heterogeneity—the varying rates at which dLLM requests reach completion—can be addressed by treating each execution block as an independent scheduling quantum. By combining block-level eviction with mixed-state memory alignment, BlockServe keeps GPU resources better utilized even as individual requests complete at different paces. Token-budget admission control further enables a 2–4\(\times\) increase in effective batch capacity under the same hardware constraints. Our evaluation focuses on offline batch inference; extending BlockServe to online serving with dynamic request arrivals remains future work.

References↩︎