1 Introduction↩︎

Autoregressive (AR) models have become the standard for generating discrete sequences. By factoring the joint distribution into a product of left-to-right conditionals, AR models enjoy a clean maximum-likelihood objective, stable training, and mature decoding infrastructure [1], properties that have powered the scaling of large language models (LLMs) from early billion-parameter systems to frontier models with hundreds of billions of parameters. Yet the AR paradigm carries fundamental limitations that grow more pressing as models are deployed at scale. First, generation is inherently sequential: each token must be produced before the next can begin, imposing \(\mathcal{O}(L)\) serial decoding steps regardless of available parallelism, a bottleneck that becomes increasingly costly as sequence lengths grow into the tens or hundreds of thousands [2]. Second, AR decoding is inherently left-to-right and irrevocable: once a token is emitted it cannot be revised in light of later context unless an explicit post-hoc correction mechanism is applied. This single-pass commitment limits the model’s ability to perform global planning, satisfy non-local constraints, or refine earlier decisions after observing downstream structure. Such capabilities are critical for tasks such as infilling, constrained editing, controllable generation, and long-horizon reasoning [3], [4].

Diffusion models offer a fundamentally different generation paradigm: instead of building a sequence token by token, they start from a fully corrupted input and iteratively refine all positions simultaneously over multiple denoising steps, with each step accessing the entire current state and thus enabling bidirectional context and global revision at every stage of generation. This iterative-refinement view is attractive for several reasons. First, parallel updates amortize wall-clock latency across positions, making generation time largely independent of sequence length given sufficient compute [5]–[7]. Second, global context at every step allows the model to plan holistically, correcting early mistakes, coordinating long-range dependencies, and satisfying constraints that span the entire output [3], [4], [8]. Third, the corruption-denoising framework also provides a natural interface for editing and infilling: by selectively re-corrupting only a subset of positions, the model can revise or complete partial outputs while conditioning on the surrounding context, without architectural changes or task-specific fine-tuning [9]–[11]. These properties make diffusion a compelling complement for AR generation whenever global coherence, parallel decoding, or fine-grained controllability is required.

Applying diffusion to discrete data, however, is not a straightforward extension of continuous diffusion. In continuous spaces like images and waveforms, the forward process adds Gaussian noise and the reverse process predicts a score or denoised signal in the same \(\mathbb{R}^d\) space. Discrete sequences instead live in a categorical space \(\{1,\dots,K\}^L\) where Euclidean notions of "small perturbations" and gradient-based score functions do not directly apply. One might consider embedding discrete tokens into a continuous space, applying standard Gaussian diffusion, and rounding back to the nearest token. However, such embed-then-diffuse approaches introduce a geometry mismatch between the embedding manifold and the discrete vocabulary, often resulting in rounding errors, representation collapse, and poor sample quality [12]–[14]. The alternative, which is the focus of this paper, is to define diffusion natively in discrete space, which requires designing a discrete corruption process together with a matching reverse denoiser that predicts clean tokens from corrupted categorical inputs [5], [15]–[17]. The choice of corruption process is not merely an implementation detail: it determines the topology of perturbations in token space, the difficulty of the denoising task at each timestep, and the resulting generation order. The discrete setting further introduces unique challenges related to exact likelihood computation, noise-schedule design for categorical variables, and the interaction between vocabulary structure and forward-process semantics [18]–[20]. These design choices constitute a rich and rapidly expanding design space that distinguishes discrete diffusion from both its continuous counterpart and from standard AR modeling.

The central premise of this paper is that this design space is best understood through a single organizing lens: the construction of the discrete state space, which we broadly refer to as tokenization. We argue that tokenization is not a preprocessing detail but a first-class design axis that shapes virtually every subsequent modeling decision. In text, the choice of subword vocabulary determines the granularity of masking, the semantics of substitution noise, and the effective sequence length seen by the denoiser. In tokenized multimodal generation, the codebook topology of a VQ-VAE or residual quantizer defines a notion of "distance" between tokens that the corruption process can either exploit or ignore. In scientific domains, the natural alphabet, such as amino acids, nucleotides, atom types, bond types, already carries rich structural and chemical semantics that interact non-trivially with noise schedules and validity constraints. By foregrounding tokenization, we unify these seemingly disparate domains under a single lens: the interaction between state-space structure and corruption-denoising dynamics. This perspective reveals recurring design patterns and recurring failure modes that are largely invisible when tokenization is treated as a fixed upstream choice.

Building on this premise, this paper develops a unified framework for discrete diffusion that spans the full pipeline from tokenization to generation, and instantiates it across three broad application families. The first domain is text and code generation via diffusion language models (dLLMs): masked, absorbing-state, and flow-matching approaches that have recently been scaled to billion-parameter regimes for open-ended generation, instruction following, reasoning, and code synthesis [6], [7], [21]–[23]. The second is tokenized multimodal generation, where continuous signals, such as images, audio, and video, are first discretized using vector quantization or codec models and then modeled with discrete diffusion in the resulting token space, often jointly with text tokens in unified architectures [24]–[27]. The third encompasses domain-intrinsic discrete structures in science and engineering: protein sequences over amino-acid alphabets, DNA/RNA over nucleotide vocabularies, molecular graphs with categorical node and edge attributes, and combinatorial objects such as graph layouts and scheduling solutions [28]–[31]. We additionally cover an emerging direction, involving agents, planning, and tool use, where discrete diffusion serves as a planner or structured-output generator for decision-making pipelines, including vision-language-action models and combinatorial optimization [3], [8], [32].

Our contributions are as follows.

1. Tokenization-centric lens. We elevate discrete state-space construction from a peripheral implementation choice to a primary organizing principle, analyzing how vocabulary design, codebook topology, and natural alphabets shape corruption processes, denoising difficulty, and downstream controllability, while introducing diffusion-facing diagnostics for tokenization quality (Section 4).

2. Unified formulation and training/inference framework. We decompose every discrete diffusion model into four components — the corruption operator, denoiser parameterization, training objective, and sampler — and show how the major formulation families instantiate this shared structure (Section 5), then organize training objectives and parameterizations (Section 6) and inference algorithms (Section 7) around the same decomposition.

3. Cross-domain instantiation. We map the framework across text and code, tokenized multimodal generation, proteins, genomics, molecules, and graphs, as well as planning and agents, surfacing shared design patterns and domain-specific constraints (Section 9).

4. Scaling, systems, and evaluation. We treat scaling behavior, training pipelines, inference acceleration, and evaluation protocols as integral parts of the design space rather than afterthoughts (Sections 8 and 10).

5. Open problems. We identify concrete open challenges: scaling behavior, the in-context-learning gap, KV-cache analogues, streaming generation, unified token spaces, and theoretical foundations, and frame these challenges as actionable research questions (Section 12).

The remainder of the paper is organized as follows. Section 2 positions our framework relative to prior overviews of diffusion-based generation. Section 3 establishes notation and provides minimal background on AR generation, continuous diffusion, and discrete Markov chains. Section 4 examines discrete state spaces and tokenization across text, code, quantized media, and scientific domains. Section 5 presents the core formulations: transition-matrix models, absorbing-state masking, substitution and mixture corruptions, continuous-time processes, score/ratio-based methods, and a unifying view. Section 6 systematizes training objectives, reverse parameterizations, conditioning and guidance, and practical training recipes. Section 7 covers inference algorithms: ancestral sampling, confidence-based remasking, semi-autoregressive and block updates, guidance and constraints, and acceleration techniques. Section 8 discusses scaling and systems considerations. Section 9 instantiates the framework across application domains. Section 10 reviews evaluation and benchmarks. Section 11 discusses discrete diffusion as global refinement, connections to optimization, trustworthiness, and when AR or hybrid approaches are preferable. Section 12 outlines future directions, and Section 13 concludes. We release a companion repository containing taxonomies and curated resources at https://github.com/AAAAA-Academia-Attractions/Discrete-Diffusion.

2 Related Work↩︎

A number of recent overviews touch on diffusion-based generation. Here we position our framework relative to them and make explicit what a tokenization-centric, cross-domain treatment adds.

Recent surveys have substantially improved coverage of diffusion-based generation, especially in text and multimodal language modeling. Text-centered reviews have organized the literature around non-autoregressive generation, continuous-versus-discrete formulations, or the emergence of diffusion language models and diffusion-based large language models [33]–[39]. Other surveys emphasize efficiency, including caching, parallel decoding, and deployment bottlenecks [2], [40]–[42]. In parallel, broad diffusion surveys and application-specific reviews in graphs, molecules, biomolecules, drug design, and recommender systems document the expansion of diffusion methods into structured scientific and industrial domains [43]–[50]. These works collectively provide strong coverage of model families, applications, and recent progress. However, several issues remain under-emphasized for a survey centered on discrete diffusion. Most notably, prior surveys rarely treat discrete state spaces and tokenization as a first-class organizing axis. Even when tokenization is discussed, it is usually framed as a local modeling choice, a future challenge, or a subtopic within a broader taxonomy rather than as a central principle that shapes corruption design, reverse parameterization, controllability, and validity [39], [42]. A second under-emphasized aspect is the lack of a unified treatment of domain-intrinsic discrete structures. Language tokens, code symbols, quantized multimodal tokens, biological alphabets, molecular components, and graph primitives are usually discussed within separate application areas rather than within a shared design space [45]–[48], [50]. A third gap is that formulations, objectives, inference, systems considerations, and evaluation are often reviewed separately, even though in practice they are coupled through the underlying discrete representation. As a result, the field still lacks a survey that treats discrete diffusion not simply as a collection of models, but as an end-to-end design space.

This paper is organized around precisely these missing links. Rather than centering only model families, we take tokenization and discrete state-space diagnostics as a primary organizing principle. This reflects our main thesis: in discrete diffusion, many of the most consequential choices arise before and beyond the denoiser architecture itself. Accordingly, this paper differs from prior work in three main ways. First, it treats tokenization, state-space factorization, and representation diagnostics as core methodological questions rather than peripheral implementation details. Second, it builds a cross-domain map spanning language, tokenized multimodal generation, and scientific discrete structures such as proteins, genomics, molecules, and graphs [38], [39], [45]–[48]. Third, it foregrounds scaling, systems, and evaluation standardization in the same framework, bringing together deployment constraints, parallelism, caching, efficiency metrics, controllability, calibration, and structural validity rather than leaving them fragmented across separate surveys [2], [40]–[42].

Beyond taxonomy, we aim to make the paper directly reusable as a practical design reference. To this end, we distill cross-domain checklists for choosing tokenization schemes, corruption operators, training objectives, reverse parameterizations, sampling procedures, constraints, and evaluation protocols. These checklists are motivated by recurring questions that appear across domains. What discrete state best captures the underlying structure. Which corruption mechanism preserves enough semantics or validity to support effective denoising. Which objective best matches the intended reverse process. When iterative remasking, block updates, or semi-autoregressive decoding are preferable. How hard and soft constraints should be imposed. And which metrics meaningfully capture not only quality, but also efficiency, controllability, calibration, and structural correctness. By making these decisions explicit and comparable across domains, this paper is intended not only to summarize the literature, but also to provide a reusable framework for designing, analyzing, and evaluating discrete diffusion systems in a principled way.

2.0.0.1 Scope and selection.

This is a narrative survey. We assembled the literature by tracking the citation graph around the foundational discrete-diffusion formulations (D3PM, multinomial diffusion, MDLM/MD4, SEDD, discrete flow matching) and the major venues and preprint servers through early 2026, prioritizing works that (i) introduce a distinct corruption/parameterization/objective/sampler choice, (ii) scale discrete diffusion to a new modality or size regime, or (iii) report a reusable evaluation or systems technique. We do not claim exhaustive coverage of every application paper, and the "under-emphasized for this paper" column of Table [tab:survey95summary] reflects our qualitative reading relative to this paper’s tokenization-centric scope rather than a quantitative coding rubric. The most closely related prior survey is [39]: relative to it, we (a) make discrete state-space construction (tokenization) the primary organizing axis rather than one topic among many, (b) add the domain-intrinsic scientific alphabets (proteins, genomics, molecules/graphs) and the planning/agent and tabular settings under one shared four-component framework, and (c) fold scaling, systems, and evaluation standardization into the same framework; conversely, that survey offers complementary depth on multimodal quantization that we treat more briefly.

3 Background and Notation↩︎

This section synthesizes two lines of prior work that underpin discrete diffusion: the probabilistic foundations of diffusion models [43], [46], [51], and earlier non-autoregressive and order-agnostic generation methods whose iterative-refinement view discrete diffusion inherits and formalizes [1], [52], [53].

3.1 Generative Modeling: AR vs.Iterative Refinement↩︎

3.1.0.1 Autoregressive factorization.

Let \(\boldsymbol{x} = (x_1, x_2, \dots, x_L)\) denote a sequence of discrete tokens, where each \(x_i\) takes values in a finite vocabulary \(\mathcal{V} = \{1, \dots, K\}\). An autoregressive (AR) model decomposes the joint distribution via the chain rule: \[\label{eq:ar} p_\theta(\boldsymbol{x}) \;=\; \prod_{i=1}^{L} p_\theta(x_i \mid x_{<i}),\tag{1}\] where \(x_{<i} = (x_1, \dots, x_{i-1})\) denotes the left context. Each conditional \(p_\theta(x_i \mid x_{<i})\) is typically parameterized by a causal (unidirectional) Transformer that outputs a categorical distribution over \(\mathcal{V}\) at position \(i\). Training maximizes the log-likelihood under teacher forcing: at each step the model receives the ground-truth prefix \(x_{<i}\) and is trained to predict \(x_i\), yielding a simple per-token cross-entropy loss.

At inference time, tokens are sampled sequentially: \(x_1 \sim p_\theta(x_1)\), then \(x_2 \sim p_\theta(x_2 \mid x_1)\), and so on. This sequential procedure requires \(L\) serial forward passes through the model, each generating exactly one token. While techniques such as KV caching (which avoids redundant recomputation of attention over the prefix) and speculative decoding (which drafts multiple tokens in parallel and verifies them against the target model) substantially reduce per-step cost and amortize latency, they do not eliminate the fundamental \(\mathcal{O}(L)\) serial dependency: the \(i\)-th token cannot be sampled until the \((i{-}1)\)-th token has been committed. For long sequences, this sequential bottleneck becomes the dominant wall-clock cost, especially on modern accelerators where parallel compute is abundant but serial throughput is constrained [2].

A second structural limitation is that AR decoding is left-to-right committed: once \(x_i\) is emitted, it becomes part of the conditioning context for all subsequent tokens and is never revisited. The model cannot correct an early mistake in light of later evidence, satisfy constraints that span the full sequence, or revise global structure after local details have been fixed. This one-pass commitment makes AR models a poor fit for tasks that require bidirectional context, such as infilling (generating text conditioned on both a prefix and a suffix), constrained editing (modifying a passage while preserving specific spans), and planning (producing a sequence whose global coherence depends on long-range coordination) [3], [4].

3.1.0.2 Iterative refinement as an alternative.

Non-autoregressive (NAR) and iterative-refinement approaches break the sequential dependency by generating or refining all positions simultaneously. The idea dates back at least to iterative conditional modes in structured prediction and was popularized for neural sequence generation by Mask-Predict [1], which alternates between predicting all masked tokens in parallel and re-masking the least confident predictions. This iterative loop (predict, evaluate confidence, re-corrupt, repeat) can be viewed as a fixed number of refinement steps \(T \ll L\), where each step updates many positions at once.

Diffusion models generalize this idea by defining a principled probabilistic framework for the corruption and refinement process. Rather than relying on a hand-designed re-masking heuristic, diffusion models specify a forward process that gradually corrupts data over \(T\) steps (or continuously in time) and learn a reverse process that progressively denoises the corrupted input back to a clean sample. The number of denoising steps \(T\) is a free parameter that trades off sample quality against generation speed, and is typically much smaller than the sequence length (\(T \ll L\)). At each reverse step, the model observes the entire current state and updates them jointly, providing bidirectional context and the ability to revise earlier decisions. This parallel, globally informed refinement is the key structural advantage of diffusion over AR generation, and it motivates the substantial body of work reviewed in this survey.

3.2 Continuous Diffusion Primer↩︎

3.2.0.1 Forward process, reverse denoising, and the score.

To motivate discrete diffusion, we briefly recall the continuous-diffusion framework that has driven progress in image and audio generation [43], [51]. Let \(\boldsymbol{z}_0 \in \mathbb{R}^d\) be a data point. A forward (noising) process gradually perturbs \(\boldsymbol{z}_0\) by adding Gaussian noise over a sequence of timesteps \(t = 1, \dots, T\): \[\label{eq:cont95forward} q(\boldsymbol{z}_t \mid \boldsymbol{z}_{t-1}) \;=\; \mathcal{N}\!\bigl(\boldsymbol{z}_t;\, \sqrt{1 - \beta_t}\,\boldsymbol{z}_{t-1},\; \beta_t \boldsymbol{I}\bigr),\tag{2}\] where \(\{\beta_t\}_{t=1}^T\) is a noise schedule controlling the rate of corruption. This process admits a closed-form marginal \(q(\boldsymbol{z}_t \mid \boldsymbol{z}_0) = \mathcal{N}(\boldsymbol{z}_t;\, \sqrt{\bar\alpha_t}\,\boldsymbol{z}_0,\, (1 - \bar\alpha_t)\boldsymbol{I})\) with \(\bar\alpha_t = \prod_{s=1}^t (1 - \beta_s)\), so that for large \(T\) the noisy sample \(\boldsymbol{z}_T\) becomes approximately standard Gaussian. A neural network \(\boldsymbol{\epsilon}_\theta(\boldsymbol{z}_t, t)\) is trained to predict the noise (or, equivalently, the score \(\nabla_{\boldsymbol{z}_t} \log q(\boldsymbol{z}_t)\)) at each step, yielding a learned reverse process that iteratively denoises \(\boldsymbol{z}_T \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})\) back toward the data distribution. The training objective, typically a reweighted mean-squared error between the predicted and true noise, can be derived as a simplified variational bound on \(\log p_\theta(\boldsymbol{z}_0)\).

Two features of this framework are worth highlighting. First, the forward process is fixed and analytic: Gaussian noise in \(\mathbb{R}^d\) has well-understood geometry, and the posterior \(q(\boldsymbol{z}_{t-1} \mid \boldsymbol{z}_t, \boldsymbol{z}_0)\) is available in closed form. Second, the score function \(\nabla_{\boldsymbol{z}_t} \log q(\boldsymbol{z}_t)\) is a continuous vector field, and the denoiser operates via gradient-like updates in the same Euclidean space as the data.

3.2.0.2 Why "embed then diffuse" is problematic for discrete data.

A natural attempt to apply continuous diffusion to discrete sequences is to embed tokens into a continuous space, run Gaussian diffusion, and then round back to the nearest token, an approach explored by several early works [12]–[14], [54], [55]. As discussed in the introduction, this introduces a geometry mismatch between the continuous embedding manifold and the discrete vocabulary, leading to rounding errors and embedding collapse. The technical root of the problem in the present notation is that the continuous score \(\nabla_{\boldsymbol{z}_t} \log q(\boldsymbol{z}_t)\) does not respect the discrete structure of the vocabulary: it can point in directions that interpolate between token embeddings rather than toward any valid token, so the gradient-based reverse update of Eq. 2 has no faithful categorical analogue.

These observations motivate defining diffusion natively in discrete space, where the forward process corrupts tokens by categorical operations (masking, substitution, or structured transitions) and the reverse process directly predicts categorical distributions over the vocabulary. This is the approach taken by the discrete diffusion models that are the focus of this survey, and we introduce the necessary formalism next.

3.3 Discrete Spaces, Markov Chains, and Notation↩︎

3.3.0.1 State space.

Throughout this paper, we consider data consisting of sequences of discrete tokens. Let \(K\) denote the vocabulary size and \(\mathcal{V} = \{1, 2, \dots, K\}\) the vocabulary. A sequence of length \(L\) is written \(\boldsymbol{x} = (x_1, x_2, \dots, x_L) \in \mathcal{V}^L\). Each token \(x_i\) can be represented as a one-hot vector \(\boldsymbol{e}_{x_i} \in \{0,1\}^K\); we will use this representation whenever it simplifies notation. In some domains the "sequence" has richer structure: for example, a molecular graph can be represented as a tuple of node-type and edge-type vectors, each taking values in a domain-specific categorical alphabet. We generally write \(\boldsymbol{x}\) for the full discrete object and \(x_i\) for its \(i\)-th component, noting that the framework extends to non-sequential structures by indexing over nodes, edges, or other discrete components. We use \(K+1\) to refer to the extended vocabulary \(\mathcal{V}_{\texttt{m}} = \{1, \dots, K, \texttt{m}\}\) when an absorbing (mask) token \(\texttt{m}\) is introduced. The data distribution is denoted \(q_{\text{data}}(\boldsymbol{x})\).

3.3.0.2 Forward process: discrete corruption via transition matrices.

A discrete diffusion model defines a forward process that progressively corrupts a clean data point \(\boldsymbol{x}_0 \sim q_{\text{data}}\) over \(T\) timesteps, producing a sequence of increasingly noisy states \(\boldsymbol{x}_1, \boldsymbol{x}_2, \dots, \boldsymbol{x}_T\). The corruption at each step is specified by a categorical Markov chain. For a single token \(x_i\) (we drop the position subscript \(i\) when the per-position structure is clear), the one-step transition is defined by a transition matrix \(\boldsymbol{Q}_t \in [0,1]^{K \times K}\) (or \(\mathbb{R}^{(K+1) \times (K+1)}\) when a mask token is included): \[\label{eq:forward95onestep} q(x_t \mid x_{t-1}) \;=\; \mathrm{Cat}\!\bigl(x_t;\; \boldsymbol{e}_{x_{t-1}}^\top \boldsymbol{Q}_t\bigr),\tag{3}\] where \([\boldsymbol{Q}_t]_{jk} = q(x_t{=}k \mid x_{t-1}{=}j)\) gives the probability of transitioning from category \(j\) to category \(k\) at step \(t\), and each row of \(\boldsymbol{Q}_t\) sums to one. Because the chain is Markov, the marginal at any step \(t\) given the clean token \(x_0\) is obtained by composing transition matrices: \[\label{eq:forward95marginal} q(x_t \mid x_0) \;=\; \mathrm{Cat}\!\bigl(x_t;\; \boldsymbol{e}_{x_0}^\top \bar{\boldsymbol{Q}}_t\bigr), \qquad \bar{\boldsymbol{Q}}_t \;=\; \boldsymbol{Q}_1 \boldsymbol{Q}_2 \cdots \boldsymbol{Q}_t.\tag{4}\] The cumulative matrix \(\bar{\boldsymbol{Q}}_t\) summarizes the total corruption from \(x_0\) to \(x_t\) and is the key object for training, since it allows direct sampling of \(x_t\) given \(x_0\) without simulating the chain step by step. The schedule is designed so that \(q(x_T \mid x_0)\) approaches a known stationary distribution, typically uniform over \(\mathcal{V}\) or concentrated on \(\texttt{m}\), that serves as the prior of the generative model.

The choice of \(\boldsymbol{Q}_t\) encodes strong inductive biases about what "corruption" means in a given domain. Three canonical designs recur: uniform substitution (any token may be replaced by any other with equal probability), absorbing/masking (tokens are progressively replaced by a special token \(\texttt{m}\)), and structured/embedding-aware transitions (substitution probabilities reflect token similarity) [15], [16]. We give the explicit matrices and analyze their trade-offs in Section 5; for the present purposes it suffices to note that absorbing-state masking dominates modern large-scale models [5], [7], [19].

3.3.0.3 Reverse process: learning to denoise.

Generation proceeds by running the forward chain in reverse. Starting from a sample \(\boldsymbol{x}_T\) drawn from the stationary distribution (e.g., all-m or uniform), the model iteratively denoises by sampling \(\boldsymbol{x}_{t-1} \sim p_\theta(\boldsymbol{x}_{t-1} \mid \boldsymbol{x}_t)\) for \(t = T, T{-}1, \dots, 1\), ultimately producing a clean sample \(\boldsymbol{x}_0\). The reverse transition for each token is parameterized as: \[\label{eq:reverse} p_\theta(x_{t-1} \mid \boldsymbol{x}_t) \;=\; \mathrm{Cat}\!\bigl(x_{t-1};\; \boldsymbol{\pi}_\theta(\boldsymbol{x}_t, t)\bigr),\tag{5}\] where \(\boldsymbol{\pi}_\theta(\boldsymbol{x}_t, t) \in \Delta^{K-1}\) (or \(\Delta^K\) with the mask token) is a categorical distribution predicted by a neural network that takes the entire noisy sequence \(\boldsymbol{x}_t\) and the timestep \(t\) as input. Note that the reverse model conditions on the full noisy sequence \(\boldsymbol{x}_t\), not just the single token at the current position, allowing it to exploit bidirectional context across all positions.

In practice, the network more commonly predicts a distribution over the clean token, written \(p_\theta(x_0 \mid \boldsymbol{x}_t, t)\), from which the reverse transition is recovered analytically using the known forward-process posterior. We refer to this dominant choice as the "\(x_0\)-parameterization" and defer its closed form, together with alternatives such as predicting \(x_{t-1}\) directly, log-probability ratios, or a score analog, to Sections 5 and 6. We write \(\boldsymbol{\pi}_\theta(\boldsymbol{x}_t,t)\) for this predicted clean-token distribution throughout; the equivalent symbols \(\mu_\theta(\boldsymbol{x}_t,t)\) (used in the masked-diffusion objectives of Section 6) and \(\hat{p}(x_0\mid \boldsymbol{x}_t)\) (used in the score/ratio discussion of Section 5) denote the same object, and we flag each on first use.

3.3.0.4 Conditioning.

In conditional generation, the reverse process is augmented with a conditioning signal \(\boldsymbol{c}\) (e.g., a text prompt, a class label, a structural constraint, or a partial sequence to be infilled): \[p_\theta(x_{t-1} \mid \boldsymbol{x}_t, \boldsymbol{c}) \;=\; \mathrm{Cat}\!\bigl(x_{t-1};\; \boldsymbol{\pi}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c})\bigr).\] The conditioning signal is typically incorporated via cross-attention, prefix concatenation, or channel-wise concatenation within the denoiser architecture. Training with classifier-free guidance, randomly dropping the conditioning signal during training so that the model learns both the conditional and unconditional distributions, enables guidance-style inference, where the predicted logits are interpolated between the conditional and unconditional predictions to amplify the influence of \(\boldsymbol{c}\). We return to conditioning and guidance in Sections 6 and 7.

3.3.0.5 Notation summary.

Table 1 collects the core notation used throughout the paper for quick reference.

4 Discrete State Spaces and Tokenization↩︎

This section develops the central thesis of the paper: that the construction of the discrete state space, tokenization, is a first-class design axis for diffusion rather than a fixed preprocessing step. Section 4.2 treats semantic tokens for text and code, Section 4.3 treats quantized tokens for images, audio, and video, and Section 4.4 treats the natural discrete alphabets of scientific domains. We begin in Section 4.1 with a cross-domain definition of a token and an argument for why tokenization shapes corruption, denoising difficulty, controllability, and cost, and close in Section 4.5 with diagnostics for evaluating tokenization quality from a diffusion-facing perspective.

4pt

4.1 What Is a Token? A Cross-Domain View↩︎

4.1.0.1 A unifying definition.

At the most abstract level, a token is a discrete symbol drawn from a finite vocabulary \(\mathcal{V}=\{1,\dots,K\}\) that serves as the atomic unit of categorical modeling. Every discrete diffusion model operates over sequences of such symbols: the forward process corrupts tokens by replacing them with other tokens or with a distinguished absorbing state, and the reverse process predicts a categorical distribution over \(\mathcal{V}\) at each position. Tokenization is the process by which raw data (text, pixels, waveforms, amino-acid chains, molecular graphs) is mapped into this categorical representation. Although the resulting symbols are always discrete integers, the origin of the vocabulary varies fundamentally across domains, and this variation has deep consequences for how diffusion should be designed. We distinguish three families of tokens that recur throughout this survey.

Semantic tokens arise in language and code, where subword segmentation algorithms (BPE, WordPiece, Unigram, byte-level) partition a character stream into a fixed vocabulary of variable-length pieces. These tokens are defined by frequency-based compression: they carry no intrinsic metric structure, and two tokens that are adjacent in vocabulary index may be semantically unrelated. This lack of topology is why most text-based discrete diffusion models rely on absorbing-state (masking) corruption, which does not require any notion of inter-token distance [5], [7], [19].

Quantized tokens arise in image, audio, and video generation, where continuous signals are discretized via vector quantization (VQ-VAE, VQ-GAN) or related codec architectures into codebook indices [24], [25], [56]. Unlike text tokens, quantized codes do possess geometric structure inherited from the codebook: codes whose embedding vectors are close in \(\ell_2\) distance typically reconstruct perceptually similar patches, frames, or spectral segments. This latent geometry creates an opportunity, largely underexploited in current practice, for structured corruption processes that treat nearby codes as more likely substitution targets, rather than treating all non-identity transitions as equally unlikely [15], [57].

Natural discrete alphabets arise in scientific domains where the data are intrinsically categorical. Proteins are sequences over the 20 standard amino-acids; DNA and RNA are sequences over 4-letter nucleotide alphabets; molecules is represented as graphs whose nodes and edges take categorical atom-type and bond-type labels. These alphabets carry rich domain-specific semantics: amino-acid substitution rates have been estimated from evolutionary data, nucleotide mutations follow biochemical biases, and chemical valence rules impose hard constraints on which atom–bond combinations are valid. In contrast to text tokens (no metric) and quantized codes (learned metric), natural alphabets come with external, domain-grounded notions of similarity and validity that can inform the design of corruption processes, guide constrained sampling, and provide interpretable evaluation metrics [28]–[30].

4.1.0.2 Why tokenization matters.

The choice of tokenization is not a preprocessing step that can be separated from the diffusion model: it shapes the entire generative pipeline in at least four interrelated ways.

First, tokenization defines the topology of perturbations. A forward process that replaces one token with another is implicitly making a statement about which corruptions are "small". In continuous diffusion, this is handled by Gaussian noise, which has a natural notion of locality in \(\mathbb{R}^d\) (Section 3). In discrete diffusion, there is no default metric: the transition matrix \(\boldsymbol{Q}_t\) is the corruption topology. When \(\boldsymbol{Q}_t\) is uniform (any token can be replaced by any other with equal probability), the model treats all errors as equally severe. When \(\boldsymbol{Q}_t\) reflects codebook geometry or substitution-matrix semantics, the model can exploit the fact that some corruptions are more informative than others. And when \(\boldsymbol{Q}_t\) is absorbing (tokens are only ever replaced by \(\texttt{m}\)), the model sidesteps the topology question entirely by reducing denoising to a fill-in-the-blank task. Each of these choices leads to a qualitatively different denoising problem and generation behavior.

Second, tokenization determines the denoising difficulty curve. The difficulty of predicting a corrupted token depends on how much information the corruption has destroyed, which in turn depends on the relationship between the vocabulary structure and the noise process. In a vocabulary where semantically similar tokens cluster, a small amount of substitution noise can be "undone" by exploiting local context; in an unstructured vocabulary the same noise level may be catastrophic. Similarly, the ratio of vocabulary size \(K\) to sequence length \(L\) affects whether the model’s capacity bottleneck is in per-token prediction (large \(K\)) or in modeling inter-position dependencies (large \(L\)). These interactions between tokenization and noise schedule are often discovered empirically through expensive hyperparameter sweeps, but they are in principle predictable from properties of the token space (Section 4.5).

Third, tokenization governs downstream controllability and validity. Constrained generation, ensuring that outputs satisfy syntactic, chemical, or structural requirements, is fundamentally a question about which regions of \(\mathcal{V}^L\) are valid. A tokenization that aligns with domain constraints makes it easier to enforce validity during sampling: for example, if each token directly corresponds to a chemically meaningful fragment, valence rules can be checked locally at each position. Conversely, a tokenization that scatters structural information across many tokens makes constraint satisfaction a global, combinatorial problem.

Fourth, tokenization sets the effective sequence length and thus the computational cost. Finer-grained tokenizations (byte-level text, pixel-level images) produce longer sequences, increasing the cost of bidirectional attention at each denoising step. Coarser tokenizations (large-vocabulary subwords, low-resolution codebooks) produce shorter sequences but require the model to make more complex per-token predictions and may lose fine-grained detail. This length-complexity tradeoff is particularly acute for diffusion models, which perform multiple forward passes over the full sequence during generation.

Taken together, these four axes explain why tokenization is not merely a practical choice but a first-class design axis for discrete diffusion. The following subsections examine how each domain instantiates this tradeoff: text and code (Section 4.2), quantized media (Section 4.3), natural scientific alphabets (Section 4.4), and diagnostics for evaluating tokenization quality (Section 4.5).

4.2 Text and Code Tokenization↩︎

In natural language processing, subword segmentation methods such as Byte Pair Encoding (BPE), WordPiece, and Unigram are fundamental to large-scale autoregressive (AR) models. However, applying these tokenization schemes directly to discrete diffusion models introduces significant computational challenges. Modern language models often use vocabularies with more than 100,000 tokens. While AR models can accommodate this with a larger final softmax layer, discrete diffusion models require operations over the entire vocabulary space at each step. As a result, constructing and manipulating such large transition spaces leads to combinatorial explosion and numerical instability [25], [56]. Beyond computational concerns, tokenization also affects sequence structure and semantic consistency. Subword methods frequently split a single semantic unit into multiple tokens of varying lengths. When uniform diffusion noise is applied to these fragments, longer or more complex words are disproportionately corrupted, leading to semantic degradation [58]. Byte-level tokenization avoids this uneven fragmentation but significantly increases sequence length, resulting in higher memory and computational costs during diffusion.

These challenges reflect a deeper issue: text tokens are discrete categorical symbols without inherent continuous structure or meaningful distance metric. To avoid transitions between unrelated tokens during the diffusion process, recent approaches adopt a designated m token for corruption [57], [59]. Within this mask-based framework, vocabulary granularity plays a critical role. Coarser vocabularies allow the model to generate higher-level semantic units in fewer steps, but they also require more accurate predictions at each step. More importantly, token granularity shapes the refinement dynamics of diffusion. The noise schedule must be aligned with token structure to determine whether generation proceeds in a coarse-to-fine manner or follows a fine-to-coarse strategy [60], [61].

The limitations of standard tokenization become even more pronounced when extending discrete diffusion models to programming languages. Unlike natural language, which can tolerate minor grammatical errors, code is highly sensitive to structural correctness. A single incorrect token such as a missing parenthesis or improper indentation, can invalidate the entire Abstract Syntax Tree (AST). In this setting, discrete diffusion models offer distinct advantages. Unlike autoregressive models, which generate tokens sequentially from left to right, diffusion models operate over the entire sequence in a bidirectional manner. This enables iterative refinement of global structure and improves the handling of long-range dependencies and strict formatting constraints, as also observed in multimodal token alignment tasks [62], [63]. Furthermore, the iterative denoising process enables new forms of constrained decoding. If an invalid token is produced during an intermediate step, external tools such as static analyzers or compilers can provide corrective feedback. This allows the generation process to be dynamically constrained within the space of syntactically valid programs, offering a level of grammar-aware control that is difficult to achieve with unidirectional autoregressive models.

4.3 Quantized Tokens for Images, Audio, and Video↩︎

4.3.0.1 Vector quantization.

Unlike text, continuous signals such as images, audio, and video do not have a natural discrete alphabet. For these modalities, quantized tokenization converts continuous signals into discrete integer indices drawn from a finite codebook.

Vector Quantized Variational Autoencoder (VQ-VAE) [64] is the dominant approach to media tokenization. It follows an encoder-codebook-decoder blueprint. The encoder compresses the input into a spatial grid of continuous latent vectors, each of which is snapped to its nearest entry in a learned codebook. The corresponding embedding vectors are then passed to the decoder for reconstruction, while the discrete indices are consumed by downstream generative models for generation tasks [25], [61]. The training objective governs both reconstruction fidelity and codebook utilization. VQ-VAE uses a pixel-level L2 reconstruction loss [64], which tends to produce blurry outputs and encourages codebook collapse, where most codebook entries go unused and the effective vocabulary shrinks. VQ-GAN [65] addresses both issues by replacing the pixel-wise L2 loss with a composite of a perceptual loss penalizing differences in intermediate VGG feature activations, and a patch-based adversarial loss from a discriminator that pushes reconstructed patches to be locally indistinguishable from real data. Together, these losses sharpen reconstructions and improve codebook utilization. Subsequent work continued to refine codebook training objectives: [66] proposes regularization on both the prior-predicted token distribution gap and the stochastic mask during inference to mitigate codebook collapse, while SeQ-GAN [57] and AlignTok [62] incorporate semantic-enhanced perceptual losses to better capture linguistic structure in the codebook. MAGVIT [56] extends the spatial codebook to 3D, enabling spatial-temporal tokenization for video sequences.

4.3.0.2 Advanced quantization.

Beyond standard vector quantization, two alternative quantization families have been explored, each trading off reconstruction fidelity, token efficiency, and training stability differently.

Residual Vector Quantization (RVQ) [67], [68] extends VQ by iteratively quantizing the residual error of each previous quantization stage, producing a hierarchy of codebook indices that together represent the input with increasing fidelity. This improves reconstruction quality without increasing sequence length, but introduces a deeper token hierarchy that complicates generative modelling. [60] addresses this by directly predicting the cumulative vector embedding of all RVQ levels at each position simultaneously, decoupling inference steps from both sequence length and depth. For audio signals, the challenge is especially acute: the high information density of speech requires multiple RVQ codebook levels to maintain fidelity, inflating the token budget and undermining language modelling efficiency. SiTok [63] addresses this by replacing the deterministic RVQ reconstruction objective with a diffusion autoencoder that explicitly models quantization uncertainty, achieving high-fidelity reconstruction at an extremely low bit rate.

At the other end of the spectrum, scalar quantization (SQ) [59] independently maps each dimension of a continuous latent vector to the nearest level on a fixed axis-aligned grid, requiring no learned codebook and avoiding the collapse and cascade instabilities of VQ and RVQ. However, because its boundaries cannot adapt to the geometry of the learned representation, SQ ignores inter-dimensional correlations and can produce systematic reconstruction artifacts when semantically meaningful directions in latent space are not axis-aligned with the quantization grid. Together, RVQ and SQ illustrate the fundamental tradeoff in advanced quantization: richer hierarchical structure improves fidelity but complicates generation, while simpler fixed-grid schemes are stable but geometrically inflexible.

4.3.0.3 Codebook topology.

Beyond quantization fidelity, the topology of the codebook has important consequences for generative modeling. The semantic distance between two codebook entries measures the perceptual or linguistic difference between the inputs they represent. In standard VQ training, geometric proximity between embedding vectors is decoupled from semantic similarity, since the reconstruction objective encourages acoustic fidelity rather than linguistic organisation. An ideally structured codebook would satisfy the property that semantically similar inputs map to geometrically nearby entries, making geometric distance a reliable proxy for semantic distance. Approaches such as semantic distillation [69] and CTC-based supervision [63] explicitly reshape the codebook topology toward this goal, producing more regular and linguistically meaningful discrete spaces. This regularity can be directly exploited in discrete diffusion by designing structured, non-uniform corruption processes. Standard discrete diffusion assumes a uniform transition distribution, where each token is equally likely to be replaced by any other, and this discards all codebook geometry. In contrast, a structured transition matrix that assigns higher corruption probability to semantically nearby codes allows intermediate noisy tokens to retain information about their origins [15]. This produces better-conditioned denoising targets and stronger training signal at intermediate corruption levels, motivating joint design of the tokenizer and the generative model so that a semantically regular codebook can be fully exploited by a structured discrete diffusion process [60], [63].

4.4 Natural Discrete Structures in Science↩︎

Biological sequences provide the most direct instances of natural discrete alphabets for diffusion models. Proteins are strings over a 20-letter amino-acid vocabulary, while DNA and RNA are composed of 4-letter nucleotide alphabets. These alphabets are compact enough to define tractable categorical state spaces, yet the sequences they form exhibit long-range dependencies that pose a distinct challenge for iterative denoising. In proteins, residues that are distant in primary sequence can be proximal in three-dimensional structure, so a diffusion model must learn co-evolutionary couplings that span hundreds of positions to produce stable and functional folds [28], [70]. Analogous long-range constraints arise in genomic sequences, where regulatory elements such as promoters, enhancers, and splice sites can modulate gene expression across kilobase-scale distances [29], [71]. Because these dependencies are non-local and often non-contiguous, absorbing-state (masking) diffusion has been the dominant corruption strategy: it avoids introducing biologically implausible substitutions during the forward process and allows the reverse process to reconstruct correlated positions jointly rather than independently [5], [72]. Domain-specific priors can further inform the noise process; for example, evolutionary substitution matrices such as BLOSUM encode empirical amino-acid exchange abilities and could serve as structured transition kernels, though most current models still rely on uniform or absorbing transitions [15].

Molecules and materials present a complementary tokenization challenge because they may be represented either as linear molecular strings or as graphs. String-based representations, including SMILES, SELFIES, and fragment-sequence representations such as SAFE, make molecules compatible with sequence models and discrete diffusion over token sequences. Recent systems such as GenMol apply masked discrete diffusion to SAFE sequences, using fragments as basic building blocks and enabling fragment remasking for molecular optimization [73]. Graph-based representations instead map atoms to categorical node types and bonds to categorical edge types, making the state space a discrete combinatorial object with hard validity constraints: each atom must satisfy valence rules, the resulting graph should be connected, and stereochemical configurations should be self-consistent. DiGress [30] operates directly on the joint node-edge categorical space, applying discrete noise to both adjacency and feature matrices, while flow-matching alternatives such as discrete flow matching [74] parameterize interpolating paths between noise and data distributions over graphs. A persistent difficulty is that even small violations of chemical validity render the generated molecule meaningless, motivating guidance and projection mechanisms at inference time (discussed in Section 7). Beyond small molecules, crystalline materials introduce additional discrete variables such as space-group symmetries and Wyckoff positions [75], further expanding the categorical design space that tokenization must accommodate.

An important design axis for scientific domains is the granularity at which the object is discretized. For molecules, atom-level graph representations preserve full chemical expressiveness but expose the denoiser to tight local constraints, increasing the computational burden of multi-step denoising. Coarser fragment-, scaffold-, or motif-level alternatives decompose molecules into chemically meaningful substructures and treat each substructure as a generation unit, reducing effective sequence or graph size and often encoding local chemical validity by construction [76], [77]. For proteins, the analogous granularity choice appears in residue-level amino-acid tokens, structure-aware residue vocabularies, structural alphabets, and learned structure tokens. For example, Foldseek-style structural alphabets and SaProt-style structure-aware vocabularies augment each residue with local geometric information, while VQ-VAE-based protein structure tokenizers discretize continuous backbone or tertiary-structure geometry into finite codebooks [78]–[80]. These representations trade off reconstruction fidelity, structural locality, sequence length, and compatibility with sequence-based language or diffusion models. The choice among these representations determines the effective state-space size, the severity of validity constraints the model must learn or enforce, and the length of the sequence the diffusion process must denoise, echoing the vocabulary-granularity tradeoffs discussed for text in Section 4.2.

4.5 Diagnostics and Metrics for Tokenizations↩︎

Evaluating a tokenization scheme requires metrics that go beyond downstream task performance to characterize how well the discrete representation preserves information, respects semantic structure, and supports iterative denoising. We organize tokenization diagnostics into three complementary families: information-theoretic and compression metrics, geometric and topological probes, and diffusion-facing diagnostics that measure the interaction between the token space and the generative process. Table 3 turns these diagnostics into a reporting checklist. We distinguish established tokenizer metrics from proposed or synthesized diffusion-facing probes; the latter should be read as actionable recommendations for tokenizer–diffusion co-design rather than as validated predictors of generation quality. We flag the status of each family explicitly: the information/compression and reconstruction metrics below are established and widely reported, whereas the geometric/topological probes and the diffusion-facing diagnostics are largely our own proposals synthesized from scattered practice rather than standardized tools; where a proposed diagnostic has, to our knowledge, not yet been systematically measured, we say so at that point. This subsection is therefore part survey and part position; we have not validated the proposed diagnostics with new experiments, and we present them as recommendations for future tokenizer-diffusion co-design.

3pt

4.5.0.1 Information and compression metrics.

For text and code, where tokenization is typically deterministic (e.g., BPE or byte-level), the relevant information-theoretic measure is the token-level entropy of the data distribution under the chosen vocabulary. Larger vocabularies compress more information per token but increase the categorical state space the diffusion model must navigate. Bits-per-character (BPC) or bits-per-byte metrics normalize by input length rather than token count, enabling comparison across tokenizers with different vocabulary sizes [17].

For quantized media tokenizers, reconstruction fidelity is the most direct indicator of information retention. Standard pixel-level metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) quantify low-level fidelity but correlate poorly with human perception; learned perceptual metrics such as LPIPS, which measures the distance between intermediate feature activations of a pretrained network, capture semantic preservation more faithfully and are widely used both as training losses and evaluation metrics in VQ-GAN-style tokenizers [65]. Reconstruction Fréchet Inception Distance (rFID), i.e., FID computed between original and reconstructed images, has become the de facto aggregate metric for image tokenizer quality [25], [56]. For audio and video, analogous distributional metrics in domain-specific feature spaces serve a similar purpose. Beyond reconstruction, it is essential to assess how effectively the codebook is utilized. Codebook perplexity (defined as the exponentiated entropy of the empirical code usage distribution, \(\mathrm{Perp} = \exp\bigl(-\sum_{k=1}^{K} p(k) \log p(k)\bigr)\), where \(p(k)\) is the fraction of encoded tokens assigned to code \(k\)) measures the effective vocabulary size [64]. A codebook with \(K\) entries but perplexity much smaller than \(K\) suffers from codebook collapse. A complementary diagnostic is the dead code ratio: the fraction of entries with zero or near-zero usage, which directly quantifies wasted capacity. Regularization strategies that penalize low-entropy usage distributions [66] or semantic-enhanced objectives [57], [62] improve utilization, and monitoring perplexity throughout training serves as an early warning for collapse.

4.5.0.2 Geometric and topological diagnostics.

As discussed in Section 4.3, the topology of the codebook, whether geometric proximity among code embeddings reflects semantic similarity, has direct consequences for structured corruption in discrete diffusion. Here we focus on how to measure this alignment, a question that remains largely open, likely because tokenizers and diffusion models are typically developed and evaluated separately, with end-to-end generation quality serving as the only joint assessment.

We suggest that a natural diagnostic is neighborhood consistency: given a distance metric in the embedding space and a semantic similarity measure (e.g., perceptual distance between the inputs reconstructed by each code), one can compute the rank correlation between the two. A well-structured codebook exhibits high correlation, meaning that \(k\)-nearest neighbors in embedding space also reconstruct semantically similar inputs. Approaches that explicitly reshape codebook geometry via semantic distillation [69] or supervision from pretrained models [62], [63] could be evaluated by tracking this correlation before and after the intervention, though to our knowledge such systematic measurements have not yet been reported.

For scientific domains with natural discrete alphabets, external references for inter-token similarity already exist. For example, evolutionary substitution matrices such as BLOSUM for amino acids, or atom-type similarity based on chemical properties for molecular graphs. We note that these could serve as ground-truth references against which to calibrate the corruption process of domain-specific discrete diffusion models, providing a principled alternative to the unstructured transitions (uniform or absorbing) that current models employ [15]. Domain-specific generation metrics (validity, uniqueness, and novelty (V.U.N.) for molecules [30], or structure-prediction confidence (e.g., pLDDT) for proteins [70]) also serve as indirect tokenization diagnostics: if a token space cannot support the generation of valid structures, the representation itself may be deficient regardless of reconstruction fidelity.

4.5.0.3 Diffusion-facing diagnostics.

The ultimate test of a tokenization scheme is how it affects the learnability and quality of the denoising process. We highlight three diagnostics that directly probe this interaction.

The first is the reconstruction-generation gap. Comparing reconstruction quality (rFID, which isolates tokenizer fidelity) with generation quality (generation FID, which reflects the full tokenizer-plus-diffusion pipeline) reveals how much performance is lost in the generative modeling stage. A large gap indicates that the token space is difficult for the diffusion model to learn, even when the tokenizer itself reconstructs well. While both metrics are routinely reported, explicitly tracking their gap across tokenizer variants would directly isolate the effect of tokenization on generative performance [25], [56].

The second is the denoising loss curves across timesteps. Plotting the per-timestep denoising loss \(\mathcal{L}(t)\) as a function of the corruption level \(t\) reveals the difficulty profile of the diffusion task. For absorbing-state (masking) diffusion, the loss at each \(t\) reflects how difficult it is to predict masked tokens given the fraction \(1-\alpha_t\) that have been masked [5], [19]. Comparing these curves across tokenizers for the same data and architecture would isolate the effect of tokenization on denoising difficulty. Tokenizations that spread difficulty relatively uniformly across timesteps are generally preferable, as they provide informative training signal at all noise levels.

The third is the sensitivity to noise schedule. For a given tokenization, sweeping over schedule families (linear, cosine, log-linear) and measuring final sample quality reveals the degree of coupling between tokenizer design and schedule tuning [5]. Tokenizations that are robust across schedules simplify model development. Recent theoretical work on masked diffusion clarifies this interaction, showing that the optimal unmasking order depends on the conditional entropy structure of token predictions, and that random-order masking during training implicitly prepares the model for worst-case orderings [81].

Together, these three families of diagnostics provide a toolkit for evaluating and comparing tokenization schemes. The geometric and diffusion-facing diagnostics remain underexplored relative to reconstruction metrics: at present, there is no established way to predict how well a tokenizer will support diffusion-based generation without training a generative model end-to-end. Closing this gap, developing lightweight and model-free proxies for diffusion-readiness, is an important open problem. In the meantime, we encourage practitioners to report metrics from each family when proposing new tokenizers for discrete diffusion.

5 Core Formulations of Discrete Diffusion↩︎

This section presents the major formulation families for discrete diffusion in detail. Building on the notation introduced in Section 3, we move from general transition-matrix models to the specific corruption families that dominate practice, then to continuous-time and score-based perspectives, and finally to a unifying view that reveals the common structure underlying all variants.

5.1 Discrete-Time Markov Chains and Transition Matrices↩︎

5.1.0.1 The D3PM framework.

The Discrete Denoising Diffusion Probabilistic Model (D3PM) [15] provides the foundational framework for discrete diffusion. Recall from Section 3 that each token \(x_i\) (we again drop the position subscript when the per-position structure is clear) evolves under a forward Markov chain parameterized by a transition matrix \(\boldsymbol{Q}_t \in [0,1]^{K \times K}\), with cumulative corruption matrix \(\bar{\boldsymbol{Q}}_t = \boldsymbol{Q}_1 \cdots \boldsymbol{Q}_t\) giving the marginal \(q(x_t \mid x_0) = \mathrm{Cat}(x_t;\, \boldsymbol{e}_{x_0}^\top \bar{\boldsymbol{Q}}_t)\). The key technical insight of D3PM is that the reverse posterior, the distribution of the previous state given the current noisy state and the clean data, is available in closed form: \[\label{eq:reverse95posterior} q(x_{t-1} \mid x_t, x_0) \;=\; \frac{q(x_t \mid x_{t-1})\, q(x_{t-1} \mid x_0)}{q(x_t \mid x_0)} \;\propto\; \bigl[\boldsymbol{e}_{x_t}^\top \boldsymbol{Q}_t^\top\bigr]_{x_{t-1}} \cdot \bigl[\boldsymbol{e}_{x_0}^\top \bar{\boldsymbol{Q}}_{t-1}\bigr]_{x_{t-1}},\tag{6}\] where the denominator \(q(x_t \mid x_0)\) serves as a normalizing constant. Here the two bracketed terms are the \(x_{t-1}\)-th entries of the row vectors \(\boldsymbol{e}_{x_t}^\top \boldsymbol{Q}_t^\top\) and \(\boldsymbol{e}_{x_0}^\top \bar{\boldsymbol{Q}}_{t-1}\), and the product is taken elementwise over the \(K\) candidate values of \(x_{t-1}\); normalizing this length-\(K\) vector by \(q(x_t \mid x_0) = \sum_{x_{t-1}} (\cdot)\) recovers the categorical posterior. This posterior plays a central role: since the reverse process \(p_\theta(x_{t-1} \mid x_t)\) must approximate \(q(x_{t-1} \mid x_t, x_0)\) averaged over \(x_0\), one can parameterize the model to predict \(\hat{x}_0 = f_\theta(\boldsymbol{x}_t, t)\) and then plug the prediction into Eq. 6 to obtain the reverse transition analytically. This predict-\(x_0\)-then-posterior strategy avoids learning the reverse distribution directly and is the dominant parameterization across most discrete diffusion models.

5.1.0.2 Design space of the transition matrix.

The choice of \(\boldsymbol{Q}_t\) is the primary degree of freedom in the D3PM framework, encoding the model designer’s prior about what kinds of corruption are meaningful. D3PM [15] introduced and compared several designs.

The first is the uniform substitution. Each token is independently replaced by a uniformly random token with probability \(\beta_t\): \[\label{eq:q95uniform} \boldsymbol{Q}_t^{\text{uniform}} \;=\; (1 - \beta_t)\,\boldsymbol{I} \;+\; \frac{\beta_t}{K}\,\boldsymbol{1}\boldsymbol{1}^\top.\tag{7}\] This treats all token substitutions as equally plausible regardless of identity. The stationary distribution is uniform over \(\mathcal{V}\), and the cumulative corruption \(\bar{\boldsymbol{Q}}_t\) has a simple closed form. The uniform distribution is stationary because \(\boldsymbol{Q}_t^{\text{uniform}}\) is doubly stochastic, so it leaves the uniform vector \(\tfrac{1}{K}\boldsymbol{1}\) invariant; iterating drives any initial distribution toward \(\tfrac{1}{K}\boldsymbol{1}\) as \(\prod_s(1-\beta_s)\to 0\). While conceptually clean, uniform substitution makes denoising difficult in practice: the model must distinguish the true token from \(K-1\) equally likely alternatives at each corrupted position, without any structural signal about which alternatives are "close."

The second is the absorbing (masking). Each token is replaced by a special absorbing token \(\texttt{m} \notin \mathcal{V}\) with probability \(\beta_t\) and remains unchanged otherwise; once masked, a token stays masked (so \(\boldsymbol{Q}_t^{\text{absorb}}\) acts on the extended vocabulary \(\mathcal{V}_{\texttt{m}}=\{1,\dots,K,\texttt{m}\}\) of Table 1 and is therefore \((K{+}1)\times(K{+}1)\)): \[\label{eq:q95absorbing} [\boldsymbol{Q}_t^{\text{absorb}}]_{jk} \;=\; \begin{cases} 1 - \beta_t & \text{if } j = k \neq \texttt{m}, \\ \beta_t & \text{if } k = \texttt{m},\; j \neq \texttt{m}, \\ 1 & \text{if } j = k = \texttt{m}, \\ 0 & \text{otherwise}. \end{cases}\tag{8}\] The stationary distribution concentrates all mass on \(\texttt{m}\). This follows because \(\texttt{m}\) is absorbing (its row is \(\boldsymbol{e}_{\texttt{m}}^\top\)) and every non-mask state has probability \(\beta_t>0\) of transitioning to it, so the only invariant distribution is the point mass \(\boldsymbol{e}_{\texttt{m}}\). The denoising task reduces to fill-in-the-blank prediction at masked positions, a problem for which bidirectional Transformers are particularly well suited. This is the design that has scaled most successfully; we discuss it in detail in Section 5.2.

The third is the embedding-aware / structured transitions. The transition probabilities depend on token identity via an external similarity measure. For example, one can define \([\boldsymbol{Q}_t]_{jk} \propto \exp(-\|\boldsymbol{v}_j - \boldsymbol{v}_k\|^2 / \sigma_t^2)\) for token embeddings \(\boldsymbol{v}_j, \boldsymbol{v}_k\), so that tokens with similar embeddings are more likely substitution targets. This encodes the inductive bias that "nearby" tokens should be confused before "distant" ones, producing a coarse-to-fine corruption trajectory. D3PM explored such designs but found them harder to tune and less stable than absorbing-state corruption in practice [15]. Subsequent work has revisited structured transitions for domains with natural similarity metrics (e.g., amino-acid substitution matrices for proteins or edge-type similarities for molecular graphs), where the external metric provides a well-grounded notion of proximity [28], [29].

Multinomial diffusion [16] independently developed a closely related framework, defining the forward process as interpolation toward a uniform categorical distribution and connecting the reverse process to a discrete analogue of the score. Together, D3PM and multinomial diffusion established the transition-matrix view as the canonical starting point for discrete diffusion, and nearly all subsequent formulations can be understood as specializations, continuous-time limits, or reparameterizations of this general framework.

5.2 Masking and Absorbing-State Diffusion↩︎

5.2.0.1 Why absorbing-state diffusion dominates in practice.

Among the transition-matrix designs introduced in Section 5.1, absorbing-state (masking) corruption has emerged as the most dominant choice for large-scale discrete diffusion, particularly in language modeling. Three properties explain this dominance.

Simplicity. The forward process has a single free parameter per step (the masking rate \(\beta_t\)), and the cumulative masking probability \(\alpha_t = \prod_{s=1}^t (1 - \beta_s)\) gives the probability that a token survives to step \(t\) in closed form. The marginal at any step is a mixture: with probability \(\alpha_t\) the token equals \(x_0\), and with probability \(1 - \alpha_t\) it equals \(\texttt{m}\). Training reduces to a reweighted masked-language-modeling (MLM) loss, where the model predicts the identity of masked tokens given the unmasked context [5], [19].

Training stability. Because the denoising task at each timestep is a classification problem over \(\mathcal{V}\) (predict the original token at a masked position), the loss landscape is well behaved: cross-entropy over a softmax output, with no need for the complex reweighting schemes or auxiliary losses that can arise with substitution-based corruption. The MDLM [5] and MD4 [19] frameworks showed that the continuous-time ELBO for masked diffusion collapses to a simple integral of reweighted cross-entropy terms, making the connection to MLM losses explicit and facilitating stable large-scale training.

Compatibility with bidirectional architectures. Masked diffusion naturally pairs with bidirectional (full-attention) Transformers: at each denoising step, the model sees a partially masked sequence and must fill in the blanks using context from both directions. This is architecturally identical to BERT-style pre-training, allowing direct reuse of well-understood Transformer architectures, positional encodings, and training infrastructure [6], [7]. The bidirectional context also provides a natural advantage for infilling and editing tasks, where the model must condition on both a prefix and a suffix.

5.2.0.2 Masking schedules and variants.

Within the absorbing-state framework, the masking schedule \(\{\beta_t\}_{t=1}^T\) (equivalently, the survival probability \(\alpha_t\)) controls the rate at which tokens are corrupted and thus the difficulty profile of the denoising task across timesteps. Common choices include linear schedules (\(\alpha_t\) decreasing linearly from 1 to 0), cosine schedules (following a shifted cosine curve), and log-linear schedules. Recent theoretical work has shown that the cosine schedule is Fisher-Rao optimal for masked discrete diffusion under certain assumptions, providing a principled justification for what was originally an empirical heuristic [82].

Several variants extend basic per-token masking: Span masking corrupts contiguous spans of tokens rather than individual positions, encouraging the model to learn phrase-level coherence and reducing the effective number of independent predictions per step. Structured masking for images masks spatial patches or frequency bands rather than individual codes, inducing coarse-to-fine generation that respects the spatial structure of the token grid [25]. Partial masking [83] generalizes absorbing-state diffusion by allowing tokens to transition to states other than \(\texttt{m}\), for example, to a semantically similar token, while retaining the simplicity of a single-step masking rate. Any-order masking [84], [85] decouples the masking (corruption) process from the unmasking (generation) order, showing that the same trained model can support diverse generation orders at inference time, bridging masked diffusion and autoregressive generation within a single framework.

5.3 Substitution Noise and Mixture Corruptions↩︎

5.3.0.1 Uniform substitution as denoising under label noise.

When the forward process replaces a token with a uniformly random category (Eq. 7 ), the denoising task at each corrupted position can be interpreted as classification under heavy label noise: the model observes a "label" (the corrupted token \(x_t\)) that has been randomized with probability \(1-\alpha_t\) and must recover the true class \(x_0\). This connection to the noise-robust classification literature [15] has two practical consequences. First, the difficulty of denoising scales with the vocabulary size \(K\): for large \(K\), even a moderate corruption rate renders the corrupted token nearly uninformative about \(x_0\), collapsing the task to unconditional prediction from context alone. Second, unlike absorbing-state corruption, the model must learn to distinguish two cases, whether the observed token is the original (with probability \(\alpha_t\)) or a random substitute (with probability \(1-\alpha_t\)), which adds a source of estimation difficulty that does not arise with masking.

5.3.0.2 Hybrid and mixture corruptions.

Pure masking and pure substitution represent two extremes of a design spectrum. Several works have explored hybrid corruption processes that combine both mechanisms, typically by partitioning the corruption probability at each step into a masking component and a substitution component. The motivation is twofold. First, pure masking can lead to degenerate dynamics: at intermediate timesteps, the model only ever sees tokens that are either perfectly clean or fully masked, with no partially corrupted intermediate states. This means the model never practices recovering from plausible-but-wrong tokens, a scenario that arises naturally during iterative sampling when the model’s own predictions (which may be incorrect) are fed back as input at the next step. Adding substitution noise forces the model to handle corrupted-but-non-masked tokens, improving robustness to its own errors during generation [83]. Second, substitution noise provides a richer training signal at intermediate corruption levels. With pure masking, the loss at a given timestep \(t\) depends only on the fraction of tokens that are masked; all information about "how close" a corruption is to the clean data is lost. With substitution noise, corrupted tokens retain partial information (a random token is occasionally the correct one by chance), and the model can learn to exploit this signal.

Concrete instantiations include D3PM’s hybrid matrices [15], which interpolate between absorbing and uniform transitions, and more recent approaches that learn or anneal the mixing ratio during training. The general finding is that small amounts of substitution noise improve generation quality, but the optimal mixing ratio is domain- and scale-dependent, and pure masking remains competitive at large scale when combined with modern training recipes [5], [6].

5.4 Continuous-Time Discrete Diffusion↩︎

5.4.0.1 The CTMC formulation.

An alternative to discrete-time Markov chains is to define the forward process as a continuous-time Markov chain (CTMC) over the categorical state space \(\mathcal{V}\). Corruption is governed by a rate matrix \(\boldsymbol{R}_t \in \mathbb{R}^{K \times K}\) (with off-diagonal entries non-negative and rows summing to zero), so that the probability of a transition from state \(j\) to state \(k \neq j\) in an infinitesimal interval \([t, t+dt)\) is \([\boldsymbol{R}_t]_{jk}\,dt\). The transition probabilities over a finite interval are obtained by solving the Kolmogorov forward equation: \[\label{eq:ctmc95forward} \frac{d}{dt}\boldsymbol{P}_t \;=\; \boldsymbol{P}_t\,\boldsymbol{R}_t, \qquad \boldsymbol{P}_0 = \boldsymbol{I},\tag{9}\] where \([\boldsymbol{P}_t]_{jk} = q(x_t{=}k \mid x_0{=}j)\) is the transition probability matrix from \(0\) to \(t\). For time-homogeneous rates (\(\boldsymbol{R}_t = \boldsymbol{R}\) constant), this simplifies to the matrix exponential \(\boldsymbol{P}_t = \exp(t\boldsymbol{R})\). The continuous-time framework subsumes the discrete-time setting: a discrete-time chain with step matrices \(\boldsymbol{Q}_1, \dots, \boldsymbol{Q}_T\) can be recovered by choosing piecewise-constant rates \(\boldsymbol{R}_t = \log \boldsymbol{Q}_t / \Delta t\) over intervals of length \(\Delta t = 1\).

Score Entropy Discrete Diffusion (SEDD) [17] developed this perspective systematically, defining rate matrices for uniform, absorbing, and general transition processes and deriving training objectives directly in continuous time. The continuous-time viewpoint was also advanced independently by [86] and [20], with the latter connecting CTMCs to discrete flow matching.

5.4.0.2 Advantages and trade-offs.

The CTMC formulation offers several theoretical and practical advantages.

Unified objective derivation. Working in continuous time allows the ELBO (or its equivalent) to be derived as an integral rather than a sum, which often leads to simpler and more interpretable expressions. For masked diffusion, this integral form reveals that the training objective is invariant to the functional form of the noise schedule beyond its endpoints [5], [19], a result that is less transparent in the discrete-time derivation.

Flexible step discretization at inference. Because the model is trained in continuous time, it is not tied to a fixed number of steps \(T\). At inference time, one can choose any discretization of \([0, 1]\) into \(T'\) steps (where \(T'\) may be much smaller than the effective \(T\) used during training), trading off quality against speed without retraining. This decoupling of training and inference granularity is a key practical benefit, enabling adaptive step schedules and accelerated sampling [17], [20].

Connection to flows and optimal transport. Continuous-time discrete processes connect naturally to discrete flow matching [20], [87] and optimal-transport formulations [88], [89], enabling the transfer of tools and theory from these neighboring fields. The Ehrenfest process [90] provides a concrete bridge between discrete and continuous state spaces by modeling binary diffusion through a physical jump process.

The main trade-off is implementation complexity: matrix exponentials are expensive for large \(K\), and numerical integration of the Kolmogorov equation requires care. In practice, most CTMC-based models sidestep this by working with rate matrices that have special structure for which \(\exp(t\boldsymbol{R})\) is available in closed form [17], [20]. For absorbing-state rates, the matrix exponential reduces to the same simple mixture formula as in the discrete-time setting, eliminating any computational overhead.

5.5 Score/Ratio-Based Characterization of Reverse Transitions↩︎

5.5.0.1 From explicit posteriors to ratio functions.

The formulations in Sections 5.1-5.4 all ultimately require specifying the reverse transition \(p_\theta(x_{t-1} \mid x_t)\), typically via the predict-\(x_0\)-then-posterior strategy. An alternative approach, developed by [91] and [17], characterizes the reverse process not by predicting \(x_0\) explicitly but by learning ratio or score functions that capture the relative probabilities of reverse transitions directly.

For a CTMC with rate matrix \(\boldsymbol{R}_t\), the time-reversed process has rate matrix: \[\label{eq:reverse95rate} [\overleftarrow{\boldsymbol{R}}_t]_{jk} \;=\; \frac{q_t(k)}{q_t(j)}\,[\boldsymbol{R}_t]_{kj}, \qquad j \neq k,\tag{10}\] where \(q_t(j) = q(x_t = j)\) is the marginal probability of state \(j\) at time \(t\). The ratio \(q_t(k)/q_t(j)\) is the concrete score [91]: it plays the same role for categorical diffusion that the Stein score \(\nabla_{\boldsymbol{z}} \log q_t(\boldsymbol{z})\) plays for continuous diffusion, characterizing how the data distribution shapes the reverse dynamics. SEDD [17] defines a score entropy loss that trains a network \(s_\theta(x_t, t)\) to estimate these ratios for all pairs of states, and shows that the resulting model admits an exact likelihood via a change-of-variables identity for CTMCs. The concrete score matching framework [91] arrives at a similar objective from a different derivation, defining a score function on the simplex of categorical distributions and training it via a denoising score-matching loss.

5.5.0.2 Connection to transition-matrix and CTMC views.

The ratio-based perspective is not a separate model family but an alternative parameterization of the same reverse process. To see this, note that the predict-\(x_0\) parameterization implicitly computes the ratios in Eq. 10 : given a predicted clean distribution \(\hat{p}(x_0 \mid x_t)\), one can compute the marginal \(\hat{q}_t(k) = \sum_{x_0} q(x_t{=}k \mid x_0)\,\hat{p}(x_0 \mid x_t)\) and form the ratio. Conversely, the ratio function can be used to recover the reverse transition probabilities in discrete time via Eq. 6 . The practical implication is that the choice between predict-\(x_0\), predict-logits, and predict-ratios is a question of parameterization convenience rather than fundamental model expressivity. This equivalence should be read as holding for a sufficiently expressive denoiser under the same factorization assumptions: the parameterizations are interconvertible at the level of the exact reverse transition, but with finite capacity, a fixed per-position factorization, and approximate optimization, the choice does affect the induced inductive bias, the tightness of the variational bound, and optimization stability (Section 6.3), as the SEDD and MD4 results below illustrate. Different parameterizations have different inductive biases: predicting \(x_0\) directly ties the model to a classification-style output, while predicting ratios allows the model to express relative preferences between states without committing to an absolute distribution. SEDD [17] found that the ratio parameterization yields tighter likelihood bounds than the \(x_0\) parameterization for the same model capacity, particularly under uniform (non-absorbing) corruption. Target concrete score matching [92] further refines this approach by providing a holistic framework that unifies several prior score-matching objectives under a single target formulation.

5.6 A Unifying View↩︎

5.6.0.1 Common structure.

Despite the diversity of formulations reviewed above, every discrete diffusion model instantiates the same four-component structure. (1) Corruption operator: a forward process that maps clean data \(\boldsymbol{x}_0\) to noisy data \(\boldsymbol{x}_t\). This is specified by a transition matrix \(\boldsymbol{Q}_t\) (discrete time) or a rate matrix \(\boldsymbol{R}_t\) (continuous time), with masking, uniform substitution, structured transitions, and hybrids as specific instantiations. (2) Denoiser parameterization: a neural network that, given \((\boldsymbol{x}_t, t)\) and optionally a conditioning signal \(\boldsymbol{c}\), predicts information about the clean data. The output may be a distribution over \(x_0\), a set of logits, or a vector of log-ratios, depending on the parameterization choice. (3) Training objective: a loss function derived from a variational bound, a score-matching identity, or a simplified surrogate. The objective determines how the denoiser is trained and what aspects of the reverse process it is optimized to approximate. (4) Sampler: an inference algorithm that, starting from the stationary distribution, iteratively applies the learned reverse transitions to produce a clean sample. The sampler may follow the reverse chain exactly, use confidence-based remasking, apply guidance, or employ acceleration techniques.

5.6.0.2 Mapping across formulations.

Table 4 summarizes how the major formulation families instantiate each component. The key takeaway is that the distinctions between formulations lie primarily in the corruption operator and the denoiser parameterization; the training objectives and samplers are often interchangeable or can be mixed across families. For example, a model trained with a score-entropy loss (ratio parameterization) can be sampled using the same ancestral-sampling or \(\tau\)-leaping algorithms as a model trained with a cross-entropy loss (\(x_0\) parameterization), and vice versa [17], [20].

This unifying perspective has several practical implications. First, it clarifies that the choice of corruption operator (masking vs.substitution vs.structured) is orthogonal to the choice of parameterization (predict \(x_0\) vs.predict ratios): any combination is in principle valid, and the optimal pairing is an empirical question that depends on the domain, vocabulary size, and computational budget. Second, it reveals that many apparently distinct models differ only in one or two components: for example, MDLM and SEDD use the same absorbing corruption but differ in parameterization (predict \(x_0\) vs.predict ratios) and objective (reweighted MLM vs.score entropy). Third, it suggests that future work should explore the off-diagonal cells of the table, combinations that have not yet been tried, as a systematic way to discover improved designs. We defer the detailed discussion of training objectives (component 3) to Section 6 and inference algorithms (component 4) to Section 7.

6 Training Objectives and Parameterizations↩︎

This section systematizes how discrete diffusion models are trained, proceeding from the most principled objective to the most practical. Section 6.1 develops the discrete ELBO and likelihood-based training; Section 6.2 shows how it simplifies to the reweighted denoising losses used in practice; and Section 6.3 covers the complementary score- and ratio-matching objectives. Section 6.4 then discusses what the reverse network should predict, Section 6.5 treats conditioning and guidance introduced at training time, and Section 6.6 collects practical recipes that determine whether these objectives train stably at scale.

4pt

6.1 Discrete ELBO and Likelihood Training↩︎

Likelihood-based training provides the most principled starting point for discrete diffusion models. For a discrete-time forward Markov chain, the standard variational decomposition can be written, equivalently, as a per-sample lower bound on \(\log p_\theta(x_0)\): \[\log p_\theta(x_0) \ge \mathcal{L}_{\mathrm{rec}} - \sum_{t=2}^{T}\mathcal{L}_t - \mathcal{L}_{\mathrm{prior}},\] where \[\mathcal{L}_{\mathrm{rec}} := \mathbb{E}_{q(x_1\mid x_0)} \!\left[\log p_\theta(x_0\mid x_1)\right],\] \[\mathcal{L}_t := \mathbb{E}_{q(x_t\mid x_0)} \!\left[ \mathrm{KL}\!\left( q(x_{t-1}\mid x_t,x_0)\,\|\,p_\theta(x_{t-1}\mid x_t) \right) \right],\] \[\mathcal{L}_{\mathrm{prior}} := \mathrm{KL}\!\left(q(x_T\mid x_0)\,\|\,p(x_T)\right).\] This form makes two points explicit. First, the training problem is governed by a sequence of reverse-time matching terms, together with a reconstruction term and a terminal prior-matching term. Second, tractable marginals \(q(x_t\mid x_0)\) are indispensable in practice: once they are available in closed form, the posterior \(q(x_{t-1}\mid x_t,x_0)\) often becomes tractable as well, which turns the objective into a denoising problem with analytically specified weights.

For masked diffusion, the above structure simplifies dramatically. In MD4, the continuous-time objective is written exactly as \[\mathcal{L}_\infty = \int_{t(1)}^{1} \frac{\alpha_t'}{1-\alpha_t}\, \mathbb{E}_{q(x_t\mid x_0)} \!\left[ \delta_{x_t,\texttt{m}}\,x_0^\top \log \mu_\theta(x_t,t) \right]dt,\] where \(\mu_\theta(x_t,t)\) denotes the model’s predicted categorical distribution over the clean token (the \(x_0\)-prediction \(\boldsymbol{\pi}_\theta\) of Section 3), and \(\delta_{x_t,\texttt{m}}\) restricts the loss to masked positions. Here \(t(1)\) is the lower endpoint of the time grid (the smallest noise level at which the loss is evaluated), so the integral runs over the corruption interval \([t(1),1]\); \(\alpha_t'=\mathrm{d}\alpha_t/\mathrm{d}t<0\) since the survival probability \(\alpha_t\) decreases in \(t\), which makes the weight \(\alpha_t'/(1-\alpha_t)\) negative and \(\mathcal{L}_\infty\) itself negative (it is an expected log-likelihood). After taking \(t(1)\to 0\), the ELBO reduces to \(-\mathcal{L}_\infty\) up to the convention adopted in the paper [19]; equivalently, \(-\mathcal{L}_\infty \le \log p_\theta(x_0)\) is a variational lower bound, and training minimizes \(\mathcal{L}_\infty\) (i.e.maximizes the bound). This formulation sharply clarifies why masked diffusion is so closely tied to reweighted cross-entropy training: only masked positions contribute, and the schedule enters through the scalar factor \(\alpha_t'/(1-\alpha_t)\). MDLM reaches a closely related conclusion from a different derivational route, showing that the continuous-time ELBO collapses to a mixture of masked-language-modeling losses and is invariant to the functional form of the noise schedule beyond its endpoints [5]. Taken together, these papers make it clear that the ELBO in masked discrete diffusion is not merely tractable; in important cases it is substantially simpler than its general Markov-chain presentation suggests.

Plaid [18] provides the strongest case for taking likelihood training seriously as an end in itself. Rather than abandoning the full variational objective, it optimizes a VLB throughout training and evaluation, uses a learned noise schedule, and shows that diffusion language models exhibit clean scaling trends under maximum-likelihood training. At the same time, it also makes the cost of this commitment plain: principled likelihood optimization is feasible, but much more compute-hungry than standard autoregressive training. This is precisely why many discrete diffusion papers prefer simplified denoising losses in practice, even when the ELBO remains the conceptual starting point.

Recent work has therefore focused on narrowing the gap between variational rigor and optimization convenience rather than treating them as mutually exclusive. [94] derive a family of increasingly tight time-dependent lower bounds and show that common reweighted diffusion losses can be interpreted as weighted sums of improved variational bounds rather than as purely heuristic surrogates. [95] extend the theory further for discrete CTMCs, deriving cross-entropy/KL identities that lead to a tighter and cheaper perplexity upper bound \(J_2\), thereby reducing one of the main practical barriers to strict likelihood-style evaluation in large-vocabulary settings.

A separate line of work enriches the ELBO itself by introducing latent variables. [96] augment masked diffusion with a global latent variable \(z\) and obtain a variational NELBO with an additional KL regularizer, using the latent to capture inter-token dependencies that factorized token predictions cannot express directly. [97] extend this idea into a hierarchical variational construction, proposing a "Double ELBO" together with KL annealing to stabilize learning and reduce posterior collapse. These models preserve the likelihood-based perspective, but relax the assumption that the reverse model should be purely token-factorized.

At the same time, the literature also contains a clear counterpoint to strict ELBO optimization. [98] show that, for Uniform State Diffusion Models, a highly simplified denoising objective can scale more favorably and yield better generation quality even when the formal variational bound becomes looser. The resulting picture is therefore not one of replacement, but of tension: ELBO-based training remains the cleanest route to likelihood estimation and principled comparison, while simplified objectives often provide a better optimization interface when generation quality and scalability are the primary concerns.

6.2 Simplified Denoising Objectives↩︎

In practice, the dominant training view is to sample a corruption level, corrupt the clean sequence, and train the model to recover the original tokens from the noisy input. This perspective is broad enough to cover masked diffusion, uniform-state corruption, and several hybrid designs that operate in continuous latent spaces but decode with categorical targets. Its appeal is obvious: compared with the full ELBO, it exposes a single supervised signal that is easy to implement, easy to batch, and often better aligned with the reverse model one actually uses at inference time.

MDLM [5] is the clearest example of how such simplification can still remain theoretically grounded. It shows that the continuous-time ELBO of masked diffusion is equivalent to a weighted masked-language-modeling objective, thereby turning a generative diffusion objective into a randomized denoising problem that can be trained with standard cross-entropy machinery. [94] then systematize this viewpoint by deriving explicit reweighting schemes and proving that monotonically increasing weights still correspond to valid variational bounds. Their analysis is useful because it reframes weighting not as an afterthought, but as a design axis with both theoretical and empirical consequences.

Several works simplify the denoising target even further by restricting supervision to genuinely corrupted positions. [99] derive, from a route-and-denoise perspective, a reweighted cross-entropy objective that is evaluated only on noisy token positions, and they find linear reweighting to work particularly well in practice. [98] push this principle to an extreme in Uniform State Diffusion Models: the loss is applied only to noise-replaced tokens, while an additional anti-uniform sharpening term is introduced to prevent the predictions from drifting toward an overly flat distribution. The conceptual shift here is small but important. Once the model is only penalized where corruption actually destroyed information, the objective becomes much closer in spirit to masked language modeling than to a generic per-position reconstruction loss.

A related but distinct simplification appears in [100]. Instead of keeping a continuous diffusion loss and then rounding back to discrete tokens, they learn a projection to vocabulary space and optimize a token-level cross-entropy directly. This removes the mismatch between a continuous reconstruction target and a discrete generation goal, and it also eliminates the need for heuristic rounding procedures. In effect, it is a simplified denoising objective designed not for masked diffusion proper, but for continuous latent diffusion models that ultimately serve discrete data.

One of the most striking conceptual simplifications comes from [101]. They show that, for masked diffusion, the NELBO can be rewritten purely in terms of the number of masked tokens \(n\), together with a natural \(1/n\) weighting. Under this view, the scalar time variable is not fundamental: what matters is the masking pattern itself and, in particular, how many tokens remain unresolved. This is a strong result because it explains why time conditioning can sometimes be weakened or even omitted without destroying the training signal.

Even so, noise-level conditioning remains useful in many practical settings. When a single denoiser must operate across corruption levels ranging from almost clean to nearly fully destroyed, explicit time embeddings often make optimization easier and improve calibration across steps. The key point is not that time conditioning is always necessary, but that its role is contingent. In some models it genuinely helps the network adapt across noise regimes; in others, especially masked models with sufficiently informative corruption patterns, the dominant variable is not continuous time but the combinatorial structure of which tokens are missing. This is precisely why simplified denoising objectives have become the default engineering choice across much of the literature: they preserve the central denoising semantics while discarding parts of the variational machinery that are not always needed for good optimization.

6.3 Score- and Ratio-Matching Objectives↩︎

A second family of objectives learns reverse dynamics through scores or probability ratios rather than directly through a categorical denoising distribution. In discrete state spaces, this amounts to modeling relative plausibilities of local transitions, which can be more natural for CTMC-style sampling rules than predicting a normalized clean-token distribution outright. Conceptually, this differs from the denoising objectives of Section 6.2, even though the two views can coincide under specific parameterizations.

For masked diffusion, MD4 gives an exact bridge between the two. In the notation of [19], for a mask state and any non-mask class \(j\), \[s_\theta(m,t)_j = \frac{\alpha_t}{1-\alpha_t}\,\mu_\theta(\texttt{m},t)_j, \qquad j\neq m.\] This identity is more than a cosmetic reparameterization. It shows that, in masked diffusion, score-based training can collapse back to a mean-parameterized denoising objective once the reverse model is constrained to be compatible with the forward process. The same paper also points out the corresponding failure mode: if the score model is left unconstrained, as in some earlier formulations, the learned reverse process need not remain consistent with the forward dynamics, which can lead to instability and poorer behavior in practice.

This observation explains why score- and ratio-based methods have often looked more attractive in theory than in implementation. They are closely tied to likelihood and continuous-time sampling, but a naive parameterization can be both expensive and fragile when the vocabulary is large. [95] address this bottleneck by showing that ratio matching can be implemented efficiently through a denoising cross-entropy objective in CEDD. In that sense, their contribution is not just a faster estimator; it is a practical reconciliation of two training philosophies that had previously seemed to pull in opposite directions.

The practical trade-off therefore becomes clearer. Compared with simplified denoising, score- and ratio-matching can provide a tighter link to continuous-time likelihood theory and reverse-process construction. Compared with direct ELBO optimization, they can be more operational once expressed in a scalable parameterization. The remaining difficulty is not the idea itself, but the form in which it is trained: unconstrained scores are elegant but brittle, whereas denoising-compatible ratio parameterizations are less free-form but much easier to scale.

6.4 Parameterizations of the Reverse Model↩︎

A central design choice in discrete diffusion is what the reverse network should predict. One option is to predict one-step reverse transitions directly, i.e., to model \(p_\theta(x_{t-1}\mid x_t)\) as the primitive object. The alternative, which has become much more common, is to predict the clean sample or its categorical distribution and then obtain the reverse step from the known structure of the forward process. In discrete settings this latter choice is especially appealing, because the corruption mechanism is often simple enough that much of the reverse transition can be reconstructed analytically once a prediction of \(x_0\) is available.

MDLM [5] exemplifies this predict-\(x_0\) perspective through its SUBS parameterization. The construction is deliberately structured: already revealed tokens are carried forward, remasking is forbidden, and the model is only asked to make meaningful predictions where information is still missing. These constraints simplify the objective analytically and, just as importantly, make the reverse process easier to cache and reuse at inference time. The paper further shows that explicit time conditioning can be removed in this setup, which is a particularly strong statement about how much of the reverse dynamics is already encoded by the masking pattern itself.

Other papers broaden the parameterization space rather than narrowing it. [99] expose a hidden route-and-denoise decomposition, in which each token is first routed, kept noisy or selected for denoising, and only then decoded through a predict-\(x_0\) mechanism. This introduces an additional latent decision into the reverse model, and it also opens the door to using different routing behavior at training and sampling time. [96] and [97] add yet another layer by injecting a global continuous latent variable \(z\) through adaptive layer normalization. In both cases, the local token predictor remains factorized conditional on \(z\), while marginalization over \(z\) restores some degree of global dependence across positions. The result is a reverse model that remains operationally simple at the token level but is no longer purely local in its representational capacity.

A related question is whether the network should predict continuous vectors or categorical logits. Plaid [18] argues strongly for the latter. Rather than forcing the model to reproduce precise embedding vectors and then infer token identities indirectly, it reparameterizes the denoiser to output a softmax over the vocabulary. This uses model capacity more efficiently and improves both likelihood and generation quality. [100] reach a similar conclusion in a different setting: replacing continuous \(\ell_2\) reconstruction and nearest-neighbor rounding with direct logit prediction substantially improves training stability and allows pretrained masked language models to be incorporated more naturally. In discrete diffusion, then, direct categorical modeling is not merely convenient; it is often the cleaner parameterization.

The same predict-\(x_0\) output can also be repurposed into other reverse-time quantities. [95] keep the model in the predict-\(x_0\) regime during training, but convert the resulting predictions into scores or ratios at sampling time using known conditional ratios from the matrix exponential. This is an instructive design choice: instead of learning a harder object directly, the model learns a simpler one and lets the analytic structure of the forward process do the rest.

Finally, self-conditioning has emerged as an effective auxiliary parameterization trick rather than a separate modeling family. In Plaid, the model feeds its own current estimate back into later denoising evaluations, which improves both generation quality and held-out likelihood [18]. Conceptually, this gives the reverse model access to its own evolving hypothesis about the clean sample, allowing refinement without changing the basic sampling framework. Latent-augmented denoisers such as VMD and VADD play a related role at a larger scale: they supplement token-level predictions with an auxiliary channel that carries global information across steps. Seen this way, reverse-model parameterization is less about choosing a single target and more about deciding which parts of the reverse process should be learned explicitly and which should be provided by structure.

6.5 Conditioning and Guidance During Training↩︎

Conditioning in discrete diffusion can be introduced either directly in the training architecture or indirectly through guidance mechanisms applied at inference time. The surveyed literature is notable in that it obtains a large amount of controllability without relying exclusively on dedicated conditional pretraining. This distinguishes diffusion from many conditional autoregressive pipelines, where the architecture often encodes the condition more explicitly from the outset.

Plaid [18] is the clearest example of this inference-time view. It is trained unconditionally, yet supports a wide range of zero-shot control settings through token guidance. The key idea is to manipulate the denoiser’s own token-level predictions during sampling so as to realize span constraints, lexical conditions, negation, and other compositional requirements. This is attractive for survey purposes because it shows that conditioning in discrete diffusion need not be reduced to a choice between prefix prompting and classifier guidance; the denoising model itself can expose enough structure to support surprisingly flexible constraint injection after pretraining.

A more explicit guidance route is taken by [100], who use plug-and-play classifier guidance in continuous latent space for controllable text generation. Their experiments span semantic content, part-of-speech patterns, syntax trees, syntax spans, and length. Although the implementation is different from Plaid’s token guidance, both works illustrate the same broader principle: the reverse process can be steered by an auxiliary signal that is external to the unconditional denoiser, provided that the denoiser remains differentiable and interpretable enough for that signal to act on.

An analogue of classifier-free guidance is explored by [98]. Their unsupervised variant constructs an unconditional branch by replacing conditioning inputs with random tokens, making it possible to recover CFG-like behavior without paired conditional data. This line is especially relevant for discrete diffusion because it turns "dropout conditioning" into a training-time mechanism whose purpose is realized mainly at inference: by learning both conditional and weakened-condition behaviors within one model, the sampler gains a simple control knob for trading off fidelity against adherence to the requested condition.

More broadly, the conditioning mechanisms used in this literature can be grouped by where the condition enters. In some cases it is encoded as observed tokens or spans, in some it is supplied through an external classifier or gradient signal, and in others it is represented implicitly through conditional versus weakened-conditioning branches. The common feature is iterative intervention. Since diffusion refines samples over multiple denoising steps, guidance is not a one-shot adjustment but a repeated modulation of the reverse trajectory. That iterative character is one reason discrete diffusion can accommodate diverse forms of control even when the underlying denoiser is trained in a comparatively generic way.

6.6 Practical Training Recipes↩︎

In discrete diffusion, training recipes matter nearly as much as the nominal objective. The first recurring theme is that the noise schedule acts as a curriculum. It determines how corruption accumulates and the order in which information tends to be reconstructed during generation. MD4 shows that schedule choice affects optimization variance and discretization quality even when the underlying continuous-time objective is invariant to the schedule shape [19]. [102] make this point especially concrete by optimizing the sampling schedule through an energy-minimization view grounded in discrete optimal transport, using a Beta-CDF parameterization that can be tuned post hoc without retraining. This is a particularly elegant result because it turns schedule design into a lightweight yet principled control interface.

Several papers go further and design the schedule to induce a specific generation order. [103] mask common token categories first, which yields a corresponding common-first reverse process. [100] propose a linguistically informed soft-masking policy based on TF-IDF and information entropy, so that more informative words are corrupted earlier and therefore reconstructed later. These choices are not just cosmetic. They encode beliefs about which parts of the sample should be fixed early and which should remain unresolved until richer context is available.

A second major theme is variance control and optimization stability. [19] use antithetic time sampling, cosine discretization, an \(\epsilon\)-shifted schedule, and EMA with decay \(0.9999\). [104] provide the most explicit variance decomposition, separating masking-pattern noise, masking-rate noise, and data noise, and then reducing them with P-POTS timestep sampling and MIRROR antithetic masking. [94] show that a simple constant weighting across noise levels can itself serve as a strong and stable recipe. [99] combine EMA, checkpoint averaging, label smoothing, and adaptive top-\(k\) routing, demonstrating that strong results can coexist with extremely short sampling chains. Read together, these works suggest that the practical bottleneck in masked diffusion is often not expressivity but the stochasticity of the training estimator.

Masking policy is another practical lever that has received increasingly principled treatment. [105] argue that the masked-diffusion objective implicitly contains a regularizing effect arising from unidentifiable masked inputs, and use this analysis to recommend a narrower, signal-favorable masking-rate window. This is a useful example of how seemingly low-level training heuristics can be justified by decomposing the objective into distinct signal and noise regimes.

Numerical precision also matters more than one might expect. [101] show that float32 Gumbel-based categorical sampling induces a truncated distribution and thereby lowers the effective temperature in masked diffusion generation. Their recommendation to use float64 for this operation is practically important, because it reveals that standard mixed-precision recipes may interact with discrete sampling in nontrivial ways. In other words, discrete diffusion inherits many large-scale training habits from contemporary language modeling, but not all of them transfer unchanged.

Plaid contributes a complementary set of large-scale lessons [18]. It learns the noise schedule jointly with the model, explicitly tying schedule optimization to variance reduction, and its scaling analysis suggests that compute-optimal diffusion language models should be smaller and trained for longer than their autoregressive counterparts. This is one of the clearest cases where the optimal training recipe is not simply "copy the AR recipe and change the objective"; the diffusion setting induces its own allocation of model size, training length, and denoising difficulty.

Finally, alignment and post-training remain comparatively underexplored, but the existing results already suggest a plausible roadmap. At a high level, supervised fine-tuning can be cast as denoising under task-specific partial observations, while preference optimization can be applied to completed samples or, more ambitiously, to intermediate refinement trajectories. Guidance-based mechanisms provide an additional lightweight option: some forms of alignment may be easier to implement by steering the sampler than by re-optimizing the denoiser from scratch. For now, the literature offers more ingredients than settled recipes, but those ingredients already indicate that post-training for diffusion models will likely involve both objective design and sampler design, rather than treating the denoiser alone as the whole system.

7 Inference Algorithms and Efficiency↩︎

Discrete diffusion inference can be viewed as a policy layer on top of the learned denoiser. In autoregressive models, generation follows a fixed append-and-cache order. In masked and other discrete diffusion models, the reverse process exposes several additional control variables. The sampler can choose which time points to visit, which tokens to reveal or revisit, which positions are grouped into a block, how external rewards or constraints steer the path, and which parts of the Transformer computation are reused or skipped. These choices expand the design space, but they also introduce characteristic failure modes: aggressive parallel updates may break dependencies, confidence scores can be miscalibrated, guidance can alter the reverse dynamics, and nominal NFE reductions may not translate into wall-clock speed.

The discussion follows the main degrees of freedom exposed by the reverse process. It begins with ancestral sampling and step schedules (Section 7.1), where the central issue is how the chain is discretized, corrected, or shortened. It then moves from time to positions: confidence-based remasking (Section 7.2) decides which tokens to trust, reopen, or advance in parallel. Semi-autoregressive and block updates (Section 7.3) change the dependency structure itself, while guidance and constraints at inference (Section 7.4) add external objectives, from rewards and search to hard validity checks and posterior conditioning. The section closes with acceleration (Section 7.5), where these algorithmic choices meet distillation, cache reuse, selective computation, speculative verification, and hardware-aware serving.

4pt

7.1 Ancestral Sampling and Step Schedules↩︎

7.1.0.1 Baseline reverse-time sampling.

A basic axis of discrete diffusion inference is how the reverse process is traversed in time. Even when the denoiser is fixed, a discrete diffusion sampler decides how many states to visit, where those states lie, and whether each reverse update is a simple local step or a corrected numerical approximation. Early discrete image-generation work illustrates this point. Discrete Predictor-Corrector samplers augment masked-token reverse updates with correction steps and show that sampler design can change sample quality at fixed model scale; for example, DPC variants improve ImageNet FID substantially under different NFE budgets [106]. Informed correctors develop the same principle for discrete diffusion: the reverse chain can be improved by adding correction dynamics rather than simply replaying the base transition [107]. These methods separate two questions that are often conflated: how good is the learned denoiser, and how well does the inference procedure use it?

Other works improve the local numerical update rather than only removing steps. High-order solvers for discrete diffusion treat sampling as numerical integration over discrete-state dynamics, yielding much better likelihood or perplexity at low NFE than Euler- or tau-leaping-style updates [108]. Neural sampler formulations such as MDNS cast masked diffusion sampling as stochastic optimal control, shifting the objective from matching a hand-designed reverse chain to learning or optimizing a path policy [109]. Reversible diffusion decoding and approximate joint sampling extend this path-centric view by asking whether the sampler can use additional reversible or joint structure to reduce wasted updates [110], [111]. These papers suggest that discrete diffusion inference is closer to controlled numerical simulation than to a single canonical decoding loop.

7.1.0.2 Step schedules and effective computation.

A complementary line reduces the number of useful denoiser evaluations by changing the time representation. Discrete Non-Markov Diffusion Models pre-sample token transition times, so the denoiser is called only when a token can actually change; in translation experiments, this collapses thousand-step nominal chains into a few dozen effective denoiser calls while preserving BLEU [112]. Few-shot temporal pruning reaches a similar conclusion empirically: many text-diffusion timesteps are redundant, and a small validation set can identify short schedules that outperform naive uniform pruning [113]. The older de-randomization formulation and its later Non-Markov version can be understood as part of the same general pattern: discrete trajectories often contain idle regions, and the inference algorithm need not treat all nominal steps uniformly [114].

Schedule design also reappears in modern masked diffusion language models. Jump Your Steps optimizes where to place a fixed number of denoising steps, while Plan for Speed uses dilated schedules to reduce the number of decoding rounds under a structured unmasking plan [115], [116]. Path Planning for masked diffusion generalizes this idea from time alone to a planner that decides which positions to unmask or remask along the reverse path, with applications ranging from language to biological sequences [117]. This creates a close connection between schedules and token-selection methods: schedules define the temporal scaffold, but effective schedules often encode assumptions about which positions will be resolved together.

This literature also clarifies that "fewer steps" can mean several different things. A method may use fewer nominal timesteps, fewer actual denoiser calls, a higher-order update with lower error per call, or a path planner that avoids unproductive parts of the trajectory. These notions have different implications for likelihood, sample quality, and serving latency. Comparisons are therefore more informative when they report not only NFE, but also whether the method changes the transition family, uses extra corrector networks or validators, and preserves quality under the same model and hardware.

7.2 Confidence-Based Remasking (Iterative Parallel Decoding)↩︎

7.2.0.1 Core confidence-based remasking.

Once the state is a partially masked sequence, a central inference axis becomes spatial: which positions are filled, reopened, or trusted at each step. This is one of the main freedoms that distinguishes masked diffusion from left-to-right generation. Train for the worst, plan for the best highlights this phenomenon: masked diffusion models are trained on hard arbitrary masks, but at inference time a sampler can choose easier subproblems first [81]. Simple token-ordering rules based on probability margins can substantially improve tasks such as Sudoku and code/math generation. These results suggest that confidence is useful because generation order itself is an inference-time resource.

This raises the question of how aggressively tokens can be decoded in parallel. Entropy-bounded unmasking links parallel-update risk to predictive entropy and dependence error, choosing more tokens only when the distribution is sufficiently certain [118]. KLASS adds a stability signal: a token is more reliable when it has both high confidence and a stable predicted distribution across consecutive denoising steps [119]. Conditional-independence testing attacks the same issue more directly by asking whether simultaneously decoded positions are conditionally safe to sample together [120]. These methods point to a useful taxonomy of token selection criteria: confidence estimates whether the current top token is plausible, entropy estimates local uncertainty, KL or trajectory stability estimates whether the decision is settled, and independence tests estimate whether parallel acceptance itself is safe.

Another class of methods allows earlier decisions to be revised. Remasking Discrete Diffusion explicitly permits an unmasked token to return to the mask state, converting decoding from a monotone reveal process into an iterative refinement process [121]. Self-reflective remasking, backtracking-enhanced samplers, and head-only look-back correction add more targeted mechanisms for reopening suspicious positions or correcting earlier commitments [122]–[124]. Local determinism propagation takes a different route: high-confidence anchor tokens make nearby positions more reliable, so the sampler can locally relax thresholds around such anchors rather than globally lowering the bar [125], [126]. Foreseeing-movement decoding and multimodal self-correction make a similar point in different settings: the sampler can use predicted future motion or cross-modal consistency to decide which positions deserve another refinement pass [127], [128]. This separates two related controls: unmasking decides when to commit a still-masked position, while remasking decides whether a commitment remains final.

7.2.0.2 Design knobs for token selection.

Several recent papers turn token selection into an explicit learned or optimized policy. NI Sampling trains a lightweight neural indicator to identify tokens that can be advanced without changing a reference trajectory, reaching large step reductions when combined with cache [129]. Learning Unmasking Policies formulates the decision as an MDP over confidence, mask status, and time [130]. Optimizing Decoding Paths scores entire candidate paths using denoising entropy and uses Best-of-N or SMC-style allocation to favor lower-uncertainty trajectories [131]. From Bits to Rounds argues that the most confident tokens are often the least informative; useful speedups require exploring medium-confidence, high-information positions rather than greedily resolving easy tokens first [132]. This perspective helps explain why confidence-only decoding can reduce rounds while still missing important positions.

Position- and frequency-aware variants broaden the same theme. Position Contrastive Guidance corrects systematic positional biases in masked diffusion sampling, while FourierSampler and FreqTS use frequency-domain signals to decide which positions are worth updating [133]–[135]. MC-DiT shows the same issue in masked image diffusion, where clean-to-clean reconstruction can improve contextual consistency during masked sampling [136]. Sink-token and token-level cross-validation papers show that local token dynamics can also be regularized or verified at inference time [137], [138]. A recurring challenge across this subsection is calibration. Token policies are most effective when their scores predict future correctness under the chosen decoding path; model confidence alone often measures ease, not importance, and can therefore produce fast but brittle generation.

7.3 Semi-Autoregressive and Block Updates↩︎

7.3.0.1 Block sampling as dependency management.

Token-selection policies decide which individual positions to update, but block and semi-autoregressive methods change the dependency topology of generation. Block Diffusion provides a representative starting point: it interpolates between autoregressive generation and fully parallel diffusion by making blocks causal across time while allowing bidirectional denoising within the active block [139]. This gives a tunable structural compromise. Smaller blocks preserve more left-to-right dependency and make the model closer to an AR LM; larger blocks expose more parallelism but increase the burden on the denoiser to model within-block dependencies.

This blockwise perspective has become a practical adaptation path for diffusion LLMs. From Next-Token to Next-Block treats an AR LM as the block-size-one limit of block diffusion, then gradually expands the active block while preserving causal context outside it [140]. ReFusion and Fast-dLLM v2 similarly use block or semi-autoregressive structure to combine diffusion-style local refinement with AR-like progression and cacheable context [141], [142]. In this view, block diffusion is more than a reduction in parallelism. It creates a different conditional factorization, one that can preserve instruction-following and long-context behavior while still permitting multiple tokens to be refined per forward pass.

7.3.0.2 When block structure helps.

Adaptive block methods make the topology state-dependent. AdaBlock-dLLM changes block size based on semantic or confidence signals, and Swordsman partitions sequences according to entropy so that easy regions can be decoded more coarsely than hard ones [143], [144]. Hierarchy Decoding recursively splits masked spans as confident positions are resolved, keeping remaining unresolved positions more sparsely distributed [145]. WavefrontDiffusion uses a dynamic schedule that propagates decoding from easier or more reliable regions, improving reasoning-oriented generation by shaping how information moves through the sequence [146]. Reviving any-subset autoregressive models connects this blockwise view back to older order-agnostic AR formulations, showing that principled parallel sampling and speculation can be interpreted as another point in the same topology design space [147]. These methods connect block topology back to token policies, but with a stronger structural prior: the sampler is choosing regions, not only positions.

Length control is another place where semi-autoregressive structure matters. Any-order flexible length diffusion, flexible-length text infilling, implicit infilling-length estimation, and bidirectional end-of-sequence control all try to avoid fixing a rigid output length before generation [11], [148]–[150]. This is particularly important for dLLMs, where a fixed canvas can waste computation on padding or make early end-of-sequence decisions unstable. Blockwise generation gives the model a way to expand, stop, or refine local regions without requiring the entire output to be predetermined.

A recurring limitation is dependency error. Generation-order theory and ParallelBench show that decoding multiple mutually dependent tokens in the same step can violate the conditional assumptions that make parallel generation attractive [151]–[153]. Multiverse and related parallelize-and-merge methods show that models sometimes contain latent preferences about parallelization, but exploiting those preferences requires careful merging and verification [154]. Fast and Fluent decoding further warns that overly large windows or poorly structured future context can degrade open-ended fluency [155]. Thus, blockwise inference can be interpreted as dependency management: it chooses which positions may interact bidirectionally, which positions are committed, and how much future uncertainty the current update is exposed to.

7.4 Guidance and Constraints at Inference↩︎

7.4.0.1 Logit shaping and soft guidance.

Guidance adds external objectives to the reverse process. The simplest family adapts classifier-free or classifier-based guidance to discrete states. Simple Guidance Mechanisms derives discrete classifier-free and classifier-based guidance rules and shows how the choice of noise process affects editability and repeated refinement [156]. Unlocking Guidance generalizes guidance to continuous-time Markov processes over discrete spaces, using process structure to make rate normalizers tractable across molecules, DNAs, and proteins [157]. Discriminator and training-free molecular guidance works provide related mechanisms for steering autoregressive or discrete diffusion transitions without retraining the base generator [158], [159]. These papers establish that guidance can operate on logits, rates, transition matrices, or proposal distributions, and that tractability depends heavily on the chosen discrete process.

Recent analysis shows that discrete guidance is not only a scalar preference adjustment. What Exactly Does Guidance Do in Masked Discrete Diffusion Models analyzes low-dimensional cases and shows how classifier-free guidance can concentrate probability on class-specific support while suppressing overlap regions, with strong guidance producing unstable concentration effects [160]. Improving Classifier-Free Guidance identifies a more concrete issue: unnormalized discrete CFG can change the total unmasking rate, not just the relative token preference [161]. Position-contrastive guidance similarly modifies token scores in ways that affect both order and content [133]. Guidance in discrete diffusion is therefore better described as modifying the reverse dynamics, including the timing of state changes, rather than only pushing samples toward a condition.

Reward and search-based methods form a second cluster. Soft Value-Based Decoding evaluates candidate states under a value function and selects higher-reward denoising paths without requiring differentiable rewards [162]. TreeG extends this idea into explicit tree search over diffusion or flow trajectories, enabling non-differentiable objectives in domains such as symbolic music, molecules, and biological sequences [163]. Importance-weighted SMC provides a more formal inference-time-scaling view: intermediate reward-tilted targets define weights, while proposal design controls variance and efficiency [164]. These approaches make extra inference compute productive, but their benefits depend on reward reliability, proposal quality, and the cost of evaluating branches.

For dLLMs, guidance increasingly uses internal confidence, verifier feedback, or reward-free references rather than a single external classifier. Self-Rewarding SMC uses trajectory-level confidence as an implicit reward [165]. RFG constructs reward-free guidance from policy-reference logit differences [166]. Reward-Weighted Sampling changes token selection order with an external reward model, and IterRef combines reward-guided noising-denoising refinement with a Multiple-Try Metropolis mechanism [167], [168]. Feedback guidance and Prism-style hierarchical search/self-verification further blur the line between guidance and test-time reasoning [169], [170]. The common theme is inference-time allocation: the sampler spends additional computation where reward, confidence, or verification suggests the current trajectory can be improved.

7.4.0.2 Validity constraints and posterior conditioning.

Hard constraints are usually treated separately because the goal is validity, not merely higher reward. Constrained Discrete Diffusion projects each denoising distribution toward the KL-nearest distribution satisfying explicit constraints, achieving zero violations in several constrained generation settings [171]. Grammar-constrained dLLM decoding checks whether a partial sequence can be completed to a CFG-valid string, while Lookahead-then-Verify samples possible continuations before accepting a token [172], [173]. DINGO compiles regular constraints into token-level automata and dynamic programs over block positions and automaton states [174]. These works illustrate why discrete diffusion is attractive for constrained generation: the model repeatedly exposes a full partially specified sequence, giving the sampler opportunities to enforce global structure before the final output is fixed.

Finally, guidance connects discrete diffusion to inverse problems and posterior sampling. G2D2 uses categorical variational distributions and Gumbel-Softmax relaxation to apply gradient guidance with discrete diffusion priors [175]. Split Gibbs, DICE, test-time anchoring, and DDPS adapt discrete diffusion to posterior sampling, controllable editing, and graph/path inverse problems [10], [176]–[178]. These methods motivate a three-way distinction: soft guidance improves preference or reward; hard constraints aim for guaranteed validity; posterior-control methods condition generation on observations or inverse-problem structure. All three exploit the same inference-time freedom, but they are evaluated with different metrics.

7.5 Acceleration: Distillation, Caching, and Parallelism↩︎

7.5.0.1 Distillation for fewer denoising steps.

Practical acceleration begins by distinguishing algorithmic step reduction from actual serving speed. Distillation methods reduce the number of refinement steps by training a new student, sampler, or consistency structure. Di4C distills dimensional correlations so low-step discrete samplers better preserve joint structure [179]. Learnable Sampler Distillation keeps the denoiser fixed but learns sampler coefficients and timesteps [180]. One-step and consistency models go further: Di[M]O distills masked image generation into a one-step generator, DLM-One is an early one-step language diffusion system, d3LLM uses pseudo-trajectory distillation, and CDLM/CD4LM train consistency or adaptive-decoding students for faster language generation [181]–[185]. These methods can yield large NFE reductions, but they are training-time interventions rather than purely training-free decoding policies.

7.5.0.2 Parallel and selective inference.

Per-step computation can also be reduced without reducing the number of denoising iterations. ReCAP reuses context features in masked generation [186]; DiCache learns sample-adaptive cache schedules for visual diffusion [187]; SparseD exploits stable sparse attention patterns across denoising steps [188]. ES-dLLM skips low-importance tokens after early layers using confidence and hidden-state variation, while SureLock stops computation for locally converged tokens but preserves their K/V states for the remaining active tokens [189], [190]. FOCUS evicts non-decodable tokens after early-layer importance scoring, and D3ToM/DToM applies selective compute to diffusion MLLMs by dynamically merging visual tokens according to decider-token salience [191], [192]. These methods share a systems observation: a denoising step need not be implemented as a full-sequence Transformer call.

Suffix and window methods expose another source of waste. DPad treats far-future suffix masks as a scratchpad and drops distant suffix tokens with sliding windows or distance-decay dropout [193]. Streaming-dLLM formalizes suffix pruning and dynamic thresholding for long generation, showing substantial throughput gains when much of the canvas is not locally useful [194]. Fast and Fluent diffusion decoding shows that long windows can also hurt fluency, motivating convolutional decoding and rejective fine-tuning [155]. Loopholing Discrete Diffusion addresses a different bottleneck: categorical sampling collapses soft information at each step, so deterministic latent bypasses can reduce idle or oscillating steps [195]. These works complement cache methods by asking not only what to reuse, but which regions belong in the computation at all.

Speculative decoding is another acceleration path. Speculative Diffusion Decoding uses diffusion to draft multiple tokens for an AR verifier [196]. DiffuSpec improves the diffusion-as-drafter approach with causal-consistency path search and adaptive draft length [197]. FailFast argues that dLLMs are useful as adaptive drafters: difficult regions can be rejected quickly, while easy regions can produce long accepted drafts [198]. DART is diffusion-inspired rather than a full diffusion LM; it trains a lightweight parallel drafter conditioned on AR target hidden states and verifies with the target model [199]. A separate line makes draft-and-verify native to dLLMs: FreeDave verifies future diffusion states, Spiffy builds directed draft graphs over future block states, and Self-Speculative Masked Diffusions split drafting and verification inside the architecture [200]–[202]. It is useful to distinguish AR-target speculation from dLLM-target speculation, since their correctness and latency accounting differ.

7.5.0.3 Caching in bidirectional denoisers.

A prominent systems line for dLLMs is cache and feature reuse. The difficulty is that bidirectional diffusion invalidates the simple append-only KV cache used by AR LMs. dLLM-Cache observes different temporal stability for prompt and response tokens and reuses cached states on different schedules [203]. dKV-Cache caches already decoded tokens with delayed updates, showing that partial KV reuse is possible even in bidirectional denoising [204]. Fast-dLLM combines approximate block-wise KV cache with confidence-aware parallel decoding [205]; FlashDLM adds efficient KV caching and a guided diffusion mechanism [206]. SlowFast sampling and learnable/adaptive parallel decoding show that cache reuse is often paired with token-acceptance policies rather than deployed alone [207]–[209]. dCache/d\(^2\)Cache, Dynamic-dLLM, and Attention-Is-All-You-Need-style cache methods refine the policy with token-, attention-, or budget-aware refresh [210]–[212]. SPA-Cache pushes this trend further by using singular-value proxies of value-state drift and layer-wise adaptive budgets [213]. The progression is from coarse cache reuse toward state-aware cache control. Open problems around KV-cache analogues and streaming generation are discussed further in Section 12.2.

7.5.0.4 Systems metrics and wall-clock evaluation.

A final consideration is evaluation discipline. Acceleration comparisons are more informative when they report the backbone, generation length, batch size, prompt length, hardware, cache refresh policy, memory overhead, number of real model calls, wall-clock latency or throughput, and quality under matched prompts. Hardware-aware work makes this especially visible: arithmetic-intensity-inspired scheduling and NPU-oriented designs show that the practical bottleneck may be memory movement, irregular sparsity, cache overhead, or verification cost rather than nominal NFE [214], [215]. This connects Section 6 to the broader efficient-inference literature for reasoning models, but dLLMs add their own complications because every denoising pass touches a mutable canvas rather than appending one token [216]. Without such details, a reported "10x speedup" may mean fewer denoising rounds, more tokens accepted per round, fewer active tokens per layer, more favorable batching, or hardware-specific throughput. Speed can therefore be treated as a systems property produced by the interaction of sampler policy, model architecture, and execution substrate.

8 Scaling and System Considerations↩︎

This section turns from formulations and algorithms to systems-level questions that govern whether discrete diffusion is practical at scale. Section 8.1 reviews backbone architectures and how bidirectionality reshapes positional encoding and length generalization. Section 8.2 examines scaling laws and compute-optimality. Section 8.3 covers the training pipeline of pretraining, supervised fine-tuning, and alignment, and Section 8.4 discusses deployment and product constraints such as latency, throughput, and hybrid serving.

8.1 Backbone Architectures↩︎

The fundamental shift from autoregressive (AR) generation to iterative denoising necessitates reevaluating the underlying neural backbones. While Transformers currently dominate the landscape, the discrete diffusion framework itself is agnostic to the specific neural architecture, opening avenues for diverse sequence modeling paradigms.

8.1.0.1 Bidirectional transformers.

Unlike AR models, which rely on left-to-right next-token prediction, discrete diffusion models learn a reverse denoising process that reconstructs clean text from corrupted or masked sequences. This paradigm naturally requires progressive full-context prediction, making bidirectional Transformers a highly intuitive backbone [7], [22]. By leveraging parallel, global attention, the model can simultaneously incorporate contextual information from all positions at once to denoise the text. However, utilizing full-context attention can sometimes introduce a "training-inference gap" if the model is later decoded in a semi-autoregressive, block-by-block manner. To address this, recent systems have proposed block-wise attention patterns that apply bidirectional modeling within individual blocks while enforcing causal masking across blocks [217], [218].

Crucially, the fully bidirectional nature of these Transformers profoundly impacts positional encodings and length generalization. Defining the relative position offset as \(\Delta = i-j\), where \(i\) is the current position and \(j\) is the attended position, standard AR models only expose the model to non-negative offsets during training, i.e., \(\Delta \in [0,\, T_{\text{train}}-1]\). By contrast, global bidirectional attention exposes diffusion models to both negative and positive offsets, yielding a substantially more symmetric range, \(\Delta \in [-(T_{\text{train}}-1),\, T_{\text{train}}-1]\) [219], [220]. By effectively learning both negative and positive relative distances, diffusion models capture a complete period of the sine and cosine components in Rotary Position Embeddings (RoPE). This symmetric coverage drastically reduces out-of-distribution embedding spaces, allowing diffusion language models to maintain remarkably stable perplexity during direct length extrapolation—avoiding the catastrophic performance collapse typical of AR models [219]. Recognizing this, researchers have successfully adapted Neural Tangent Kernel (NTK)-based RoPE scaling into "Diffusion-aware NTK," extending context windows up to 128K tokens via lightweight post-training [220].

8.1.0.2 State space models (SSMs).

Although bidirectional Transformers provide a strong foundation, the diffusion training objective does not impose strict constraints on the choice of the denoising network [221]. State Space Models (SSMs) and recurrent architectures, such as Mamba, present a compelling alternative backbone for diffusion language models [221], [222]. SSMs are particularly notable for their ability to perform linear-time sequence modeling using selective state spaces [221]. While full-attention Transformers scale quadratically with sequence length—often bottlenecking the generation of massive contexts—Mamba-like backbones could alleviate this computational burden. Consequently, integrating SSMs into the diffusion framework holds significant promise for supporting long sequences and enabling efficient linear-time inference, potentially marrying the arbitrary-order generation capabilities of diffusion with the scalable context mechanics of modern recurrent models [222].

8.2 Scaling Laws and Compute-Optimality↩︎

8.2.0.1 Scaling questions.

A central question for discrete diffusion language models (DLMs) is whether they follow scaling trends comparable to those that have underpinned the success of autoregressive (AR) models. Current empirical evidence suggests that DLMs can scale competitively both when trained from scratch and when adapted from pre-trained AR models. For instance, LLaDA reports that, at the 8B scale, masked diffusion models can achieve performance competitive with strong AR counterparts under matched compute budgets [7]; this competitiveness is as self-reported by the originating work and has not been independently audited here. Similarly, DiffuLLaMA shows that increasing the size of adapted DLMs, up to the 7B scale, is associated with clear gains on downstream tasks [21].

However, recent analyses suggest that compute-optimal token-to-parameter ratios and training dynamics of DLMs may differ from those of AR models. According to [223], different diffusion noise types behave similarly in strictly compute-bound settings, whereas in data-bound regimes uniform diffusion appears to favour a more parameter-heavy and data-light scaling profile. If this pattern hold at larger scale, it could make DLMs particularly relevant in settings where high-quality training data become a limiting factor. Furthermore, Efficient-DLM suggests that training choices affect the downstream efficiency frontier: stronger AR-to-DLM conversion, together with blockwise attention and position-dependent masking, can improve the trade-off between generation quality, throughput, and the number of function evaluations (NFE) at inference time [218]. Finally, recent work on overparameterised diffusion models suggests that generalisation and memorisation compete over training time: across image and language diffusion settings, the onset of memorisation is empirically observed to scale approximately with dataset size, providing a motivation for early stopping when generalisation is the priority [224].

8.2.0.2 Empirical evidence.

Beyond training-time scaling, DLMs also exhibit distinctive inference-time trade-offs, among generation quality, decoding steps, and test-time compute. Unlike AR decoding, diffusion generation does not require a strict one-step-per-token correspondence between output length and inference depth. Reported trends indicate that more denoising steps can improve generation quality on complex logical and planning tasks, as illustrated by Dream 7B on Countdown and Sudoku [22].

Another distinctive inference-time issue in DLMs concerns the initial generation length budget. Unlike AR models, where output length is typically determined dynamically through <eos>, masked DLMs are often initialised with a fixed budget of mask tokens. Rather than treating longer budgets as universally beneficial, recent work frames the denoising trajectory itself as a source of reasoning-time computation, where intermediate reverse-diffusion states can act as latent reasoning actions optimised by outcome-based reinforcement learning [225]. However, this form of test-time scaling also has practical limits. Naively allocating an excessively long generation budget can trigger an empirical failure mode known as <eos> overflow, in which the model terminates prematurely or degenerates into repeated <eos>-like padding outputs [226].

A related issue concerns the limits of step reduction. Naively compressing inference steps with uniform schedules can degrade generation quality, since early denoising stages are typically lower-confidence and more uncertainty-sensitive. To mitigate this problem, recent work has proposed more adaptive scheduling strategies, such as the Ascending Step-Size (ASS) scheduler. Under its proposed decoding design, ASS is reported to reduce the required number of inference steps from a linear bound \(O(L)\) to a logarithmic scale \(O(\log L)\) while maintaining competitive generation quality [227]; this \(O(\log L)\) claim is the originating paper’s analysis under its own assumptions and decoding design, not a worst-case guarantee verified here.

Finally, early evidence suggests that DLM scaling behaviour is sensitive to the chosen corruption strategy and may also vary across task regimes. Current evidence does not yet support a universal best corruption strategy; instead, different diffusion variants may behave differently across reasoning-heavy and knowledge-heavy task regimes [223]. Taken together, these observations suggest that scaling choices for DLMs should likely be matched to the downstream task regime rather than treated as universally optimal.

8.3 Training Pipeline: Pretraining, SFT, and Alignment↩︎

8.3.0.1 Pretraining.

Pretraining masked discrete diffusion language models can be viewed as a non-autoregressive analogue of next-token prediction. Rather than generating tokens sequentially, these models corrupt an input sequence and learn to recover the clean text through a parallel denoising objective related to the ELBO [7]. Because training such models from scratch is expensive, many recent studies instead adapt pre-trained autoregressive (AR) models into diffusion language models (DLMs). For example, DiffuLLaMA [21] and Dream [22] introduce a shift operation to preserve aspects of the AR model’s next-token dependency structure during diffusion training, while LAD [228] adopts a parameter-efficient LoRA-based adaptation strategy together with structured perturbations such as token swaps, duplication, and span shifts.

Pretraining also involves diffusion-specific choices in corruption schedules and representation design. Dream [22] proposes Context-Adaptive Token-Level Noise Rescheduling (CART), which assigns noise levels according to contextual informativeness. Efficient-DLM [218] further introduces position-dependent masking to better reflect the left-to-right structure of language and reduce training–inference mismatch. To enrich intermediate representations, Soft-Masked Diffusion [229] replaces hard masking with a soft mixture over the top-\(k\) predicted embeddings from earlier denoising steps.

8.3.0.2 Instruction tuning.

Supervised fine-tuning (SFT) adapts diffusion denoisers to follow natural-language instructions and prompts. In a standard dLLM SFT setup, the prompt is typically kept clean while the response tokens are randomly masked. However, this still mismatches the semi-autoregressive blockwise decoding strategy often used at inference time, where the model conditions on a clean prefix and a fully hidden future [217]. Blockwise SFT [217] addresses this mismatch by hiding all future tokens and computing the loss only over the currently decoded block. Similarly, Planner Aware Path Learning (PAPL) [230] incorporates planned denoising trajectories into the training objective.

To better support later reasoning-oriented post-training, IGPO [231] introduces Length-Aligned SFT, which synthetically rewrites verbose reasoning traces into more concise forms that better match the generation profile of full-attention dLLMs during RL sampling. In addition, Rainbow Padding [226] mitigates early termination under large decoding budgets by decoupling padding from termination and replacing repeated <eos> placeholders with a cyclic set of distinct padding tokens. Beyond pure text generation, Co-GRPO [232] reformulates masked diffusion generation as a Markov Decision Process and jointly optimises model parameters and inference schedules, with reported gains in generation quality and alignment in broader masked-diffusion settings.

8.3.0.3 Preference optimization.

Preference optimization and reinforcement learning have become an increasingly active area of DLM post-training. Relative to autoregressive models, the main complication is that rewards and preferences must be assigned over an iterative denoising trajectory rather than a single left-to-right generation path. This makes both credit assignment and likelihood estimation more difficult, and has led to several distinct but related lines of work.

One line of research extends DPO-style offline preference alignment to diffusion processes. Here, the core challenge is how to construct tractable and stable preference objectives over denoising trajectories or their variational surrogates. LLaDA 1.5 [233] addresses this issue with Variance-Reduced Preference Optimization (VRPO). SDPO [234] further decomposes sequence-level trajectory alignment into stepwise posterior-matching objectives, making optimization more tractable. D2-DPO [235] also adapts DPO to continuous-time diffusion dynamics, although its empirical validation is mainly conducted on structured binary sequences rather than full language generation. It is also important to distinguish these methods from superficially similar work such as DiffPO [236], which applies diffusion-inspired iterative refinement to inference-time alignment for AR LLMs rather than performing preference optimization for native dLLMs.

A second line of work studies online RL and policy-gradient training for dLLMs. The main bottleneck here is that exact per-step likelihoods are typically unavailable or expensive to compute, so most methods rely on approximations or alternative objectives. d1 [237] estimates likelihood ratios through a single-step mean-field approximation for GRPO-style updates, while d2 [238] develops trajectory-likelihood estimators such as d2-AnyOrder and d2-StepMerge to improve policy-gradient training for masked DLMs. Other methods instead seek to reduce or avoid the instability introduced by ratio estimation. wd1 [239] proposes a ratio-free weighted policy optimization objective, while SPG [240] introduces sandwiched variational bounds to reduce bias in policy-gradient estimation. At a broader theoretical level, SEPO [241] provides a general policy-gradient framework for discrete diffusion models based on an explicit score-entropy formulation.

Beyond stepwise optimization, recent work increasingly treats the denoising trajectory itself as a central object of alignment. This shift is motivated in part by the possibility that useful intermediate predictions may later be degraded or overwritten by subsequent denoising steps [242]. MDPO [242] and TraceRL [243] therefore optimise trajectory-level behaviour rather than treating each denoising step in isolation. d-TreeRPO [244] further improves trajectory evaluation through tree-structured rollouts, enabling verifiable step-wise reward propagation and more precise transition-probability estimation. At inference time, EntRGi [245] shows that generation can also be steered without additional training through entropy-aware reward guidance. At the same time, the trajectory-centric and any-order nature of DLM decoding introduces new safety concerns. A2D [246] shows that arbitrary-order decoding can enlarge the attack surface for jailbreaks, and proposes token-level safety alignment as a corresponding defence. Taken together, these studies suggest that post-training for DLMs is gradually shifting from token-level preference matching toward broader control over the denoising trajectory itself.

8.4 Deployment and Product Constraints↩︎

8.4.0.1 Latency and throughput targets.

Discrete diffusion models (DDMs) have a different deployment trade-off from autoregressive (AR) models. AR models generate text one token at a time, while DDMs update many tokens in parallel through iterative denoising. Because of this, DDMs can make better use of highly parallel hardware such as GPUs and TPUs in some serving settings. For example, Mercury [221] reports that diffusion-based coding models can reach very high throughput, with self-reported figures exceeding 1,000 tokens per second on its own serving stack and hardware, claimed to outperform optimized AR baselines. As with the other headline numbers in this section, these are figures reported by the originating work rather than independently reproduced, and the hardware, batch size, and baselines differ across papers; they should be read as indicative rather than directly comparable. Efficient-DLM [218] further shows that converting pre-trained AR models into diffusion models with block-wise attention can improve throughput while keeping competitive accuracy. In addition, systems such as DiRL [247] combine practical inference engines and efficient attention implementations to reduce the cost of iterative diffusion decoding. However, DDMs still face an important product constraint: Time-To-First-Token (TTFT). Since bidirectional diffusion needs several denoising steps before producing readable text, it may be less suitable than AR models for interactive streaming applications where fast first response matters.

8.4.0.2 Hybrid models.

To balance the cost of full diffusion decoding with the benefit of parallel generation, recent work has explored hybrid and semi-autoregressive designs. In this line of work, sequences can be processed block by block, while tokens inside each block are refined in parallel. Blockwise SFT [217] improves this setting by reducing the mismatch between standard supervised fine-tuning and blockwise diffusion inference, which leads to better performance on complex reasoning tasks. Beyond hybrid decoding, DDMs are also naturally useful for refinement. Their bidirectional structure makes them well suited for editing, inpainting, and self-correction. For example, PRISM [170] adds an inference-time self-correction module that identifies low-quality tokens and re-masks them for another round of revision. In reinforcement learning, IGPO [231] uses this inpainting ability to insert partial ground-truth reasoning traces during exploration, which can improve sample efficiency in sparse-reward settings. These results suggest that DDMs can be used not only as standalone generators, but also as refinement and editing modules on top of existing language modeling pipelines.

9 Applications↩︎

This section instantiates the framework across application domains, showing how the same corruption-denoising machinery specializes to very different discrete state spaces. The first four subsections cover modalities with engineered tokenizations: language and text (Section 9.1), code (Section 9.2), multimodal foundation models (Section 9.3), and tokenized media (Section 9.4). The next four cover domains with natural discrete structure: proteins (Section 9.5), genomics (Section 9.6), molecules and graphs (Section 9.7), and planning and agents (Section 9.8). We then treat tabular data and imputation (Section 9.9) and close with recommended case studies and a reporting checklist for new domains (Section 9.10). The domains differ in maturity: text/code and tokenized media are well established with large-scale systems and standard benchmarks; proteins, genomics, and molecules/graphs are active but more specialized; and planning/agents and tabular generation are comparatively emergent, with results that are promising but preliminary. We try to flag this evidence strength as we go rather than presenting all domains at a uniform cadence. Table 7 summarizes these domains and explicitly separates well-established settings from active but specialized or still-emerging ones.

3pt

9.1 Language Modeling and Text Generation↩︎

Text generation is among the earliest and most active applications of discrete diffusion. Existing work spans unconditional generation, sequence-to-sequence generation, controllable generation, rewriting and paraphrasing, summarization, data augmentation, instruction following, and emerging reasoning-oriented applications. Early continuous or simplex-based approaches such as Diffusion-LM and SSD-LM diffuse in continuous embedding or simplex spaces and demonstrate the potential of diffusion for controllable text generation and modular control [12], [55]. Sequence-to-sequence diffusion models extend this idea to conditional generation, where the source sequence is kept as conditioning context and the target sequence is generated through denoising [13], [248], [249]. Other works use diffusion for non-autoregressive generation and machine translation, often combining iterative refinement with masked prediction to improve parallel decoding quality [250]–[252].

A second major direction is controlled generation. Because diffusion generation proceeds through iterative refinement rather than one-pass left-to-right decoding, control signals can be injected at multiple stages of generation. This has been used for attribute control, conversation-structure control, commonsense knowledge generation, controllable synthetic data generation, and text editing [9], [253]–[256]. Diffusion language models have also been applied to summarization and document-oriented tasks, including perspective-preserving summarization, efficient text summarization, and generative document retrieval [257]–[259]. More recently, large-scale diffusion language models have been explored for instruction following and general-purpose language modeling, suggesting that diffusion is moving from task-specific text generation toward broader LLM-style applications [6], [7], [22], [260], [261].

Why diffusion is attractive here (global revision, bidirectional conditioning, and repeatedly applicable guidance) and when it is not, follows the general analysis of Section 11.1; we do not repeat it. The way these systems instantiate the four-component pipeline (corruption operator, denoiser parameterization, training objective, sampler) likewise follows the unified view of Section 5 and Table 4, with text-specific choices being absorbing-state masking over a subword or byte vocabulary, a bidirectional Transformer denoiser, a reweighted denoising objective, and a confidence-based or block sampler.

Language modeling and text generation provide a central testbed for discrete diffusion. The area has progressed from early embedding-space and simplex diffusion models to masked diffusion LMs and large-scale diffusion LLMs. Across tasks, the main advantages are global revision, bidirectional conditioning, controllable refinement, and robustness to local generation errors. The main challenges are multi-step latency, weaker streaming behavior, difficulty matching AR models in in-context learning and reasoning, and the need for standardized evaluation. These trade-offs make text generation a useful lens for understanding the broader design space of discrete diffusion models.

9.2 Code: Infilling, Repair, and Structured Constraints↩︎

Code generation has become an active testbed for discrete diffusion [23], [262]–[266], and in four-component terms these systems share a common instantiation: an absorbing (masking) corruption over a code-subword vocabulary, a bidirectional Transformer denoiser, a reweighted denoising objective, and a sampler augmented with syntactic or test-based constraints, differing mainly in whether the denoiser is trained from scratch or adapted from an AR code model, and in how constraints enter the sampler. Software development frequently involves editing existing codebases through tasks such as fill-in-the-middle completion and bug fixing. Autoregressive models face challenges in these scenarios due to their left-to-right decoding constraints. Discrete diffusion models address this by treating the target edit region as an absorbing masked state, allowing the reverse denoising process to simultaneously condition on both preceding and succeeding code segments. Recent masked diffusion models indicate that this bidirectional awareness facilitates non-causal reasoning and flexible generation orders, such as drafting the structural outline before filling in specific details [23], [265]. This bidirectional capability can also be implemented by adapting existing autoregressive models into diffusion models via specialized masking strategies [262].

Beyond context modeling, the iterative denoising trajectory of diffusion supports the injection of structural constraints during generation to enforce the validity of the outputs. These constraints are applicable to code synthesis as well as tasks requiring strict syntactic correctness, such as discovering mathematical equations [264]. The iterative framework also accommodates dynamic validation techniques. For instance, unit test generation can be accelerated by mining structural patterns from abstract syntax trees and incorporating them into intermediate decoding steps to guide the model toward functionally correct solutions [263]. Finally, applying these infilling techniques to repository-level contexts requires handling varying edit lengths. Recent work addresses this through dynamic length adjustment mechanisms, allowing the diffusion process to expand or contract the target sequence while processing localized blocks within long files [266].

9.3 Multimodal Foundation Models (Text + Vision/Audio)↩︎

A growing line of research generalizes discrete diffusion from text-only dLLMs to diffusion multimodal large language models (dMLLMs), which jointly handle text and additional modalities, including images, video, and audio. These methods share a core objective: substituting the conventional autoregressive (AR) backbone (used in standard MLLMs) with a bidirectional, masked-denoising backbone, thereby enabling understanding and generation to be performed by one unified decoder. We categorize this emerging area into four primary dimensions: the objectives of dMLLMs, the way different modalities are embedded and organized in the token space, how corruption schedules are synchronized across multiple streams, and the distinctive failure modes that stem from employing a single shared denoiser.

9.3.0.1 Objectives of diffusion MLLMs.

Early multimodal masked generative transformers, such as MAGVLT [267], together with unified discrete diffusion methods [268], introduced a key shift: they demonstrated that vision-language tasks can be solved by predicting masked tokens across multiple modalities, instead of relying on strictly left-to-right token decoding. Show-o [26] operationalized this idea by employing a single transformer backbone for both understanding and generation, a strategy that was subsequently adopted in instruction-tuned frameworks. MMaDA [27] and LLaDA-V [269] were subsequently proposed as multimodal analogues of text-only masked dLLMs. Collectively, these models demonstrate that pairing bidirectional attention with iterative remasking allows the system to repeatedly revisit and incrementally refine its predictions. More recent approaches, such as Lavida-O [270], Lumina-DiMOO [271], and FUDOKI [272], instead emphasize "omni" backbones, which perform both generation and understanding within a single denoising step. Meanwhile, Muddit [273] and [274] extend generative modeling beyond basic text-to-image applications. Overall, the twofold advantage of dMLLMs compared with AR MLLMs becomes clear: they support the simultaneous decoding of multiple modalities, and they enable globally consistent revision of multimodal outputs, rather than committing to an irreversible left-to-right generation process.

9.3.0.2 Unified tokens vs.cross-attention.

The central idea concerns how to fuse multiple modalities. One strategy converts each data stream into a shared set of discrete tokens that a single denoiser processes; the other keeps separate encoders per modality and connects them using cross-attention. Models in the first category—including Show-o [26], Unified Multimodal Discrete Diffusion [275], MMaDA [27], Muddit [273], and Lumina-DiMOO [271]—rely on a VQ tokenizer or neural codec to convert non-text inputs into discrete tokens, which are then concatenated with textual tokens. A bidirectional self-attention mechanism is subsequently applied to capture dependencies both within each modality and across different modalities. This approach keeps the denoiser straightforward, but it introduces design decisions around codebook resolution, vocabulary fusion, and positional encoding schemes for mixed-modal streams. In contrast, cross-attention-based dMLLMs like LLaDA-V [269] and LaDiC [276] maintain image representations in a continuous encoder and apply diffusion only to the text tokens; this sacrifices a single, unified generative process in exchange for tighter alignment with existing visual-instruction-tuning workflows. Hybrid approaches fall between these extremes: SDAR-VL [277] applies block-wise denoising to improve stability, Sparse-LaViDa [278] prunes the multimodal context to cut computation, and Lavida-O [270] adds flexible token budgets so a single backbone can process everything from brief captions to long videos.

9.3.0.3 Corruption schedules across streams.

When several token streams rely on a single denoiser, their corruption schedules must be carefully aligned. Because text and image tokens inherently carry different amounts of information—with a masked image patch typically holding less semantic meaning than a masked word—applying the same masking ratio across modalities is suboptimal. It can squander model capacity on easy image inpainting while degrading the text stream so severely that learning becomes ineffective. To mitigate this, recent approaches introduce modality-specific schedules. Prominent instances include distinct text and image mask ratios in MMaDA [27] and its parallel thinking–aware extensions [279], [280]; block-wise frame masking for video dMLLMs such as VidLaDA [281] and VDT-style transformers [282]; and flow-based velocity scheduling in FUDOKI [272]. How these schedules are coordinated forms an important dimension of model design. Some architectures draw a single diffusion timestep and apply it across all modalities via separate noise functions, while others assign distinct timesteps to each stream, training the model to reconstruct one modality from the partially corrupted information in another. In addition, as illustrated by masked diffusion captioning [283], the masking distribution itself plays a decisive role in shaping which visual features the model ultimately learns to emphasize, even when the objective is purely generative.

9.3.0.4 Failure modes and open issues.

Despite rapid advances, dMLLMs display distinctive failure patterns that diverge from those of AR MLLMs. First, because later iterations are conditioned on partially denoised latents, early mistakes can cascade through subsequent steps; corrective methods [284] address this by reintroducing masks mid-process so the model can revise earlier errors. Second, context tokens may act as anchors, leading to repetitive or degenerate generations in dMLLMs [285], [286], a phenomenon that becomes more pronounced as sequence length increases. Third, the quality of decoded outputs is fundamentally capped by the tokenizer: limited codebooks can cause visual or auditory artifacts that cannot be fixed by any amount of denoising in code space. Finally, because diffusion-based sampling distributes computation across many iterations, how test-time computation is allocated becomes critical; for instance, self-verified test-time scaling for dMLLM-based TTS [287] shows that inference-time remasking and verification can be as influential as the underlying architecture. Overall, these observations indicate that the advantages of parallel, globally consistent multimodal generation introduce new tuning burdens related to scheduling, repetition control, and verification, and that dMLLMs should be regarded as complementary to, rather than direct drop-in replacements for, AR multimodal foundations.

9.4 Tokenized Media Generation (Images, Audio, Video)↩︎

Beyond text, a substantial body of work applies discrete diffusion to continuous-valued media such as images, audio, and video. The basic recipe first compresses the raw signal into a sequence of discrete codes using a vector-quantized autoencoder or neural codec, then trains a categorical denoiser over those codes, and finally decodes the predicted codes back into the original signal through the tokenizer’s decoder. We organize this area along four themes: the shared pipeline, how it is instantiated for each modality, the strengths of working in token space, and the limitations inherited from the codec.

9.4.0.1 The shared pipeline.

The canonical pipeline for discrete diffusion in media is continuous data \(\to\) tokenizer or codec \(\to\) discrete diffusion in token space \(\to\) decoder. This template was established by VQ-Diffusion [24] and its improved version [288], which replace the Gaussian transition with a categorical one over VQ-VAE codes; the resulting model achieves competitive text-to-image quality and supports natural parallel infilling. Muse [289] and Unleashing Transformers [290] push the parallel-decoding view further, using more iterations only at sampling time, and Effective and Efficient Masked Image Generation [291] reports that careful schedule design can narrow the gap between discrete token diffusion and continuous latent diffusion at fewer sampling steps. Subsequent variants add guidance and conditioning machinery on top of this backbone (text-conditioned sampling [292], global context modeling [293], self-guided sampling [294], and layout- or typography-conditioned generation such as TextDiffuser [295]) without changing the basic "tokenize then denoise" structure. A more recent thread [296] studies how to unify several such denoisers under a single discrete backbone across modalities, connecting the tokenized view to the dMLLMs discussed above.

9.4.0.2 Images and videos.

For images, tokenized discrete diffusion tends to be most effective in settings where the main goal is editing, infilling, or otherwise constrained generation. Masked generative transformers [289], [291] enable rapid editing by allowing arbitrary subsets of tokens to be remasked, and text-conditioned variants [292], [295] employ parallel decoding to jointly manage global composition and fine-grained details. For video, the basic approach remains similar: spatio-temporal tokens produced by a VQ codec are denoised in parallel, while block- or frame-level corruption is applied to preserve temporal structure. VDT [282] employs masked modeling on video transformers for general-purpose prediction, CineTrans [297] tailors this pipeline to handle cinematic transitions, and Lumos-1 [298] integrates AR and discrete-diffusion video generation into a unified framework. Discrete diffusion has similarly been leveraged for structured visual outputs that are inherently discrete—including scene text recognition [299], human mesh recovery [300], 3D pose estimation [301], motion generation [302]–[304], sign language [305], [306], co-speech gesture [307], and talking-face synthesis [308], contexts in which a categorical formulation of the outputs naturally matches the label structure.

9.4.0.3 Audio and speech.

For audio, tokenization is usually performed by a neural codec, commonly using residual vector quantization (RVQ), which generates multiple parallel streams of discrete tokens for each frame. Applying discrete diffusion to these tokens offers the same advantages seen in image-token diffusion—such as parallel decoding, infilling, and editing—while aligning naturally with the architecture of contemporary codec-based speech and music models. Representative work includes Diffsound [309] and MDSGen [310] for text-to-sound generation; VampNet [311] and diffusion-based symbolic music models [312]–[314] for music synthesis; as well as Metis [315], an efficient high-fidelity speech system [316], BAD [317], wavelet-domain speech diffusion [318], and VibeVoice [319] for expressive, long-form speech generation. The same framework naturally enables token-level audio inpainting [320], and has further been adapted to discriminative settings such as automatic speech recognition—Drax [321], dLLM-ASR [322], audio-conditioned diffusion LLMs [323], and cross-modal speech diffusion [324], in which iterative refinement over hypothesis tokens replaces standard greedy or beam-search decoding. In parallel, multiple studies repurpose vector-quantization mechanisms for diffusion operating directly in the acoustic space [325]–[327].

9.4.0.4 Strengths and limitations.

The main advantages of tokenized discrete diffusion for media are its parallel decoding, straightforward editing via remasking chosen tokens, and the ability to reuse backbones from text and multimodal models. Together, these features support inpainting, outpainting, and localized editing workflows without any retraining: one simply re-masks and re-decodes a token region under the desired conditioning. However, two recurring limitations are reported in the literature. First, the tokenizer fundamentally limits generation quality: once the codec throws away a detail, it can never be reconstructed, and small codebooks typically introduce recognizable artifacts, including blocky images, metallic or tinny speech, and temporal flickering in video. Improved tokenizers and residual codecs [288], [327] can mitigate these issues, but the fundamental upper limit remains unchanged. Second, artifact patterns are linked to the topology of the codebook: since transitions operate over a fixed set of code indices, the model’s effective "neighbors" during denoising are determined by which codes lie nearby in the codebook. If the codebook is not smooth with respect to the underlying signal space, this can lead to unnatural transitions. Overall, these findings indicate that tokenized discrete diffusion is best understood as a complement to continuous latent diffusion, excelling in scenarios where editing, infilling, integration with other discrete modalities, or highly parallel decoding are the primary goals, and underperforming in cases where ultimate output quality depends on perceptual fidelity that the tokenizer is not yet able to maintain.

9.5 Proteins and Biological Sequences↩︎

Proteins are perhaps the most natural fit for discrete diffusion: a protein is a sequence over the twenty-letter amino-acid alphabet, so generation reduces to denoising in a small, well-defined categorical space whose "tokens" already carry biochemical meaning. The dominant approach adapts masked and absorbing-state diffusion to this alphabet, treating protein design as conditional sequence modeling under stability, motif, and functional constraints. DPLM established that a discrete diffusion objective yields a versatile protein language model whose representations transfer across generative and predictive tasks [70], and its successor DPLM-2 extends this to a multimodal model that jointly generates sequence and structure tokens [328]. Guided discrete diffusion provides a general mechanism for steering generation toward desired properties by combining a sequence-space denoiser with a property predictor [28], an idea instantiated across antibody design [329]–[332], therapeutic-peptide generation under multiple objectives [333], [334], combinatorial functional design [335], membrane proteins [336], and even the problem of shrinking a protein while preserving its function [337].

A second major thread conditions generation on three-dimensional structure. Inverse folding, recovering a sequence that folds to a target backbone, has been cast as discrete flow matching over the amino-acid alphabet [72], [338] and improved through mask-prior and denoising strategies that inject structural priors into the corruption process [339]. Other work conditions on geometric constraints for backbone inpainting [340], generates protein conformational ensembles via structure language models [341], and frames property optimization itself as a denoising problem over pretrained protein language models [342]. Robustness to distribution shift, which is critical when designs are pushed beyond the training distribution, has been addressed through context-guided diffusion [343].

Evaluation in this domain is distinctive because the ground truth is ultimately physical. In-silico proxies (sequence perplexity, recovery rate against native sequences, motif accuracy, and predicted structural metrics such as scTM and pLDDT from folding models) are standard for rapid iteration, but the most meaningful signals come from downstream structure predictors and, where available, wet-lab measurements of stability, binding, and function. We refer readers to a recent survey dedicated to controllable protein sequence design for a broader taxonomy of conditioning mechanisms and benchmarks [344].

9.6 Genomics (DNA/RNA) and Regulatory Sequences↩︎

Nucleic-acid sequences present a different regime from proteins: the alphabet is even smaller (four nucleotides), but the functionally relevant signals (regulatory grammar, splicing motifs, secondary structure) depend on long-range interactions that span hundreds or thousands of positions. This is precisely where global, bidirectional refinement is attractive, since regulatory elements can be coordinated across the whole sequence rather than committed left to right. Dirichlet flow matching adapts continuous flow-matching machinery to the probability simplex over nucleotides and demonstrates controllable DNA-sequence design, including generation conditioned on target regulatory activity [71].

Beyond DNA, discrete diffusion has been applied to RNA secondary-structure prediction, where the reverse process refines a discrete base-pairing representation [345], and to three-dimensional RNA inverse folding, where a hyperbolic discrete diffusion model captures the hierarchical geometry of functional RNA [346]. SE(3)-equivariant discrete diffusion has been used to jointly generate sequence and structure for nucleic-acid and protein complexes [347]. At the single-cell and functional-genomics scale, masked discrete diffusion jointly models cell identity and expression [348], while unifying multimodal masking frameworks [349] and large-scale RNA foundation models [350] point toward general-purpose generative models of regulatory sequence. A recurring practical consideration across these applications is measurement noise: experimental readouts of expression, accessibility, or structure are themselves noisy, so conditioning signals must be treated as soft targets rather than hard constraints.

9.7 Molecules and Graph Generation↩︎

Molecules and, more generally, graphs are an intrinsically discrete and combinatorial domain: a molecular graph is defined by categorical node attributes (atom types) and categorical edge attributes (bond types, including the absence of a bond). Discrete diffusion is therefore a natural fit, and DiGress, which corrupts node and edge categories with independent transition matrices and learns a graph denoiser, is the canonical instantiation [30]. In four-component terms, the graph setting fixes the state space (categorical node/edge attributes with hard validity) and a joint node-edge corruption, and most subsequent work varies one component at a time; the recurring design lesson is that the binding constraint here is not denoiser capacity but validity, which is enforced through the sampler (guidance, projection, constrained decoding) far more than through the corruption operator. Organized by which component they modify, a large body of work refines this setup: on the corruption side, continuous-time variants [351], [352] and Schrödinger-bridge or auto-encoder reformulations [353], [354]; on the sampler/ordering side, autoregressive and layerwise orderings that improve scalability [355]–[359]; on the state space itself, higher-order and hypergraph generation [360]–[362]; on the denoiser/representation side, sparse and degree-guided formulations for large graphs [363]–[368] and stability-oriented recastings of the denoising process [369]–[371]; and on the systems side, post-training quantization for efficiency [372].

A central challenge specific to 2D molecular graph generation is validity: a randomly denoised graph need not correspond to a chemically valid molecule. Much of the 2D literature therefore focuses on enforcing valence rules, connectivity, and property constraints, either through guidance or through constrained sampling. MolDiff explicitly addresses the atom–bond inconsistency that arises when atoms and bonds are generated independently [373]. For 3D structure-based drug design, the modeling bottleneck is different: many methods generate atom types and coordinates while chemical bonds are inferred or reconstructed afterward, so the main challenges include geometric equivariance, target-aware conditioning, steric and interaction constraints, and the limited flexibility caused by fixing the molecular size or atom count during generation. A range of methods incorporate three-dimensional and target-aware structure for structure-based drug design [374]–[377], fragment- and motif-level tokenizations that build in chemical validity by construction [73], [76], [378], and linker design for fragment joining [379]. Conditional and multi-conditional generation has been pursued through Bernoulli and discrete-graph guidance [380], [381], graph diffusion transformers [382], [383], composable score-based conditioning [384], and unified frameworks for dual-target molecule generation [385]. Language-to-molecule translation connects text prompts to molecular generation [386], [387], and plug-and-play projection enables controllable generation without retraining [388].

The same machinery extends naturally to adjacent problems where the object of interest is a discrete structure scored by an external objective. Retrosynthesis, predicting reactants from products, has been framed as categorical diffusion and discrete flow matching [389]–[391]; mass-spectrum-conditioned generation recovers molecules from spectra [392]; and inorganic-materials and crystal generation incorporate symmetry and electronic-structure constraints [75], [393]–[395]. Discrete graph diffusion has further been applied to protein co-design [74], neural-architecture search [396], graph-edit-distance computation [397], end-to-end path planning [398], social-graph generation [399], node classification [400], GNN explanation [401], and de novo drug design more broadly [371], [402]. Viewed through the optimization lens of Section 11, property-guided molecular generation is naturally read as search over a discrete configuration space for high-scoring structures under a learned predictor. Finally, as graph diffusion models mature, questions of fairness [403] and security, including watermarking [404] and backdoor attacks [405], have begun to receive attention.

9.8 Planning, Agents, and Decision Making↩︎

Planning, agents, and decision-making applications of discrete diffusion have recently emerged across language agents, embodied control, multimodal reasoning, and combinatorial optimization settings [3], [8], [31], [32], [406]–[409]. These works explore how iterative refinement in discrete spaces can be leveraged for planning, structured action generation, and constraint-aware decision-making. We organize this emerging area along four themes.

9.8.0.1 Diffusion agents as planners.

A central distinction between diffusion-based agents and conventional autoregressive (AR) agents lies in how plans or reasoning traces are constructed. AR agents generate sequences left-to-right, committing to earlier decisions without the ability to revise them, which can lead to error accumulation in long-horizon reasoning or planning tasks. In contrast, discrete diffusion models generate plans through iterative global refinement, allowing later context to influence and revise earlier tokens. Recent works explore this idea in both language-based and multimodal agents [3], [8], [32], [410], [411]. For example, diffusion-based reasoning frameworks generate intermediate plans that are repeatedly refined, enabling implicit backtracking and correction. In embodied or vision-language settings, diffusion planners have been used to iteratively refine action sequences or trajectories conditioned on observations [406], [412]–[414]. It is important to emphasize that this “global revision” capability is a potential advantage rather than a guaranteed outcome: its effectiveness depends on the training objective, noise schedule, and inference strategy (e.g., remasking policies or guidance). Without appropriate design, diffusion models may still exhibit local inconsistencies or converge slowly.

9.8.0.2 Offline RL & trajectory generation.

Discrete diffusion provides a natural framework for modeling trajectories in offline reinforcement learning (RL), where sequences of states, actions, and optionally rewards are tokenized and treated as structured discrete objects. Instead of learning a policy directly, diffusion models learn a generative distribution over trajectories, which can then be sampled or guided toward high-reward behaviors. Several works demonstrate that diffusion over trajectories can act as a stochastic planner that improves upon suboptimal datasets [415]–[418]. By conditioning on reward signals or value estimates, the model can generate trajectories that are unlikely under the behavior policy but achieve higher returns. This connects to broader perspectives of diffusion as an optimization process in discrete spaces, where denoising corresponds to iteratively improving candidate solutions. In robotics and control settings, diffusion-based trajectory planners have also been explored for structured action generation and robustness [407], [419], [420]. However, practical performance depends critically on how trajectories are tokenized and how constraints (e.g., dynamics feasibility) are enforced during sampling.

9.8.0.3 Tool use & API calls.

Modern language agents frequently rely on structured tool use, such as generating JSON outputs or function calls that must satisfy strict schemas. Discrete diffusion is well-suited for such settings because it operates directly over categorical tokens and supports constraint-aware decoding. Diffusion-based approaches to tool use and structured generation highlight two key advantages. First, global context can be used to jointly refine all fields in a structured output, rather than committing to arguments sequentially as in AR decoding. Second, constraint enforcement (e.g., valid JSON syntax or schema compliance) can be integrated into the denoising process via masking, projection, or guided sampling [408], [421], [422]. Recent systems explore diffusion-based language agents that interleave reasoning and tool invocation, enabling iterative correction of tool arguments and outputs [423], [424]. Nevertheless, reliability in practice depends not only on the generative model, but also on external constraint mechanisms such as validators, execution feedback, and verification loops. Diffusion alone does not guarantee correctness without such auxiliary components.

9.8.0.4 Combinatorial optimization.

Many planning problems can be formulated as combinatorial optimization over discrete configurations, such as traveling salesman problems (TSP), scheduling, or graph construction. Discrete diffusion offers an alternative to classical search methods by treating solutions as tokens that are iteratively refined. In this view, diffusion acts as a stochastic refinement process over candidate solutions, analogous to local search but with global context and learned transition dynamics. Applications include graph optimization, routing, and structured design problems [31], [409], [425]. Compared to traditional approaches such as beam search, evolutionary algorithms, or simulated annealing, diffusion models can learn problem-specific priors and produce diverse candidate solutions in parallel. However, feasibility constraints remain a central challenge. Many works incorporate repair operators, constraint-aware transitions, or projection steps to ensure validity during denoising [407], [426]. The effectiveness of diffusion in combinatorial settings depends on how constraints are encoded, either implicitly through training data or explicitly through inference-time mechanisms.

Across these domains, discrete diffusion reframes planning and decision making as iterative refinement in structured discrete spaces. Its main promise lies in global consistency, flexibility in conditioning, and compatibility with constraints. At the same time, its advantages are contingent on careful design of tokenization, corruption processes, and inference strategies, and should be viewed as complementary to, rather than replacements for, autoregressive and classical planning methods.

9.9 Tabular Data and Imputation↩︎

Tabular data sit at an interesting boundary for discrete diffusion: a single row mixes categorical fields with numerical fields, so a generative model must handle both modalities coherently. Categorical columns are a natural fit for the discrete machinery developed throughout this paper, each column is simply a small vocabulary, and the forward process corrupts category labels exactly as it does tokens. Numerical columns require an additional decision: they can be discretized into bins and absorbed into the discrete state space, or modeled with a continuous diffusion process running alongside the discrete one in a hybrid design. Early work such as TabDDPM took the latter route, combining Gaussian diffusion for numerical features with multinomial diffusion for categorical features under a shared schedule [427]. More recent models pursue tighter integration of the two modalities: TabDiff formulates a mixed-type, multi-modal diffusion process that jointly generates numerical and categorical columns [428], [429], TabNAT couples continuous and discrete generation in a single non-autoregressive framework [430], and Unmasking Trees casts tabular generation as a masked-prediction problem solved with tree-based predictors [431].

Missing-value imputation is a particularly natural application, because it maps directly onto the masking view of discrete diffusion: the observed entries of a row play the role of unmasked context, the missing entries play the role of masked positions, and the reverse process fills them in conditioned on the rest of the row. This framing requires no special-purpose architecture, imputation and generation are the same task with different masking patterns, and it accommodates arbitrary, non-contiguous missingness, since any subset of cells can be designated as the target. Evaluation in this domain attends to three distinct criteria that are sometimes in tension. Fidelity measures how closely generated columns and their joint correlations match the real data distribution; privacy asks whether synthetic rows leak information about real records (e.g., via membership-inference or distance-to-closest-record analyses), which is a primary motivation for synthetic tabular data in the first place; and downstream utility measures whether models trained on synthetic or imputed data perform comparably to those trained on real data. A recurring theme is that optimizing fidelity alone can degrade privacy, so the most useful evaluations report all three together rather than a single aggregate score.

9.10 Other Case Studies↩︎

Beyond the applications outlined above, discrete diffusion model is also employed in settings where data are inherently discrete or can be represented via compact codebooks. We treat these as case studies rather than applications. While they frequently reuse modeling strategies from text or media generation, their unique constraints, evaluation criteria, and typical failure patterns warrant separate treatment. We close by providing a standardized checklist to guide the reporting of design choices in future case studies.

9.10.0.1 Layout and document generation.

Document, slide, GUI, and scene layouts are an obvious early application area for discrete diffusion, because layout components have categorical types and quantized coordinates. LayoutDM [432] introduces the standard approach of modeling element classes and bounding-box bins jointly as categorical variables, evolved under masked or absorbing transitions. Subsequent research develops this template along three main directions: conditioning on partial layouts or predefined graph structures [433], [434]; correction of layout-decoder-specific failure modes, including layout sticking and pattern collapse [435], [436]; and specialization to particular application domains such as floor plans, vector graphics, and landmark-based layouts [437]–[439]. A common finding is that layout-oriented metrics—like overlap, alignment, and validity—pose challenges for greedy AR decoders, while iterative remasking enables the model to jointly manage both global alignment and fine-grained local placement.

9.10.0.2 3D assets generation.

Representing 3D content as sequences of discrete primitives—such as part labels, voxel or cube indices, or VQ shape tokens—allows 3D generation to leverage the same masked denoising architectures. 3DQD [440], [441] discretizes shapes at the part level. Cube-Absorb discrete diffusion [442] employs absorbing transitions to cope with the very long token chains found in large scenes. Mixed Diffusion for 3D Indoor Scenes [443] combines continuous and discrete variables to model object placement, while PartDiffuser [444] synthesizes 3D meshes part by part. Complementary research investigates 2D-to-3D alignment [445], scan-based generative modeling [446], and instance-level 2D-to-3D representation learning [447]. A key design trade-off across these methods lies between having an expressive codebook rich enough to capture geometric detail and the resulting long token sequences, which in turn motivate block-wise or absorbing diffusion schedules.

9.10.0.3 Sketches, fonts, and stylized media.

Discrete diffusion is also effective for structured creative tasks in which the components are small and well-defined. Representative applications include sketch-conditioned image inpainting [448], joint sketch-and-context generation [449], glyph-style synthesis [450], high-quality font creation [451], Chinese calligraphy synthesis and recognition within a unified diffusion framework [452], and arbitrary style transfer [453]. In these scenarios, the approach retains the parallel decoding and editability benefits of image-token diffusion, but operates in a smaller, more tightly constrained code space. This makes it easier to enforce properties such as legibility or stroke order compared to diffusion carried out directly in pixel space.

9.10.0.4 Recommendation and information retrieval.

A growing body of work treats user–item interaction histories as discrete sequences, enabling the application of the same masked-denoising framework commonly used in NLP. Within this view, sequential recommendation has been reformulated as fuzzy modeling in a discrete state-space diffusion framework [454]; generative recommendation has been implemented via LLaDA-Rec [455] and DiffGRM [456]; and further extensions targeting reranking, bundle generation, and continuous-time recommendation continue to elaborate and refine this overarching pipeline [457]–[459]. Closely related approaches apply discrete diffusion at the retrieval stage in graph-based RAG through query-aware flow diffusion [460], and for imputing graph features using fractional subgraph diffusion [461]. These research avenues are closely related to the planning and decision-making subsection (Section 9.8). Consequently, we only summarize them briefly here and refer the reader to that subsection for planning-oriented applications of discrete diffusion.

9.10.0.5 Specialized scientific and structured domains.

A final group of case studies highlights discrete diffusion beyond standard image and text modalities. Biomedical imaging is explored through LLaDA-MedV [462] and via discrete-diffusion methods for prostate MRI super-resolution [327], [463]. In wireless communications, residual vector quantization is applied to channel symbols [464], and spiking neural networks have been integrated with VQ-based discrete diffusion to enable low-power generative models [465]. Structured knowledge generation is addressed by DiSK [466]; LEAF [467] reformulates time series forecasting as a diffusion-based language model; and object-centric representation learning is advanced through masked generative modeling [468]. Vision–language applications that go beyond straightforward generation also fall into this category. These include image captioning [469], caption editing [470], segmentation refinement [471], remote-sensing image captioning [472], defect detection [473], source-mask optimization for lithography [474], uncertainty-aware Bernoulli diffusion [475], and guided discrete diffusion for scientific applications [476]. Although these domains differ substantially, prior work follows a similar template: a pretrained or manually crafted tokenizer converts the domain-specific signal into a discrete set of tokens, and a categorical denoiser then performs parallel, editable, and constraint-aware generation over this token vocabulary.

9.10.0.6 A checklist for new case studies.

Based on these examples, we suggest that future case studies report at least the following elements, to enable consistent comparison of results across domains. This checklist is an authors’ recommendation rather than an established community standard; we offer it to improve cross-domain comparability, not as a description of current reporting practice. (1) The tokenization setup, including codebook size, level of granularity, and pretraining strategy. (2) The corruption process, specifying whether mask, uniform, or absorbing transitions are used, along with the noise schedule. (3) The training objective, for example cross-entropy, ELBO, score-entropy, or a flow-matching variant. (4) The model architecture, including how it incorporates auxiliary modalities. (5) The sampling procedure, including the number of sampling steps, the remasking strategy, and any guidance mechanisms. (6) Constraint mechanisms, such as validators, projection methods, or classifier-based guidance. (7) The evaluation protocol, including domain-specific metrics for cases where standard diffusion metrics are not applicable. (8) Typical failure modes, such as sticking, repetition, codebook collapse, or violations of validity constraints. Describing results along these dimensions facilitates cross-domain comparison and clarifies which design choices generalize across case studies and which remain specific to a particular domain.

10 Evaluation and Benchmarks↩︎

This section reviews how discrete diffusion models are evaluated and why several autoregressive evaluation habits do not transfer unchanged. Section 10.1 discusses likelihood, perplexity, and calibration. Section 10.2 surveys quality metrics across text, media, science, and agentic domains. Section 10.3 addresses speed, cost, and efficiency metrics, arguing for quality–latency frontier reporting, and Section 10.4 treats editing, infilling, and conditional fidelity.

10.1 Likelihood, Perplexity, and Calibration↩︎

Likelihood, perplexity, and calibration are central but subtle evaluation dimensions for discrete diffusion language models. Unlike autoregressive (AR) models, whose likelihood is given by a left-to-right product of next-token probabilities, diffusion models define generation through a corruption-denoising process. As a result, likelihood evaluation depends on the chosen formulation, parameterization, and sampler. Some models admit tractable or efficiently estimable variational bounds, while others are primarily evaluated by denoising losses, task metrics, or sample quality. This makes likelihood reporting useful but also easy to misinterpret, especially when comparing diffusion language models with AR baselines.

10.1.0.1 Likelihood reporting.

For AR language models, evaluation commonly reports negative log-likelihood or perplexity under the factorization \(p_{\theta}(x)=\prod_{i=1}^{L}p_{\theta}(x_i\mid x_{<i})\). This provides a direct token-level measure of predictive performance under teacher forcing. Discrete diffusion models generally do not use this factorization. Instead, they define a forward corruption process \(q(x_t\mid x_0)\) and a learned reverse process \(p_{\theta}(x_{t-1}\mid x_t)\), leading to likelihood objectives based on variational lower bounds or continuous-time analogues. In D3PM-style models, the evidence lower bound decomposes into reconstruction, prior, and timestep-wise KL terms, where the reverse model is trained to approximate the posterior \(q(x_{t-1}\mid x_t,x_0)\) [15], [16]. In masked diffusion language models, the objective often simplifies to a reweighted masked-token prediction loss, which can be interpreted as a continuous-time or discrete-time variational bound under an absorbing-state corruption process [5], [19], [477].

Because of this difference, diffusion likelihoods and AR perplexities are not automatically apples-to-apples. First, many diffusion models report a variational bound rather than exact likelihood. The bound may be loose, and its tightness depends on the number of timesteps, the weighting scheme, the parameterization, and whether the reverse posterior is computed exactly or approximated. Second, likelihood units may differ across papers: some report bits per dimension, bits per token, negative log-likelihood, ELBO, or perplexity derived from a bound. Third, tokenization can change the apparent value of perplexity. A model evaluated with byte-level tokens, BPE tokens, or a different vocabulary size may have substantially different token-level perplexity even if the underlying text distribution is similar. For cross-tokenizer comparison, normalized quantities such as bits-per-byte or bits-per-character are often more meaningful than raw token perplexity.

A further complication is that likelihood and generation quality may diverge. In AR modeling, lower perplexity often correlates with stronger language modeling, although there is not a complete measure of instruction-following or human preference. For diffusion language models, the mismatch can be larger because the training objective measures denoising quality across corruption levels, whereas generation quality depends on the full reverse trajectory and the sampler. A model may achieve a strong denoising loss but still produce lower-quality samples if the sampler commits tokens too aggressively, accumulates errors across denoising steps, or uses a poor remasking schedule. Conversely, inference-time techniques such as confidence remasking, guidance, or test-time scaling can improve sample quality without changing the underlying likelihood bound [121], [156], [168]. Therefore, likelihood should be reported together with sample quality and decoding-budget metrics rather than used as the sole evaluation criterion.

For fair comparison with AR baselines, we recommend that papers state explicitly: (1) whether the reported value is an exact likelihood, an ELBO, an upper bound on perplexity, or a surrogate denoising loss; (2) the tokenization scheme and normalization unit; (3) the number of diffusion steps used for likelihood estimation if applicable; and (4) whether the same data preprocessing and context length are used for AR and diffusion models. When possible, diffusion models should report both likelihood-style metrics and task-level generation metrics, since these capture complementary aspects of model behavior. Recent work on likelihood-based diffusion language models and efficient perplexity bounds provides useful tools for making such comparisons more principled [17], [18], [95].

10.1.0.2 Calibration.

Calibration measures whether a model’s predicted probabilities correspond to empirical correctness. For discrete diffusion, calibration is important at two levels: the calibration of token posteriors at each denoising step, and the calibration of the final generated sequence. At a given timestep \(t\), the model predicts a distribution over clean tokens, reverse states, or score/ratio values. If the model assigns high confidence to incorrect predictions, the sampler may prematurely freeze wrong tokens; if it is under-confident, the sampler may repeatedly remask correct tokens and waste computation. Thus, calibration directly affects the efficiency and reliability of iterative generation.

This connection is especially clear in confidence-based remasking. Many masked diffusion samplers generate candidate tokens for masked positions, estimate confidence from token probabilities or entropy, freeze high-confidence tokens, and remask low-confidence ones. The effectiveness of this strategy depends on whether confidence is well calibrated across positions and timesteps. Ideally, high-confidence predictions should be likely to be correct, and the confidence scale should remain comparable as the corruption level changes. In practice, calibration may vary substantially across timesteps: early denoising steps operate under high uncertainty and broad context ambiguity, while later steps condition on more fixed tokens and may become overconfident in locally fluent but globally inconsistent predictions. Miscalibration can lead to error locking, where an incorrect token is frozen early and then treated as reliable context in later denoising rounds.

Calibration also interacts with masking schedules and selective refinement. A well-calibrated model can support adaptive computation: easy tokens can be fixed early, while uncertain spans receive more denoising steps. This is important for efficient generation because diffusion models trade quality for the number of refinement rounds. If token confidence accurately predicts correctness, the sampler can allocate computation non-uniformly across the sequence, focusing on difficult regions such as rare entities, syntactically constrained spans, or long-range dependencies. Recent methods that use remasking, adaptive block sizes, local determinism, or entropy-bounded unmasking are viewed as attempts to improve the calibration of intermediate predictions [118], [121], [125], [143].

Calibration should therefore be evaluated explicitly rather than assumed from likelihood or sample quality. Standard tools include reliability diagrams, expected calibration error (ECE), negative log-likelihood of token predictions at each timestep, entropy-accuracy curves, and selective prediction metrics. For diffusion language models, these metrics can be computed separately for different corruption levels, token types, and decoding iterations. Timestep-wise calibration curves are particularly informative: they reveal whether the model is overconfident early in generation, underconfident near convergence, or poorly calibrated only under certain masking ratios. In conditional generation, calibration can also be measured with respect to constraint satisfaction, e.g., whether high-confidence tokens are more likely to preserve required entities, attributes, or fixed spans.

Beyond token-level calibration, sequence-level uncertainty remains an open evaluation problem. Since diffusion models generate through multiple stochastic refinement steps, uncertainty arises not only from the final token distributions but also from trajectory-level choices such as masking order, resampling, and guidance strength. Two samples with similar final token probabilities may have very different denoising histories: one may converge steadily, while another may oscillate across several refinements. Such trajectory information can be useful for detecting unstable generations, hallucinations, or constraint violations. Reporting convergence diagnostics, remasking rates, entropy decay, and agreement across multiple denoising trajectories may therefore provide a richer picture of reliability than final likelihood alone.

In summary, likelihood and perplexity remain important for evaluating diffusion language models, but they must be interpreted carefully. Diffusion models often report variational bounds or denoising-based surrogates rather than AR-style likelihoods, and comparisons are sensitive to tokenization, normalization, and inference procedure. Calibration is equally important because intermediate confidence estimates determine which tokens are frozen, remasked, or refined. A comprehensive evaluation should therefore combine likelihood-style metrics, task-level generation quality, timestep-wise calibration, and sampler-dependent diagnostics.

10.2 Quality Metrics Across Domains↩︎

10.2.0.1 Text.

Evaluating generation quality in dLLMs requires sophisticated and task-specific metrics. Early n-gram overlap statistics such as BLEU [478] and ROUGE [479] are systematically ill-suited to non-autoregressive generators: they reward lexical proximity to a single reference, thereby penalizing the diverse outputs that iterative stochastic denoising naturally produces. [480] illustrates that a generation can be semantically correct, factually grounded, and globally coherent yet score poorly simply because its wording diverges from the reference. They show that a dLLM can achieve equal or higher BLEU and ROUGE scores than a comparably-sized AR model while simultaneously exhibiting stylometric properties such as burstiness and perplexity under an external LM that cause standard AI-generated-text detectors to fail. This demonstrates that single surface metrics, applied without awareness of how diffusion generation differs architecturally from autoregressive generation, therefore risk producing misleading assessments of output naturalness.

Task-based evaluation on closed-form benchmarks, where correctness does not depend on reference-matching, is therefore the preferred primary quality signal for dLLMs. These benchmarks yield binary or ordinal correctness signals across diverse task families: mathematical reasoning, general knowledge, and code generation evaluated by execution. When closed-form correctness is insufficient, they should be complemented by instruction-following benchmarks, factuality-sensitive evaluations, and human preference comparisons. [481] open a further evaluative axis orthogonal to generation: they demonstrate that the bidirectional pretraining of dLLMs can be assessed through downstream text embedding tasks such as long-document and reasoning-intensive retrieval, where bidirectional context provides advantages that generation-focused benchmarks do not expose.

A further dLLM-specific evaluation concern is the attribution and detection of diffusion-generated text. Standard AR watermarking schemes fail in this setting because they condition each new token on previously generated tokens. This is an assumption that dLLMs’ arbitrary generation order breaks. Three concurrent works address this from different angles: [482] compute watermark signals in expectation over undetermined context; [483] introduce order-agnostic watermarking that operates with both predictive and bidirectional context strategies; and [484] propose biasing tokens on both their available left and right neighbours to retain near-zero runtime overhead. That three independent proposals were needed to rehabilitate a single AR evaluation mechanism underscores a broader point: evaluation tools calibrated for autoregressive generation cannot be applied to dLLMs without first verifying that their underlying assumptions still hold.

10.2.0.2 Media.

For tokenized image generation, the Fréchet Inception Distance (FID; [485]) measures the Wasserstein-2 distance between Inception-v3 feature distributions of real and generated images and is the standard quality metric for image generation. For token-based discrete diffusion, a key limitation is that FID conflates two distinct sources of error: tokenizer reconstruction quality and generative model quality. [57] shows that optimizing a VQ tokenizer for reconstruction does not guarantee better downstream generation quality, motivating the practice of reporting reconstruction metrics separately from generation FID. Alongside FID, Inception Score and CLIP score are used as complementary metrics for sharpness and text-image alignment respectively [486], [487]. Evaluating the full encode-generate-decode pipeline additionally requires reconstruction-consistency metrics, such as LPIPS, to detect artefacts introduced by the mismatch between the tokenizer’s training distribution and the generative model’s output distribution.

Similar to FID for images, Fréchet Video Distance (FVD; [488]) and Fréchet Audio Distance (FAD; [489]) extend the same distributional summary principle to video and audio respectively, using domain-appropriate feature networks. For video, a known limitation is that FVD is insensitive to certain temporal distortions, such as frame-level flickering introduced by codebook-boundary transitions in discrete video generation [56]. For audio, FAD is used alongside perceptual metrics such as word error rate and speaker similarity to capture distinct quality dimensions that a single distributional metric cannot summarize [43], [310].

Across all three modalities, no single metric captures the full range of failure modes in token-based discrete generation. A principled evaluation protocol therefore requires three coordinated measurements: tokenizer reconstruction fidelity evaluated independently of the generative model; downstream generation quality measured by the appropriate distributional metric; and reconstruction-consistency evaluation of the full encode-generate-decode pipeline.

10.2.0.3 Science and graphs.

Protein and genomic sequence generation require evaluation protocols that combine sequence plausibility with downstream functional or structural criteria. For protein design, sequence recovery and perplexity are useful for inverse-folding benchmarks, but they are insufficient on their own because many sequences with low recovery may still fold correctly, while high-recovery sequences need not satisfy the intended design objective. Consequently, structure-aware evaluation commonly includes refoldability or self-consistency checks using structure-prediction models, RMSD or TM-score to the target backbone or motif, confidence metrics such as pLDDT, diversity and novelty relative to natural proteins, and, when available, experimental stability or binding measurements [490]–[492]. For genomics, validity is usually less about syntactic correctness, since any string over the nucleotide alphabet is formally valid, and more about biological plausibility and target activity. Evaluation therefore emphasizes sequence statistics such as GC content and k-mer or motif composition, novelty relative to training sequences, conservation of regulatory grammar, and predicted or measured functional activity, including chromatin accessibility, promoter strength, enhancer activity, or gene-expression targets [29], [71], [493]. Thus, as in molecular generation, biological-sequence evaluation is layered: low-level sequence realism must be paired with task-specific structural or functional validation.

Molecular generation evaluation follows a constraint-first hierarchy. The validity, uniqueness, and novelty must be satisfied before considering any other distributional metric, since a model generating chemically invalid or memorized structures provides little scientific value regardless of its distributional scores. The MOSES benchmark [494] and GuacaMol [495] operationalize this with complementary emphases, with MOSES centering on Fréchet ChemNet Distance (FCD) and scaffold statistics, and GuacaMol focusing on KL divergence across physicochemical property distributions. [30] establish these benchmarks as the standard evaluations for molecular discrete diffusion, demonstrating that the validity-uniqueness-novelty hierarchy combined with FCD exposes failure modes that neither metric type reveals in isolation.

For graph generation, the standard suite uses maximum mean discrepancy (MMD), a kernel-based distributional distance, on degree distributions, clustering coefficients, and orbit statistics. Discrete diffusion’s sparse intermediate states keep structural features well-defined throughout denoising, a property [496] show reduces average MMDs relative to continuous noise. For combinatorial optimization, optimality gap is the primary criterion [31], [409], and its monotone improvement with denoising steps provides a quality-compute tradeoff with no direct AR analogue. This illustrates that scientific evaluation for discrete diffusion is not a single metric problem, but a layered framework where constraint satisfaction and distributional fidelity serve complementary diagnostic roles.

10.2.0.4 Agents and tool use.

For agentic tool use and code generation, execution-based evaluation is the most dominant quality signal. Unlike surface-based metrics, executing the generated output is reference-free and directly measures functional correctness. Pass@\(k\) on coding benchmarks such as HumanEval [497] and MBPP [498] is the standard execution-based metric, measuring the probability that at least one of \(k\) generated samples passes all unit tests. A dLLM-specific concern is that non-AR generation can produce syntactically malformed outputs even when the intended logic is correct. Syntactic validity rate must therefore be reported alongside pass@\(k\) for dLLMs, as left-to-right AR generation enforces local syntactic consistency by construction.

For function-calling and API use, pass@\(k\) alone is insufficient. A model can produce fluent but structurally invalid function calls that fail downstream execution entirely [174]. Structured outputs must additionally be evaluated on schema-level metrics, including schema validity, field completeness, and type compliance. Benchmarks such as BFCL [499] provide standardized harnesses for these metrics across diverse tool-use scenarios. End-to-end task success rate measures whether a full multi-step agent loop completes its goal, providing a coarser but more ecologically valid signal that aggregates individual tool-call quality into an outcome metric.

Iterative self-correction is another dLLM-specific evaluation dimension. The ability to re-mask and regenerate low-confidence or erroneous spans within a single generation is structurally unavailable to left-to-right AR models without explicit rollback. [3] demonstrate that discrete diffusion’s bidirectional context provides an advantage on tasks requiring global constraint satisfaction, where early positions can be revised once later context clarifies the requirements. Two metrics characterize this dimension: correction frequency, defined as the proportion of denoising steps that revise previously generated tokens, and pass@\(k\) as a function of denoising steps, which traces how generation quality improves with additional compute. However, standardized protocols for reporting these quantities across dLLMs are currently lacking, making it difficult to attribute improvements in end-to-end task success to better initial generation or better self-correction. Establishing such protocols is an important open evaluation gap as the generate-verify-correct loop becomes the central operating mode for diffusion-based agents.

10.3 Speed, Cost, and Efficiency Metrics↩︎

Speed and efficiency are central evaluation dimensions for discrete diffusion models because their main practical promise is to reduce the sequential bottleneck of autoregressive generation. However, efficiency comparisons are often difficult to interpret: different papers report different numbers of denoising steps, different samplers, different hardware, different batch sizes, and different definitions of generated tokens. A diffusion model may use fewer serial decoding steps than an autoregressive model, but each step usually processes the full sequence with bidirectional attention; conversely, an autoregressive model may require many more decoding steps but benefits from KV caching and streaming. Therefore, speed evaluation should distinguish algorithmic step complexity, actual model forward passes, wall-clock latency, throughput, memory cost, and end-to-end system cost.

10.3.0.1 Standardizing speed.

The first quantity to report is the number of denoising steps. For a masked diffusion language model, this is the number of refinement iterations used to transform an initially corrupted sequence into a final output. For discrete-time samplers, this may correspond directly to the reverse diffusion steps; for continuous-time models, it corresponds to the chosen discretization of the reverse process; for confidence-based or blockwise samplers, it may refer to the number of remasking and update rounds [5], [17], [19], [121], [139]. Step count is useful because it captures the serial depth of generation, but it is not sufficient by itself: two samplers with the same number of steps may require different numbers of neural forward passes, different sequence lengths per pass, or different auxiliary computations.

A second metric is the number of model forward passes. Some samplers perform one denoiser call per step, while others require multiple calls for guidance, verification, rejection, speculative decoding, or self-correction. Classifier-free guidance, for instance, may require both conditional and unconditional predictions unless implemented with batching or approximation. Similarly, speculative or hybrid AR-diffusion methods may draft with one model and verify with another, so their cost cannot be summarized by diffusion step count alone [196], [197], [500]. Reporting forward-pass count makes the computational budget more comparable across samplers and model families.

A third metric is wall-clock latency, including time-to-first-output and time-to-final-output. Autoregressive models are naturally streamable: they can emit tokens as soon as decoding begins, even if the full sequence takes many steps to complete. Diffusion models typically require several denoising rounds before producing a coherent output, so their time-to-first-readable-output may be worse even when final-output latency is competitive. For interactive systems, this distinction matters. Papers should therefore report both end-to-end generation latency and, when relevant, the earliest point at which a usable partial output can be returned.

A fourth metric is throughput, measured as generated tokens per second or sequences per second. Throughput is strongly affected by batch size and sequence length. Diffusion models benefit from parallel updates across positions and from batching many sequences, making them attractive for offline or high-throughput settings. However, the per-step cost of bidirectional attention over the full sequence can be high, especially for long contexts. AR models, by contrast, have a sequential dependency over output length but exploit KV caching to reduce the marginal cost of each new token. Thus, throughput comparisons should specify whether the setting is single-query interactive decoding, batched offline generation, or long-context generation.

Hardware and implementation details should also be reported. Efficiency depends on GPU/TPU type, memory bandwidth, kernel implementation, precision, batch size, maximum sequence length, attention implementation, and whether caching or compilation is used. Recent work has begun to design diffusion-specific acceleration methods, including adaptive caching, local determinism propagation, token-wise feature caching, quantization, consistency distillation, adaptive block sizes, and one-step or few-step generation [125], [143], [182]–[185], [203], [204], [501], [502]. These methods make it even more important to separate model-intrinsic efficiency from system-level engineering. At minimum, papers should report the model size, hardware, batch size, precision, context length, output length, number of denoising steps, number of forward passes, and whether caching or acceleration tricks are enabled.

10.3.0.2 Frontier plots.

We note up front that this subsection advocates a reporting methodology rather than carrying out a new empirical comparison: we do not compile a normalized cross-paper results table, because the cited results use heterogeneous hardware, tokenizers, and budgets that we cannot re-run or fairly normalize here. The reporting recommendations below are therefore authors’ proposals aimed at future work. Because diffusion sampling exposes an explicit quality-speed trade-off, single-point efficiency numbers are often misleading. A model sampled for \(4\) denoising steps may be fast but low quality; the same model sampled for \(64\) steps may be much slower but substantially better. Similarly, confidence-based remasking, adaptive schedules, guidance strength, and block size can all shift the trade-off between latency and quality. Therefore, we recommend reporting quality-latency frontier plots rather than only a single speed number.

A useful frontier plot places a task-level quality metric on the vertical axis and latency or forward-pass budget on the horizontal axis. For language generation, quality may be measured by task-specific metrics such as ROUGE, BLEU, BERTScore, exact match, pass@\(k\) for code, constraint satisfaction rate, human preference, or LLM-judge score. The horizontal axis may be wall-clock latency, number of forward passes, denoising steps, or FLOPs. Reporting multiple x-axes can be helpful: denoising steps reveal algorithmic serial depth, while wall-clock latency reflects implementation reality. For deployment-oriented papers, tokens per second and memory footprint should also be included.

Frontier plots are especially important when comparing diffusion models with AR baselines. AR models define a natural quality-latency curve through decoding parameters such as temperature, nucleus sampling, beam size, speculative decoding, and draft-model configuration. Diffusion models define a corresponding curve through denoising steps, remasking schedules, block sizes, guidance scales, and early stopping. Comparing one diffusion setting against one AR setting can obscure the true trade-off. A stronger comparison is to plot the best achievable quality at each latency budget for each family. This makes clear whether diffusion dominates AR in a particular regime, such as batched generation or fixed-length infilling, or whether AR remains preferable, such as low-latency streaming chat.

Frontier reporting also helps diagnose where efficiency gains come from. If a method improves latency without changing the quality-latency frontier, it is primarily an implementation or constant-factor improvement. If it shifts the frontier upward, it improves sample quality at a fixed budget. If it shifts the frontier leftward, it achieves the same quality with fewer steps or less wall-clock time. This distinction is important for evaluating acceleration methods such as distillation, caching, adaptive unmasking, or speculative diffusion decoding [118], [181], [184], [196], [218]. In addition, frontier plots should include error bars or confidence intervals when results depend on stochastic sampling, because diffusion samplers can exhibit variance across random seeds, masking orders, and guidance settings.

10.3.0.3 End-to-end agent latency.

For tool-using systems, planning systems, and agentic applications, model decoding latency is only one component of the total cost. A diffusion model may be used to propose a plan, generate a set of candidate actions, refine intermediate reasoning states, or produce structured tool calls. In such settings, the full system cost depends on the number of denoising steps, the number of generated candidates, the number of tool calls, and the number of verification or repair rounds. Evaluating only raw token-generation speed can underestimate the true latency experienced by users or downstream systems.

We recommend reporting the full planning-cycle cost: \[\text{Total cost} \approx N_{\text{denoise}} C_{\text{forward}} + N_{\text{tool}} C_{\text{tool}} + N_{\text{verify}} C_{\text{verify}},\] where \(N_{\text{denoise}}\), \(N_{\text{tool}}\), and \(N_{\text{verify}}\) denote the numbers of denoising steps, tool calls, and verification rounds, respectively, and \(C_{\text{forward}}\), \(C_{\text{tool}}\), and \(C_{\text{verify}}\) denote their corresponding costs.

For systems that sample multiple candidate trajectories, the cost should also include the number of parallel or sequential candidates. If candidates are generated in parallel, wall-clock latency may be lower than total compute cost; if they are generated sequentially, both latency and compute grow with the number of samples. Tool calls may dominate the total runtime when they involve web search, code execution, theorem proving, database queries, or external simulators. Verification can also be expensive when it uses another large model, a compiler, a unit-test suite, or a domain-specific validator.

Diffusion can be attractive in agentic settings because it can revise a whole plan or structured output rather than committing to actions one at a time. For example, a model can generate a full action sequence, evaluate global consistency, remask invalid steps, and refine the plan. This global revision may reduce the number of failed tool calls or repair loops, even if the model decoding itself is slower. Therefore, the right efficiency metric is not always tokens per second, but successful task completion per unit cost. For tool-using agents, useful metrics include total wall-clock time to task completion, number of model calls, number of tool calls, number of failed tool calls, number of verification failures, total GPU time, and monetary cost.

This end-to-end perspective is also important for constrained generation and code generation. A diffusion model may generate candidate programs or structured outputs through multiple denoising rounds, after which an external compiler, parser, unit-test suite, or constraint checker filters or repairs the output. In such systems, decoding speed should be reported together with pass rate, repair rate, and validation cost. A model that produces more valid outputs may reduce downstream verification cost even if its raw decoding is slower; conversely, a faster sampler that produces many invalid candidates may be inefficient end-to-end. Thus, for agentic and tool-augmented applications, efficiency evaluation should couple generation latency with downstream execution and verification outcomes.

In summary, efficiency evaluation for discrete diffusion should go beyond denoising step count. A complete report should include model forward passes, wall-clock latency, throughput, memory, hardware, batch size, sequence length, and acceleration settings. Because diffusion exposes a continuous quality-speed trade-off, quality-latency frontier plots are preferable to single-point comparisons. Finally, for agents and tool-using systems, evaluation should measure the full planning cycle, including denoising, candidate generation, tool execution, and verification. These metrics make it possible to determine when diffusion provides a genuine systems-level advantage over AR generation and when its apparent parallelism is offset by multi-step refinement or downstream costs.

10.4 Editing, Infilling, and Conditional Fidelity↩︎

10.4.0.1 Edit benchmarks.

Editing is a task that requires models to produce outputs that are simultaneously high-quality, consistent with their context, and minimally disruptive to unmodified regions. Benchmarks for this task typically measure infilling consistency, minimal-change editing, and constraint adherence. Several benchmarks already exist across modalities. HumanEval-Infilling [503] evaluates infilling consistency for code, measuring pass@\(k\) on span-masked Python functions where the model must generate a middle segment conditioned on both a left prefix and a right suffix. SARI [504] benchmarks text editing by comparing system outputs against both the source sentence and multiple human references. It computes F1 scores over tokens that are correctly added, kept, or deleted, making it the standard metric for minimal-change rewriting tasks such as simplification and paraphrasing. For media, image and audio inpainting benchmarks assess reconstruction quality within masked regions via pixel- or token-level fidelity metrics, along with the boundary coherence between edited and preserved regions [320], [448]. For constraint adherence, COLLIE [505] provides a compositional benchmark measuring the success rate of outputs satisfying rich lexical and structural constraints across word, sentence, and paragraph levels. These existing benchmarks were designed and validated primarily on AR models. They remain applicable in discrete diffusion settings to the extent that the model is evaluated on span-level quality: HumanEval-Infilling’s bidirectional conditioning is in fact a natural fit for masked denoising, and SARI’s decomposition of adds, keeps, and deletes remains informative. However, a critical evaluation gap is that none of these benchmarks explicitly penalize out-of-span changes, which is a failure mode specific to discrete diffusion models where the denoising process may revise tokens that lie outside the targeted region. Within the discrete diffusion literature, EdiText [9] takes a step toward closing this gap by proposing a controllable coarse-to-fine text editing evaluation that tracks attribute adherence and out-of-region token preservation jointly. Establishing out-of-span preservation rate as a first-class reported quantity alongside infilling quality and constraint satisfaction is a necessary step toward reliable editing evaluation for discrete diffusion models. Concretely, given an input \(\boldsymbol{x}\), an edit region \(S\subseteq\{1,\dots,L\}\), and an output \(\hat{\boldsymbol{x}}\), we suggest reporting the out-of-span preservation rate \[\label{eq:oosp} \mathrm{OOSP} \;=\; \frac{1}{|\,\bar{S}\,|}\sum_{i\in \bar{S}} \mathbb{1}[\hat{x}_i = x_i], \qquad \bar{S}=\{1,\dots,L\}\setminus S,\tag{11}\] i.e.the fraction of tokens outside the intended edit region that are left unchanged (\(\mathrm{OOSP}=1\) means a perfectly localized edit). This is straightforward to compute, complements infilling quality on \(S\), and exposes the diffusion-specific failure mode in which denoising rewrites tokens it was not asked to touch.

10.4.0.2 Conditional alignment.

Conditional generation should be evaluated in two quantities: conditioning fidelity, which measures how closely the output follows the signal, and unconditional quality, which measures whether the output remains coherent independent of that signal. These two quantities should be tracked simultaneously and neither alone is sufficient. Across modalities, standard practice has converged on pairing one metric from each axis. For text, an externally trained attribute classifier provides fidelity scores on dimensions such as sentiment or topic, while perplexity or fluency serves as the quality control. For media, CLIP score [487] measures semantic alignment between a text prompt and a generated image or video, and is paired with FID [485] to jointly capture fidelity and distributional quality. [292] adopt this dual-reporting protocol in the context of text-conditioned masked generative image models. For structured domains such as molecules, property satisfaction rates measures the conditioning fidelity, and validity and diversity measures global quality. Since all of these metrics evaluate the final output independently of how it was generated, they transfer to discrete diffusion settings without modification. However, the gap emerges not in what is measured, but in what is left unreported. In continuous diffusion, classifier-free guidance [506] interpolates scores linearly at inference. In masked discrete diffusion, the analogous operation acts on categorical logits and is sensitive when the denoising chain guidance is applied. [161] show that high guidance applied early in the denoising chain, when most tokens are still masked, causes excessive unmasking and degrades sample quality, whereas late-stage guidance improves fidelity. This trade-off is invisible to metrics that evaluate only the finished output. [507] further show that fidelity must be measured by a held-out attribute classifier rather than the model used for conditioning, to avoid circular evaluation. For discrete diffusion models, the guidance weight used should therefore be reported as a first-class experimental variable alongside fidelity and quality scores, rather than treated as an implementation detail.

11 Discussion: Optimization, Controllability, and Trustworthiness↩︎

This section steps back from individual methods to synthesize cross-cutting themes. Section 11.1 draws together the recurring contrast between diffusion as global, iterative refinement and autoregressive generation, and reframes the reverse process as a form of discrete optimization. Section 11.2 then examines trustworthiness, distinguishing established findings from open hypotheses.

11.1 Diffusion as Global Discrete Refinement vs.Autoregressive Generation↩︎

Across the formulations, applications, and inference algorithms surveyed above, a single conceptual thread recurs: discrete diffusion generates by iterative global refinement rather than by left-to-right commitment. This subsection draws that thread together and contrasts it with autoregressive (AR) generation, then reframes the reverse process as a form of discrete optimization.

11.1.0.1 Planning interpretation.

Each denoising step conditions on the entire current sequence and updates many positions at once. The trajectory from a fully corrupted state to a clean sample can therefore be read as a coarse-to-fine planning process: early steps fix global structure under high uncertainty, while later steps resolve local detail once the surrounding context has stabilized. This stands in contrast to AR decoding, where the factorization \(p_\theta(\boldsymbol{x})=\prod_i p_\theta(x_i\mid x_{<i})\) commits to each token before the next is considered. In agentic and reasoning settings, this difference is consequential: a diffusion planner can revise an earlier part of a plan or reasoning trace after inspecting a later part, whereas AR decoding must rely on explicit revision machinery (self-correction loops, external verification, or re-decoding) to change content it has already emitted [3], [8].

11.1.0.2 Error correction across refinement steps.

Because positions are revisited over multiple steps, an early mistake need not propagate irreversibly the way it can under AR exposure bias. Remasking and confidence-based decoding (Section 7) make this explicit: tokens committed under high uncertainty can be returned to the masked state and re-predicted once neighboring tokens supply more context. The practical upside is improved global coherence and constraint satisfaction; the practical caveat is that this benefit is realized only when the sampler actually budgets steps for revision rather than greedily freezing high-confidence tokens. The empirical picture is therefore nuanced, and we frame revision as a potential advantage that depends on the decoding policy rather than a guaranteed property of the model class.

11.1.0.3 Optimization lens.

The following framing is an interpretive perspective we propose rather than an established result. The reverse chain can be viewed as a stochastic optimizer over the discrete configuration space \(\mathcal{V}^L\): it begins from a high-entropy prior and progressively concentrates probability mass on high-likelihood (or, under guidance, high-reward) regions. This perspective connects discrete diffusion to combinatorial search and offline black-box optimization, where the denoiser plays the role of a learned proposal distribution and guidance plays the role of an objective. Compared with classical heuristics such as beam search or evolutionary methods, diffusion offers learned, globally informed proposals and a natural mechanism for soft constraint handling via logit shaping; compared with exact combinatorial solvers, it trades optimality guarantees for amortized speed and the ability to exploit learned structure. We expand on these connections, including offline RL and combinatorial optimization, in Section 9.

11.1.0.4 When AR or hybrid approaches win.

Diffusion is not uniformly preferable. AR models retain decisive advantages for token-by-token streaming, incremental user interaction, and settings where mature KV-cache infrastructure makes per-token latency negligible. Many of the most effective recent systems are therefore hybrids: AR planning followed by diffusion refinement, diffusion used for editing or infilling on top of an AR backbone, block/semi-autoregressive schemes that interpolate between the two regimes, and two-stage pipelines that use diffusion for global drafting and AR for fluent local decoding. The choice between paradigms is best framed not as a winner-take-all question but as a matching of generation mechanism to task structure (latency profile, need for revision, constraint density, and streaming requirements).

11.2 Trustworthiness: Hallucination, Uncertainty, and Safety Constraints↩︎

The trustworthiness properties of discrete diffusion differ from those of AR models in ways that are only beginning to be characterized. We summarize three threads, hallucination, uncertainty, and safety, and are careful to distinguish established findings from open hypotheses.

11.2.0.1 Hallucination.

It is sometimes hypothesized that bidirectional context and multi-step revision should reduce hallucination relative to AR generation, since the model can reconcile a claim against the full surrounding context before committing. This is plausible but not yet established: the evidence base is thin, confounded by differences in scale and training data, and complicated by the fact that parallel decoding can introduce its own failure mode in which jointly sampled tokens are individually plausible but mutually inconsistent. We therefore treat reduced hallucination as a research question requiring controlled, scale-matched comparisons rather than as a settled advantage.

11.2.0.2 Uncertainty-aware generation.

A more concrete opportunity is that discrete diffusion exposes per-token, per-step confidence signals as a native byproduct of decoding. These signals, already used to drive confidence-based remasking, can in principle support uncertainty-aware behaviors: abstaining on low-confidence spans, requesting clarification, or triggering retrieval or external verification before committing. Turning these raw confidences into calibrated, actionable uncertainty estimates is an open and practically valuable direction.

11.2.0.3 Safety constraints and adversarial robustness.

Guidance, constrained decoding, and policy shaping during denoising provide levers for steering generation toward safe outputs, and the ability to enforce hard constraints (valid schemas, grammars, valence rules) at each step is a genuine strength for structured domains. At the same time, a growing body of work shows that the same mechanisms that make diffusion flexible also create new attack surfaces and defense considerations: jailbreak and red-teaming studies targeting diffusion language models [508]–[510], analyses of intrinsic safety and the role of the masking process [511]–[513], membership-inference and privacy analyses [514], [515], and adversarial prompting specific to the diffusion setting [516]. A recurring theme is that safety properties cannot be inherited wholesale from the AR literature: bidirectional, multi-step generation changes both where interventions can be applied and where vulnerabilities arise. Establishing whether safety is fundamentally easier or harder to enforce in discrete diffusion than in AR models remains open.

12 Future Directions↩︎

This section outlines open problems that we view as both consequential and tractable. Section 12.1 concerns scaling laws and benchmark standardization. Section 12.2 discusses closing the gap with autoregressive models in in-context learning, caching, and streaming. Section 12.3 considers unified token spaces, and Section 12.4 surveys theoretical questions of expressivity, convergence, and identifiability.

12.1 Scaling Laws, Compute Trade-offs, and Benchmark Standardization↩︎

Despite rapid progress, the scaling behavior of discrete diffusion remains far less understood than that of AR models. Several first-order questions remain open: whether diffusion language models exhibit scaling laws comparable to those of AR models in the large-data limit, how the number of denoising steps trades off against parameter count and training tokens, and whether the reported compute-optimal frontier — favoring smaller models trained longer — holds across domains and scales. Answering these requires standardized ablations that vary model size, step budget, noise schedule, and tokenization independently, rather than the single-point comparisons that dominate the current literature.

A closely related obstacle is the absence of shared evaluation standards. Because diffusion decoding exposes a quality-compute frontier rather than a single operating point, single-number comparisons are easy to misinterpret (see Section 10). We see benchmark standardization as a prerequisite to, rather than a complement of, settling the scaling questions above: a minimum reporting set — including the number of steps, wall-clock latency, hardware, decoding algorithm, batch and prompt lengths, and tokenization details — together with reproducible suites spanning text, graphs, and biological sequences would let the field compare methods on a common footing and transform scaling claims into testable hypotheses.

12.2 Closing the Gap with AR: In-Context Learning, Caching, and Streaming↩︎

Three capabilities that AR models provide almost for free remain difficult for discrete diffusion, and closing these gaps is among the most consequential open problems.

12.2.0.1 In-context learning.

Few-shot in-context learning appears to be weaker in current diffusion language models than in comparable AR models. The reasons for this gap remain poorly understood. Possible explanations include a mismatch between the denoising objective and the next-token prediction that ICL appears to rely on, interference between the prompt and the iteratively refined response under bidirectional attention, and insufficient scale or instruction tuning. Promising directions include multi-pass prompting, retrieval-augmented denoising, step-wise conditioning schemes that protect the prompt from corruption, and hybrid AR-prompted diffusion decoding.

12.2.0.2 Caching and streaming.

As discussed in Section 7.5, bidirectional attention and whole-canvas updates invalidate the append-only KV cache that underpins efficient AR serving, and they also complicate streaming output, since no token is final until the reverse process completes. Recent work on block-wise and delayed KV reuse, incremental unmasking caches, speculative refinement, and confidence-gated freezing shows that partial reuse is achievable, but a clean analogue of the AR KV cach, one that delivers comparable per-token serving efficiency without sacrificing the revision flexibility that motivates diffusion, has not yet emerged. Architectural changes that enable partial reuse while preserving bidirectional refinement, and decoding schemes that emit committed prefixes incrementally, are both worth pursuing.

12.3 Unified Token Spaces and Cross-Modal/Scientific Integration↩︎

A recurring theme of this survey is that the discrete state space is a first-class design axis. A natural frontier is therefore the design of unified token spaces: shared discrete vocabularies that span modalities (text, image, audio, video tokens) versus modality-specific codebooks coordinated through a common diffusion process. Unified spaces promise simpler architectures and cross-modal transfer, but raise hard questions about how to balance vocabulary granularity, codebook topology, and per-modality corruption schedules within a single model. In scientific domains, an analogous opportunity is the joint modeling of sequence and structure tokens — for example, coupling amino-acid sequence with structural descriptors, or molecular graphs with 3D conformer tokens. Another promising direction is to incorporate physics- and chemistry-based constraints directly into the corruption and reverse processes rather than enforcing them only at sampling time.

12.4 Theory: Expressivity, Convergence, and Identifiability↩︎

The theoretical foundations of discrete diffusion are advancing but remain incomplete. Regarding convergence, recent analyses establish discrete-time and uniformization-based guarantees and error bounds for score-based and flow-matching discrete samplers [517]–[522], but a unified account of how schedule, transition design, and parameterization jointly govern sampling error is still missing. Regarding expressivity, an important question is what classes of distributions factorized reverse models can represent, and how the gap to fully joint models scales with sequence length and step count; results on the parallel-decoding and information-theoretic limits of diffusion language models [523]–[526] begin to address this. Regarding identifiability, the bias introduced by approximate objectives and the conditions under which different parameterizations recover the same reverse process deserve systematic study [527], [528]. Beyond their intrinsic interest, such results would have practical payoff: a sharper theory would tell us how to choose schedules, design transition matrices, and anticipate failure modes rather than discovering them empirically.

13 Conclusion↩︎

This survey has presented a unified treatment of discrete diffusion models, organized around the thesis that the construction of the discrete state space, namely tokenization, is a first-class design axis rather than merely a preprocessing detail. We reviewed the major formulation families under a common four-component structure (corruption operator, denoiser parameterization, training objective, and sampler), surveyed training objectives and inference algorithms, examined scaling and systems considerations, and mapped applications spanning text and code, tokenized multimodal generation, proteins, genomics, molecules, and graphs, planning and agents, and tabular data.

The central message is that the state-space design, the corruption process, the learning objective, and the sampling algorithm form an inseparable system: a choice made for one component constrains the others, and the most effective designs are those in which these choices are co-adapted to the structure of the data and the demands of the task. Where discrete diffusion most clearly shines is in parallel refinement, infilling and editing, and constraint-aware generation across domains, capabilities that follow directly from its bidirectional, iterative nature and that are awkward to obtain from purely left-to-right models.

Looking ahead, we expect autoregressive and diffusion paradigms to coexist rather than one displacing the other, with hybrid systems (AR planning composed with diffusion refinement, or diffusion editing layered atop AR backbones) likely to dominate practical deployments. The open problems we have highlighted, from scaling laws and in-context learning to caching, unified token spaces, and theoretical foundations, are concrete and tractable, and progress on them will determine how far the paradigm can be pushed. We hope that the unified perspective and cross-domain map offered here will help researchers navigate this rapidly evolving field and establish discrete diffusion as a powerful framework for controllable discrete generation and structured discovery.

Broader Impact↩︎

This survey covers generative modeling techniques that, while not introducing new capabilities themselves, map and make more accessible a fast-moving design space with real dual-use and deployment considerations. We summarize the principal concerns by domain and point to mitigations discussed in the technical sections, without claiming to resolve them.

13.0.0.1 Dual use in scientific generation.

The protein, genomics, and molecule/graph applications (Sections 9.5-9.7) lower the barrier to designing biological and chemical sequences with targeted properties. The same conditioning and guidance machinery that steers a model toward a therapeutic peptide or a high-binding ligand can in principle be redirected toward harmful targets. The discrete, constraint-aware nature of these methods is relevant here in both directions: hard validity and property constraints (Section 7.4) can be used to exclude hazardous regions of sequence space as readily as to include desirable ones. We encourage practitioners releasing scientific generators to pair them with property filters, access controls appropriate to the risk, and documentation of intended use, consistent with established responsible-research-in-the-life-sciences norms; we do not provide operational detail on any harmful application.

13.0.0.2 Safety of diffusion language models.

Section 11.2 reviews evidence that the bidirectional, multi-step, any-order nature of discrete diffusion changes the safety surface relative to autoregressive models: arbitrary-order decoding can enlarge the jailbreak attack surface, and watermarking and membership-inference defenses calibrated for left-to-right generation do not transfer unchanged. Translating those findings into practice, we suggest that deployment-time evaluation of dLLMs report (i) jailbreak and adversarial-prompting results under diffusion-specific, order-varying attacks rather than only AR-style attacks; (ii) membership-inference and privacy tests appropriate to the training regime; and (iii) whether any watermarking scheme used is order-agnostic. Whether safety is ultimately easier or harder to enforce in discrete diffusion than in AR models remains, as noted in Section 11.2, an open question; we flag it as a priority rather than a solved problem.

13.0.0.3 Synthetic-data privacy.

The tabular generation and imputation setting (Section 9.9) is explicitly motivated by privacy-preserving data sharing, yet the same fidelity that makes synthetic tables useful can leak information about real records. As discussed there, fidelity, privacy, and downstream utility are in tension, and optimizing fidelity alone can degrade privacy. We recommend that work in this area report membership-inference or distance-to-closest-record analyses alongside fidelity and utility, rather than a single aggregate score, so that the privacy cost of a given fidelity level is visible.

13.0.0.4 Accessibility and concentration of capability.

Finally, by consolidating scattered techniques into a single design reference, this survey is intended to broaden access to discrete diffusion methods. We view wider, better-documented access, together with the reporting and evaluation standards proposed throughout, as net positive for reproducibility and scrutiny, while recognizing that the same accessibility applies to the dual-use concerns above. We have tried to keep the level of detail at the level of design principles and citations to published work, and to avoid step-by-step guidance toward misuse.

References↩︎