1 Introduction↩︎

Diffusion Large Language Models (dLLMs) [1]–[3] generate text by iteratively denoising a masked sequence, refining many positions at once instead of strictly left to right. Block-wise variants [4], [5] decode consecutive blocks autoregressively while denoising within each block, which restores KV-cache reuse. Reinforcement learning has become the standard way to improve their reasoning [6]–[13].

Early work applies RL to randomly masked targets [6], [14], but random masking yields high-variance gradient estimates [13], and recent work has therefore moved toward trajectory-aware methods [15], [16]. The current state of the art, TraceRL [7], trains not on random masks but on the model’s exact decoding trajectory, on the view that under random masking the post-training objective is mismatched with the trajectory the model follows at inference.

Preserving the trajectory is not free. To train on each decoding step in order, TraceRL slices a single rollout into up to \(\lceil K/s\rceil\) trajectory-aligned training samples, one forward each, where \(K\) is the block size and \(s{\ge}1\) a shrinkage factor [7]. At full fidelity (\(s{=}1\)) one rollout becomes up to \(K\) samples, so the data grows with \(K\). Raising the shrinkage \(s\) bounds the cost but aggregates \(s\) consecutive decoding steps into one slice and discards their internal order, trading trajectory fidelity for cost. A larger block therefore makes a trade-off between slicing cost and trajectory fidelity, which we quantify in Section 4.5.

The question is whether reconstructing the exact trajectory is necessary. Without slicing, a dLLM-RL run still faces two weaknesses at two independent stages, rollout generation and optimization. Rollouts are generated with dynamic sampling [17], [18], the common rollout decoder across dLLM-RL methods including TraceRL, which commits every token whose confidence exceeds a fixed threshold \(\tau\). Because this rule is pointwise, a step that unmasks many positions only marginally above \(\tau\) over-commits and injects several errors into the rollout. At optimization, random masking makes the policy-gradient estimate high-variance [13].

In this work, we address both stages and call the resulting recipe SLIM-RL, which combines three components developed in Section 3. A \(\tau\)-budget dynamic-unmasking decoder, a single-pass training-free rule shared by rollout and inference, caps each step’s cumulative confidence-based uncertainty. A variance-reduced framework then updates the policy, built from a sequence-level length-normalized ratio [19], deterministic quadrature over masking levels [20], and an unnormalized advantage [21]. A mean-preserving, monotonically decreasing per-block mask schedule front-loads the policy-gradient signal onto the earliest, most-conditioned block.

On SDAR-4B, SLIM-RL outperforms trajectory-aware TraceRL on math at block size 16 and on both math and code at block size 4, where the 4B model also surpasses the larger LLaDA-8B and Dream-7B diffusion models, and reaches TraceRL’s best accuracy on under half the training data (Table 6). The margin widens with block size, and Section 4 reports the per-benchmark results.

Our contributions are summarized as follows:

• We show that reconstructing the exact decoding trajectory is not required to match trajectory-aware RL at equal training cost. With rollout commit risk bounded and the masking objective variance-reduced, trace-free random masking is on par with trajectory-aligned slicing at block size 4 and outperforms it at block size 16.

• We introduce a \(\tau\)-budget dynamic-unmasking decoder that commits only the largest low-uncertainty subset of positions whose cumulative uncertainty \(\sum_i(1-p_i)\) stays within a calibrated budget \(m(1-\tau)\), in contrast to dynamic sampling’s per-position threshold. It is training-free and transfers across LLaDA, Dream, and SDAR.

• We introduce a mean-preserving, monotonically decreasing per-block mask schedule that concentrates the policy-gradient signal on the earliest, most-conditioned block, adding \(5.91\%\) on MATH500 over random masking.

• On SDAR-4B at block size 16, SLIM-RL achieves \(6.32\%\) and \(11.05\%\) higher accuracy than TraceRL on MATH500 and GSM8K under matched decoding, and reaches TraceRL’s best accuracy on \(0.46\times\) the training data. At block size 4, it achieves higher accuracy than the larger LLaDA-8B and Dream-7B diffusion models across math and code, and improves over TraceRL on code by \(4.20\%\) on MBPP and \(3.65\%\) on HumanEval. On SDAR-1.7B, SLIM-RL reaches TraceRL’s best on \(0.76\times\) the training data.

2 Preliminaries↩︎

2.1 Diffusion Large Language Models↩︎

2.1.0.1 Forward and reverse processes.

Given a prompt \(x\) and a clean response \(y=(y^{1},\ldots,y^{L})\), a diffusion large language model (dLLM) [1], [3] defines a forward corruption process and a learned reverse generation process. The forward process independently replaces each token with the mask symbol \([\mathrm{MASK}]\) at masking level \(t\in[0,1]\): \[q_{t}(y_{t}\mid y) = \prod_{i=1}^{L} q_{t}(y_{t}^{i}\mid y^{i}), \label{eq:forward}\tag{1}\] where each token is independently masked according to \(q_{t}(y_{t}^{i}{=}[\mathrm{MASK}]\mid y^{i})=t\) and \(q_{t}(y_{t}^{i}{=}y^{i}\mid y^{i})=1-t\), so that \(y_{0}\) is the clean response and \(y_{1}\) is fully masked. For reverse process, starting from \(y_{1}\), the model iteratively predicts masked positions and resamples a less-noised \(y_{r}\) from \(y_{t}\) (\(r<t\)) until \(y_{0}\) is recovered.

2.1.0.2 Block-wise dLLMs.

Vanilla dLLMs denoise the full response under bidirectional attention, which prevents KV-cache reuse and is expensive for long responses. Block-wise dLLMs [4], [5] partition the response into \(B\) consecutive blocks \(Y_{1},\ldots,Y_{B}\) of size \(K\), where \(Y_{b}=(y^{(b-1)K+1},\ldots,y^{bK})\). Attention is bidirectional within a block and causal across blocks, so blocks are generated sequentially while tokens within each block are denoised in parallel. This yields the block-causal factorization \[\pi_{\theta}(y\mid x) = \prod_{b=1}^{B} \pi_{\theta}(Y_{b}\mid x,Y_{<b}) \label{eq:block95factorization}\tag{2}\] which preserves KV-cache compatibility across blocks while retaining intra-block parallelism.

2.1.0.3 Confidence-driven decoding.

At each denoising step \(s\), the model scores every masked position by its max-confidence \(p_{i}=\max_{v\in\mathcal{V}} p_{\theta}(y_{0}^{i}=v\mid y_{s},x)\), where \(y_{s}\) is the partially denoised sequence at step \(s\), and commits position \(i\) only when \(p_{i}>\tau\) for a fixed threshold \(\tau\). Aggressive thresholds (small \(\tau\)) commit more tokens per step but inject errors that propagate through later denoising. Conservative thresholds preserve quality but require more refinement steps, eroding the parallelism advantage of dLLMs.

2.2 GRPO for dLLMs↩︎

For each prompt \(x\sim\mathcal{D}\), the old policy samples a group of \(G\) responses \(\{y_{j}\}_{j=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)\) with scalar rewards \(\{r_{j}\}_{j=1}^{G}\) and the standardized advantage \(\hat{A}_{j}=(r_{j}-\operatorname{mean}\{r_{i}\})/\operatorname{std}\{r_{i}\}\). Standard GRPO [22] defines a token-level importance ratio \(\rho_{j}^{k}(\theta)=\pi_{\theta}(y_{j}^{k}\mid x,y_{j}^{<k})/\pi_{\theta_{\mathrm{old}}}(y_{j}^{k}\mid x,y_{j}^{<k})\) and maximizes the per-token clipped surrogate \(\frac{1}{G}\sum_{j}\frac{1}{|y_{j}|}\sum_{k}\min\!\bigl(\rho_{j}^{k}\hat{A}_{j},\operatorname{clip}(\rho_{j}^{k},1{-}\epsilon,1{+}\epsilon)\hat{A}_{j}\bigr)\), regularized by \(\beta D_{\mathrm{KL}}[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}]\) toward a reference policy.

For dLLMs, the per-token conditional \(\pi_{\theta}(y_{j}^{k}\mid x,y_{j}^{<k})\) is intractable in closed form because the model parameterizes \(p_{\theta}(\cdot\mid y_{t},x)\) at masked positions rather than via a left-to-right factorization. Random-masking dLLM-RL methods [14], [23] factorize the sequence probability into per-token denoising terms and approximate the ratio on sampled corruptions \(\widetilde{y}_{j}(t)\) of each response at masking level \(t\sim \mathrm{U}[0,1]\): \[\rho_{j}^{k}(\theta) \;\approx\; \frac{p_{\theta}\bigl(y_{j}^{k}\mid x,\widetilde{y}_{j}(t)\bigr)}{p_{\theta_{\mathrm{old}}}\bigl(y_{j}^{k}\mid x,\widetilde{y}_{j}(t)\bigr)},\quad k\in\mathcal{U}_{j}(t), \label{eq:dllm95ratio}\tag{3}\] where \(\mathcal{U}_{j}(t)\) denotes the masked positions in \(\widetilde{y}_{j}(t)\).

3 Method↩︎

SLIM-RL separates dLLM RL into two stages: risk-controlled rollout collection, then trace-free low-variance optimization. A \(\tau\)-budget decoder (Section 3.1) shapes the rollouts by bounding the cumulative uncertainty committed at each denoising step; a low-variance random-masking objective then updates the policy on them (Sections 3.2 and 3.3), with a mean-preserving decreasing per-block mask schedule (Section 3.4) concentrating the gradient signal on the earliest, most-conditioned block.

3.1 \(\tau\)-Budget Dynamic Unmasking↩︎

\(\tau\)-budget dynamic unmasking is training-free and used for both RL rollout generation and test-time inference. At denoising step \(s\), let \(\mathcal{M}_s\) denote the currently masked positions in the active block, and let \(p_i = \max_{v\in\mathcal{V}} p_{\theta}(y_0^{i}=v\mid y_s, x)\) be the model’s max-confidence at position \(i\). The threshold-based candidate set \[\mathcal{A}_s = \{i \in \mathcal{M}_s : p_i > \tau\} \label{eq:candidate95set}\tag{4}\] is the set of positions standard dynamic sampling would commit at step \(s\). We assign each candidate the confidence-based uncertainty \(u_i = 1-p_i\) and define the cumulative step-level uncertainty of a set \(\mathcal{S}\subseteq\mathcal{A}_s\) as \(U(\mathcal{S})=\sum_{i\in\mathcal{S}} u_i=\sum_{i\in\mathcal{S}}(1-p_i)\). Given a risk-budget schedule \(m\ge 1\), we set the step-level budget \(\mathcal{B}_s = m(1-\tau)\). We sort the candidates by ascending uncertainty \(u_{i_1}\le u_{i_2}\le\cdots\le u_{i_{|\mathcal{A}_s|}}\) and commit the largest prefix of this ordering whose cumulative uncertainty stays within the budget: \[\begin{align} k_s &= \max\left\{ k: \sum_{r=1}^{k} u_{i_r} \le \mathcal{B}_s \right\},\\ \mathcal{C}_s &= \{i_1,\ldots,i_{k_s}\}. \end{align} \label{eq:budget95constraint}\tag{5}\]

The remaining positions stay masked for subsequent denoising steps. If \(\mathcal{A}_s = \emptyset\), we commit the single masked position with the highest confidence. Figure 1 contrasts \(\tau\)-budget with dynamic sampling on a single denoising step, and Algorithm 7 gives the full single-pass decoder.

3.1.0.1 Interpretation: a calibrated wrong-commit budget.

Let \(\mathrm{WrongCommit}_s=\sum_{i\in\mathcal{C}_s}\mathbf{1}\{\hat{y}_i\neq y_i^{\ast}\}\) count committed positions that disagree with the correct token. If max-confidence approximated the probability that the argmax is correct, then \(\mathbb{E}[\mathrm{WrongCommit}_s]\approx\sum_{i\in\mathcal{C}_s}(1-p_i)=U(\mathcal{C}_s)\le\mathcal{B}_s\), so the budget caps a calibrated proxy for the expected wrong commitments per step. Pointwise thresholding instead commits all of \(\mathcal{A}_s\), whose aggregate uncertainty grows with the number of accepted tokens, so many marginally-above-threshold positions can together inject large step-level risk. The calibration assumption holds only approximately, so we treat it as motivation and \(\tau\)-budget rollouts commit fewer expected-wrong tokens per step than dynamic sampling (Figure 5).

3.2 Training Objective: Variance-Reduced Random Masking↩︎

After rollouts are collected, SLIM-RL does not reconstruct the token-level decoding order. The intrinsic weakness of random masking for RL is a high-variance policy gradient. The first source of variance is the granularity of the importance ratio. A token-level ratio attaches a response-level advantage to many independently clipped per-position ratios. As argued by GSPO [19], such token-level importance weights introduce high-variance training noise that can accumulate with response length. We therefore use a sequence-level ratio, which is computed for the whole response at a given masking level.

For a masking level \(t\), let \(\widetilde{y}_{j}(t)\) be a randomly masked version of \(y_j\), let \(\mathcal{U}_{j}(t)\) denote its masked positions, and let \(M_{j}(t)=|\mathcal{U}_{j}(t)|\). The normalized sequence denoising log-score is \[\ell_{\theta}^{j}(t) = \frac{1}{M_{j}(t)} \sum_{i\in\mathcal{U}_{j}(t)} \log p_{\theta} \left( y_j^{i} \mid x,\widetilde{y}_{j}(t) \right), \label{eq:block95log95score}\tag{6}\] where \(y_j^{i}\) is the clean token at masked position \(i\).

The corresponding sequence-level denoising ratio is the geometric mean of the per-position denoising ratios over the masked positions, \[\rho_{j}^{(t)}(\theta) = \exp\!\left( \ell_{\theta}^{j}(t) - \ell_{\theta_{\mathrm{old}}}^{j}(t) \right) = \exp\!\left( \frac{1}{M_{j}(t)} \sum_{i\in\mathcal{U}_{j}(t)} \log \frac{ p_{\theta}( y_j^{i} \mid x,\widetilde{y}_{j}(t) ) }{ p_{\theta_{\mathrm{old}}}( y_j^{i} \mid x,\widetilde{y}_{j}(t) ) } \right). \label{eq:block95importance95ratio}\tag{7}\]

3.3 Deterministic Quadrature and Unnormalized Advantage↩︎

The second source of variance is the masking level \(t\), over which the random masking objective is an expectation. Prior dLLM-RL estimates this expectation stochastically, drawing \(t\) at random for each update [14], [23], and [20] show that this random time is the dominant source of the estimator’s variance. We instead evaluate the expectation deterministically with Gauss–Legendre quadrature over \((0,1)\). At \(Q\)-point nodes and weights \(\{(t_q,\omega_q)\}_{q=1}^{Q}\), \(\rho_{j}^{(t_q)}(\theta)\) is the ratio of Eq. 7 at \(t{=}t_q\), which removes the variance of the random \(t\). We also use the unnormalized advantage to remove bias [21], and let \(A_j\) denote the response-level advantage assigned to \(y_j\).

\[A_j = r_j - \frac{1}{G} \sum_{i=1}^{G} r_i. \label{eq:dr95centered95advantage}\tag{8}\]

3.3.0.1 Final objective.

The final objective combines the per-quadrature-node sequence ratios \(\rho_{j}^{(t_q)}(\theta)\), the unnormalized advantage \(A_j\) of Eq. 8 , and a KL regularizer toward the old policy.

\[\mathcal{J}_{\mathrm{SLIM}}(\theta) = \mathbb{E} \Biggl[ \frac{1}{G} \sum_{j=1}^{G} \sum_{q=1}^{Q} \omega_q\, \psi_{\epsilon}\!\left(\rho_{j}^{(t_q)}(\theta),\,A_j\right) - \beta\, D_{\mathrm{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{\theta_{\mathrm{old}}}\right) \Biggr]. \label{eq:final95objective}\tag{9}\]

We define the clipped policy objective term as \(\psi_{\epsilon}(\rho, A)=\min\!\left(\rho A,\,\operatorname{clip}(\rho,1-\epsilon,1+\epsilon)A\right)\). Algorithm 6 details the training pipeline, from risk-budgeted rollout collection to the quadrature-based policy update.

3.4 Monotonically Decreasing Per-Block Masking Schedule↩︎

Vanilla random-masking RL masks each position independently. We instead apply a monotonically decreasing per-block schedule \(p_1\ge p_2\ge\cdots\ge p_B\) across the \(B\) blocks \(Y_1,\dots,Y_B\) of a response, masking block \(Y_b\) at rate \(p_b\). Under the block-causal factorization of Eq. 2 the earliest block conditions every later block, so masking it most concentrates the learning signal on the positions the rest of the response depends on, while later blocks stay progressively less masked. The per-block rates average to the global masking level \(t\) (\(\tfrac1B\sum_b p_b=t\)) and the explicit cosine form is given in Appendix 8.1. It reaches \(32.91\) on MATH500 against \(27.00\) for uniform masking (Table 2).

4 Experiments↩︎

4.1 Setup↩︎

4.1.0.1 Data, models, and training.

Our base model is the block-wise dLLM SDAR-4B-Chat [5] with block size 16 and block size 4, SDAR-1.7B-Chat is at block size 4. Following [7] we train on the MATH set [24] (level 3–5) and PrimeIntellect-verified coding problems [25], keeping a separate model per task. On SDAR-4B-Chat this runs for \(160\) steps at block size 16 and \(100\) steps at block size 4, and the SDAR-1.7B-Chat model at block size 4 trains to convergence. The code model continues from the 4B math model at block size 4 and also trains to convergence. Each RL step samples \(128\) prompts with \(G{=}8\) responses each at temperature \(1.0\). We use \(Q{=}3\) Gauss–Legendre nodes per step, AdamW at constant learning rate \(1{\times}10^{-6}\), clip \(\epsilon{=}0.1\) (the range \([1{-}0.1,1{+}0.1]\)), and KL coefficient \(\beta{=}0.01\) (\(k_3\) estimator), on \(8\times\)A100 (40 GB) GPUs across two nodes of four (full configuration in Appendix 8, Table 7, and the evaluation details in Appendix 7.5).

4.1.0.2 Evaluation.

We evaluate on two math benchmarks, MATH500 [24] and GSM8K [26], and two code benchmarks, MBPP [27] and HumanEval [28]. Generation uses length \(256\), temperature \(1.0\), and the two decoders, dynamic sampling and \(\tau\)-budget at both block size 16 and block size 4. We compare against an autoregressive reference, Qwen2.5-7B-Instruct [29]; the full-attention dLLMs LLaDA-8B-Instruct [1] and Dream-7B-Instruct [3]; and two RL baselines, random-masking and TraceRL.

4.2 Main Results↩︎

3pt

4.2.0.1 SLIM-RL substantially outperforms the RL baselines on math.

On the SDAR-4B-Chat base at block size 16, SLIM-RL reaches \(31.36\) on MATH500 and \(71.04\) on GSM8K under dynamic sampling, and \(32.91\) and \(75.03\) under \(\tau\)-budget, well above TraceRL at \(25.04\) and \(59.99\) and random-masking RL, which barely moves off the base at \(12.82\) and \(47.71\). At block size 4 it stays ahead of TraceRL on both math sets under dynamic sampling, and the 4B model surpasses the larger LLaDA-8B and Dream-7B diffusion models, exceeding LLaDA-8B by \(10.76\%\) on MATH500 while remaining below the autoregressive Qwen2.5-7B (Table 1). The advantage also carries to a smaller model, where on SDAR-1.7B at block size 4, SLIM-RL again outperforms TraceRL on MATH500 (\(36.71\) vs.\(35.29\) under dynamic sampling) and matches it on GSM8K, with cross-scale data-efficiency reported in Section 4.5.

4.2.0.2 SLIM-RL also outperforms TraceRL on code.

Under code RL trained from the block size 4 math model, SLIM-RL improves over TraceRL by \(4.20\%\) on MBPP and \(3.65\%\) on HumanEval under dynamic sampling, while preserving math accuracy. Block size 4 parallelism (tokens-per-forward, TPF) and the decode-time block-size change are reported in Appendix 7.3.

4.3 Ablation Study↩︎

Each component contributes as a leave-one-out drop from the full recipe on SDAR-4B-Chat at block size 16, measured on MATH500 under the \(\tau\)-budget decoder where the full recipe peaks at \(32.91\) (Table 2). The largest drop is the quadrature, where using two Gauss–Legendre quadrature nodes instead of three lowers it to \(26.56\). Generating the rollouts with dynamic sampling instead of \(\tau\)-budget lowers accuracy to \(27.09\). The sequence-level importance ratio attains the full-recipe \(32.91\), whereas a token-level ratio degrades it to \(28.09\). Replacing the monotonically decreasing per-block schedule with uniform random masking lowers accuracy to \(27.00\). The \(\tau\)-budget rollout’s benefit does not transfer to TraceRL. Trained on \(\tau\)-budget rollouts, TraceRL reaches only \(22.91\) on MATH500, below the \(26.09\) of its usual dynamic sampling rollouts, so the gain is specific to trace-free random masking.

4.4 Scaling Block Size↩︎

SLIM-RL’s advantage over TraceRL grows with the block size. With both methods trained natively at each block size (Table 3), the two are on par at block size 4, where TraceRL keeps the exact (\(s{=}1\)) trajectory at low slicing cost and SLIM-RL matches it on math to within \(0.33\%\) on MATH500 and \(0.23\%\) on GSM8K under matched decoding. At block size 16 the math margin widens to \(6.32\%\) and \(11.05\%\). The same widening shows on code benchmarks. TraceRL loses code ability at block size 16, its MBPP falling to \(35.96\) and HumanEval to \(44.65\) from the base’s \(38.80\) and \(54.48\), while SLIM-RL holds near the base at \(38.84\) and \(53.39\) (Table 1). Random-masking RL improves far less than TraceRL and SLIM-RL on math at every block size, while on code it stays near the base.

4.5 Training Efficiency: Trace-Free Optimization Avoids Trajectory Slicing↩︎

4.5.0.1 Trace-free optimization attains TraceRL’s accuracy at a constant per-rollout cost.

On SDAR-4B the baseline produces \(3.55\) samples per response at block size 16 (\(s{=}4\)) and \(4.00\) at block size 4 (\(s{=}1\)), both higher than our constant \(Q{=}3\); the full trajectory (\(s{=}1\)) at block size 16 instead costs \(12.92\), with \(18.5\%\) of responses reaching the full \(16\) samples.² At block size 16, plain random-masking RL is near the base model while SLIM-RL overtakes TraceRL, reaching its best MATH500 accuracy on only \(0.46\times\) the training data (Table 6, Figure 2). At block size 4 and \(1.7\)B, it reaches TraceRL’s best on \(0.81\times\) and \(0.76\times\) the data, and uses less total data at every scale, \(0.85\times\), \(0.70\times\), and \(0.74\times\). At block size 4 the baseline keeps the full trajectory and both methods are near their best accuracy, where SLIM-RL still stays slightly ahead.

4.5.0.2 SLIM-RL also trains to a more parallel model.

Beyond accuracy, SLIM-RL commits more tokens-per-forward than TraceRL at matched decoding. At block size 4 it reaches TPF \(1.94\) vs.\(1.72\) on MATH500 under dynamic sampling and leads on every benchmark (Table 5), and at block size 16 it sustains the higher TPF throughout training (Figure 4). Because the decoder is fixed, this TPF gain comes from the trained model, not from the \(\tau\)-budget rule, which at \(m{=}1\) slightly lowers TPF in exchange for accuracy. The other panels of Figure 4 report the training dynamics, where generation length shortens for all methods, and the mask ratio for Random-masking and ours stays near \(0.5\) while TraceRL’s stays near \(0.25\).

4.5.0.3 The \(\tau\)-budget decoder transfers across architectures.

Because it reads only per-position confidences, the \(\tau\)-budget decoder transfers training-free across LLaDA, Dream, and SDAR, a drop-in replacement for dynamic sampling (Table 4). On the same model its more conservative commits trade a little throughput for accuracy, leaving tokens-per-forward just below dynamic sampling. Dream-7B is the exception, where \(\tau\)-budget instead raises tokens-per-forward above dynamic sampling on every benchmark except GSM8K, so it is faster while its code accuracy also rises.

5 Related Work↩︎

5.0.0.1 Reinforcement learning for dLLMs.

Existing RL for dLLMs either reconstructs the decoding trajectory or trains on randomly masked corruptions of the completed response. On the trajectory side, TraceRL [7] optimizes the objective on the exact decoding trace, DCoLT [15] rewards the full denoising trajectory through outcome-based RL, and d2 [16] estimates the likelihood of the decoding trajectory. The masking view instead discards the decoding order, with d1 [6] factorizing the policy through a mean-field decomposition, DiffuCoder’s coupled-GRPO [23] pairing complementary masking realizations, and MMaDA’s UniGRPO [14] adapting GRPO to masked diffusion. Closest to our objective, [20] replace the random masking level with Gaussian quadrature on the diffusion ELBO, and ESPO [30] reduces the ratio to a single sequence-level weight per response. SLIM-RL stays on the random-masking side and controls its gradient variance by recombining existing components, a sequence-level length-normalized ratio [19], deterministic quadrature over the masking level [20], and an unnormalized advantage [21]. Both act on the training objective alone, whereas SLIM-RL pairs it with the \(\tau\)-budget rollout decoder and the mean-preserving per-block mask schedule, controlling rollout commit risk together with optimization variance.

5.0.0.2 Diffusion language models.

Diffusion language models [31]–[33] differ in architecture and in how they commit tokens at inference. Full-attention dLLMs (LLaDA [1], [34], Dream [3], MMaDA [14]) scale iterative denoising to the 7–8B range but cannot natively reuse KV caches across denoising steps. Block-wise hybrids (Block Diffusion [4], SDAR [5]) generate autoregressively across blocks while denoising the active block in parallel, restoring KV-cache reuse and serving as the substrate for recent dLLM RL [7], [8], [10]. A separate line accelerates inference by adaptively committing tokens at each denoising step. Fast-dLLM [17] and Dimple [18] commit every position whose confidence exceeds a threshold (dynamic sampling), D2F [35] adds discrete diffusion forcing for faster-than-autoregressive inference, and S2D2 [36] adds training-free self-speculation. Dynamic sampling bounds only per-position confidence, so the wrong-commit risk of a step grows with the number of positions committed. The \(\tau\)-budget decoder (Section 3.1) instead bounds the cumulative step-level risk directly in a single pass.

6 Conclusion↩︎

We asked whether reconstructing the exact decoding trajectory is necessary to match TraceRL at equal training cost in diffusion language models, and found that it is not. SLIM-RL is trace-free, pairing a \(\tau\)-budget rollout decoder that caps a confidence-based proxy for step-level commit risk with a variance-reduced random-masking objective built from sequence-level ratios, deterministic quadrature, and a decreasing per-block mask schedule. On block-causal SDAR-4B, this recipe outperforms TraceRL on both math and code, reaching its best accuracy on \(0.46\times\) the training data. Whether exact trajectory reconstruction remains worthwhile at larger blocks or longer responses is left to future work.

7 Additional Experimental Results↩︎

This appendix collects the parallelism/TPF, cross-architecture, block size 4, code-RL, cross-scale, and training-dynamics results referenced from the main text. All numbers are at response length \(256\).

7.1 Ablation and Block-Size Tables↩︎

These tables support the analysis in Sections 4.3–4.4: Table 2 gives the leave-one-out component ablations and Table 3 reports the block-size dependence of the advantage over TraceRL.

6pt

4pt

7.2 Cross-Architecture Decoder Transfer and Parallelism↩︎

Because \(\tau\)-budget depends only on per-position confidences, it applies unchanged to full-attention (LLaDA, Dream) and block-causal (SDAR) diffusion models as a training-free, drop-in replacement for dynamic sampling (Table 4). Full SLIM-RL is excluded because it already trains with \(\tau\)-budget, so we include its dynamic-sampling-trained ablation in its place. On parallelism, at the conservative budget \(m{=}1\) the \(\tau\)-budget tokens-per-forward is uniformly slightly below dynamic sampling across LLaDA and every SDAR model in Table 4, the small throughput cost it pays for its accuracy gains, so we report it as an accuracy decoder rather than a speed one. Dream-7B at block size 4 is the one architecture where \(\tau\)-budget speeds up TPF on MATH500, MBPP, and HumanEval, with only GSM8K nearly the same at \(2.61\) versus \(2.60\); on the two code sets accuracy rises in step, MBPP from \(36.93\) to \(37.33\) at TPF \(4.45\) to \(4.59\) and HumanEval from \(35.57\) to \(36.18\) at TPF \(5.46\) to \(5.72\). For Dream the decoder is both faster and more accurate on code, while on MATH500 accuracy is nearly unchanged and on GSM8K it drops slightly.

4pt

7.3 Block Size 4 and Code RL↩︎

Table 5 reports the full block size 4 results: the two methods reach comparable accuracy, with SLIM-RL slightly ahead, and replacing dynamic sampling with \(\tau\)-budget decoding leaves accuracy nearly unchanged. The code-RL results, with code RL continued from the block size 4 math-RL model, are reported in the main-text Table 1 (lower group).

4pt

7.4 Training Dynamics and Cross-Scale↩︎

At \(1.7\)B scale (Figure 3, Table 6), SLIM-RL stays ahead of TraceRL on MATH500 (\(36.71\) vs.\(35.29\)) and reaches TraceRL’s best accuracy on \(0.76\times\) the training data, a data-efficiency gain that holds at a second model scale. Figure 4 shows training dynamics: SLIM-RL maintains a stable tokens-per-forward while TraceRL’s declines, so SLIM-RL ends higher, and generation length shortens for all methods.

6pt

7.5 Evaluation Details↩︎

We report the average accuracy over \(n\) sampled responses per task. For the SDAR family we set \(n{=}9\) on MATH500, \(n{=}3\) on GSM8K, and \(n{=}9\) on both MBPP and HumanEval. A coding response counts as correct only when it passes every functional test. Each diffusion model is evaluated under both decoders, dynamic sampling and the \(\tau\)-budget decoder of Section 3.1.

For the SDAR-4B-Chat and SDAR-1.7B-Chat models, we keep the pretrained block-diffusion decoding, a response length of \(256\), temperature \(1.0\), top-\(p{=}1.0\), and top-\(k{=}0\). Dynamic sampling unmasks every position above the confidence threshold \(\tau{=}0.9\), and the \(\tau\)-budget decoder reuses that \(\tau\) with budget multiplier \(m{=}1\). The 4B models are evaluated at block size \(16\) and block size \(4\), and the 1.7B model at block size \(4\).

For the full-attention baselines, we use temperature \(0.1\) and sample \(n{=}3\) responses per task across all four benchmarks. LLaDA-8B-Instruct decodes at block size \(32\) and Dream-7B-Instruct at block size \(4\), both over a length-\(256\) response. Both baselines run dynamic sampling and the \(\tau\)-budget decoder at threshold \(\tau{=}0.95\) with \(m{=}1\).

Qwen2.5-7B-Instruct is the autoregressive reference. We sample \(n{=}3\) responses at temperature \(0.1\), and raise the generation length to \(512\).

8 Algorithm Pipeline↩︎

Algorithm 6 details the SLIM-RL training loop. Each outer step samples \(G\) rollouts via \(\tau\)-budget dynamic unmasking (Algorithm 7), scores them with a verifiable reward \(r(\cdot)\), forms the unnormalized advantage \(A_j\) (Eq. 8 ), evaluates the sequence-level ratio \(\rho_{j}^{(t_q)}(\theta)\) (Eq. 7 ) at \(Q\) Gauss–Legendre quadrature nodes per response, and updates \(\theta\) on the clipped objective (Eq. 9 ).

6pt

8.1 Per-Block Mask Schedule↩︎

The monotonically decreasing per-block schedule (Section 3.4) is a cosine schedule centered at the masking level \(t\): \[p_b = t + \tfrac{\delta}{2}\cos(\pi \xi_b),\qquad \xi_b=\frac{b-1}{B-1},\] where \(\xi_b\) is the normalized position of block \(b\) among the \(B\) blocks of a response. The rate decreases from \(t+\tfrac{\delta}{2}\) at the first block to \(t-\tfrac{\delta}{2}\) at the last. Since \(\cos(\pi \xi_b)\) is antisymmetric about \(\xi_b=\tfrac12\), the per-block rates average to \(t\) exactly (\(\tfrac1B\sum_b p_b=t\)), so the schedule only reallocates a fixed expected mask count across blocks. The spread is bounded by \(\delta\le 2\min(t,1-t)\) to keep every \(p_b\in[0,1]\); we set \(\delta=0.2\) and apply the schedule at each Gauss–Legendre quadrature node \(t\).

References↩︎