1 Introduction↩︎

Knowledge graphs encode information as structured networks of entities and their relations. Generating natural language descriptions from such graphs requires a model to interpret this relational structure and render it as fluent, faithful text. Autoregressive large language models (LLMs) have demonstrated strong performance across many natural language generation tasks and have become the dominant approach for graph-to-text generation, achieving strong results on benchmarks such as WebNLG [1]–[3]. However, text-only autoregressive models struggle to fully capture the structure of graphs, often leading to information omission or hallucination [4].

These models generate text one token at a time in a fixed left-to-right order, meaning they must commit to a linearization of the graph structure from the very first token, without the ability to revisit or revise earlier choices. Furthermore, prior work has shown that autoregressive LLMs frequently struggle with complex graph planning, triplet ordering, and factual grounding, especially when the number of graph triplets increases [5], [6]. Whether this sequential constraint is merely an inconvenience or a fundamental limitation for graph-to-text generation remains an open question.

Masked diffusion language models (MDLMs) [7], [8] offer a fundamentally different generation process. Rather than producing tokens sequentially, MDLMs such as LLaDA [9] and Dream-7B [10] begin with a fully masked sequence and iteratively unmask tokens over multiple refinement steps. Unlike autoregressive models, MDLMs select which positions to resolve at each step, making irreversible commitments in a flexible, content-dependent order rather than a predetermined sequential one. This flexibility is intuitively well aligned with how humans approach the same task: when translating a graph into text, we typically first identifies which entities are present and how they relate before composing full sentences. In principle, an MDLM could mirror this process, first establishing which entities to mention and then filling in the relational language connecting them.

Despite this potential, MDLMs remain largely unexplored in the graph-to-text generation literature. It is unclear how they construct text from graph-structured inputs, whether their generation trajectories exhibit systematic patterns, and whether they offer practical advantages over autoregressive LLMs on this task. To address these questions, we apply LLaDA to the WebNLG [11] and LAGRANGE [12] benchmarks and conduct a detailed analysis of its generation trajectory—the order in which tokens are predicted over the course of iterative unmasking. Our analysis reveals that LLaDA follows a clear content-first generation pattern: it first predicts entity tokens, then resolves relational and function-word tokens together, and finally commits structural tokens such as punctuation. The entity-first behavior replicates across both benchmarks and on another MDLM, Dream-7B [10], indicating that it is a property of masked diffusion rather than of a particular backbone or benchmark.

Our trajectory analysis also uncovers a previously unidentified failure mode: after supervised fine-tuning (SFT), LLaDA tends to commit structural tokens, such as the period token (“.”) and the end-of-sentence (EOS) token, very early in the denoising process, before substantive content has been resolved. To mitigate premature structural commitment, we propose \(\lambda\)-scaled structural decoding, a training-free inference-time modification that downweights the confidence of structural tokens, delaying their commitment and substantially improving generation quality.

Motivated by these findings, we propose Graph-LLaDA, which augments the LLaDA generation pipeline with a Graph Transformer [13] encoder that explicitly incorporates graph structure into the diffusion process. The encoder produces structure-aware node representations that are injected into LLaDA’s input sequence alongside the linearized triples, enabling the model to attend to both the relational topology and the lexical content of the input graph.

Our main contributions are as follows:

• We present the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation and establish strong baselines on WebNLG and LAGRANGE.

• We conduct a detailed analysis of MDLM generation trajectories on graph-to-text, characterizing the order in which the denoising process commits to entities, relations, and structural tokens.

• We identify premature structural commitment as a failure mode of SFT-trained MDLMs and propose \(\lambda\)-scaled structural decoding to mitigate it.

• We propose Graph-LLaDA, which augments the LLaDA generation pipeline with a Graph Transformer encoder that explicitly incorporates graph structure into the decoding process.

2 Methodology↩︎

We first define the graph-to-text generation task, then present our methodology for analyzing the generation trajectories of masked diffusion models, propose a decoding modification motivated by this analysis, and finally describe how we extend LLaDA with a graph encoder to incorporate structural information from knowledge graphs.

2.1 Problem Formulation↩︎

Graph-to-text generation is the task of automatically generating fluent and faithful natural language text from structured graph inputs. Formally, each triple \(t = (s_i, r_i, o_i)\) represents a directed edge in the graph, where \(s_i\), \(r_i\), \(o_i\) denote the subject, relation, and object, respectively. A set of such triples \(S = \{t_1,t_2,\ldots,t_n\}\) forms a directed graph that encodes structured factual knowledge. Given \(S\) as input, the goal is to generate a natural language description \(D\) that accurately and completely conveys the semantic information encoded in the graph.

2.2 Generation Trajectory Analysis↩︎

LLaDA is a masked diffusion language model that generates text by iteratively unmasking a fully masked sequence, treating mask tokens as the noise to be removed. At each denoising step, the model predicts all masked positions simultaneously using bidirectional attention and commits the most confident predictions. This process repeats until every token is unmasked. A central question of this work is whether masked diffusion models follow systematic patterns when constructing text from graph-structured inputs—specifically, whether core elements such as entities and relations are resolved at particular stages of the diffusion trajectory. To answer this question, we record the complete unmasking trajectory of LLaDA and Dream-7B during inference.

2.2.0.1 Trajectory Recording.

We record each committed token along with its position in the output sequence and the denoising timestep at which it was committed. We normalize each token’s commitment timestep to \([0, 1]\), where \(0\) corresponds to the first denoising step and \(1\) to the final step. This allows trajectory statistics to be compared across generations of different lengths and across models that use different numbers of denoising steps. The resulting trajectory provides a complete temporal record of the generation process, capturing when each output token was decided.

2.2.0.2 Token Classification.

To analyze whether the generation trajectory reflects the structure of the input graph, each generated token is classified into one of five categories: entity, relation, stop word, punctuation, or other. Entity and relation tokens correspond to content derived from the input triples, stop words capture function words, punctuation captures sentence-level delimiters, and the other category contains tokens — typically connective phrasing or paraphrastic content — that could not be mapped into any of the preceding categories. In practice, other tokens constitute a small minority of all generated tokens and are heavily concentrated at the end of the trajectory, so they do not undermine our analysis (see Table 1). We apply a priority-based procedure in which each token is tested against a sequence of classification criteria, and the first match determines its label. Details of this procedure can be found in Appendix 7.2.

2.3 \(\lambda\)-Scaled Structural Decoding↩︎

A second observation from our trajectory analysis motivates an inference-time intervention. After supervised fine-tuning, LLaDA shifts toward committing structural tokens—sentence-terminating punctuation, end-of-sequence, and end-of-turn markers—very early in the denoising process, before the bulk of the content has been resolved. This premature commitment effectively fixes the length and segmentation of the output in advance, producing generations that are either padded with redundant content or truncated. We defer the supporting quantitative evidence—first-period timing, content-before-period proportions, and per-\(\lambda\) recovery curves—to Section 4.1.0.1. To the best of our knowledge, this failure mode has not been previously identified, and we view its diagnosis as one of the main contributions of this work: it isolates a concrete mechanism by which SFT can degrade generation quality in MDLMs despite improving token-level accuracy.

To address this premature structural commitment, we introduce a simple inference-time modification that downweights the confidence of structural tokens. At each denoising step, before tokens are committed, we multiply the predicted confidence of any structural token by a scalar \(\lambda \in (0, 1]\). This causes structural tokens to require higher raw confidence before being selected (or rather it requires that all other tokens have sufficiently low confidence), delaying their commitment relative to content tokens and reducing the risk of premature sequence termination. A potential alternative is using an MDLM supporting flexible-length generation [14]; however, their approach requires significant additional training, while our modification requires no retraining and can be applied to any MDLM at inference time.

2.4 Graph-Augmented LLaDA↩︎

Orthogonal to the decoding modification above, we also investigate whether explicitly encoding graph structure improves generation quality. We propose Graph-LLaDA, which extends LLaDA with a Graph Neural Network (GNN) encoder that provides structural information during generation. We chose LLaDA as the base model because it was the first masked diffusion language model scaled to competitive performance with many autoregressive models. Dream-7B [10] was another candidate for this work; however, as shown in Table 3, its zero-shot performance was significantly lower than LLaDA. LLaDA 2 [15] was also considered but was not chosen due to its significantly larger compute requirements. Our approach does not place any constraints on model architecture or selection and can in principle be extended to other MDLM backbones.

As illustrated in Figure 1, the model receives two parallel representations of the input knowledge graph: a graph stream that encodes relational topology through learned embeddings, and a text stream that includes the raw triples as linearized text for lexical grounding. This dual representation allows the GNN to capture graph structure while ensuring that entities and relations remain directly accessible to the language model.

2.4.0.1 Graph Encoding.

The input triples are parsed into a directed graph where entities are nodes and relations are edges. Node and edge features are initialized by tokenizing each entity or relation with LLaDA’s tokenizer, retrieving the corresponding pretrained token embeddings, and mean-pooling over the token sequence. This produces a semantically meaningful initialization grounded in LLaDA’s representations. Since LLaDA’s embedding dimension is larger than what the GNN requires, a learned linear projection maps the pooled embeddings down to the GNN’s hidden size \(d_g\). Edge features are derived the same way through an analogous projection.

The resulting graph is then encoded using a Graph Transformer, which updates node representations by attending over their neighborhoods. Crucially, edge features participate in this attention mechanism: relation embeddings are used to modulate the attention weights between connected nodes, so that the model can distinguish, for example, a birthPlace edge from a nationality edge between the same pair of entities. This is particularly important for graph-to-text generation, where relations often correspond to the main verbs or predicative content of the target sentence, and faithfully preserving their semantics is essential for factual accuracy. Finally, a learned MLP projects the GNN’s output embeddings back into LLaDA’s embedding space, so that the structure-aware node representations are semantically aligned and dimensionally consistent with LLaDA’s token embeddings and can be directly inserted into the sequence.

2.4.0.2 Embedding Injection.

The text prompt includes a <graph> marker that expands into a sequence of \(N\) placeholder tokens (one per node in the input graph), delimited by <g_start> and <g_end> markers. After tokenization, each placeholder position is replaced with the corresponding projected node embedding from the GNN.

We adopt this per-node injection strategy, following GraphGPT [16], rather than compressing the entire graph into a single token as in G-Retriever [17]. Per-node tokens allow the model to attend to individual entity representations, which is particularly important for multi-hop reasoning over knowledge graphs. Recent work [18] has confirmed that single-token pooling creates an information bottleneck and that increasing the number of graph tokens recovers the lost performance.

The prompt additionally contains the raw triples rendered as bracketed tuples (e.g., [subject, relation, object]), providing lexical grounding alongside the structural signal. During both training and inference, the graph token positions are never masked by LLaDA’s forward diffusion process and are excluded from the diffusion loss.

2.4.0.3 Training.

We follow a two-stage training strategy similar to the approach of [16]. In stage (1), we freeze the LLM and train only the GNN and the projection layers. This forces the projector to learn a mapping from the GNN’s output space into LLaDA’s embedding space, aligning the two representations. In stage (2), we freeze the GNN and train the projection layer alongside LoRA [19] weights on the LLM, allowing the language model to adapt to the graph-conditioned inputs.

3 Experimental Setup↩︎

3.0.0.1 Datasets.

We evaluate on two benchmarks. Our primary benchmark is WebNLG v3.0 [1], a manually curated dataset for graph-to-text generation. Each input consists of 1–7 RDF triples extracted from DBpedia, paired with one or more reference descriptions. We train on the WebNLG training split and evaluate on the English test set, which contains 1,779 input triple sets. All results are reported using multi-reference evaluation. Although WebNLG is a high-quality benchmark, its ontology coverage is limited and its graph structures are relatively constrained [4].

To assess generalization beyond WebNLG’s domain, we additionally evaluate on LAGRANGE [12], a large-scale graph-text aligned corpus created by aligning Wikipedia sentences with Wikidata. Models are not trained on LAGRANGE; we evaluate using either zero-shot inference or models already trained on WebNLG. The one exception is GAP, which we also train on LAGRANGE to examine whether its strong in-domain performance transfers out of domain on WebNLG.

3.0.0.2 Models.

We adopt LLaDA-8B as our primary model to investigate how masked diffusion language models handle entities and relational structures in graph-to-text generation. For MDLMs, we also evaluate Dream-7B [10] in a zero-shot setting. For autoregressive baselines, we evaluate Qwen3-8B [20], LLaMA 3.1-8B [21], and LLaMA 3.3-70B under both zero-shot and LoRA-based supervised fine-tuning. For supervised baselines, we compare against GAP [22], JointGT [23], and MGSA [24]. We retrain GAP ourselves to ensure a fair comparison; results for the remaining supervised baselines are taken from published numbers.

3.0.0.3 Implementation.

All models are fine-tuned with LoRA; full hyperparameters and training details are provided in Appendix 7.1. Graph-LLaDA uses a Graph Transformer encoder followed by a 2-layer MLP projector, trained in two stages as described in Section 2.4. Unless otherwise noted, we apply \(\lambda\)-scaled structural decoding with \(\lambda=0.25\) at inference time. This is chosen as it offered the best performance, as shown in Table 2. We use a shared prompt template across all models without per-model tuning (Appendix 7.3).

3.1 Evaluation.↩︎

3.1.0.1 Metrics.

We report four automatic metrics: BLEU [25], METEOR [26], ROUGE-L [27], and CIDEr [24]. These capture lexical overlap with the reference descriptions but cannot fully reflect semantic quality or detect hallucinations, so we complement them with two preference-based evaluations.

3.1.0.2 Human Evaluation.

First, we conduct a human evaluation on a random sample of 50 WebNLG outputs from our best model. Two English speakers independently rate each output on three dimensions: fluency, hallucinations, and omissions. The samples are uniformly distributed by KG size to allow comparison across input complexity. We additionally ask Claude Sonnet 4.6 and GPT 5.5 to perform the same annotation for comparison with the human judgments. Results are reported in Table 12.

3.1.0.3 LLM-Judge Evaluation.

Second, we scale the preference evaluation across all analyzed models using an LLM judge in a pairwise head-to-head setup. For each model pair, the judge evaluates 1,000 outputs drawn from both WebNLG and LAGRANGE, rating each comparison on the same three dimensions. We use two judges: Gemma 4 31B [28] and Qwen 3.6 [29]. To mitigate positional bias, we evaluate each pair twice with the output order reversed, with preference retained only when the judge’s decision is consistent across both orderings, and inconsistent judgments are treated as ties.

The resulting pairwise preferences are converted into Elo ratings. We compute separate Elo ratings for fluency, hallucinations, and omissions. Because Elo scores can be sensitive to the order in which comparisons are processed, we repeat the computation over 1,000 random permutations of the comparison sequence and report the mean score across permutations.

4 Results and Analysis↩︎

We organize our results around three questions: how MDLMs construct text from graph-structured inputs, whether their generation trajectories exhibit exploitable patterns, and whether they offer practical advantages over autoregressive LLMs. We first analyze the generation trajectories of LLaDA and Dream-7B, showing that \(\lambda\)-scaled structural decoding corrects the premature structural commitment induced by SFT. We then benchmark Graph-LLaDA against autoregressive and supervised baselines under both in-domain and out-of-domain conditions. Finally, we complement automatic metrics with human and LLM-based preference evaluations, which reveal that surface-level overlap metrics substantially understate MDLM generation quality and obscure meaningful differences in hallucination and omission rates.

4.1 Analysis of Generation Trajectories↩︎

4.1.0.1 MDLMs exhibit strong entity-first generation.

As shown in Table 1 (and visually in Figure 2), ARLMs left-to-right decoding process naturally follows the compositional structure of the target text, such as entity–relation–entity, with punctuation and boundary markers typically produced near the end of the sequence. In contrast, MDLMs are not constrained by this sequential decoding order. On both WebNLG and LAGRANGE, LLaDA and Dream consistently exhibit an entity-first unmasking pattern. Rather than constructing the output strictly from left to right, they preferentially recover high-confidence entity tokens at early denoising steps, and subsequently fill the remaining masked positions with relational, punctuation, and auxiliary content.

4.1.0.2 SFT induces premature structural commitment.

When compared with base LLaDA (Table 8), SFT predicts structural tokens significantly earlier in the generation trajectory. Because the structure tokens are predicted early, it effectively fixes the output length before filling the content tokens. This leads to omitted or hallucinated outputs if the prematurely fixed length is too short or long. After SFT, the proportion of content tokens generated before the first period drop sharply from 60.9% to 17.8% and the mean and median first period tokens are generated significantly earlier (0.720 \(\rightarrow\) 0.385). Furthermore, the number of period tokens doubles from 1.22 to 2.44 (due to the misplaced initial period) and the effective length of the sentence drops from 21.8 tokens to 20.3.

4.1.0.3 \(\lambda\)-scaled structural decoding mitigates premature commitment.

Applying \(\lambda\)-scaled structural decoding effectively mitigates the tendency of SFT-tuned LLaDA to generate structural tokens at earlier stages of the trajectory by delaying structural commitment during generation. Table 7 illustrates how \(\lambda\) controls the generation order of period and EOS tokens during unmasking. As \(\lambda\) decreases, both period and EOS tokens are predicted progressively later in the generation trajectory. Meanwhile, the proportion of content tokens generated before the first period rises from 17.8% to 89.9%, indicating that lower \(\lambda\) values encourage the model to resolve semantic content before committing to sentence structure.

Interestingly, as shown in Figure 4, the generation trajectory of \(\lambda\)-scaled MDLMs more closely resembles the sentence generation behavior observed in LLaMA models. This is largely because MDLMs use structure tokens as anchors for later content. Without them, MDLMs are less confident in predicting sparser and longer-range outputs.

4.2 Benchmark Comparison↩︎

In the zero-shot setting, LLaDA-8B underperforms every autoregressive baseline including LLaMA 3.1-8B and Qwen3-8B². SFT alone is insufficient to close this gap: LLaDA-8B\(_\text{SFT}\) without \(\lambda\)-scaling loses 9.4 BLEU-4 points relative to the \(\lambda\)-scaled variant (Table 3), directly implicating the premature structural commitment failure mode identified above. With both SFT and \(\lambda=0.50\), LLaDA-8B reaches performance comparable to LLaMA 3.1-8B\(_\text{SFT}\). Published supervised baselines substantially outperform all LLM/MDLM models in-domain (best: 66.5 vs. BLEU-4).

4.2.0.1 Out-of-Domain Generalization.

Out-of-domain evaluation on LAGRANGE (Table 4) reveals a markedly different picture. GAP exhibits the largest performance degradation across all models and is consistently outperformed by every LLM and MDLM. This indicates that small encoder-decoder supervised models learn dataset-specific stylistic patterns rather than generalizable graph-to-text mappings. LLM and MDLM models retain substantially more of their in-domain performance out of domain, confirming that scale and pretraining generalize more robustly than task-specific supervision alone.

4.3 Human and LLM Evaluation↩︎

Both humans and LLMs consistently assign high scores (\(\geq\)​90%) to Graph-LLaDA on fluency, omissions, and hallucinations (Table 12), indicating that its generation quality is considerably stronger than automatic metrics alone suggest. Agreement between annotators is very high, though Cohen’s \(\kappa\) remains low for most categories (Table 13). This is expected under skewed score distributions: when nearly all outputs receive the highest rating on a given dimension, there is little disagreement for \(\kappa\) to measure, and the statistic collapses toward zero even when annotators are in near-perfect agreement. The high raw agreement rates are therefore more informative than \(\kappa\) in this setting.

4.3.0.1 Pairwise LLM-based Evaluation.

As shown in Tables 10 and 9, GAP ranks among the worst-performing models under both in- and out-of-domain conditions despite its strong automatic metric scores, owing to hallucinations and omissions that lexical overlap metrics do not penalize. This further shows the inadequacy of surface-level metrics for graph-to-text evaluation.

In-domain, the dominant factor is model size: LLaMA 3.3-70B leads by a wide margin over all 8B-class models except Graph-LLaDA, which closes most of that gap despite being nearly nine times smaller. This suggests that the graph encoder provides a meaningful quality signal that partially compensates for the capacity disadvantage. Out-of-domain, the ranking shifts in favor of MDLMs. LLaDA-8B and Graph-LLaDA take the top two spots, with LLaMA 3.3-70B falling to third—a reversal that points to stronger generalization in masked diffusion models relative to their autoregressive counterparts, at least on this distribution. LLaDA’s advantage is concentrated in fluency, while Graph-LLaDA’s edge over base LLaDA is more pronounced on omissions, consistent with the graph encoder improving coverage of input triples.

5 Related Work↩︎

5.0.0.1 Decoding Dynamics and Structural Token Behavior in MDLMs.

Several concurrent works have observed related pathologies in MDLM decoding. [30] note that LLaDA’s SFT instruction model commits EOS tokens early before meaningful content is resolved, framing this as a length-control problem. [31] identify an “EOS Trap” in which EOS confidence is systematically higher than non-EOS tokens, and address it via block-wise decoding constraints. More broadly, [32] and [33] propose training-free decoding strategies to mitigate premature commitment of unstable or high-uncertainty tokens across the trajectory. Our work differs in attributing the failure mode specifically to SFT, characterizing it via concrete trajectory statistics, and proposing \(\lambda\)-scaled structural decoding as a targeted fix.

5.0.0.2 Graph-to-text generation.

Early systems relied on dedicated graph encoders [34], but [2] showed that linearizing triples and fine-tuning BART [35] or T5 [36] reaches state-of-the-art on WebNLG without graph-specific inductive bias—at the cost of brittleness to meaning-preserving input permutations [37]. A second line of work re-injects graph structure into a pretrained backbone: JointGT [23] adds structure-aware aggregation and graph–text alignment objectives, GAP [22] uses only graph-aware attention masks, and MGSA [24] combines entity- and word-level structural attention. More recently, LAGRANGE [12] introduces a much larger Wikipedia/Wikidata-aligned KG–text corpus, constructed under a cyclic-equivalence criterion to improve graph–text faithfulness; we use it in Section 4.1.0.1 as an out-of-distribution test set to probe whether WebNLG-trained models actually generalize. Our work follows the structural-encoding tradition but is the first to pair it with a masked diffusion decoder.

5.0.0.3 Masked diffusion language models.

Discrete diffusion for text traces back to D3PM [7], whose absorbing-state variant connects diffusion to masked language modeling. MDLM [8] reduces the objective to a mixture of masked-LM losses competitive with autoregressive perplexity, and SEDD [38] surpasses GPT-2 via a score-entropy reformulation. LLaDA [9] is the first 8B masked diffusion LLM trained from scratch and is competitive with LLaMA-3 8B on instruction following; Dream-7B [10] reaches comparable performance by initializing from an autoregressive checkpoint with context-adaptive noise rescheduling. Closer to our setting, [14] propose any-order, flexible-length masked diffusion as a training-time fix for fixed-length generation; our \(\lambda\)-scaled structural decoding addresses a related symptom—premature commitment to length-fixing structural tokens—but requires no retraining, and we return to this comparison in the conclusion as the natural principled alternative to our inference-time intervention.

5.0.0.4 Graph-augmented LLMs.

Several recent systems couple a GNN encoder with a pretrained autoregressive LLM. G-Retriever [17] extracts query-relevant subgraphs and feeds them to a frozen LLM via a single pooled graph token, while GraphGPT [16] uses a per-node sequence of graph tokens delimited by start/end markers; [18] show that G-Retriever’s single-token pooling creates an information bottleneck that more graph tokens recover. Graph-LLaDA inherits GraphGPT’s per-node injection pattern but combines it with a masked diffusion decoder. To the best of our knowledge, no prior work has paired an MDLM with a graph encoder for any task, and no discrete-diffusion language model of any kind has been applied to KG-to-text generation: Graph-LLaDA is novel on both axes.

6 Conclusion↩︎

We presented the first systematic study of masked diffusion language models for graph-to-text generation. By recording the full unmasking trajectories of LLaDA and Dream-7B on WebNLG and LAGRANGE, we showed that masked diffusion naturally adopts a content-first decoding order—entities first, then relations and function words, with structural tokens last—without any explicit decoding-order supervision.

In analyzing the same trajectories under supervised fine-tuning, we identified a previously undocumented failure mode of SFT-tuned MDLMs: premature commitment to sentence-ending tokens, which fixes output length and segmentation before substantive content has been resolved. We addressed this with \(\lambda\)-scaled structural decoding, a training-free inference-time fix that recovers up to \(+9.4\) BLEU-4. Building on these findings, we proposed Graph-LLaDA, which integrates a Graph Transformer encoder into LLaDA via a two-stage training pipeline. Under pairwise LLM-judge evaluation, Graph-LLaDA closes most of the in-domain gap to LLaMA 3.3-70B and ranks second out-of-domain on LAGRANGE—ahead of LLaMA 3.3.

Limitations↩︎

6.0.0.1 Single-run experiments.

Due to the large number of experiments run and the known computational inefficiencies of LLaDA, we only run our experiments a single time. Because LLaDA uses confidence-based generation, we expect outputs to remain consistent across runs and our prior experience supports this, though we did not validate it in this work.

6.0.0.2 Single MDLM backbone for SFT and \(\lambda\)-scaling.

Our SFT, \(\lambda\)-scaling, and Graph-LLaDA experiments all use LLaDA-8B as the backbone. While we replicate the entity-first trajectory finding on Dream-7B (Appendix 8.4), the SFT failure mode and the \(\lambda\)-scaling fix have only been validated on LLaDA, and we have not tested them on Dream-7B, LLaDA-MoE, or LLaDA 2. We expect the findings to generalize to other MDLMs but do not claim this is established.

6.0.0.3 Single-token-per-step decoding.

All experiments use single-token-per-step generation, since multi-token decoding degrades quality substantially in our setting. Combining \(\lambda\)-scaling with recent multi-token decoding strategies is left to future work.

6.0.0.4 English graph-to-text only.

Our evaluation is limited to two graph-to-text benchmarks (WebNLG and LAGRANGE) in English; we do not investigate multilingual generation or knowledge-graph question answering settings.

6.0.0.5 LLM-judge Elo bias.

The LLM-judge Elo evaluation, while controlled for positional bias and aggregated over many permutations, inherits whatever biases the judge models hold. We treat it as a complement to, not a replacement for, the (smaller) human evaluation.

6.0.0.6 Rule-based token classification.

Our token-classification procedure for trajectory analysis is rule-based and has a small residual other category that we exclude from the categorical means. This residual is concentrated late in the trajectory and is reported in Table 1.

7 Methodology Details↩︎

7.1 Implementation Details↩︎

For LLaDA SFT, we fine-tune with LoRA [19] (\(r=16\), \(\alpha=32\)) for 5 epochs using random-token masking. For Graph-LLaDA, the GNN encoder uses a Graph Transformer with a hidden dimension of 512 (See Table 5), and the projector is a 2-layer MLP. Stage 1 training (frozen LLaDA, GNN and projector only) runs for 5 epochs; Stage 2 (joint fine-tuning with attention-only LoRA) runs for 3 epochs. During inference, we use iterative unmasking with confidence-based token selection. Unless otherwise noted, we apply \(\lambda\) scaling (\(\lambda = 0.50\)) to structural tokens and set the generation length to 128 tokens. Qwen3-8B is evaluated with chain-of-thought reasoning enabled, as disabling it substantially degrades performance (BLEU-4 drops from 48.35 to 25.10).

All training and inference was done on nodes with 1x–8x H100 GPUs. Only LLaMA 3.3 required 2 or more GPUs, all LLaDA models can fit on a single H100. Claude Code, OpenAI Codex, and/or Github Copilot were used in the development of this work.

While masked diffusion models can generate multiple tokens per step, in this paper we study single-token generation only, as multi-token generation leads to significant performance degradation. Additionally, many recent works [39] have explored different strategies for improving multi-token generation, thus any findings related to standard multi-token generation may not be applicable to real-world implementations.

7.2 Token Classification Details↩︎

This section expands the priority-based token classification procedure summarized in Section 2. Each generated token is assigned to exactly one of five categories (entity, relation, stop word, punctuation, other) by the first matching rule.

7.2.0.1 Surface-form matching against the input triples.

For each triple, subject and object names are normalized (underscores replaced with spaces, parenthetical suffixes stripped) and augmented with common format variants (numeric comma formatting, date expansions). Predicate names are similarly normalized and split on camelCase and underscore boundaries into individual content words. A case-insensitive search over the decoded output text marks spans that match an entity or relation surface form, with entity matches taking priority over relation matches when spans overlap.

7.2.0.2 Linguistic classification with spaCy.

For tokens not resolved by surface-form matching, we apply spaCy-based (en_core_web_sm) linguistic classification. Tokens whose named-entity label falls within a standard set (person, org, gpe, date, etc.) are labeled as entities; proper nouns and numerals are also labeled as entities. Verbs are labeled as relations, and nouns and adjectives that fuzzy-match a relation content word are likewise labeled as relations. Punctuation tokens are identified by spaCy’s is_punct flag, and function words by is_stop or membership in a closed set of part-of-speech tags (det, adp, aux, pron, cconj, sconj, part).

7.2.0.3 Fallback lexicon.

As a final fallback, tokens still labeled as other are checked against a conservative manually-built lexicon of common relation realizations (e.g., married, band, city) and reclassified as relations if matched. End-of-sequence and end-of-turn special tokens are excluded from the analysis throughout.

7.3 Prompt Templates↩︎

We use a single instruction template across all models. The template is wrapped in each model’s chat format using tokenizer.apply_chat_template with add_generation_prompt=True. We deliberately did not tune prompts per model so that performance differences reflect model capability rather than prompt engineering.

7.3.0.1 Shared instruction (LLaDA zero-shot / SFT, LLaMA, Qwen).

The triples list is interpolated as one bullet per triple, using the WebNLG subject | predicate | object format verbatim.

You are given a set of knowledge graph triples. Generate a single natural-language sentence that describes the information in the triples.
Requirements:
- The sentence must be fully consistent with the triples.
- The sentence must cover all facts in the triples.
- The sentence must not add any facts not present in the triples.
- Only output the sentence and nothing else.
Triples:
{list_of_triples}

For Qwen3-8B the reasoning mode is disabled by appending /no_think after the rendered chat template, following the model’s documented switch [20]. LLaMA and Dream use the rendered template unchanged.

7.3.0.2 Graph-LLaDA.

For Graph-LLaDA the user message contains both a bracketed rendering of the triples (for lexical grounding) and a single <graph> marker whose embedding is replaced at the input-embedding layer by the projected GNN node sequence (Section 2.4). The expanded token stream the model actually sees is <g_start> <g_patch> ... <g_patch> <g_end>, with one <g_patch> per graph node.

You are given a knowledge graph. Generate a single natural-language sentence that describes the information in the graph.
Requirements:
- The sentence must be fully consistent with the graph.
- The sentence must cover all facts in the graph.
- The sentence must not add any facts not present in the graph.
- Reuse the entity and relation names from the triples when possible.
- Only output the sentence and nothing else.
Triples:
{list_of_triples}
Graph tokens: <graph>

8 Additional Results↩︎

8.1 Ablation Studies↩︎

8.1.0.1 Stage 1 LoRA Configuration.

Table 5 reports the effect of GNN hidden dimension on Stage 1 performance (frozen LLaDA, 5 epochs). Performance is non-monotonic, peaking at dimension 512 (53.61 BLEU-4) and declining at higher dimensions. At 5 epochs, dimension 512 already matches dimension 256 trained for 8 epochs (53.56 BLEU-4). The gap widens with graph complexity: at 5 triples, dimension 512 leads by 7.34 BLEU-4. Larger dimensions (1024–4096) consistently underperform, likely because the high-dimensional GNN representations do not project effectively into LLaDA’s embedding space.

8.1.0.2 Stage 2 LoRA Configuration.

Starting from a dimension-512 Stage 1 model, we compare attention-only LoRA against all-linear LoRA for Stage 2 fine-tuning. Attention-only LoRA provides a modest gain (+0.52 BLEU-4 to 54.13), while all-linear LoRA slightly hurts overall performance (\(-\)​0.77 BLEU-4 to 52.84). The additional capacity from adapting MLP layers introduces instability on complex graphs: 7-triple BLEU-4 drops from 41.17 to 36.10. The WebNLG training set (35K examples) appears too small to benefit from the larger parameter budget of all-linear LoRA (21M vs 12.6M trainable parameters).

8.2 Per-Triple Performance Breakdown↩︎

As shown in Table 6, BLEU-4 consistently decreases as the number of input triples increases across all models. The degradation is most pronounced for zero-shot LLaDA-8B, which drops sharply from 62.41 to 32.53, indicating its limited ability to handle larger graphs. SFT mitigates this: SFT LLaDA-8B achieves the smallest performance spread of 13.48 across triple counts. Graph-LLaDA (\(\lambda=0.25\)) obtains the best BLEU-4 scores in buckets 1–4 and 7, while maintaining a spread of 16.72, suggesting that the graph encoder improves both overall quality and robustness to graph complexity.

8.3 Trajectory Visualizations and Detailed Metrics↩︎

To better understand why supervised fine-tuning yields smaller performance gains for LLaDA than for other models, we further analyze the graph-to-text generation process, as illustrated in Figure 4 and Figure 3. This subsection collects the trajectory figures and supporting tables that accompany the trajectory analysis of Section 4.1.0.1. Figure 4 shows aggregated generation-step distributions by token category across LLaMA, base LLaDA, SFT LLaDA, SFT LLaDA with \(\lambda\)-scaling, and Graph-LLaDA. Figure 3 provides a direct base-vs-SFT comparison. We quantify this phenomenon in Table 8. Tables 8 and 7 report the first-period and content-before-period statistics that underpin the SFT failure-mode discussion and the \(\lambda\)-scaling recovery curve.

8.4 Dream-7B Trajectory Analysis↩︎

We replicate the trajectory analysis on Dream-7B’s WebNLG predictions using the procedure of Section 2, with Dream’s own tokenizer. Mean unmasking steps are reported alongside LLaDA in Table 1; we summarize the comparison here.

8.4.0.1 The entity-first ordering generalizes.

Dream commits entities at mean step \(0.423\) with slope \(-0.90\), closely matching LLaDA’s \(0.385\) (\(-1.35\)) on WebNLG and \(0.427\) (\(-0.87\)) on LAGRANGE. Entities are clearly separated from every other category in both backbones, supporting the claim that content-first decoding is a property of masked diffusion rather than of any particular MDLM.

8.4.0.2 Fine-grained structural token ordering diverges.

In LLaDA, punctuation is the latest-committed category (mean \(0.631\)), producing a clean three-tier ordering (entities, relations/stop words, then punctuation). In Dream, punctuation (\(0.616\)) is interleaved with stop words (\(0.613\)) and slightly precedes relations (\(0.655\)), collapsing to a two-tier structure (entities, then everything else).

We do not interpret the Dream-vs-LLaDA divergence as evidence about which architecture is “correct”—it may instead reflect differences in pretraining objective, instruction tuning, or output-length calibration. We did not run Dream SFT or \(\lambda\)-scaling experiments and so cannot directly compare its post-SFT failure mode to LLaDA’s.

8.5 Generation Examples↩︎

Representative examples are shown in Figure 5. Base LLaDA tends to produce fluent outputs with few hallucinations but omits facts from the input graph. SFT alleviates omissions but can introduce repetition. Graph-LLaDA without \(\lambda\)-scaling is less stable in controlling output length, leading to truncated or hallucinated outputs in some cases. Graph-LLaDA with \(\lambda{=}0.25\) consistently produces fluent sentences that cover all facts in the input graph.

8.6 Pairwise LLM Win Rates↩︎

7pt

As shown in Figures 6, 7, and 8, GAP achieves competitive fluency under in-distribution conditions but is consistently outperformed by all other models on faithfulness and coverage—consistent with the automatic metric results discussed in Section 4.1.0.1. LLaDA similarly scores highly on fluency while tending to omit graph content, as illustrated in Figure 5.

Table 11 reveals that both Gemma 4 and Qwen 3.6 exhibit substantial positional bias, consistently favoring Position-1 responses across all evaluation dimensions, with the effect most pronounced for fluency. This bias persists after response-order swapping and is weakest for faithfulness, suggesting that more subjective dimensions are disproportionately susceptible to presentation-order effects. We mitigate this by retaining a preference only when the judge’s decision is consistent across both orderings, treating inconsistent judgments as ties (Section 2).

9 Extended Discussion↩︎

9.1 Alternatives to \(\lambda\)-Scaled Structural Decoding↩︎

Before settling on \(\lambda\)-scaled structural decoding, we considered three other inference-time interventions for the early-structural-commitment problem. We summarize them here for completeness; each is straightforward to implement, but none addressed the root cause as cleanly as \(\lambda\)-scaling.

9.1.0.1 Confidence-threshold remasking.

A natural baseline—and the strategy underlying Mask-Predict [40]—is to refuse to commit any token whose confidence falls below a fixed threshold, regardless of whether it is in the top-\(k\) for the step. This does not address our failure mode: by construction, the structural tokens we want to delay are exactly the tokens whose confidence is highest early in the trajectory (Table 8). A confidence threshold thus has no effect on premature period/EOS placement unless it is set so high that legitimate content tokens are also rejected, in which case the trajectory grinds to a halt. \(\lambda\)-scaling instead directly downweights structural-token confidence so they no longer dominate the top-\(k\) ranking.

9.1.0.2 Position-conditional commitment.

A second alternative is to forbid commitment of structural tokens until some fraction of positions have been unmasked—e.g., “never commit a period while more than \(32\) positions remain masked.” This works in principle but has the same flavor of arbitrariness one usually wants to avoid: the threshold is in absolute positions, so it does not generalize across generation lengths (a threshold tuned for \(\mathit{gl}=128\) is wrong for \(\mathit{gl}=32\) or \(\mathit{gl}=256\)), and the boundary between “allowed” and “forbidden” is a step function rather than a soft penalty. \(\lambda\)-scaling replaces this with a single dimensionless scalar that scales naturally across generation lengths.

9.1.0.3 Entropy-aware commitment.

A third alternative is to commit a structural token contingent on low entropy of the model’s distribution over non-structural alternatives at that position—intuitively, ’placing a period only when the model is confident there is nothing else to say. We initially found this attractive because it appears principled, but on reflection, it is operationally close to \(\lambda\)-scaling: in both cases, structural tokens require a higher implicit bar to be committed, and the bar is satisfied roughly when the rest of the sequence has been resolved. The entropy formulation adds a per-position entropy computation and an extra hyperparameter (the entropy threshold) without changing the qualitative effect. We therefore prefer the simpler \(\lambda\) formulation.

Together these alternatives clarify the design space and motivate \(\lambda\)-scaling as the minimal intervention that targets the root cause—inflated confidence on a small set of structural tokens—without depending on absolute positions or auxiliary entropy estimates. We do not claim \(\lambda\)-scaling is optimal (cf.the conclusion), only that it is the simplest fix in this family that we found to work reliably.

9.2 Future Work↩︎

We view \(\lambda\)-scaled structural decoding as a diagnosis-driven workaround rather than a final solution to premature structural commitment. Its optimal value depends on training quality, it requires a per-checkpoint sweep, and it modifies inference rather than fixing the underlying calibration of the SFT-tuned model.

More principled alternatives include training-time fixes such as flexible-length MDLMs [14], calibration losses targeting structural tokens specifically, or learned remasking schedules that adapt to the unmasking trajectory. Investigating these, and extending the SFT failure-mode analysis to other MDLMs (Dream-7B, LLaDA-MoE, LLaDA 2)—we have only confirmed the base-model entity-first pattern on Dream—are the most natural follow-ups to this work.

References↩︎

---

1. Equal contribution authors↩︎

2. We focus subsequent discussion on LLaMA 3.1-8B as the primary autoregressive baseline, as Qwen3-8B’s reasoning-oriented training makes standard LoRA fine-tuning difficult.↩︎