1 Introduction↩︎

Recent progress in LLM reasoning has increased the demand for reasoning supervision [1]–[3]. However, scaling such supervision remains bottlenecked by trade-offs among reliability, diversity, and cost [4], [5].

Prior approaches address this trade-off in three ways. (1) Human expert annotation yields reliable reasoning chains but is prohibitively expensive at the process level [4]–[6]. (2) Synthetic data from teacher models improves scalability [7], [8], but inherits and propagates latent reasoning errors [9], [10]. (3) Rule- or template-based generation reduces cost through programmatic construction, but narrow templates restrict factual coverage [11]. Despite these differences in how reasoning data is generated, a structural limitation emerges once it is used for contrastive supervision: candidate differences are often concentrated in a few entity-slots, while supervision is applied at the full-sequence level.

We formalize this gap as the Resolution Mismatch Problem (Fig. 1): preference is often determined by a few discriminative entity-slots, while optimization is applied to the full sequence. The bottleneck is not how many candidates we compare, but at what resolution we compare them. Standard DPO [12] (Fig. 1-a) and its multi-candidate extensions (Fig. 1-b) widen the comparison set but retain sequence-level granularity, which dilutes the signal on informative entity-slots [13]–[15].

Knowledge graphs provide a natural way to address this mismatch: under fixed schemas, alternative paths yield candidates that share the same template, with differences localized to entity-slots [16]–[18]. We instantiate this idea in KARMA (Knowledge graph-based Automated Reasoning Materialization and Alignment), a unified framework whose synthesis pipeline produces such candidates and whose learning component, Slot-Parallel Alignment (SPA), operates directly on them (Fig. 1-c).

The synthesis pipeline verbalizes KG paths under a shared template [19]–[21] and uses recurrence-based support as a structural prior for preference assignment. This exposes explicit discriminative entity-slots while expanding factual coverage beyond hand-written templates.

On the learning side, SPA applies a decoupled slot-level objective that supervises construction-exposed entity slots while retaining the shared template as a language-modeling signal. For efficient packed evaluation, SPA can optionally use slot-aware masked attention to approximate per-candidate log-likelihoods in a single forward pass without changing the slot-level supervision signal. Unlike prior token-level methods [22]–[25] that learn or estimate which tokens matter, SPA targets entity-slots that are explicitly exposed by construction in KARMA-synthesized data.

When data and optimization share the same slot-level resolution, preference learning can focus on discriminative evidence while preserving shared templates. We evaluate KARMA across biomedical, chemistry, and computer-science benchmarks, where it yields consistent gains over base LLM and SFT baselines.

Our contributions are:

• Aligned contrastive synthesis from KGs: We propose a KG-grounded synthesis pipeline that constructs chosen-rejected candidates sharing a verbalization template by design, localizing preference differences to entity-slots and using KG path recurrence as a structural prior for preference assignment.

• Slot-Parallel Alignment (SPA): We introduce a preference optimization method whose resolution matches the data: a decoupled slot-level objective applies preference and anchor losses to construction-exposed entity slots while preserving shared templates as a language-modeling signal. We further provide an optional slot-aware masked attention implementation that reduces redundant candidate evaluation without estimating token importance.

2 Related Work↩︎

2.0.0.1 Trade-offs in Reasoning Supervision

Reasoning supervision faces a cost-coverage trade-off: human process supervision yields reliable step-level signals but is prohibitively expensive [4]–[6], teacher-distilled data scales better but degrades under recursive use [7], [9], [10], [26], and template-based generation lowers cost but restricts factual coverage and collapses contrastive differences onto a few entity-slots [11], [27]. KARMA turns this collapse into a resource: by grounding candidate construction in KG paths, it expands factual coverage while keeping discriminative variation localized to entity-slots by construction.

2.0.0.2 Fine-grained Preference Alignment

Sequence-level preference optimization [12], [28]–[30] dilutes supervision on the few discriminative positions when candidates share most tokens [14], [25]. Token-level methods address this by estimating which tokens matter via importance scoring, learned masks, or oracle selection [22]–[24], [31], [32], but require additional learning or external signals to identify them. SPA instead exploits a structural prior from KARMA-synthesized data: discriminative entity-slots are known by construction, so supervision routes to them directly without estimation.

2.0.0.3 Knowledge Graph Verbalization

Knowledge graphs ground language models by verbalizing structured facts into natural language [19]–[21], [33], with reasoning-on-graph approaches further leveraging KG paths as faithful intermediate traces [16]–[18]. Both lines treat each verbalized path as a single positive instance, leaving the structural property that alternative paths share endpoints and schema unused. KARMA exploits exactly this property to expose slot-level discriminative structure for SPA.

3 Methodology↩︎

3.1 Overview↩︎

We propose KARMA, a knowledge graph (KG)-grounded contrastive synthesis pipeline, and SPA, a slot-level preference optimization method co-designed for KARMA-synthesized data. KARMA enumerates alternative KG paths between fixed source-target entities under a fixed relational schema and verbalizes them into structurally aligned contrastive candidates that share a template scaffold while differing only at discriminative entity-slots. SPA then aligns preference optimization with this entity-slot-level structure through two core learning mechanisms: (i) slot-parallel organization of candidate variants under shared context, and (ii) a decoupled hybrid objective that applies preference supervision to entity slots while preserving the shared template. Slot-aware masked attention is an optional packed-evaluation mechanism that reduces redundant forward passes by approximating per-candidate log-likelihoods in a single forward pass. Together, KARMA and SPA realize a single principle: align the granularity of supervision with the structural granularity of the data.

3.2 KARMA Data Synthesis: KG-Grounded Contrastive Candidate Construction↩︎

KARMA constructs slot-aligned contrastive candidates from a domain knowledge graph in three steps: it enumerates schema-constrained paths between source-target entity pairs, selects top-\(K\) alternatives per candidate pool by balancing support and structural diversity, and verbalizes the selected paths with a schema-specific template shared within the pool. The resulting candidates share the same source and target entities and relation schema, differing only at their intermediate entity slots. This slot-localized variation is the interface to SPA: it confines contrastive information to entity slots, exposing preference signal at the same granularity as the data variation.

3.2.1 Path Enumeration↩︎

KARMA first constructs candidate pools by enumerating schema-constrained multi-hop paths from a domain knowledge graph (KG) \(\mathcal{G}=(\mathcal{N},\mathcal{E})\), where \(\mathcal{N}\) is the set of entities and \(\mathcal{E}\subseteq \mathcal{N}\times\mathcal{R}\times\mathcal{N}\) is the set of typed edges over relation types \(\mathcal{R}\). A relation schema specifies an ordered pattern of typed edges that candidate paths must instantiate. KARMA uses such schemas to collect alternative paths that share the same schema and source and target entities while differing only in their intermediate entities (Fig. 2A).

For a given source entity \(e_s\), target entity \(e_t\), and relation schema, KARMA enumerates a schema-constrained candidate pool \(\mathcal{P}\), a source-to-target subgraph of \(\mathcal{G}\), whose elements take the form \[{ p^{m} = \left( e_s, e_1^{(m)}, e_2^{(m)}, \ldots, e_{n-1}^{(m)}, e_t \right) },\] where \(m=1,\dots,M\) indexes paths within the pool, \(n\) denotes the number of hops in the relation schema, \(e^{(m)}_j\) denotes the entity at the \(j\)-th intermediate position of path \(p^m\) for \(j=1,\dots,n-1\), and \(e_s,e_t\) denote the source and target entities.

Each such pool corresponds to one source-target pair and serves as the candidate set for one training instance. Paths in the same pool share the same relation schema and source and target entities, but may differ in their intermediate entities. Equivalently, each intermediate slot \(j\) induces a set of distinct slot entities \(\mathcal{S}_j=\{e_{j,1},\dots,e_{j,d_j}\}\), from which each path selects one entity, so \(e_j^{(m)}\in\mathcal{S}_j\). This property allows KARMA to construct contrastive alternatives whose differences are localized to intermediate entity slots.

3.2.2 Top-\(K\) Candidate Selection↩︎

Given a candidate pool \(\mathcal{P}\) of size \(M\), written \(\mathcal{P}=\{p^1,\dots,p^M\}\), KARMA selects \(K\) paths by balancing path support and structural diversity. We measure the support of each path by the cumulative recurrence of its intermediate entities within the pool: each intermediate entity contributes its occurrence count across \(\mathcal{P}\), and the path’s support is the sum of these contributions. Paths whose intermediate entities frequently recur thus receive higher support, reflecting structural connections that are commonly shared under the same source-target pair.

KARMA then greedily admits paths in descending order of support, subject to an overlap budget \(\delta\) that limits the number of intermediate entities shared with previously admitted paths (Fig. 2B). This budget enforces structural diversity by preventing near-duplicate candidates. The procedure stops when \(K\) paths have been admitted; pools with fewer than \(K\) admissible paths are discarded.

We denote the admitted paths by \(p^{\star}_k\) (\(k=1,\dots,K\)), indexed in admission order, with intermediate entities written as \(e_j^{(k)}\) for slot \(j\) of the \(k\)-th selected path. The ordered selected path sequence \[\mathcal{P}^{\star}=(p^{\star}_1,\dots,p^{\star}_K)\] preserves this ordering through verbalization and provides the preference ordering used by SPA: \(p^{\star}_1\) corresponds to the chosen candidate and \(p^{\star}_{2:K}\) to the rejected candidates. We empirically validate this chosen-rejected partition with a GPT-4 judge study in Appendix 11, and report end-to-end ablations against random path selection in Section 4.3.

3.2.3 Template Verbalization↩︎

Each selected path \(p^{\star}_k\in\mathcal{P}^{\star}\) is verbalized into a natural-language sequence \(y^{k}\) by populating a schema-specific template with the entities along the path. The template provides a fixed scaffold of textual fragments and entity slots determined by the relation schema, while the entities of \(p^{\star}_k\) are realized as multi-token spans following a domain-specific rule. Within a candidate pool, all selected paths share the same source and target entities and the same relation schema, so they are verbalized under an identical template scaffold. Consequently, the resulting ordered candidate sequence \(\mathcal{Y}=(y^{1},\dots,y^{K})\) differs only at the intermediate entity slots, while all template fragments and endpoint entities remain shared (Fig. 2C). This slot-localized variation is precisely the structural property SPA exploits in the next section.

Repeating this procedure over all retained source-target pairs yields the training corpus \[\mathcal{D}=\{(x_i,\mathcal{Y}_i)\}_{i=1}^{N}, \qquad \mathcal{Y}_i=(y_i^{1},\dots,y_i^{K}),\] where \(x_i\) is a prompt constructed from the source and target entities of the \(i\)-th instance, and \(\mathcal{Y}_i\) is the ordered verbalized candidate set inheriting the chosen/rejected ordering from the corresponding selected path sequence.

3.3 Slot-Parallel Alignment for Slot-Level Preference Learning↩︎

SPA realigns preference optimization from sequence-level to slot-level resolution, matching the granularity at which KARMA candidates actually differ. It concentrates supervision on discriminative entity slots through a decoupled hybrid objective: slot-level preference and anchor losses supervise entity tokens, while a template LM loss preserves the shared scaffold. Slot-parallel packing evaluates variants under shared context, and slot-aware masked attention optionally obtains per-candidate log-likelihoods in one forward pass instead of \(K{+}1\), at a small accuracy cost.

3.3.1 Slot-Parallel Packing with Shared Context↩︎

For a single training instance, we drop the corpus index \(i\) and write \(\mathcal{Y}=(y^{1},\dots,y^{K})\). Following the path notation in Section 3.2.2, the candidates in \(\mathcal{Y}\) differ only at the intermediate slots \(j=1,\dots,n-1\), where slot \(j\) holds the entity \(e_j^{(k)}\) in candidate \(y^{k}\). Let \(t_0,\dots,t_{n-1}\) be the shared template fragments, and let \(e_j^{(1)}\) denote the chosen entity at slot \(j\), which we refer to as the anchor. SPA constructs the packed sequence \[\mathbf{s} = \bigl[ x;\; t_0,\; e_1^{(1{:}K)},\; t_1,\; \dots,\; e_{n-1}^{(1{:}K)},\; t_{n-1} \bigr],\] where \(e_j^{(1{:}K)}\) denotes the ordered concatenation of slot-\(j\) entity variants \(e_j^{(1)},\dots,e_j^{(K)}\). As illustrated in Fig. 2E, SPA groups the variants of each intermediate slot into a local slot bundle while sharing the surrounding template fragments. We shuffle the intra-slot variant order during training to prevent positional shortcuts.

3.3.2 Slot-Aware Masked Attention↩︎

Since candidates in \(\mathbf{s}\) share all template fragments and differ only at the intermediate entity slots, \(K{+}1\) standalone forward passes would redundantly recompute identical template representations. SPA replaces the causal attention mask with a slot-aware mask so that all per-candidate log-likelihoods are obtained from a single forward pass while approximating their standalone values.

We organize the mask by what each query token is allowed to attend to within the causal prefix. Template tokens at fragment \(t_j\) attend only to the preceding template fragments and to the anchor entity of each past slot. An entity token \(e_j^{(k)}\) at slot \(j\) in candidate \(k\) attends to the preceding template fragments, to the past anchors \(e_1^{(1)},\dots,e_{j-1}^{(1)}\), and to itself. Consequently, each candidate’s log-likelihood within the packed sequence is computed along its own anchored path, approximating the value it would have under an independent forward pass.

Fig. 3 (left) illustrates the resulting mask for a packed sequence with two intermediate slots and three variants per slot. With a plain causal mask (top right), later tokens can attend to multiple variants from previous slots, so a candidate’s log-likelihood within the packed sequence diverges from its standalone value. The SPA mask (bottom right) restricts each candidate to its own anchored path while keeping the template representation shared, so one forward pass over \(\mathbf{s}\) approximates \(K{+}1\) independent passes while encoding the template scaffold only once.

3.3.3 Decoupled Hybrid Objective↩︎

Given per-candidate log-likelihoods obtained from the packed sequence, applying a sequence-level preference objective to \(\mathbf{s}\) would still spread supervision over many template tokens that are identical across candidates. Since KARMA candidates differ only at intermediate entity slots, SPA decouples the objective according to this structure: a slot-level preference loss supervises entity choices, an anchor loss stabilizes the chosen entities, and a template language modeling loss preserves the shared scaffold (Fig. 2F).

3.3.3.1 Slot-level preference.

For each slot \(j\), the anchor entity \(e_j^{(1)}\) should be preferred to the rejected entities \(e_j^{(k)}\) from candidates \(k=2,\dots,K\). Let \(\pi_\theta\) denote the policy model and \(\pi_{\mathrm{ref}}\) a fixed reference model. For an entity span \(e_j^{(k)}\), we write \(\ell_\theta(e_j^{(k)}\mid\mathbf{s})\) and \(\ell_{\mathrm{ref}}(e_j^{(k)}\mid\mathbf{s})\) for its length-normalized log-likelihood under \(\pi_\theta\) and \(\pi_{\mathrm{ref}}\), computed under the slot-aware mask. We define the reference-normalized score \[r_\theta(e_j^{(k)}) = \ell_\theta(e_j^{(k)} \mid \mathbf{s}) - \ell_{\mathrm{ref}}(e_j^{(k)} \mid \mathbf{s}).\] This score is the policy’s log-likelihood gain over the reference on the entity span \(e_j^{(k)}\). SPA applies a squared margin loss between the anchor and each rejected entity at the same slot: \[\label{eq:pref} \begin{align} \mathcal{L}_{\mathrm{pref}} &= \frac{1}{(n-1)(K-1)} \\ &\quad \times \sum_{j=1}^{n-1} \sum_{k=2}^{K} \left( r_\theta(e_j^{(1)}) - r_\theta(e_j^{(k)}) - \frac{1}{2\beta} \right)^2, \end{align}\tag{1}\] where \(\beta\) controls the target margin. Here, \(k\) follows the original top-\(K\) candidate order: \(k=1\) is the chosen candidate and \(k=2,\dots,K\) are rejected candidates. This term provides the discriminative signal at the slots where candidates differ, rather than spreading preference supervision over the shared template.

3.3.3.2 Anchor confidence.

The preference term in Eq. 1 constrains relative scores, but it does not directly ensure that the chosen entity has high likelihood. SPA therefore adds an anchor confidence term: \[\label{eq:anchor} \mathcal{L}_{\mathrm{anchor}} = -\frac{1}{n-1} \sum_{j=1}^{n-1} \ell_\theta(e_j^{(1)} \mid \mathbf{s}).\tag{2}\] This term stabilizes the likelihood of the selected path entities, complementing the relative comparison imposed by \(\mathcal{L}_{\mathrm{pref}}\).

3.3.3.3 Template language modeling.

The template fragments encode the relation scaffold shared by all candidates, but they are not discriminative for the slot-level preference comparison. Let \(\mathcal{T}\) denote the set of shared template token positions in \(\mathbf{s}\), and let \(s_u\) denote the \(u\)-th token of \(\mathbf{s}\). SPA preserves the scaffold with a standard causal language modeling loss: \[\label{eq:template} \mathcal{L}_{\mathrm{template}} = -\frac{1}{|\mathcal{T}|} \sum_{u\in\mathcal{T}} \log \pi_\theta(s_u \mid s_{<u}).\tag{3}\] This term maintains the fluency and logical structure of the verbalization while leaving entity discrimination to the slot-level losses.

3.3.3.4 Full objective.

The final SPA objective combines the slot-level preference term with the two auxiliary losses: \[\label{eq:spa95full95objective} \mathcal{L}_{\mathrm{SPA}} = \mathcal{L}_{\mathrm{pref}} + \lambda_{\mathrm{anchor}}\mathcal{L}_{\mathrm{anchor}} + \lambda_{\mathrm{template}}\mathcal{L}_{\mathrm{template}},\tag{4}\] where \(\lambda_{\mathrm{anchor}}\) and \(\lambda_{\mathrm{template}}\) are scalar loss weights. We average \(\mathcal{L}_{\mathrm{SPA}}\) over training instances.

4 Experiments↩︎

We evaluate KARMA across three knowledge-intensive domains to answer three research questions: (RQ1) Does the co-design of KARMA-synthesized contrastive candidates and SPA’s slot-level objective outperform both conventional SFT and sequence-level preference optimization on the same data? (RQ2) How important is graph-guided path selection for synthetic reasoning supervision? (RQ3) Does aligning the preference objective’s resolution with the entity-slot structure of the data yield consistent gains over sequence-level and token-level alternatives?

4.1 Experimental Setup↩︎

We evaluate on three domains: Biomedical, CS, and Chemistry, using 11 benchmarks that mix MMLU subsets [34], MMLU-Pro subsets [35], and reasoning-heavy benchmarks (MedQA [36], PubMedQA [37], GPQA [38], CSBench [39]). The base model is Qwen2.5-7B-Instruct with LoRA fine-tuning, and all evaluations use zero-shot chain-of-thought prompting. Full benchmark lists (9.1), dataset sources (7), and implementation details (8) are provided in Appendix.

4.2 Main Results↩︎

As shown in Tab. 1, KARMA yields consistent gains over the baseline across the majority of benchmarks, with the largest improvements on reasoning-heavy ones. KARMA also outperforms Open-Data SFT on the majority of benchmarks, indicating that the gains do not merely reflect additional in-domain text but the structural properties of KARMA-synthesized supervision.

The contrast between KARMA-Data SFT and KARMA further isolates the role of the training objective. KARMA-Data SFT uses identical KARMA-synthesized chosen candidates but optimizes them with a flat sequence-level SFT objective, and fails to consistently improve over the baseline. This reflects an inherent incompatibility between KARMA’s data structure and sequence-level supervision: chosen candidates from the same source-target pair share almost all tokens except at the intermediate entity-slots, so SFT on these candidates assigns equal weight to all positions and the model is dominantly supervised on the shared template tokens, which form a small set of patterns repeated across many examples. This concentrates SFT updates on a low-diversity distribution, causing the model to collapse toward the template phrasing while losing pretraining-acquired knowledge in the long tail [40], [41]. We view this as a data-side analog of the Resolution Mismatch Problem, where the mismatch arises not between preference signal and optimization granularity, but between data structure and supervision granularity. SPA resolves this by decoupling slot-level preference supervision from template language modeling, converting the same data into the strongest signal in the comparison. This confirms that the gains of KARMA arise from the co-design of structurally aligned data and slot-level supervision, not from data scale or content alone. This answers RQ1: KG-grounded contrastive synthesis improves domain reasoning beyond conventional SFT, but only when the training objective operates at the entity-slot-level resolution at which the candidates actually differ.

4.3 Ablation Study↩︎

4.3.0.1 Ablation of Graph-Guided Path Selection.

Tab. 1 shows that the full KARMA framework outperforms baselines, but does not isolate which component drives the gain. RQ2 asks whether KARMA’s support-based path selection itself contributes, or whether any sampling from the same KG would suffice. We replace support-based selection with uniform random sampling, holding all other components fixed (KG, source-target pairs, schema, template, SPA objective, hyperparameters). Tab. 2 shows that KARMA outperforms random sampling on every Biomedical benchmark, with an average gain of +16.5 points.

The gain pattern is itself informative. Improvements are largest on benchmarks that require chaining multiple facts, such as GPQA-Bio (+26.7), MMLU-Pro-Bio (+19.4), and MedQA (+16.0), and remain substantial on knowledge-heavy multiple-choice reasoning (MMLU-Bio +14.7), while PubMedQA, which often admits surface-level factual lookup, shows only a marginal gain. This selectivity is consistent with the view of KG paths as a structural analogue of reasoning chains: support-based selection acts as a simple evidence prior, where intermediate entities that recur across the pool reflect connections corroborated by multiple traversals in the graph, yielding chosen-rejected contrasts that are informative for reasoning rather than mere surface distractors. That such a lightweight, training-free heuristic, relying only on entity recurrence statistics already implicit in the KG, suffices to produce these gains answers RQ2: KARMA’s advantage comes from how paths are selected, and the KG itself already carries much of the structure needed to make that selection effective.

4.3.0.2 Ablation on Preference Objective Resolution.

Tab. 3 compares SPA with representative preference optimization baselines operating at different resolutions. The goal is to isolate the effect of preference objective resolution: whether the preference signal is applied at the sequence, token, or slot level. All methods are trained on the same KARMA-synthesized candidates and share the same training pipeline and hyperparameters; only the preference objective differs.

We consider three baselines. DPO (\(K=1\)) performs preference optimization using pairwise chosen-rejected comparisons. MDPO (\(K=4\)) extends this formulation to multiple rejected candidates while still operating at the sequence level. TDPO (\(K=1\)) introduces token-level KL regularization, providing a finer-grained signal than sequence-level objectives but without explicitly targeting the entity-slots where KARMA (\(K=4\)) candidates differ. In contrast, SPA applies supervision directly at the slot level, aligning the optimization objective with the structural units of KARMA-synthesized data.

The results show that SPA achieves the best or tied-best performance across all benchmarks, improving over the strongest baseline (TDPO) on reasoning-heavy ones such as CSBench (+2.0) and GPQA-Chem (+2.2). Sequence-level objectives assign supervision to the entire candidate, token-level regularization makes the signal more fine-grained, and SPA further aligns it with the entity-slot structure where the actual candidate differences arise. This answers RQ3: aligning the preference objective’s resolution with the data’s entity-slot structure yields consistent gains over both sequence-level (DPO, MDPO) and token-level (TDPO) alternatives, indicating that resolution alignment, beyond granularity alone, accounts for the improvement.

4.3.0.3 Ablation of Slot-Aware Masked Attention.

Tab. 4 characterizes the accuracy-efficiency trade-off of slot-aware masked attention, which we use as the default packed-evaluation implementation in SPA. The default w/ Mask variant packs all candidates into one sequence and approximates their log-likelihoods in a single forward pass, whereas w/o Mask evaluates candidates independently with \(K{+}1\) forward passes per step. Both variants share identical objectives, data, and hyperparameters; only the attention masking and packing differ. Efficiency is measured on Qwen2.5-7B-Instruct with \(K{=}4\) rejected candidates.

The two variants exhibit a clear trade-off between accuracy and training cost. On the accuracy side, disabling the mask matches or slightly improves performance on five of the six benchmarks, while the w/ Mask configuration remains within one accuracy point of the independent-evaluation variant on those benchmarks and outperforms it on GPQA-Chem. On the efficiency side, the default packed evaluation under the slot-aware mask reduces the number of forward passes per step from \(2(K{+}1)\) to \(2\), yielding a 1.38\(\times\) wall-clock speedup. The cost is marginally higher peak GPU memory, since the packed sequence concentrates all candidates into a single forward and materializes the slot-aware mask, while reserved memory remains comparable. Thus, slot-aware masked attention is the default efficiency-oriented implementation, not the source of SPA’s slot-level supervision. Users can trade wall-clock speed for peak accuracy by enabling or disabling the mask, while the preference, anchor, and template losses remain unchanged.

5 Conclusion↩︎

We propose KARMA, a KG-based pipeline that synthesizes structurally aligned contrastive candidates with explicit entity-slot-level differences. By leveraging KG paths, KARMA provides broad factual coverage while preserving structural alignment across candidates. We further introduce SPA, a slot-level preference optimization method designed for KARMA-style data. SPA addresses the resolution mismatch of sequence and token-level objectives by aligning supervision with the entity slots where candidates differ, enabling more precise preference attribution and improved reasoning performance. Together, KARMA achieves substantial gains on most knowledge-intensive reasoning benchmarks, while our analysis highlights the importance of KG path selection strategies.

6 Limitations↩︎

KARMA shares two limitations with prior KG-grounded LLM research [16]–[19]. First, the synthesis pipeline relies on a domain knowledge graph with sufficient coverage and connectivity to support multi-hop path sampling, so domains without such a KG would require automatic KG construction or hybrid retrieval-based grounding. Second, KARMA assumes that the target capability is expressible through an explicit schema and verbalization template, which restricts the framework to capabilities decomposable into discrete entity-slot structures and leaves open-ended generation outside its current scope.

Beyond these, slot-aware masked attention is an efficiency-oriented implementation choice rather than the source of SPA’s supervision signal: the default masked setting reduces per-step forward passes from \(2(K{+}1)\) to \(2\) and yields a 1.38\(\times\) wall-clock speedup, while disabling it recovers independent candidate evaluation at higher training cost. Users can choose either setting based on whether wall-clock time or peak accuracy is the binding constraint. Finally, all experiments use Qwen2.5-7B-Instruct with LoRA fine-tuning, and generality across other model families and scales remains to be verified.

Appendix↩︎

7 Dataset Construction Details↩︎

7.1 Knowledge Graph Sources↩︎

KARMA constructs training data from domain-specific knowledge graphs from GRBench [42], summarized in Table 5. Each graph is represented as a typed node-edge structure, where nodes contain textual attributes and outgoing typed neighbors. The biomedical graph uses heterogeneous node types such as diseases, symptoms, compounds, genes, anatomy, side effects, pharmacologic classes, biological processes, molecular functions, cellular components, and pathways. The academic graph uses paper nodes with citation edges, where each paper contains title, abstract, and keyword fields.

7.2 Relation Schemas↩︎

A relation schema defines the edge-type pattern used to enumerate paths from a source entity to a target entity. In the academic graph, schemas are metapaths over citation edges. We use four-hop schemas by default, generated from all combinations of reference and cited_by. In the biomedical graph, schemas are typed metapaths over biomedical node and relation types.

7.3 Path Enumeration and Filtering↩︎

For each schema, KARMA enumerates paths that follow the prescribed edge-type sequence, as described in Algorithm 4. Each path connects a source entity \(e_s\) to a target entity \(e_t\) through a sequence of intermediate entities. For a fixed source-target pair, all paths satisfying the same schema form a candidate pool. Candidate pools with fewer than \(K\) valid paths are discarded. We remove loops in intermediate positions and apply intermediate caps during traversal to avoid path explosion.

7.4 Support-based Top-K Selection↩︎

For each source-target candidate pool, we select \(K\) paths using an entity-support score and an overlap constraint, as detailed in Algorithm 5. The support score of a middle path \(m\) is the sum of the occurrence counts of its intermediate entities within the same source-target pool: \[\mathrm{score}(m)=\sum_{e \in m}\mathrm{freq}(e).\] We then greedily select high-score paths while allowing at most \(\delta\) overlapping intermediate entities with the already selected paths. In the academic-domain generator, we use \(K=4\) and \(\delta=1\).

7.5 Template Verbalization↩︎

Each selected path is verbalized into a natural-language candidate using a schema-specific template. Entity mentions are marked with bold delimiters, e.g., **entity**. These marked spans define the entity slots used by SPA. The selected paths for the same source-target pair share the same template scaffold and differ only in their intermediate entity slots.

7.6 Dataset Statistics↩︎

Table 6 reports dataset statistics. For each domain, we report the number of nodes and edges in the source knowledge graph, the number of relation schemas used for path enumeration, the number of generated training samples, and the number of candidates per pool size \(K\).

8 Training Details↩︎

8.1 Model and LoRA Configuration↩︎

All experiments use Qwen2.5-7B-Instruct as the base model. We fine-tune the model with LoRA adapters. The LoRA rank is 32 and the LoRA alpha is 32. Adapters are inserted into the attention and feed-forward projection modules. The full configuration is reported in Table 7. The reference model \(\pi_{\mathrm{ref}}\) used by SPA and all DPO-family baselines is obtained by disabling the LoRA adapter on the policy model \(\pi_{\theta}\); that is, \(\pi_{\theta}\) and \(\pi_{\mathrm{ref}}\) share identical base parameters and differ only in whether the trained adapters are active.

8.2 Training Protocol↩︎

All training runs share the same general-purpose hyperparameters (learning rate, optimizer, batch size, epochs, gradient clipping, LoRA configuration) across baselines and KARMA, so that performance differences reflect the training objective rather than tuning. Method-specific hyperparameters, such as \(\lambda_{\mathrm{anchor}}\), \(\lambda_{\mathrm{template}}\), and \(\beta\) for SPA, are set as in Table 8 for KARMA, and the corresponding original defaults are used for DPO, MDPO, and TDPO. We do not perform validation-based checkpoint selection; the final epoch checkpoint is used for evaluation. Each domain dataset is used for training only, and evaluation is performed on the held-out benchmarks listed in Table 11.

All results reported in this paper, including the main results in Table 1 and all ablations, are computed as the mean of 5 independent runs with different random seeds.

8.3 Compute Infrastructure↩︎

All experiments are run on NVIDIA A100 GPU. A single SPA training run for one domain (1M samples, 3 epochs) takes approximately 40 GPU-hours on 1 GPU. The total compute budget across 5 random seeds, three domains, all baseline methods, and ablation studies is on the order of a few thousand A100 GPU-hours.

8.4 Baseline Implementations↩︎

We compare KARMA with the base model, open-data SFT, KARMA-data SFT, and preference-optimization baselines, as listed in Table 9. All fine-tuned baselines share the same base model, LoRA configuration, learning rate, batch size, epochs, and optimizer as KARMA; they differ only in the training objective and, where applicable, in the structure of the input data. Each baseline retains its original method-specific hyperparameters (e.g., the KL coefficient in DPO, the token-level regularization weight in TDPO) at the values recommended in their respective papers.

8.5 Slot-aware Mask Implementation↩︎

SPA packs candidate variants into a single sequence by replacing each entity slot with a local bundle of chosen and rejected variants. The attention mask starts from a causal mask and then blocks cross-candidate contamination according to Table 10. PAD separators inserted between variants are blocked as both queries and keys.

8.6 SPA Loss Computation↩︎

The full SPA loss computation procedure is summarized in Algorithm 6. Given a packed sequence with slot-aware masking, the policy and reference models compute log-probabilities for each variant. Slot-level preference and anchor confidence losses are accumulated across slots, and a template language modeling loss is computed on non-entity tokens.

9 Evaluation Details↩︎

9.1 Benchmark Sources↩︎

We evaluate on biomedical, chemistry, and computer-science benchmarks. The benchmark groups are summarized in Table 11.

9.2 Prompt Templates↩︎

All benchmarks are evaluated using the same model wrapper and benchmark-specific prompt templates, summarized by category in Table 12.

9.3 Answer Extraction↩︎

Generated responses are normalized to benchmark-specific answer formats before scoring. For multiple-choice tasks, the predicted option is extracted from the generated text and compared against the gold option. For label-based QA tasks such as yes/no/maybe classification, the generated text is mapped to the corresponding normalized label.

9.4 Inference Configuration↩︎

The inference configuration used across all benchmarks is reported in Table 13.

10 Hyperparameter Sensitivity↩︎

Table 14 reports sensitivity to the loss weights \(\lambda_{\mathrm{pref}}\), \(\lambda_{\mathrm{anchor}}\), and \(\lambda_{\mathrm{template}}\) on the Biomedical domain. The analysis examines whether SPA is robust to the relative weighting of slot-level preference learning, anchor confidence, and template language modeling. Based on this analysis, we fix \(\lambda_{\mathrm{pref}}=1.0\) in all main experiments and absorb it into \(\mathcal{L}_{\mathrm{pref}}\) in the final objective (Eq. 4 ), tuning only \(\lambda_{\mathrm{anchor}}\) and \(\lambda_{\mathrm{template}}\). The selected configuration is \((\lambda_{\mathrm{pref}}, \lambda_{\mathrm{anchor}}, \lambda_{\mathrm{template}})=(1.0, 0.5, 0.1)\), which achieves the best average accuracy across MedQA, PubMedQA, and MMLU-Pro-Bio.

11 Chosen Path Validation↩︎

To verify that the support-based Top-\(K\) selection (Algorithm 5) yields a chosen path that is genuinely the most plausible among the candidates in a pool, we conduct an LLM-judge validation study using GPT-4. For each candidate pool, GPT-4 is provided with the source-target pair, the shared schema scaffold, and all \(K\) verbalized candidates without preference labels, and is asked to identify the candidate whose intermediate entities form the most factually and semantically plausible reasoning chain between the source and target. We then measure the agreement rate between GPT-4’s top-ranked candidate and the chosen path selected by our support-based procedure. Ties are conservatively counted as disagreement.

As reported in Table 15, GPT-4 agrees with our chosen selection on \(74.8\%\) of sampled pools, well above the \(25\%\) chance baseline for \(K=4\) candidates. This indicates that entity-support scoring is a reliable proxy for path plausibility, and that the resulting chosen-rejected partition reflects a meaningful preference signal rather than an artifact of the heuristic. The remaining disagreement reflects cases in which multiple candidates within a pool are comparably plausible, which is consistent with the resolution mismatch motivating SPA: at the entity-slot resolution, several variants may be defensible, and the learning objective must be capable of expressing fine-grained preferences among them.

12 Qualitative Analysis↩︎

Table 16 shows a representative example of a KARMA-generated candidate pool from the biomedical domain, and Table 17 provides the corresponding path-only view.

References↩︎

---

1. Corresponding author.↩︎