1 Introduction↩︎

Grading a student’s exam response is a rule application task. A grader needs to read the student answer, compare it with an explicit mark scheme, and assign a holistic mark. Large language models (LLMs) are increasingly deployed as automatic graders [1]–[4]. However, training them to grade reliably remains difficult. The final mark depends on a sequence of intermediate judgements, such as which rubric criteria are satisfied, what evidence supports them, which mark band applies, and how partial credits compose.

This makes rubric grading a demanding credit-assignment problem. Outcome-based reinforcement learning (RL) methods such as GRPO [5] assign a a single final reward to the whole reasoning trajectory [6], and they cannot identify which reasoning step caused an incorrect mark. More fine-grained credit assignment methods, such as token-reallocation methods [7] or intervention-style training [8] offer a possible remedy.

However, these methods were primarily designed for self-contained reasoning tasks like mathematics, where the answer can often be checked by a verifier and the error is usually local. They are less suited to rule-application reasoning tasks, where judging an answer requires checking whether each step correctly invokes and applies an external rule, rubric, or policy. Rubric grading has two additional requirements. First, it requires external grounding. Rather than just maintaining internal logical consistency, each judgement step must be anchored in both the marking scheme and the student’s written evidence. Second, rule-based reasoning is an uncertainty reduction process. The model’s belief about the final mark should progressively converge. Existing methods do not directly control this decision trajectory. Outcome-only RL ignores harmful belief shifts that occur in the middle of reasoning process, while token-level constraints can over-penalise benign exploratory phrasing.

To address these challenges, we introduce Edit (Evidence-Diagnosed Intervention Training), a novel two-phase framework for training rubric-faithful LLM graders. As shown in Figure 1, rather than relying on prompt-based self-audits or rigid token-level rewards, Edit uses the model’s internal decision signals to locate grading failures, revise them locally, and calibrate how the model’s belief over the final mark evolves during reasoning.

To enforce strict external grounding, the first phase, Edit-SFT, replaces prompt-based self-audits with internal-state localisation. At each reasoning step, it inspects the model’s posterior belief over the final mark and computes mask-based grounding signals that measure whether the step depends on the student answer, the rubric, and the preceding reasoning steps. These signals identify where the reasoning first moves away from the correct mark or becomes weakly grounded. The selected step is then revised as a small atomic edit under a locality constraint. A rubric checklist is provided as privileged context, helping the reviser use the mark scheme without forcing the output to follow a rigid checklist format.

To manage the uncertainty reduction process, the second phase, Edit-RL, introduces belief-guided reward shaping for calibration RL. Instead of reallocating sequence-level reward across tokens, Edit-RL augments the standard mark-distance outcome reward with a penalty for large mid-trajectory belief drifts. Small fluctuations are allowed as benign exploration, but belief drifts that move too far from the gold mark are penalised in proportion to their excess.

Experimental results on two real-world, multi-subject student response grading benchmark demonstrate that Edit significantly outperforms strong SFT and RL baselines on both in-domain and out-of-domain splits. Furthermore, ablation studies confirm that our internal-state diagnostics are the primary source of these performance gains.

In conclusion, our contributions are three-fold:

• We introduce Edit, a novel two-phase training framework for rubric-faithful grading, that adapts credit assignment to externally grounded rubric grading rather than closed-system reasoning.

• We replace unreliable prompt-based self-audits with robust posterior and grounding signals (Internal State Diagnostics), and use rubric-checklist context to support local atomic revisions without imposing a rigid output template.

• We propose a calibration RL mechanism, called Belief-Guided Reward Shaping, that penalises severe mid-trajectory belief drifts while tolerating benign exploration, improving grading accuracy and out-of-domain generalisation.

2 Methodology↩︎

Edit(Evidence-Diagnosed Intervention Training) fine-tunes an LLM grader in two phases (Fig. 2). Phase 1 (Edit-SFT, §2.2) builds a supervised training set by revising small, local parts of incorrect reasoning steps. The atomic steps to revise are chosen using the model’s own internal belief signals and grounding signals. The revised content is guided by a rubric-checklist, which is extracted from the marking scheme and the student answer. Phase 2 (Edit-RL, §2.3) then calibrates the resulting policy with group-relative outcome-RL. The reward is shaped by the posterior belief signal used in Phase 1, giving a marking-task-aligned formulation that we call Belief-guided Reward Shaping.

2.1 Internal State Signals↩︎

Edit first extracts two step-level internal-state signals from the grading model (i.e., policy), both computed offline from existing rollouts. Consider a rollout be generated for the question \(q\), marking rules \(\rho\), and a student answer \(a\). The rollout contains \(T\) reasoning steps \(s_{1:T}\). For each step boundary, we compute two signals.

2.1.0.1 Posterior belief probe.

At step boundary \(k\!\in\!\{0,\dots,T\}\), we estimate the policy’s belief about the final integer mark, \(m\!\in\!\{0,\dots,M_q\}\), where \(M_q\) is the maximum available mark for question \(q\). To do this, we append the final marking scaffold to the partial reasoning chain and read the next-token logits over the valid mark tokens: \[\label{eq:probe} \hat{p}_k(m) \propto \exp(\mathrm{logit}_\theta(m | q, \rho, a, s_{1:k}, \textit{scaff.}))\tag{1}\] From \(\hat{p}_k\), we derive the expected mark \(\hat{m}_k = \mathbb{E}_{m \sim \hat{p}_k}[m]\) and the expected absolute error \(\epsilon_k = \mathbb{E}_{m \sim \hat{p}_k}\bigl[|\hat{m} - m^*|\bigr ]\), where \(m^*\) is the gold mark. We then define the signed per-step change as: \[\label{eq:eegain} \Delta \epsilon_k = \epsilon_{k-1} - \epsilon_k\tag{2}\] A positive value means that step \(s_k\) moved the policy’s belief closer to the gold mark, while a negative value means that the step moved the belief farther away. Thus, the sign shows whether the reasoning step locally helped or hurt the model’s marking belief.

2.1.0.2 Mask-based grounding audit.

For each reasoning step \(s_k\), we compute mask-based support scores that measure how much the likelihood of \(s_k\) depends on three information sources: the marking rule \(\rho\), the student answer \(a\), and the reasoning prefix \(s_{<k} = (s_1,\ldots,s_{k-1})\). Let \(\ell_k(q,\rho,a,s_{<k}) = -\log p_\theta(s_k \mid q,\rho,a,s_{<k})\) denote the negative log-likelihood of step \(s_k\) under the full context. We write \[\ell_k^{\mathrm{full}} = \ell_k(q,\rho,a,s_{<k}).\] The support scores are then defined as \[\begin{align} G_k^{\rho} &= \ell_k(q,\texttt{[MASK]},a,s_{<k}) - \ell_k^{\mathrm{full}}, \notag\\ G_k^{a} &= \ell_k(q,\rho,\texttt{[MASK]},s_{<k}) - \ell_k^{\mathrm{full}}, \notag\\ G_k^{p} &= \ell_k(q,\rho,a,\varnothing) - \ell_k^{\mathrm{full}}. \label{eq:support} \end{align}\tag{3}\]

Larger values indicate that \(s_k\) is more strongly supported by the corresponding masked input source. For example, if masking the student answer makes the step much harder to predict, then the step is likely grounded in the student answer. Because \(G_k^p\) naturally increases for later steps (later steps tend to depend more on the earlier reasoning prefix), we use a position-residualised \(\widetilde{G}_k^{p}\) as the reasoning-prefix-support score. Let \(u_{i,k}=k/T_i\) denote the relative position of step \(k\) in example \(i\), and let \(\mu_p(u)=\mathbb{E}_{(i,j)\sim\mathcal{D}} \!\left[ G_{i,j}^{p} \,\middle|\, u_{i,j}=u \right]\) be the dataset-level mean prefix-support score at relative position \(u\). We then define \[\label{eq:pg95resid} \widetilde{G}_{i,k}^{p} = G_{i,k}^{p} - \mu_p(u_{i,k}).\tag{4}\]

This audit is an input-ablation probe. We replace one input section with a placeholder and measure how much harder it becomes to predict the already-generated step. We validate this audit using external per-step oracle annotations from a stronger annotator model, as reported in appendix 9. In short, the grounding-audit scores correlate with the oracle labels in the expected direction.

2.2 Phase 1: Edit-SFT↩︎

Edit-SFT builds a supervised training pool by revising small parts of incorrect rollouts. Each rollout passes through four sub-stages: rubric-checklist generation, candidate step selection using internal signals, atomic revision, and outcome filtering. Successful revisions are then mixed with naturally correct rollouts and used as SFT targets.

2.2.0.1 Rubric-checklist.

For each sample \(x_i = \{q_i, \rho_i, a_i\}\), we generate a rubric-checklist by prompting the policy with the gold mark \(m^*_i\) as previliged information (available only during training-pool construction, not at inference). The policy is asked to list every marking point verbatim and to judge whether each point is covered, supported by a quoted evidence span. This checklist is later used as context during intervention.

2.2.0.2 Candidate steps selection.

In this phase, we locate steps in an incorrect rollout that should be edited. For each incorrect rollout (\(m \neq m^*\)), we first extract the internal-state signals at each step boundary described in §2.1. We take the top-\(k\) steps ranked by the most-negative \(\Delta_{\epsilon_k}\), while filtering out those with a positive or near-tie value. This selects steps that most strongly moved the model’s belief away from the gold mark. For these candidates, we compute \(G_k^{\rho}, G_k^{a}\), and \(\widetilde{G_k^{p}}\). We then identify weakly grounded steps by checking whether all grounding-audit values fall below the dataset-wide 25th-percentile threshold. These steps are promoted in our candidate pool. Implementation details are given in Appendix 7.

2.2.0.3 Atomic revision under locality constraint.

For each selected candidate step \(s_k\), we ask the policy to revise it into 1–3 new sub-steps that fix only the local reasoning error. The prompt includes the sample context \(\{q, \rho, a\}\), the full reasoning attempt with \(s_k\) highlighted, the internal-signal diagnostic, and the previously generated rubric checklist.

The revision must satisfy a locality constraint, ensuring that the intervention does not destroy the reasoning structure or coherence of the original rollouts. The constraint also prevents the revised step from simply copying dense information or oracle knowledge from the gold mark or rubric checklist. The details for the prompt and checklist-mimicry detector are shown in Appendix 11.

2.2.0.4 Pool composition and training.

For each revision, we sample \(N\) continuations and retain the revision if any continuation reaches the correct mark, \(m = m^*\). The surviving rollouts are used as SFT samples. To preserve the policy’s performance on easy or already-correct cases, we also mix the originally successful rollouts in the training pool. The policy is then trained on this combined dataset.

2.3 Phase 2: Edit-RL↩︎

Initialised from the Edit-SFT policy, Edit-RL applies distance-aware GRPO [5] augmented by a belief-guided reward shaping term. By re-using the same posterior probe, this shaping term calibrates marking accuracy while penalising harmful drifts in intermediate beliefs.

2.3.0.1 Distance-aware outcome reward.

We use a dense reward based on mark distance: \[r_\text{out} = \max\Bigl(0,\, 1 - \tfrac{|m - m^*|}{M_q}\Bigr)\] with \(r_\text{out}=0\) on parse failure. This reward gives higher scores to rollout whose predicted mark \(m\) is closer to the gold mark \(m^*\), providing distance-sensitive signal for RL calibration.

2.3.0.2 Belief-guided reward shaping.

After Edit-SFT, the model has learned to avoid reasoning steps that cause large harmful drifts in its posterior marking belief. In Edit-RL, we strengthen this behavior by adding a per-rollout shaping signal that uses the posterior probe to check whether the model’s belief drifted too much from the gold mark at any intermediate step: \[r_\text{shape} = - \frac{1}{M_q}\frac{1}{T} \sum_{k=1}^{T}\max\bigl(0, |\bar{m}_k - m^*| - \beta_\text{drift}\bigr), \label{eq:prs}\tag{5}\] where \(\bar{m}_k\) is the expected posterior mark at step boundary \(k\) (Eq. 1 ). The threshold \(\beta_\text{drift}\) is an exploration tolerance, measured in raw marks, which accepts small belief fluctuations that may reflect useful exploration. However, when the belief moves beyond this tolerance, the rollout is penalised in proportion to the size of excess drift. The total reward is: \[r = r_\text{out} + \gamma_\Phi \cdot r_\text{shape}\] This reward is used directly in the distance-aware GRPO objective. As a result, a rollout can be penalised relative to others in the same group if it contains a large belief deviation, even when its final mark is close to \(m^*\).

The posterior belief probe is run on the frozen Edit-SFT checkpoint rather than the active policy. This maintains a stationary shaping signal, reduces computational overhead, and prevents reward hacking. We validate this choice by proving a small KL divergence between the trained policy with the initialised reference in Appendix 8. Although this relaxes the strict policy-invariance of classic PBRS [9], it provides a stable, task-aligned constraint that significantly enhances out-of-distribution generalisation.

3 Experiment↩︎

3.1 Experiment Setup↩︎

3.1.0.1 Datasets

We evaluate policy grading on two complementary benchmarks, covering five datasets in total. The first benchmark is SAS, drawn from SAS-Bench [3], a Chinese short-answer scoring benchmark. We use its three diverse subjects (History, Geography, and Physics). The second benchmark is Private-Science³, a proprietary collection of student responses to GCSE-level science exam questions in two subjects (Biology and Physics), each marked by trained examiners against an official mark scheme. The two benchmarks are deliberately complementary. SAS comprises many questions with only a handful of responses each, while Private-Science contains fewer questions, but each is answered by hundreds of students. Full dataset statistics are given in Appendix 6.

3.1.0.2 Evaluation protocol

For Private-Science, we use three splits: a train set, an in-distribution (ID) test set of held-out responses to questions that appear in training, and an out-of-distribution (OOD) test set of responses to questions that are never seen in training. Separating ID from OOD tests whether a grader generalises to unseen questions rather than fitting question-specific surface patterns. SAS has no ID split. Each SAS question receives only two to five responses, so we evaluate the SAS only on the OOD questions, treating the SAS dataset as a subject-level fitting question.

Following standard practice in automated scoring [2], we use quadratic weighted kappa (QWK) as the main metric, macro-averaged over questions. We also report exact-match accuracy, within-1 accuracy, and mean absolute error (MAE).

3.1.0.3 Baselines

All methods use Qwen3-8B as the policy model, adapted with LoRA. Training details are reported in Appendix 7. We keep the backbone fixed across methods so that performance differences mainly reflect the training objective rather than model capacity. We compare our method after each phase (Edit-SFT and Edit-RL) against four baselines:
(1) Base, the off-the-shelf Qwen3-8B grader.
(2) GRPO [5], RL on standard group-relative distance-aware outcome-reward.
(3) DGPO [7], RL with distribution-guided token-level advantage reallocation.
(4) InT [8], using SFT and RL stages based on self-proposed interventions.

Edit and InT contain an SFT stage followed by an RL stage, whereas GRPO and DGPO apply RL directly to the base model.

5pt

3.2 Grading Performance↩︎

5pt

We first discuss the public SAS benchmark, where every evaluation question is unseen during training. As shown in Table 1, Edit-RL achieves the best results on average across all metrics. In particular, it improves macro-QWK from \(0.727\) for Base and \(0.731\) for InT to \(0.794\). These results show that the method improves grading accuracy in a fully OOD setting.

The subject-level results show some variation. Edit-RL gives the best QWK on History and Physics. InT-RL remains strongest on History for several metrics, while Edit-SFT performs best on Geography, where all RL variants degrade the performance. However, Edit-RL remains the only RL calibrated model that strongly outperforms Base. This variation suggests that different subjects benefit from different forms of adaptation. Nevertheless, our full method provides the most consistent overall improvement across SAS.

We next evaluate Private-Science, where both ID and OOD splits are available. As shown in Table 2, Edit-SFT beats all baselines across metrics on Private-Biology, improving macro-QWK by \(+0.027\) on the ID-test split and \(+0.04\) on the OOD-pool over the strongest baseline (GRPO). InT-RL outperforms across multiple metrics in Private-Physics ID setting, while degrades drastically in the OOD, suggesting an overfitting on the trained types of questions. In contrast, Edit-SFT shows steady gains across both ID and OOD. This suggests that the improvement is not simply due to fitting question-specific patterns seen during training. We attribute the improvement from Edit-SFT to three design choices. First, internal signals help locate reasoning steps that are likely to need revision. Second, atomic revisions keep the intervention local and avoid disrupting the full reasoning chain. Third, the rubric-checklist context gives the revisor task-specific guidance without forcing it to copy the checklist directly. These designs together preserve the model’s original reasoning capabilities while making its marking reasoning chain more grounded and better aligned with the rubric.

Edit-RL further improves performance over Edit-SFT on both test pools. It improves macro-QWK by \(+0.016\) and \(+0.027\) on Biology’s ID and OOD, respectively. It also improves ACC, within-1, and MAE, giving the best results across all metrics. The larger OOD gain suggests that belief-guided reward shaping is especially useful when the model must grade unseen questions.

Table 3 further separates the effect of the SFT and the RL. Applying Edit-RL directly to Base does not improve OOD performance over standard GRPO. In contrast, applying RL after Edit-SFT gives stronger results, and the full combination Edit-SFT\(+\)Edit-RL is best on every metric. Figure 3 illustrates the QWK dynamics on the ID and OOD with RL proceeds. Although the improvement matches with a GRPO on Edit-SFT in ID, Edit-RL shows steady improvement on the OOD setting. This suggests that belief-guided reward shaping is most effective when the model has first been trained on internally located, rubric-aware atomic revisions.

4pt

3.3 Ablation for Rule-Aware Atomic Intervention↩︎

We ablate the main design choices of Edit-SFT on SAS-History, the largest dataset in the SAS suite we adopted. The ablation focuses on three components: internal-signal localisation, rubric-checklist context, and atomic locality constraints. These components test whether performance gains come from selecting better edit locations, giving the revisor better rule context, and keeping the revision local.

As shown in Table 4, removing any module degrades performance. Excluding the rubric-checklist context leads to the largest drop in QWK and MAE while leaving ACC largely unchanged, suggesting increased variance in grading behavior. Removing atomic locality constraints during revision results in a slight relative degradation. This is likely because SAS scoring is naturally point-based and therefore already imposes contextual constrains. Edit-SFT without internal-signal localisation exhibits the largest drop across all metrics, highlighting the critical role of our step-candidate selection for reivision. Figure 4 further shows that the base model tends to select earlier steps, whereas our localisation module selects middle or final steps, where reasoning synthesis typically occur.

5pt

3.4 Rule-Faithfulness under Interventions↩︎

We also test whether a grader truly follows the rubric, rather than relying on its internalised scoring scale. To do this, we apply deterministic edits to the mark scheme where the gold-score change can be computed exactly. We then measure the Rule-Sensitivity Ratio (RSR), defined as the ratio between the model’s score change and the gold score change. (\(\mathrm{RSR}{=}1\) means that the model follows the rule edit exactly; \({<}1\) under-reacts, \({>}1\) over-reacts). The edits include level-of-response band shifts (LoR) and points-total rescaling (PTS). The detailed intervention strategies are reported in Appendix 10.1. One complication is that a predicted score of \(0\) is unchanged by every edit, so we decompose RSR into two parts: 1. Floor Rate (FR), the fraction of predictions equal to zero, and 2. In-Rubric RSR (IB-RSR), responsiveness only among non-zero predictions.

Table 5 presents the results averaged across the intervention strategies for LoR and PTS, respectively. Base demonstrates strong responsiveness as a naturally pretrained model, reflecting its inherent ability to follow instructions consistently. Edit-SFT achieves the strongest average responsiveness across all interventions, with IB-RSR biases of only \(-0.005\) and \(-0.014\) from 1 compared to Base.

InT underperforms across both settings, with its SFT variant performing worst on PTS and its RL variant performing worst on LoR. Although all RL-based paradigms reduce faithfulness to some extent, Edit-RL still significantly outperforms all trained baselines, demonstrating effective rule-following control paired with Edit-SFT. Overall, our method not only strengthens rubric-following capability under strongly imposed rule interventions, but also preserves this capability under belief-control signals.

4pt

4 Related Work↩︎

4.0.0.1 Credit Assignment for Reinforcement Learning

Credit assignment refers to the problem of attributing a delayed outcome to specific intermediate decisions, which remains a core challenge in LLM reasoning. Standard sequence-level objectives like GRPO [5] broadcast a single scalar reward uniformly across a trajectory. To refine this coarse signal, recent work generally falls into three paradigms. Return redistribution methods, such as DGPO [7], reallocate trajectory-level returns to individual steps. Process reward models (PRMs), like Math-Shepherd [10], train explicit step-level verifiers to provide fine-grained supervision. Alternatively, hindsight interventions, such as InT [8], prompt the policy to self-audit and rewrite erroneous steps. However, a growing body of work shows that LLM-generated chains-of-thought frequently diverge from actual computation, often necessitating external rewards to enforce faithfulness [11], [12]. While prior methods rely on external verifiers or textual self-critique, our framework directly localises critical steps using the model’s well-calibrated internal signals and guides the policy with belief-guided reward shaping.

4.0.0.2 LLM-based Automated Student Assessment

Automated essay scoring (AES) has increasingly transitioned from training dedicated classifiers to utilizing pretrained LLMs as evaluators or distilling their rationales into smaller models [13], [14]. For short-answer and science grading, recent benchmarks and systems primarily rely on prompt engineering or verbal reflection of frozen models [2], [3]. Other approaches explore aligning evaluation rationales via preference optimization on thought trees to improve explainability [15]. While most existing systems treat grading as a zero-shot prompting task or rely on external solvers to follow rules, our work frames grading as an end-to-end trainable credit-assignment problem. By supplying the rubric checklist as a privileged input during the rewriting phase, our approach decouples analytical reasoning from rigid structural formats to ensure faithful adherence to external grading criteria.

5 Conclusion↩︎

In this paper, we propose Edit, a novel two-phase framework addressing the unique credit-assignment challenges of rule-faithful LLM grading. To enforce strict external grounding, Edit replaces unreliable prompt-based self-audits with internal-state localisation and privileged-context rewriting. Furthermore, our belief-guided reward shaping guides the uncertainty reduction process by penalising severe belief excursions while tolerating benign exploration. Experiments on two real-world grading datasets demonstrate that Edit significantly outperforms state-of-the-art baselines across in-domain and out-of-domain splits. Future work will explore generalizing the framework to broader domains, streamlining the training pipeline, and dynamically decomposing holistic rubrics.

Limitations↩︎

We identify three key limitations of the Edit framework.

First, our evaluation scenarios are currently restricted in scope. Although we frame student answer scoring as a complex rule-based reasoning task, we evaluate Edit on two datasets primarily consisting of short-answer and fill-in-the-blank questions. We have not yet tested our framework on long-form essay scoring tasks, such as those in the Automated Student Assessment Prize (ASAP) benchmark, nor have we explored its applicability in other high-stakes rule-based reasoning domains like legal judgment or medical diagnosis. An important direction for future work is to verify and extend the generalizability of our method across broader domains and longer, more unstructured text formats.

Second, the proposed training pipeline involves considerable complexity. While our probing-based internal-state localisation reduces inference overhead compared to prior prompt-elicited self-audits, the overall framework still relies on a heavy multi-stage process: rigorous SFT data construction followed by a calibration RL phase (GRPO). This multi-step orchestration increases both computational complexity and engineering effort. Future research should investigate more streamlined, end-to-end training paradigms that simplify the pipeline while preserving the precise credit-assignment capabilities of evidence-diagnosed interventions.

Third, Edit relies on the availability and granularity of explicit rule sets. Specifically, Edit-SFT utilises a per-criterion rubric checklist as privileged input to constrain atomic rewrites. However, in many real-world settings, rubrics can be vague, holistic, or implicitly defined by human graders. The effectiveness of our internal-state diagnostics and rule-faithful rewriting may degrade when applied to such unstructured or noisy criteria. Future work could explore mechanisms to dynamically decompose holistic rubrics into structured checklists, or adapt the framework to handle fuzzy and implicit rule constraints.

Ethics Statement↩︎

This research has been approved by our institutional ethics committee. Our dataset comprises LLM-augmented open-source question banks and proprietary data from a collaborative institution (which has also passed their internal rigorous ethical review). All data has been strictly de-identified to remove Personally Identifiable Information (PII). Given that the open-source subset relies on LLM generation, it may inherit sociopolitical or linguistic biases from pre-training corpora. We strongly recommend manual review before using this data for downstream training to avoid penalising diverse linguistic expressions or propagating cognitive distortions.

Furthermore, our framework is intended solely for researching LLM reasoning in rule-constrained environments. Automated grading involves high-stakes decisions with profound psychological impacts on students’ academic trajectories. We strictly advise against deploying these models in real-world educational settings without human-in-the-loop to ensure fairness, accountability, and pedagogical empathy.

Acknowledgements↩︎

This work was supported in part by the UK Engineering and Physical Sciences Research Council (EPSRC) through the Prosperity Partnership scheme (grant no. UKRI566) and a Turing AI Fellowship (grant no. EP/V020579/1, EP/V020579/2).

6 Dataset Statistics↩︎

Table 6 reports, for each dataset, the number of responses and distinct questions in every split, together with the range of per-question maximum marks. The three splits play different roles across the two benchmarks. For Private-Science, each question is answered by hundreds of students: the Test split holds out responses to questions that are seen during training (in-distribution, ID), whereas the Holdout split consists of entirely unseen questions (out-of-distribution, OOD). For SAS, each question has only a handful of responses, so questions are partitioned disjointly across all three splits; both the Test and Holdout splits are therefore out-of-distribution, and SAS has no ID split.

7 Implementation Details↩︎

7.0.0.1 Training

All methods adapt Qwen3-8B using a shared LoRA recipe: rank \(r{=}64\), \(\alpha{=}128\), applied to all attention and MLP projection matrices (\(q,k,v,o,\text{gate},\text{up},\text{down}\)). We train for \(2\) epochs at learning rate \(2\times10^{-4}\) (cosine schedule, \(10\%\) warmup). For the SAS dataset, we set the effective batch size to \(8\) due to the small amount of data, and to \(32\) on Privacy-Science. The same recipe is used for all subjects and methods, so comparisons are not confounded by hyperparameter differences.

7.0.0.2 Decoding and evaluation

At evaluation, we obtain \(1\) completion per marking using greedy sampling. This ensures consistency across all datasets and methods. QWK is computed per question over the integer score range \([0, M_q]\), where \(M_q\) is the question’s maximum mark, using quadratic weights, and then macro-averaged across questions.

8 Validity of the Frozen-Probe Reference↩︎

Edit-RL reads the per-step mark belief \(E_k\) from a frozen copy of the Edit-SFT policy and penalises its drift from gold. We freeze the reference for two reasons. (i) Non-gameability: RL could lower the shaping penalty by shifting its own belief readout rather than by grading more faithfully. A fixed, exogenous reference removes this shortcut. (ii) Calibration: the Edit-SFT is already a competent grader, so its step-boundary mark posterior is a meaningful target for “where belief should sit,” unlike the raw policy. This design is sound only if the trained policy’s belief remains well-described by the frozen readout, so we verify this directly.

We probe the mark posterior \(\hat{p}_k(m)\) (Eq. 1 ) at every step boundary of \(200\) Privacy-Biology test rollouts under the frozen Edit-SFT and under each SFT-initialised RL policy, and measure their divergence (Table 7). The Edit-RL policy stays very close to the frozen reference (mean \(\mathrm{KL}{=}0.026\), \(\mathrm{JSD}{=}0.0055\)) and is flat across chain position, confirming that the frozen probe remains a faithful proxy for the policy’s belief. Plain GRPO, the same SFT initialisation and KL-to-init term, but without belief shaping, drifts about twice as far (\(\mathrm{KL}{=}0.064\)), and increasingly so deeper in the chain. Belief shaping thus anchors the intermediate trajectory to the calibrated reference, not merely the final mark.

4.5pt

9 Masked-Support Audit Validation↩︎

We validate the masked-support audit against an external annotator’s step labels, which tag each grading step as a grounding step (a pointwise judgement read off the answer against the rubric) or a synthesis step (one that composes prior conclusions). 5 shows two clean dissociations. Grounding steps occupy the high Answer-Grounding region (panel a), whereas synthesis steps collapse to \(G^a\approx0\); conversely, synthesis steps carry markedly higher Prefix Grounding (panel b, mode at \(G^p\approx1.5\) vs.a near-zero mode for grounding steps). Both dissociations are highly significant (8): Answer Grounding separates the two classes with a large effect (\(r{=}0.71\), \(p<10^{-300}\)) and Prefix Grounding likewise (\(r{=}0.58\), \(p<10^{-300}\)), and both survive collapsing to per-rollout means. Rubric Grounding, in contrast, is statistically significant but practically negligible (\(r{=}0.08\)): both step types lean equally on the rubric, so the discriminating axes are which evidence a step reads (the answer, for grounding steps) and whether it integrates prior reasoning (the prefix, for synthesis steps). This confirms that \(G^a\) and \(G^p\) behave as intended and are well separated from zero.

4.5pt

10 Rule-Faithfulness Intervention↩︎

10.1 Intervention Strategies↩︎

We probe rule-faithfulness using six deterministic, gold-computable edits to the mark scheme, grouped into two families that correspond to the two Privacy-Biology scheme types. For each edit we recompute the reference (gold) mark under the edited scheme, regrade the same student response with the edited scheme supplied in-prompt, and compare the model’s score change to the gold change via the Rule-Sensitivity Ratio \(\mathrm{RSR}=\overline{|\Delta\mathrm{pred}|}/\overline{|\Delta\mathrm{gold}|}\) (§3.4).

10.1.0.1 Level-of-response (LoR) band shifts.

For LoR questions, the scheme maps each response-quality level to a mark band. We apply: A1 (uniform \(+2\)): adding a constant \(2\) marks to every level (\(\overline{|\Delta\mathrm{gold}|}{=}1.85\)); A2 (expand \(\times2\)): doubling each level’s mark, stretching inter-level spacing (\(\overline{|\Delta\mathrm{gold}|}{=}3.00\)); B1 (differential): a non-uniform per-level shift (e.g. \(+1/+2/+2\)) from the lowest to the highest band,

10.1.0.2 Points-total (PTS) rescales.

For PTS questions, the scheme awards a fixed mark per creditable point. We multiply every point’s value by a constant factor: \(\times0.5\) (\(\overline{|\Delta\mathrm{gold}|}{=}0.35\)), \(\times1.5\) (\(0.90\)), and \(\times2.0\) (\(1.25\)).

10.1.0.3 Floor confound.

A prediction of \(0\) is a fixed point of every edit (it cannot move), so a grader can appear faithful merely by predicting \(0\). We therefore decompose RSR into a floor rate (fraction of perturbed predictions equal to \(0\)) and an in-rubric RSR (RSR computed only over rows with a non-zero perturbed prediction), and report both.

10.2 Full Intervention Results↩︎

7pt

Table 9 gives the full breakdown. LoR edits: all graders are near-faithful. In-rubric RSR stays in \([0.78,1.14]\) and floor rates are low (\(6\)–\(12\%\)); the model tracks band shifts as intended. The Edit family is the most faithful trained family here (in-rubric RSR \(0.86\)–\(0.96\), closest to \(1\)) and floors the least (\(6.4\)–\(7.8\%\)), whereas Base mildly over-reacts (\(1.13\)–\(1.14\) on A2/B1). PTS rescales: unfaithful for everyone. Multiplicative point rescaling collapses in-rubric RSR to \(0.1\)–\(0.35\) on the up-rescales (\(\times1.5,\times2.0\)) and inflates floor rates to \(21\)–\(32\%\), evidence of absolute-count anchoring that no method removes. InT floors hardest (\(26\)–\(32\%\)); Edit-SFT retains the highest PTS in-rubric RSR among the trained arms (\(0.25\)–\(0.35\) on the up-rescales). The decomposition also exposes a confound in the headline RSR: InT’s competitive aggregate RSR on PTS is partly a flooring artefact (its high floor rate removes responsive rows), which the floor-rate / in-rubric split makes explicit.

11 Prompt↩︎

We document the core prompt behind Edit’s Phase-A3 Atomic rewriting under locality constraint (shown in Figure 6). For space we show only its structural skeleton: the section headers, the per-instance privileged inputs (rendered as {\(\cdot\)} placeholders), and the output contract; the verbose per-section guidance is elided ([…]).

None

References↩︎

---

1.  Equal contribution.↩︎

2.  Corresponding author.↩︎

3. Dataset name and provider details are anonymised during review. The dataset cannot be publicly released because of licensing and privacy constraints.↩︎