¹

1 Introduction↩︎

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse terminal rewards, typically final-answer correctness [1]–[5]. This terminal anchor reliably certifies success but provides no direct feedback on the reasoning path, leading to two failure modes usually treated separately: training stalls on hard problems where outcome-only rollouts rarely succeed [6]–[8], and models produce correct but inefficient reasoning on problems they can already solve [9]–[11]. Before success, prerequisite progress is invisible unless it reaches the final answer, so near-misses provide little signal; after success, endpoint rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. Two diagnostic probes provide evidence for this interpretation (Figure 1; details in Appendix 10): scaffolded prompts reveal prerequisite signal that endpoint rewards leave unexposed, and among final-answer-correct rollouts outcome-only GRPO increases response length while retaining low-value content.

This sparse-anchor view reframes prior work as addressing different consequences of the same anchor: exploration-oriented methods reshape inputs, rollouts, or updates on hard problems [6], [7], [12]–[20], efficiency methods reduce overthinking [10], [11], [21], and process-supervision methods train PRMs, derive implicit rewards, or reassign dense feedback [22]–[27]. Less explored is how the rule-verifiable signal itself can be densified through independently verified sub-answer targets before success and correctness-gated process composition after success.

We instantiate this view as SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage RLVR framework that densifies the sparse anchor while retaining the GRPO update. Adaptive Scaffolded RL (ASR) targets the before-success phase: for problems where outcome-only rollouts yield low success rates, it converts answer-hidden scaffolded sub-question chains into prefix-decomposed verifiable rewards, crediting partial progress without leaking answers. Quality-Aware Process RL (QPR) targets the after-success phase: on trajectories that reach the correct answer, it applies correctness-gated process-shape rewards that prefer useful, well-organized reasoning over redundant or locally flawed reasoning. The stages are naturally sequential—ASR first raises the supply of verified trajectories, after which QPR refines their process quality—and by design neither allows answer leakage nor brevity to substitute for verified correctness. To evaluate reasoning processes beyond final-answer accuracy, we introduce a Step-Quality Evaluation Protocol that preserves rule-based answer verification while adding post-hoc LLM-judged diagnostics of useful-step density, low-value step ratios, error localization, and token efficiency.

Across mathematical and scientific benchmarks and two training corpora (DAPO-Math [3] and Big-Math [28] ), SCOPE-RL improves accuracy, useful-step density, and token efficiency over outcome-only GRPO on Qwen3-8B-Instruct [29] . Average accuracy rises from 55.80% to 66.35% on DAPO-Math and from 53.58% to 64.79% on Big-Math, with 16.2% and 27.1% token reductions, respectively. The gains hold under a GSPO backend (61.60%\(\to\)​66.93%) and on Qwen3-0.6B-Instruct (26.06%\(\to\)​32.06%; both verified on DAPO-Math), indicating that reward-signal densification is complementary to policy-update-level RLVR advances.

1.0.0.1 Contributions.

(i) A unifying sparse-anchor view of RLVR: outcome-only verification leaves the reasoning path under-specified before and after success, calling for phase-specific densification around the same verified endpoint. (ii) SCOPE-RL, instantiating this view with prefix-decomposed verifiable rewards (ASR) and correctness-gated process-shape rewards (QPR) in a naturally sequential procedure. (iii) An expert-validated Step-Quality Evaluation Protocol, with consistent gains across two training sources and multiple benchmarks.

2 Preliminary↩︎

Given a prompt-answer pair \((q,a)\), outcome-only RLVR samples \(y \sim \pi_\theta(\cdot \mid q)\) and assigns the rule-verified reward \[r_{\mathrm{out}}(y,a)=\mathbf{1}[\hat{a}(y)=a].\] More generally, GRPO admits any scalar reward \(R(q,y)\) in place of \(r_{\mathrm{out}}\). For \(G\) sampled responses \(\{y_g\}_{g=1}^{G}\), we use \[\label{eq:grpo95advantage} A_g = \frac{R(q,y_g)-\mathrm{mean}_{j} R(q,y_j)}{\mathrm{std}_{j} R(q,y_j)}.\tag{1}\] When the within-group reward is constant (\(\sigma^2_R(q)=0\)), Eq. 1 is undefined; following standard GRPO implementations [1], we set \(A_g\equiv 0\), so the group produces no reward-driven policy-gradient signal.

To quantify where GRPO receives usable reward variation, define \[\begin{align} \sigma^2_R(q) &= \mathrm{Var}_g\, R(q,y_g), \tag{2}\\ \eta_R &= \mathbb{P}_{q\sim\mathcal{D}}\!\big[\sigma^2_R(q) > 0\big]. \tag{3} \end{align}\] \(\eta_R\) measures the fraction of prompts that can produce non-zero group-relative advantages under reward \(R\); for \(r_{\mathrm{out}}\!\in\!\{0,1\}\), \(\sigma^2_{r_{\mathrm{out}}}(q)=0\) exactly when the group is uniformly correct or uniformly wrong.

3 The SCOPE-RL Framework↩︎

SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency) instantiates a single design principle on GRPO: the policy update is left unchanged, and the sparse anchor is densified only by modifying what enters the scalar reward on each side of the success boundary.

Figure 2 illustrates the complete pipeline. Both stages follow the same reward-densification template—where new signal is placed, when it is triggered, and how it is composed. Stage 1, Adaptive Scaffolded RL (ASR), places verifiable signal at scaffolded sub-endpoints, triggers it through on-policy routing, and composes prefix-consistent rewards for weak-signal problems. Stage 2, Quality-Aware Process RL (QPR), places signal on already-correct trajectories, triggers it through a correctness gate, and composes bounded process-shape rewards.

3.1 Two-Stage Coupling↩︎

ASR and QPR are coupled by a sequential dependency. Because QPR’s correctness gate (§3.3) returns zero on incorrect rollouts, it provides little signal on hard problems before the policy can reliably reach verified answers. ASR first lifts success rates through prefix-decomposed verifiable rewards, after which QPR has a denser set of correct trajectories to shape. Jointly mixing the two rewards would weaken this gate by applying QPR penalties before enough correct rollouts exist to provide useful contrast among them. We therefore train sequentially: Stage 1 produces a checkpoint \(\theta_1\) from the routed mixed reward \(R_1\), and Stage 2 initializes from \(\theta_1\) and optimizes \(R_{\mathrm{QPR}}\) on the original undecomposed problems. The only state passed across stages is the policy parameters.

3.2 Stage 1: Adaptive Scaffolded RL↩︎

Stage 1 addresses the before-success regime, where few rollouts reach the final answer and endpoint rewards provide little information about prerequisite progress. Since scaffold-prefix correctness is predictive of main-answer success (Figure 1 (a)), ASR treats ordered scaffold prefixes as verifiable proxies for prerequisite progress, increasing supervision density through answer-hidden scaffold construction, on-policy scaffold routing, and prefix-consistent scaffold rewards (Figure 2, Stage 1).

3.2.1 Answer-Hidden Scaffold Construction↩︎

Scaffold construction turns hidden prerequisite progress into verifiable sub-endpoints, each checkable by the same rule-based verifier as the final answer. Construction is performed offline, with answer-hidden non-leakage enforced as a validity predicate (Eq. 4 below). The complete decomposition prompt template is provided in Appendix 13.

Before RL training, ASR generates candidate scaffolded sub-question chains offline for each training problem. Given a problem \(q\) with answer \(a\), a valid scaffold has the form \[\mathcal{S}(q) = \bigl\{(q_1, a_1),\, \dots,\, (q_m, a_m),\, (q_{\mathrm{main}}, a)\bigr\},\] where \(q_{\mathrm{main}}\) is the final-target sub-problem whose verifiable answer is the original problem’s answer \(a\), and each \(q_i\) targets a prerequisite concept needed for the final solution. During RL training, the policy observes only the ordered sub-questions and the main problem \(q_{\mathrm{main}}\); the sub-answers \(a_i\) are verifier-only and used solely for reward computation.

We cache a scaffold only when it satisfies \[\label{eq:scaffold95valid} \mathrm{Valid}(\mathcal{S}) = V(\mathcal{S}) \land P(\mathcal{S}) \land D(\mathcal{S}) \land H(\mathcal{S}),\tag{4}\] where \(V\) denotes automatic verifiability, \(P\) prerequisite relevance, \(D\) dependency awareness, and \(H\) answer-hidden non-leakage. These predicates filter out scaffolds that are unverifiable, irrelevant, dependency-inconsistent, or answer-leaking, preserving the verifiable-answer invariant of RLVR (the policy is rewarded only for answers it produces itself, not for answers that appear in the prompt) by construction. A human audit of 200 random scaffolds confirms strict prerequisite-chain integrity (Appendix 21); invalid decompositions are discarded before RL training.

3.2.2 On-Policy Scaffold Routing↩︎

Applying scaffolds to every problem would dilute supervision and waste rollout cost on already-solvable problems. ASR routes groups by on-policy outcome reward, a self-adapting alternative to static difficulty labels or a learnable router.

During RL, each problem is first rolled out on the original prompt and scored with the outcome reward. The group mean outcome reward \[\label{eq:group95mean95reward} \bar{r}(q)=\frac{1}{G}\sum_{g=1}^{G} r_{\mathrm{out}}(y_g,a)\tag{5}\] serves as an on-policy estimate of whether endpoint-only training provides sufficient positive signal for \(q\). A problem is routed to the scaffolded path if \[\bar{r}(q) < \tau,\] where \(\tau\) is a fixed routing threshold; otherwise, the original rollouts and outcome rewards are retained. We use \(G=8\) and \(\tau=0.5\) in all experiments, with sensitivity analysis in Appendix 18.

3.2.3 Prefix-Consistent Scaffold Rewards↩︎

Once routing has selected a scaffold, the question is how to convert sub-answer matches into reward. We use a prefix product so that a sub-question contributes reward only when itself and all earlier sub-questions in its dependency chain are correct—this stays inside GRPO while preventing the policy from harvesting credit on later sub-answers it can guess without solving the prerequisites.

For routed problems, the scaffolded rollout asks the model to answer all sub-questions and then the original problem in a single generation, using explicit answer tags for extraction. Let \(\hat{a}_i\) be the model’s answer to \(q_i\), and define the prefix-consistency indicator \[\Pi_i = \prod_{j=1}^{i} \mathbf{1}[\hat{a}_j = a_j].\] The scaffold reward is \[\label{eq:stage195reward} R_{\mathrm{ASR}} = \beta \sum_{i=1}^{m} \frac{1}{m} \Pi_i + (1-\beta) \Pi_m \mathbf{1}[\hat{a}_{\mathrm{main}} = a],\tag{6}\] where \(\beta\) controls the weight assigned to scaffold checkpoints; we set \(\beta=0.5\) in our experiments. Since \(\Pi_m=\prod_{j=1}^{m}\mathbf{1}[\hat{a}_j=a_j]\) requires all \(m\) sub-answers to be correct, valid prerequisite prefixes receive partial credit and the main-answer match contributes only when the entire scaffold prefix is satisfied.

Stage 1 therefore uses the mixed scalar reward \[\label{eq:stage195mixed95reward} R_1(q,y)= \begin{cases} R_{\mathrm{ASR}}(y,\mathcal{S}(q)), & \bar{r}(q)<\tau, \\ r_{\mathrm{out}}(y,a), & \bar{r}(q)\ge\tau. \end{cases}\tag{7}\] Here, \(y\) denotes the rollout used in the selected branch: scaffolded rollouts for routed groups and original-prompt rollouts otherwise.

3.2.3.1 Optimizing ASR with GRPO.

Routed groups are re-rolled out on cached scaffolded prompts and scored with \(R_{\mathrm{ASR}}\); non-routed groups keep their original rollouts and outcome rewards.

3.2.3.2 ASR enlarges the effective gradient support.

The mixed reward \(R_1\) can introduce effective gradient support in routed groups that are degenerate under \(r_{\mathrm{out}}\) (Eq. 3 ). In particular, routed scaffold rewards produce non-zero advantages whenever an outcome-uniform group contains prefix-level disagreement, consistent with the gap between GRPO and SCOPE-RL Stage 1 in Figure [fig:dyn95effective].

3.3 Stage 2: Quality-Aware Process RL↩︎

Stage 2 addresses the after-success regime, where final-answer rewards cannot distinguish concise, useful, and well-organized correct trajectories from redundant or locally flawed ones. We use process shape to denote the compositional structure of an outcome-correct trajectory, including useful-step density, low-value-step ratios, local flaws, and soft conciseness. QPR returns to the original undecomposed problems and applies these preferences only after correctness is established (Figure 2, Stage 2).

3.3.1 Correctness-Gated Process-Shape Reward↩︎

Any process-shape signal that fires on incorrect rollouts could encourage short or fluent wrong answers, reproducing the failure mode of naive length penalties (Figure 1 (b)). QPR prevents this with a multiplicative correctness gate: the rule-based verifier decides final-answer correctness, and the LLM Judge only annotates how a verified-correct trajectory was reached. QPR is initialized from \(\theta_1\) and trains on the original, undecomposed problems. Concretely, for each rollout \(y\), \[\label{eq:qpr95gate} R_{\mathrm{QPR}}(y, a) = \begin{cases} 0, & \hat{a}(y) \neq a, \\ Q_{\mathrm{process}}(y), & \hat{a}(y) = a. \end{cases}\tag{8}\] The judge model, prompt, and reward formula are fixed throughout QPR training.

3.3.2 Step-Level Category Annotation↩︎

Conditioned on the gate in Eq. 8 , QPR uses a fixed LLM Judge to annotate step-level role among correct trajectories. Unlike a learned PRM, the Judge is not trained against the policy and never decides final-answer correctness, so its annotations cannot override the verifier.

For each correct-answer rollout, the judge parses the reasoning trajectory into \(N\ge 1\) atomic steps \(\{s_1, \dots, s_N\}\). Each step receives a category label \(c_i\) from five mutually exclusive classes—useful, mechanical, redundant, reversion, and error. Only useful contributes positive reward, while the remaining four labels carry penalties of different severities. Label definitions, the judging prompt, and parsing details are provided in Appendix 16.

From these annotations, we compute the useful-step ratio \(S_u\): \[S_u = \frac{1}{N}\sum_i \mathbf{1}[c_i = u],\] where \(u\) denotes the useful label. \(S_u\) rewards trajectories with a high density of substantive reasoning steps.

3.3.3 Quality-Aware Reward Composition↩︎

The annotations from the previous section give us a positive useful-step signal and four kinds of low-value step. We compose them with bounded multiplication, which avoids the failure modes of additive composition (length canceling quality) and pure-product composition (a single category zeroing the reward). Concretely, for each undesirable category \(k \in \mathcal{K}=\{\mathrm{mec},\mathrm{red},\mathrm{rev},\mathrm{err}\}\), let \(r_k\) denote the fraction of steps assigned label \(k\), and define \[\begin{align} \Phi &= \prod_{k \in \mathcal{K}} (1 - \lambda_k r_k), \label{eq:qpr95penalty} \\ \kappa &= \frac{1}{1 + \alpha \ln(N+1)}. \end{align}\tag{9}\] The overall process-shape reward is \[\label{eq:qpr95process} Q_{\mathrm{process}} = S_u \cdot \Phi \cdot \kappa,\tag{10}\] where \(S_u\) rewards useful reasoning density, \(\Phi\) suppresses low-value or locally flawed steps, and \(\kappa\) softly encourages conciseness with a logarithmic shape that avoids over-penalizing already-short solutions while still discouraging unbounded length. We set all \(\lambda_k<1\), so each penalty factor lies in \((0,1]\) and the product \(\Phi\) is strictly positive. Since \(\kappa\) multiplies the positive term, conciseness alone cannot raise the reward; shortening helps only when \(S_u\) is preserved. Hyperparameter values are reported in Appendix 16.

3.3.3.1 Optimizing QPR with GRPO.

Stage 2 uses \(R_{\mathrm{QPR}}\) as the scalar reward in the same GRPO backend. Because QPR keeps all training prompts in their original form, the model must internalize the process-shape preference rather than rely on explicit sub-question structure at inference time.

The complete two-stage training procedure is summarized in Algorithm 5 (Appendix 8).

4 Step-Quality Evaluation Protocol↩︎

We propose a general Step-Quality Evaluation Protocol for diagnosing reasoning processes beyond final-answer accuracy. Accuracy alone cannot reveal whether a training method changes the reasoning trajectories that lead to an answer. The protocol preserves rule-based answer verification while adding post-hoc LLM-judged step annotations, and reports decomposed measures of useful reasoning, low-value steps, error localization, and token efficiency. These metrics are diagnostic-only and never feed back into training. The protocol shares QPR’s five-class step taxonomy (§3.3) but uses independent evaluation judges and prompts (Appendix 15). We instantiate it on standard scientific and mathematical reasoning benchmarks; a scaling validation across five model sizes confirms the protocol’s discriminative power (Appendix 12).

4pt

5 Experiments↩︎

We evaluate SCOPE-RL in a math/science-oriented RLVR setting with Qwen-family backbones, testing whether its stage-specific supervision—ASR before success and QPR after success—improves learnability, trace quality, and efficiency, and whether these gains hold across two training sources, four evaluation domains, and independent expert preferences.

5.0.0.1 Setup.

We use Qwen3-8B-Instruct [29] as the base model and outcome-only GRPO [2] as the primary baseline, trained for the same total optimizer budget as the two SCOPE-RL stages combined. SCOPE-RL is trained sequentially on 2,400 problems from each of DAPO-Math [3] and Big-Math [28]: Stage 1 applies ASR; its best checkpoint initializes Stage 2 QPR. We evaluate GPQA [30] and MATH500 [22] with single-sample decoding, and report 8-sample average accuracy on AIME 2024 [31] and AIME 2025 [32]. Full training, fairness, and evaluation-protocol details are in Appendix 17.

5.1 Main Results and Reasoning Efficiency↩︎

Across both training sources, ASR raises final-answer accuracy over outcome-only GRPO; QPR then improves correct-trace quality and efficiency without sacrificing correctness (Table 1).

5.1.0.1 Accuracy.

ASR raises average accuracy on both sources (DAPO-Math: 55.80\(\to\)​65.86%; Big-Math: 53.58\(\to\)​62.36%), with the largest gains on AIME24/25 where endpoint rewards are sparsest. QPR adds further headroom (66.35% / 64.79%) and improves process diagnostics: Useful% rises monotonically across the two stages, and 1st-Err-Pos shifts later. Pass@128 analysis (Appendix 9) rules out a capacity ceiling from ASR scaffolding; QPR largely preserves this ceiling, with a minor coverage–quality trade on one AIME subset.

5.1.0.2 Efficiency.

Total token reduction is 16.2% (DAPO-Math) and 27.1% (Big-Math) without sacrificing accuracy or useful-step density. The split is source-dependent: on DAPO-Math, Stage 1 preserves length (6,936\(\to\)​6,948) and Stage 2 carries the full reduction; on Big-Math, Stage 1 already cuts 24.1% (8,228\(\to\)​6,245) and Stage 2 refines a further 4.0%. QPR thus shifts from primary compressor to fine-grained refiner with Stage 1’s residual redundancy.

5.2 Ablation Studies↩︎

Our ablations test whether the gains depend on prefix-decomposed verifiable rewards, correctness-gated process-shape rewards, and sequential training rather than auxiliary prompting, static routing, or simple brevity pressure.

5.2.0.1 ASR needs both scaffolded rewards and adaptive routing.

Removing scaffolded decomposition (No Decomp.), applying it indiscriminately (All Decomp.), or replacing on-policy routing with offline difficulty labels each costs 7–9 pp of average accuracy (Table 2); both prefix-decomposed rewards and capability-tracking routing matter. A reward-signal ablation (Appendix 19) isolates the prefix-consistency gate as the largest contributor.

2pt

4pt

5.2.0.2 Step-level decomposition, all three terms, and sequential staging are each load-bearing.

Table 3 reports three coupled ablations on the shared ASR backbone. Panel (a): step-level decomposition is the load-bearing choice—Holistic Judge, which reuses the same correctness gate and LLM judge but collapses per-step labels into a single trace score, nearly matches SCOPE-RL on Useful% (64.14%) yet trails 2.37 pp on Avg and uses 25% more tokens, while Concise Prompt sacrifices useful-step density (58.69%) and Overlong Penalty weakens accuracy without reliably controlling length. Panel (b): the three multiplicative terms in \(Q_{\mathrm{process}}\) (Eq. 10 ) target disjoint failure modes—removing the positive term collapses Avg (58.81%), removing the penalty leaves redundancy (Useful% 61.00%) despite the latest 1st-Err position (0.751), and removing the length factor preserves accuracy but inflates tokens to 7,882. Panel (c): Mix-Stage reaches only 60.98% Avg, confirming that prefix-decomposed verifiable rewards must precede process-shape refinement. Although individual ablations lead on isolated metrics (No Penalty on 1st-Err, Mix-Stage on Useful%), only SCOPE-RL jointly maximizes accuracy and overall trace quality.

5.3 Training Dynamics↩︎

Figure 3 corroborates the two-stage mechanism: ASR sustains a higher effective gradient ratio—the fraction of groups with non-degenerate advantages—by routing low-pass-rate problems to scaffolded prompts; response length grows under ASR then drops once QPR engages, while validation reward rises, indicating that QPR removes low-value reasoning rather than necessary steps.

5.4 Qualitative Analysis↩︎

Figure 4 decomposes correct traces into the five-category taxonomy. The progression matches Table 1: GRPO inflates length without raising useful-step density, ASR shifts steps toward useful reasoning, and QPR drops length while pushing Useful% to its peak.

A side-by-side trace comparison on the same Lagrange interpolation problem (Appendix 25) makes the two-stage mechanism concrete: GRPO reaches an incorrect intermediate conclusion and recovers only via lengthy self-correction, ASR identifies the correct mathematical structure earlier though some redundant verification remains, and QPR distills the trace into a direct partition-of-unity derivation.

5.5 Robustness Across Optimizer Backend and Model Scale↩︎

SCOPE-RL’s gains hold under a GSPO backend (61.60\(\to\)​66.93%, 13.1% token reduction) and on Qwen3-0.6B-Instruct (26.06\(\to\)​32.06%, 22.9% token reduction), indicating that the before/after-success decomposition is not tied to a single optimizer or model scale. Full results are in Appendix 24.

5.6 Additional Process-Quality Validation↩︎

We further validate that QPR improves perceived trace quality rather than only automatic diagnostic scores. In a pairwise evaluation on 200 examples where both GRPO and SCOPE-RL are answer-correct, three domain experts prefer SCOPE-RL across all five dimensions: conciseness (72% vs.%), non-redundancy (69% vs.%), clarity (65.5% vs.%), logical coherence (59.5% vs.%), and overall quality (70.5% vs.%). Full results are reported in Appendix 23.

6 Related Work↩︎

RLVR and hard-problem learning. Recent RLVR systems and GRPO-style optimizers [1]–[5] improve LLM reasoning under verifiable final-answer rewards but face sparse signal on hard problems, motivating curricula [6], [7], [12], difficulty filtering [13], demonstrations [20], hint or prefix injection [16]–[19], reformulation or decomposition [14], [15], and staged pipelines [8]. These methods reshape inputs, rollouts, or GRPO updates [18], [20], but none install independently verifiable sub-answer rewards. ASR instead places rule-based verifiers on each scaffolded sub-answer, hidden from the policy. EvoCoT [19] conditions rollouts on self-generated, verified CoT prefixes progressively shortened as a curriculum, not on independently verified sub-answer targets.

Process supervision and reasoning efficiency. Step-level supervision spans PRMs trained on step labels [22], [23], implicit process rewards from policy logits [24], refined aggregation or credit assignment of PRM rewards [25], [27], and analytical decomposition of the outcome objective [26]; PRIME [33] supports verifier evaluation and selection rather than step labels, and efficiency methods reduce overthinking via length-aware or compression objectives [10], [11], [21]. QPR differs in how it gates and composes process signal: conditional on a correct answer, a fixed (not learned) LLM judge labels each step’s functional role over a closed taxonomy (useful, mechanical, redundant, reversion, error) and shapes the trajectory reward accordingly, with zero process reward on incorrect rollouts. QPR operates on the reward-signal axis while ASR exposes additional verifiable targets through answer-hidden scaffolds; both leave the policy-update rule unchanged.

7 Conclusion↩︎

We framed outcome-only RLVR as resting on a sparse anchor—reliable at the endpoint but under-specified on the reasoning path before and after success. SCOPE-RL densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden scaffolded sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine verified-correct trajectories after success. Together with the Step-Quality Evaluation Protocol, experiments on Qwen3-8B-Instruct show gains in accuracy, token efficiency, and process quality across DAPO-Math and Big-Math, with robustness checks under GSPO and on Qwen3-0.6B-Instruct suggesting that reward-signal densification is complementary to policy-update-level RLVR advances.

Limitations↩︎

Our experiments are limited to Qwen-family backbones and math/science-oriented RLVR settings. The main results use Qwen3-8B-Instruct, with additional validation on Qwen3-0.6B-Instruct and a GSPO backend, but do not cover other model families, larger-scale models, or broader verifiable-reward domains.

We do not exhaustively compare against all hint-based exploration or length-aware reasoning-compression methods. This is a deliberate scope choice: our experiments isolate whether verifier-compatible reward densification improves RLVR when the policy optimizer and training interface are fixed. Combining SCOPE-RL with these orthogonal techniques is a promising direction for future work.

Finally, our hyperparameter analysis is limited to a sensitivity sweep for the routing threshold \(\tau\) (Appendix 18); broader tuning of \(\beta\), \(\alpha\), and \(\{\lambda_k\}\) remains future work.

8 Training Algorithm↩︎

Algorithm 5 presents the complete pseudocode for the SCOPE-RL two-stage training procedure described in Section 3.

9 Pass@k Analysis↩︎

Since ASR employs an offline teacher model to generate scaffolded decompositions for difficult problems, a natural concern is whether this auxiliary supervision constrains the model’s exploration capacity. We address this with a pass@\(k\) analysis on AIME 2024 [31], AIME 2025-I, and AIME 2025-II [32]; we read pass@\(k{=}128\) as an upper-bound proxy for reasoning capacity, since at this budget any solution the model can in principle find is likely to surface in at least one of 128 samples.

At \(k{=}128\), ASR matches or exceeds GRPO on every subset (AIME24: 90.00% vs.%; AIME25-I: 93.33% vs.%; AIME25-II: 93.33% vs.%) and exceeds the base model by 17–27 pp. Scaffolded training therefore does not cap reasoning potential—if anything, it raises the ceiling on the harder AIME 2025 subsets.

SCOPE-RL retains the GRPO pass@128 capacity on AIME 2024 (90.00%) and AIME 2025-II (86.67%), but drops on AIME 2025-I (86.67%\(\to\)​80.00%). This is consistent with QPR’s design as a quality refiner rather than a capacity expander: by concentrating probability mass on cleaner, shorter derivations, the policy trades a small amount of solution diversity for higher useful-step density and lower token cost (Tab. 1). Capacity itself is established at ASR; SCOPE-RL then operates on top of that capacity. For applications that prioritize raw coverage at large sampling budgets, the ASR checkpoint is the appropriate operating point; SCOPE-RL is shipped because the deployment regime targets accuracy and efficiency at small \(k\) rather than the high-\(k\) frontier.

12pt

10 Motivation Probing Experiment Details↩︎

To verify that scaffold-prefix progress reflects genuine path-level learning signal rather than merely correlating with intrinsic problem easiness, we design the following probing experiment.

10.0.0.1 Data construction.

We sample 755 problems from the training set whose scaffold decompositions contain exactly 4 sub-questions. Fixing the number of sub-questions eliminates chain-length variation as a confound in the binning analysis.

10.0.0.2 Evaluation procedure.

We evaluate the base model (Qwen3-8B-Instruct [29]) on these 755 problems under two conditions: (1) Scaffold prompt: the model receives the full sub-question chain together with the original main problem and is asked to answer all sub-questions and the main problem sequentially; (2) Original prompt: the model receives only the original, undecomposed problem.

10.0.0.3 Binning and metric computation.

From the scaffold-prompt rollouts, we determine the longest correct prefix for each sample—i.e., the number of consecutive sub-questions answered correctly before the first failure (ranging from 1 to 4). Samples are grouped into four bins by this prefix length. For each bin, we report two main-answer accuracy values: (a) accuracy under the scaffold prompt (green line in Figure 1 (a)), and (b) accuracy when the same set of problems is evaluated under the original prompt (red line in Figure 1 (a)). The comparison between the two lines on the same bin isolates the effect of scaffold-provided structure from problem-level difficulty.

11 Reward Properties: Bounded Scale and Effective Gradient Support↩︎

We analyze two properties of the stage-specific rewards. Bounded scale (§11.1) keeps both rewards on the same \([0,1]\) range as \(r_{\mathrm{out}}\), so the GRPO clip range and KL coefficient need not be re-tuned per stage. Effective gradient support (§11.2) shows that bounded scale alone does not ensure learning: on the prompt subsets where outcome-only training is degenerate (\(\sigma^2_{r_{\mathrm{out}}}=0\)), ASR and QPR can each provide strictly positive reward variance under the heterogeneity conditions stated below.

11.1 Bounded Scale↩︎

For ASR, with \(m\ge1\), \(\beta\in[0,1]\), \(\Pi_i\in\{0,1\}\), and \(\mathbf{1}[\hat{a}_{\mathrm{main}}=a]\le1\), Eq. 6 gives \[0 \le R_{\mathrm{ASR}} \le \beta + (1-\beta) = 1.\]

For QPR, with \(S_u,r_k\in[0,1]\), \(\lambda_k\in[0,1)\), and \(\alpha\ge0\), each penalty factor satisfies \((1-\lambda_k r_k)\in(0,1]\), so \(\Phi\in(0,1]\); also \(\kappa=1/(1+\alpha\ln(N+1))\in(0,1]\) for \(N\ge1\). Eq. 10 gives \[0 \le Q_{\mathrm{process}} \le S_u \le 1.\] Since \(R_{\mathrm{QPR}}=Q_{\mathrm{process}}\) for verifier-correct rollouts and \(R_{\mathrm{QPR}}=0\) otherwise, \(0\le R_{\mathrm{QPR}}\le1\) follows.

11.2 Effective Gradient Support↩︎

A bounded reward stays on a learnable scale, but GRPO requires within-group reward variation to produce non-zero advantages. Concretely, the std-zero convention (§2) makes the per-rollout advantage piecewise: \[A_g(q) = \begin{cases} \dfrac{R(q,y_g)-\bar R(q)}{\sigma_R(q)}, & \sigma^2_R(q) > 0, \\[0.4em] 0, & \sigma^2_R(q) = 0, \end{cases} \label{eq:advantage95piecewise}\tag{11}\] where \(\bar R(q)=\mathrm{mean}_j R(q,y_j)\) and \(\sigma_R(q)=\sqrt{\sigma^2_R(q)}\). Throughout this appendix, \(\mathrm{Var}_g\) denotes the empirical population variance over the \(G\) sampled rollouts. The policy receives reward-driven gradient on \(q\) if and only if \(\sigma^2_R(q)>0\). The effective gradient ratio \(\eta_R = \mathbb{P}_q[\sigma^2_R(q)>0]\) (Eq. 3 ) measures the fraction of prompts that pass this threshold.

For the outcome-only reward \(r_{\mathrm{out}}\!\in\!\{0,1\}\), let \(p(q) = \frac{1}{G}\sum_g r_{\mathrm{out}}(y_g,a)\) denote the empirical pass rate in the sampled group. Then \[\label{eq:rout95variance} \sigma^2_{r_{\mathrm{out}}}(q) = p(q)\bigl(1-p(q)\bigr),\tag{12}\] which vanishes exactly when \(p(q)\in\{0,1\}\), i.e.when the group is uniformly correct or uniformly wrong. The fraction of prompts contributing reward-driven gradient under outcome-only training is therefore \(\eta_{r_{\mathrm{out}}}=\mathbb{P}_q[0<p(q)<1]\), the prompts where the current policy is neither saturated nor fully blocked.

11.2.0.1 ASR recovers signal via prefix disagreement.

Let \(L(y_g)=\sum_{i=1}^m \Pi_i(y_g)\) denote the length of the correct scaffold prefix. Since the prefix indicators are nested (\(\Pi_i\) requires consecutive correctness on sub-questions \(1\) through \(i\), so \(\Pi_1\!\ge\!\Pi_2\!\ge\!\cdots\!\ge\!\Pi_m\)), \(\Pi_m=\mathbf{1}[L(y_g)=m]\), and Eq. 6 rewrites as \[\label{eq:asr95via95L} \begin{align} R_{\mathrm{ASR}}(y_g) ={}& \tfrac{\beta}{m}\,L(y_g) \\ &{}+ (1{-}\beta)\,\mathbf{1}[L(y_g){=}m]\,\mathbf{1}[\hat{a}_{\mathrm{main}}{=}a]. \end{align}\tag{13}\] If \(L(y_g)\) differs across rollouts in a routed group and \(\beta>0\), then \(R_{\mathrm{ASR}}\) takes distinct values across rollouts, so \(\sigma^2_{R_{\mathrm{ASR}}}(q)>0\). This holds in particular on routed groups with \(\sigma^2_{r_{\mathrm{out}}}(q)=0\): ASR recovers gradient signal on this degenerate subset whenever scaffold-prefix lengths disagree across rollouts, without requiring any assumption on the main-answer distribution under the scaffolded re-rollouts.

Remark on \(\Pi_m\) as a verification gate. The factor \(\Pi_m\) in the second term means that when \(\Pi_m=0\) on all rollouts, the main-answer match contributes nothing even if some \(\hat{a}_{\mathrm{main}}(y_g)=a\) holds by coincidence. By design, this prevents \(R_{\mathrm{ASR}}\) from rewarding lucky main-answer guesses without prerequisite progress, in line with ASR’s design intent of crediting verifiable progress rather than coincidence; on such “lucky” rollouts \(r_{\mathrm{out}}\) would have variance, but the second term of \(R_{\mathrm{ASR}}\) intentionally contributes none. The empirical gap between GRPO and SCOPE-RL Stage 1 in Figure [fig:dyn95effective] indicates that ASR’s prefix signal outweighs the lucky-guess variance suppressed by \(\Pi_m\) in practice.

11.2.0.2 QPR recovers signal on outcome-uniform-correct groups.

Let \(p_c(q)=\frac{1}{G}\sum_g \mathbf{1}[\hat{a}(y_g)=a]\) denote the correct-rollout fraction in the sampled group (equivalently, \(p_c\equiv p\)). If \(p_c(q)=0\), then \(R_{\mathrm{QPR}}\) is identically zero and \(\sigma^2_{R_{\mathrm{QPR}}}(q)=0\). For \(p_c(q)>0\), let \(\mu_c, \nu_c\) denote the mean and variance of \(Q_{\mathrm{process}}(y)\) conditioned on \(\hat{a}(y)=a\). The QPR gate (Eq. 8 ) makes \(R_{\mathrm{QPR}}\) a mixture of \(\{0\}\) and \(Q_{\mathrm{process}}\), and a law-of-total-variance decomposition gives \[\label{eq:qpr95var95decomp} \sigma^2_{R_{\mathrm{QPR}}}(q) = p_c\,\nu_c + p_c(1-p_c)\,\mu_c^2,\tag{14}\] where \(p_c, \mu_c, \nu_c\) are evaluated at \(q\) and \(\mu_c\ge 0\) since \(Q_{\mathrm{process}}\in[0,1]\). Eq. 14 is strictly positive iff \(p_c(q)>0\) and either (i) \(\nu_c(q)>0\) (heterogeneity of \(Q_{\mathrm{process}}\) among correct rollouts), or (ii) \(0<p_c(q)<1\) and \(\mu_c(q)>0\) (mixed group with a non-zero typical correct-rollout reward). The first term captures intra-correct variance in process shape; the second is the binary-pass-rate variance scaled by \(\mu_c^2\). On groups with mixed correct and incorrect rollouts (\(0<p_c<1\)), \(\sigma^2_{r_{\mathrm{out}}}\) is already positive; QPR additionally provides finer-grained variation under condition (i) or (ii), and loses signal only in the boundary case where the judge returns identically zero on all correct rollouts (\(\mu_c=\nu_c=0\)). The genuine recovery happens on outcome-uniform-correct groups (\(p_c=1\), where \(\sigma^2_{r_{\mathrm{out}}}=0\)): \[\sigma^2_{R_{\mathrm{QPR}}}(q) \;=\; \nu_c(q),\] which is positive whenever correct rollouts differ in process shape; if the judge collapses to a constant value on all correct rollouts of such a group, \(\nu_c=0\) and QPR provides no signal on that prompt.

The two mechanisms target the two regimes that outcome-only training cannot resolve: ASR provides signal on routed groups via prefix-length disagreement, recovering gradient on the routed subset with \(\sigma^2_{r_{\mathrm{out}}}=0\); QPR provides signal on outcome-uniform-correct groups via heterogeneous trajectory quality, recovering gradient on \(\{p_c=1\}\). Both recovered subsets are degenerate under \(r_{\mathrm{out}}\).

12 Protocol Validation: Scaling Discrimination↩︎

Table 5 validates the Step-Quality Evaluation Protocol by applying it to the Qwen3.5 model family [34] across five scales. Metrics improve consistently with model capacity, confirming the protocol’s discriminative power.

13 Scaffold Decomposition Prompt↩︎

The system prompt used in the automated decomposition pipeline (Section 3.2; statistics in Appendix 20) is shown in Figure 6.

None

14 Concise Reasoning Prompt↩︎

To establish a non-RL baseline for reasoning compression, we prepend the following concise-reasoning instruction to the system prompt when evaluating the base model without any RL training:

15 Step-Quality Evaluation Protocol Details↩︎

15.0.0.1 Evaluation set instantiation.

As a concrete instantiation of the protocol, we select four established datasets spanning different reasoning domains and difficulty levels: GPQA [30] for graduate-level scientific reasoning, MATH500 [22] for competition mathematics, and AIME 2024 [31] and AIME 2025 [32] for high-difficulty olympiad-style problems. The protocol itself is not tied to these specific datasets and can be applied to any STEM evaluation set with verifiable answers. GPQA and MATH500 are evaluated with single-sample generation. To reduce variance on the smaller and harder AIME subsets, each AIME problem is sampled 8 times from the evaluated model and scores are averaged across rollouts.

15.0.0.2 Step-level judging protocol.

For each generated response, final-answer correctness is computed independently using a rule-based answer checker. Process evaluation is applied to the reasoning trace rather than to the final answer, and the LLM Judge is never used to decide whether the final answer is correct. The evaluation judge (Gemini-3-flash-preview [35]) is also a different model from the one used to produce QPR training rewards (GPT-4.1-mini), so the protocol’s diagnostic measurements are not generated by the same judge that shaped the policy. The judge first parses the reasoning trace into \(N\) atomic steps \(\{s_1, \dots, s_N\}\) without summarizing or rewriting the model’s solution. It then assigns each step a category label \(c_i\) from the same five-way taxonomy used in QPR: useful, mechanical, redundant, reversion, and error.

The five labels separate distinct process behaviors. Useful steps make genuine cognitive progress toward the solution. Mechanical steps are correct but mostly executional. Redundant steps repeat information or re-derive already established facts. Reversion steps occur when the model unnecessarily second-guesses or backtracks on previously correct reasoning, causing the solution path to regress. Error steps contain locally invalid reasoning, even when the final answer is eventually correct. This taxonomy allows the protocol to distinguish shorter high-quality reasoning from outputs that are merely short, lucky, or under-explained.

15.0.0.3 Diagnostic metrics.

The protocol reports decomposed metrics rather than relying only on the aggregate reward used for training. Let \(\mathcal{P}\) be the set of evaluated instances, \(\hat{a}_p\) the extracted final answer, and \(a_p\) the ground-truth answer. We report final-answer accuracy as \[F = \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \mathbf{1}[\hat{a}_p = a_p],\] with truncated responses counted as incorrect in the overall accuracy setting.

For process quality, we report the useful-step ratio \[S_u = \frac{1}{N}\sum_i \mathbf{1}[c_i = u],\] and the negative-step ratios \[r_k = \frac{1}{N}\sum_i \mathbf{1}[c_i = k], \quad k \in \mathcal{K}.\] Here, \(u\) denotes the useful label, and \(\mathcal{K}=\{\mathrm{mec},\mathrm{red},\mathrm{rev},\mathrm{err}\}\) denotes mechanical, redundant, reversion, and error steps. These metrics directly quantify whether a model uses more substantive reasoning steps and fewer low-value or harmful steps after training.

We additionally measure where the reasoning chain first breaks down. For instances containing at least one error step, we compute \[S_{\mathrm{first\text{-}err}} = \frac{1}{|\mathcal{P}_{\mathrm{err}}|} \sum_{p \in \mathcal{P}_{\mathrm{err}}} \frac{i^{(p)}_{\mathrm{err}}}{N_p},\] where \(i^{(p)}_{\mathrm{err}}\) is the first error-step index and \(N_p\) is the number of steps in instance \(p\). Higher values indicate that errors occur later in the reasoning chain, reflecting longer locally coherent derivations.

For efficiency, we report average step count and average completion tokens. These efficiency metrics are interpreted jointly with accuracy and useful-step density: a reduction in length is considered beneficial only when final accuracy and useful-step density are preserved or improved.

15.0.0.4 Evaluation set statistics.

Table 6 summarizes the composition of the evaluation set used in this work. In total, it contains 392 unique problems and produces 812 evaluation instances per model.

15.0.0.5 Implementation pipeline.

The protocol is implemented as a three-stage pipeline. We deliberately use a different model for evaluation than for QPR training: all evaluation-time judging (step parsing, step scoring, and the IAA study in Appendix 22) uses Gemini-3-flash-preview, whereas QPR training rewards are produced by GPT-4.1-mini (Appendix 16). This decoupling ensures that the diagnostic metrics reported in the main results are not produced by the same model that shaped the policy during training. Given a set of problems with ground-truth answers and a target model to evaluate, the pipeline proceeds as follows:

1. Model Rollout. Each problem is sent to the target model. The model generates a complete reasoning trace followed by a final answer. Responses that exceed the maximum token budget are marked as truncated and counted as incorrect in overall accuracy, but excluded from process-quality evaluation (since their reasoning chains are incomplete).

2. Step Parsing. An LLM Judge (Gemini-3-flash-preview) segments each non-truncated reasoning trace into atomic steps. The judge is instructed to faithfully reconstruct the original reasoning process without summarizing, merging, or optimizing — preserving redundancy, hesitation, and self-correction as separate steps.

3. Step Scoring. A second LLM Judge call evaluates each parsed step by assigning a mutually exclusive category label applied in strict priority order (reversion \(>\) error \(>\) redundant \(>\) mechanical \(>\) useful). The judge also renders a binary final-answer correctness decision by comparing the model’s answer against the ground truth.

All metrics reported in the main paper are computed from the Step Scoring outputs: category ratios are step-count fractions, and first-error position is computed per-problem then averaged.

15.1 Step Parsing Prompt↩︎

The following prompt instructs the LLM Judge to segment a model’s reasoning trace into atomic steps without any modification or judgment.

15.2 Step Scoring Prompt↩︎

The following prompt instructs the LLM Judge to assign category labels to each parsed step, and to determine final-answer correctness.

16 QPR Reward Details↩︎

16.0.0.1 Step labels.

QPR uses five mutually exclusive labels to characterize the role of each reasoning step after final-answer correctness has been verified. Useful steps make genuine cognitive progress toward the solution. Mechanical steps are correct but primarily executional computations. Redundant steps restate or re-derive information that has already been established. Reversion steps occur when the model unnecessarily second-guesses or backtracks on previously correct reasoning, causing the solution path to regress. Error steps contain locally invalid reasoning even when the final answer is eventually correct.

16.0.0.2 Judge annotation.

For each correct-answer rollout, the LLM Judge parses the trace into atomic reasoning steps without rewriting the solution and assigns one category label to each step. These annotations are used only to construct the process-quality reward; final-answer correctness remains determined by the rule-based verifier.

16.0.0.3 Judge prompt template.

The LLM Judge (GPT-4.1-mini) receives each problem, its ground-truth answer, and the model’s complete reasoning trace; it returns a structured JSON with per-step category labels.

16.0.0.4 Reward hyperparameters.

In our experiments, we set \(\alpha=0.5\). The penalty weights are \(\lambda_{\mathrm{mec}}=0.05\), \(\lambda_{\mathrm{red}}=0.20\), \(\lambda_{\mathrm{rev}}=0.25\), and \(\lambda_{\mathrm{err}}=0.40\), assigning the strongest penalty to locally invalid reasoning and smaller penalties to low-value but correct computation.

17 Training Implementation Details↩︎

All experiments are conducted on a single node equipped with 8 NVIDIA H800 GPUs, using the verl 0.7.0 training framework with vLLM 0.9.2 as the rollout engine. The base model is Qwen3-8B-Instruct [29]. Both stages use GRPO [2] with \(n{=}8\) rollout samples per prompt, a prompt batch size of 48, and a maximum prompt length of 1,024 tokens. KL penalty is disabled in both stages, including both reward-side KL and loss-side KL loss, and the entropy coefficient is set to 0.

17.0.0.1 Stage 1 — Adaptive Scaffolded RL.

Each batch begins with rollouts on the original problems under the outcome reward. Any group whose mean outcome reward falls below the threshold \(\tau\) is re-rolled out on the cached scaffolded prompt and scored with the prefix-consistent scaffold reward defined in Section 3.2. Within a routed group, scaffolded rollouts replace rather than augment the original rollouts, so each parameter update aggregates the same number of loss-contributing trajectories (\(G{=}8\) per prompt) as outcome-only GRPO; this preserves compute and gradient-batch parity with the baseline. Scaffold chains are generated offline with a teacher decomposer and cached on disk; no decomposition model is invoked during RL. The retained original-problem trajectories and the scaffolded trajectories are combined into a single batch for the GRPO [2] parameter update. No separate value network is used; the mixed reward serves directly as the scalar signal for advantage estimation. The learning rate is set to \(1 \times 10^{-6}\) with a 60-step linear warmup and no decay, and the maximum response length is 8,192 tokens.

17.0.0.2 Stage 2 — Quality-Aware Process RL.

QPR is initialized from the best ASR checkpoint and trained with a lower learning rate of \(5 \times 10^{-7}\) under a cosine decay schedule. The maximum response length is extended to 16,384 tokens to accommodate longer reasoning traces during the early phase of QPR training. Each rollout is scored by the correctness-gated process reward defined in Eq. 10 : incorrect final answers receive zero reward, while correct-answer rollouts are annotated by an LLM Judge implemented using GPT-4.1-mini. The judge decomposes each reasoning trace into atomic steps and assigns one of five mutually exclusive labels (useful, mechanical, redundant, reversion, error).

18 Scaffold Routing Threshold Sensitivity↩︎

Table 7 reports the effect of the scaffold routing threshold \(\tau\) on Stage 1 (ASR) performance. We vary \(\tau \in \{0.25, 0.5, 0.75\}\) across both training sources and report per-benchmark accuracy and average accuracy. The default threshold used in the main experiments is \(\tau=0.5\).

12pt

19 ASR Reward-Signal Ablation↩︎

To separate scaffold-provided structure from reward densification, Table 8 compares two additional ASR variants on the same scaffolded prompts. Scaffold + Final Reward keeps the scaffolded input format but rewards only the final main answer, removing all sub-answer rewards; it improves only modestly over outcome-only GRPO and remains far below ASR, indicating that the scaffold prompt itself is not the main source of the gain. Scaffold + Independent Rewards rewards each correct sub-answer independently, without the prefix-consistency gate; it performs better than final-only scaffolding but remains below ASR, showing that dense sub-answer rewards are important and that enforcing prerequisite order further improves credit assignment.

3pt

20 Scaffold Construction Statistics↩︎

Table 9 summarizes the coverage and granularity of the automated scaffold construction pipeline across the two training sources. “Filtered” refers to problems automatically excluded because their ground-truth answers do not satisfy the verifiability constraint, such as proof-based or open-ended problems. “Avg.Subs” denotes the average number of sub-problems per successfully decomposed question.

21 Human Validation of Scaffold Prerequisite Chains↩︎

To verify that the generated scaffolds genuinely form prerequisite chains rather than loosely related sub-questions, we randomly sampled 200 scaffolds from the training data and conducted manual annotation. Two annotators independently examined each scaffold and judged whether every later sub-question explicitly depends on the result of at least one earlier sub-question (i.e., no dependency breakage or redundant sub-questions that bypass the chain). A scaffold is marked invalid if any posterior sub-question can be answered without using the result of its designated predecessor.

All 200 inspected scaffolds (100%) satisfy strict prerequisite-chain integrity: each sub-question builds upon previous sub-question results, and no dependency breakage is observed. This confirms that the automated validity filter (Eq. 4 ) effectively ensures well-formed prerequisite structures before RL training.

22 Protocol Validation: Step-Label Agreement with Human Experts↩︎

The Step-Quality Evaluation Protocol’s headline metric (useful-step ratio \(S_u\)) and QPR’s positive reward term both depend on the binary distinction between useful steps and the four low-value categories. We therefore directly validate this binary distinction against human experts.

22.0.0.1 Annotation setup.

We randomly sample 50 reasoning trajectories generated by Qwen3-8B-Instruct (the model used in our main experiments) across our evaluation benchmarks (GPQA, MATH500, AIME). All trajectories are pre-parsed into atomic steps by the same LLM Judge used in evaluation; human annotators see the trajectory with step boundaries already drawn and do not re-segment. We deliberately scope this study to label-level agreement conditional on the protocol’s segmentation, since this matches how the protocol is used at evaluation time; segmentation reliability is checked separately by the scaling-discrimination study (Appendix 12), where the protocol cleanly separates five model sizes. The 50 trajectories contain approximately 1,250 atomic steps in total (mean 25 steps per trajectory), approximately balanced across the three benchmarks.

22.0.0.2 Asymmetric annotation protocol.

To control annotation cost while keeping the LLM Judge consistent with how it is actually used during evaluation, we adopt an asymmetric setup:

• Human annotators mark each step as useful or non-useful only. Two domain experts with PhD-level training in mathematics and physics annotate independently. We do not require them to discriminate among the four low-value categories, since the binary distinction (a) determines the protocol’s headline metric \(S_u\) and the sign of QPR’s positive reward, (b) is the boundary that experts can adjudicate quickly and reliably, and (c) is the only process-quality distinction we report and analyze as an evaluation metric. The four low-value categories do shape the QPR training reward through their penalty weights (Eq. 9 ); we therefore do not claim their boundaries are irrelevant, only that validating the reported diagnostic (\(S_u\)) requires validating the useful/non-useful boundary, which is what this study targets.

• LLM Judge produces the same five-class labels (useful, mechanical, redundant, reversion, error) it produces during evaluation, with no special configuration for this study. For comparison against the human binary labels, we collapse the four low-value classes into non-useful; this is the same aggregation convention used in QPR’s \(\Phi\) penalty and in reporting the useful-step ratio \(S_u\).

Annotators receive the same rubric used by the LLM Judge (Appendix 15) but are blinded to the LLM Judge’s outputs and to which method produced each trajectory.

22.0.0.3 Step-level binary agreement.

Table 10 reports binary agreement between expert consensus and the LLM Judge, with inter-annotator agreement (IAA) between the two human experts as the human ceiling. Expert consensus is defined as the agreed label when both experts match (covering 90.0% of steps); on the remaining 10.0% the LLM is compared against each expert separately and scored values are averaged.

35pt

22.0.0.4 Per-class binary metrics.

Table 11 reports per-class precision/recall/F1 of the LLM Judge against expert consensus for the useful and non-useful classes, alongside the corresponding inter-annotator F1.

22.0.0.5 Scope of this validation.

This study validates the protocol only: it shows that the LLM Judge’s per-step useful/non-useful labels are consistent with expert labels at near-ceiling levels. The pairwise preference results in Section 5.6 are a separate piece of evidence about model output quality (SCOPE-RL vs.GRPO under expert preference) and are not used here as protocol-validation evidence.

23 Pairwise Expert Evaluation↩︎

Table 12 reports the full dimension-level results of the pairwise expert evaluation. Three domain experts (Ph.D.-level mathematicians) independently judge 200 examples where both GRPO and SCOPE-RL produce the correct final answer. For each example and dimension, the majority vote among the three annotators determines the winner.

24 Robustness Results: GSPO Backend and 0.6B Scale↩︎

Table 13 reports results when SCOPE-RL is instantiated with the GSPO optimizer instead of GRPO (panel a), and on the smaller Qwen3-0.6B-Instruct model (panel b). Both experiments use DAPO-Math as the training source and follow the same two-stage protocol as the main experiments. The GSPO results (Table 13a) confirm that SCOPE-RL’s gains are not tied to a single optimizer implementation: ASR improves accuracy from 61.60% to 66.04%, and QPR further raises it to 66.93% while improving the useful-step ratio from 60.89% to 63.27% and reducing token usage by 13.1%, both relative to the GSPO baseline. The 0.6B results (Table 13b) show that even under limited model capacity, the stage-wise decomposition remains effective: ASR lifts average accuracy from 26.06% to 30.39%, and QPR improves it to 32.06% while raising Useful% from 51.79% to 58.43% and reducing tokens by 22.9%, both relative to the GRPO baseline.

4pt

25 Case Study: Reasoning Trace Comparison↩︎

Useful insight Error Reversion Redundant Mechanical / omitted

References↩︎

---

1. \(^*\)Equal contribution. \(^\dagger\)Corresponding author.↩︎