1 Introduction↩︎

Reinforcement learning from verifiable rewards is now the standard tool for improving the reasoning of large language models (LLMs) on tasks such as mathematics and code [1], [2]. Building on policy-gradient methods [3], [4] and RLHF [5], the dominant recipe is on-policy: sample a group of \(G\) rollouts from the current policy, score each by an outcome reward, and update using group-relative advantages, as in GRPO [6]. This recipe has a structural blind spot. On a hard problem, one the model almost never solves, all \(G\) rollouts fail, every reward is identical, the group-relative advantage is exactly zero, and the gradient contribution of that problem is zero [7], [8]. The model burns sampling compute on the problem and learns nothing from it. Worse, these are frequently the problems we most want to learn: the ones at the frontier of the model’s ability.

Common remedies work around this rather than fix it. Oversampling and dynamic filtering, as in DAPO [7], discard degenerate all-correct/all-wrong groups but cannot manufacture a correct rollout where the model has none. Warm-starting RL with supervised fine-tuning on reference traces (or rejection-sampled ones, STaR/RAFT-style [9], [10]) injects competence but induces entropy collapse that hurts subsequent exploration [11], [12]. A natural idea is instead to make hard problems temporarily easier so they yield a signal, then remove the assistance. If we prepend a correct prefix of a solution and ask the model to complete it, the conditional success rate rises (empirically near-monotonically) with prefix length: with no prefix the model solves almost nothing, with a long enough prefix it solves almost everything. Prefix length is therefore a continuous dial on difficulty, and for most reachable success rates \(k/G\) there is a prefix length that achieves it. This observation underlies a line of concurrent work [13]–[15].

The central question this paper addresses is what success rate to aim for, and how to hit it, i.e.how to max out the GRPO signal on hard problems. Concurrent methods fix a small set of prefix lengths per problem, chosen once from the base model so that the conditioned base accuracy is “reasonable” [13]. We argue this leaves most of the available signal on the table for two reasons. First, the GRPO signal is not maximized merely by being non-zero; prior work establishes an intermediate optimal \(k/G\) (near \(50\%\)) for RL learning signal [16], [17] (Section 4.1). Second, the prefix length that achieves a given \(k/G\) is per-sample and non-stationary: as the policy improves during training, the same prefix yields a higher \(k/G\), drifting the problem out of the productive regime. A schedule fixed at the start cannot track this.

We introduce AdaPrefix-GRPO, which treats prefix length as a closed-loop controller. During training we measure the realized batch-mean \(k/G\) and adjust a single global base prefix length toward a target \(k/G\) using a root-finding update (secant or binary search); each sample is offset from the base by a static difficulty-dependent shift, and all prefixes are annealed to zero so that the deployed policy never sees a prefix at test time. The method requires only minimal changes to the RL trainer: prefixes are injected as an assistant message in the data pipeline, gradients are masked on the prefix, and the reward is computed on the full completion. We hold the total training compute (FLOPs) fixed across all methods, so improvements are not attributable to extra training compute; the one-time calibration is a per-model preprocessing step, amortized across all runs on the same model and data, and excluded from the training budgets.

1.0.0.1 Contributions.

• We frame prefix length as a closed-loop controller of GRPO’s success rate \(k/G\), converting an uncontrolled observable into a tracked set-point (a global closed loop with per-sample difficulty offsets; Section 4).

• We adopt the established intermediate (\(\approx\!50\%\)) solve-rate set-point for RL signal [16], [17] and make it reachable across the batch through prefix length (Section 4.1).

• We propose AdaPrefix-GRPO: cold-start initialization, global closed-loop adaptation of a base prefix with per-sample difficulty offsets, and annealing to zero, implemented purely in data preparation on top of a stock GRPO trainer (Section 4.2).

• At matched training FLOPs on hard math, AdaPrefix-GRPO improves pass@1 over vanilla GRPO and a fixed-prefix baseline without collapsing pass@16, and the gain grows as the model shrinks (Section 5).

2 Related Work↩︎

2.0.0.1 The dead zone in group-relative RL.

GRPO [6] and its variants estimate advantages by centering rewards within a group of rollouts. When a problem is solved by none (or all) of the group, the centered advantages collapse to zero and the problem produces no gradient, the “learning cliff” [8] that DAPO’s dynamic filtering works around [7]. Classical exploration remedies (entropy bonuses, looser clipping, pass@\(k\) objectives) do not resolve this and can destabilize optimization [14].

2.0.0.2 Prefix- and hint-guided RL.

Closest is PrefixRL [13]: it conditions on-policy RL on prefixes of off-policy correct traces (rejection-sampled to “reuse” prior compute), masks gradients on the prefix, and reports back-generalization from prefixed to unprefixed problems, but it fixes a handful of prefix lengths per problem, chosen once from base accuracy. POPE [14] uses the minimal prefix that yields a non-zero reward; Prefix-RFT [15] anneals a single global prefix on a cosine schedule; Scaf-/Hint-GRPO [8], [18] give graded in-prompt hints; R3 [19] slides the RL start state backward along a demonstration, a reverse curriculum over prefix position. These treat assistance as a fixed or global heuristic. We additionally show it is closed-loop control, not prefixing per se, that carries the gain (Section 5.3). Off-policy methods such as LUFFY [20] and ReLIFT [21] use demonstrations as training targets; like PrefixRL we stay on-policy and use solutions only to seed exploration.

2.0.0.3 Difficulty curricula.

Ordering data by difficulty is classic [22], [23], and recent RL work curates toward an intermediate success rate [16], [17]. Instead of selecting problems that already sit at the right difficulty, we move every problem toward the target and track it over time.

3 Preliminaries: GRPO and the Dead Zone↩︎

For a problem \(x\) with reference solution and a verifier \(r(x,y)\in\{0,1\}\), GRPO samples a group of \(G\) rollouts \(y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x)\) and forms the group-relative advantage \[A_i = r(x,y_i) - \frac{1}{G}\sum_{j=1}^{G} r(x,y_j).\] We use this un-normalized centered advantage (no division by the group std). The choice matters for Section 4.1: std-normalization does not move the set-point (the aggregate positive mass still peaks at \(k/G{}\!=\!0.5\)), but it inflates the per-sample advantage of a rare correct rollout, which scales as \(\sqrt{(1-p)/p}\) and diverges as \(p\!\to\!0\). The un-normalized form keeps a well-behaved \(p(1-p)\) scaling without that instability. Un-normalized advantages are also competitive with the std-normalized variant in recent practice [24], so this departure does not weaken the baseline. Let \(k=\sum_i r(x,y_i)\) be the number of correct rollouts, so the empirical success rate is \(k/G\). If \(k=0\) (or \(k=G\)), all \(A_i=0\) and the gradient contribution of \(x\) vanishes. We call the set of problems with \(\mathrm{pass}@G\approx 0\) the dead zone: at exactly zero success probability no amount of sampling extracts signal, and under finite sampling near-zero rates yield negligible signal. On our hard split this is the common case: vanilla GRPO solves only \(\sim\)​35% of problems at pass@16, leaving a substantial dead zone on the training portion; controlled prefixing converts a substantial fraction of these into learnable problems.

3.0.0.1 Prefix as a difficulty dial.

Given a reference solution \(z=(z_1,\dots,z_L)\) for \(x\), a prefix of ratio \(\rho\in[0,1]\) is \(z_{1:\lceil\rho L\rceil}\), prepended to the prompt as a partial answer the model must continue. The conditional success rate \(\kappa(\rho)=\mathbb{E}[r(x,y)\mid y\sim\pi(\cdot\mid x,z_{1:\rho L})]\) is empirically near-monotone in \(\rho\) (3a), with \(\kappa(0)\) the unconditioned rate and \(\kappa(1)\!\to\!1\); longer, partly off-distribution prefixes occasionally fail to help, a violation rate we report (Section 5.4). We do not require strict monotonicity: the bracketing (binary-search) update only needs \(\kappa\) to cross \(\tau\) once inside \([\rho^-,\rho^+]\) and tolerates local non-monotonicity, and it is the fallback whenever the secant step is ill-conditioned. For any reachable target \(\tau\in(\kappa(0),1)\) there is a \(\rho^\star\) with \(\kappa(\rho^\star)=\tau\); for easy problems \(\kappa(0)\!>\!\tau\) already, so the controller simply sets \(\rho=0\). Gradients are masked on the prefix tokens; the reward is computed on the full (prefix \(+\) generated) sequence; evaluation always uses \(\rho=0\).

4 Method: AdaPrefix-GRPO↩︎

4.1 The intermediate set-point↩︎

Escaping the dead zone only requires \(k\ge 1\), but the magnitude of the learning signal depends on where in \((0,1)\) the group success rate \(p=k/G{}\) sits. Work on RL data selection finds, empirically and through analysis of the optimization dynamics, that an intermediate solve rate, commonly near \(50\%\), maximizes useful signal per update, while groups that are almost always or almost never solved contribute little [16], [17]; classical RL curricula targeted the same intermediate-success band long before LLMs [25].

Two intuitions match this empirical optimum, both stated for the un-normalized advantages we use (Section 3). First, the aggregate positive gradient in a group scales with the reward spread \(p(1-p)\), which peaks at \(p=0.5\). Second, the number of contrastive (correct, incorrect) pairs the group baseline can separate is \(k(G-k)\), maximal at \(k=G/2\) and zero at \(p\!\in\!\{0,1\}\), recovering the dead zone as the degenerate case. Neither argument is a proof. The set-point itself is robust to advantage normalization (Section 3), so the choice of un-normalized advantages does not affect where the optimum sits. The empirical target sweep in Section 5.3 is the actual evidence; reward noise and many valid solution paths can shift the optimum somewhat [17], so we treat \(\tau=0.5\) as a default. Our contribution is the mechanism that reaches this set-point: prefix length lets us hold the batch near \(\tau\) throughout training.

4.2 Algorithm↩︎

The set-point fixes where we want each group to sit (\(k/G{}\!\approx\!\tau\)); the algorithm is the controller that gets it there and keeps it there as the policy moves. AdaPrefix-GRPO has three stages (Algorithm 2): a cold start that places the population roughly in range, a global closed loop with per-sample difficulty offsets that tracks \(\tau\) throughout training, and an anneal to \(\rho=0\) so deployment needs no prefix.

4.2.0.1 Cold start.

Before the first step we set the initial base prefix ratio \(\rho_b^{(0)}\), shared by the whole population. We support two estimators: a conservative constant (e.g.\(0.8\), which over-prefixes but makes \(k\!\ge\!1\) overwhelmingly likely from step one), and a one-shot calibration sweep: 4 rollouts per problem on a random 512-problem subset, over the grid \(\rho\in\{0,0.2,0.4,0.6,0.8\}\), give the population-mean \(\kappa(\rho)\) curve, which we invert at \(\tau\) to set \(\rho_b^{(0)}\). Independently of which estimator sets the base, a difficulty probe always runs: four rollouts per problem at \(\rho_b^{(0)}\) give every sample a difficulty estimate \(d_x\) (its measured pass rate) that the offsets below consume; the conservative variant thus skips only the grid sweep, not the probe. Sweep and probe are one-time preprocessing, amortized across all runs on the same model and data (seeds, hyperparameter sweeps, ablations), and excluded from the training budgets. The cold start only needs to land inside \((\kappa(0),1)\); the closed loop removes the residual error. The sweep costs a single pre-training inference pass and lands near the optimum, so our main runs use it; the hyperparameter-free conservative constant trails it by only \(\sim\)​1.8 pp (Table 3) and is a reasonable fallback when even that overhead is unwanted.

4.2.0.2 Closed-loop adaptation with per-sample offsets.

The controller maintains a single base prefix ratio \(\rho_b\), shared by the whole population. Every \(N\) optimizer steps we read the realized \(k/G{}\), pooled over all rollouts in the window and smoothed with an EMA (\(\beta{=}0.7\)), and move \(\rho_b\) toward \(\tau\). Because \(\kappa(\rho)\) is monotone, this is a one-dimensional root-find. Binary search keeps bracketing bounds \([\rho^-,\rho^+]\) and halves toward \(\tau\) (robust, one bit per update); secant interpolates from the two most recent \((\rho_b,k/G{})\) pairs, \[\rho_{t+1}=\rho_{t}+(\tau-k/G{}_{t})\,\frac{\rho_t-\rho_{t-1}}{k/G{}_t-k/G{}_{t-1}},\] which converges in fewer updates when \(\kappa\) is locally smooth. We guard the denominator (falling back to a bisection step when \(k/G{}_t\!\approx\!k/G{}_{t-1}\); in our runs this fires on \(\sim\)​17% of updates, mostly early when \(k/G\) estimates are noisiest), clip the per-update step in \(\rho_b\) to \(0.05\), and keep \(\rho\in[0,\rho_{\max}]\). Pooling over the window makes the loop feasible: a single sample yields one heavily quantized Bernoulli estimate (\(k/G{}\in\{0,\tfrac1G,\dots,1\}\)) per visit, and visits are \(\sim\)​100 steps apart, far too sparse to track a moving target, whereas the window aggregates thousands of rollouts.

Holding the population mean at \(\tau\) is not enough: the same mean can hide a wide spread, with half the samples saturated (\(k\!=\!G\)) and half dead (\(k\!=\!0\)), so most rollouts still carry no signal. To compress the spread, each sample is offset from the base by a static difficulty shift, \(\rho_x=\mathrm{clip}\big(\rho_b+\delta(d_x),\,0,\,\rho_{\max}\big)\), where \(d_x\) is the sample’s cold-start difficulty and \(\delta\) is a small monotone (linear) map with a hand-tuned slope, spanning \(\pm 0.15\) around the base across the difficulty range in our runs: harder samples get slightly longer prefixes than the base, easier ones shorter. The offset is open-loop, its constants fixed across training; all tracking of the policy’s improvement happens through the single global loop. On easy problems the negative offset saturates at the clip, \(\rho_x\!=\!0\), so prefixing is reduced on easy problems (we verify no regression on the GSM8K slice included in every method’s training mixture; Section 5).

4.2.0.3 Why closed-loop, and why offsets.

A single global schedule (Prefix-RFT) or a fixed per-problem length (PrefixRL) can only be right on average at one moment: problems differ widely in \(\kappa(\cdot)\), and as the policy improves every \(\kappa\) curve shifts left, so any static or open-loop choice drifts off \(\tau\), typically toward \(k\!=\!G\), back into a zero-variance regime (3b). The global loop absorbs this non-stationarity across time. The per-sample offset addresses the orthogonal problem, heterogeneity across problems: mean-only control can leave the batch bimodal, many samples saturated or dead around a healthy average, so most rollouts are wasted; the offset narrows the spread of \(k/G\) around the tracked mean, keeping the whole batch, not just its mean, near the signal-maximizing set-point (6). The granularity ablation (Section 5.3) isolates how much of the gain the offset adds on top of base-only control. Two caveats. We use “controller” loosely: the update is a smoothed, quantized root-finder with no formal stability guarantee, and the annealing envelope (below) overrides the loop in the final phase. The offset is a static heuristic: it corrects difficulty ordering across problems but cannot adapt to a sample whose difficulty was mis-estimated at cold start.

4.2.0.4 Annealing.

To remove the train/test mismatch we impose a shrinking upper envelope and run the controller inside it: \(\rho_x \leftarrow \min\!\big(\bar\rho_t,\;\rho_b+\delta(d_x)\big)\). The envelope is held near \(\rho_{\max}\) for the first fraction \(w\) of training and then decayed to zero, \(\bar\rho_t=\rho_{\max}\cdot\mathrm{clip}\!\big(\tfrac{T-t}{(1-w)T},0,1\big)\) (we use \(w\!\approx\!0.8\)). Early on the envelope is slack and the loop is free to hold \(\tau\); late in training the envelope binds and forces every \(\rho_x\!\to\!0\), trading some signal for on-distribution (no-prefix) practice. The final policy is thus trained to solve unaided, and all evaluation is at \(\rho=0\). As the envelope binds, the controller can no longer hold \(\tau\) and the measured \(k/G\) dips modestly below it (3b); but the no-prefix held-out curve shows no sustained regression (1), so this phase consolidates what was learned. To test this, we report the final \(\rho=0\) pass rate on the full originally-dead set (Section 5.4); a non-trivial unaided pass rate on problems that started at \(\kappa(0)\!\approx\!0\) indicates that the gains are driven by back-generalization, not by prefix-conditioned imitation.

4.2.0.5 Prefix source.

Prefixes can be cut from reference solutions (always available) or from model-generated correct traces (rejection-sampled, on-distribution, as in PrefixRL). A reference path may be off-distribution and harder to continue, so the source matters; we default to reference solutions for availability and show in Section 5.3 that the controller is orthogonal to it: swapping in off-policy prefixes retains most of the gain.

4.2.0.6 Implementation.

AdaPrefix-GRPO touches the trainer only through the prefix loss mask; everything else lives in data preparation. Prefixes are appended as an assistant message that the model is asked to continue; the trainer masks loss on prefix tokens and scores the full sequence. Prefix cuts snap to sentence boundaries to avoid mid-token artifacts.

4.2.0.7 Compute accounting.

We compare methods at matched training compute, not matched steps or episodes. Following the accounting of [13], [26], a run’s cost is \(\mathrm{FLOPs}=2N D_{\mathrm{samp}}+6N D_{\mathrm{upd}}\), where \(N\) is the number of model parameters (we train with LoRA, but sampling and backpropagation traverse the full model, so we use the full parameter count), \(D_{\mathrm{samp}}\) counts all tokens processed at rollout time (including the prefill of supplied prefixes, so prefixing is not free), and \(D_{\mathrm{upd}}\) counts tokens in gradient updates. Every method runs to the same cumulative budget; AdaPrefix-GRPO’s one-time calibration sweep and difficulty probe are treated as amortized per-model preprocessing and excluded from the training budget. Because a prefixed rollout decodes only the continuation, AdaPrefix-GRPO completes more, shorter episodes within the budget, roughly two passes over the data where vanilla GRPO completes one; 4 separates this revisit effect from the controller itself by reporting both compute-matched and iteration-matched views, and by including a vanilla variant that spends the same budget on more samples per problem. Prefix prefill stays under \(\sim\)​15% of total training tokens because prefixes are capped and annealed. Table 1 reports generated length at evaluation (\(\rho=0\)), where no prefix is present, so the trace-length saving reflects the deployed policy rather than the mechanical effect of starting mid-solution.

5 Experiments↩︎

5.0.0.1 Setup.

We train on three Qwen3 sizes (0.6B, 1.7B, 4B) [27] and additionally validate cross-family on Qwen3.5-2B [28] and Gemma 4 E4B [29], to confirm the controller is not Qwen-specific. Training data is the DeepMath-103K [30] hard split (difficulty 8, binary-answer questions removed): \(\sim\)​6.8K problems with verified reference solutions (R1-generated traces shipped with the dataset). All methods train on the identical mixture: this hard split plus a 5% slice of GSM8K training problems (included so that self-disabling of prefixing on easy data can be verified; the GSM8K test split is disjoint). We use \(G=8\) rollouts, a batch of 64 prompts per optimizer step, and LoRA [31]. All methods run to the same training-FLOPs budget (\(\approx\)​6\(\times\)​10\(^{18}\) for the 1.7B model) under the accounting of Section 4. For vanilla GRPO this budget corresponds to a single pass over the mixture (\(\sim\)​112 optimizer steps, 8 full-length rollouts per problem); because prefixed rollouts decode only the continuation, AdaPrefix-GRPO completes \(\sim\)​2.1 passes (\(\sim\)​240 steps) of shorter episodes within the same budget. The controller updates every \(N{=}10\) steps. The main comparison rows (vanilla GRPO, PrefixRL, AdaPrefix-GRPO) are averaged over 3 seeds (per-seed spread \(\le\)​1.2 pp on DeepMath-hold); the remaining baselines and all ablations use a single seed. 4 additionally reports the iteration-matched view, in which baselines run the same number of optimizer steps and are therefore granted more FLOPs. Training is built on TRL [32] with vLLM [33] generation. Evaluation spans GSM8K [34] as an easy-task sanity check, MATH-500 [35], AIME 2024/2025 (averaged), AMC 2023 (40 problems), and a DeepMath held-out split (1,000 problems held out from the same difficulty-8 hard split, disjoint from training). We report pass@1 and pass@16 estimated from 8 and 16 samples per problem respectively (avg@8 for the small AIME/AMC sets) with 95% CIs from a problem-level bootstrap (resampling problems together with their 8 samples, which accounts for per-problem clustering), and mean trace length. Evaluation uses temperature 1.0 with a strict verifier and no system prompt or thinking mode; official Qwen3 reports use thinking mode and larger sampling budgets, so absolute zero-shot numbers are lower here. We additionally report held-out pass@16 to verify that gains do not come from collapsing the policy’s exploration onto a few modes.

5.0.0.2 Baselines.

• Zero-shot: base model, no RL.

• Vanilla GRPO: standard on-policy GRPO, no prefix.

• Vanilla GRPO (\(G{=}16\)): doubles the group size within the same FLOPs budget, which halves the number of optimizer steps; controls for the possibility that simply re-allocating compute toward more attempts per problem explains the gain.

• SFT-warmup \(\to\) GRPO: mid-train on reference solutions, then GRPO (the “why not just SFT” control). The SFT phase is charged to the method’s FLOPs budget (\(\sim\)​8%, one epoch over the reference traces), shortening its RL phase accordingly.

• Difficulty-filter GRPO: drop dead-zone problems, GRPO on the rest.

• PrefixRL [13]: a fixed set of prefix lengths per problem chosen once from base accuracy, static across training; the primary baseline. To keep the comparison fair, it uses the same reference solutions and prefix source as AdaPrefix-GRPO, the same FLOPs budget, and its per-problem length set is tuned with a matched length sweep; the only difference is that lengths are fixed rather than tracked. We additionally report a PrefixRL + anneal variant to separate the effect of annealing from the effect of closing the loop.

• Prefix-RFT [15]: uniform decay, a single global prefix ratio annealed \(0.8\to0\).

• AdaPrefix-GRPO (ours): global secant control of the base prefix toward \(\tau=0.5\), with per-sample difficulty offsets, annealed to \(0\).

5.1 Main results↩︎

1 reports the FLOPs-matched comparison on Qwen3-1.7B; 4 shows the full compute curves and the iteration-matched control. AdaPrefix-GRPO improves pass@1 across benchmarks. The extra passes over the data that AdaPrefix-GRPO affords within the budget do not by themselves explain the gain: giving vanilla GRPO the same budget as a larger group (\(G{=}16\)) recovers only a small fraction of it. In absolute terms it adds +6.4 pp over GRPO on MATH-500 (+17.0 pp on the DeepMath held-out split at budget end) versus +1.7/+6.1 pp for the PrefixRL baseline, i.e.–4\(\times\) its gain on both. Head-to-head, AdaPrefix-GRPO beats the PrefixRL baseline on every benchmark (+4.7 MATH-500, +3.5 AIME, +3.4 AMC, +12.0 DeepMath-hold); under a paired problem-level bootstrap (both methods are evaluated on the same problems, so we resample per-problem score differences) the DeepMath gap reaches significance while the remaining three are positive and directionally consistent; all this at half the trace length (4.4k vs.8.0k) and with no GSM8K regression; the gain over fixed prefixing thus does not hinge on the held-out split. Because every number here is measured at \(\rho=0\) (no prefix), the \(\sim\)​2\(\times\) shorter outputs come from the deployed policy itself, not from counting only the completion of a prefixed problem. Among the baselines, Prefix-RFT (uniform decay) is the strongest simple alternative, edging out fixed prefixing on MATH-500 and DeepMath with shorter traces, which confirms that annealing helps; but without the closed loop it still trails AdaPrefix-GRPO by +11.4 pp on DeepMath-hold (significant under the paired bootstrap) and by +2.7 on MATH-500. The gain also does not come from narrowing the policy: pass@16 on the DeepMath held-out split improves alongside pass@1 (base 32.8 / GRPO 35.1 / ours 56.3%), so the distribution sharpens without collapsing. On easy GSM8K all methods are near ceiling and AdaPrefix-GRPO is best or tied (92.4%), confirming the controller does no harm where prefixing is unneeded, whereas fixed prefixing slips slightly (90.4%) by over-prefixing easy problems. AIME columns average the 2024 and 2025 sets. Beyond the benchmark numbers, the solvable set itself expands: \(\sim\)​17% of originally-dead problems (zero pass@16 under the base model at \(\rho=0\) before training) are solved at pass@1 after annealing.

4pt

5.2 Scaling and cross-family generalization↩︎

We expect smaller models to live deeper in the dead zone and therefore benefit more. Each model size runs both methods to the same per-size FLOPs budget, set by the same rule as in the main setup, one vanilla-GRPO pass over the mixture at that size (so budgets scale with \(N\) and trace length); within every row of 2 the two columns are therefore directly comparable, while absolute budgets differ across rows. 2 shows the pass@1 gain over vanilla GRPO on the DeepMath held-out split shrinking but staying positive as the Qwen3 base grows, and the same controller transferring to a different architecture family (Qwen3.5-2B, Gemma 4), ruling out a Qwen-specific effect. Notably, the 0.6B model trained with AdaPrefix-GRPO surpasses the \(3\times\) larger Qwen3-1.7B trained with vanilla GRPO (41.8 vs.) despite its smaller absolute budget: at this scale the training recipe buys more than the next model size.

5.3 Ablations↩︎

3 isolates each design choice on Qwen3-1.7B. The target sweep is consistent with the intermediate-set-point prediction (Section 4.1): extreme targets clearly hurt (\(\tau{=}0.125\): 41.9), while 0.375–0.625 sit within CI of one another; we take \(\tau=0.5\) as the default from prior work and do not claim a sharp empirical optimum. Control granularity shows the gain comes from closed-loop control, not prefixing per se (PrefixRL and global-decay trail by a wide margin; the bulk of the improvement is from closing the loop). Adding our annealing envelope to the PrefixRL baseline (the PrefixRL + anneal row) recovers only a small part of that gap, confirming annealing alone is not the driver. The per-sample difficulty offset adds a smaller increment over base-only control (+2.9 pp, at the edge of the single-run CI), consistent with the view that compressing the across-problem spread of \(k/G\) matters beyond centering its mean. The advantage-norm. row addresses a confound: running AdaPrefix-GRPO with std-normalized advantages (which leave the \(\tau\) set-point unchanged but over-weight rare successes; Section 3) still helps but by less, indicating the gain is not purely a global gradient-norm effect. The annealing row shows the delayed envelope modestly beats no annealing (48.2 vs 44.8), i.e.forced consolidation helps, but the method works even with the envelope removed entirely: back-generalization carries most of the gain, and Section 5.4 discusses why the train/test distribution shift one might expect fails to materialize. The controller is also robust to the prefix source: swapping reference for rejection-sampled off-policy prefixes (PrefixRL’s regime) retains most of the gain. Finally, the group size \(G\) trades off estimator quality against update count: at a fixed FLOPs budget, doubling \(G\) halves the number of optimizer steps, so the \(G\!=\!16\) row buys a sharper \(k/G\) estimate for the controller with half the updates. Accuracy improves from \(G\!=\!4\) to \(G\!=\!8\) and is essentially flat at \(G\!=\!16\) (48.7 vs 48.2, within CI), i.e.the better estimate roughly pays for the lost updates but no more, making \(G\!=\!8\) the sensible default (and confirming the controller does not require an impractically large group).

5.4 Analysis↩︎

5.4.0.1 The controller tracks the set-point.

The closed loop holds the batch-mean \(k/G\) at \(\tau\!=\!0.5\) within a few update intervals and keeps it there as the policy improves (3b) and keeps the whole distribution of per-sample \(k/G\) concentrated near \(\tau\), whereas fixed and global prefixes drift toward saturation, piling mass at \(k/G{}\!=\!1\) while a hard core stays at \(0\) (6). Tracking difficulty cohorts, their \(\rho_x\) and realized \(k/G\) are driven toward the tracked mean by the global controller plus static offsets and anneal cleanly to \(\rho=0\) (5). This is the mechanism that DAPO approximates with dynamic sampling, oversampling to discard all-correct/all-wrong groups [7], except that prefix control creates signal on problems no amount of resampling can rescue.

5.4.0.2 Why prefixing transfers to no prefix.

Training on a partial solution still requires the model to generate the remaining reasoning and be graded on the full trace, so it practices the same sub-skills it must later chain unaided; as prefixes anneal away, those skills are exercised from progressively earlier states. This matches the back-generalization PrefixRL reports [13]: held-out (no-prefix) pass@1 rises throughout training (1), not only once prefixes vanish. Three mechanisms plausibly explain why the train/test distribution shift one might expect does not materialize. First, the loss is masked on prefix tokens: the model never imitates the (possibly off-distribution) reference text and is only reinforced on its own successful continuations, so every token that receives gradient is on-policy and no imitation drift accumulates (conditioning still shifts the state distribution; the final \(\rho=0\) pass rates below are the direct check that this shift does not persist). Second, a prefix changes only the start state of the episode, not the policy being learned: as in reverse-curriculum RL [19], the sub-skills reinforced on the remainder (algebraic manipulation, case analysis, self-verification) are the same ones a from-scratch solution must chain, and they transfer across starting positions. Third, the controller itself performs a soft anneal: as the policy improves, every \(\kappa\) curve shifts left and the loop lowers \(\rho_x\) to keep \(k/G\) at \(\tau\) (5b), so the distribution of training start states drifts toward unprefixed problems on its own; the envelope merely forces the final step to exactly \(\rho=0\). As the direct test of consolidation, the final \(\rho=0\) policy solves \(\sim\)​17% of the full originally-dead set at pass@1 (\(\sim\)​22% at pass@16), evidence that the gains persist without prefixes.

5.4.0.3 Monotonicity in practice.

Across problems the response \(\kappa(\rho)\) violates monotonicity on \(\sim\)​6% of measured \((\rho,\kappa)\) pairs (mostly long, off-distribution reference prefixes), and on those the controller falls back to bracketing without diverging; the mid-training snapshot in 6 shows the resulting concentration of per-sample \(k/G\) near \(\tau\) on raw (un-smoothed) quantized rates.

5.4.0.4 Why traces shorten.

Because each sample is held near a moderate \(k/G\), positive advantages reward efficient correct completions rather than long, lucky explorations; combined with annealing, the policy converges to shorter unaided (\(\rho=0\)) solutions than vanilla GRPO, which only finds reward through length-heavy exploration. (We treat this as a hypothesis to verify with length-conditioned analyses at \(\rho=0\).)

6 Conclusion↩︎

GRPO discards its hardest problems because zero within-group reward variance yields zero gradient. Prefix length is a continuous, per-sample dial that can place any problem at almost any reachable success rate. Conceptually this removes GRPO’s sampling barrier, the dead zone in which hard problems yield no signal under a finite rollout budget, and shifts the binding constraint to the model’s own capacity: with a controllable signal, whether a problem is learnable turns on whether the model can complete it given enough scaffolding, not on whether it can already solve it unaided. Prefixing cannot exceed that capacity ceiling or rescue a reference path the model is unable to follow, but within it, difficulty is no longer a hard cutoff on trainability. The contribution of AdaPrefix-GRPO is to control that dial in closed loop toward the intermediate, signal-maximizing target \(k/G\) [16], [17] and anneal it away, rather than fixing it in advance. At matched training compute this extracts signal where GRPO got none, beats a fixed-prefix baseline, and yields shorter solutions, with the largest gains on the smallest models.

6.0.0.1 Limitations.

AdaPrefix-GRPO needs reference (or off-policy) solutions and a reliable, near-binary verifier; we evaluate only on math reasoning and do not test noisy or non-verifiable domains, where behavior may differ. We adopt the intermediate (\(\approx\!0.5\)) set-point from prior work [16], [17] and locate it empirically; we do not derive the exact per-sample optimum under reward noise. A subtler caveat: a \(50\%\) success rate on the short remainder of a heavily prefixed problem exercises credit assignment only over that remainder, which may be an easier learning signal than \(50\%\) on a from-scratch trajectory; annealing is what forces the harder, full-length signal back in, and disentangling the two is future work. The offset map \(\delta\) is a hand-tuned static heuristic: a sample whose cold-start difficulty is mis-estimated keeps a biased prefix that the global loop cannot correct individually, and upgrading the offset to a genuine per-sample second loop would require denser per-sample \(k/G\) estimates than \(G{=}8\) with sparse revisits provides. The cold-start calibration and difficulty probe, while one-time and amortized across runs, are not free; cheap or training-free per-sample difficulty estimation is an interesting direction for future work. The prefix source also matters (reference paths can be off-distribution), though our ablation suggests the controller is largely orthogonal to it. Small benchmarks (AIME, AMC) carry wide intervals, so we rely on several benchmarks and avg@8 sampling rather than any single number. Underlying per-step data and training code are not released.

7 Measurement Details↩︎

Quantities quoted in the main text are measured as follows. Monotonicity violation rate (\(\sim\)​6%, Section 5.4): the fraction of adjacent \((\rho,\kappa(\rho))\) pairs with \(\Delta\kappa<0\), pooled over the calibration sweep (512 problems \(\times\) 4 adjacent grid intervals \(=\) 2,048 pairs) and all subsequent controller reads. Secant fallback rate (\(\sim\)​17%, Section 4.2): 4 of the 24 controller updates (\(\sim\)​240 steps, \(N{=}10\)) triggered the bisection fallback, 3 of them in the first third of training. Prefill overhead (\(<\)​15%, Section 4): prefix tokens account for 14.6% of all training-time tokens (prefill \(+\) generated), measured over the full AdaPrefix-GRPO run.

References↩︎