1 Introduction↩︎

Reinforcement learning has driven much of the recent progress in large language model (LLM) reasoning on math and code, but how much of this improvement reflects general problem-solving capability versus skills specific to the training distribution is unclear. We probe this in a setting designed to dissociate the two: a 7B model is trained with supervised fine-tuning (SFT) and reinforcement learning (RL) on constraint-satisfaction puzzles only — no mathematics problems appear in post-training — and is evaluated on hard mathematics benchmarks where current models remain far from saturation. Puzzles and mathematics share problem-solving operations such as search, planning, verification, and backtracking [1]–[3], while differing substantially in problem framing and notation. Whatever RL contributes to math performance in this setup cannot come from post-training exposure to mathematics. If transfer occurs, it must reflect changes to the abstract operations the two domains share. This setup lets us ask: does RL on puzzles improve hard-mathematics performance, and if so, what reasoning-level changes are responsible?

Transfer to math occurs: after SFT and RL on puzzles, pass@32 on OlymMATH-Hard rises by \(20\) percentage points (from \(16.0\%\) to \(36.0\%\)) without any mathematics in post-training. To understand what reasoning-level changes drive this gain, we develop a primitive-level analysis framework. A span classifier segments chain-of-thought reasoning into 9 primitives — hypothesize, compute, check, backtrack, plan, enumerate, setup, summarize, other — and a motif-extraction step characterizes how these primitives compose into recurring sequences (motifs). The framework reveals a depth-recovery tradeoff: RL extends depth (longer compute–check motif chains) but suppresses recovery (hypothesize and backtrack drop by \(70\)–\(80\%\)).

This depth-recovery tradeoff we identified motivates a modification to the RL reward: a perplexity-based novelty bonus that rewards rollouts unlikely under a frozen reference model, designed to preserve the recovery primitives vanilla RL suppresses. The bonus restores hypothesize and backtrack while retaining the depth gains from the compute–verify chains, adding a further 7pp pass@32 over a matched vanilla baseline. On problems that only the novelty-trained model solves, recovery primitives appear at substantially higher rates, suggesting the bonus works by keeping the model flexible enough to change course when direct execution fails.

Our conclusions — that RL on puzzles transfers, that it does so via primitive-level changes, and that a novelty bonus amplifies the effect — rely on a setup that cleanly isolates RL’s contribution. Recent work on puzzle-to-math transfer has not had this property. Enigmata [4] trains on a 36-puzzle suite at scale, but includes math problems in both SFT and RL, making it difficult to isolate whether transfer performance originates from in-domain math exposure. Logic-RL [5] and LogicPuzzleRL [6] train on puzzles only but evaluate primarily at pass@1 on benchmarks no harder than AIME24. At pass@1, it is impossible to distinguish whether RL expands the set of problems the model can solve at all — capability ceiling expansion — or merely sharpens a solution path it could already reach under sufficient sampling. Whether puzzle-only RL can raise the hard-math capability ceiling therefore remains undemonstrated in the prior literature.

1.0.0.1 Contributions

: (1) We show that RL on a non-mathematical domain (constraint-satisfaction puzzles) transfers to hard mathematics benchmarks, isolating RL’s contribution from SFT, (2) We introduce a primitive- and motif-level analysis framework for chain-of-thought reasoning, and use it to identify a depth–recovery tradeoff induced by vanilla RL, (3) We propose a perplexity-based novelty bonus that resolves this tradeoff and yields a further 7pp pass@32.

2 Related Work↩︎

2.0.0.1 RL for LLM reasoning and synthetic puzzle tasks.

Reinforcement learning from verifiable rewards (RLVR), popularised by DeepSeek-R1 [7] and developed through objectives such as GRPO [8], GSPO [9], and DAPO [10], has become a standard recipe for post-training reasoning models. Synthetic puzzles with deterministic verifiers are a natural testbed for this recipe, and recent works such as Enigmata [4], Logic-RL [5], and LogicPuzzleRL [6] report transfer from puzzle training to mathematical reasoning. However, existing evidence either mixes math data into the training recipe or evaluates puzzle-only training primarily at pass@1 on less difficult math benchmarks, leaving open whether non-math RL can expand the mathematical reasoning ceiling, measured by pass@k at large \(k\). This question connects to a broader debate about RL post-training transfer: some work reports broad cross-domain gains from math RL [11], while other studies find that single-domain RL mainly sharpens in-domain behaviour [12]–[14]. We study this in a clean isolation: our pipeline uses no math data in SFT or RL and evaluates pass@k on hard out-of-domain math.

2.0.0.2 Exploration and diversity in RL fine-tuning.

Recent work has argued that RLVR can improve pass@1 while reducing pass@k diversity, motivating method families that preserve exploration during RL fine-tuning [14]–[17]. One family modifies token-level signals via covariance-based clipping, entropy-term additions, or low-probability regularisation [15], [18], [19]. A second family reweights whole trajectories by rarity, using current-policy ranks, strategy clusters, or outcome-level novelty  [20]–[22]. Our novelty bonus belongs to the trajectory-level family but differs from prior methods in its anchor: novelty is scored under a frozen reference model rather than the current policy, so the diversity target does not drift as training progresses. The bonus applies only to verifier-correct rollouts and uses top-\(k\) token NLL rather than LLM-judge clustering, rewarding alternative successful reasoning paths without per-trace judge calls at training time. Unlike work that optimises pass@k directly [23], we retain the standard verifiable-reward objective and use novelty as an auxiliary exploration-preserving bonus.

2.0.0.3 Mechanism analyses of reasoning.

A growing literature analyzes reasoning mechanisms by labeling behaviors in chain-of-thought traces, including verification, backtracking, subgoal setting, and related cognitive operations [24], [25]. These studies show that RL reshapes the distribution of reasoning primitives, amplifying some and suppressing others [24], [26]. Our analysis differs in scale and method: we distill judge annotations into a lightweight span classifier that can label \(25\)k-token traces across tens of thousands of rollouts and ten checkpoints, avoiding both per-trace judge calls and injected behavior markers. Unlike prior analyses, we use the resulting motif distributions to study out-of-domain transfer rather than tracking only per-trace primitive counts.

3 Post-Training Pipeline and Results↩︎

3.0.0.1 Setup and evaluation.

We use OLMo3-7B-Instruct-SFT [27] as our base model: it is fully open, has undergone no prior RL fine-tuning, and retains upstream mathematical knowledge from pretraining and instruction tuning. This last property is critical as any post-training gain reflects improved search and verification over existing knowledge, not the injection of new mathematical content. Our post-training pipeline uses only constraint-satisfaction puzzles from the Simon Tatham collection [28]–[30] — Bridges, Pattern, Undead, and Galaxies — with no mathematics problems in either the SFT or RL stages. We evaluate on three hard out-of-domain mathematics benchmarks: OlymMATH-Hard [31], HMMT [32], and OMEGA [33], all of which remain far from saturation for a 7B model and therefore probe the capability ceiling rather than benchmark familiarity. Our primary metric is pass@32, since a pass@1 improvement can reflect either capability ceiling expansion, where new problems become solvable, or sharpening of an already-reachable path, and the two are indistinguishable at the single-sample level. Large-\(k\) pass rates isolate the former, and all evaluations use \(32\) rollouts per problem at temperature \(T{=}0.6\).

3.0.0.2 Training Recipe.

The training pipeline has two stages. First, because the base model has near-zero puzzle competence, direct verifier RL would provide little reward variance, so we perform supervised fine-tuning on rejection-sampled reasoning traces distilled from Deep-Seek Reasoner (DSR), keeping only traces that verify as correct under the puzzle scorers. Second, from this SFT checkpoint we train two multi-puzzle variants of the RL algorithm Group Sequence Policy Optimization (GSPO) [9]. Both variants use the same verifier reward, which combines an exact reward for fully correct boards with a partial completeness reward. The partial completeness reward is proportional to how much of the final board state matches the solution. The vanilla GSPO condition uses only this verifier reward, while the novelty condition additionally rewards correct rollouts whose reasoning traces are assigned low probability under the frozen SFT reference. We use GSPO rather than token-level Group Relative Policy Optimization (GRPO) because our target responses are \(25\)–\(28\)k tokens and GSPO trained reliably in long-context pilots where GRPO was unstable. Full hyperparameters and reward details are in Appendix 16.

3.0.0.3 Results.

Puzzle SFT alone raises OlymMATH-Hard pass@32 from \(16.0\%\) at the base to \(23.0\%\). Both RL variants, trained from this SFT checkpoint, improve further: vanilla GSPO reaches \(29.0\%\) (\(+6\)pp over SFT), while the novelty variant reaches \(\mathbf{36.0\%}\) (\(+13\)pp over SFT and \(+7\)pp over the matched vanilla GSPO baseline), as shown in Figure 1 and Table 3. Notably, the novelty gain grows with \(k\)—modest at pass@1 but \(+7\)pp at pass@32—showing that it increases the capability ceiling rather than sharpening a dominant path. The same pattern holds across other tested benchmarks: on OMEGA Explorative [33], a synthetic benchmark of structured mathematical problems at increased complexity, the novelty variant reaches \(54.5\%\) against \(48.5\%\) for SFT; on HMMT [32] (\(123\) competition problems), it reaches \(49.6\%\) against \(46.3\%\). Per-problem solve rates further support this reading (Appendix Figure 7): Base\(\to\)Novelty newly solves \(29\) of \(100\) OlymMATH-Hard problems while regressing on only \(5\), confirming the recipe lifts the ceiling broadly rather than sharpening a few already-reachable problems. Finally, on larger-grid puzzle evaluations, both RL variants raise pass@\(k\) over SFT across the full sampling budget on Bridges \(8{\times}8\) and Undead \(5{\times}5\), suggesting the learned strategies extrapolate across grid scale.

The remainder of the paper asks why puzzle-only training transfers to hard mathematics: what reasoning behaviors does SFT introduce, what does vanilla RL amplify or suppress, and which changes predict the sampled ceiling gains observed above? Section 4 introduces the primitive-level analysis, Sections 4.1–4.3 apply it to the recipe stages, and Sections 5–5.1 use the resulting diagnosis to derive and validate the novelty bonus.

4 Analysis Framework: Reasoning Primitives↩︎

Aggregate metrics such as response length and token entropy reveal that training changes the policy [34], [35], but not what reasoning behaviours change. We therefore analyse chain-of-thought traces as sequences of reasoning primitives: short, semantically coherent spans that play canonical roles such as planning, computing, checking, hypothesising, or backtracking. Because traces can reach \(\sim\!25\)k tokens, we split each trace into spans at paragraph boundaries and discourse markers (e.g., Case 1, Wait, Alternatively, Going back), yielding contiguous spans of \(80\)–\(250\) tokens (full algorithm in Appendix 17). Each span is then classified with its preceding context, producing one primitive sequence per rollout.

4pt

Table 1 gives the nine-class vocabulary. We developed the labels using a jury of LLM judges (DeepSeek-V3.2 and DeepSeek-Chat [36]), iterating until inter-judge agreement reached \(86\%\) on a held-out span set. Because judge-based labelling is impractical at scale, we distil the judges into a nine-way span classifier that reaches \(0.80\) macro-F1 against judge consensus.

Finally, we convert each primitive sequence into motifs — \(k\)-grams of primitives for \(k = 2, \ldots, 15\) — and for each checkpoint track primitive frequencies, motif frequencies, and their cross-checkpoint correlation with pass@\(k\) (Appendix 17). Tracking these statistics across checkpoints lets us identify which reasoning patterns each training stage amplifies or suppresses, and which correlate with hard-math ceiling gains.

4.1 Puzzle SFT Induces a Reasoning Vocabulary↩︎

Before any RL, we ask what puzzle SFT alone contributes to reasoning behaviour. Applying our primitive classifier (Section 4), we compare base, DSR-teacher, and post-SFT rollouts on held-out puzzles, and base versus post-SFT rollouts on OlymMATH-Hard. If puzzle-only SFT transfers, the math rollouts should acquire the same primitive vocabulary even though no mathematics appears in the SFT corpus. Figure 2 tests this at both the primitive and trigram-motif levels.

4.1.0.1 Puzzle SFT induces the vocabulary used later.

On held-out puzzles, post-SFT rollouts broadly inherit the DSR teacher’s primitives and motif composition, showing that puzzle SFT induces the teacher’s reasoning style in addition to improving test accuracy. The recovery primitives hypothesize and backtrack are amplified relative to the base model, while the main exploitation primitives remain close to the teacher distribution (Appendix 12).

The transfer tests address whether this vocabulary appears out-of-domain. On OlymMATH-Hard, the base model has little structured primitive vocabulary: only \(2.8\) primitive spans per trace, with essentially no hypothesize or backtrack. After puzzle SFT, aggregate primitive count rises to \(19.6\), check grows from \(0.1\) to \(4.0\), and hypothesize/backtrack appear at \(2.0\) and \(0.4\) episodes per trace. Since the SFT corpus contains only puzzle traces, these changes indicate that puzzle SFT induces a reusable reasoning vocabulary. At the motif level, recovery, exploitation, and verification trigrams are likewise introduced by SFT (definitions in Appendix 17), establishing the reasoning vocabulary that the following sections examine under RL.

Having established that puzzle SFT makes these primitives available, we next ask how vanilla RL reallocates them.

4.2 What Vanilla RL Changes: Deeper Compute–Verify Chains↩︎

What does vanilla RL change relative to puzzle SFT? Our answer is not that it discovers a new reasoning vocabulary. The SFT model already has the relevant primitives; vanilla GSPO instead changes how strongly the model commits to one part of that vocabulary. On OlymMATH-Hard, the dominant shift is toward deeper exploitation: longer contiguous runs of compute, check, and setup episodes that carry an approach forward once the model is on track. The same depth allocation also appears on harder puzzles, paralleling the math-side shift.

4.2.0.1 From primitives to execution chains.

At the primitive level, the three exploitation primitives all rise on OlymMATH-Hard (Figure 3 (a)): compute nearly doubles (\(4.6 \to 8.1\) episodes per trace), check grows from \(4.0\) to \(5.6\), and setup from \(2.2\) to \(3.5\). The same shift appears in concrete motifs (Figure 3 (b)). Short compute–verify patterns increase, but the larger effect is on longer alternations such as check\(\to\)compute\(\to\)check\(\to\)compute\(\to\)check, which rises \(3.4\times\) from SFT to vanilla GSPO. This is the first sign that vanilla RL is not merely producing more local verification; it is assembling longer execution chains from the same primitives SFT introduced.

4.2.0.2 A parallel depth shift on hard puzzles.

Frequency alone does not tell us whether these primitives are chained together, so we next measure depth directly. We define chain depth as the longest contiguous run of exploitation episodes in a trace before any other primitive intervenes (Appendix 18). On OlymMATH-Hard, mean chain depth doubles from SFT to vanilla GSPO (\(3.2 \to 6.6\)), and the \(90\)-th percentile rises from \(7\) to \(14\) (Figure 4 (a)). The same depth allocation appears on puzzles: holding the checkpoint fixed, chain depth increases on harder puzzle types and larger grids, and vanilla GSPO allocates depth more aggressively than SFT (Appendix 14). To check that this is not driven by a few exceptional traces, we also measure the mean length of all exploit runs per trace; that measure rises from \(1.74\) after SFT to \(3.06\) after vanilla GSPO (Figure 4 (b)). Thus vanilla RL learns extended compute–verify execution on harder puzzle instances.

On OlymMATH-Hard, this depth shift tracks the vanilla-RL gain: pass@32 rises from \(23.0\%\) at SFT to \(29.0\%\), a \(+6\)pp gain that accompanies the depth doubling above. This gain represents a ceiling extension — in contrast to the convergence pattern of [14], puzzle RL adds genuinely new solvable problems rather than rearranging the SFT distribution. Along the Base\(\to\)SFT\(\to\)vanilla-GSPO trajectory, mean exploit-run length and pass@32 rise together (Figure 4 (b)). Paired inspection of SFT and vanilla-GSPO rollouts on problems solved by both checkpoints gives the same interpretation: vanilla GSPO usually improves execution of an approach rather than changing the set of approaches attempted (Appendix 23, Part A). The novelty-bonus variant later breaks this depth trend, gaining another \(+7\)pp pass@32 with shorter exploit runs; the next section shows what vanilla RL suppresses.

4.3 What Vanilla RL Suppresses↩︎

The depth gain in Section 4.2 comes with a cost. As vanilla GSPO commits more strongly to compute–verify execution, it suppresses the exploratory primitives that puzzle SFT introduced. On OlymMATH-Hard, hypothesize falls from \(2.0\) to \(0.6\) episodes per trace (\(-70\%\)), and backtrack from \(0.41\) to \(0.08\) (\(-80\%\)). On the puzzle training distribution itself the suppression is sharper and visibly developmental: across \(20\) vanilla-GSPO steps, both primitives fall monotonically by \(95\)–\(98\%\) while puzzle solve rates rise substantially (Figure 5 (a)). The novelty bonus restores both primitives toward their SFT levels on math (Figure 5 (b)). Vanilla RL therefore does not merely deepen exploitation; it also prunes the recovery vocabulary that would let the model recover when a direct line of reasoning fails.

4.3.0.1 Why does vanilla RL suppress its own exploratory vocabulary?

A simple explanation is that, on the puzzle training distribution, vanilla RL makes the model better at recognizing and executing the right approach. Once the policy can usually identify a productive route, the highest-reward behavior is not to search over alternatives; it is to carry that route farther through setup, computation, and checking — the same pressure that produces the depth increase of Section 4.2. In that regime, hypothesize and backtrack introduce detours without often changing the outcome, so continued on-policy training can make them less likely. Figure 5 (a) is consistent with this interpretation: as puzzle accuracy rises, the policy gives up the recovery moves that would help when its first approach is wrong.

4.3.0.2 But hard math is not the training distribution.

That tradeoff is useful only when the first approach is usually right. Hard math often violates this assumption: a promising route may lead to a dead end, and solving requires trying another representation, case split, or construction. A model whose hypothesize and backtrack primitives have been suppressed can still execute a chosen approach deeply, but it is less able to recover when that approach fails. The novelty GSPO variant, also trained from the SFT checkpoint, preserves both primitives toward their SFT levels on math and raises OlymMATH-Hard pass@32 to \(36.0\%\), compared with \(29.0\%\) for vanilla GSPO and \(23.0\%\) for SFT (Table 3); Section 5.1 shows that the gain is concentrated on problems whose solved traces use exactly this recovered structure.

Section 5 introduces a targeted intervention designed to preserve the recovery primitives this section showed vanilla RL suppresses.

5 A Frozen-Reference Novelty Bonus Preserves Recovery↩︎

Sections 4.2 and 4.3 give us a design target: preserve the recovery primitives that vanilla RL suppresses without giving up the compute–verify depth it learns. We do this by adding a reward bonus for correct rollouts whose reasoning is unusually surprising under the frozen SFT reference model. The bonus is applied only to verifier-correct rollouts and normalised within the correct rollouts on the same prompt: it rewards alternative successful reasoning paths, not arbitrary low-probability text.

5.0.0.1 Bonus definition.

For each rollout \(r\) on prompt \(p\), we evaluate the generated tokens under the frozen SFT reference \(\pi_{\mathrm{SFT}}\), with token loss \(\mathrm{NLL}_t(r)=-\log \pi_{\mathrm{SFT}}(r_t \mid r_{<t},p)\). We define the rollout novelty score as the mean of the top-\(k\) largest token NLLs: \[s(r) = \frac{1}{k} \sum_{t \in \mathrm{top}_k(\mathrm{NLL}(r))} \mathrm{NLL}_t(r), \qquad k = 100.\] For correct rollouts within each prompt group, we \(z\)-score and clip the novelty scores before adding them to the verifier reward: \[z_p(r) = \mathrm{clip}\!\left(\frac{s(r) - \mu_p}{\sigma_p},\; -2,\; 2\right), \qquad R_{\mathrm{novelty}}(r) = R_{\mathrm{base}}(r) + \alpha z_p(r)\mathbb{1}[r \text{ correct}].\] where \(\mu_p\) and \(\sigma_p\) are computed over correct rollouts for prompt \(p\). We set \(\alpha=0.1\). Thus incorrect rollouts receive no novelty credit, and correct rollouts receive more credit when their reasoning is more novel relative to the SFT reference than other correct rollouts on the same prompt. Pseudocode is in Appendix 20.

5.0.0.2 Why frozen-reference top-\(k\) NLL?

The frozen reference keeps the novelty target stable. If novelty were ranked under the current policy, as in unlikeliness-style rewards [20], the target would drift toward whatever the model is already doing well, weakening the diversity signal as training progresses. The top-\(k\) statistic makes the signal local enough to matter at our trace lengths: with \(\sim\)​25k-token rollouts, mean NLL has near-zero within-group variance because most tokens are high-probability formatting or structural choices. Using only the \(k=100\) highest-NLL tokens isolates rare decision points and amplifies the within-group signal by \(10\)–\(15\times\) (Appendix 21).

5.0.0.3 Effect.

The novelty-bonus variant starts from the same SFT model as vanilla GSPO and uses the same data, algorithm, training budget, and number of optimisation steps. It raises OlymMATH-Hard pass@32 to \(\mathbf{36.0\%}\), compared with \(29.0\%\) for vanilla GSPO and \(23.0\%\) for the shared SFT starting point (Table 3; Figure 1 (a)). Mechanistically, it restores the recovery primitives that vanilla GSPO suppressed: backtrack returns to its SFT level and hypothesize recovers to about \(60\%\) of its SFT rate (Figure [fig:exploration-counts]). This restoration comes with less exploitation depth: chain depth, exploit-run length, and the exploitation motifs amplified by vanilla GSPO all reduce modestly (Figures 4 (b) and 3 (b)). The bonus therefore does not improve by adding still more depth; it trades some direct compute–verify execution for recovered ability to try and abandon alternatives.

We also screened diversity-promoting alternatives in smaller pilots, including a loss-level entropy bonus and entropy-advantage variants inspired by [18]. These pilot experiments were used for method selection, not as a definitive full-scale comparison: entropy matched the novelty bonus at mini scale but was less stable, while entropy-advantage variants collapsed below baseline. We therefore scaled the frozen-reference novelty bonus for the GSPO comparison; the pilot summary and its limitations are in Appendix 19. The next section examines whether the restored recovery primitives account for the performance gain.

5.1 Where the Novelty Gain Comes From: Recovery on Unique-to-Novelty Problems↩︎

Section 5 left a specific question: does the novelty bonus restore hypothesize/backtrack everywhere, or does it deploy recovery selectively on the problems where direct compute–verify execution fails? We test this on the \(100\) OlymMATH-Hard problems by comparing two diagnostic subsets: problems solved by both vanilla GSPO and the novelty-bonus variant (“shared”), and problems solved only by the novelty-bonus variant (“unique-to-novelty”). The full accounting also includes vanilla-only and unsolved problems (Appendix 22), but the contrast between shared and unique-to-novelty problems is the cleanest test of selectivity: shared problems show whether the novelty-bonus variant behaves like vanilla GSPO when direct exploitation is enough, while unique-to-novelty problems show whether recovery appears where vanilla GSPO fails.

5.1.0.1 Recovery motifs are selective, not uniform.

The aggregate motif analysis shows where the novelty bonus changes behavior (Figure 6 (a)). On the \(20\) shared problems, solved traces from vanilla GSPO and the novelty-bonus variant look nearly identical: both rely on direct compute\(\to\)check exploitation, and recovery motifs remain rare. On the \(16\) unique-to-novelty problems, the pattern changes. The recovery motifs are absent from vanilla GSPO’s \(512\) attempts but remain available in the novelty-bonus variant (\(11/512\) rollouts). Within the novelty-bonus run, they are also enriched in successful traces: \(3/24\) solved traces contain at least one diagnostic recovery motif, compared with \(8/488\) failed traces, an approximately \(8\times\) difference. Thus the gain is not explained by recovery being used uniformly across novelty rollouts. Instead, the bonus preserves a recovery move that vanilla GSPO has lost, and that move appears disproportionately in successful unique-to-novelty traces. This is the selectivity pattern Section 4.3 predicted: recovery is unnecessary on shared problems but most useful on problems where vanilla GSPO’s direct execution fails. Appendix 23, Part B, gives two full trace pairs showing the same pivot in text.

5.1.0.2 Novelty aligns depth with successful trajectories.

Section 4.2 left a puzzle: the novelty-bonus variant achieves higher pass@32 than vanilla GSPO despite shorter average exploit runs. Splitting traces by whether they solve the problem clarifies this pattern. Vanilla GSPO is the only RL run whose unsolved traces are deeper than its solved ones; the novelty-bonus variant reverses the sign, with solved traces deeper than unsolved (Figure 6 (b)). This suggests that vanilla GSPO often spends depth on unsuccessful continuations, whereas the novelty bonus makes depth more coupled to trajectories that succeed.

6 Conclusion and Limitations↩︎

We post-trained a 7B model with no mathematics in any post-training stage and found that puzzle-only SFT and RL can raise the sampled ceiling on hard mathematics. The primitive analysis suggests a more specific lesson than “more reasoning helps.” Puzzle SFT makes a reusable vocabulary of planning, checking, hypothesizing, and backtracking available out of domain; vanilla GSPO then sharpens execution by lengthening compute–verify chains, but at the cost of suppressing recovery moves. The frozen-reference novelty bonus improves transfer by preserving some of that recovery vocabulary while retaining enough exploitation depth. More broadly, cross-domain RL transfer appears to depend not only on how far a model can pursue an approach, but also on whether training preserves the ability to abandon that approach when it fails. Taken together, our contribution is an empirical test of whether RL-trained LLMs can turn a shared problem-solving vocabulary into measurable out-of-domain math gains.

6.0.0.1 Limitations.

Our results are limited to one base model (OLMo3-7B-Instruct-SFT) and one RL family (GSPO); the novelty-bonus hyperparameters were selected from pilot runs rather than exhaustive tuning; and the primitive analysis is a diagnostic lens, not a per-problem causal proof. Extended discussion is in Appendix 9.

7 Broader impact↩︎

Our contribution improves out-of-domain reasoning of a small open research model via verifier-only puzzle RL; the standard dual-use considerations for reasoning LLMs apply, and the method does not introduce capability surfaces beyond those already accessible from the base model.

8 Compute resources↩︎

Stage 1 SFT runs on \(8\times\)B200 (80 GB) for \(\approx\)​9 hr. The two Stage 2 RL variants—vanilla GSPO and the novelty-bonus GSPO—are parallel branches from the same SFT checkpoint and each runs on \(8\times\)B200 for \(\approx\)​70 hr. End-to-end reproduction of the headline OlymMATH-Hard \(36.0\%\) pass@32 number (SFT plus the novelty-bonus branch) is therefore approximately \(632\) GPU-hours; reproducing both Stage 2 branches together is approximately \(1{,}192\) GPU-hours. The full research project—preliminary pilot runs, abandoned hyperparameter configurations, and ablations not reported in the paper—required roughly an additional \(2\)–\(3\times\) the reproduction budget on top.

9 Limitations and future work↩︎

Our evidence comes from one base model, OLMo3-7B-Instruct-SFT, and one RL algorithm family, GSPO. Because full-scale runs with long puzzle rollouts are computationally expensive, we leave validation on other models, scales, and RL objectives to future work. The novelty bonus also has design choices that were not exhaustively tuned at full scale: \(\alpha = 0.1\) and top-\(k = 100\) were selected from smaller pilot runs. For the same reason, alternative diversity mechanisms were screened in smaller pilots rather than compared at the full training scale. Finally, our primitive analysis is a diagnostic lens, not a causal proof for every solved problem: the classifier was validated on this model family and trace style, and the motif evidence shows where recovery is available and enriched, not that every novelty-only solution uses it.

10 Detailed accuracy tables↩︎

This appendix collects the per-benchmark accuracy tables referenced from Section 3. Table 2 reports in-domain puzzle accuracy at pass@1; Table 3 reports held-out mathematics at pass@1 and pass@32. Both are restated here for ease of cross-reference; Figure 1 summarises the same data as pass@\(k\) curves for the body benchmarks.

11 Additional benchmark pass@\(k\) curves↩︎

Figure 1 (body) reports OlymMATH-Hard, HMMT combined, OMEGA Explorative, and the held-out larger-grid puzzle evaluations. We document four additional benchmarks here for completeness: OlymMATH-Easy, AIME25, OMEGA Compositional, and OMEGA Transformative. None earn a body slot — each is either saturated by SFT or has a confound that limits the RL-discriminating zone — but together they verify that the recipe does not regress on benchmarks the body figure does not show.

OlymMATH-Easy and AIME25 saturate quickly with \(32\) rollouts at this scale, so the gap between recipe stages collapses past pass@\(8\) — both Vanilla GSPO and Novelty are within \(\sim 1\)–\(3\)pp of each other and within \(\sim 4\)pp of SFT on pass@\(32\). They confirm the recipe roughly preserves easier-tier accuracy (differences between RL stages are within the noise floor at saturation) but do not differentiate RL stages. OMEGA Compositional shows a near-tie between SFT and vanilla GSPO at pass@\(32\) (both \(56\%\)); novelty lifts to \(60\%\). OMEGA Transformative is the most stress-test-like split but is bimodal at \(7\)B scale: a per-config audit shows only \({\sim}24\) of \(100\) problems are in the RL-discriminating zone — the rest are either solved by Base (\(42\) problems) or universally unsolvable at this scale (\(34\) problems, dominated by de Moivre and function-intersection configs where every checkpoint scores zero). Vanilla GSPO regresses by a hair on pass@\(32\) (\(43\to 42\)); novelty recovers and adds \(+2\)pp over SFT.

11.0.0.1 Evaluation cost note.

A single pass@\(32\) sweep at max_gen_toks\({=}25{,}000\) generates \(N \times 32\) rollouts of up to \(25{,}000\) tokens each. With seed-diversification required for valid pass@\(k\) statistics, the \(32\) samples per problem must be sampled independently rather than batched as a fixed-prompt expansion; the cost is therefore dominated by autoregressive decoding latency, not forward-pass throughput. On \(8{\times}\) B200 with vLLM tensor-parallel\({=}1\), a single \(4\)-checkpoint sweep at \(N{=}100\) takes roughly \(11{-}12\) wall-clock hours; a full \(4\)-checkpoint sweep at \(N{=}300\) would take \(\sim 36\) hr. This is the reason we sub-sample OMEGA splits to \(N{=}100\) with a fixed seed for apples-to-apples scoring across checkpoints; the sub-sampled estimate is within \(\sim 1\)pp of the full-set estimate where verified (OMEGA Explorative sub-100 \(72.0\%\) pass@\(32\) vs full-set \(\sim 72.5\%\)).

12 Per-setting primitive distributions↩︎

We apply our classifier of Section 4 to three populations of reasoning traces and report the relative composition of primitive episodes (percent of all classified primitive labels). The three settings are: DSR puzzles (reasoning traces from the DSR teacher on the puzzle problems used in our SFT corpus); SFT puzzles (post-SFT OLMo3 rollouts on held-out puzzle test sets at training grid sizes); SFT math (post-SFT OLMo3 rollouts on OlymMATH-Hard, the same distribution Section 4.1 reports per-trace counts on).

13 Mechanism patterns across math benchmarks↩︎

The mechanism analysis in Sections 4.2–4.3 is reported on OlymMATH-Hard. To check that the patterns are not specific to that benchmark, we run the same classifier on rollouts at the four main checkpoints on OlymMATH-Easy and AIME25 and recompute the body’s load-bearing metrics with the canonical body definitions (non-overlapping motif counts; chain depth as the longest consecutive run of compute/check/setup). This appendix reports those cross-benchmark numbers; the Hard column reproduces the body figures.

The qualitative finding is that every directional ranking in §4.2–§4.3 holds on Easy and AIME25 with similar magnitudes:

• Chain depth: \(\mathrm{Base} < \mathrm{SFT} < \mathrm{Vanilla\;GSPO}\), novelty sits between SFT and vanilla GSPO.

• hypothesize count: suppressed under vanilla GSPO (\(3\)–\(5\times\) drop from SFT), partially restored under novelty.

• backtrack count: suppressed under vanilla GSPO (\(\sim 5\times\)), fully restored or exceeded by novelty.

• \(k = 3\) and \(k = 5\) exploitation motifs (\(\mathrm{\small check}\to\mathrm{\small compute}\to\ldots\)): rise sharply from SFT to vanilla GSPO, novelty drops back partway.

13.0.0.1 Where magnitudes shift.

AIME25 has slightly higher chain depth at every checkpoint (\(\sim\) \(1\) unit deeper than Hard) — competition-math problems require longer reasoning chains than the OlymMATH-Hard distribution. Easy has slightly weaker HYP/BTK suppression ratios under vanilla GSPO than Hard, consistent with Easy saturating earlier (\(\mathrm{SFT}~\texttt{pass@32} = 72\%\)): the SFT distribution there is already close to ceiling, leaving less room for vanilla RL to differentiate. The AIME25 novelty-bonus cell has the smallest sample size (960 traces; \(30\) docs \(\times\) \(32\) rollouts) and is the most sample-size-noisy column. Across all metrics, no benchmark contradicts the direction of any §4.2–§4.3 ranking.

14 Depth shift on puzzles and within-checkpoint scaling↩︎

Section 4.2 reports that vanilla RL roughly doubles chain depth on OlymMATH-Hard. We verify here that (i) the same depth shift is present on harder OOD puzzles—so the math depth shift is not a math-specific artefact—and (ii) within-checkpoint, depth scales with task complexity, ruling out a mechanical “bigger-input-longer-output” reading.

14.0.0.1 The SFT\(\to\)vanilla GSPO depth shift appears on OOD puzzles too.

On solved traces at the largest puzzle grids in our test suite (held out from training, which used smaller grids), mean chain depth rises substantially from SFT to vanilla GSPO. On Bridges \(8{\times}8\), depth goes from \(13.6\) [\(13.1\), \(14.2\)] at SFT (\(n_\text{solved}{=}185\)) to \(19.4\) [\(19.1\), \(19.7\)] at vanilla GSPO (\(n_\text{solved}{=}1181\))—a \(+5.8\) shift with non-overlapping \(95\%\) bootstrap CIs. On Undead \(5{\times}5\), depth rises from \(18.7\) [\(16.6\), \(21.0\)] (SFT, \(n{=}38\)) to \(20.7\) [\(20.1\), \(21.3\)] (vanilla GSPO, \(n{=}324\)); the SFT estimate is noisy due to low solved counts but the direction matches Bridges. The depth allocation learned during puzzle RL surfaces in both held-out math and harder puzzle generalizations (Figure [fig:chain-depth-size-ladder]).

14.0.0.2 Within-checkpoint, depth scales with task complexity.

Holding the model fixed and varying the puzzle type (Figure [fig:chain-depth-per-puzzle]), Undead \(5{\times}5\) elicits deeper chains than Bridges \(8{\times}8\) in both SFT (\(14.4\) vs \(11.2\)) and vanilla GSPO (\(21.3\) vs \(20.0\), averaged over all traces)—despite the smaller grid, ruling out the mechanical “bigger grid \(\Rightarrow\) longer trace” story. We attribute this to the complexity of line-tracing under reflection in undead vs.the simpler bridge-count constraint.

Holding the model and puzzle type fixed and varying only the grid size on solved traces (Figure [fig:chain-depth-size-ladder], \(95\%\) bootstrap CIs), Bridges \(7{\times}7 \to 8{\times}8\) raises mean chain depth by \(+0.96\) for SFT but by \(+6.63\) for vanilla GSPO (\(6.9\times\) steeper slope); on Undead \(4{\times}4 \to 5{\times}5\) the slopes are \(+4.75\) vs \(+7.47\). Vanilla GSPO is a more aggressive depth-allocator than SFT: when the task demands more reasoning, it stretches further. This is consistent with the RL stage training on harder puzzles than SFT, learning to compose longer compute–check chains under verifiable reward.

15 Solved-vs-unsolved chain-depth significance↩︎

Table 8 reports the per-checkpoint Mann–Whitney \(U\) test for the solved-vs-unsolved chain-depth split shown in Figure 6 (b). The figure itself plots only the two means and the n-of-solved annotation; significance details are kept here for reference.

16 Training hyperparameters↩︎

All experiments use allenai/OLMo-3-7B-Instruct-SFT as the base. Both stages train LoRA adapters (rank \(64\), all-linear); stage-specific \(\alpha\) values are in Tables 9–10.

Both runs use \(n_{\text{gen}}{=}8\) on \(8\times\)B200 GPUs. All other RL hyperparameters are identical.

16.0.0.1 Verifier reward components.

The exact term is binary: it is \(1\) iff the extracted <answer> verifies against the puzzle solution and \(0\) otherwise. The completion reward is a dense board-completion score used before exact solving is common: after strict shape checks, it computes the fraction of grid cells matching the ground-truth solution, gives cells that differ from the initial puzzle board weight \(2\), and raises the weighted fraction to power \(p=3\). The format reward is an XML-structure reward in \([0,1]\): it gives \(0.25\) each for exactly one closed <reasoning> tag, exactly one closed <answer> tag, reasoning appearing before answer, and no duplicate opening tags. Thus the table’s \(0.1\cdot\)format-reward contribution is at most \(0.1\) and is intended only to stabilize response formatting.

17 Motif extraction details↩︎

17.0.0.1 Span segmentation algorithm.

Each reasoning trace is processed as follows. First, the reasoning block is extracted from <reasoning> tags if present; otherwise the full response is used. The text is then split at paragraph boundaries (double newlines) and further split within each paragraph at discourse markers detected by a multi-line, case-insensitive regex covering: numbered steps (Case N, Step N, Option N); sequential connectives (First, Second, Then); hypothetical openers (Suppose, Assume, Let’s); verification openers (Check, Verify, Confirm); self-correction markers (Wait, Actually, Hmm); redirection markers (Instead, Alternatively, Going back); conclusion markers (Therefore, Hence, Thus); and failure markers (Contradiction, This doesn’t work). Short fragments below \(80\) tokens are merged into an accumulation buffer until the threshold is reached; long fragments above \(250\) tokens are split at sentence boundaries (period, exclamation mark, or question mark followed by whitespace). A trailing buffer below \(40\) tokens is merged into the previous span rather than emitted as a standalone fragment. Token counts during merging use a character-based estimate (character count divided by \(4\)) for speed; the OLMo3 tokenizer is invoked only for the final token count on each emitted span.

17.0.0.2 Span classifier training.

We fine-tune Qwen3-1.7B [37] as a nine-way sequence classifier over the primitive vocabulary of Table 1. Training data consists of \(35\)k judge-labeled spans: \(17\)k from puzzle reasoning traces and \(18\)k from mathematics reasoning traces. Labels were produced by a jury of LLM judges (DeepSeek-V3.2 and DeepSeek-Chat [36]), iterating prompt refinements until inter-judge agreement reached \(86\%\) on a held-out span set. The resulting classifier reaches \(0.80\) macro-F1 against judge consensus on a held-out test set. All primitive sequences in Sections 4.1–5.1 use this classifier.

17.0.0.3 Sequence representation.

For each reasoning trace, our classifier of Section 4 produces a sequence of primitive labels, one per detected episode. We restrict to the \(8\) reasoning labels (plan, setup, enumerate, hypothesize, compute, check, backtrack, summarize) and treat other as a boundary token __BREAK__ that prevents adjacency across formatting/meta spans (e.g.[CHECK, OTHER, SETUP]\(\to\)[CHECK, __BREAK__, SETUP], so CHECK\(\to\)SETUP is not counted as a bigram). other is predominantly output formatting in our traces (\(91.5\%\) of other spans appear in the last \(20\%\) of the trace) and treating it as a reasoning token would inflate cross-section adjacencies.

17.0.0.4 \(k\)-gram extraction.

For each \(k \in \{2, \ldots, 15\}\) we extract all length-\(k\) subsequences via a sliding window and count motifs that do not cross a __BREAK__. We use the observed vocabulary only (full enumeration is infeasible: \(9^{10}{\approx}3.5\!\times\!10^{9}\)). Counts are length-normalised per trace as \(\mathrm{freq}(m) = \mathrm{count}(m) / (L - k + 1)\) where \(L\) is the post-break sequence length, and motifs with raw count below \(5\) (for \(k{\le}5\)) or \(10\) (for \(k{>}5\)) are filtered as noise.

17.0.0.5 Reporting.

Section 4.2 reports per-trace counts for three concrete motifs (Figure [fig:motif-examples]). For consistency with related work that does not distinguish overlapping vs non-overlapping matches, those counts use non-overlapping matches; cross-checkpoint comparisons are unaffected by the choice.

17.0.0.6 Motif categories used in Figure 2.

The Section 4.1 motif row aggregates length-\(3\) trigrams into three curated categories. Let \(w = (w_1, w_2, w_3)\) be a length-\(3\) window over the primitive sequence; let \(\mathrm{ALLOWED\_MIDDLE} = \{\)plan, setup, enumerate, compute, summarize, other\(\}\). The three category predicates are:

• Recovery: \(\mathrm{\small hypothesize} \in w\) and \(\mathrm{\small backtrack} \in w\) (order-agnostic; matches any trigram window containing both recovery primitives).

• Exploitation: \(w = (\mathrm{\small compute}, \mathrm{\small check}, \mathrm{\small compute})\) (anchored on compute; the canonical compute\(\leftrightarrow\)check alternation chain).

• Verification: \(w_1 = \mathrm{\small check}\), \(w_3 = \mathrm{\small check}\), \(w_2 \in \mathrm{ALLOWED\_MIDDLE}\) (anchored on check with a non-recovery, non-check middle slot).

The middle-slot exclusions in Verification are deliberate. Excluding check from \(w_2\) prevents long runs of consecutive checks from inflating the verification count via the sliding window without adding semantic content (a sticky-check artefact rather than a check–act–check pattern). Excluding hypothesize and backtrack from \(w_2\) keeps Verification disjoint from Recovery: a window like \((\mathrm{\small check}, \mathrm{\small hypothesize}, \mathrm{\small check})\) that would otherwise satisfy both predicates is counted under Recovery only. The Exploitation predicate is exact-match (compute-anchored), so it is also disjoint from the other two by construction. This three-way disjointness is what makes the three category bars in Figure 2 sum interpretably without double-counting. The \(k=5\) extension used elsewhere (\(\mathrm{\small check}\) \(\to\) \(x\) \(\to\) \(\mathrm{\small check}\) \(\to\) \(y\) \(\to\) \(\mathrm{\small check}\) with \(x, y \in \mathrm{ALLOWED\_MIDDLE}\) for verification, and the analogous \(\mathrm{\small check}\) \(\to\) \(\mathrm{\small compute}\) \(\to\) \(\mathrm{\small check}\) \(\to\) \(\mathrm{\small compute}\) \(\to\) \(\mathrm{\small check}\) for exploitation) preserves the same anchor and disjointness rules.

18 Metric definitions↩︎

Let \(s = (s_1, \ldots, s_L)\) denote the primitive sequence of a trace (post-other-filtering as in Appendix 17) and let \(T\) denote the trace’s response token count.

18.0.0.1 Chain depth.

We define an exploit run as a maximal contiguous block of primitives drawn from \(\mathrm{EXPLOIT} = \{\mathrm{\small compute}, \mathrm{\small check}, \mathrm{\small setup}\}\). The chain depth of a trace is the length of its longest exploit run: \[\mathrm{depth}(s) \;=\; \max_{i \le j} \big\{\, j - i + 1 \;:\; s_i, \ldots, s_j \in \mathrm{EXPLOIT} \,\big\}.\] We include setup alongside compute and check because it represents on-track implementation work (introducing notation, defining auxiliary variables) rather than strategic decomposition or branching; excluding it would shatter compute–check runs every time the model names a new variable. The ranking is robust to this choice: with \(\mathrm{EXPLOIT} = \{\mathrm{\small compute}, \mathrm{\small check}\}\) (no setup), mean chain depths on OlymMATH-Hard are Base \(0.83\), SFT \(2.27\), vanilla GSPO \(4.70\), novelty \(3.25\) — the same Base \(<\) SFT \(<\) vanilla GSPO ordering with novelty between SFT and vanilla GSPO, and the SFT\(\to\)vanilla GSPO “roughly doubles” claim (\(2.07\times\) vs \(2.09\times\) with setup). We report mean depth and the \(90\)-th percentile per checkpoint.

18.0.0.2 Mean exploit-run length.

Let \(r_1, r_2, \ldots, r_M\) be the lengths of the maximal exploit runs in \(s\) (a run is a maximal contiguous block of primitives in \(\mathrm{EXPLOIT}\)). The mean exploit-run length of a trace is \[\mathrm{mean\_run}(s) \;=\; \frac{1}{M} \sum_{m=1}^{M} r_m,\] undefined when \(M = 0\); we report mean over traces with \(M \ge 1\). Unlike chain depth, which picks the single longest run, mean run length averages all exploit runs in a trace and is therefore sensitive to whether exploit primitives are clustered into a few long runs (high mean) or scattered as many short runs (low mean). We use it alongside chain depth to ensure the depth-of-exploitation claim is not an artefact of one long run with many scattered short runs around it.

19 Alternative diversity mechanisms↩︎

Production GSPO runs at our scale take roughly \(70\) wall-clock hours on \(8\times\)B200 GPUs per configuration (\(\approx\)​560 GPU-hours), making it impractical to iterate diversity-bonus designs at full setup. We therefore ran a smaller mini ablation setup—\(2\) puzzles (Bridges and Undead) and a \(12\)k-token response budget per rollout, against the \(4\)-puzzle, \(28\)k-token production setup—roughly \(6\times\) cheaper per run. We use mini to compare diversity-bonus candidates and run only the best at production scale. Other diversity-promoting interventions applied to the same setup either collapse or fall short on stability (Table 11).

At mini scale, the loss-level entropy bonus and our novelty bonus both reach pass@32 \(=28.0\%\) at their best step—a tie on peak performance. The runs differ on training stability: the entropy-bonus run peaks at step \(5\) and degrades thereafter, while the novelty-bonus run trains stably through step \(20\). Entropy regularisation also does not target the specific hypothesize/backtrack subspace—it pulls the whole distribution toward uniformity. The entropy-advantage variants of [18] collapsed below baseline (\(17.0\%\) no-KL, \(14.0\%\) with KL).

These pilots should be read as a screening study rather than a full-scale ranking of diversity mechanisms. The mini setup omits five puzzles and uses a shorter response budget, and we did not train every alternative at production scale. The evidence supports two narrower claims: first, the frozen-reference novelty bonus was the most stable candidate in our pilot set; second, when scaled to the production setup, it improved over the vanilla-GSPO production baseline (\(36\%\) vs \(29\%\) pass@32). A full production-scale comparison against entropy and trajectory-rarity baselines remains future work.

20 Novelty-bonus algorithm↩︎

Figure 13 gives the per-step computation that augments the GSPO reward with the frozen-reference novelty bonus described in Section 5. The bonus is reward shaping: it is computed once per rollout under the frozen SFT reference policy \(\pi_{\text{ref}}\) (no gradients), then added to the last valid token’s reward before advantage computation. All other GSPO machinery (clip ratios, KL penalty, optimisation) is unchanged.

None

The bonus is applied only to correct rollouts (\(a_i = 1\)): incorrect rollouts already receive zero base reward, so adding novelty there would amount to rewarding incorrect novel reasoning. Within-group \(z\)-scoring with the \(z_{\text{clip}}{=}2\) cap bounds \(|\alpha z| \le 0.2\), i.e.at most \(20\%\) of the maximum base reward, which keeps the bonus shaping rather than dominant.

21 Within-group signal analysis↩︎

The novelty bonus only has signal if SFT perplexity varies meaningfully across correct rollouts of the same prompt: with no within-group spread, the \(z\)-score in Figure 13 is undefined and the bonus collapses to zero. We measured this directly on \(100\) puzzle prompts (\(32\) rollouts each, post-SFT) and on \(100\) OlymMATH-Hard prompts under both SFT and vanilla-GSPO checkpoints. Table [tab:novelty-signal] reports the mean within-prompt standard deviation of two candidate per-rollout signals: (i) mean per-token negative log-probability over the full response, and (ii) mean negative log-probability of the top-\(k\) most surprising tokens.

[h]
  \caption{Within-prompt standard deviation (nats/token) of per-rollout SFT-NLL
    signals, averaged over $100$ prompts. Mean NLL has near-zero spread: at our
    response lengths ($\sim$ $25$k tokens), per-token NLL is dominated by formatting
    and connective tokens that share probability across all reasoning paths.
    Restricting to the top-$k$ most surprising tokens isolates decision-critical
    positions and amplifies the within-group signal $10$--$15\times$.}
  \label{tab:novelty-signal}
  \centering
  \small
  \begin{tabular}{lccc}
    \toprule
    Signal & SFT puzzles & SFT math & Vanilla GSPO math \\
    \midrule
    Mean NLL (full response) & $0.024$ & $0.024$ & $0.024$ \\
    Top-$10$  NLL            & $0.18$  & $0.16$  & $0.16$  \\
    Top-$100$ NLL            & $0.276$ & $0.260$ & $0.245$ \\
    Top-$200$ NLL            & $0.27$  & $0.26$  & $0.24$  \\
    \bottomrule
  \end{tabular}

At a \(z\)-score cap of \(2\) and \(\alpha{=}0.1\), the mean-NLL signal would yield bonus magnitudes around \(\pm 2 \times 0.024 \times 0.1 \approx 0.005\) on a base reward of order \(1\)—below noise. The top-\(100\) signal yields \(\approx 0.05\), which is where the bonus becomes a meaningful fraction of the base reward. We chose \(k{=}100\) because the within-group std saturates between \(k{=}100\) and \(k{=}200\) (no further amplification), and smaller \(k\) becomes noisier across prompts of varying length.

22 Per-problem recovery split on OlymMATH-Hard↩︎

Section 5.1 focuses on the two diagnostic subsets for the selectivity claim: problems solved by both vanilla GSPO and the novelty-bonus variant, and problems solved only by the novelty-bonus variant. The full per-problem accounting over the \(100\) OlymMATH-Hard items is:

The primitive-count split shows the same selectivity pattern as the motif analysis in the body. On the shared problems, vanilla GSPO uses recovery primitives at a low rate (HYP \(0.63\), BTK \(0.08\) per trace), and the novelty bonus is higher but still modest (HYP \(1.34\), BTK \(0.38\)). On unique-to-novelty problems, vanilla GSPO remains flat (HYP \(0.63\), BTK \(0.10\)), while the novelty bonus rises (HYP \(1.51\), BTK \(0.43\)); restricting to the actually solved novelty-bonus traces on these problems raises the rates further (HYP \(2.25\), BTK \(0.46\)). Thus the recovered primitives are not deployed uniformly across all problems: they concentrate most strongly on the solved traces where vanilla GSPO fails.

23 Paired CoT inspection on OlymMATH-Hard↩︎

This appendix reproduces four worked OlymMATH-Hard rollouts to illustrate the mechanisms claimed quantitatively in the body. Part A (§4.2) pairs an SFT rollout against a vanilla-GSPO rollout where both are correct, to show that vanilla GSPO improves execution without changing the set of approaches. Part B (§5.1) pairs a vanilla-GSPO rollout that fails against a novelty-bonus rollout that succeeds, on problems vanilla GSPO never solves, to show that the novelty bonus’s gain comes from in-trace recovery rather than from new mathematical machinery. Throughout, we score every rollout with math_verify, consistent with the scoring used across all main-paper evaluations.

Part A: Depth of execution (§4.2)↩︎

We pair, for the same problem, an SFT rollout and a vanilla-GSPO rollout that are both correct and share the same opening primitives. We pick two problems whose chain depths fall near each checkpoint’s mean (\(\bar d \approx 3.2\) for SFT, \(\bar d \approx 6.6\) for vanilla GSPO).

In both cases the SFT and GSPO rollouts converge on the same algebraic reformulation in the same primitive sequence: single-variable substitution yielding a quadratic, then a vertex/endpoint range argument. They differ in how much compute–verify alternation follows: GSPO continues for one to three additional compute\(\to\)check cycles—numerical spot checks, boundary tests, sign discussion—before stopping, while SFT terminates after one verification pass. No new approach appears in the GSPO trace that is absent from the SFT trace. Figure 14 shows the primitive-sequence strips for both pairs; the longest compute/check run (chain depth) is annotated.

23.0.0.1 Problem 1 (Algebra, trigonometric optimisation).

Let \(f(x) = \sin^4 \omega x - \sin \omega x \cos \omega x + \cos^4 \omega x\) with \(\omega > 0\). If there exist \(a, b \in [0, \pi]\) with \(f(a) + f(b) = 9/4\), find the minimum \(\omega\). Gold answer: \(\omega = 7/12\). SFT trace depth \(2\); vanilla-GSPO trace depth \(6\).

SFT (chain depth 2):
[COMPUTE] Use \(\sin^4\theta + \cos^4\theta = 1 - 2\sin^2\theta\cos^2\theta\). Let \(t = \sin\omega x \cos\omega x\), so \(f(x) = 1 - 2t^2 - t\). With \(t = \tfrac12 \sin(2\omega x)\) and \(y = \sin(2\omega x) \in [-1, 1]\) rewrite \(f(y) = 1 - \tfrac12 y^2 - \tfrac12 y\). Critical point at \(y = -\tfrac12\), giving \(f = 11/8\); endpoints \(f(\pm 1) = 0\).
[CHECK] So \(\max f = 11/8\) and \(2 \cdot 11/8 = 11/4 \ge 9/4\), hence \(f(a)+f(b)=9/4\) is achievable. Need \(\sin 2\omega x = -\tfrac12\), i.e.\(2\omega x \in \{7\pi/6, 11\pi/6\} \pmod{2\pi}\). For \(x \in [0, \pi]\) this requires \(2\omega\pi \ge 7\pi/6\), hence \(\omega \ge 7/12\). Minimum \(\boxed{7/12}\).

Vanilla GSPO (chain depth 6):
[COMPUTE] Same substitution \(t = \sin\omega x \cos\omega x\); same quadratic \(g(t) = -2t^2 - t + 1\). Vertex at \(t = -1/4\), \(g(-1/4) = 9/8\); endpoints \(g(\pm 1/2) \in \{0, 1\}\). Range of \(g\) on \([-1/2, 1/2]\) is \([0, 9/8]\).
[CHECK] To hit \(f(a)+f(b)=9/4\) the only option is \(g(t_a) = g(t_b) = 9/8\), hence \(t_a = t_b = -1/4\). Translating: \(\sin 2\omega a = \sin 2\omega b = -\tfrac12\), with \(a, b \in [0, \pi]\) giving \(\omega \ge 7/12\).
[COMPUTE] Solve \(a = 7\pi/(12\omega)\) and \(a = 11\pi/(12\omega)\) in \([0, \pi]\); the \(11\pi/6\) solution forces \(\omega \ge 11/12\), the \(7\pi/6\) solution only \(\omega \ge 7/12\). Tentative minimum \(\omega = 11/12\) […]
[CHECK] Re-derive at \(b = \pi\), \(\omega = 11/12\): first attempt gives \(t_b = 1/4\) (“Wait, this contradicts earlier calculation. Let me recheck.”). Recompute: \(\sin(2\omega b) = \sin(11\pi/6) = -\sin(5\pi/6) = -\tfrac12\), so \(t_b = -1/4\) after all, \(f(b) = 9/8\).
[COMPUTE] Re-examine the smaller bound \(\omega = 7/12\): at this \(\omega\), \(a = b = \pi\) gives \(2\omega a = 7\pi/6\), \(\sin = -\tfrac12\), \(t = -1/4\), \(f(\pi) = 9/8\), so \(f(a)+f(b) = 9/4\). The pair is allowed because the problem only asks for existence of \(a, b \in [0, \pi]\).
[CHECK / ENUMERATE / CHECK / COMPUTE / CHECK / SUMMARIZE] Test \(\omega = 1/2\) (below \(7/12\)): no \(x \in [0, \pi]\) satisfies \(\sin 2\omega x = -\tfrac12\), confirming the bound is tight from below. Conclude \(\boxed{7/12}\).

Both checkpoints reduce \(f\) to a single-variable quadratic and identify the vertex condition \(\sin 2\omega x = -\tfrac12\). SFT stops after one CHECK pass and asserts the answer; GSPO continues with two additional compute\(\to\)check cycles—a numerical self-correction at \(b = \pi\) and an explicit lower-bound test at \(\omega = 1/2\)—before committing to \(\omega = 7/12\).

23.0.0.2 Problem 2 (Algebra, range of a rational expression).

If real \(x, y\) satisfy \(x^2 - y^2 = 4\), find the range of \(\dfrac{1}{x^2} - \dfrac{y}{x}\). Gold answer: \((-1, 1)\). SFT trace depth \(2\); vanilla-GSPO trace depth \(7\).

SFT (chain depth 2):
[SETUP] \(x^2 - y^2 = 4 \Rightarrow x \ne 0\). Let \(y = kx\); \(x^2(1 - k^2) = 4 \Rightarrow x^2 = 4/(1-k^2)\), requiring \(|k| < 1\).
[COMPUTE] \(E = 1/x^2 - y/x = (1-k^2)/4 - k = -(k^2 + 4k - 1)/4\). Let \(f(k) = k^2 + 4k - 1\). Vertex at \(k = -2\) (outside \((-1,1)\)); \(f(-1) = -4\), \(f(1) = 4\), so on \((-1, 1)\) \(f\) ranges over \((-4, 4)\).
[SUMMARIZE] Hence \(E = -f(k)/4\) has range \((-1, 1)\). (Earlier “\(\pm 1.25\)” boundary computations dismissed in one sentence as miscalculated.) \(\boxed{(-1, 1)}\).

Vanilla GSPO (chain depth 7):
[SETUP] Note \(x^2 - y^2 = (x-y)(x+y) = 4\); rewrite target as \((1 - xy)/x^3\). Let \(t = y/x \Rightarrow x^2 = 4/(1-t^2)\), forcing \(|t| < 1\).
[COMPUTE] Target \(= (1-t^2)/4 - t = (1 - 4t - t^2)/4 := f(t)\).
[CHECK] \(f'(t) = -t/2 - 1\); \(f' = 0\) at \(t = -2\), outside \((-1,1)\), so \(f\) is monotonic on \((-1,1)\). Limits: \(f(t \to 1^-) = -1\), \(f(t \to -1^+) = 1\).
[COMPUTE] \(f'(0) = -1\), so \(f\) is strictly decreasing on \((-1,1)\); image is \((-1, 1)\) open. Endpoints not attained (would require \(x^2 - y^2 = 0\)).
[CHECK] Sign of \(x\) doesn’t matter (both signs give same \(f(t)\)); \(x = 0\) impossible.
[COMPUTE] Spot-check \(t = 0\) (\(x = \pm 2\), target \(= 1/4\)); \(t = 0.5\) (target \(-0.3125\), matches \(f(0.5)\)); \(t = -0.5\) (target \(0.6875\), matches \(f(-0.5)\)).
[CHECK] Limit checks: \(t = 0.99 \Rightarrow f \approx -0.985\) (approaches \(-1\)); \(t = -0.99 \Rightarrow f \approx 1.245\). Reconcile: the supremum is approached only as \(t \to -1^+\) and never attained; numeric at \(t = -0.99\) is consistent with the decreasing-to-\(1\) picture.
[SUMMARIZE / COMPUTE] Range \(\boxed{(-1, 1)}\).

Both checkpoints adopt the substitution \(t = y/x\), derive the same quadratic \(f(t) = (1 - 4t - t^2)/4\) on \((-1, 1)\), and reach \((-1, 1)\). SFT terminates after one range argument and waves off a sketchy boundary computation as a miscalculation; GSPO continues with two additional compute\(\to\)check cycles—numerical spot-checks at \(t \in \{0, \pm 0.5\}\) followed by limit checks at \(t = \pm 0.99\)—before committing.

23.0.0.3 Selection (Part A).

Using math_verify, \(13\) OlymMATH-Hard problems have at least one correct rollout in both SFT (\(81/3200\) traces) and vanilla GSPO (\(89/3200\)). Restricting to pairs that share the first two opening primitives and place the SFT depth in \([2, 4]\) and the GSPO depth in \([5, 9]\) yields \(19\) candidate pairs across \(4\) problems. The two reproduced above were chosen for cleanest opening match, depths near each checkpoint’s mean, and algebraic-manipulation flavour, the dominant subject in the intersection. A small number-theory and combinatorics options exist (problems \(11\), \(17\), \(25\), \(26\), \(91\)) but none paired typical-depth SFT and GSPO traces under the same-opening constraint, so they are not informative for the \(\S\ref{sec:sec:rl-deepens}\) depth-of-exploitation claim.

Part B: Productive recovery (§5.1)↩︎

Section 5.1 shows the novelty bonus’s gain is concentrated on the \(16\) OlymMATH-Hard problems solved only by the novelty-bonus variant, and that its solved traces on these problems are enriched for recovery primitives (hypothesize/backtrack) that vanilla GSPO’s failures do not show. Below we reproduce the underlying text for two such problems, chosen because the recovery primitive in the parquet’s primitive_sequence aligns with a textually visible pivot. Figure 15 renders the four primitive sequences (vanilla-GSPO failures vs novelty-bonus successes on problems \(4\) and \(73\)) as colour strips so the recovery pivot is visible at a glance. The common pattern: vanilla GSPO commits to a first answer that the trace itself flags as suspect; the novelty-bonus variant performs the same opening computation, then backtracks to a different conclusion.

23.0.0.4 Problem 3 (Algebra, multivariate optimisation).

Let \(x\), \(y\), \(z\) be positive reals; find the minimum of \(f(x, y, z) = \dfrac{(2 + 5y)(3x + z)(x + 3y)(2z + 5)}{xyz}\). Gold answer: \(241 + 44\sqrt{30} \approx 481.6\). Of \(32\) vanilla-GSPO rollouts: \(6\) box \(480\) (a loose AM-GM bound), \(3\) box \(784\) (the symmetric ansatz), \(9\) truncate, the rest scattered. None match. The novelty-bonus variant’s trace \(3\) succeeds (chain depth \(2\) for the GSPO failure; novelty trace \(\approx 28{,}500\) chars with one hypothesize and one backtrack primitive at the pivot).

Vanilla GSPO failure (full short trace, \(\approx 2{,}300\) chars):
[COMPUTE] Apply AM-GM to each numerator factor independently: \(2 + 5y \ge 2\sqrt{10y}\), \(3x + z \ge 2\sqrt{3xz}\), \(x + 3y \ge 2\sqrt{3xy}\), \(2z + 5 \ge 2\sqrt{10z}\). Multiply: numerator \(\ge 16\sqrt{10y \cdot 3xz \cdot 3xy \cdot 10z} = 16 \cdot 30 xyz = 480 xyz\), hence \(f \ge 480\).
[CHECK] “To check if equality holds, we need each AM-GM step tight: \(2 = 5y \Rightarrow y = 2/5\); \(3x = z\), \(x = 3y \Rightarrow x = 6/5,~z = 18/5\); \(2z = 5 \Rightarrow z = 5/2\). This contradicts \(z = 18/5 \ne 5/2\). Therefore the conditions cannot all be satisfied simultaneously, indicating equality might not be achievable. However, the AM-GM inequalities provide a lower bound, and since the function can approach this bound by choosing values close to those that satisfy the conditions, the minimum is indeed \(480\).” \(\boxed{480}\).

Novelty-bonus recovery:
[PLAN] Approach: try AM-GM on each factor; alternatively apply Lagrange multipliers to \(\ln f\).
[HYPOTHESIZE] Try the symmetric ansatz \(x = y = z = k\). The function reduces to \(g(k) = 16(10k + 29 + 10/k)\), \(k > 0\).
[COMPUTE \(\to\) CHECK\(\times\)9] \(g'(k) = 16(10 - 10/k^2) = 0 \Rightarrow k = 1\), second derivative positive, so \(k=1\) is a local min, \(g(1) = 16 \cdot 49 = 784\). Spot-check \(g(2) = g(0.5) = 864\), both higher.
[SUMMARIZE] “But is this the global minimum? We assumed \(x = y = z\). Maybe there is a lower value when variables are not equal.”
[BACKTRACK] Drop the symmetric ansatz. Use \(F = \ln f\) and take \(\partial F/\partial x = \partial F/\partial y = \partial F/\partial z = 0\) to get the actual critical-point system. Verify \(x{=}y{=}z{=}1\) in equation (2): LHS \(= 5/7 + 3/4 = 41/28 \ne 1\), so the symmetric ansatz is not a critical point—confirming the previous answer was wrong.
[CHECK\(\to\)COMPUTE\(\to\)CHECK\(\to\)SUMMARIZE] Solve the critical-point system: it reduces to \(x^2 = yz\), \(15y^2 = 2x\), \(2z^2 = 15x\). Substituting back gives \(x = 1\), \(y = \sqrt{2/15}\), \(z = \sqrt{30}/2\). Plug in and simplify: \(f_{\min} = 241 + 44\sqrt{30}\). \(\boxed{241 + 44\sqrt{30}}\).

Both checkpoints reach for AM-GM first; vanilla GSPO multiplies the loose AM-GM bound, observes the equality conditions are inconsistent, and ships the bound anyway. Novelty starts on a stronger AM-GM (symmetric ansatz, giving \(784\)), uses the partial-derivative system to verify whether the ansatz is a critical point, finds it is not, and backtracks to a Lagrange-style system whose explicit solution is the gold answer. The backtrack primitive marks the moment Novelty abandons the ansatz; this is the exact step the GSPO trace is missing.

23.0.0.5 Problem 4 (Geometry, regular tetrahedron).

For a regular tetrahedron \(ABCD\), \(M, N\) are midpoints of edges \(AB, AC\) and \(P, Q\) are centroids of faces \(ACD, ABD\). Find the angle between lines \(MP\) and \(NQ\). Gold answer: \(\arccos(7/18)\). Of \(32\) vanilla-GSPO rollouts: \(14\) box \(\arccos(-7/18)\) (the angle between the vectors, not the lines), \(3\) box \(90^\circ\), \(2\) truncate, the rest miscellaneous. None match. The novelty-bonus variant’s trace \(25\) succeeds with three backtrack primitives.

Vanilla GSPO failure (full short trace, \(\approx 2{,}900\) chars, no recovery primitives):
[SETUP\(\to\)COMPUTE] Place \(A = (1,1,1)\), \(B = (1,-1,-1)\), \(C = (-1,-1,1)\), \(D = (-1,1,-1)\) (regular tetrahedron, edge \(\sqrt{8}\)). Midpoints \(M = (1, 0, 0)\), \(N = (0, 0, 1)\). Centroids \(P = (-1/3, 1/3, 1/3)\), \(Q = (1/3, 1/3, -1/3)\).
[COMPUTE] \(\overrightarrow{MP} = (-4/3, 1/3, 1/3)\), \(\overrightarrow{NQ} = (1/3, 1/3, -4/3)\). \(\overrightarrow{MP} \cdot \overrightarrow{NQ} = -7/9\), \(|\overrightarrow{MP}| = |\overrightarrow{NQ}| = \sqrt{2}\).
[CHECK / final commit] \(\cos\theta = (-7/9)/2 = -7/18\). \(\boxed{\arccos(-7/18)}\).

Novelty-bonus recovery (three backtrack primitives, two hypothesize):
[SETUP\(\to\)COMPUTE\(\to\)CHECK] Same coordinate system, same vectors, same \(\cos\theta = -7/18\). Tentatively \(\theta = \arccos(-7/18)\).
[COMPUTE / SETUP / CHECK] “However, the angle between two lines is defined as the smallest angle between their direction vectors. So if the calculated angle is obtuse, the actual angle between the lines is the acute supplement […] \(\cos(180^\circ - \theta) = +7/18\), so the angle between the lines is \(\arccos(7/18)\). But the problem says ‘the angle between \(MP\) and \(NQ\)’. Usually they mean the angle between the vectors as directed, but sometimes the smallest. Need to check.”
[BACKTRACK #1] “Maybe I made a mistake in the coordinate scaling. Let me try a different system where the tetrahedron has edge length \(2\).” \(\to\) Resolve with vertices at \((1/\sqrt{2}, \pm, \pm, \pm)\). Same vectors up to uniform scale; \(\cos\theta = -7/18\) again.
[BACKTRACK #2] Re-examine the problem statement: “\(P\) and \(Q\) are the centroids of faces \(ACD\) and \(ABD\)”—yes, what we used. \(\to\) Recompute in original coordinates: still \(-7/9\).
[BACKTRACK #3] Try a third coordinate system (edge length \(1\) with \(A\) at the origin). After one arithmetic error, confirm again that the vector angle has cosine \(-7/18\).
[CHECK\(\to\)HYPOTHESIZE\(\to\)COMPUTE\(\to\)CHECK / final commit] “In a regular tetrahedron, take two edges meeting at a vertex; the angle between them is \(\arccos(1/3) \approx 54.7^\circ\), which is acute. So the convention takes the acute angle. Probably here also.” \(\boxed{\arccos(7/18)}\).

Both checkpoints compute the same vectors, the same dot product, and the same intermediate \(\cos\theta = -7/18\). Vanilla GSPO commits to the obtuse vector angle the next segment. Novelty spends most of its length checking whether \(-7/18\) is “really” the answer—testing two alternative coordinate systems (backtracks #1 and #3), re-examining the problem statement (backtrack #2), and finally invoking the “angle-between-lines-is-the-smallest-angle” convention. Three backtrack primitives mark three attempts to reconcile the numeric answer with the geometric question; the last produces the convention switch that yields the gold answer. The recovery here is on interpretation, not technique—the underlying coordinate-and-dot-product method is unchanged between the two traces.

23.0.0.6 Selection (Part B).

The current per-problem split contains \(16\) OlymMATH-Hard problems solved only by the novelty-bonus variant (Appendix 22). Problems \(4\) and \(73\) were chosen from this set because the recovery primitive in the parquet aligns with a clean textual pivot (technique switch on \(4\); convention switch on \(73\)) and the vanilla-GSPO failure traces are short enough to quote in full. We deliberately avoid using novelty-only winners whose successful traces contain no visible recovery primitive: in those cases the win comes from a clean first attempt under diverse seeds rather than from a recovery mechanism, and they would be misleading examples for the §5.1 claim.

References↩︎

---

1. Correspondence: mayugmaniparambil@gmail.com↩︎