1 Introduction↩︎

Reinforcement learning from verifiable rewards has become the standard tool for sharpening the reasoning of large language models (LLMs) on tasks such as mathematics and code [1], [2]. The dominant recipe is on-policy and outcome-based: sample a group of rollouts from the current policy, score each by whether its final answer is correct, and update with group-relative advantages, as in GRPO [2]. The reward is attached to the answer; the chain of thought that produced it is never graded. On problems the model can already partly solve this is benign, but on hard problems it creates a strong incentive to write more: a longer chain affords more chances to stumble onto the right answer, so additional text is the cheapest way to raise expected reward. The trace inflates with hedging and backtracking (“hmm,” “wait,” “let me reconsider”), and accuracy grows far slower than length (accuracy rises only modestly while length inflates by an order of magnitude [3]). The model ends up producing more reasoning per problem without producing better reasoning per token.

The obvious fix, grading the reasoning directly, is intractable. There is no ground truth for “good thinking”: no label says which step was the insight and which was filler, and a learned process reward model is expensive, brittle, and itself unverifiable [4]. So the trace stays unscored, and the length pathology persists. We ask instead: can a second model supply the missing signal? If a different policy attempts the same problem and we reward each model for out-reasoning the other, then the trace is graded implicitly by the rival, without any process labels. A step that leads to a win is reinforced; filler that the opponent exploits is punished.

The grader must be a different model, and the game must be competitive. Post-training RL is, by construction, self-improvement: the policy is optimized on signal it generates itself, which reinforces the very blind spots that created its errors: a model auditing its own work tends to plateau [5]. A second model with different failure modes breaks this closed loop. But merely having two models is not enough. Naively averaging or letting models agree, as in self-consistency and mixture-of-agents [6], [7], tends to regress toward consensus and can dilute quality when the models differ in strength [8], while two near-identical models gain little from one another. The combination that works is a pair of comparable strength hypothesized to have different blind spots, placed under competitive pressure: each model must not only solve the problem but beat a peer that sees its work, so the objective rewards out-reasoning the peer rather than agreeing with it. Under this pressure the pair co-improves: each win raises the bar for the other on the next step, and both models end up past the ceiling where a single self-graded policy plateaus.

We instantiate this as Agon (Greek agōn, contest). Two distinct policies are trained head-to-head: a drafter produces candidate solutions; a challenger reads an opponent’s worked solution and is rewarded for solving correctly and beating it; roles rotate every step so both models learn both roles, and at inference the pair deploys the same way, one model drafting and the other answering after reading the draft. In principle the recipe applies to any pair of comparably strong models; for efficiency we realize the pair as two low-rank adapters over one frozen base (\({\sim}2\%\) memory overhead, not \(2\times\)), kept divergent by different initialization and update streams. In this work the exchange happens in text: we first establish that the cross-model signal is useful before optimizing the channel (latent-space exchange; 6). We test two claims under a matched generation budget: that cross-model information exchange raises accuracy (the exchange claim), and that adversarial pressure raises it further than cooperation (the competition claim), using a \(2\times2\) design that crosses competition with exchange (one cell of which is provably inert under group-relative normalization; 4.4), plus a single-model self-refinement control, a zero-training orchestration control, and a budget-matched GRPO self-cascade control.

1.0.0.1 Contributions.

• We reframe reasoning quality as an implicit reward that a competing peer model can supply, turning the unlabeled “good thinking” problem into a head-to-head game played during RL (4).

• We propose Agon: a recipe for jointly training two distinct policies against each other with competitive GRPO, via draft-and-challenge rollouts, per-step role rotation, and a competitive reward (correctness plus a conversion bonus for out-solving the opponent). It runs on a stock GRPO trainer with no changes to the optimization core; a shared-base dual-adapter instantiation makes the pair nearly as cheap as a single model (4).

• We lay out a \(2\times2\) matrix (competition \(\times\) information exchange) with self-refinement and zero-training controls (5). One cell, competition without exchange, is inert by construction under group-relative normalization (4.4), so the competition claim rests on the adversarial-vs-cooperative comparison; the fully isolated cell (per-rollout hidden drafts) is left for future work.

• Under a matched generation budget on hard math, competition \(+\) exchange beats cooperation, self-refinement, and vanilla GRPO on pass@1, with the largest gains on the smallest models. At the same doubled inference budget, the trained cascade nearly doubles the accuracy of an untrained Mixture-of-Agents pass over the same base, which itself barely improves on GRPO.

2 Related Work↩︎

2.0.0.1 Outcome-only RL and the length pathology.

GRPO [2] and the R1 recipe [1] reward only the verified final answer. A well-documented side effect is uncontrolled growth of trace length (“overthinking”), with accuracy rising far more slowly than tokens [3], [9]. Remedies act on the symptom: explicit length penalties or length-controlled objectives [3], and dynamic sampling that discards degenerate all-correct/all-wrong groups [10]. None grades the content of the trace, because step-level labels do not exist and learned process reward models are costly and themselves unverifiable [4]. Agon leaves the verifier untouched and instead derives a content signal from a competing peer.

2.0.0.2 Self-play and self-improvement.

A model can improve against versions of itself: SPIN [11] plays the current policy against its own past generations; Self-Rewarding LMs [12] use the model as its own judge; and self-play reasoners such as Absolute Zero [13] and R-Zero [14] co-evolve a proposer and a solver instantiated from one model. These methods are powerful but share a structural ceiling: a policy optimized on its own signal reinforces the blind spots that caused its errors, and self-correction without external feedback is unreliable [5]. Agon differs by training two distinct policies with different initialization and update streams, so the grader is a genuinely different agent rather than a copy of the graded. Multiagent finetuning [15] also diversifies several models from one base via distinct data streams, but toward cooperative ensembling; Agon maintains divergence for adversarial grading, where the distinct rival is what supplies the reward.

2.0.0.3 Multi-model collaboration, debate, and ensembles.

Combining several models (self-consistency [6], multi-agent debate [16], and Mixture-of-Agents [7]) can improve robustness, but the aggregation is cooperative: it pushes toward consensus, which dilutes quality when members differ in strength [8], and yields little when they are near-identical. These are inference-time procedures over frozen models. Agon instead applies cross-model pressure during RL and makes it competitive: a model is rewarded for out-solving a peer that has seen its work, so the objective pays for beating the peer, not for matching it. We use a cooperative variant only as an ablation to isolate the value of competition, and include a two-agent MoA layer as a zero-training control; it recovers roughly one-eighth of Agon’s gain over GRPO (3). Closest in spirit is in-context self-refinement [17], where a model revises its own draft; Agon differs in that the draft comes from a competing, distinct peer and is selected by outcome under RL, not by self-feedback.

2.0.0.4 Adversarial and generator–verifier training.

Discriminator/verifier signals have long shaped generators, from GANs [18] to prover–verifier games that improve the legibility of LLM solutions [19] and adversarial games over language [20]. Debate [21] first framed a judged competition between two agents as a scalable proxy for supervising reasoning itself. Agon is symmetric and rotational rather than role-fixed: both models alternate between drafting and challenging, so neither degenerates into a pure critic, and both improve as solvers; the judge is a ground-truth verifier rather than a learned or human one. The eventual move from text to latent-space exchange connects to work on inter-model communication in representation space [22], which we leave as future work.

3 Preliminaries: GRPO and the Unscored Trace↩︎

For a problem \(x\) with a verifier \(r(x,y)\in\{0,1\}\) that checks only the final answer of a completion \(y=(\text{trace},\text{answer})\), GRPO [2] samples a group of \(G\) rollouts \(y_1,\dots,y_G\sim\pi_\theta(\cdot\mid x)\) and forms the group-relative advantage \[A_i \;=\; \frac{r(x,y_i)-\mu}{\sigma},\qquad \mu=\tfrac{1}{G}\textstyle\sum_j r(x,y_j),\quad \sigma=\mathrm{std}_j\,r(x,y_j), \label{eq:grpo-adv}\tag{1}\] updating \(\pi_\theta\) to raise the log-probability of above-average rollouts, with the usual KL regularization toward a reference (the base) omitted here for brevity; groups with zero reward variance (all-correct or all-wrong, where \(\sigma=0\)) carry no signal and are dropped. The signal is entirely a function of the answer: two completions with the same answer receive the same advantage regardless of how they reasoned.

3.0.0.1 The unscored trace.

On hard problems the per-problem solve rate \(\mathbb{E}_{y\sim\pi_\theta}[r(x,y)]\) is small, and the policy raises expected reward most cheaply by increasing the number of distinct attempts inside one trace: each restart, case split, or “let me reconsider” is another chance at the answer. Because the trace is never scored, this padding is not penalized and is in fact reinforced through any correct rollout that contains it. The result is the empirically observed regime where length grows much faster than accuracy [3], [9]; in our own runs, GRPO training grows the mean trace from the zero-shot \(6.1\)k to \(8.1\)k tokens for \(+7\) pp accuracy (3). The policy increases the quantity of reasoning while its density, the useful signal per token, stays flat. The bottleneck, then, is not quantity. The prevailing axes of scale (data, parameters, context, and sampled tokens) all add more reasoning; Agon instead targets a different axis, signal per token, by making low-density traces lose the competition.

3.0.0.2 The missing grader.

Closing this gap requires a reward that distinguishes a concise, sound trace from a long, lucky one, i.e.a signal on the reasoning itself. Verifiers do not provide it, and process reward models require labels we do not have [4]. Agon obtains the signal relationally: a peer policy attempts the same \(x\), reads the other’s worked solution (the post-reasoning summary, not the raw trace; 4.3), and the reward depends on which model out-reasons the other. A trace whose flaws the opponent exploits (a sign error, a dead end it avoids) loses; a trace that leads to a win is reinforced. The grader is thus a second model, and, because both models are optimized, the bar rises over training. We formalize this next.

4 Method: Agon↩︎

Agon trains two policies, \(A\) and \(B\), head-to-head on the same problems. The design has four parts: a draft-and-challenge rollout that routes one model’s reasoning into the other’s context (4.3); a competitive reward that grades the challenger against the drafter (4.4); a compute-parity accounting so all comparisons are fair (4.5); and an efficient instantiation of the two policies as divergent adapters over a shared base (4.2).

4.1 Why a second model↩︎

Post-training RL is self-improvement: the gradient in 1 is computed from rollouts the policy itself produced, so it amplifies whatever the policy already does, including the systematic errors that define its blind spots. A model checking its own work inherits the bias that produced the mistake, which is why unaided self-correction often fails to add signal [5]; we expect the same closed loop to cap self-play, however many rounds it runs (illustrated in 3, left). A different model is outside this loop: a peer hypothesized to have different failure modes audits what the policy cannot see in itself. In principle the peer could come from a different model family: the strongest models, open ones included, now cluster within a few points of one another on standard reasoning benchmarks [23], so pairs of matched strength are easy to find; whether such pairs also differ enough in behavior is an empirical question we leave to future work. Our instantiation instead engineers the divergence structurally from one base (4.2).

Pairing helps only under two conditions. The models must be of comparable strength (otherwise, we expect, the weaker simply imitates the stronger and the game collapses into distillation; we do not ablate the strength gap), and they must have different failure modes, so that one can catch what the other misses (we make this concrete in 4.2, 4). We therefore keep the pair matched in capacity but divergent in behavior, and we make the interaction competitive so the objective rewards beating the peer rather than agreeing with it.

4.2 Instantiating the two policies↩︎

Agon itself only requires two policies of matched strength and different blind spots; how they are parameterized is an implementation choice. We instantiate both as LoRA adapters [24] over a single frozen base model: \(\pi_A=\mathrm{base}\!+\!\Delta_A\) and \(\pi_B=\mathrm{base}\!+\!\Delta_B\). The second policy costs one extra rank-16 adapter (\({\sim}10\)M parameters, \({\sim}2\%\) of the base, plus its optimizer state) rather than a full second model, and the shared base keeps the two policies in compatible representation spaces, useful for the eventual latent-space exchange (6). Two models alone are not enough, though: the useful regime is the one motivated in 4.1, equal strength with hypothesized different blind spots (4). We maintain divergence by (i) different initialization (\(B\) starts with small Gaussian noise around the zero matrix; \(A\) uses the standard LoRA zero-init) and (ii) different update streams induced by role rotation (4.3): on any given problem one adapter drafts while the other challenges, so the two never receive the same gradient. All reported numbers use the same base checkpoint for both adapters. We stress that this instantiation is minimally divergent: at initialization the two policies differ only by adapter noise, so any complementarity must emerge from the diverging update streams during training rather than from pre-existing blind-spot differences. Different blind spots are therefore a hypothesis about the trained pair, not a measured property. The gap between cooperative exchange and single-adapter self-refinement (46 vs 32; 3) shows that training to read a peer’s draft beats training to read one’s own, but the comparison varies more than one factor (trainable adapter capacity, whose draft is read, the refiner prompt), so it does not isolate blind-spot divergence as the cause.

4.3 The draft-and-challenge step↩︎

Each optimizer step (2, 7) designates a drafter and a challenger. The drafter generates \(N\) rollouts \(\{a_i\}\sim\pi_{\text{draft}}(\cdot\mid x)\) from the plain problem prompt and receives a vanilla GRPO update on them (correctness and format only, no opponent term); this standalone stream trains each adapter to solve from scratch, which is the first stage of deployment (4.5). The challenger then conditions on its paired opponent’s solution summary: the answer section the model writes after its private reasoning block, which recaps the derivation. The raw thinking trace is dropped, since the summary carries the distilled solution at a fraction of the tokens (cheaper to prefill, and free of the exploration noise that dominates long traces), and the final answer is withheld as a precaution against copying (5), though a worked summary often implies its answer anyway. The summary is inserted as an attempt that “may or may not be correct,” and the challenger generates one rollout per pair, \(b_i\sim\pi_{\text{chal}}(\cdot\mid x, a_i)\), forming a group of \(N\). The challenger receives the competitive gradient; each stream’s GRPO advantage is computed within its own group of \(N\). Because each \(b_i\) sees a different opponent \(a_i\), the group spans varied opponent contexts for the same \(x\); this is what gives the conversion bonus within-group variance. Roles rotate every step: on even steps \(A\) drafts and \(B\) challenges, on odd steps the reverse. Without rotation, each adapter receives only one kind of gradient, one only ever solving from scratch and the other only ever solving against an opponent; the pair still trains, but ends up weaker (52 vs 61 pass@1; 7). With rotation, both adapters alternate between the two streams every step and co-improve. The challenger prompt instructs the model not to trust or paraphrase the opponent but to verify it and solve independently, winning by being correct when the opponent is wrong; the prompt is identical across the coop and adv reward stacks (4.4), so they differ only in the reward.

4.4 Reward: correctness plus a conversion bonus↩︎

Let \(c(y)=r(x,y)\in\{0,1\}\) be correctness. We report two reward stacks, applied to the challenger rollout \(b_i\) given its paired drafter solution \(a_i\): \[\begin{align} \boldsymbol{coop:}\quad & R(b_i) \;=\; 2\,c(b_i) \;+\; \lambda\,\phi(b_i), \tag{2}\\ \boldsymbol{adv:}\quad & R(b_i) \;=\; 2\,c(b_i) \;+\; \underbrace{c(b_i)\,\big(1-c(a_i)\big)}_{\text{conversion bonus}} \;+\; \lambda\,\phi(b_i), \tag{3} \end{align}\] where \(\phi(\cdot)\) is a format term (well-formed <think>/<answer> structure) with weight \(\lambda=0.5\) (25% of the correctness term), identical across stacks so it introduces no bias between them. In coop the opponent appears only in the context, not in the reward, which isolates the value of information exchange (the exchange claim). In adv the conversion bonus \(c(b_i)\,(1-c(a_i))\) raises the effective correctness weight from \(2\) to \(3\) exactly on the paired opponent attempts that failed: solving where the opponent could not is worth more, which is the competitive pressure.

The form matters. A naïve margin \(c(b_i)-c(a_i)\) works poorly: since \(c(a_i)\) is action-independent (fixed by the pairing \(a_i\), not by the challenger’s rollout \(b_i\)), its subtractive term contributes no expected policy gradient (exactly in the unnormalized objective; up to the nonlinearity of group standardization after it): it acts as a per-sample baseline, shifting each reward by the drafter’s outcome without adding any competitive direction. The tested margin stack, \(R=2c(b_i)+\big(c(b_i)-c(a_i)\big)=3c(b_i)-c(a_i)\), makes this concrete: the scalar \(3\) cancels under group-standard-deviation normalization and the \(-c(a_i)\) term is action-independent (zero expected policy gradient), so in expectation the margin points in the same direction as the cooperative gradient. The ablation is consistent with this prediction (\(49\) vs coop \(46\)), though the \(3\) pp difference lies within binomial sampling noise. The conversion bonus instead modulates the correctness weight by opponent difficulty: because it multiplies the action-dependent \(c(b_i)\), the reweighting shifts the gradient itself, and, given the within-group spread of opponent contexts \(\{a_i\}\), tilts it toward the opponent attempts that failed. We confirm this empirically (7): the margin form barely improves on cooperation, whereas the conversion bonus clearly does. More generally, any reward that is affine in \(c(b_i)\) with group-constant coefficients (e.g., a shared opponent draft for the entire challenger group, making \(c(a)\) identical for all \(N\) rollouts) normalizes to exactly the advantage of the conversion-free reward in the same contexts under group-relative standardization; the conversion term then contributes zero expected policy gradient. Within-group variation in opponent difficulty is what makes the competitive term trainable. In the shared-opponent ablation we therefore fix one opponent draft per problem, hidden from the challenger’s context and used only for the reward (so the contexts are plain prompts, as in vanilla GRPO); the observed result (32) is consistent with the theoretical prediction that a group-constant conversion bonus adds no directional signal. This departs from the standard GRPO assumption that all rollouts in a group share the same prompt; here each challenger rollout conditions on its own opponent context. The resulting credit assignment is not merely noisier but systematically confounded: a rollout’s success probability depends on the quality of the draft it happened to draw (an action-independent draw), so group standardization mixes context difficulty with policy quality, and the conversion bonus, which pays exactly in the harder contexts, partially offsets that difficulty penalty. This admits an alternative reading of the adversarial gain: rather than “competitive pressure” in a game-theoretic sense, the bonus acts as difficulty-weighted reward shaping that upweights correct completions in hard contexts. Our experiments do not distinguish these two interpretations; we use the competitive framing for its constructive role in the design and leave the disentanglement to future work.

The resulting reward ladder (adv) is \(R-\lambda\phi\in\{0,2,3\}\). The drafter’s standalone stream has no opponent, so 2 applies as-is.

As an optional density lever, studied separately in 5.6 and not part of the headline recipe, we add a length tiebreak that fires only when both solutions are correct: \(R(b_i)\mathrel{+{=}}\gamma\,c(b_i)\,c(a_i)\,\mathbf{1}\!\left[\,|b_i|<|a_i|\,\right]\) with weight \(\gamma\), rewarding the challenger for matching the opponent’s correctness in fewer tokens. Because the term requires \(c(b_i)=1\), it never directly penalizes correctness, only breaking ties among already-correct traces, so accuracy stays essentially flat while length drops, converting the emergent shortening (5.5) into a directly tunable target.

4.5 Compute parity↩︎

Per step Agon generates \(N\) drafter rollouts plus \(N\) challenger rollouts (\(2N\)), versus \(N\) for vanilla GRPO. Both adapters are updated every step, the drafter on its standalone group and the challenger on its opponent-conditioned group; rotation swaps which adapter gets which stream. Our primary invariant is the generation budget: every method generates \(2N\) rollouts per training problem (vanilla GRPO gets \(2N\) instead of its default \(N\)) under the same completion-length cap and the same single pass over the training problems. Realized token counts then differ only through learned trace length, bounded for every method by the same cap; we do not audit total generated tokens (the drafter-stage length is not reported), but with the rollout count and length cap equalized, the gains are not attributable to a larger generation budget. The challenger additionally prefills the opponent summary, which is extra prefill compute we do not equalize (the summary is short relative to a full trace, is amortized cheaply against generation, and is absent at single-adapter inference). At inference Agon deploys exactly as it trains: a two-stage cascade in which one adapter drafts from the plain prompt and the other reads the draft’s summary and produces the final answer. This is two sequential generation passes, a doubled budget relative to a single-pass baseline, so 3 includes two-pass controls at the same budget (cross-refinement, self-refinement). Reported trace lengths refer to the final (challenger) stage. Both cascade directions (\(A{\to}B\) and \(B{\to}A\)) are evaluated and the better is reported, selected post hoc on the held-out set.

5 Experiments↩︎

5.0.0.1 Setup.

We train Qwen3 models [25] (primarily 0.6B; 1.7B/4B for scaling) on the hard split of DeepMath-103K [26] (difficulty 8, binary-answer questions removed), with a held-out set of \(300\) problems. We use group size \(G=N=8\) for both streams, LoRA rank \(16\), learning rate \(5\times10^{-5}\) (linear decay, \(10\%\) warmup), one training epoch over the \(3{,}000\) training problems, format weight \(\lambda=0.5\), length tiebreak \(\gamma=0.5\) (used in 5.6 only), and a standard format reward (well-formed <think>/<answer>) as \(\phi\). Generation uses vLLM [27] and training uses TRL [28]. The primary metric is pass@1 on the DeepMath held-out set; we additionally sanity-check on GSM8K [29] and MATH-500 [30] (2). For the second-domain study (5.3) we train on the easy-difficulty subset of CodeContests [31] with a unit-test verifier (\(r=1\) iff all tests pass), under the same protocol and budget. Evaluation protocol: sampled decoding (temperature \(0.6\), top-\(p\) \(0.95\)), one rollout per stage, generation budget matched to training (\(15\)k completion tokens per stage), the answer parsed from the <answer> block; the \(300\) held-out problems are difficulty-8 only with binary-answer questions removed, split with a fixed seed before any training. Pair-trained methods are evaluated as the two-stage cascade (4.5) in both directions, reporting the better direction; mean trace length reports the final-stage completion. Reported pass@1 values include 95% Clopper–Pearson binomial confidence intervals over the 300-problem held-out set (4).

5.0.0.2 The \(2\times2\) design.

We cross two binary factors, competition and information exchange, giving four cells plus three controls (1): self-refinement, an untrained MoA at inference, and a budget-matched GRPO two-pass self-cascade. One caveat is structural: the “competition without exchange” cell uses a shared opponent draft, hidden from the challenger and used only for the reward, which by the analysis in 4.4 makes the conversion bonus group-constant and hence gradient-free, so this cell is theoretically equivalent to vanilla GRPO and serves as a consistency check rather than a true isolation of competition. Consequently, the exchange claim (exchange helps) is carried by the GRPO-vs-cooperative comparison, and the competition claim (competition helps beyond exchange) by the cooperative-vs-adversarial comparison, holding everything else fixed at a matched generation budget; a competition-without-visible-exchange cell with per-rollout hidden drafts is left for future work.

Controls: self-refinement (model sees its own prior attempt); untrained MoA at inference (the base model’s two independent

samples critique each other, no training); GRPO two-pass self-cascade (a trained GRPO model refines its own first pass, matching the cascade’s inference budget).

5.1 Main results↩︎

3 reports the main comparison on Qwen3-0.6B. All deltas below are read from the pass@1 column. The trained pair methods and the refinement controls are two-pass at inference; vanilla GRPO and zero-shot are single-pass (see the footnote of 3). Information exchange alone (cooperative exchange: \(46\)) already improves over vanilla GRPO (\(30\)) and over self-refinement (\(32\)), supporting the exchange claim; adding competition (Agon: \(61\)) improves further over cooperation, supporting the competition claim. The untrained MoA control is a two-agent Mixture-of-Agents layer at inference [7]: each model reads the other’s attempt and refines its own. It lifts GRPO by \(+4\) pp (\(30\!\to\!34\)); Agon lifts it by \(+31\) pp, roughly eight times the gain of untrained aggregation. We note that the two levers contribute comparably: information exchange adds \(+16\) pp (GRPO \(\to\) cooperative) and competition a further \(+15\) pp (cooperative \(\to\) Agon). The dual-adapter architecture alone contributes little: the competitive-shared-opponent variant sits at \(32\), barely above single-model GRPO at \(30\), so the gain from exchange is attributable to the shared context, not to going from one model to two.

Each method is a single training run; the confidence intervals in 4 cover sampling variation over the 300 held-out problems (\(\pm{\sim}5.5\) pp) but not run-to-run training variance. Both headline deltas (exchange: \(+16\) pp; competition: \(+15\) pp) exceed the width of these intervals. The Agon cascade reaches roughly twice vanilla GRPO’s held-out pass@1 while shortening final-stage traces; in gain terms, GRPO lifts the base by \(+7\) pp (\(23\!\to\!30\)) while Agon lifts it by \(+31\) pp over GRPO (\(30\!\to\!61\)), and \(+26\) pp over a two-pass GRPO self-cascade at matched inference budget, nearly doubling the untrained MoA cascade.

To decompose the cascade gain, we evaluate the drafter adapter standalone after training (plain prompt, no opponent summary): it reaches 46 pass@1, a +16 pp lift over vanilla GRPO. This is notable given the budget asymmetry: each adapter drafts on only half the steps with group size \(N\), so it receives roughly a quarter of the standalone-rollout gradient volume of the vanilla GRPO baseline (which uses \(2N\) on every problem), plus challenger-stream gradients on the alternating steps. Our working hypothesis is that the challenger-stream updates transfer to standalone solving (the adapter is trained to verify and re-derive, skills that a plain prompt also exercises), but we have not tested this with a control (e.g.the standalone accuracy of a cooperative-trained or self-refinement adapter), so the attribution of the drafter’s +16 pp to the competitive objective is a hypothesis, not an isolated measurement. The challenger then converts a net 15 of the drafter’s 54 failures into correct answers (with possible regressions among the drafter’s 46 successes), yielding the final 61. Under this reading, the drafter’s lift (+16 pp) and the challenger’s conversion (+15 pp) contribute comparable shares of the total gain, the former via the hypothesized transfer of challenger-stream skills to standalone solving.

To check that the gains are not an artifact of the training distribution, 2 reports the same checkpoints on GSM8K and MATH-500 without any further tuning: the ordering is preserved with smaller absolute margins than in-distribution (\(+13\) and \(+19\) pp over zero-shot, vs \(+38\) on DeepMath-hard), as expected for a method whose pressure targets the hard regime.

pass@1 is the primary metric. Pair-trained methods and the refinement controls run two sequential generation passes per problem (for and cooperative: draft, then challenge reading the draft’s summary); both cascade directions are evaluated and the better is reported. Zero-shot and vanilla GRPO are single-pass. avg.len is the final-stage completion length only (full cascade uses \(\sim\)2\(\times\) tokens at inference). Zero-shot and the untrained MoA involve no training and are reported as single evaluations. 95%

confidence intervals are Clopper–Pearson binomial over 300 held-out problems.

5.2 Scaling and generality↩︎

We expect smaller models, which live deeper in the hard regime, to benefit most. 5 shows the gain over vanilla GRPO shrinking but remaining positive as the base grows, and the same pattern holding on two other model families, Qwen3.5 [32] and Gemma 4 [33]: the delta tracks zero-shot strength, not the family. The Agon cascade on the 0.6B base (61) exceeds not only zero-shot Qwen3-4B (52) but vanilla GRPO on the 4B base (59), a model \({\sim}7\times\) larger, on the same held-out set. In every case the pair is two adapters over one base (our instantiation, not a requirement of the method); pairing across families is future work.

5.3 A second domain: competitive-programming code↩︎

The recipe is not specific to math; any domain with a clean programmatic verifier qualifies. We repeat the core comparison on Qwen3-1.7B trained on the easy-difficulty subset of CodeContests [31], with \(r(x,y)=1\) iff the generated program passes all unit tests, a held-out set of \(300\) problems, and the same protocol and generation budget. 6 shows the same ordering as 3, exchange beats the baseline and competition beats cooperation, with traces again shortening. The absolute delta is smaller than on math: unit-test rewards are sparser at this scale, and code traces make the opponent prefill longer relative to generation. Across domains the claims should be read as directional rather than magnitude-matched.

5.4 Ablations↩︎

7 isolates each design choice on Qwen3-0.6B (held-out pass@1). The competition claim is carried by the competition row (adv vs coop). The information exchange row (Agon vs the shared-opponent variant) should be read with care: switching to a shared opponent removes the visible draft and, by the analysis in 4.4, deactivates the competitive gradient, so this comparison changes two things at once and is a consistency check rather than an isolated exchange effect; the isolated exchange delta is GRPO vs cooperative in 3. In the “shared opponent” ablation the challenger group is scored against one fixed opponent draft, hidden from its context (group-constant \(c(a)\)), so the conversion bonus normalizes away (4.4); the result (\(32\) vs \(30\)) is consistent with theory and shows that within-group variance in opponent difficulty is required for the competitive term to contribute gradient. A per-rollout opponent pairing (distinct hidden drafts) is the honest ablation of competition without visible exchange; that variant is left for future work. Role rotation is ablated explicitly: with fixed roles each adapter receives only one gradient stream, yielding 52 pass@1 versus 61 with rotation.

5.5 Analysis: why traces shorten↩︎

Empirically, Agon’s traces are shorter than GRPO’s (8a): the final (challenger) stage averages \(3.5\)k tokens against \(8.1\)k for a single GRPO pass. Note that this is a per-stage comparison: the cascade also runs a drafting stage, whose length we do not report, so total generated tokens per problem are not directly compared here and a full token-cost accounting is left for further analysis. The like-for-like comparison is against the two-stage untrained MoA control, where the same (final) stage is measured on both sides: Agon’s is half the length (\(3.5\)k vs \(6.9\)k). The headline reward contains no length term. The observed shortening is consistent with the challenger having a candidate solution in context, so length-heavy exploration is no longer the only route to a correct answer. Challenger completions conditioned on a correct opponent summary are \({\sim}35\%\) shorter than the challenger-stage average (\(2.3\)k vs \(3.5\)k tokens on the final checkpoint), and 6 shows the typical pattern: the challenger pinpoints the flawed step, re-derives it, and stops.

5.6 An optional density lever (auxiliary study)↩︎

The shortening above is emergent: nothing in the headline reward asks for it. The same game can, however, also grade brevity: instead of taxing tokens with a penalty, we make being correct in fewer tokens than the rival a way to win. As a small side study, separate from the main recipe and from the two headline claims, we ask whether the shortening can thus be driven directly, using the length-tiebreak term from 4.4 (a bonus \(\gamma\) when the challenger is correct and shorter than an equally-correct opponent). On Qwen3-0.6B, turning it on cuts mean trace length from \(3.5\)k to \(2.6\)k at essentially unchanged accuracy (\(61\!\to\!60\) pass@1), so the emergent shortening becomes directly controllable (8). Because the term only breaks ties among already-correct traces, it compresses without measurably trading away accuracy. We report this as an auxiliary knob; all headline numbers elsewhere use the length-free reward.

6 Conclusion↩︎

Agon trains two models to reason over each other’s work, and the pair co-improves past where a single self-graded policy stalls. The competing peer supplies the signal outcome-only RL lacks: it reads the policy’s solution and grades it implicitly, with no process labels and no change to the verifier. Two distinct policies, trained head-to-head with rotating draft-and-challenge roles (instantiated cheaply as two adapters over one frozen base), let us test the idea. Under a matched generation budget, information exchange beats vanilla GRPO and self-refinement (the exchange claim), competition beats cooperation (the competition claim), and the resulting traces are shorter, a by-product of the competition rather than of an explicit penalty. At the same doubled inference budget, combining two copies of the base without training buys \(4\) points over GRPO, while training the pair to compete buys \(31\); the gain comes from the learned collaboration rather than from having two models.

6.0.0.1 Limitations.

Agon needs a verifier and reference problems, and we study math plus a smaller code study, both with clean programmatic rewards; noisier domains may behave differently. Gains depend on the pair being matched in strength but divergent in behavior (too large a gap collapses into distillation, too small a gap into self-play), and our divergence is maintained heuristically (init \(+\) rotation) rather than guaranteed; complementarity is not quantified beyond the cascade win rate. Text exchange bounds the channel bandwidth. Inference is a two-stage cascade, i.e.two sequential generation passes and added latency; reported trace lengths cover the final stage only, and the better cascade direction is selected post hoc on the held-out set. Compute parity holds for generated tokens only: the challenger’s extra prefill of the opponent summary is not equalized (4.5). Groups with zero within-group variance (all-correct or all-wrong) yield undefined \(\sigma\) in GRPO advantage; such groups are dropped. Every trained number in the paper is a single training run evaluated with a single sampled rollout per problem; the better cascade direction was selected post hoc on the held-out set. Run-to-run training variance is unquantified, and the reported deltas should be read with that in mind. The reported magnitudes come from a hard, decontaminated subset and should not be read as DeepMath-wide pass@1.

6.0.0.2 Future work.

This work establishes that the cross-model signal helps while exchanging it in text; the channel itself is the next axis. Exchanging hidden states directly, via KV-cache injection or gated latent bridges between the two adapters, removes the serialization bottleneck and lets models share information they cannot verbalize [22]. Beyond accuracy, our auxiliary density lever (5.6) is a first step; scaling it into a full multi-objective, compression traded explicitly against correctness, could make density a first-class, tunable target, and a diversity term over divergent solution strategies could make the pair complementary enough to ensemble.

References↩︎