1 Introduction↩︎

Reinforcement learning with verifiable rewards (RLVR) has substantially advanced performance in domains like mathematics and programming by providing a simple yet effective recipe for improving reasoning [1]–[5]. A notable example is Group Relative Policy Optimization (GRPO [6]), which streamlines the RL formulation by eliminating the need for a separate critic model [7]. GRPO estimates advantages by sampling multiple outputs per prompt and normalizing rewards within each group. However, its binary correctness-based reward scheme introduces two fundamental limitations: (1) it neglects the distinct contributions of individual tokens throughout the reasoning trajectory [8], and (2) it offers no mechanism for differentiating quality among responses in zero-advantage groups, wasting vast training data [9], [10].

Recent work has identified entropy as a critical signal for advantage shaping in GRPO [11]. Subsequent methods incorporate entropy to refine optimization signals, either by selectively updating high-entropy tokens [12] or by augmenting advantage estimates with entropy-based terms [8], [13], [14]. They effectively improve exploration at uncertain decision points and introduce finer-grained advantage differentiation that better distinguishes token importance both within and across trajectories [10]. However, entropy quantifies uncertainty while remaining agnostic to the quality of underlying reasoning. A model may exhibit high entropy either when productively exploring alternative solutions or when becoming confused by incorrect paths [12]. Thus, entropy and correctness are intrinsically misaligned. This creates an inherent challenge for optimization: entropy alone cannot distinguish valuable exploration from detrimental errors. This competing effects have led to contradictory design choices across existing methods: some reward high-entropy tokens in correct responses [10], while others instead favor low-entropy tokens [14]. Fundamentally, what is needed is a correctness-aware signal that can reliably identify tokens aligned with correct reasoning.

This intuition can be illustrated through human-problem-solving. As shown in Figure 1, students often struggle to identify mistakes without guidance, but once a reference answer is provided, incorrect steps become immediately apparent upon review. This contrast highlights a simple principle: correctness emerges through comparison between self-review and reference-guided review. A similar pattern appears in LLMs. When evaluating their own generations alone, LLMs lack fine-grained awareness of the erroneous parts. In contrast, conditioning on a reference answer induces distinct probability shifts on specific tokens: increasing token probabilities aligned with the reference and suppressing those that contradict it. Such contrastive probability changes may provide a more faithful, correctness-aligned signal than entropy.

Building on these insights, we propose Contrastive Policy Optimization (CPO), a framework for correctness-aware advantage shaping in RLVR. CPO introduces a token-level contrastive disagreement that quantifies the divergence between reference-guided and vanilla generation distributions. Through theoretical and empirical analysis, we show that this contrastive disagreement reliably indicates token-level correctness. Since both contrastive disagreement and RLVR rewards are correctness signals operating at different granularities, token-level and trajectory-level respectively, their combination is principled rather than heuristic, motivating our advantage shaping design. To ensure stability, we constrain the disagreement magnitude to prevent it from overwhelming or reversing the trajectory-level advantage. CPO’s token-level advantage shaping directly mitigates the zero-advantage issue prevalent in existing RLVR approaches [10], [15], enabling more informative gradients and effective exploration across large-scale training corpora.

Interestingly, contrastive disagreement also offers a principled theoretical grounding for on-policy distillation (OPD  [16], [17]): the reverse KL used in OPD is a special case of contrastive disagreement. More broadly, our theoretical framework subsumes diverse OPD variants under a single correctness-driven perspective: whether the teacher is an external model [17], a self-teacher conditioned on reference answers [18], critic feedback [19], or other contextual signals [20], what matters is that the teacher distribution is more correctness-informed than the current policy.

Results show that CPO substantially enhances reasoning capabilities over entropy-based RLVR methods while largely preserving out-of-domain performance. On Qwen2.5-Math-7B and Qwen3-Base-4B, CPO outperforms GRPO by 7.7% and 8.5% on average, respectively. We analyze the distinction between contrastive disagreement and entropy, finding that CPO focuses on discriminative features tied to correctness, whereas entropy-based methods emphasize linguistic variability. We show that correct and incorrect responses in CPO naturally support exploration and exploitation, and that balancing these roles leads to superior performance. Our contributions are as follows: (1) We propose contrastive disagreement as a more reliable token-level correctness signal than entropy, supported by both theoretical and empirical evidence. (2) Our contrastive disagreement lays a theoretical foundation for OPD: diverse OPD variants, whether based on external teachers, reference-guided, or critic-guided self-teachers, can be understood as different instantiations of a correctness-informed distribution, unifying RLVR and OPD under a single correctness-driven objective. (3) We introduce CPO, a correctness-aware advantage shaping framework that is computationally efficient and effectively addresses the zero-advantage problem in RLVR. (4) We show CPO’s effectiveness on in-domain math reasoning and out-of-domain generalization, along with in-depth analysis of its learning dynamics.

2 Contrastive Policy Optimization↩︎

We begin by validating the reliability of contrastive disagreement to provide a correctness-aligned signal for token-level advantage shaping, both theoretically and empirically. Building on this foundation, we introduce CPO framework.

2.1 Contrastive Disagreement↩︎

2.1.1 Theoretical Analysis↩︎

Given the question \(x\), the sampled trajectory \(y=(y_1,\dots,y_{|y|})\)² from the policy \(\pi\) and the environment reward \(R(x,y)\in\{-1,1\}\), we define the position-wise probability of the oracle correctness event \(C := \{R(x,Y)=1\}\): \[\small \begin{align} g(x,y_{<t},y_t) := \mathbb{P}\big(C | X=x, Y_{<t}=y_{<t}, Y_t=y_t \big), \end{align}\] which measures the probability that choosing token \(y_t\) at position \(t\) will lead to a correct completion, averaging over all possible future continuations.

We define the ideal next-token distribution conditioned on the oracle correctness as: \[\small \tilde{\pi}_{\mathrm{post}}(y_t\mid x,y_{<t}) := \mathbb{P}\!\left( Y_t=y_t \,\middle|\, X=x,\; Y_{<t}=y_{<t},\; C \right).\] By Bayes’ rule, this distribution is regarded as a reweighting of the prior by correctness probability: \[\small \tilde{\pi}_{\mathrm{post}}(y_t\mid x,y_{<t}) = \frac{\mathbb{P}\big(Y_t=y_t | X=x, Y_{<t}=y_{<t}\big)\, g(x,y_{<t},y_t)}{Z(x,y_{<t})} = \frac{\pi(y_t\mid x,y_{<t})\, g(x,y_{<t},y_t)}{Z(x,y_{<t})}, \label{eq:bayes95reweight}\tag{1}\] where \(Z(x,y_{<t}) = \sum_b \pi(b\mid x,y_{<t})\, g(x,y_{<t},b)\) is the prior-average correctness probability at prefix \(y_{<t}\). Taking logarithms yields: \[\small \begin{align} \log \frac{\tilde{\pi}_{\mathrm{post}}(y_t\mid x,y_{<t})}{\pi (y_t\mid x,y_{<t})}= \log g(x,y_{<t},y_t) - \log Z(x,y_{<t}). \end{align} \label{eq:ideal95disagreement}\tag{2}\] Crucially, for fixed \((x,y_{<t})\), this log-ratio is monotonically increasing in \(g(x,y_{<t},y_t)\). Therefore:

• Tokens with below-average correctness (\(g(x,y_{<t},y_t) < Z(x,y_{<t})\)) have negative log-ratio.

• Tokens with above-average correctness (\(g(x,y_{<t},y_t) > Z(x,y_{<t})\)) have positive log-ratio.

In practice, we do not have direct access to the ideal distribution \(\tilde{\pi}_{\mathrm{post}}\). However, we can condition the model on the oracle reference \(y^\star\) (e.g., prompting the model to refine based on \(y^\star\)) to obtain \(\pi_{\mathrm{post}}(y_t\mid x,y^\star,y_{<t})\). Since \(y^\star\) encodes the ground-truth that determines correctness and the prompt also asks the model to refine, conditioning on such prompt serves as a practical proxy for conditioning on the correctness event \(C\), i.e., \(\pi_{\mathrm{post}}(\cdot\mid x,y^\star,y_{<t}) \approx \tilde{\pi}_{\mathrm{post}}(\cdot\mid x,y_{<t})\). Consequently, the contrastive disagreement \[\small \begin{align} \delta_t(x,y) &:= \log \frac{\pi_{\mathrm{post}}(y_t\mid x,y^\star,y_{<t})}{\pi (y_t\mid x,y_{<t})} \propto \log g(x,y_{<t},y_t) \end{align} \label{eq:delta}\tag{3}\] provides a good estimate of the ideal log-ratio in Eq. 2 . Importantly, this holds regardless of whether the rollout \(y\) is correct or not—the correctness probability \(g(x,y_{<t},y_t)\) is defined over future continuations, not individual trajectories. Full derivations are provided in Appendix 8.

2.1.2 Empirical Studies↩︎

We empirically validate Eq. 3 , showing that contrastive disagreement \(\delta_t(x,y)\) reliably indicates token-level correctness \(g(x,y_{<t},y_t)\). Since \(g(x,y_{<t},y_t)\) represents the expected correctness averaging over all possible future continuations from token \(y_t\), it is intractable to compute exactly. We approximate it with Accuracy@16, i.e., sampling 16 continuations from \(y_t\) and measuring their average score.

Experimental setup. We adopt Qwen2.5-7B-Math [21] as our prior policy \(\pi\) and evaluate on queries from Math-500 [22]. For each query \(x\), we first sample a vanilla generation \(y \sim \pi (\cdot \mid x)\) and record the token-level generation probabilities \(\pi (y_{t} \mid x, y_{<t})\) for each token \(y_{t}\). Next, we compute reference-guided generation probabilities \(\pi_{\mathrm{post}}(y_{t} \mid x, y^\star, y_{<t})\) by forward-passing the sampled sequence \(y\) through the model conditioned on the reference \(y^\star\). Both vanilla and reference-guided prompts are in Appendix 9. We then compute the token-level contrastive disagreement \(\delta_t(x,y)\).

We identify the first token position \(t^\star\) where the disagreement deviates from a predefined threshold \(\delta\): \[\small t^\star = \begin{cases} \min \left\{ t \in \{1, 2, \ldots, |y|\} \mid \delta_t < \delta \right\}, & \delta < 0, \\ \min \left\{ t \in \{1, 2, \ldots, |y|\} \mid \delta_t > \delta \right\}, & \delta > 0. \end{cases}\] Note that \(\delta_t = 0\) denotes the boundary where no disagreement exists. For \(\delta <0\), more negative values indicate greater incorrectness, while for \(\delta >0\), more positive values indicate greater correctness.

We then resample continuations starting from these high-disagreement positions: \(\hat{y} \sim \pi(\cdot \mid x, y_{\le t^\star})\). By conditioning on the prefix up to \(t^\star\) (which includes the high-disagreement token \(y_{t^\star}\)), each resampled completion \(\hat{y}\) is directly influenced by the token choice at this critical fork point. We repeat this resampling process 16 times for each query and compute the Accuracy@16 as the empirical approximation of the correctness probability \(g(x,y_{<t^\star},y_{t^\star})\).

Results. Figure 2 shows the Accuracy@16 of \(\hat{y}\) as a function of the threshold \(\exp(\delta)\). The results show clear asymmetry. As we select tokens with increasingly negative disagreement (\(\exp(\delta)<1\)), accuracy drops substantially from 57% at the boundary (\(\exp(\delta) = 1\)) to approximately 48% at \(\exp(\delta) = 0.1\). This strongly confirms that negative contrastive disagreement reliably identifies incorrect-leaning tokens. In contrast, the positive disagreement (\(\exp(\delta) > 1\)) maintains stable accuracy near baseline level. This asymmetry is expected: selecting correct-leaning tokens merely maintains the baseline performance, which is fundamentally bounded by the backbone model’s capabilities. Nevertheless, the results validate our theoretical analysis, showing that \(\delta_t(x,y)\) correlates with \(g(x,y_{<t},y_t)\). Notably, this correlation is particularly strong for incorrect tokens, confirming the theoretical findings in a more realistic scenario.

Implication. The contrastive disagreement in Eq. 3 resembles the reverse KL used in OPD, which can be viewed as a special case by with an external teacher as \(\pi_{\mathrm{post}}\). More generally, CPO admits any distribution more correctness-informed than the current policy as \(\pi_{\mathrm{post}}\), including external teachers, critic-guided policies [23], or reference-conditioned policies [18]. In this work, we use reference answers because they provide direct correctness signals. We also show that reference-conditioned policies can even outperform external teachers, while avoiding reliance on stronger models and enabling self-improvement within the same policy in Sec 3.3.

2.2 Advantage Shaping with Contrastive Disagreement↩︎

Under the above view, contrastive disagreement and RLVR rewards serve as complementary correctness signals at token and trajectory levels, motivating our advantage-shaping design. Since trajectory-level advantage \(A_i\) cannot distinguish correct from erroneous tokens, we use contrastive disagreement \(\delta_t(x,y)\) to shape advantages for finer-grained credit assignment.

The overall framework is depicted in Figure 3. The policy model generates the response with prior likelihood \(\pi_{\theta_{\text{old}}}(y_t\mid x,y_{<t})\). Then, we concatenate it with a reference-guided prompt, detailed in Appendix 9, which asks the model to regenerate a first-person response based on the reference answer. This prompt simulates real-time problem-solving tone, making the sampled response an appropriate continuation. This combined sequence is resent to the policy model to obtain the posterior likelihood \(\pi_{\theta_{\text{old}}}(y_t\mid x, y^\star,y_{<t})\). Following standard RLVR methods [6], we calculate the trajectory-level advantage \(A_{i,t} = A_i\), and additionally compute the contrastive disagreement as posterior-prior KL: \[\small \label{eq:cpo95delta} \delta_{i,t} = \log \frac{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x, y_i^\star,y_{i,<t}) }{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x,y_{i,<t})} \text{.}\tag{4}\] Advantage Magnitude and Direction. The trajectory-level advantage \({A}_{i,t}\) should be the primary determinant, as the goal is to reward the generation of correct solutions. Meanwhile, the disagreement magnitude \(|\delta_{i,t}|\) indicates how decisively a token leans toward correctness or incorrectness. We therefore propose the base form of advantage shaping as \({A}_{i,t} + \alpha \cdot \delta_{i,t}\), where \(\alpha > 0\) controls the strength of the token-level signal. The primacy of trajectory-level advantage \({A}_{i,t}\) extends beyond magnitude to direction as well. However, in practice, conflicts can arise between trajectory-level and token-level signals. For instance, a good response (\({A}_{i,t} > 0\)) may contain tokens with strongly negative disagreement (\(\delta_{i,t} \ll 0\)), potentially reversing the advantage sign. To ensure the shaped advantage direction remains consistent with \({A}_{i,t}\), we introduce a clipping operation: \[\small \tilde{A}_{i,t} = \begin{cases} {A}_{i,t} + \max(\alpha^{+} \cdot \delta_{i,t}, -\frac{1}{2}{A}_{i,t} ), & {A}_{i,t} > 0, R_i=1, \\ {A}_{i,t} + \min(\alpha^- \cdot \delta_{i,t}, -\frac{1}{2}{A}_{i,t} ), & {A}_{i,t} < 0, R_i=-1, \\ \alpha^+ \cdot \delta_{i,t}, & {A}_{i,t} = 0, R_i=1, \\ \alpha^- \cdot \delta_{i,t}, & {A}_{i,t} = 0, R_i=-1, \\ \end{cases} \label{eq:shaped95advantage}\tag{5}\] where \(\alpha^{+}\), \(\alpha^{-}\) denote the combination strength for correct and incorrect responses. We do not clip the original zero-advantage responses, because the original advantage offers no value for policy optimization. However, our token-level disagreements still offer meaningful learning signals, enabling more informative gradients and more effective exploration across large-scale training corpora. The final CPO objective follows the standard GRPO, with \(\tilde{A}_{i,t}\) replacing the original advantage: \[\small \begin{align} \label{eq:cpo} \mathcal{J}_{\text{CPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y_i \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \Bigg[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min \left( r_{i,t}(\theta) \textcolor{red}{\tilde{A}_{i,t}}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \textcolor{red}{\tilde{A}_{i,t}} \right) \Bigg] \text{.} \end{align}\tag{6}\]

3 Experiment↩︎

3.1 Experimental Setup↩︎

3.1.0.1 Training and evaluation.

During training, we conduct experiments on two base models: the general-purpose Qwen3-Base-4B [2] and the domain-specific Qwen2.5-Math-7B [21]. We train models on the MATH dataset [22] with 7.5k problems. We implement CPO based on GRPO [6]. Full hypeparameters are in Appendix 10. We evaluate on both in-domain and out-of-domain benchmarks. In-domain math reasoning tasks include MATH500 [22], AIME2024/2025 [24], and AMC23 [25]. For out-of-domain evaluation, we use GPQA [26], MMLU-Pro [27], and Knowlogic [28], covering knowledge-driven and logical reasoning tasks. In inference, we set temperature to 0.6 and top-p to 0.95 for sampling, and report Pass@16 as our primary evaluation metric.

Baselines. We compare CPO with two baseline categories ³: Standard RLVR methods including GRPO [6] and DAPO [15]. Entropy-intervened methods that incorporates entropy into GRPO: (1) Entropy-Tokens [12], which optimizes tokens only in the top 20% entropy range; (2) Entropy-Adv [8], which directly augments the advantage with an entropy term; (3) EM-RL-token [13], directly uses negative entropy as rewards in RL; (4) RL-ZVP [10], which applies entropy regularization exclusively to zero-advantage prompts for better data utilization; and (5) IB-reg [29], which modulates entropy regularization based on advantage values.

3.2 Main Results↩︎

We present comparative results in Table 1 and Table 2 for Qwen2.5-math-7B and Qwen3-Base-4B. CPO achieves superior performance across both backbones, outperforming GRPO by 7.7% and 8.5%, respectively. For in-domain math reasoning, CPO yields particularly notable gains on AIME2025, the most challenging benchmark requiring high per-step accuracy over long reasoning chains. This validates that inducing fine-grained supervision signals is particularly well-suited for complex reasoning-intensive tasks. More strikingly, while baselines show substantial out-of-domain degradation (except KnowLogic on Qwen2.5-Math-7B), CPO preserves and even improves generalization, exceeding Qwen3-Base-4B, on GPQA and MMLU-PRO by 1.5% and 2.8%, respectively. This likely stems from its self-contrastive correctness signals that avoid external preference biases, while token-level advantage shaping reduces overfitting by targeting critical tokens rather than penalizing all tokens uniformly. Due to computational constraints, we report mean \(\pm\) std over 3 independent runs for CPO, GRPO and the strongest baseline DAPO in Appendix 11.1, showing that CPO consistently outperforms both baselines with improvements well beyond one standard deviation.

3.3 Unifying OPD into CPO via Posterior Design↩︎

We study different instantiations of \(\pi_{\text{post}}\) under the CPO framework in Figure 4. OPD employs a stronger teacher, Qwen2.5-Math-72B, to guide the student via a reverse KL objective. OPD clear out-of-domain gains but limited in-domain improvements. We then consider three variants: (1) CPO-\({\pi_{\text{teacher}}}\), using the teacher directly as \(\pi_{\text{post}}\); (2) CPO-\(\pi_{\text{self}}(\cdot|ref)\), our method that conditions the current policy on reference answers; and (3) CPO-\(\pi_{\text{teacher}}(\cdot|ref)\), conditioning the teacher on reference answers. CPO-\(\pi_{\text{teacher}}\) underperforms the reference-based variant, suggesting that teacher guidance alone show weaker instance-level correctness signals. In contrast, reference conditioning provides more direct correctness to serve as \(\pi_{\text{post}}\). Combining both yields the best performance, where CPO-\(\pi_{\text{teacher}}(\cdot|ref)\)) consistently achieves the strongest results, indicating that teacher generalization and reference-based correctness signals are complementary.

3.4 Ablation Study↩︎

We conduct ablation studies to investigate the components design in CPO in Table 3. We focus on three key aspects: data effectiveness, reference-guided prompt design, and advantage design.

Data Effectiveness. We examine how response correctness affects CPO. Applying CPO only on incorrect responses maintains most in-domain math reasoning but reduces out-of-domain generalization (e.g., \(-\)​4.2% on MMLU-PRO). In contrast to the incorrect-only setting, shaping only correct responses shows the opposite trend. This suggests that negative examples primarily drive in-domain improvement by targeting the model’s weaknesses. Meanwhile, positive examples play a crucial role in maintaining out-of-domain capabilities. A plausible explanation is that shaping positive examples encourages the model to explore diverse solution paths, thereby preventing overfitting to narrow reasoning patterns and preserving broader generalization.

Reference-guided Prompt Design. Besides the default single-turn first-person prompt, we test two alternatives: (1) 2-turn gold-shot, a two-turn structure where the question-reference pair forms the first turn and the model’s response the second; (2) Motivated by the few-shot prompting [30], 2-turn 1-shot, which replaces the first turn with a fixed training question-reference pair. Full prompt formats are in Appendix 9. Results show that our default prompt works best. The improvement of CPO over 2-turn gold-shot may stem from stronger single-turn problem-solving capabilities in current LLMs. Meanwhile, 2-turn gold-shot outperforms 2-turn 1-shot, suggesting that using the reference to the current question induces more reliable contrastive signals than using an unrelated example.

Advantage Design. We investigate advantage shaping methods to identify the most effective one.

• Weighted disagreement \(\pi_{\text{post}}\log \frac{\pi_{\text{post}}}{\pi}\). \(\pi_{\text{post}}\) is designed to prevent the Eq. 3 from exploding when both \(\pi\) and \(\pi_{\text{post}}\) are very small ( \(\pi \ll \pi_{\text{post}} \ll 0.1\)). Compared to the defaulted CPO, the weighting shows a moderate decline. This suggests two findings: (1) extreme cases where both probabilities are rare, making the protective weighting unnecessary; (2) the weighting suppresses informative cases where \(\log \frac{\pi_{\text{post}}}{\pi}\) carries a critical signal.

• Motivated by GSPO [31], we explore trajectory-level advantage shaping by averaging token-level disagreement in a response. But this causes substantial performance drops on out-of-domain tasks compared to our token-level design. This suggests that fine-grained, token-level shaping better preserves crucial variations in token-wise credit assignment and is more robust to distribution shifts.

• We remove the advantage clipping in Equation 5 . Performance degrades mostly in AIME2024 & 2025, the most challenging reasoning tasks. This indicates that preserving consistent advantage directions is crucial for difficult problems, where misalignment between token-level and trajectory-level directions can severely disrupt learning.

• Instead of using contrastive disagreement for advantage shaping, we follow [18] in treating it as an additional regularization term in the final objective. Results show that the use of contrastive disagreement in CPO is more effective than [18] across most in-domain and out-of-domain tasks.

3.5 In-depth Analysis↩︎

3.5.0.1 How CPO improves RL training.

To understand the learning mechanisms of CPO and how it improves RL training, we analyze the training dynamics of CPO and GRPO in Figure 5. From (a)/(b), we observe that CPO achieves more effective training than GRPO on both in-domain and out-domain tasks. While the difference in greedy decoding accuracy appears modest, (c)/(d) reveal larger gaps in Pass@\(K\) performance (\(K=\)​1 to 16). GRPO primarily optimizes for pass@1 performance compared to the Base model [32], with diminishing gains as \(K\) increases and even performance degradation on MMLU-PRO. In contrast, CPO improves both individual sample quality and diversity, with consistent improvements across all \(K\) values. The entropy and response length curves in (e)/(f) provide insight into these differences. CPO converges to a relatively high entropy plateau while producing longer responses, indicating sustained exploration throughout training. GRPO, however, drives entropy near zero, severely constraining the model’s exploratory capacity.

How response correctness shapes CPO. To investigate how correct and incorrect responses contribute to training CPO, we vary the ratio of combination strengths for correct and incorrect responses, i.e., \(\alpha^{+}:\alpha^{-}\) in Figure 6. A clear pattern emerges: higher ratios of \(\alpha^{+}\) yields increasing entropy throughout training. This entropy-increasing behavior stems from two factors. First, CPO activates all-correct groups that contribute zero gradient in GRPO, injecting additional learning signal into otherwise wasted data. Second, unlike sequence-level updates that amplify already-dominant tokens and collapse the distribution, token-level updates selectively reward correct-leaning tokens regardless of their prior probability, reinforcing diverse reasoning paths and maintaining a broader output distribution. We also provide a controlled experiment isolating CPO’s contribution on zero-advantage prompts in Appendix 11.2, where CPO outperforms RL-ZVP by +3.2% average, confirming the superiority of contrastive disagreement over entropy on zero-advantage problems. Conversely, incorrect responses focus on exploitation and confident predictions, as evidenced by steadily decreasing entropy for higher \(\alpha^{-}\) ratios. Downstream performance in (b)/(c) verifies that balanced ratios work best, with 1:1 achieving the highest Pass@K on both AIME2025 and MMLU-PRO, outperforming exploration-heavy (5:0, 4:1) and exploitation-heavy (0:5, 1:4) configurations. This suggests that incorrect and correct responses are mutually beneficial: correct responses allow the model to explore novel paths, while incorrect responses take these opportunities to exploit and refine the best paths.

Contrastive disagreement vs. entropy. While CPO is not derived from entropy, prior analyses show that it strongly impacts entropy. To understand their relationship and key differences, we show word clouds of fork tokens in Figure 7. 76% of fork tokens overlap between both measures. This is expected as high-entropy tokens indicate uncertainty, which typically correlates with potential errors. However, CPO uniquely identifies execution-oriented tokens, including programming constructs (python, output) and mathematical symbols (tau, equiv), reflecting focus on correctness-critical features. Entropy emphasizes discourse markers (Please, Consider) and communicative expressions (Human, What), capturing linguistic variability. This distinction makes CPO ideal for RL training: prioritizing task-critical tokens over linguistic diversity enables efficient RL training that closely aligns policy learning with task success.

4 Related Work↩︎

Entropy-Intervened RL. Using entropy in RL was first introduced by [33] to encourage exploration and prevent premature convergence. Recently, several works have incorporated entropy into RLVR objectives. [11] analyze entropy dynamics during training, establishing a foundation for entropy-based interventions. A straightforward approach is entropy regularization, adding entropy terms to the RLVR objective [34], [35]. [29] extend this with bi-directional regularization to prevent both entropy collapse and explosion. However, these methods can suffer from instability [36], motivating approaches that focus on entropy changes rather than absolute values [36], [37]. Another line of work uses entropy for advantage shaping, including awarding high entropy for exploration [8], suppressing entropy to consolidate well-performing patterns [13], and hybrid approaches [10], [14]. Beyond regularization and advantage shaping, entropy enables additional control mechanisms. [12] and [38] use entropy to identify critical fork tokens or chunks to narrow training focus. [39] employ it to control trust regions during updates.

Fine-grained Correctness Signals. Recent work explores fine-grained rewards to address sparse reward problem in RLVR [40]. [41], [42] provide step-level correctness but require intensive human labor. Tree-based search methods automate step-level annotation [43]–[47], but the repeated sampling makes them computationally impractical in RL settings. Recent advances in on-policy distillation (OPD) enable students to learn from teacher feedback on self-generated outputs [16], enabling more effective learning. To eliminate external teacher dependency, self-distillation methods have emerged. Rejection sampling uses output rewards to select superior responses [48], [49], while others simulate synthetic teachers by conditioning the model on critics [19], [23], reference [18], [50] or any other correctness-informed information [20].

5 Conclusion↩︎

We propose Contrastive Policy Optimization (CPO), a novel framework that leverages token-level contrastive disagreement between reference-guided and vanilla generation as a correctness-aware signal for advantage shaping in RLVR. Both theoretical and empirical results demonstrate that this contrastive disagreement reliably indicates token-level correctness. CPO achieves significant improvements on challenging math reasoning tasks while maintaining out-of-domain generalization compared to entropy-based methods. Our analysis reveals that CPO outperforms GRPO by better preserving the exploration capability of the backbone model. This advantage stems from the distinct roles of different response types: correct responses drive exploration of novel solutions, while incorrect responses exploit the most promising paths. Balance both leads to superior performance.

6 Limitations↩︎

The primary computational overhead of CPO stems from the additional reference-guided forward pass required to compute the posterior likelihood \(\pi_{\mathrm{post}}\), which increases training time by approximately 20% compared to standard GRPO. One promising direction to mitigate this is to decouple the reference-guided forward pass from the main training loop via asynchronous computation, at the cost of slightly stale posterior estimates that introduce a degree of off-policy bias.

Additionally, CPO currently relies on the availability of reference answers to construct the correctness-informed distribution \(\pi_{\mathrm{post}}\), which limits its applicability to settings with verifiable rewards. However, as discussed in Section 2, reference answers are just one instantiation of a correctness-informed distribution. A natural extension is to replace reference-guided prompting with other contextual signals that can serve the same role without requiring ground-truth answers, such as critic feedback that judges the quality of the model’s own outputs, or process-level annotations that identify reasoning errors at intermediate steps. Exploring such alternatives would broaden the applicability of CPO to open-ended generation and subjective reasoning tasks, where reference answers are unavailable.

7 Impact Statements↩︎

The primary goal of this work is to improve the accuracy of credit assignment during reinforcement learning. This method has the potential for social impact in domains requiring high accuracy, such as industrial workflow agent and scientific assistance. While improving reasoning capabilities is generally beneficial, we acknowledge that advanced models carry inherent risks related to dual-use potential and the amplification of biases present in training data. We do not foresee specific negative consequences unique to this method beyond these broader considerations that apply to general-purpose LLM development.

8 Theoretical Analysis of Contrastive Generation Disagreement↩︎

Let \(x\) denote a prompt/question and \(y=(y_1,\dots,y_T)\) denote an autoregressive rollout from the prior (vanilla) policy \[\pi(y\mid x)\;=\;\prod_{t=1}^T \pi_{\theta}(y_t\mid x,y_{<t}).\] A verifiable environment returns a binary reward \[R(x,y)\in\{-1,1\},\] indicating whether the final answer in \(y\) is correct. GRPO optimizes the standard RLVR objective \[J(\theta)\;=\;\mathbb{E}_{x}\,\mathbb{E}_{y\sim \pi(\cdot\mid x)}\big[ R(x,y)\big].\]

8.0.0.1 Justification (empirical).

We implement this proxy via teacher-forcing: for a prior rollout \(y\sim \pi(\cdot\mid x)\), we re-compute \(\log \pi_{\mathrm{post}}(y_t\mid x,y^\star(x),y_{<t})\) on the same tokens. Empirically, in Sec. 2.1.2, large negative disagreements are concentrated on incorrect trajectories and align with error-causing steps, supporting \(\pi_{\mathrm{post}}(\cdot\mid x,y^\star(x),y_{<t})\) as an effective approximation to \(\tilde{\pi}_{\mathrm{post}}(\cdot\mid x,y_{<t})\).

9 Generation Prompts↩︎

In this section, we provide the complete prompts used in our experiments: vanilla generation (Table ¿tbl:tab:prompt95vanilla?), reference-guided generation (Table ¿tbl:tab:prompt95rgg?), and two variants from the ablation study in Sec. 3.4: multi-turn gold-shot (Table ¿tbl:tab:prompt95mtgs?) and multi-turn 1-shot generation (Table ¿tbl:tab:prompt95mt1s?). While the exact prompt format is not the core contribution of CPO, it plays a practical role in determining how effectively the reference-guided distribution shifts toward correctness-oriented reasoning. As long as the prompt elicits a posterior distribution that is more correctness-informed than the vanilla generation, the contrastive disagreement remains a meaningful signal regardless of the specific format. The performance differences across prompt variants in Table 3 reflect how effectively each format elicits correctness-oriented reasoning, rather than a fundamental sensitivity to prompt design. Based on our ablation study, we suggest the following practical guidelines for reference-guided prompt design: (1) use a single-turn format, as LLMs have stronger single-turn instruction-following capability; (2) generate in a first-person, real-time problem-solving tone to encourage genuine reasoning; (3) instruct the model not to acknowledge the reference answer, ensuring the model reasons rather than copies; (4) require complete reasoning with intermediate steps and error-correction to capture meaningful token-level differences.

10 Detailed training Implementations↩︎

To verify the effectiveness of CPO in enhancing RLVR performance, we conduct experiments on two base models: the general-purpose Qwen3-Base-4B [2] and the domain-specific Qwen2.5-Math-7B [21]. We train both models on the MATH dataset [22], which contains 7.5k problems. We implement CPO based on GRPO [6]. During training, we use a prompt batch size of 1024 with 8 rollouts per prompt, and set the sampling temperature to 1.0. The maximum context length is set to 4096 for Qwen2.5-Math-7B and 8192 for Qwen3-Base-4B. The combination strength \(\alpha^{+}\) and \(\alpha^{-}\) are both set to 0.025. We perform policy updates with a mini-batch size of 256 and a learning rate of 1e-6. The KL penalty coefficient between the policy and reference models is set to 1e-3, and the PPO clip ratio is set to 0.2. We conduct all experiments on 8 A800 GPUs. The additional forward pass required to calculate reference-guided generation probabilities increases training time by approximately 20%.

11 Additional Experiments and Analyses↩︎

11.1 Statistical Significance.↩︎

To verify the reliability of our results, we conduct 3 independent runs for CPO, GRPO, and DAPO and report mean \(\pm\) std in Table 4. CPO consistently outperforms both baselines with improvements well beyond one standard deviation across all benchmarks. Notably, CPO exhibits substantially smaller variance than DAPO on challenging benchmarks such as AIME2024 (±1.9 vs. ±10.3), suggesting more stable training dynamics. We also note that our primary evaluation metric Pass@16 inherently mitigates evaluation variance by aggregating over 16 sampled responses per problem, further reducing the influence of random sampling.

11.2 CPO on the Zero-Advantage Problem↩︎

For zero-advantage prompts, existing entropy-based methods such as RL-ZVP attempt to assign a high advantage for high-entropy tokens. However, since entropy only captures uncertainty without distinguishing whether a token contributes to correct or incorrect reasoning, the resulting signal remains correctness-agnostic: it may inadvertently reinforce confused or erroneous token choices that happen to exhibit high entropy. CPO addresses this in a more principled manner. CPO uses the token-level contrastive disagreement \(\delta_{i,t}\) as the shaped advantage, which is theoretically and empirically demonstrated in Section 3.1 to serve as a reliable proxy for token-level correctness probability. This provides a more meaningful and directionally reliable advantage signal compared to existing entropy-intervened methods. In our experiments, this advantage is reflected in the consistently superior performance of CPO over RL-ZVP across both in-domain and out-of-domain benchmarks (e.g., +9.5% average on Qwen2.5-Math-7B and +6.6% on Qwen3-Base-4B). To further isolate the contribution of CPO on zero-advantage prompts, we conduct a controlled experiment where only rollouts from zero-advantage groups are used for training, comparing advantage shaping of CPO and RL-ZVP under this setting. The results demonstrate that CPO consistently outperforms RL-ZVP under this controlled zero-advantage setting, with a +3.2% average improvement. This confirms that the advantage signals of CPO is more useful over entropy-based methods.

11.3 Comparison with SFT and Distillation Methods↩︎

To ensure fair comparison under equivalent data conditions, we conduct additional experiments comparing CPO with SFT and distillation baselines on Qwen2.5-Math-7B, where the distillation baseline is constructed by fine-tuning the model on trajectories self-sampled with reference-guided prompts.

As shown in Table 6, vanilla SFT degrades substantially (-12.7 avg.) compared to the original Qwen2.5-Math-7B, while distillation recovers much of the performance (+3.5 avg.), and CPO achieves the best results by a significant margin (+7.9 avg.). SFT suffers from a fundamental distribution mismatch: the concise ground-truth trajectories are far from the model’s own generation distribution, making them difficult to learn from directly. Distillation mitigates this by using reference-guided self-sampling to generate trajectories that are initially much closer to the model’s own distribution. Moreover, the additional reference information guides the model to produce more reasonable and learnable samples, which explains the consistent performance gains over vanilla SFT. However, distillation remains inherently off-policy: the sampled trajectories are collected once and fixed as static supervised targets. As training progresses and the policy evolves, these fixed targets become increasingly misaligned with the current model distribution, ultimately limiting its performance. CPO avoids this issue entirely by operating fully on-policy: the reference answer is never used as a imitation target, but solely to compute a token-level contrastive signal at each training step using the current policy, allowing the contrastive signal to evolve alongside the model and preserving full exploration capability.

References↩︎

---

1. Corresponding authors↩︎

2. Without loss of generality, we drop the subscript \(i\) in this subsection. Here \(y_t \equiv y_{i,t}\) denotes the token at position \(t\) of \(y_i\).↩︎

3. We also compare CPO with SFT and distillation-based methods in Appendix 11.3, where all methods utilize the same reference answer during training for fair comparison.↩︎