1 Introduction↩︎

As the use of large language models (LLMs) continues to expand, the demands placed on them have become increasingly diverse, driving continuous advances in training methods. While next-token prediction [1], [2] on vast amounts of data remains a promising paradigm for pre-training, post-training processes have been extensively explored and remain key to aligning LLMs with downstream requirements. Post-training commonly involves supervised fine-tuning (SFT), which trains models on predefined desired outputs often generated by a larger and stronger LLM, and reinforcement learning (RL), which evaluates the quality of model-generated responses and optimizes the model toward more favorable outputs. Recently, On-Policy Distillation (OPD) has been actively studied as an alternative to reinforcement learning. With the benefits of large, high-performing models, OPD trains the model on dense signals from model-generated responses, reducing concerns about reward design and the noisy signals RL faces. Studies show that OPD can be competitive with RL in the post-training reasoning process, with reduced computational cost and improved performance, when a high-performing model is used as the teacher model.

On-Policy Distillation (OPD) is a variant of knowledge distillation (KD) similar to RL. Traditional KD [3] is designed to transfer outputs from a high-performing teacher model to a student model. It is combined with the generative nature of LLMs and splits into two sub-categories. The first way is to distill from teacher-generated sequences. This approach was originally named Sequence-KD [4] and has been generalized to Supervised Fine-Tuning (SFT) on large model-generated sequences. It is widely used for the post-training stage just before RL. The second way is the on-policy distillation of student-generated sequences. For on-policy training, it generates sequences using the student network and uses the teachers to score the sampled tokens. The generated tokens that agree with the teacher network are enhanced, while tokens with low probability on the teacher network are degraded. As in RL, the learning signal is applied only to the sampled tokens; thus, the student’s original knowledge and ability are preserved throughout the post-training process. There are extensive studies [5]–[7] that extend the OPD’s learning signals beyond sampled tokens, but we will not address these variants in this paper.

Despite its importance and prominence in post-training research, OPD’s fundamental area remains underexplored. The rewards for OPD are simply set to log the difference in probability between the teacher and the student. Even though OPD achieves remarkable performance with the original loss function, there may be room for improvement in the fundamental design of its loss function. Among the various directions for exploration, we focus on the learning trace of reasoning tuning [8]. Most LLMs have multiple stages for training. It starts with pre-training on next-token prediction tasks using massive amounts of data. Then, tuned for reasoning and chat ability with post-training such as SFT and RL. As a post-training method, OPD only requires knowledge from the post-training phase. Thus, as shown in Fig. 1, we design a new loss function for on-policy distillation based on the difference between the base and the reasoning tuning model of the teacher network.

Our core concept is described in Fig. 1. In OPD, the teacher model acquires reasoning ability through reasoning tuning (e.g., instruction tuning). However, the reasoning-tuned teacher also retains natural preferences and stylistic tendencies acquired before the reasoning tuning. We argue that distilling the reasoning-tuned teacher alone does not necessarily reveal the learning trajectory that led the teacher to acquire reasoning capabilities. Thus, we can provide reasoning knowledge by utilizing the delta between the reasoning-tuned teacher and the base teacher (i.e., the model before post-training). We propose On-Policy Delta Distillation (OPD\(^2\)), defining the delta signal using it as the main reward for OPD. In contrast to OPD, which uses the log probability difference between teacher and student as the reward signal, OPD\(^2\) uses the difference between teacher and teacher base as the reward signal. We first conduct three analyses to show the differences and benefits of the delta signal relative to the original reward: word clouds for distillation signals, token-level visualizations, and word-level statistical analyses. These analyses highlight the differences and characteristics of \(\Delta\), providing insight into changes in reward. Based on these analyses, we find that the delta signal offers an effective alternative distillation signal with distinct and desirable properties.

Building on this observation, we introduce two reward designs: centering and joint conditioning, to effectively utilize the delta signal for OPD. First, centering is subtracting the expected rewards from the policy model’s sampling probabilities to handle the signals more easily. In the original OPD, it is automatically removed through training on-policy probabilities. But, hard to figure out the actual learning signal’s direction and causes negative effects without a single sign-biased condition. Using zero-centered rewards, we introduce a joint condition between OPD and the delta signal to address the convergence-point issue in delta-signal-based distillation. With these two design points, we complete the rewards design for our OPD\(^2\) and successfully introduce the delta signal into the OPD framework.

In our experiments, we develop an extensive verification framework across three reasoning domains: Math, Science, and Code. We construct a mixed-domain training set by sampling an equal number of questions from one dataset for each of the Math [9], Science [10], and Code [11] domains, resulting in a 1:1:1 domain ratio. We then perform on-policy distillation on this mixed training set. The models trained by on-policy distillation methods are evaluated on 7 Math, 3 Science, and 4 Code benchmarks. Our verification covers various sizes of Qwen3 [12], both non-thinking and thinking modes, and the recently released Gemma-4 [13]. In extensive verification, OPD\(^2\) consistently outperforms other on-policy distillation methods, demonstrating an impressive performance gap. It is noteworthy that OPD\(^2\) significantly improves on-policy distillation across various sizes (1.7B - 8B), both in no-think and think modes, on cutting-edge models with superior reasoning performance [12], [13].

2 Method↩︎

2.1 Preliminary↩︎

On-Policy Distillation (OPD), a post-training method for reinforcement learning, such as GRPO, is based on efficient per-sample training costs with token-level supervision. Qwen3 highlights the practical efficiency of OPD on its small models [12]. Subsequent studies [14], [15] have developed detailed OPD formulations and implementations. Basically, knowledge distillation trains the student to reduce the output probability distance from the teacher. Since the output is a probability, Cross-Entropy or Kullback-Leibler divergence is employed for the distance function. The loss function of OPD is defined as knowledge distillation loss on on-policy data, i.e., the student’s rollout samples as

\[\mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_{\theta}(\cdot \mid x)} \left[ D_{\mathrm{KL}} \left( \pi_{\theta}(y \mid x) \,\middle\|\, \pi^{*}(y \mid x) \right) \right], \label{eq:pre95opd95loss}\tag{1}\]

where \(\pi_{\theta}\) indicates the student and \(\pi^{*}\) is the teacher model. For a given question \(x\), OPD generates a student response \(y \sim \pi_{\theta}(\cdot \mid x)\) and trains the student’s next-token prediction to mimic that of the teacher by minimizing KL divergence.

OPD is often used to replace reinforcement learning in the post-training stage. Thus, OPD implementation also follows an RL-like framework to leverage optimized on-policy training codes and settings. Similar to the RL setting, the OPD objective is defined as a reward applied only to sampled tokens. For the question \(x\) and given context \(y_{<t}\), the reward for \(t\)-th token \(y_t\) is defined as

\[R_t = \log \pi^{*}(y_t \mid x, y_{<t}) - \log \pi_{\theta}(y_t \mid x, y_{<t}). \label{eq:pre95opd95reward}\tag{2}\]

Note that the reward \(R_t\) is only applied to determine whether to enhance or degrade the sampled token \(y_t\). It doesn’t make gradients for unsampled probabilities. While there are studies on top-k OPD that address unsampled tokens for both rewards and training, we will not cover them in the paper. Eq. 2 is obtained from the partial derivative of the KL divergence with respect to the sampled-token probability \(\pi_\theta(y_t \mid x, y_{<t})\). Since the reward is maximized whereas the KL divergence is minimized, we reverse the sign and omit the additive constant \(-1\).

Using \(R_t\) as token-level rewards, OPD trains the student with the following RL-like gradient computation: \[\nabla_{\theta} J_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_{\theta}(\cdot \mid x)} \left[ \sum_{t=1}^{T} R_t \nabla_{\theta} \log \pi_{\theta}(y_t \mid x, y_{<t}) \right]. \label{eq:pre95opd95rl}\tag{3}\]

With this implementation, the student network is trained to maximize the token-level rewards \(R_t\), which is equivalent to minimizing the KL divergence between the teacher and the student on on-policy data.

2.2 Delta Signal for Distillation↩︎

OPD uses the log-probability difference on the sampled logit \(y_t\) as rewards for the token as

\[R_t^\mathrm{OPD} = \log \pi^{*}(y_t \mid x, y_{<t}) - \log \pi_{\theta}(y_t \mid x, y_{<t}), \label{eq:opd95reward}\tag{4}\]

where \(\pi^{*}\) presents the teacher model and \(\pi_{\theta}\) means the student model. Following the teacher’s signal is a fundamental approach to knowledge distillation. But, we propose to go one step further with the base model (i.e., the model prior to post-training) of the teacher \(\pi^{*}_{base}\). In other words, we define the delta signal:

\[R_t^{\Delta}= \log \pi^{*}(y_t \mid x, y_{<t}) - \log \pi^{*}_{base}(y_t \mid x, y_{<t}) \label{eq:delta95reward}\tag{5}\]

and use it as a primary learning objective of on-policy distillation. \(R_t^{\Delta}\) represents the learning trace of the teacher model from its own base model. In general, LLMs are pre-trained on next-token prediction on massive amounts of data, then tuned with SFT and RL to improve their reasoning ability. As shown in Fig 1, the target knowledge for on-policy distillation is reasoning ability, not basic next-token prediction. Thus, in our on-policy distillation, \(R_t^{\Delta}\) serves as the primary learning signal, extracting the teacher’s knowledge to enable complex reasoning beyond simple next-token prediction.

Before applying \(R^\Delta\) for on-policy distillation, we analyze the difference between \(R^{OPD}\) and \(R^\Delta\) to give intuition behind this change. We provide three analyses of \(R^\Delta\): word clouds to show signal patterns, signal differences with a simple question, and statistical signal changes from \(R^{OPD}\) to \(R^\Delta\).

2.2.0.1 Word clouds.

Fig. 2 shows the representative words with strong signal patterns for teacher, base, and delta. For the analysis, we employ Qwen3-1.7B as the student and Qwen3-4B-Thinking-2507 as the teacher. Also, Qwen3-4B-Base is selected as the base model for the teacher. To simulate an OPD environment, we sample 10k problems from OpenMathReasoning [9] and generate responses with Qwen3-1.7B. The generated tokens are used to build OPD (log\(\pi^{*}\) - log\(\pi_{\theta}\)), base (log\(\pi^{*}_{base}\) - log\(\pi_{\theta}\)) and \(\Delta\) (log\(\pi^{*}\) - log\(\pi^{*}_{base}\)) signals. The signals are illustrated as word clouds using intensity on positive signals. Note that since distillation signals are highly biased to negative, we apply centering as in Eqs. 6 and 7 .

Fig. 2 shows the signal pattern for OPD, Base, and \(\Delta\). OPD represents words enhanced by conventional on-policy distillation loss. These words are preferred by the teacher model among the sampling candidates in the student policy. Base signal means the signals preferred by the base model, which is not connected with reasoning ability and described as a negative signal in Fig. 1. We conjecture that these natural preferences are also included in the teacher model and don’t need to be transferred to the student for reasoning performance. The last \(\Delta\) presents words enhanced when distillation is conducted with Eq. 5 . Compared to OPD, \(\Delta\) enhances reasoning connection words such as hence, note, instead, however, and yet. On the other hand, words that verify and explore the solution are degraded from OPD: see, try, verify, and confirm. As expected, the degraded words also appear in the base model’s word cloud.

2.2.0.2 Token-level signals.

The most representative characteristics of on-policy distillation are the token-level rewards and supervision. It is hard to achieve with other RL-based on-policy training. We provide examples to show the token-level signal difference between OPD and \(\Delta\). It would be close to the actual training scenario that uses challenging questions. But, the challenging scenario requires high-level knowledge and is hard to figure out the wrong parts. Thus, we use simple reasoning questions with intentionally incorrect reasoning and answers to demonstrate how OPD and \(\Delta\) signals cope with the incorrect reasoning. The incorrect reasoning and answer were synthetically generated using an external LLM and manually curated for clarity. They are fixed inputs rather than rollouts sampled from either the student or the teacher. The experiment is conducted for three simple questions from three domains: math, science, and code. The question, reasoning, and answer inputs to the student (Qwen3-1.7B), teacher (Qwen3-4B-Thinking-2507), and base (Qwen3-4B-Base) to build the distillation signals \(R_t^{OPD}\) and \(R_t^\Delta\). Fig. 3 shows the token-level reward signals for OPD and \(\Delta\). Blue indicates promoting signals that increase the token probability, while red indicates suppressing signals. To improve visualization, we clip the signal at \(\pm\)​6 as shown in the color bar in Fig. 3.

OPD and \(\Delta\) show a meaningful difference in various areas of reasoning. In general, \(\Delta\) is more negative than OPD on generic reasoning expressions such as There, Since, and The variable. In the first question, the reasoning goes wrong from multiply. At those parts, \(\Delta\) shows relatively negative signals compared to OPD including the "\(\times\)" token. It is similar to other questions. In the second, into steam is positive on OPD while negative on \(\Delta\). The critical reasoning parts in the third question are joins and values together, which also show different signal directions for OPD and \(\Delta\). OPD keeps showing positive signals in the incorrect reasoning part, which is due to the magnitude difference. The teacher and student present strong negative signals on that part. But, simply, student signals are stronger than those of the teacher. As a result, OPD produces positive signals even in traces of wrong reasoning. In the case of \(\Delta\), two models originate from the same weights, and the reasoning-tuned teacher is more sensitive to incorrect tokens. Thus, the teacher and base model have a superior-inferior relationship in reasoning about related words, which makes \(\Delta\) more robust than OPD in the correlation between the distillation signal and reasoning correctness.

2.2.0.3 Statistical Analysis.

We further analyze token-level signal changes using statistical values when the distillation signal is changed from OPD to \(\Delta\). For this analysis, Qwen3-8B is used for the student, Qwen3-30B-A3B-Thinking-2507 for the teacher, and Qwen3-30B-A3B-Base for the base. We select a large model setting to show the clear signal statistics reducing noisy signals from failure reasoning traces. We use the questions from OpenMathReasoning [9] for the math domain, OpenCodeReasoning [11] for the code domain, and OpenScienceReasoning-2 [10] for the science domain. In each domain, we randomly sample 10k questions and generate reasoning responses with the student model. The generated reasonings are evaluated on the teacher and base models and converted into distillation signals: OPD and \(\Delta\). For each token in the vocabulary, we measure how often its signal changes by at least 1 when replacing OPD with \(\Delta\). We separately count increases and decreases as enhanced and suppressed cases, respectively. The thresholds of \(+1\) and \(-1\) correspond approximately to the top and bottom 4% of OPD signal values and 5% of \(\Delta\) signal values, respectively. For example, an enhanced ratio of 50% indicates that half of the occurrences of a token receive a positive signal change of at least 1. Such a token is thus strongly biased toward enhancement due to changes in the distillation signal.

Table 1 shows the words with high distillation strength changes when the on-policy distillation reward is replaced with \(\Delta\). We conduct analysis on three domains, Math, Code, and Science, where each domain contains 10k questions, and the number of generated tokens is denoted beside the domain name. Across domains, \(\Delta\) tends to enhance explicit logical connection words: hence, thus, however, and regardless. In suppressed words, a vague expression of uncertainty, perhaps, is commonly suppressed with a high ratio. Also, \(\Delta\) suppresses words frequently used in generic problem-solving narration, including tackle, look, consider, analyze, and computing. In Math, note, why, and depending are enhanced, suggesting stronger emphasis on observation, verification, and conditional reasoning, while conversational expressions such as i’m and we’re are suppressed. In Code and Science, reflective expressions such as imagine, oh, hmm, and another are also more frequent, suggesting increased consideration of alternative reasoning paths.

2.3 On-policy Delta Distillation (OPD\(^2\))↩︎

Although the delta signal (\(R^\Delta\)) can be beneficial for distillation, it may have an issue with its convergence behavior. Specifically, unlike the original reward, \(R^\Delta\) does not consider the student signal \(\pi_{\theta}\) in the reward computation; as a result, the training based on \(R^\Delta\) converges to the maximum reward token with a one-hot vector. Even though this convergence point is not reachable in a practical strong-to-weak setting, it might still cause instability during training. Thus, we design a convergence point for \(R^\Delta\) such that no gradients are computed when the student models match the teacher.

On-policy distillation shares the on-policy nature with RL. Thus, a bias invariant to the action (i.e., token) in the rewards is ineffective for training [16]–[18]. We therefore begin with the bias-subtracted OPD formulations, which obtain the advantage by subtracting the expected reward under the on-policy distribution:

\[A_t^\mathrm{OPD} = R_t^\mathrm{OPD} - \mathbb{E}_{\tilde{y}_t \sim \pi_{\theta}(\cdot \mid x, y_{<t})} \left[ \log \pi^{*}(\tilde{y}_t \mid x, y_{<t}) - \log \pi_{\theta}(\tilde{y}_t \mid x, y_{<t}) \right], \label{eq:opd95adv}\tag{6}\]

\[A_t^\Delta = R_t^\Delta - \mathbb{E}_{\tilde{y}_t \sim \pi_{\theta}(\cdot \mid x, y_{<t})} \left[ \log \pi^{*}(\tilde{y}_t \mid x, y_{<t}) - \log \pi^{*}_{base}(\tilde{y}_t \mid x, y_{<t}) \right]. \label{eq:delta95adv}\tag{7}\]

\(A_t^\mathrm{OPD}\) and \(A_t^{\Delta}\) presents rewards advantages of sampled action \(y_t\) over expected rewards values over its original probability \(\pi_{\theta}(\cdot \mid x, y_{<t})\). During implementation, we compute the expected reward over the top-k (k=1024) tokens of \(\pi_{\theta}\) to save GPU memory. We use \(A_t^\Delta\) to replace \(R_t^{\mathrm{OPD}}\) for the on-policy distillation driven by \(\Delta\) signals. For the convergence point issue, we add a stop condition in alignment with \(A_t^\mathrm{OPD}\) as

\[A_t^{D^2} = \begin{cases} A_t^{\Delta} & \text{if } A_t^{\Delta} A_t^{\mathrm{OPD}} > 0, \\ 0 & \text{otherwise.} \end{cases} \label{eq:opd2}\tag{8}\]

\(A_t^{D^2}\) is our advantage function for OPD\(^2\). It basically follows \(A_t^{\Delta}\) but is turned off when it conflicts with original distillation signals. The condition (\(A_t^{\Delta} A_t^{\mathrm{OPD}} > 0\)) prevents the student from being over-trained by the trace signal and being distant from the teacher’s signal. When the student probability is the same as that of the teacher, i.e.,\(\pi_\theta = \pi^{*}\), the advantage \(A_t^{D^2}\) is turned off by the condition. On-policy distillation with \(A_t^{D^2}\) restricts updates to sign-consistent common descent directions between the trace \(A_t^{\Delta}\) and distillation \(A_t^{\mathrm{OPD}}\) signals, while using \(A_t^{\Delta}\) to control the gradient magnitude. Thus, in RL form, the training gradient is described as

\[\nabla_{\theta} J_{\mathrm{OPD^2}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_{\theta}(\cdot \mid x)} \left[ \sum_{t=1}^{T} A_t^{D^2} \nabla_{\theta} \log \pi_{\theta}(y_t \mid x, y_{<t}) \right].\]

The effectiveness of OPD\(^2\) is demonstrated in the upcoming experiment section, with extensive evaluation benchmarks across various models compared to OPD and the advanced on-policy distillation method, ExOPD.

3 Experiment↩︎

We design an extensive evaluation framework to verify the performance improvements of our OPD\(^2\). Since on-policy distillation is primarily used as a post-training process, we target small reasoning models, such as Qwen3 [12] and Gemma4 [13], as student models, using large models from the same family as the teacher. As a post-training process, a notable benefit of on-policy distillation is that it can be applied to any domain without requiring domain-specific reward design. To elaborate on this advantage, we build a multi-domain training dataset by mixing sampled questions from three reasoning domains: Math, Science, and Code. After on-policy distillation, the trained model is evaluated on various benchmarks, including 7 Math, 3 Science, and 4 Code benchmarks. Note that our benchmark covers both non-thinking and thinking modes for the Qwen3 architecture. We also report training dynamics, an ablation study, and computational analysis for OPD\(^2\), which help to understand detailed characteristics of OPD\(^2\) training.

3.1 Experiment settings↩︎

3.1.0.1 Models.

We target open-source models with strong reasoning abilities: Qwen3 [12] and Gemma4 [13]. Note that all student and teacher models are instruct-tuned, while the teacher-base model is -Base. Recently, on-policy distillation studies have reported non-thinking mode performance rather than thinking-mode performance for Qwen3. We find that both settings offer distinct and complementary value. Qwen3 is not well-tuned for reasoning tasks in non-thinking mode. Thus, there is enough gap for improvement, and on-policy distillation has to make substantial changes on student models to achieve comparable performance. On the other hand, thinking mode already has top-level reasoning ability. Thus, on-policy distillation faces the challenges of improving or altering the student model without compromising its existing performance. Therefore, we evaluate both modes for Qwen3, whereas only the thinking mode is verified for Gemma4.

For the Qwen3 family, Qwen3-1.7B, Qwen3-4B, and Qwen3-8B are used as student models, and Gemma4-E4B-it is selected for the student in Gemma4. The models specialized for non-thinking are used as teachers for non-thinking mode case: Qwen3-4B-Instruct-2507 for Qwen3-1.7B and Qwen3-30B-A3B-Instruct-2507 for Qwen3-4B and -8B. For thinking mode, the teacher model is switched to thinking specialized models: Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507. ExOPD [15] and our OPD\(^2\) require teacher base. Thus, we utilize the corresponding base models for these methods: Qwen3-4B-Base and Qwen3-30B-A3B-Base. For Gemma4, we use Gemma-4-31B-it as the teacher for Gemma4-E4B-it, while Gemma-4-31B serves as the teacher-base. The model setting is also summarized in Table 10.

3.1.0.2 Training data.

We introduce a multi-domain post-training scenario for on-policy distillation. Three datasets are selected for three reasoning domains: OpenMathReasoning [9], OpenScienceReasoning-2 [10], and OpenCodeReasoning [11]. These datasets contain challenging questions, which are enough to extract meaningful distillation signals with on-policy distillation. We conduct 1:1:1 balanced sampling for each domain and make a subset with 100k questions for on-policy distillation. Since the training uses fewer than 30k samples, the model experiences only a fraction of the available questions, corresponding to less than one training epoch. Each question is used at most once. Note that we used only the questions from these datasets, and the reasoning traces and answers included in the datasets are discarded during sampling.

3.1.0.3 Training setting.

We implement on-policy distillation methods, i.e., OPD, ExOPD [15], and OPD\(^2\), based on TRL [19]’s GRPOTrainer with proper revision. The revision includes enabling single completion for a question, token-level signals, and disabling group normalization. For each question, only 1 completion is generated and forwarded to student, teacher, and teacher-base models to build corresponding distillation signals. We use a temperature of 0.7 for softmax after the model forward function. The number of training steps is set to 100, and all reported results are measured at the final training step. The expected rewards computed in the centering operation (Eq. 6 and 7 ) are conducted on top-k (k\(=1024\)) tokens to reduce GPU memory cost for full-vocabulary average. The maximum completion length is set to 8k, with a temperature of 0.7 and KL regularization with a reference model disabled. The models are optimized by AdamW [20] using a learning rate of \(5 \times 10^{-6}\), cosine lr decay, and gradient clipping at 1.0. To prevent excessively frequent gradient clipping, we uniformly scale all rewards by a factor of 0.1 for all on-policy distillation methods. vLLM [21] rollout backend integrated in TRL is used to generate responses in colocate mode for Qwen3-1.7B and in server mode on 1 node for others. Note that this setting is also summarized in Table 11.

3.1.0.4 Evaluation.

The models trained with the on-policy distillation methods are evaluated on various benchmarks. We select several benchmarks that are available in Evalchemy [22] or lm-eval-harness [23]. 7 Math, 3 Science, and 4 Code benchmarks are selected as the evaluation set. For math, the models are evaluated on AIME24 [24], AIME25 [25], AMC23 [26], HMMT25 [27], MATH500 [28], OlympiadBench [29], and ReasoningGym Math [30]. For coding, we use CodeContests [31], CodeForces [32], LiveCodeBench [33], and ReasoningGym Algorithm [30]. GPQA [34], SuperGPQA [35], and SciBench [36] are selected for science. All benchmarks use the pass@1 metric with averages across repetitions. The repetition numbers for each benchmark are listed in Table 12

5pt

6pt

3.2 Reasoning performance↩︎

Based on the extensive evaluation, we compare the performance of OPD\(^2\) with the original model, OPD, and ExOPD [15]. Since the original models are reasoning-tuned and have strong reasoning abilities, on-policy distillation methods are first evaluated to determine whether they are beneficial to the model. OPD means basic on-policy distillation training with Eq. 3 . Although it is the most fundamental on-policy distillation method, it still shows promising performance improvements and is applied in various practical settings [12], [14]. ExOPD [15] is a recently proposed advanced method for on-policy distillation. It leverages a teacher–base model to amplify the extrapolation signal and represents a state-of-the-art approach in this area. We implement ExOPD with \(\lambda=1.25\), following the setting reported in the original paper.

3.2.0.1 Non-thinking mode.

We first evaluate all methods in the non-thinking mode, where the models generate answers without explicit reasoning traces. As shown in Tables 2 and [tbl:tab:qwen95other], all on-policy distillation methods substantially improve the original Qwen3 models across Math, Code, and Science, suggesting that on-policy distillation is particularly effective at strengthening the relatively weak non-thinking capabilities of reasoning-tuned models.

Among the methods compared, OPD\(^2\) achieves the best average performance across all model sizes and domains. The improvements are particularly notable in Math. For Qwen3-1.7B, OPD\(^2\) increases the average score from 34.8 to 54.6, outperforming OPD and ExOPD by 3.6 and 3.2 points, respectively. Similar gains are observed for Qwen3-4B and Qwen3-8B, where OPD\(^2\) achieves average scores of 70.3 and 71.6, compared with 66.4 and 67.8 for ExOPD. Notably, the 4B model trained with OPD\(^2\) already surpasses the 8B model trained with both OPD and ExOPD, demonstrating that the proposed method can provide gains comparable to, or greater than, those obtained by increasing the model size.

OPD\(^2\) also consistently improves performance in Code and Science. On Code, it achieves average scores of 29.4, 40.1, and 39.9 for the 1.7B, 4B, and 8B models, respectively. The gain is especially large for Qwen3-1.7B, improving the original model by 18.9 points and ExOPD by 4.8 points. On Science, OPD\(^2\) obtains the highest average score for all three model sizes, reaching 38.8, 50.5, and 51.6. These results indicate that the advantages of OPD\(^2\) are not restricted to mathematical reasoning but generalize across diverse reasoning domains.

5pt

6pt

3.2.0.2 Thinking mode.

We next evaluate the methods in the thinking mode, where the original Qwen3 models already exhibit strong reasoning performance. As shown in Tables 4 and [tbl:tab:qwen95think95other], improving these strong baselines through on-policy distillation is substantially more challenging than improving their relatively weak non-thinking capabilities. In particular, standard OPD frequently degrades performance, while ExOPD provides only limited or inconsistent improvements. For example, on Math, both methods underperform the original models for all three model sizes in terms of average performance.

In contrast, OPD\(^2\) consistently achieves the highest average performance across all model sizes and domains. On Math, OPD\(^2\) improves the original 1.7B, 4B, and 8B models by 3.5, 1.5, and 2.2 points, respectively, reaching average scores of 62.7, 74.8, and 75.9. The improvements are particularly notable on challenging competition-level benchmarks such as AIME25 and HMMT25. For instance, OPD\(^2\) improves the HMMT25 score of Qwen3-8B from 44.3 to 52.3, whereas OPD and ExOPD achieve 43.3 and 46.3, respectively.

A similar trend is observed in Code and Science. On Code, OPD\(^2\) improves the average scores of the 1.7B, 4B, and 8B models from 29.3, 48.7, and 50.8 to 40.4, 50.8, and 57.8, respectively. In particular, the 8B model obtains consistent gains across all four code benchmarks. On Science, where the original models are already competitive, OPD\(^2\) still achieves the best average performance for every model size, whereas OPD and ExOPD often reduce the baseline performance. These results demonstrate that OPD\(^2\) can improve not only relatively weak capabilities but also the already strong reasoning capabilities of reasoning-tuned models without sacrificing their overall performance.

5pt

6pt

3.2.0.3 Gemma4.

Finally, we evaluate the methods on Gemma4-E4B-it to examine whether their effectiveness generalizes beyond the Qwen3 model family. As shown in Tables 6 and [tbl:tab:gemma95other], standard OPD substantially degrades performance across all three domains, indicating that directly applying conventional on-policy distillation can be unstable when applied to a strong model from a different family. ExOPD alleviates this degradation and yields meaningful improvements in Math, but its effectiveness remains inconsistent across domains.

OPD\(^2\) achieves the best Math performance by a clear margin, improving the average score from 60.6 to 67.8. It outperforms ExOPD by 2.5 points on average and achieves the highest score on six of the seven benchmarks. The gain is particularly remarkable on AIME24, where OPD\(^2\) improves the original model from 51.7 to 69.2, compared with 59.6 for ExOPD. These results show that the benefit of OPD\(^2\) is not specific to Qwen3 and transfers effectively to a distinct model architecture and training recipe.

The results on Code present a more challenging case. Gemma4-E4B-it already exhibits strong coding performance, and none of the distillation methods surpasses the original model in terms of average score. Nevertheless, OPD\(^2\) retains substantially more of the original capability than OPD and ExOPD, achieving an average score of 49.5 compared with 36.9 and 45.1, respectively. It also improves RGAlgo from 62.2 to 64.0. On Science, OPD\(^2\) improves the average score from 47.0 to 48.8 and achieves the best results on SuperGPQA and SciBench. Overall, these results demonstrate that OPD\(^2\) generalizes across model families and provides a more robust trade-off between improving target capabilities and preserving the strong capabilities of the original model.

3.3 Training dynamics↩︎

OPD\(^2\) demonstrates substantial performance improvements across all domains and consistently outperforms the other on-policy distillation methods by a meaningful margin. To understand how these gains emerge during training, we compare the performance trajectories of OPD, ExOPD, and OPD\(^2\) under the Qwen3-4B non-thinking setting in Fig. 4. All methods exhibit a rapid increase in performance during the early stages of training, indicating that most of the benefits of on-policy distillation are achieved within a relatively small number of optimization steps. After this initial improvement, however, their trajectories diverge considerably.

OPD and ExOPD tend to reach their peak performance early and subsequently plateau or degrade as training proceeds. This behavior is particularly evident on Math and Code, where the final performance can be noticeably lower than the intermediate peak. In contrast, OPD\(^2\) achieves a larger initial improvement and maintains a consistently higher performance throughout the remaining training steps. Although its performance also fluctuates after the early peak, it remains clearly above that of OPD and ExOPD across all three domains.

The separation is especially notable on CodeContests, where OPD\(^2\) maintains a large advantage over the competing methods throughout training. A similar gap is observed in Math and Science. These results suggest that the improvements of OPD\(^2\) do not arise from an isolated peak at a particular checkpoint, but reflect a persistent advantage over the entire training trajectory. Please note that all results in the main tables are reported at the final training step. We do not select the best-performing checkpoint.

3.4 Ablation study↩︎

6pt

We conduct an ablation study to analyze the contribution of each component in OPD\(^2\). As shown in Table 8, replacing the delta signal with the standard OPD signal causes the largest performance degradation in both non-thinking and thinking modes. This result indicates that the delta signal is the primary source of the performance gains achieved by OPD\(^2\). Removing the agreement condition or the centering operation has a comparatively smaller effect. Although these components generally contribute to the overall performance, their impact is not as substantial or consistent as that of the delta signal. Overall, the ablation results show that the delta signal plays the central role in OPD\(^2\), while the condition and centering provide additional but relatively modest improvements.

3.5 Computational costs↩︎

7pt

OPD\(^2\) introduces additional computation compared with standard OPD because it requires a teacher-base model forward pass to construct the delta signal. As shown in Table 9, this increases the wall-clock training time by approximately 24-28% for the Qwen3 models and by 8% for Gemma4-E4B. For OPD\(^2\), this additional computation represents a modest overhead compared with standard OPD. However, the computation cost of OPD\(^2\) remains comparable to that of ExOPD, with only a small difference in training time across all model sizes. Our current implementation is not optimized for the reward computation in Eq. 5 , suggesting that the reported overhead can be further reduced. Moreover, on-policy distillation typically converges within a relatively small number of training steps, and prolonged training can even lead to performance degradation, as observed in Fig. 4. Thus, although OPD\(^2\) requires additional computational cost compared to standard OPD, its additional overhead remains limited in the typical short-training regime of on-policy distillation.

4 Related Work↩︎

Post-training commonly relies on SFT and RL to align LLMs with reasoning tasks. On-policy distillation (OPD) has recently emerged as an alternative post-training method built on knowledge distillation. The distillation transfers the outputs of a high-performing teacher to a student [37]. In fact, the SFT above is itself a form of KD. Its first form learns from teacher-generated sequences, named Sequence-KD [38], and later generalized to SFT on large model-generated sequences [39], [40]. However, training on the teacher’s sequences has a limitation. The student learns from the teacher’s distribution but generates from its own distribution at inference. This train-inference mismatch is known as exposure bias [41]. On-policy distillation addresses this issue. Instead of teacher sequences, it uses the student’s own rollout samples and lets the teacher score the sampled tokens [42]–[44]. Thus, the learning signal is applied to the states the student actually visits at inference. Recent studies show that OPD is competitive with RL in reasoning post-training, with dense token-level signals and reduced cost [45], and its efficiency has been reported on small models in recent releases [12], [14].

Following OPD, various studies extend or analyze the on-policy distillation framework. Some studies modify the learning signal, such as reweighting it by uncertainty [46] or allowing a controlled degree of off-policy data for efficiency [47]. Other studies remove the external teacher and use a single LLM as both the teacher and the student, in which the teacher gets a verified solution as privileged information while the student sees only the problem [48], [49]. There are also studies that analyze OPD rather than extend it [50]–[52]. These methods share a common point. They all set the teacher’s output as the target for the student. The most related work to ours uses the teacher’s base model for the distillation signal. ExOPD [15] uses the teacher-base model to build an extrapolation signal. It amplifies the difference between the teacher and its base with a factor \(\lambda > 1\), so the student is trained beyond the teacher. Our OPD\(^2\) also uses the teacher base model, but in a different way. We do not extrapolate the teacher’s output. Instead, we use the difference between the teacher and its base, the delta signal, as the main reward. The delta signal represents the knowledge that the teacher learned from reasoning tuning. With centering and the joint condition, OPD\(^2\) focuses the signal on reasoning-related tokens while keeping the student’s original ability.

5 Conclusion↩︎

In this paper, we have examined the potential of the delta signal, the difference between the teacher and the base model, as the major training signal for on-policy distillation. We conjectured that the delta signal can be an effective way to transfer reasoning knowledge obtained through reasoning tuning of the teacher model, starting from base models. Through various analyses, we showed that the delta signal includes meaningful reasoning signals and has beneficial points compared to the original distillation rewards. Based on this observation, we have proposed OPD\(^2\), on-policy distillation based on the delta signal. Our OPD\(^2\) is verified on an extensive framework comprising three reasoning domains (Math, Science, and Code), 14 evaluation benchmarks, and networks of various sizes (1.7B-8B) across cutting-edge models (Qwen3 and Gemma4). In all settings, OPD\(^2\) achieved substantial performance improvements over conventional on-policy distillation, demonstrating the effectiveness of the delta signal for on-policy distillation. We believe that OPD\(^2\) can serve as an important milestone in the development of on-policy distillation.

6 Appendix↩︎

Tables 10 and 11 summarize the model-specific configurations and common optimization settings used in our experiments. We use NVIDIA H100 GPUs for all experiments. Qwen3-1.7B is trained on a single 8-GPU node, while Qwen3-4B, Qwen3-8B, and Gemma4-E4B are trained using four nodes. For the multi-node settings, 24 GPUs are used for student training and 8 GPUs are allocated to the vLLM rollout server.

For Qwen3, we use a larger model from the same model family as the teacher. The Qwen3-1.7B student uses Qwen3-4B as its teacher, while the Qwen3-4B and Qwen3-8B students use Qwen3-30B-A3B. We select the corresponding Instruct or Thinking checkpoint according to the target generation mode. Gemma4-E4B-it is evaluated only in the thinking setting and uses Gemma4-31B-it as the teacher. The corresponding pretrained checkpoints without instruction tuning are used as the teacher-base models for ExOPD and OPD\(^2\).

All methods are trained for 100 optimization steps with an effective global batch size of 256 or 288. We use AdamW with a learning rate of \(5\times10^{-6}\), cosine lr decay, a warmup ratio of 0.1, and a minimum learning-rate ratio of 0.1. The gradient norm is clipped to 1.0, and all the rewards are scaled by 0.1 to prevent excessive gradient clipping. We set the maximum generation length to 8192 and the sampling temperature to 0.7. The KL coefficient is set to \(\beta=0\), as we do not apply an additional KL regularization term with the reference model during training.

7pt

Table 12 summarizes the evaluation configurations for all benchmarks. We report pass@1, interpreted as single-sample accuracy, and average the scores over multiple independent repetitions to reduce the variance introduced by stochastic generation. The repetitions are not used for best-of-\(k\) selection. AIME24 and AIME25 are evaluated with 16 repetitions because of their relatively small number of problems, whereas larger benchmarks are evaluated with fewer repetitions.

6pt

References↩︎