1 Introduction↩︎

Large reasoning models (49B+) achieve strong performance but are prohibitively expensive for deployment. Knowledge distillation compresses these capabilities into smaller students. The dominant off-policy paradigm—training on static teacher-generated traces—suffers from exposure bias: prediction errors compound autoregressively at inference [1], [2].

On-policy distillation (OPD) addresses this by training on student-generated rollouts scored by the teacher [1], [3], matching or exceeding RL methods like GRPO [4]–[6]. However, existing OPD methods apply supervision uniformly—the same divergence, on every token, at every phase, for every prompt.

We identify a single structural problem underlying this uniformity: in OPD, supervision quality depends on student competence. Unlike off-policy KD (teacher traces are always coherent) or RL (binary rewards are equally reliable), OPD’s per-token teacher corrections are only informative when the student’s rollouts are sufficiently coherent. This competence-dependent degradation manifests at three scales:

• Token level: \({\sim}50\%\) of tokens are deterministic for both models—supervising them wastes compute. Among the remaining, some call for sharpening (RKL) while others require diversity preservation (FKL) [5].

• Temporal level: The optimal divergence evolves from mode-covering (early) to mode-seeking (late), yet prior solutions use manual two-stage switches [4], [7].

• Prompt level: Problems beyond the student’s capability produce incoherent rollouts on which teacher supervision is noise [8], [9].

These are not three independent problems but three symptoms of ignoring how supervision quality varies with competence. We observe that entropy provides a unified observable for this quantity across all three scales.

We introduce SEAD, a framework operationalizing this principle:

1. Token-level: Sparse Entropy-Adaptive Divergence. Joint teacher-student entropy partitions tokens into Zone A (skip), Zone B (RKL), Zone C (FKL)—jointly determining selection and divergence type.

2. Temporal: Competence-Driven Annealing. Continuous FKL\(\to\)RKL transition tracking the evolving active token composition.

3. Prompt-level: Competence-Gated Curriculum. First prompt-level curriculum for OPD, addressing an open problem flagged by [8], [9].

Crucially, these are symbiotically necessary: token selection requires coherent rollouts (curriculum), and annealing requires monotonic competence growth (also curriculum). The ablation (Section 4) confirms super-additive interactions.

We validate SEAD on OLMo-3 (7B\(\to\)​32B) and Nemotron (8B\(\to\)​49B) across MATH-500, Minerva-Math, AIME 2024/2025, AMC 2023, and OlympiadBench. The full framework achieves +4.8 average over vanilla OPD.

2 Method↩︎

2.1 Preliminaries and Problem Setting↩︎

Let \(\pi_\theta\) and \(\pi_{\text{te}}\) denote the student’s and teacher’s policies respectively. For a given prompt \(\mathbf{q}\), let \(\mathbf{c}_t = (\mathbf{q}, x_1, \dots, x_{t-1})\) denote the context up to step \(t\), and \(x_t \in \mathcal{V}\) the generated token. Each OPD iteration: (1) samples rollouts \(\mathbf{x} \sim \pi_{\theta_{\text{old}}}(\cdot | \mathbf{q})\); (2) queries teacher logits on the rollout; (3) updates \(\pi_\theta\) via a divergence loss. The two standard choices are forward KL (mode-covering): \[\mathcal{L}_{\text{FKL}}(t) = \sum_{v \in \mathcal{V}} \pi_{\text{te}}(v | \mathbf{c}_t) \log \frac{\pi_{\text{te}}(v | \mathbf{c}_t)}{\pi_\theta(v | \mathbf{c}_t)} \label{eq:fkl}\tag{1}\] and reverse KL (mode-seeking): \[\mathcal{L}_{\text{RKL}}(t) = \sum_{v \in \mathcal{V}} \pi_\theta(v | \mathbf{c}_t) \log \frac{\pi_\theta(v | \mathbf{c}_t)}{\pi_{\text{te}}(v | \mathbf{c}_t)} \label{eq:rkl}\tag{2}\] Forward KL encourages the student to cover all modes of the teacher, preserving diversity at reasoning branch points. Reverse KL drives the student to concentrate on the teacher’s high-probability modes, yielding sharper outputs but risking premature entropy collapse [4].

Intuitively, gradient signal scales with competence: when the student cannot solve a problem at all (\(p_i \approx 0\)), its rollouts are incoherent and \(\phi(p_i) \approx 0\), so the lower bound is almost a zero regardless of the teacher’s quality. Entropy serves as the observable proxy for this principle: joint teacher–student entropy reveals whether a token position is informative, redundant, or noise. Assumption 1 motivates a unified principle: allocate compute only where supervision quality is high. SEAD operationalizes this at three granularities: (i) token-level—skip tokens where both teacher and student are already confident (zero information gain); (ii) temporal—shift from exploratory FKL to sharpening RKL as competence grows; (iii) prompt-level—restrict training to prompts where rollouts are coherent enough for the teacher to provide meaningful corrections. We present these in order of increasing scope, noting that the prompt-level curriculum (Sec. 2.5) is the foundational enabler: it ensures the entropy landscape is well-behaved, which the token-level selector requires, and it guarantees monotonic competence growth, which the temporal annealer assumes.

2.2 The Unified SEAD Objective↩︎

SEAD integrates token-level selection, temporal annealing, and prompt-level curriculum into a single loss. Let \(\mathcal{B}\) and \(\mathcal{C}\) denote the sets of token indices assigned to RKL and FKL respectively (formally defined in Sec. 2.3), and let \(\lambda > 0\) be a hyperparameter balancing the intrinsic scale differences between the two divergence terms: \[\mathcal{L}_{\text{SEAD}}(\theta, t_{\text{step}}) = \frac{1}{|\mathcal{B} \cup \mathcal{C}|} \left( \sum_{t \in \mathcal{B}} \mathcal{L}_{\text{RKL}}(t) + \alpha(t_{\text{step}}) \cdot \lambda \sum_{t \in \mathcal{C}} \mathcal{L}_{\text{FKL}}(t) \right), \quad \mathbf{q} \sim \text{Uniform}(\mathcal{D}(t_{\text{step}})) \label{eq:sead95unified}\tag{4}\] Three control variables—all functions of training progress, all serving the same principle (allocate compute only where supervision quality is high):

• Zone partition \(\{\mathcal{A}, \mathcal{B}, \mathcal{C}\}\): which tokens receive gradient and which divergence (Sec. 2.3);

• Annealing coefficient \(\alpha(t_{\text{step}})\): FKL/RKL balance evolving with competence (Sec. 2.4);

• Eligible set \(\mathcal{D}(t_{\text{step}}) = \{\mathbf{q}_i : d_i \leq c(t_{\text{step}})\}\): prompts within the competence frontier (Sec. 2.5).

2.3 Token-Level: Sparse Entropy-Adaptive Divergence↩︎

We define token-level entropy \(H_{\text{te}}(t) = -\sum_{v} \pi_{\text{te}}(v | \mathbf{c}_t) \log \pi_{\text{te}}(v | \mathbf{c}_t)\) and \(H_{\theta}(t)\) analogously. To maintain computational tractability and avoid the prohibitive overhead of a full softmax over the entire vocabulary \(\mathcal{V}\) during training, \(H_{\text{te}}(t)\) is approximated using the top-\(k\) vocabulary subset. SEAD partitions tokens into

• Zone A (Skip, \({\sim}\rho_A\%\)): Both \(H_{\text{te}}(t)\) and \(H_\theta(t)\) low. Zero gradient—the vast majority of tokens (connectives, formatting, deterministic steps) fall here.

• Zone B (RKL, \({\sim}\rho_B\%\)): \(H_{\text{te}}(t)\) low, \(H_\theta(t)\) high. Student should sharpen toward the confident teacher.

• Zone C (FKL, \({\sim}\rho_C\%\)): \(H_{\text{te}}(t)\) high. Genuine reasoning forks—student should cover teacher modes.

The per-step loss is: \[\mathcal{L}_{\text{SEAD}}^{\text{token}} = \frac{1}{|\mathcal{B} \cup \mathcal{C}|} \left( \sum_{t \in \mathcal{B}} \mathcal{L}_{\text{RKL}}(t) + \lambda \sum_{t \in \mathcal{C}} \mathcal{L}_{\text{FKL}}(t) \right) \label{eq:sead}\tag{5}\] with defaults \(\rho_A = 50\), \(\rho_B = 40\), \(\rho_C = 10\). Unlike EOPD [5] (teacher entropy only) and TIP [10] (weighting, single divergence), SEAD jointly determines both selection and divergence type via joint teacher-student entropy, with extreme sparsity (\({\sim}20\%\) active).

2.3.0.1 Sparse selection is approximately lossless.

The following theorem shows that Zone A tokens contribute negligible gradient, justifying their exclusion. The key precondition is that Zone A tokens have genuinely low entropy for both teacher and student—a property that holds when the curriculum (Sec. 2.5) ensures coherent rollouts, preventing spurious low-entropy assignments from misaligned contexts.

The bound vanishes as \(\tau \to 0\): Zone A tokens are selected precisely for near-zero entropy, making their gradient contributions negligible. Crucially, this guarantee requires the entropy landscape to be well-behaved—which holds when curriculum ensures coherent rollouts.

2.3.0.2 Connection to temporal scheduling.

As the student improves, Zone A expands and the active set shifts toward Zone B. The optimal FKL/RKL ratio among active tokens thus evolves, motivating a temporal schedule.

2.4 Temporal: Competence-Driven Divergence Annealing↩︎

As the student masters more tokens, the active token composition shifts: early in training Zone C (both uncertain) is large, calling for mode-covering FKL; late in training Zone B (teacher confident, student uncertain) dominates, calling for mode-seeking RKL. The annealing schedule tracks this: \[\alpha(t_{\text{step}}) = \alpha_{\text{end}} + \frac{\alpha_{\text{start}} - \alpha_{\text{end}}}{2}\left(1 + \cos\!\left(\frac{t_{\text{step}}}{T_{\text{total}}} \cdot \pi\right)\right) \label{eq:anneal}\tag{7}\] decreasing from \(\alpha_{\text{start}} = 0.8\) to \(\alpha_{\text{end}} = 0.0\). Combined with SEAD zones, \(\alpha\) modulates Zone C’s FKL weight in Eq. 4 , implementing a continuous exploration\(\rightarrow\)refinement transition that generalizes the discrete two-stage switches of [4] and [7].

Temporal annealing assumes monotonically improving competence—an assumption that can be violated under uniform prompt sampling. The following curriculum ensures this precondition holds.

2.5 Prompt-Level: Competence-Gated Curriculum↩︎

The deepest manifestation of the supervision quality principle (Assumption 1): on prompts far beyond the student’s capability, \(p_i(\theta) \approx 0\) implies \(\phi(p_i(\theta)) \approx 0\), so the modulated PL condition yields vanishing gradient signal. The rollouts are incoherent and teacher corrections amount to noise. This degradation is absent in off-policy KD (teacher traces are always coherent, \(\phi \equiv 1\)) and RL (binary rewards are equally reliable regardless of rollout quality). The curriculum keeps \(\phi(p_i(\theta))\) bounded away from zero on all eligible prompts—a precondition for both Theorem 1 (the entropy landscape must be well-behaved for Zone A tokens to genuinely have low entropy) and temporal annealing (competence must grow monotonically for the schedule to track the correct FKL/RKL ratio).

2.5.0.1 Difficulty estimation.

We sample \(K\) rollouts per prompt from \(\pi_{\theta_0}\) and compute difficulty \(d_i = 1 - p_i\) where \(p_i = \frac{1}{K}\sum_k \mathbf{1}[\mathrm{correct}(\mathbf{x}^{(k)}_i)]\). Pass rate is a binary discretization of rollout entropy, maintaining the entropy-as-unified-signal principle.

2.5.0.2 Competence-based progression.

Following [11], a competence function \(c(t) = \min(1, c_0(1 + t/T)^p)\) grows monotonically, with eligible set \(\mathcal{D}(t) = \{\mathbf{q}_i : d_i \leq c(t)\}\). In OPD, \(c(t)\) delineates prompts where rollouts are coherent enough for informative teacher supervision.

2.5.0.3 Staleness of difficulty scores.

Because \(d_i\) is computed from the initial student \(\pi_{\theta_0}\), the relative ordering of prompts may drift as training progresses. While monotonic improvement guarantees that initially “easy” prompts remain solvable, mid-tier prompts may improve at different rates. Consequently, we rely on the curriculum as a coarse filter rather than a strict total ordering, trading precise difficulty tracking for computational efficiency. The primary risk is minor sample inefficiency, not gradient corruption, as the curriculum’s only theoretical requirement is maintaining a minimum competence floor \(\phi(p_i(\theta)) \geq c > 0\) on all eligible prompts.

Under Assumption 1, and assuming \(\beta\)-smoothness of the loss and bounded gradient variance \(\sigma^2\) (see Appendix 7.3), we have the following theorem:

3 Evaluation↩︎

3.1 Experimental Setup↩︎

3.1.0.1 Models.

We evaluate our approach on two distinct model families to demonstrate generalizability. The first is Nemotron (Nano-8B student, Super-49B teacher)and the second is OLMo (7B-Instruct-SFT student, 32B-Instruct teacher), which allows us to rigorously test our method in the challenging small teacher-student capability gap regime.

3.1.0.2 Training.

We trained our models using 4 nodes. Each node is equipped with 8 H100 80GB GPUs and 1TB memory. We utilize a rigorously deduplicated version of the DAPO-Math-17K dataset [12] for all training phases. More training details could be found in Appendix 8.2.

3.1.0.3 Evaluation.

Our evaluation suite comprehensively assesses mathematical reasoning capabilities across varying difficulty levels. We report greedy Pass@1 accuracy on standard reasoning benchmarks, including MATH-500, Minerva-Math, and OlympiadBench. To evaluate performance on highly complex, competition-level mathematics, we utilize the AMC 2023 and AIME 2024/2025 datasets, reporting the Pass@32 metric (the average pass rate over 32 independently sampled trajectories per problem).

3.1.0.4 Baselines.

We benchmark our approach against several strong paradigms: Group Relative Policy Optimization (GRPO) [13], Vanilla OPD utilizing full-token Reverse Kullback-Leibler (RKL) divergence [3], and On-Policy Self-Distillation (OPSD) [9].

3.2 Main Results↩︎

We present the results of OLMo model pair in Table 1 and leave Nemotron results in Appendix [app:exp95nemotron].

3.2.0.1 The Distillation Challenge.

As shown in Table 1, bridging the massive 17.2-point performance gap between the OLMo teacher and student is challenging. While the teacher establishes a strong upper bound (75.4 average), standard off-policy distillation (OPD) baselines fail to meaningfully improve upon the base student. OPSD and GRPO stagnate near the student’s 58.2 average, while Vanilla OPD (RKL \(k=1\)) yields a marginal \(+1.0\) improvement.

3.2.0.2 Efficacy of Proposed Components.

KL Annealing alone provides a modest gain (+0.2 avg), confirming that divergence scheduling without token-level structure has limited impact. Token Zones alone similarly yields marginal improvement (+0.3), as zone-based sparsity without a matching temporal schedule leaves the FKL/RKL balance untuned. The critical finding is their synergy: combining Token Zones with KL Annealing produces a +4.5 jump to 63.7 average—a \(20\times\) amplification over either component in isolation. This validates the symbiotic design: zone sparsity concentrates gradient on informative tokens, while annealing ensures the divergence applied to those tokens evolves appropriately. Adding the competence-gated curriculum (SEAD full) further improves to 64.0, with the easy-to-hard ordering providing the coherent rollouts that stabilize entropy-based zone assignment.

3.2.0.3 Combined Method Performance.

Combining Token Zones with KL Annealing yields robust performance, achieving a 63.7 average (a \(+5.5\) absolute increase over the base student), with pronounced gains on the most rigorous datasets, including \(+7.7\) points on AIME24 and \(+7.5\) points on AIME25. Ultimately, the full SEAD method further pushes the average to 64.0. These results indicate that stabilizing the token zone strategy with an annealed KL penalty effectively mitigates baseline stagnation, drastically elevating student capabilities and closing the teacher-student gap.

3.3 Ablation Study↩︎

We conduct a \(2^3\) factorial ablation on OLMo-3-7B/32B to isolate three components: T (Token SEAD, 50/40/10 zone assignment), A (KL annealing from 0 to target), and C (curriculum). Table 2 reveals three findings: Curriculum (C) is the single strongest factor, yielding +4.20 avg.accuracy alone, while annealing (A) in isolation provides negligible gain (+0.22). T+A jumps to +4.52, a 20\(\times\) amplification over A alone, demonstrating that entropy-adaptive zone sparsity requires a matching schedule to be effective. The three-way combination T+A+C achieves the best overall result (64.00 avg., +5.2 AIME25), with each pairwise interaction contributing complementary gains. T+A matches curriculum-only (63.70 vs.), and A+C (63.97) narrows the gap to the full system. This confirms that SEAD’s token-level zone assignment, annealing, and curriculum address orthogonal bottlenecks: zone sparsity targets which tokens receive gradient, annealing controls when sparsity activates, and curriculum determines what order examples are presented.

4 Analysis↩︎

4.1 Token-Level Analysis↩︎

4.1.0.1 Entropy distribution and zone assignment.

Zone A tokens overwhelmingly correspond to deterministic computation steps and connectives, e.g., on OLMo-3 7B/32B (50/40/10 config), 73.2% of computation tokens and 54.2% of syntax tokens are assigned to Zone A. Zone B tokens cluster at reasoning transitions where the student holds partial knowledge (17.2% strategy forks, 52.7% common tokens receiving moderate gradient), and RKL sharpens the student toward the teacher’s peak mode. Zone C tokens appear at genuine reasoning forks—63.9% are strategy fork positions where both teacher and student entropy exceed the 75th percentile, representing critical decision points with multiple valid continuations where FKL’s mode-covering property prevents catastrophic collapse. Crucially, the 50/40/10 configuration routes zero strategy fork tokens to Zone A, ensuring every reasoning-critical position receives active supervision (Table 3). Figure 3 (b) visualizes this on actual model output: Zone C tokens (red) concentrate at branching points in the reasoning chain, while Zone A tokens (green) cover predictable syntax and arithmetic with near-zero loss contribution.

4.1.0.2 Selection overlap and signal complementarity.

With a Jaccard index of 63.7% between the top-20% teacher entropy and top-20% student entropy positions, approximately 36% of high-entropy tokens are unique to one signal (Figure 3 (c)). Teacher entropy identifies positions where guidance is most valuable (multiple valid continuations), while student entropy identifies positions the student has already mastered. This partial overlap justifies joint conditioning: neither signal alone captures the full picture.

4.1.0.3 Gradient concentration.

Figure 3 (d) quantifies the concentration effect: Zone A (50% of tokens) carries only 9.7% of total FKL loss, while Zone B+C (50% of tokens) concentrates 90.3% of gradient signal. The per-zone ratios are stark: Zone C tokens carry \(3.19\times\) their fair share of loss, Zone B tokens carry \(1.46\times\), and Zone A tokens carry only \(0.19\times\). This means skipping Zone A discards half the tokens while retaining over 90% of the training signal, a highly efficient compute-quality tradeoff. Figure 3 (a) shows this visually: the sorted FKL curve drops precipitously, with Zone B and C tokens (orange, red) dominating the high-loss region while Zone A tokens (green) cluster near zero.

3.5pt

4.2 Training Dynamics↩︎

A known failure mode of vanilla OPD is premature entropy collapse, where the student model overly concentrates probability mass on the teacher’s peak mode before exploring alternative reasoning paths. SEAD effectively mitigates this collapse. As illustrated in Appendix 9.2, vanilla OPD exhibits a clear decreasing trend in average token-level entropy during training, whereas SEAD maintains stable, consistently higher entropy levels. This preserves necessary distributional diversity while still sharpening the student’s outputs.

5 Related Work↩︎

5.0.0.1 On-Policy Distillation.

GKD [1] established on-policy distillation for LMs. Extensions include RKL for reasoning [3], self-distillation (OPSD, [9]), and theoretical unification with dense KL-constrained RL (G-OPD, [6]). Recent systems work focuses on scaling [4], [14].

5.0.0.2 Token-Level Selection.

EOPD [5] switches FKL/RKL based on teacher entropy alone. TIP [10] uses student entropy \(\times\) divergence for weighting with a single divergence. SE-KD [15] and SCOPE [16] explore other selection criteria. SEAD differs via (i) joint teacher-student partitioning, and (ii) per-zone divergence switching.

5.0.0.3 Divergence Scheduling.

Reopold [4] uses a discrete two-stage schedule; PACED [7] finds forward-then-reverse optimal; DRKL [17] addresses entropy collapse. Our annealing generalizes these to continuous transitions motivated by evolving zone composition.

5.0.0.4 Curriculum for Reasoning.

Curriculum has been studied for RL-based reasoning [18], [19]. However, No prior work applies curriculum to OPD, despite it being flagged as an open problem [8], [9]. Our contribution is showing why it is uniquely necessary: the modulated PL condition (Assumption 1) formalizes supervision quality degradation specific to OPD.

6 Conclusion and Limitations↩︎

In conclusion, SEAD addresses the dependence of teacher supervision on student competence in on-policy distillation by utilizing entropy as a unified signal. It establishes a mutually reinforcing framework operating across three levels: applying adaptive divergences to informative tokens, temporally annealing from exploration to refinement, and employing a competence-gated prompt curriculum. However, this work has several limitations. First, the curriculum relies on static difficulty scores rather than computationally expensive adaptive re-estimation. Second, our method may apply to other domains (e.g., code) for future work. Finally, SEAD’s efficacy on very long reasoning chains requires further investigation.

7 Theoretical Proofs↩︎

7.1 Full Assumptions for Theorem 1↩︎

7.2 Proof of Theorem 1↩︎

7.3 Full Assumptions and Proof of Theorem 2↩︎

The analysis requires four assumptions. Assumptions 5 and 6 are standard; the novelty is Assumption 1.

8 Additional Experiment Setup↩︎

8.1 Licenses for Existing Assets↩︎

We use only existing models, datasets, benchmarks, and software frameworks whose terms permit research use. Table 4 summarizes the main assets used in this work. We use these assets for training, evaluation, and comparison only, and we do not redistribute third-party model weights, benchmark data, or dataset contents unless their licenses explicitly permit redistribution. When releasing code, we will include links to the original asset sources, preserve copyright and license notices, and specify the exact versions used for all reproducibility-critical dependencies.

All third-party assets are used consistently with their intended research or evaluation purposes. For benchmarks derived from mathematical competitions or external providers, we report aggregate evaluation results only and do not claim ownership over the underlying problems. The proposed method itself does not require any proprietary dataset or private user data.

8.2 Training Details.↩︎

During each iteration, we sample \(B = 96\) prompts and generate one rollout per prompt using a sampling temperature of 0.7 and a maximum generation length of 16,384 tokens. Following teacher scoring, we perform \(K = 1\) gradient update with a clipping parameter \(\epsilon = 0.2\). We use a peak learning rate of \(5 \times 10^{-9}\) with a linear warmup and decay schedule over a single training epoch. For the SEAD configuration, the regularization weights are set to \(\rho_A = 50\), \(\rho_B = 40\), and \(\rho_C = 10\). The annealing factor follows a cosine schedule, decaying smoothly from \(\alpha_{\text{start}} = 0.8\) to \(\alpha_{\text{end}} = 0.0\). For curriculum learning, we adopt a smooth competence-based ordering inspired by [11]. We first estimate per-problem difficulty as \(d_i = 1 - \hat{a}_i\), where \(\hat{a}_i\) is the pass rate computed from \(K = 8\) independent rollouts of the student model prior to training. Examples are then sorted by difficulty in ascending order and presented easy-to-hard in a single pass without replacement, with small Gaussian noise (\(\sigma = 0.02\)) added to difficulty scores for stochastic tie-breaking. This provides a smooth, continuous curriculum that naturally increases problem difficulty as training progresses, without requiring discrete phase boundaries or explicit competence thresholds.

9 Additional Experimental Results↩︎

9.1 Results on Nemotron↩︎

We present the results on Nemotron model pair in Table 5.

9.2 Training Dynamics↩︎

A known failure mode of OPD is premature entropy collapse: if the student is trained with reverse KL from the beginning, it is encouraged to concentrate probability mass on the teacher’s current peak mode before it has explored alternative valid reasoning paths. SEAD addresses this through the interaction of token selection and KL annealing. Early in training, the coefficient \(\alpha\) is high, so Zone C tokens receive forward-KL supervision; later, \(\alpha\) decays toward zero, shifting the active objective toward reverse KL refinement.

The key distinction from a global annealing baseline is that SEAD does not apply this uniformly across all tokens. Roughly 50% of tokens are assigned to Zone A and skipped, because both teacher and student are already confident on them. Consequently, annealing only affects the subset of tokens where divergence choice matters: Zone C reasoning forks, where forward KL preserves the teacher’s support, and Zone B partially mastered tokens, where reverse KL sharpens the student toward the teacher’s preferred continuation.

Figure 4 shows the average entropy for different methods during the training. Among these methods, vanilla OPD shows a clear decreasing trend in entropy, where all our SEAD-related methods keep the entropy at a similar level and higher than vanilla OPD, which proves the ability of our method to prevent entropy collapse.

10 Broader Impacts↩︎

This work aims to improve the efficiency of distilling reasoning capabilities from larger language models into smaller student models. By reducing the amount of token-level supervision required during on-policy distillation, SEAD may lower the computational cost of post-training and make strong reasoning models more accessible to researchers and practitioners with limited hardware resources. This could have positive impacts by reducing energy consumption, lowering deployment costs, and enabling broader participation in research on reasoning-oriented language models.

At the same time, improving the efficiency of reasoning-model distillation may also make capable models easier to train, adapt, and deploy. Such models could be misused for generating misleading content, automating deceptive reasoning, assisting with harmful planning, or increasing the scale of low-cost model deployment without sufficient safety evaluation. Although our experiments focus on mathematical reasoning benchmarks, the same distillation principles may transfer to other domains where stronger reasoning capabilities require careful governance.

Our method does not directly address model safety, factuality, robustness, bias, privacy, or misuse prevention. In particular, SEAD optimizes the efficiency and effectiveness of teacher–student distillation, but it does not guarantee that the student inherits only desirable behaviors from the teacher. If the teacher model exhibits unsafe, biased, or unreliable behavior, the student may also learn such behavior. Therefore, practical deployment of models trained with SEAD should include standard safety evaluations, red-teaming, domain-specific risk assessment, and monitoring for unintended behavior.

The empirical scope of this paper is limited to mathematical reasoning benchmarks and two teacher–student model families. As a result, the broader societal implications of applying SEAD to open-domain assistants, code-generation systems, or high-stakes decision-support systems remain uncertain. We encourage future work to evaluate competence-aware distillation under broader safety, robustness, and alignment criteria before deployment in sensitive settings.

11 Declaration of LLM Usage↩︎

This paper studies large language models as the primary subject of research. The proposed method uses teacher and student language models during on-policy distillation, as described in the main text.

In addition, the authors used large language model tools to assist with non-substantive writing tasks, including improving clarity, editing grammar, drafting checklist responses, and revising explanatory text. All technical ideas, algorithmic design choices, theoretical claims, proofs, experimental results, tables, and conclusions were produced, checked, and approved by the authors. The authors take full responsibility for the content of the paper, including any text that was edited with LLM assistance.

References↩︎