1 Introduction↩︎

Large language models (LLMs) have achieved rapid progress on mathematical and logical reasoning tasks [1]–[3]. This progress is largely driven by reasoning-oriented post-training techniques, such as supervised fine-tuning (SFT) on chain-of-thought traces [4]–[6], on-policy distillation (OPD) from larger models [7]–[9], and reinforcement learning (RL) with verifiable rewards [10], [11]. Despite their methodological differences, these methods are still commonly evaluated through single-attempt final-answer accuracy.

Final-answer accuracy, however, provides only a partial view of reasoning behavior. It measures whether a model eventually reaches the correct answer, but does not reveal whether the model can reliably estimate its own uncertainty during the reasoning process [12], [13]. This distinction is especially important for inference-time compute. Before reasoning begins, confidence can serve as a signal of problem difficulty [14], [15]; during reasoning, it can help identify low-quality traces and terminate them early [16], [17]; and after reasoning, it can support the selection or aggregation of multiple sampled solutions [18], [19]. In these settings, well-calibrated confidence can improve not only final accuracy, but also the efficiency and reliability of reasoning [20], [21].

Existing research leaves two key questions unresolved. First, work on confidence estimation often treats calibration as an aggregate model property or a final-output signal, rather than studying where confidence is reliable within a chain-of-thought trace [22]–[27]. Second, prior comparisons of reasoning-oriented post-training methods mainly focus on final-answer accuracy, leaving unclear how SFT, OPD, and RL affect confidence calibration and uncertainty estimation. As a result, we still lack a systematic understanding of how different post-training paradigms reshape confidence before, during, and after reasoning.

To bridge this gap, we systematically investigate how reasoning-oriented post-training affects confidence calibration across the entire reasoning pipeline. Rather than treating calibration as a monolithic property of the final output, we decompose the reasoning process into a stage-wise framework. This allows us to evaluate the utility of confidence signals for specific inference-time decisions through three targeted calibration tasks: In the Pre-CoT stage, we evaluate whether confidence signals before chain-of-thought generation can predict problem difficulty. In the Intra-CoT stage, we examine whether confidence signals during generation can identify reasoning traces that are unlikely to lead to correct answers and support early termination. In the Post-CoT stage, we test whether confidence signals from complete reasoning traces can improve answer aggregation across multiple candidates.

To our knowledge, this is the first controlled study of how SFT, RL, and OPD reshape confidence calibration in reasoning models. We compare variants derived from the same Qwen2.5-7B-Instruct backbone and trained on the same problem mixture drawn from DeepScaler and SimpleRL, holding the model family and data source fixed so that calibration differences can be attributed directly to the post-training phase. The training setup is constructed to make the three paradigms comparable: SFT learns from DeepSeek-R1-Distill-Qwen-32B teacher traces retained only when the final answer is correct; OPD distills teacher token-level supervision onto student-generated rollouts; and RL optimizes verifiable outcome rewards. We then evaluate these models on diverse reasoning benchmarks.

This controlled setup reveals a sharper picture than final-answer accuracy alone. OPD provides the most reliable Pre-CoT signal for difficulty estimation, SFT yields the strongest Intra-CoT signal for early termination, and RL produces the best Post-CoT signal for answer aggregation. More surprisingly, we find that confidence reliability is strongly position-dependent. SFT maintains a relatively stable confidence gap between correct and incorrect traces, which explains why it is effective for online early stopping. RL confidence is initially uninformative, but becomes reliable after the model commits to a reasoning direction. OPD shows the opposite pattern: its early confidence is useful, but later confidence can become inversely calibrated, meaning that incorrect traces may appear more confident than correct ones. These patterns explain why naive confidence averaging over the full trace can be misleading: it mixes reliable and unreliable regions into one score.

Motivated by this observation, we introduce PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf is simple but effective because it matches confidence extraction to each model’s calibration dynamics. For RL, it avoids making early decisions before confidence becomes meaningful, improving early stopping by up to about 4 points under matched token budgets and further raising confidence-based aggregation to a 6.1-point average gain over majority voting. For OPD, it avoids the later inverse-calibration region and instead relies on the earlier part of the trace, consistently improving low-budget early stopping across all benchmarks, with gains of more than 3 points on AIME-style problems. These results show that confidence in reasoning models should be used not only stage-wise, but also position-awarely.

1.0.0.1 Contributions.

We make three main contributions. First, we introduce a three-stage framework for reasoning models that connects confidence from Pre-CoT, Intra-CoT and Post-CoT to inference-time compute decisions. Second, we provide a controlled comparison of SFT, RL, and OPD, showing that reasoning-oriented post-training induces distinct confidence shifts. Third, we further show that confidence reliability is position-dependent and introduce PosConf, a position-aware strategy that improves confidence-based early stopping and aggregation.

2 Confidence Calibration Framework↩︎

We partition confidence by what decision it can affect: before generation for query-level routing, during generation for online pruning, and after generation for candidate aggregation. This view yields three stages: Pre-CoT, Intra-CoT, and Post-CoT.

Let \(x\) denote an input problem with ground-truth answer \(y\), and let \(P_x\) denote the corresponding prompt. For each problem, the model samples a set of reasoning traces \(\mathcal{T}_x=\{\tau_i\}_{i=1}^{N}\), where we use the simplified notation \(\mathcal{T}\) and \(\tau=(t_1,\dots,t_m)\) when the context is clear. Each trace \(\tau_i\) produces an extracted answer \(\mathrm{answer}(\tau_i)\) and receives a trace-level confidence score \(C(\tau_i)\). As our foundational internal confidence signal, we rely on token-level predictive statistics. Given a prefix \(s_\ell = (P_x, t_1, \dots, t_{\ell-1})\), let \(p(v \mid s_\ell)\) be the model’s next-token distribution over the vocabulary \(\mathcal{V}\). Following [16], we derive confidence from the top-\(k\) candidate tokens, \(\mathcal{V}_k(s_\ell)\). The token-level confidence at position \(\ell\) is defined as: \[c_\ell = -\frac{1}{k} \sum_{v \in \mathcal{V}_k(s_\ell)} \log p(v \mid s_\ell). \label{eq:token95conf}\tag{1}\] Intuitively, this score captures the concentration of the next-token distribution: a highly peaked distribution results in a larger confidence value, whereas a flatter, more uncertain distribution yields a lower value. We also use relative token position to compare confidence dynamics across traces of different lengths. For a token at index \(t\) in a trace of length \(m\), we define its relative token position as \[\rho_t = t/m \in [0,1].\] Thus, \(\rho_t \approx 0\) corresponds to the beginning of the reasoning trace, whereas \(\rho_t \approx 1\) corresponds to the final-answer region.

2.0.0.1 The Pre-CoT Stage.

The Pre-CoT stage evaluates whether confidence before generation begins can serve as a query-level signal to predict problem difficulty. We compute a query-level confidence score before any chain-of-thought tokens are generated by applying the token-level confidence formula to the prompt itself (\(s_1=P_x\)): \[C_{\mathrm{pre}}(x) = c_1(P_x) = -\frac{1}{k} \sum_{v \in \mathcal{V}_k(P_x)} \log p(v \mid P_x). \label{eq:pre95conf}\tag{2}\] To proxy actual difficulty, we sample \(\mathcal{T}_x\) and aggregate answers by majority voting [18]. A problem is “solved” if the majority-voted answer \(\hat{a}_x\) matches \(y\), and “unsolved” otherwise. We then measure how well Pre-CoT confidence ranks these outcomes using AUROC and PRR [28]. Higher values indicate that \(C_{\mathrm{pre}}(x)\) captures model-specific difficulty before reasoning starts.

2.0.0.2 The Intra-CoT Stage.

The Intra-CoT stage evaluates whether confidence during generation can serve as a step-level signal to detect unreliable reasoning paths. For a partial trace \(\tau_{1:\ell}=(t_1,\dots,t_\ell)\), we compute token-level confidence scores \(\{c_1,\dots,c_\ell\}\). To reduce lexical and formatting noise in individual token probabilities, we adopt the group-confidence approach from [16] and smooth these values over a sliding window of size \(w\). For any position \(\ell \ge w\), the local window confidence is defined as: \[G_{\ell,w}(\tau) = \frac{1}{w} \sum_{r=\ell-w+1}^{\ell} c_r. \label{eq:group95conf}\tag{3}\] This metric acts as an online reliability signal. If \(G_{\ell,w}(\tau)\) drops below a predefined threshold \(\theta\), it suggests the trace has entered a highly uncertain segment, triggering early termination. These truncated traces are excluded from final answer aggregation. We evaluate this stage by sweeping the threshold \(\theta\) to map the token-accuracy frontier.¹ A well-calibrated Intra-CoT signal should significantly reduce token usage without compromising accuracy.

2.0.0.3 The Post-CoT Stage.

The Post-CoT stage evaluates whether confidence after generation concludes can serve as a trace-level signal to improve answer aggregation. Here, we use the mean token confidence as a trace-level score for each completed trace in \(\mathcal{T}_x\). A reliable Post-CoT signal should assign higher scores to traces that yield correct answers. To evaluate this, we compare standard majority voting against confidence-filtered majority voting. While the standard approach selects the most frequent answer across all samples, our filtered approach first ranks traces by \(C_{\mathrm{post}}(\tau_i)\), retains only the top \(\eta\)%, and performs majority voting on this refined subset. We additionally report “most-confident selection”, which simply extracts the answer from the single highest-confidence trace. Improved accuracy from these methods confirms that trace-level confidence successfully isolates reliable candidates.

3 Experimental Setup↩︎

3.0.0.1 Models.

We compare four models based on the Qwen2.5-7B-Instruct backbone [29]. We refer to them as Qwen-Instruct, Qwen-SFT, Qwen-RL, and Qwen-OPD. Qwen-Instruct is the original instruction-tuned backbone and serves as the reference model without additional reasoning-oriented post-training. The other three models are trained on the same mixture of DeepScaler [30] and SimpleRL [31] data, but with different post-training paradigms.

Qwen-SFT is trained with standard cross-entropy on long chain-of-thought traces generated by DeepSeek-R1-Distill-Qwen-32B [7]. To construct high-quality supervised targets, we use rejection sampling and keep only teacher responses whose final answers are correct. Qwen-RL is trained with group relative policy optimization [11] using binary outcome rewards [32]. Qwen-OPD first samples rollouts from the student model and then distills token-level supervision from DeepSeek-R1-Distill-Qwen-32B through dense KL divergence [33], [34]. Holding the backbone and data source fixed makes calibration differences more attributable to the post-training objective. Detailed training configurations are provided in Appendix 9.

3.0.0.2 Benchmarks.

We evaluate all models on four mathematical reasoning benchmarks with different difficulty profiles. AIME 2024 [35] and AIME 2025 [36] are competition-level benchmarks consisting of challenging short-answer problems. AMC 2023 provides a broader set of moderate-difficulty contest problems, while MATH500 [37] offers a larger and more statistically stable evaluation set covering diverse mathematical settings.

4 Results↩︎

We present results across three stages. Overall, we find that reasoning-oriented post-training induces stage-dependent changes in confidence calibration. In Pre-CoT, OPD provides the most informative pre-reasoning signal. In Intra-CoT, SFT yields the most effective token-level signal for early stopping. In Post-CoT, RL shows the largest benefit when confidence is used to filter completed reasoning traces. These results show that post-training does not uniformly improve confidence calibration.

4.0.0.1 Pre-CoT: Difficulty Estimation

3pt

Table 1 shows that Pre-CoT confidence differs substantially across model variants. Qwen-OPD achieves the strongest overall Pre-CoT calibration, with the highest average AUROC of 0.644 and the highest average PRR of 0.345. Its advantage is especially clear on MATH500, where it obtains an AUROC of 0.734 and a PRR of 0.611. Qwen-OPD also performs strongly on AIME 2024, where it achieves the best AUROC and PRR among all models. On AIME 2025, Qwen-Instruct achieves the highest AUROC, but Qwen-OPD still obtains the best PRR, indicating that its confidence ranking remains more useful for selective prediction.

In contrast, Qwen-RL shows substantially weaker Pre-CoT calibration. Its AUROC falls below 0.5 on all four benchmarks, and its PRR is negative across all benchmarks. This suggests that Qwen-RL confidence before reasoning does not reliably distinguish easy problems from hard ones. In some cases, it may even mis-rank problem difficulty, making low-confidence rejection harmful rather than helpful. Qwen-SFT also does not improve over Qwen-Instruct in the Pre-CoT stage, suggesting that supervised fine-tuning on correct reasoning traces does not necessarily preserve query-level difficulty perception.

Overall, Qwen-OPD provides the best-calibrated Pre-CoT signal, whereas Qwen-RL shows a weaker ability to assess problem difficulty.

4.0.0.2 Intra-CoT: Early Stopping

Figure 2 shows the matched-budget Intra-CoT frontiers, including both standard confidence-based early stopping and the PosConf variants introduced in Section 5.0.0.2. The 100% point corresponds to the full online budget, while moving right applies more aggressive early stopping down to a 30% retained budget.

Standard confidence-based filtering improves the accuracy–compute trade-off for all post-trained models, but the improvement profiles differ across training paradigms. Qwen-SFT gives the clearest Intra-CoT gains on the competition-style benchmarks: from the 100% to 30% budget, accuracy rises from 18.84 to 26.90 on AIME 2024 and from 15.14 to 23.68 on AIME 2025. On AMC 2023, it peaks at 61.63 under a 40% budget, compared with 52.80 at the full budget. The MATH500 gain is smaller but still positive, increasing from 68.54 to 73.17 at the 30% budget.

Qwen-RL also benefits from filtering, but its gains are less stable under very aggressive stopping. On AIME 2024, it improves from 21.59 to 25.13 at the 50% budget, but drops to 22.88 at the 30% budget. On AIME 2025 and AMC 2023, the 30% budget still improves accuracy from 18.76 to 21.74 and from 53.19 to 59.55, respectively. On MATH500, Qwen-RL peaks at 77.06 under a 40% budget and then falls back to 74.83 at 30%.

Qwen-OPD exhibits a more aggressive low-budget frontier. At the 30% budget, it improves from 20.78 to 26.49 on AIME 2024, from 18.49 to 23.46 on AIME 2025, and from 72.62 to 79.17 on MATH500. However, on AMC 2023 it remains below Qwen-SFT and Qwen-RL even after filtering, reaching 54.79 at 30% compared with 60.21 for Qwen-SFT and 59.55 for Qwen-RL. This indicates that OPD confidence can preserve strong traces under tight budgets for some benchmarks.

4.0.0.3 Post-CoT: Answer Aggregation

3pt

Table 2 evaluates whether Post-CoT confidence can improve answer aggregation after complete reasoning traces are generated. Under standard majority voting, Qwen-SFT achieves the highest average accuracy of 59.8%, followed by Qwen-OPD at 58.5% and Qwen-RL at 57.4%. This shows that majority voting primarily reflects the base quality and diversity of sampled reasoning traces, rather than the usefulness of trace-level confidence.

Confidence-based aggregation changes this ranking. Qwen-RL benefits the most from confidence filtering: its Top-10% confidence voting reaches an average accuracy of 62.8%, improving over its own majority-vote baseline by 5.4 percentage points. The gain is especially clear on AIME 2025, where Qwen-RL improves from 29.3% under majority voting to 40.7% under Top-10% confidence voting. Qwen-RL also improves on AIME 2024 and MATH500, indicating that its trace-level confidence is useful for selecting reliable completed traces.

In contrast, Qwen-SFT and Qwen-OPD do not benefit from the same confidence filtering strategy. For Qwen-SFT, both Top-5% and Top-10% confidence voting reduce average accuracy relative to majority voting. For Qwen-OPD, all confidence-based strategies underperform its majority-vote baseline in average accuracy. This suggests that stronger majority-vote accuracy does not necessarily imply better Post-CoT calibration: a model may generate correct traces, but its trace-level confidence may still fail to rank those traces reliably.

Overall, Qwen-RL provides the best-calibrated Post-CoT confidence signal for answer aggregation. Unlike Pre-CoT and Intra-CoT, where Qwen-OPD and Qwen-SFT are more effective, Post-CoT favors Qwen-RL because confidence filtering improves the quality of the retained reasoning traces.

5 Analysis↩︎

The results above suggest that post-training does not simply make confidence more or less reliable. Instead, it changes where confidence is calibrated within the reasoning process. To better understand these differences, we analyze confidence dynamics across token positions and trace-level distributions. This analysis helps explain why the same confidence signal can support Pre-CoT, Intra-CoT, or Post-CoT differently across post-training methods.

5.0.0.1 Position-Dependent Confidence Separation

Figure 3 shows the mean token-level confidence trajectories of correct and incorrect traces across relative token positions. Models with these post-training methods exhibit different patterns.

For Qwen-SFT, correct traces maintain higher confidence than incorrect traces across most of the reasoning process, with a sharper increase near the final-answer region. This pattern is consistent with the SFT objective: token-level cross-entropy directly constrains the next-token distribution along teacher-generated reasoning traces. As a result, local confidence fluctuations remain meaningful during generation, making SFT particularly suitable for Intra-CoT. However, because SFT encourages sampled traces to follow similar teacher-like trajectories, its completed traces can have highly overlapping confidence distributions, as shown in Figure 4. This limits the usefulness of trace-level confidence for Post-CoT answer aggregation.

Qwen-RL shows the opposite pattern. At the beginning of reasoning, correct and incorrect traces are almost indistinguishable, which explains its weak Pre-CoT calibration. After the model commits to a reasoning path, the two trajectories diverge rapidly. The trace-level distribution in Figure 4 further shows a clear separation: correct traces concentrate in a high-confidence region, while many incorrect traces form a lower-confidence tail. This explains why RL benefits substantially from Post-CoT confidence filtering, even though its early-stage confidence is weak.

Qwen-OPD exhibits the most counterintuitive behavior. It starts with well-calibrated early confidence, which is consistent with its strong Pre-CoT performance. However, its confidence trajectories cross in the later part of reasoning: incorrect traces gradually become more confident and eventually surpass correct traces. The trace-level distribution confirms this inverse calibration, where incorrect traces can receive higher confidence than correct ones. This explains why Post-CoT confidence selection hurts Qwen-OPD.

5.0.0.2 PosConf: Confidence Extraction from Reliable Intervals

The trajectory analysis shows that confidence is not equally informative across relative token positions. For SFT, confidence is informative across much of the trace and becomes especially discriminative near the final answer. For RL, initial confidence is noisy, but the signal becomes reliable after a path-commitment phase. For OPD, the initial part of the trace is informative, while later confidence becomes inversely calibrated after the confidence-transition point. These observations explain why uniformly averaging confidence over an entire trace, or using a fixed tail window for all models, can be suboptimal.

To test whether confidence can be recovered from reliable relative-position intervals, we consider a position-aware confidence score, which we call PosConf. For a completed trace \(\tau=(t_1,\ldots,t_m)\), we define \[C_w(\tau) = \frac{\sum_{t=1}^{m} w(t/m)c_t}{\sum_{t=1}^{m} w(t/m)}, \label{eq:position95weighted95conf}\tag{4}\] where \(c_t\) is the token-level confidence at position \(t\), and \(w(\cdot)\) is a position-dependent weighting function. This formulation extracts trace-level confidence from relative-position intervals where the signal is both discriminative and correctly oriented. In our analysis, SFT and RL use tail confidence, while OPD uses initial confidence.

The same principle applies to Intra-CoT. A standard early-stopping rule applies the same threshold throughout generation. This can be misleading when confidence estimates are miscalibrated in certain positions. For RL, applying the threshold too early may stop traces before confidence becomes discriminative. For OPD, applying the threshold after the transition point may preferentially stop correct traces, because incorrect traces have become more confident. Therefore, early stopping should also be position-dependent.

Early stopping is activated only when the relative token position \(\rho_t=t/m\) lies within an active interval \(I\) where confidence is well-calibrated. Once this condition is satisfied, the trace is stopped if its windowed confidence \(\mathrm{WinConf}_w(\tau,t)\) falls below a threshold \(\theta\). We parameterize the reliable intervals as two relative-position intervals: \(\mathcal{I}_{\mathrm{intra}}=[\alpha_{\mathrm{intra}},\beta_{\mathrm{intra}}]\) for Intra-CoT, and \(\mathcal{I}_{\mathrm{post}}=[\alpha_{\mathrm{post}},\beta_{\mathrm{post}}]\) for Post-CoT. Table 3 summarizes the PosConf parameters derived from AIME 2024 trajectory analysis.

The PosConf rows in Table 2 show the resulting Post-CoT aggregation accuracy. For Qwen-RL, PosConf further improves confidence-based aggregation, with PosConf-Top-5% confidence voting reaching the strongest average performance. For Qwen-OPD, extracting confidence from the pre-transition region mitigates inverse calibration. For Qwen-SFT, PosConf scoring remains less effective than majority voting, suggesting that its confidence separation is too weak for aggressive filtering.

PosConf also improves Intra-CoT early stopping, as shown in Figure 2. For Qwen-RL, the 15% skip period avoids premature stopping in the initial phase where correct and incorrect traces are difficult to distinguish from each other. This is most visible when standard early stopping becomes too aggressive: at the 40% budget, PosConf improves AIME 2024 from 22.11 to 26.10 and MATH500 from 77.06 to 78.97; at the 60% budget, it improves AIME 2025 from 20.61 to 24.26. On AMC 2023, the best matched-budget value rises from 59.55 to 60.22 with PosConf.

For Qwen-OPD, activating early stopping only before the confidence-transition region consistently improves low-budget performance. At the 30% budget, PosConf raises accuracy from 26.49 to 28.85 on AIME 2024, from 23.46 to 26.72 on AIME 2025, from 54.79 to 55.73 on AMC 2023, and from 79.17 to 80.09 on MATH500. The gain is especially clear on AIME 2025, where PosConf adds 4.32 points at the 40% budget and 3.26 points at the 30% budget.

Overall, these results show that confidence in reasoning models is a dynamic and position-dependent signal. A model may provide reliable confidence early in generation but become less reliable later, or provide uninformative confidence early while becoming useful after committing to a reasoning path. PosConf offers a practical way to align the use of confidence with where it is most reliable within the reasoning trace.

6 Discussion↩︎

6.0.0.1 Calibration is stage-dependent.

Our results suggest that confidence calibration in reasoning models should not be treated as a single global property of a model. A model can provide useful confidence before reasoning begins, but unreliable confidence after a full trace is generated; conversely, a model can be poorly calibrated at the query level while still producing useful trace-level confidence for answer aggregation. This distinction is important because confidence is used for different inference-time decisions at different stages. Our findings show that these uses are not interchangeable. The same confidence signal can be helpful for one decision and misleading for another.

This stage-dependent view also changes how post-training should be evaluated. Reasoning-oriented post-training is usually judged by final-answer accuracy, but final accuracy alone does not determine whether a model’s confidence is reliable. For example, majority voting accuracy mainly reflects the quality and diversity of sampled traces, whereas confidence-filtered aggregation tests whether the model can identify which completed traces are reliable. Similarly, early stopping requires confidence to be locally meaningful during generation, not only correlated with the final answer after the trace is complete. Therefore, evaluating calibration only at the final output can obscure important differences among post-training methods and variants.

6.0.0.2 Post-training objectives shape where confidence is reliable.

The different behaviors of SFT, RL, and OPD suggest that confidence dynamics are closely tied to the training objective. SFT directly optimizes token-level likelihood on correct teacher-generated reasoning traces. This objective appears to preserve local confidence information throughout generation, which explains why SFT is particularly effective for Intra-CoT early stopping. However, because SFT encourages traces to follow similar teacher-like patterns, completed traces may have overlapping confidence distributions, limiting the usefulness of trace-level confidence for Post-CoT aggregation.

RL produces a different pattern. Since RL is driven by outcome rewards rather than dense token-level supervision, its confidence before reasoning begins is a weak indicator of whether the problem will be solved. This helps explain the poor Pre-CoT calibration of the RL model. However, once the model commits to a reasoning path, correct and incorrect traces become more separable in confidence space. As a result, RL provides the strongest Post-CoT signal: confidence filtering can identify reliable completed traces and improve answer aggregation.

As for OPD, its pre-reasoning confidence is relatively informative, suggesting that distillation can preserve useful query-level uncertainty. However, later in the reasoning trace, OPD confidence can become misleading: incorrect traces may become more confident than correct traces. This inverse calibration explains why naive confidence filtering hurts OPD in Post-CoT aggregation. More broadly, this shows that distillation can transfer or amplify confidence patterns that are useful in one region of the reasoning process but unreliable in another.

7 Conclusion↩︎

We introduced a three-stage framework for evaluating confidence calibration in reasoning models before, during, and after chain-of-thought generation. Our results show that post-training induces stage-dependent confidence shifts: OPD is most effective for Pre-CoT difficulty estimation, SFT for Intra-CoT early stopping, and RL for Post-CoT answer aggregation.

We further find that confidence reliability is position-dependent within reasoning traces. Motivated by this, PosConf extracts confidence from reliable relative-position intervals, improving confidence-based early stopping and aggregation. These findings highlight the need to evaluate confidence not as a global model property, but as a stage-specific signal tied to different decisions.

Limitations↩︎

Our study focuses on mathematical reasoning benchmarks and models derived from a single Qwen2.5-7B-Instruct backbone. This controlled design helps isolate the effect of post-training paradigms, but future work should test whether the same stage-dependent calibration patterns hold across larger model scales, different model families, and non-mathematical reasoning tasks. In addition, we focus on internal confidence derived from token probabilities. Other uncertainty signals, such as entropy, probability margins, consistency across samples, or hybrid confidence estimators, may capture complementary aspects of reasoning reliability.

8 Related Work↩︎

8.0.0.1 Chain-of-Thought Reasoning.

Chain-of-thought (CoT) prompting [4] demonstrated that providing step-by-step reasoning examples enables large language models to solve complex problems. Zero-shot variants [38] and scratchpad-based approaches [39] further showed that reasoning capabilities can be elicited without task-specific exemplars. Self-consistency [18] improved reliability by sampling multiple reasoning paths and aggregating answers through majority voting, establishing a connection between sampling diversity and answer quality. More structured approaches, such as Tree of Thoughts [40], extend chain-of-thought reasoning by exploring multiple reasoning branches. These works focus on how to generate or structure reasoning traces; our work complements them by studying how confidence signals along these traces can be used for inference-time decisions.

8.0.0.2 Reasoning-Oriented Post-Training.

Several post-training paradigms have been developed to improve reasoning in LLMs. Supervised fine-tuning on chain-of-thought traces, using data generated by stronger models or bootstrapped from the model itself [5], [6], [41], [42], directly trains models to produce step-by-step solutions. Reinforcement learning with outcome-based rewards [7], [10], [11], [30], [31] optimizes reasoning through trial and error, rewarding correct final answers without requiring step-level supervision. On-policy distillation [33], [34], [43], [44] combines elements of both by distilling token-level supervision from a teacher model onto student-generated rollouts. Process reward models [45]–[47] provide step-level feedback but require additional verifier training. Prior comparisons of these paradigms have focused on final-answer accuracy [41], [48]. Our work differs by comparing how each paradigm reshapes the model’s internal confidence signal across different stages of reasoning.

8.0.0.3 Calibration and Confidence Estimation in LLMs.

Calibration measures whether a model’s expressed confidence aligns with its actual accuracy [12], [49], [50]. Classical post-hoc approaches such as temperature scaling [12] and Platt scaling [50] were developed for classification tasks with well-defined output probabilities, while more recent work has revisited their effectiveness in modern deep networks [51]. For LLMs, confidence estimation has been studied through both internal signals and verbalized outputs. Internal approaches extract confidence from token logits, hidden states, or predictive entropy [13], [22], [52]. Verbalized approaches elicit confidence through prompting strategies [23]–[25]. [53] showed that pre-trained transformers exhibit miscalibration that persists after fine-tuning, and [54] studied how calibration evolves across model layers. Recent surveys [26], [27] provide comprehensive overviews of this area. [14] showed that hidden representations before generation can estimate question difficulty. [16] introduced group-level token confidence for reasoning models. Our work extends calibration analysis to reasoning models by introducing a stage-wise framework that connects confidence to specific inference-time decisions, rather than measuring calibration as a single aggregate property.

8.0.0.4 Test-Time Compute Scaling.

Recent work has shown that allocating more compute at inference time can substantially improve reasoning performance. [20] demonstrated that scaling test-time compute through repeated sampling and selection can be more effective than scaling model parameters. [19] showed that increasing the number of sampled solutions follows predictable scaling laws for problem coverage. [21] analyzed compute-optimal inference strategies for problem-solving. These approaches rely on mechanisms such as majority voting [18], [55], best-of-\(N\) selection with reward models [45], [56], and adaptive compute allocation. Entropy-based signals have also been explored for guiding RL training [57] and identifying critical reasoning tokens [58]. Our three-stage framework provides a structured view of when internal confidence supports different test-time strategies: Pre-CoT for difficulty-based routing, Intra-CoT for early termination, and Post-CoT for confidence-based answer aggregation.

9 Training Details↩︎

9.0.0.1 Backbone and training data.

All model variants are derived from Qwen2.5-7B-Instruct. Qwen-Instruct is the unmodified instruction-tuned backbone. Qwen-SFT, Qwen-RL, and Qwen-OPD are trained on the same reasoning-oriented data mixture based on DeepScaler and SimpleRL, so that differences in confidence behavior are primarily attributable to the post-training objective rather than model family or data domain.

9.0.0.2 Supervised fine-tuning.

Qwen-SFT is trained with full-parameter supervised fine-tuning using LlamaFactory. The supervised targets are long chain-of-thought traces generated by DeepSeek-R1-Distill-Qwen-32B and retained only when the extracted final answer is correct. We use a cutoff length of 16,384 tokens, learning rate \(1\times10^{-6}\), two epochs, per-device batch size 1, gradient accumulation 8, cosine learning rate schedule, warmup ratio 0.1, maximum gradient norm 1.0 and bf16 training.

9.0.0.3 Reinforcement learning.

Qwen-RL is trained with group relative policy optimization using verifiable answer rewards. The RL run uses maximum response length 16,384, train batch size 16, rollout count 8, KL coefficient \(1\times10^{-4}\), entropy coefficient \(1\times10^{-3}\), rollout tensor parallel size 2, and 8 H20 GPUs. The reward is based on whether the extracted final answer matches the ground truth.

9.0.0.4 On-policy distillation.

Qwen-OPD uses on-policy distillation from DeepSeek-R1-Distill-Qwen-32B. The student first generates its own rollouts, and the teacher provides dense token-level supervision on those rollouts. We use the GOLD-style on-policy setup with \(\lambda=1.0\) and universal logit distillation to support teacher–student tokenizer differences.

10 Inference and Evaluation Protocol↩︎

10.0.0.1 Generation setup.

For each model and benchmark problem, we generate multiple reasoning traces with temperature 0.6, top-\(p\) 0.95, and maximum generation length 32,768 tokens. The generation scripts use an offline trace budget of 320 traces per problem. Unless otherwise stated, aggregation experiments evaluate a fixed budget of 256 traces per problem such that majority voting and confidence-filtered voting are compared under the same sampled-trace budget. We use top-\(k=20\) next-token log-probabilities for all confidence computations.

10.0.0.2 Repeated evaluation.

To reduce variance from stochastic generation, we repeat each experiment five times with different random seeds. Each repetition independently generates a new set of reasoning traces and runs the same extraction, aggregation, and evaluation pipeline. Unless otherwise stated, we report the mean performance over the five repetitions.

10.0.0.3 Prompt template.

All models use the same chat-style prompt. The user message appends the instruction: “Please reason step by step, and put your final answer within \boxed{}.” This keeps answer formatting consistent across models and benchmarks.

10.0.0.4 Answer extraction.

We extract the final answer from each generated trace and compare it with the benchmark ground truth. When mathematical equivalence checking is available, we use it before falling back to normalized exact match. Invalid, missing, or unparseable answers are counted as incorrect. The same extraction and evaluation pipeline is used for all models.

10.0.0.5 Pre-CoT labels.

For each model–problem pair, we label a problem as solved if the majority-voted answer over sampled traces matches the ground truth, and unsolved otherwise. This makes difficulty labels model-specific: a problem may be easy for one post-training method and hard for another. We then evaluate whether first-token confidence ranks solved problems above unsolved problems using AUROC and PRR.

10.0.0.6 Intra-CoT sweep.

For online early stopping, we compute token confidence from the top-\(k\) next-token probabilities and smooth it with a sliding window. A trace is stopped when the windowed confidence falls below a threshold. Thresholds are derived from warmup traces and increased to produce matched-budget token–accuracy frontiers. For each problem, we construct \(M=50\) thresholds between the minimum warmup window confidence and the 100 percentile of warmup window confidence. We then select the operating points whose observed retained-token ratios are closest to token budgets in Figure 2. In each sweep, we use 32 warmup traces to estimate the minimum confidence and adaptive window size, evaluate 256 online samples, and repeat the experiment with 5 independent resamples unless otherwise specified. To avoid future-information leakage, the online Intra-CoT variant does not normalize token positions by the final length of the current trace. Instead, it estimates an expected trace length \(\hat{m}_x\) from warmup traces and uses the online relative position \(\hat{\rho}_t=\min\{t/\hat{m}_x,1\}\) when activating position-restricted stopping rules. We set the adaptive divisor to \(D=10\), so that the window size is \(w_x=\max\{1,\mathrm{round}(\bar L_x/D)\}\), where \(\bar L_x\) is the mean warmup trace length for problem \(x\).

10.0.0.7 Post-CoT aggregation.

For answer aggregation, majority voting uses all sampled valid answers. Confidence filtering first ranks traces by trace-level confidence, keeps the top 5% or top 10%, and then performs weighted majority voting over the retained answers. We also report most-confident selection, which returns the answer from the single highest-confidence trace. We repeat the post-CoT aggregation experiment with 5 independent subsamples and report the averaged results unless otherwise specified.

11 Additional Metric Definitions↩︎

11.0.0.1 AUROC.

Let \(s_i\) be a confidence score where larger values indicate higher confidence. Let \(\mathcal{P}\) and \(\mathcal{N}\) denote correct and incorrect traces, respectively. We compute trace-level AUROC as \[\begin{align} \mathrm{AUROC} &= \frac{1}{|\mathcal{P}||\mathcal{N}|} \sum_{i\in\mathcal{P}}\sum_{j\in\mathcal{N}} a_{ij}, \\ a_{ij} &= \mathbf{1}\{s_i>s_j\} +\frac{1}{2}\mathbf{1}\{s_i=s_j\}. \end{align}\] An AUROC above 0.5 means confidence tends to rank correct traces above incorrect traces; an AUROC below 0.5 indicates inverse calibration.

11.0.0.2 Prediction-Rejection Ratio.

For selective prediction, sort examples by decreasing confidence and let \(\mathrm{Acc}(r)\) be the accuracy of the top \(r\) retained examples. We define the ranked area under the selection curve as \[\mathrm{AUC}_{\mathrm{rank}} = \frac{1}{n}\sum_{r=1}^{n}\mathrm{Acc}(r).\] Let \(\mathrm{AUC}_{\mathrm{rand}}\) be the expected area under random rejection and \(\mathrm{AUC}_{\mathrm{oracle}}\) be the area obtained by ranking all correct examples before incorrect examples. The prediction-rejection ratio is \[\mathrm{PRR} = \frac{ \mathrm{AUC}_{\mathrm{rank}}-\mathrm{AUC}_{\mathrm{rand}} }{ \mathrm{AUC}_{\mathrm{oracle}}-\mathrm{AUC}_{\mathrm{rand}} }.\] PRR is therefore a normalized area for the selective prediction curve. Negative PRR indicates that confidence ranking is worse than random rejection.

11.0.0.3 Supplementary figures.

In addition to the AIME 2024 trajectory and histogram figures in the main text, we report the same diagnostics for AIME 2025, AMC 2023, and MATH500. Together, these figures show that the position-dependent pattern is consistent across all benchmarks.

References↩︎