¹

1 Introduction↩︎

Recently, large reasoning models (LRMs) have rapidly emerged and made substantial progress on complex natural language processing (NLP) tasks, as exemplified by OpenAI-o1 [1] and DeepSeek-R1 [2]. These models are equipped with the ability to generate long reasoning chains, demonstrating strong potential on challenging reasoning problems such as mathematical competitions [3]. However, while LRMs heavily rely on long chain-of-thought (CoT) traces to perform well on difficult tasks, they tend to produce redundant reasoning and self-reflection for simple inputs, incurring pronounced overthinking and token overhead [4]–[7]. This behavior leads to verbose thought chains that increase computation cost and reduce overall inference efficiency. Accordingly, how to enable LRMs to dynamically adjust token consumption based on the input difficulty has attracted increasing attention, determining the practical industrial usability of LRMs in terms of the balance between accuracy and latency [8].

Most of the existing approaches focus on reasoning compression and length control predominantly, which treat shortening reasoning chains as the primary objective [9] and apply a uniform reduction of reasoning tokens to all the queries [4], [10]–[12]. While such methods can substantially decrease generation length, they often incur non-trivial performance degradation on difficult problems, since complex tasks still require sufficient reasoning depths and lengths to sustain accurate answers [13], [14]. Another line of work resorts to difficulty-adaptive reasoning to mitigates the imbalance between overthinking for easier instances and underthinking for harder ones. This category of methods tends to dynamically adjust the budget of output tokens based on the model performance [8].

However, existing works on adaptive reasoning still face a severe challenge of coarse-grained difficulty estimation. Current methods utilize the accuracy of model outputs to measure the problem difficulty and roughly determine the output length [8]. We argue that this coarse-grained estimation heavily relies on external labels and provides a partial assessment merely on the answer, rather than measuring the quality of the whole reasoning chains generated by LRMs.

To address this limitation, we propose CAT (Confidence-Adaptive Thinking), an adaptive reasoning framework driven by the model’s intrinsic confidence. Inspired by recent works on the quality estimation from the model’s internal token distributions [15]–[17], our main idea is to leverage self-certainty [18] as the intrinsic fine-grained indicator to distinguish high-quality reasoning trajectories from erroneous ones. Firstly, CAT employs self-certainty as the model’s intrinsic confidence metric to estimate the quality of generated reasoning trajectories, which reflects the problem difficulty. Based on the separation of confidence and lengths between different trajectories, we further construct preference data to make the model capture the relationship between problem difficulties and output lengths. Secondly, we devise a confidence-weighted preference optimization (CWPO) method, which weights the vanilla preference optimization objective with confidence. This encourages the model to compress reasoning steps under high confidence while retaining necessary exploration otherwise, thereby mitigating overthinking for simple cases and maintaining reasoning performance especially for hard ones.

In summary, our main contributions are²:

• We introduce the confidence-adaptive thinking (CAT) framework that shifts the paradigm of efficient reasoning from external supervision to intrinsic confidence awareness. CAT enables reasoning models to autonomously perceive problem difficulty and modulate their thinking depth.

• We propose the confidence-weighted preference optimization (CWPO) objective that dynamically weights the vanilla objective based on the calibration ratio of confidence to length. CWPO mitigates overthinking while preserving the model’s ability to explore complex reasoning paths if necessary.

• We conduct extensive experiments across three challenging benchmarks and show superior performance of CAT over state-of-the-art baselines on the balance between inference efficiency and reasoning accuracy.

2 Related Work↩︎

Efficient Reasoning in LRMs. Recent studies have increasingly focused on the phenomenon of overthinking in large reasoning models [7], [19], [20]. Existing efficient reasoning methods can generally be categorized into two streams. The first involves training strategies to equip LRMs with the ability to generate concise reasoning chains, spanning from supervised fine-tuning [10], [21] to reinforcement learning [8], [22]–[24]. The second category comprises inference-time methods, including prompting [25]–[27], task routing [28], [29], latent space compression [30], [31], and dynamic decoding [32], [33].

Compared with existing works on training methods of efficient reasoning, our work utilizes the model’s confidence as the estimation of problem difficulty, instead of solely depending on external reward models and extrinsic metrics. This makes the full usage of the model’s intrinsic property to achieve adaptive reasoning.

Confidence Utilization in LRMs. Recent works have shown that the model confidence potentially indicate the quality of reasoning chains [15]–[18]. As one of the representative metrics to reflect confidence, self-certainty [18] has been primarily applied to Best-of-N selection [15]. For comparison, our work uses self-certainty as the model’s confidence to self-evaluate the quality of generated reasoning chains, which guides the adaptive thinking via preference optimization, instead of merely injecting it into the inference stage.

3 Methodology↩︎

3.1 Task Definition and Method Overview↩︎

Given an input question \(x\), our goal is to acquire a reasoning trajectory \(y\) that contains a multi-step reasoning process and a final answer. Under the precondition of accuracy, \(y\) is required to become short for simple problems while being long for hard ones if necessary.

An overview of our framework is presented in Figure 1. Firstly, we sample multiple reasoning trajectories for each question and compute their path-level self-certainty scores as confidence via a dedicated forward pass (Section 3.2.1). Secondly, we construct preference pairs based on the confidence and lengths, and apply dynamic selection to prioritize more informative supervision (Section 3.2.1). Finally, we fine-tune the base LRM with a confidence-weighted preference optimization objective, which incorporates confidence and lengths to further modulate the preference strength, achieving conditional length regulation (Section 3.2.2).

3.2 Confidence-Adaptive Thinking↩︎

Our confidence-adaptive thinking framework consists of two stages, including confidence-aware preference labeling and confidence-weighted preference optimization. While the first stage aims to incorporate confidence as intrinsic signals to construct fine-grained preference pairs, the second stage further utilizes confidence to further enhance the preference optimization objective.

3.2.1 Confidence-Aware Preference Labeling↩︎

To build the preference dataset, we first sample \(K\) reasoning trajectories \(\{y^{(k)}\}_{k=1}^{K}\) for the question \(x\) from the base reasoning model, each of which is a token sequence \(y^{(k)}=\left(y^{(k)}_1,\dots,y^{(k)}_{n_k}\right)\) with the length of \(n_k\). The goal of this stage is to construct a preference dataset \(\mathcal{D}=\{(x, y_w, y_l, s)\}\), where \(y_w\) and \(y_l\) denote the winning and losing trajectories for the same input \(x\), and \(s\) indicates the confidence-calibrated preference score.

3.2.1.1 Self-Certainty as Intrinsic Confidence.

To capture the model’s intrinsic confidence during reasoning, we follow [18] to employ self-certainty, which can also serve as a trajectory-level quality measure. Formally, assuming that \(\boldsymbol{p}_\theta(\cdot \mid x, y_{\le i})\) denotes the next-token distribution at \(i\)-th position, \(V\) indicates the vocabulary size, and \(\mathcal{U}\) represents the uniform distribution over \(V\), self-certainty (SC) can be computed as follows: \[\label{eq:sc95definition} \text{SC}(x,y)\;=\;-\frac{1}{nV}\sum_{i=1}^{n}\sum_{j=1}^{V}\log\!\Big(V\cdot \boldsymbol{p}_\theta(j\mid x,y_{\le i})\Big)\tag{1}\] which corresponds to measuring the KL divergence \(D_{\mathrm{KL}}\!\bigl(\mathcal{U}\,\|\,p_\theta(\cdot\mid x, y_{\le i})\bigr)\) and averaging this quantity over \(i\). Intuitively, a larger divergence from the uniform distribution implies a more peaked (and thus more certain) predictive distribution, leading to higher SC. Conversely, a distribution closer to uniform is flatter, indicating greater uncertainty and yielding lower SC.

3.2.1.2 Preference Pair Construction.

We consider three important factors of each trajectory to construct the preference dataset: (i) the correctness of the answer, (ii) the length, and (iii) the intrinsic confidence based on SC in Eq. (1 ). We emphasize that SC is a complementary to external factors, which estimates trajectory-level fine-grained qualities and determines the strength of pairwise preferences.

Inspired by [8], we categorize preference pairs into two types: Conciseness Pairs (CPs), formed by two correct trajectories where the preferred one is shorter; and Deliberation Pairs (DPs), formed by two incorrect traces where the preferred one is longer. Unlike prior approaches that calibrate preference strength using per-question fixed budgets or external difficulty estimation, CAT uses only model-internal evidence to modulate pairwise preference score \(s\).

For each input question \(x\) and its \(K\) candidate reasoning paths, we consider the margin between both lengths and self-certainty to acquire the preference score as \(s\). Specifically, given a candidate pair \((x, y_w, y_l)\), we first compute the margin in terms of self-certainty, lengths, and correctness: \[\begin{align} \Delta r & \;=\; r(y_w) - r(y_l) \\ \Delta \mathrm{SC} & \;=\; \mathrm{SC}(x,y_w) - \mathrm{SC}(x,y_l) \end{align}\] where \(\mathrm{SC}(\cdot)\) can be acquired by Eq.(1 ) and \(r(\cdot)\) is a factor with respect to reasoning lengths and correctness: \[\label{eq:path95score} r(y) = \begin{cases} +\frac{1}{|y|} & \text{if y is correct} \\ -\frac{1}{|y|} & \text{if y is incorrect} \end{cases}\tag{2}\] This design assigns the highest reward to short, correct paths while imposing the lightest penalty on long, incorrect paths. Conversely, short but incorrect paths receive the most severe penalty.

For CPs, our intent is to favor short and confident solutions and reject long and unconfident ones. We therefore multiply \(r\) and \(\mathrm{SC}\) so that a pair receives stronger strength precisely when the winning path \(y_w\) is not only much more efficient but also more internally decisive: \[s_{\text{CP}}(x,y_w,y_l) \;=\; \Delta r \cdot \Delta \mathrm{SC} \label{eq:cp95score}\tag{3}\]

For DPs, we want to prefer long and unconfident attempts over short and confident failures, discouraging premature yet decisive mistakes. Accordingly, we reverse the confidence term, making wrong trajectories with larger certainty receive stronger penalties: \[s_{\text{DP}}(x,y_w,y_l) \;=\; \Delta r \cdot \big(-\Delta \mathrm{SC}\big) \label{eq:dp95score}\tag{4}\]

In both cases, larger scores of preference pairs indicate potentially stronger and more discriminative preference signals for subsequent optimization. Thus, we devise a Dynamic Pruning strategy to select the preference optimization dataset based on \(s\). Concretely, for each query \(x\), we rank the CP and DP sets by their score \(s\) in descending order, respectively, and retain only those pairs whose scores fall within the top three highest score levels. We then pool candidates from all the queries and sort them globally based on the preference score \(s\), truncating the list by removing the bottom \(\tau\) fraction, where \(\tau\) denotes the truncation ratio. Finally, to prevent over-representing queries that produce many high-scoring pairs, we enforce a per-query cap and retain at most one CP and one DP per query in the final preference dataset.

max width=,center

3.2.2 Confidence-Weighted Preference Optimization (CWPO)↩︎

To adjust the model’s reasoning depth conditionally on its internal certainty, rather than applying a uniform length bias to all the samples, we propose the Confidence-Weighted Preference Optimization (CWPO) objective that pioneers the use of intrinsic self-certainty directly within the alignment loss landscape. Compared with the vanilla SimPO objective [34], we dynamically modulated the scaling factors of the winning and losing terms. Formally, the CWPO loss is computed as:

\[\label{eq:CWPO95loss} \begin{align} &\mathcal{L}_{\text{CWPO}}(\pi_\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \Bigl[ \log \sigma\Bigl(\\ &\tfrac{\beta_w}{|y_w|}\log \pi_\theta(y_w|x) -\tfrac{\beta_l}{|y_l|}\log \pi_\theta(y_l|x) -\gamma \Bigr) \Bigr] \end{align}\tag{5}\] where the dynamic weights \(\beta_w\) and \(\beta_l\) are acquired by the original \(\beta_{base}\) in SimPO and a calibration ratio (\(\rho\)) based on self-certainty and lengths: \[\rho(x,y) \;=\; \frac{\text{SC}(x,y)}{|y|^{\alpha}} \label{eq:calibration95ratio}\tag{6}\] where \(\alpha \in (0,1)\) is a length-aware exponent to keep \(\mathrm{SC}\) and \(|y|\) on a comparable scale for numerical stability. This ratio imposes an additional tunable length penalty so that confidence-guided scaling can better align gradient allocation with efficiency.

The CWPO loss sets different weights for conciseness pairs (CPs) and deliberation pairs (DPs): For CPs, where the model compares two correct responses, we define \(\beta_w = \beta_{base} \cdot \sigma(\rho(x,y_w))\), while symmetrically scaling the loser’s weight using the inverse ratio \(\beta_l = \beta_{base} \cdot \sigma(\rho(x,y_l)^{-1})\). This specifically incentivizes the model to commit to reasoning paths that are both correct and concise. Conversely, for DPs, we focus on penalizing short and erroneous answers with unearned confidence. We set the penalty weight \(\beta_l = \beta_{base} \cdot \sigma(\rho(x,y_l))\) and the winner’s weight \(\beta_w = \beta_{base} \cdot \sigma(\rho(x,y_w)^{-1})\). By integrating these internal signals, CWPO moves beyond static length penalties, allowing the model to autonomously judge when to compress reasoning and when to deliberate, achieving a balance between efficiency and accuracy.

4 Experiments↩︎

4.1 Settings↩︎

Models and Datasets. We conduct comparative experiments on three LRMs: DeepSeek-R1-Distill-Qwen-7B (R1-7B) / 1.5B (R1-1.5B) [2] and Qwen3-8B [35]. For the training dataset, following [36], we randomly select 2,000 questions from the MATH training set [37], maintaining diversity in both difficulty and response length.

Benchmarks. We follow [8] to select three benchmarks, including MATH-500 [38], AIME24 [39], and GPQA [40].

Baselines. We select several state-of-the-art methods for efficient reasoning as baselines, including OverThink [4], DAST [8], and ConCISE [36]. For ConCISE, we choose the best-performing alternative ConCISE\(_{\text{SimPO}}\) as the comparison baseline. All of these methods are under the paradigm of preference optimization with the SimPO objective.

Implementation Details. Following [8], we generate 20 candidate responses per question in our training set, and set the maximum sequence length to 4,096 tokens. Based on the hyperparameter analysis in Appendix 7.2, the truncation ratio (\(\tau\)) was set to 0.15. The preference optimization is conducted within the SimPO framework [34]. We adopt low-rank adaption (LoRA) [41] with the rank of \(r=32\), scaling factor of \(\alpha=64\), dropout rate of 0.05. The training epoch is 1 while the batch size is 16. The learning rate is 5e-5 for DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, and 5e-6 for DeepSeek-R1-Distill-Qwen-1.5B, as the weaker 1.5B backbone yields more DPs after CAPL and thus benefits from more conservative optimization. All the experiments are conducted on 2 NVIDIA A800 GPUs. More training details are provided in Appendix 6.1. Decoding processes are executed using the the OpenR1 evaluation scripts [42], with comprehensive decoding details provided in Appendix 6.2. The experimental results are presented with mean values over 3 runs.

4.2 Results and Analysis↩︎

4.2.1 Overall Results↩︎

The results in Table ¿tbl:tab:main? show that CAT achieves the highest accuracy (exceeding the backbone model) on all the three benchmarks while maintaining an acceptable compression rate (CR), suggesting that CAT can allocate reasoning steps adaptively to obtain better performance. Although OverThink and ConCISE attain the most substantial compression rates, they still incur an unavoidable loss in accuracy relative to the backbone model. DAST and CAT exhibit similar balancing trends between task performance and compression, as both aim to achieve adaptive compression while preserving the model’s reasoning capability. Compared with DAST, CAT delivers higher for all the base models, and achieves higher CR and C-CR in most settings. These results suggest that CAT is more effective at adaptive reasoning, demonstrating effectiveness of our proposed confidence-aware adaptive reasoning approach based on the model’s intrinsic signals.

4pt max width=

4.2.2 Ablation Study↩︎

To assess the key components in CAT, including Confidence-Aware Preference Labeling (CAPL) and Confidence-Weighted Preference Optimization (CWPO), we conduct an detailed ablation study by removing either CAPL (w/o CAPL, where preferences are scored only by \(\Delta r\)) or CWPO (w/o CWPO, where we replace CWPO with vanilla SimPO). The results in Table ¿tbl:tab:ablation95CAT? show that all these parts contribute to the final performance. We observe that constructing preference pairs solely based on length differences (w/o CAPL) yields a higher compression ratio but leads to a larger degradation in reasoning performance on most tasks. This observation highlights the importance of high-quality training data that provides difficulty-adaptive reasoning signals.

Due to the paper limit, we further explore the effect of Self-Certainty on CAT in Appendix 7.1.

4.2.3 Analysis of Self-Certainty↩︎

To better understand how self-certainty helps the model achieve an optimal balance between reasoning accuracy and length, we conduct a detailed analysis on the reasoning trajectories generated by Qwen3-8B of the MATH dataset.

SC effectively distinguishes correct and incorrect reasoning paths. In Figure 2 (a), we analyze the distribution of self-certainty for correct and incorrect responses on MATH (Level 4). The distributions for correct and incorrect responses concentrate around distinct means, with correct responses exhibiting a higher mean. This suggests that SC can effectively distinguish correct from incorrect reasoning trajectories and is strongly correlated with response quality.

SC is robust to reasoning lengths. We analyze self-certainty across responses of varying lengths. As illustrated in Figure 2 (b), SC is not noticeably affected by response lengths: across the entire length range, the median (blue line) shows only a very slight downward trend, which is largely attributable to the fact that shorter responses contain a higher proportion of correct trajectories, indicating that SC is stable regardless of reasoning lengths.

4.2.4 Generalization Across Preference Optimization Methods↩︎

To test the generalization ability of our method, we further apply our method to DPO in addition to SimPO, and assess the performance on DeepSeek-R1-Distill-Qwen-7B using the same preference pairs constructed by CAPL. The DPO-version CWPO objective (denoted as CWPO\(_{\text{DPO}}\)) is slightly different from vanilla CWPO, which is detailed in Appendix 8. The results in Table ¿tbl:tab:CWDPO? indicate that CWPO\(_{\text{DPO}}\) beats standard DPO in most of the metrics on three benchmarks, demonstrating the promising generalization ability to different preference optimization methods.

4pt max width=

4.2.5 Case Study↩︎

To intuitively illustrate how CAT affects reasoning behaviors, we present a case study on DeepSeek-R1-Distill-Qwen-7B in Figure 3. We observe that all the three methods reach the correct answer but exhibit different reasoning lengths and self-certainty. The backbone model repeatedly verifies the same inequalities and explicitly checks invalid alternatives. DAST reduces the reasoning length with lower self-certainty, while still retaining additional verification beyond the core derivation. In comparison, CAT achieves the shortest reasoning chain with higher self-certainty. It does not eliminate reflection entirely: after deriving the feasible interval, it only keeps a brief validity check rather than exploring invalid candidates. This qualitative case supports the design of CAT, which incorporates self-certainty together with correctness and length signals to favor concise and confident correct reasoning paths.

5 Conclusion↩︎

This work proposes confidence-adaptive thinking (CAT), which addresses the pronounced overthinking and token overhead in large reasoning models through intrinsic confidence awareness. CAT integrates self-certainty as LRMs’ intrinsic confidence to enable them to compress confident responses while deliberating on uncertain ones. Extensive experiments demonstrate that CAT consistently achieves a superior balance between inference efficiency and accuracy.

Limitations↩︎

While CAT demonstrates a superior balance between reasoning accuracy and efficiency, we identify the following areas for future improvement:

5.0.0.1 Path-Level Aggregation.

Our current framework utilizes path-level Self-Certainty to score reasoning traces. While this metric effectively differentiates high-quality responses, aggregating token-level signals into a single scalar for the entire sequence may overlook variations in confidence at specific reasoning steps. Future work could explore integrating token positions with their specific Self-Certainty scores to enable more precise step-level compression.

5.0.0.2 Domain-Specific Evaluation.

Our experiments focus on STEM disciplines such as mathematics and physics that allow rigorous correctness verification. Although Self-Certainty is an intrinsic signal independent of ground truth, our preference labeling strategy currently utilizes verification results. We aim to extend this approach to open-ended generation tasks where Self-Certainty can guide alignment without reliance on external answers.

5.0.0.3 Offline Optimization Paradigm.

CAT employs Confidence-Weighted Preference Optimization on static datasets constructed from pre-sampled trajectories. This offline setting limits the ability of the policy to dynamically update its confidence estimates during the training process. Future research will investigate transitioning from offline optimization to online reinforcement learning variants, allowing the model to iteratively refine its reasoning efficiency through continuous interaction.

Acknowledgments↩︎

This work was supported by Sichuan Science and Technology Program (2025ZNSFSC1488), Noncommunicable Chronic Diseases-National Science and Technology Major Project (2023ZD0501806), Fundamental Research Funds for the Central Universities (ZYGX2025XJ041), and CIPS-SMP-Zhipu Large Model Fund (CIPS-SMP20250314).

6 Experimental Details↩︎

6.1 Training details↩︎

6.1.0.1 Training Set.

While the construction methodology of the training set is detailed in Section 4, Figures 4 (a) and  4 (b) illustrate its diversity in terms of difficulty and length. Specifically, the difficulty distribution aligns with the Level metric of the 2000 selected questions from the MATH dataset. The length distribution reflects the reasoning chains generated by DeepSeek-R1-Distill-Qwen-7B, obtained by sampling 20 paths per question with a temperature of 0.6 and a top_p of 0.95.

6.1.0.2 Training Configurations.

Following Dynamic Pruning with a truncation ratio of \(\tau=0.15\), our constructed preference pairs for R1-7B yielded 1,765 Conciseness Pairs (CPs; 93.1%) and 130 Deliberation Pairs (DPs; 6.9%), a distribution similar to that reported in DAST [8]. Similarly, for R1-1.5B, we identified 1,742 CPs (88.8%) and 221 DPs (11.2%). For Qwen3-8B, the resulting dataset comprised 1,517 CPs (97.4%) and 40 DPs (2.6%). These results indicate that for models with stronger reasoning capabilities, such as Qwen3-8B, Confidence-Aware Preference Labeling generates a higher proportion of CPs to facilitate the learning of conciseness. Conversely, for models with weaker reasoning abilities, such as R1-1.5B, the method produces more DPs to encourage cautious exploration. Furthermore, following DAST, the original SimPO hyperparameters were set to \(\beta=200\) and \(\gamma=1\) for all three models. All baselines use comparable data budgets to CAT.

6.2 Evaluation Details↩︎

In our evaluation setup, we use a unified decoding configuration for all experiments, with temperature = 0.6 and top_p = 0.95 [2]. The maximum generation length is capped at 32,768 tokens for all the models.

7 Additional Experiments↩︎

7.1 Ablation Study on Self-Certainty↩︎

4pt max width=

To further investigate the effects of Self-Certainty (SC) in CAT, we conduct an additional ablation study to remove SC in CWPO or CAPL & CWPO, respectively. Note that removing SC in CAPL only is equivalent to w/o CAPL in Table ¿tbl:tab:ablation95CAT?, which is not repeatedly analyzed in this section.

Table ¿tbl:tab:ablation95SC? shows that SC matters in both data construction and model optimization. Although removing SC commonly brings a higher compression rate, the reasoning accuracy largely degrades on all the benchmarks. This directly supports that SC contributes non-redundant value to our method.

7.2 Analysis of Hyperparameters↩︎

To examine the effects of the Length-Aware Exponent(\(\alpha\)) and the Truncation Ratio (\(\tau\)), we performed a grid search on MATH-500 using the R1-7B model for validation. As shown in Figure 5, R1-7B achieves the optimal results when \(\alpha\)=0.5 and \(\tau\)=0.15. In both single-parameter sweeps, these settings yield the best Acc and compression ratio. For the other backbone models, we conducted the same hyperparameter search and selected the final hyperparameters accordingly.

7.3 Validation of SC as a Difficulty Indicator↩︎

Because MATH Level 4 does not provide a sufficiently diverse difficulty distribution, we analyze SC across different difficulty levels on the training set. As shown in Figure 7, as the difficulty level increases, the average SC over all trajectories (including both correct and incorrect ones) exhibits a clear downward trend. This suggests that, beyond separating correct from incorrect trajectories, SC also captures differences in problem difficulty. This property is central to CAT: it enables difficulty-aware control.

7.4 Performance Analysis Across Difficulty Levels↩︎

We analyze the performance of CAT, DAST, and CONCISE on the MATH-500 dataset using the R1-7B model across varying difficulty levels. As shown in Figure 6. CAT achieves the highest accuracy in the two most challenging difficulty tiers, demonstrating a substantial advantage over other baselines and highlighting its robust reasoning capabilities for complex problems. While DAST also exhibits difficulty adaptability, it underperforms CAT in both accuracy and length compression. Furthermore, although CONCISE achieves the most significant length reduction, its reasoning performance deteriorates sharply as problem difficulty increases, creating a marked gap compared to the other methods and indicating a lack of capability in handling complex reasoning tasks.

8 Details of CWPO\(_{\text{DPO}}\) Objective↩︎

To examine whether the benefits of CWPO-style SC weighting can generalize beyond the original CWPO setting, we incorporate the same SC-based dynamic weighting strategy into the DPO objective. The resulting objective, denoted as CWPO\(_{\text{DPO}}\), is formulated as follows: \[\label{eq:CWDPO95loss} \begin{align} &\mathcal{L}_{\text{CWPO_{\text{DPO}}}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \Bigl[ \log \sigma\Bigl(\\ &\beta_w \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} -\beta_l \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \Bigr) \Bigr] \end{align}\tag{7}\] Here, we use the same definitions of \(\beta_w\) and \(\beta_l\) as in Section 3.2.2.

References↩︎

---

1. \(^*\)Corresponding author.↩︎

2. Our codes are available at https://github.com/Jiang9732/CAT-code.↩︎