1 Introduction↩︎

Reasoning-focused language models, such as DeepSeek-R1 [1], achieve strong performance through extended chains of thought. However, longer reasoning does not always help: models frequently exhibit overthinking behaviors such as hedging, re-verifying, or switching approaches [2], [3] which can lead the model to an incorrect final answer. This motivates a natural question: can we train models to retain productive self-reflection while surpressing unproductive self-reflection? A significant challenge is cost: identifying whether self-reflection is helpful at each reasoning step would require step-wise labels via a process reward model [4], [5], LLM-as-a-judge, or manual annotation [4]. In this work, we propose a cheaper alternative.

Our key observation is that reasoning models can commit to intermediate answers within their thinking traces–for example, writing "the answer is X" or boxing a result before continuing to reason. These commitments provide verifiable demonstrations of productive and unproductive self-reflection: by comparing each to the ground truth, we know whether subsequent reflection improved or degraded the answer, without any external supervision. When a model reaches a correct intermediate answer and then reflects its way to an incorrect one, we have direct evidence that this self-reflection was harmful.

Based on this intuition, we propose DASH (Drift-Aware advantage SHaping), which converts traces where the answer drifts from correct to incorrect intermediate examples from wasted negatives into informative training examples. Rather than assigning a single scalar advantage to the entire rollout, DASH divides each trace into segments bounded by consecutive answer checkpoints and assigns advantages based on whether each segment moves towards or away from the correct answer. A single drift trace simultaneously teaches the model to reinforce the reasoning that found the correct answer and to suppress the overthinking that abandoned it–extracting dual training signal from what GRPO would treat as a flat negative example.

We complement DASH with six lightweight linguistic overthinking signals—repetition, hedging, abandonment, contradiction, recomputation, and length outlier—that characterize reasoning quality without requiring intermediate answer extraction. These signals serve as evaluation metrics to verify that accuracy gains correspond to genuine behavioral improvements rather than superficial shortcuts.

Experiments on Nemotron-4B across four competition-level math benchmarks show:

• Best accuracy where drift is worst. DASH achieves 50.8% on AIME25 (vs.% GRPO, 46.1% base)—the benchmark with the highest drift prevalence.

• Self-correction over spiraling. DASH’s correct traces exhibit twice as many contradiction-then-resolution patterns as GRPO’s (0.92 vs./trace), while showing fewer blind approach abandonments—reasoning longer but more productively.

2 Analyzing Overthinking in Reasoning Traces↩︎

Prior work on reasoning efficiency has primarily characterized overthinking through response length: longer traces are treated as less efficient, and length penalties or early-stopping mechanisms are used to encourage brevity [6]. However, length alone does not provide detail signals for what overthinking behaviors are exhibited by the model.

To address this, we first introduce a set of linguistic signals to analyze overthinking patterns. All signals are regex- or \(n\)-gram-based, requiring no learned components:

• S1 (Repetition): Maximum \(n\)-gram overlap between sliding windows. Captures circular reasoning loops where the model rephrases without progressing [7].

• S2 (Hedging): Density of uncertainty markers (“wait, no,” “let me reconsider”) per 100 tokens. Operationalizes the self-doubt mechanism preceding negative flips [8].

• S3 (Abandonment): Count of explicit strategy switches (“this approach is wrong,” “let me try another way”). The strongest individual failure predictor: 4.1–4.3\(\times\) more common in incorrect traces across all models tested.

• S4 (Contradiction): Count of self-contradiction markers (“which is impossible,” “can’t be right”). Captures unresolved inconsistencies [9], [10].

• S5 (Recomputation): Numerical values derived 3+ times in computation contexts. Targets confirmatory re-checking that rarely catches errors [11].

• S6 (Length outlier): Per-prompt group \(z\)-score of response length. Adaptive: hard problems warrant long responses, but within-group outliers indicate pathology.

Using these signals, we analyze reasoning traces generated by Llama-3.1-Nemotron-Nano-4B-v1.1 [12] on AIME 2025. We bucket generated responses by length and analyze the correlation between correctness, length, and overthinking behaviors. We present results for abandonment and self-contradiction dimensions in Figure 2 and a full analysis in Appendix 10.

Figure 2 (a) shows that accuracy decreases monotonically with response length, confirming that the model’s additional computation in longer traces is largely unproductive. Panels (b) and (c) reveal a more nuanced finding: even controlling for response length, incorrect traces exhibit higher rates of approach abandonment and self-contradiction than correct traces within the same length bucket. The gap is most pronounced for approach abandonment, where incorrect traces show consistently elevated density across all buckets, while self-contradiction shows a similar but more moderate trend.

These patterns suggest that much of the model’s unproductive computation involves cycles of self-reflection that actively steer reasoning in the wrong direction—abandoning viable approaches or arriving at contradictions that undermine earlier progress. This motivates our central hypothesis: if we can discourage self-reflective behavior that ultimately leads to incorrect answers while preserving reflection that aids error correction, we may improve both reasoning efficiency and accuracy without simply truncating generation length.

3 Drift-Aware Advantage Shaping↩︎

One challenge is that it is difficult to identify where self-reflective behavior is needed for correcting errors in reasoning steps and where it hurts performance. In this section, we consider a cheap proxy: rather than obtaining annotations across each step containing self-reflection and whether it is helpful or not, we instead extract signals from self-reflection that occurs after arriving at a candidate final answer, which can be verified with the ground truth as a measure of whether self-reflection was helpful or not.

3.1 Preliminaries: GRPO↩︎

In Group Relative Policy Optimization [13], for each prompt \(x\), the model generates \(n\) rollouts \(\{y_1, \ldots, y_n\}\). Each rollout receives a reward \(r_i\), and advantages are computed by group normalization: \[A_i = \frac{r_i - \text{mean}(\{r_j\}_{j=1}^n)}{\text{std}(\{r_j\}_{j=1}^n)}\]

This scalar \(A_i\) is broadcast identically to every token position in rollout \(y_i\), yielding per-token advantages \(a_t = A_i\) for all \(t \in \{1, \ldots, |y_i|\}\). The policy gradient loss is: \[\mathcal{L} = -\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(\rho_t A_i,\;\text{clip}(\rho_t, 1\pm\epsilon) A_i\right)\] where \(\rho_t = \pi_\theta(y_t | x, y_{<t}) / \pi_{\text{ref}}(y_t | x, y_{<t})\).

3.2 Identifying Answer Drift↩︎

Our proposed algorithm is centered around the key idea of detecting portions of self-reflection where the final answer is incorrect even though the trace had reached the correct solution at some point. Formally, we define answer drift as:

In order to identify drift occurrence, we extract intermediate answer commitments within the reasoning trace by matching patterns including \(\backslash\)boxed{...}, “the answer is \(X\)”, and natural-language answer commitments. Instances where drift occurs are prime examples of where self-reflection harms performance that we aim to penalize.

3.3 Segment-Based Advantage Shaping↩︎

One important consideration is that we would like to maintain positive self-correction even within traces with drift. For example, a trace can oscillate between a correct answer candidate and an incorrect answer candidate, and we would like to encourage behavior where the model’s self reflection changes to the correct answer while penalizing shifts from correct to incorrect. To handle this, rather than broadcasting a single advantage to all tokens, we divide each rollout into segments bounded by consecutive answer checkpoints (\(\hat{a}_{j-1}, \hat{a}_{j}\)] and assign segment-specific advantages.

3.3.0.1 Segment construction.

Given checkpoints at token positions \(p_1 < p_2 < \ldots < p_K\) within the response, we define segments:

• Neutral segment \(S_0\): tokens before the first checkpoint (\(t < p_1\))

• Positive segment \(S_j^+\): tokens in segment \((\hat{a}_{j-1}, \hat{a}_{j}]\) where checkpoint \(\hat{a}_{j}\) is correct

• Negative segment \(S_j^-\): tokens in segment \((\hat{a}_{j-1}, \hat{a}_{j}]\) where checkpoint \(\hat{a}_{j}\) is incorrect

3.3.0.2 Advantage assignment.

For each token at position \(t\): \[a_t = \begin{cases} +|A_i| \cdot \alpha_+ & \text{if } t \in S_j^+ \\ -|A_i| \cdot \alpha_- \cdot w(t) & \text{if } t \in S_j^- \\ A_i \cdot \alpha_n & \text{if } t \in S_0 \text{ (conditional)} \end{cases} \label{eq:segment95advantages}\tag{1}\]

where \(\alpha_+, \alpha_-\) are positive and negative scale factors, \(\alpha_n\) is the neutral scale, and \(w(t)\) is a length penalty weight defined below.

3.3.0.3 Length penalty within negative segments.

To encode “the longer you continue past a correct answer, the worse it gets,” tokens in negative segments receive an escalating penalty: \[w(t) = 1 + \alpha \cdot \frac{t - t_{\text{start}}^{(k)}}{t_{\text{end}}^{(k)} - t_{\text{start}}^{(k)}}\] where \(\alpha\) controls the ramp rate and the penalty is capped at \(w_{\max}\) to prevent gradient explosion.

3.3.0.4 Conditional neutral mode.

Tokens before the first answer checkpoint (the “neutral” segment) present a design choice. We implement three modes:

• zero: No gradient on neutral tokens (conservative).

• standard: Apply the standard GRPO advantage.

• conditional: Apply a scaled advantage based on whether the trace eventually drifts, providing a weak signal that the initial reasoning trajectory correlates with drift.

3.4 Reward Shaping for Drift Traces↩︎

Before advantage computation, drift traces receive a shaped reward that reflects their partial correctness: \[r_{\text{drift}} = r_{\text{incorrect}} + \delta \cdot \left(1 - \frac{L_{\text{post-drift}}}{L_{\text{total}}}\right)\] where \(\delta\) is the drift partial credit and the formula naturally decreases as the post-drift section grows. The shaped reward is capped: \(r_{\text{drift}} \geq r_{\text{incorrect}}\), ensuring drift traces are never penalized more harshly than pure-incorrect traces.

4 Experimental Results↩︎

4.1 Experimental Setup↩︎

4.1.1 Model and Data↩︎

We evaluate on Llama-3.1-Nemotron-Nano-4B-v1.1 [12], a 4B-parameter reasoning model that exhibits “detailed thinking” mode triggered by a system prompt. This model exhibits high drift rates (\(\sim\)​20% on AIME-level problems), making it a natural testbed for drift-aware training. Training data consists of 16.5K mathematical reasoning problems sampled from OpenR1-Math-220K [14], sourced from NuminaMath 1.5 [15] with reasoning traces from DeepSeek-R1 [1] verified by Math-Verify. We restrict to problems with both correct and incorrect verified traces to ensure meaningful GRPO signal.

4.1.2 Evaluation↩︎

We evaluate on four competition-level mathematical reasoning benchmarks: OlympiadBench [16], AMC 2023 [17], AIME 2024 [18], and AIME 2025 [19]. For AMC23, AIME24, and AIME25, we report avg@32 (average accuracy over 32 sampled solutions per problem). For OlympiadBench, we report pass@1.

4.1.3 Training Configuration↩︎

All runs use GRPO with group size \(n=16\) trained on 4 nodes of 8 H100 GPUs. For our drift-aware (DASH) runs, we set positive and negative advantage scales to \(\alpha_+ = 1.0, \alpha_- = 1.0\), use a length penalty with \(\alpha = 3.0\) and \(w_{\max} = 3.0\), and apply a conditional neutral mode with \(\alpha_n = 0.1\). Full training hyperparameters are provided in Appendix 13.

4.1.4 Baselines↩︎

Along with the base model and standard GRPO, we compare against two baselines designed to address lengthy reasoning traces:

4.1.4.1 DR-GRPO [20].

DR-GRPO is a debiased variant of GRPO that addresses an implicit length bias in the optimization objective. It makes two modifications: (1) replacing per-response length normalization in the policy loss with a constant scaling factor, which ensures that longer and shorter responses receive equal per-token gradient weight, and (2) computing advantages as \(\hat{A}_i = r_i - \text{mean}(\{r_j\}_{j=1}^n)\) without dividing by \(\text{std}(\{r_j\}_{j=1}^n)\), which removes a question-level difficulty bias. These changes have been shown to improve token efficiency and reduce the length of incorrect responses. This baseline tests whether an explicit credit assignment over the reasoning trajectory is more effective at reducing drift and overthinking behavior compared to using a debiased optimizer.

4.1.4.2 GRPO + Brevity Bonus.

We introduce a simple reward shaping baseline that explicitly encourages shorter correct traces. For each prompt group, we identify the length of the shortest correct response \(l_{\min}\). Each correct trace of length \(l_i\) then receives an additive per-token bonus: \(b_i = \frac{\beta \cdot l_{\min}}{l_i^2}\), where \(\beta\) is a scale hyperparameter. The shortest correct trace receives the maximum total bonus of \(\beta\), while longer correct traces receive proportionally less (scaling as \(l_{\min}/l_i\)). Incorrect traces receive no bonus. This baseline tests whether a simple length pressure on correct traces–without any fine-grained credit assignment over the reasoning trajectory–suffices to reduce drift and overthinking behavior. In our experiments, we use \(\beta = 0.2\)

4.2 Results↩︎

4.2.1 Main Results↩︎

Table 1 presents accuracy across benchmarks.

6pt

4.2.1.1 Key observations.

(1) Strongest performance on the hardest benchmark. DASH achieves the highest AIME25 accuracy (50.8%), outperforming GRPO and DR-GRPO. AIME25 is the benchmark with the highest drift prevalence and longest reasoning traces, suggesting that segment-level credit assignment is most impactful where overthinking is most severe.

(2) GRPO’s blind spot on AIME25. Standard GRPO improves over the base model on OlympiadBench (+6.4) and AIME24 (+1.3) but degrades on AIME25 (\(-\)​0.7). DASH avoids this regression, improving AIME25 by +4.7 over base. This aligns with our drift analysis: AIME25 has the highest correct-to-wrong drift rate, and GRPO’s uniform negative advantage on drift traces inadvertently penalizes the valid reasoning prefix, discouraging the strategies that found the correct intermediate answer.

4.2.2 Drift Reduction↩︎

In Figure 3, we report plot the rate of positive changes in answer (Wrong → Right) as a percentage of all flips in answer correctness for generated traces on AIME across DASH and baselines. We observe that DASH maintains the high combined accuracy and self-correction rate among methods that do not sacrifice accuracy, while Brevity Bonus achieves a high W→R rate but at the cost of reduced accuracy.

5 Analysis↩︎

5.1 Ablation Studies↩︎

6pt

5.1.0.1 Ablation results.

Table 2 isolates the contribution of two DASH components. Removing the length penalty on negative segments (which escalates the penalty for tokens further past a correct answer) reduces AIME25 by \(-\)​2.2 and average by \(-\)​1.18, confirming that the “the longer you continue past a correct answer, the worse” signal is load-bearing. Removing the conditional neutral mode (which provides a weak gradient on tokens before the first answer checkpoint) has a smaller effect on AIME25 (\(-\)​1.9) but interestingly improves OlympiadBench (+1.5), suggesting that pre-answer tokens on OlympiadBench contain less drift-predictive signal. The full DASH configuration achieves the best average and the best AIME25 score, indicating that both components contribute complementary value.

5.2 Overthinking Signal Analysis↩︎

Figure 4 compares five training methods across six overthinking signals on AIME25. DASH achieves the tightest profile: lowest repetition (s1: 0.40 vs. Nemotron-4B), lowest abandonment (s3: 2.59 vs. GRPO), and lowest length-outlier (s6: 0.008 vs. GRPO)—while producing 11.8% longer traces and achieving the highest accuracy (50.8% vs.% GRPO avg@32).

5.2.0.1 Contradiction as self-monitoring.

The one axis where DASH exceeds GRPO is contradiction (s4: 2.67 vs.). Rather than indicating overthinking, this reflects a qualitative shift in error-handling strategy. In GRPO, contradictions are 6.1\(\times\) more frequent in incorrect than correct traces—they are almost exclusively failure symptoms. In DASH, this discrimination ratio drops to 3.9\(\times\) because DASH’s correct traces contain nearly twice as many contradictions (0.92 vs. per trace), using them as an error-monitoring mechanism: detect inconsistency \(\to\) diagnose root cause \(\to\) resolve \(\to\) proceed. GRPO instead responds to confusion by silently abandoning approaches (8 abandonments per trace in the example of Figure 6) without ever identifying what went wrong. Full discrimination statistics are in Table 4 (Appendix 11), and a head-to-head trace comparison on AIME25 Problem 20 is in Figure 6.

In summary: DASH’s reasoning is longer but not more wasteful—it exhibits less repetition, less abandonment, and more deliberate error-checking, consistent with a model that has learned to self-correct rather than spiral.

5.3 Qualitative Analysis↩︎

To demonstrate how our drift-aware algorithm mitigates answer drift, we analyze two concrete case studies from the AIME 2025 evaluation set in Table 3. In both instances, the base model successfully uncovers the correct solution in its intermediate reasoning trace but drifts away from the answer (often numerous times), choosing alternative approaches. It ultimately exhausts the token limit without generating a final answer. In contrast, our drift-hybrid model stabilizes after a single validation phase and successfully commits to the correct solution.

6 Related Work↩︎

6.1 Overthinking and Reasoning Efficiency↩︎

Excessive reasoning in LLMs has been documented across multiple studies. [21] first named the “overthinking” phenomenon in o1-like models, showing that models over-allocate compute to simple problems. [22] demonstrated a U-shaped relationship between reasoning length and accuracy, and the “Reasoning Completion Point” framework [23] formally showed that once a model reaches its peak correctness probability, continued reasoning almost never improves the answer—it primarily re-confirms or flips to wrong. [2] link drift to thought-switching, and [3] trace the mechanism to self-doubt after correct answers. Rather than hoping continued reasoning self-corrects, we train the model to recognize when to stop within a trace.

6.2 Length Control in RL for Reasoning↩︎

Several methods address the length problem in GRPO training.

6.2.0.1 Length-penalized rewards

: Dr. GRPO [20] corrects a per-token normalization bias that dilutes penalties for long incorrect traces; DAPO [24] uses token-level loss aggregation; ShorterBetter [25] introduces “Sample Optimal Length” as a reward signal.

6.2.0.2 Progressive constraints

: ThinkPrune [26] iteratively tightens token budgets, and L1/LCPO [27] adds explicit length-controlled objectives.

6.2.0.3 Decoupled normalization

: DRPO [28] normalizes correct and incorrect rollouts separately, preventing length effects from corrupting advantage estimates. Our method adopts decoupled normalization as a component but adds segment-level granularity within individual traces.

All of these approaches use token count as a proxy for overthinking—a lossy signal that penalizes all length equally, including legitimate complex reasoning. Our method instead targets why a response is long (detecting actual answer drift) rather than that it is long.

6.3 Token- and Segment-Level Credit Assignment↩︎

Standard GRPO broadcasts a single advantage to all tokens. Several methods provide finer-grained credit through learned weighting (GTPO [29], \(\lambda\)-GRPO [30]), external judges (CAPO [31]), or Monte Carlo value estimation (VinePPO [32], SPO [33]).

Most closely related to our work, VPPO [34] uses a process reward model (PRM) to localize the first incorrect step in a failed rollout, partitioning the trajectory into a verified correct prefix (rewarded) and an erroneous suffix (penalized). This shares our intuition that not all tokens in a failed trace deserve equal blame. However, VPPO relies on an external PRM for error localization and targets correctness boundaries, whereas our method uses the model’s own intermediate answers to detect drift boundaries—requiring no additional models and applying to both correct and incorrect traces, since a correct trace that drifts before self-correcting still wastes compute. VPPO’s binary prefix/suffix partition also does not capture the richer structure of traces with multiple answer changes, which our segment-based formulation handles naturally.

6.4 Inference-Time Early Exit↩︎

Orthogonal to our training-time approach, several methods diagnose overthinking at inference [6], [35]–[37]. These are complementary to ours: a model trained with drift-aware shaping could additionally use inference-time early exit for further efficiency gains. We discuss extended related work in Appendix 14.

7 Conclusion↩︎

We presented DASH, a segment-level credit assignment method for reducing overthinking in reasoning language models. The core idea is to use the model’s own answer checkpoints as free supervision: when a trace reaches a correct answer and later moves away from it, the trace reveals both a useful reasoning prefix and a harmful reflective suffix. DASH preserves this structure by rewarding segments that lead to correct checkpoints and penalizing segments that lead away from them.

Our analysis shows why this distinction matters. Overthinking is not simply a matter of response length; even among traces of comparable length, incorrect solutions contain more abandonment and unresolved contradiction. On competition math benchmarks, DASH is most effective where this failure mode is most prevalent, achieving the best AIME25 accuracy while reducing repetition, abandonment, and length outliers. The remaining contradiction signal becomes more productive: DASH uses contradictions for diagnosis and recovery rather than as a prelude to spiraling.

These results suggest that efficient reasoning training should focus less on making models universally shorter and more on teaching them when reflection has stopped being useful. Segment-level credit from answer checkpoints offers a simple way to make that distinction without process labels or external judges.

Limitations↩︎

Our work has several limitations. First, experiments are conducted on a 4B model; while our analysis suggests drift patterns are scale-invariant, training dynamics may differ at larger scales. Second, our method requires extractable intermediate answers for drift detection, which limits applicability to domains with verifiable checkpoints (mathematics, code with test cases). Open-ended reasoning tasks without clear answer markers would require alternative drift indicators. Third, the slight accuracy decrease on some easy benchmarks for Nemotron (MATH-500: \(-1.7\)pp vs.base) represents a real trade-off that practitioners must weigh against gains on hard problems. Finally, our evaluation is limited to mathematical reasoning; generalization to other reasoning domains (logical, scientific, commonsense) remains to be validated.

Future work could explore difficulty-adaptive methods that modulate drift penalty strength based on problem difficulty or model uncertainty.

As with any RL fine-tuning method, DASH inherits the safety properties and failure modes of the base model; we do not introduce safety mitigations beyond those of the upstream models.

8 Signal Implementation Details↩︎

8.0.0.1 S1: Repetition.

Window size 200 tokens, stride 50. Jaccard similarity over 5-grams between non-adjacent window pairs. Score = max similarity, clipped to \([0,1]\). Threshold: \(>0.4\).

8.0.0.2 S2: Hedging.

Case-insensitive regex: wait, hmm, actually, hold on, let me reconsider, I’m confused, not sure, on second thought. Density = count per 100 tokens.

8.0.0.3 S3: Abandonment.

Regex: this approach is wrong, let me try another, let’s restart, going back to, scrapping this, dead end, alternatively. Raw count per trace.

8.0.0.4 S4: Contradiction.

Regex: contradicts the previous, which is impossible, this is impossible, can’t be right, that’s not possible, inconsistent with, but we just showed. Raw count per trace.

8.0.0.5 S5: Recomputation.

Numeric values extracted via [-+]?\d+(?:\.\d+)?. Values appearing 3+ times within 10-token computation contexts (near \(=\), \(+\), \(-\), \(\times\)) are flagged. Count of unique repeated values.

8.0.0.6 S6: Length outlier.

\(z\)-score of thinking-section length within the GRPO prompt group. Flag threshold: \(z > 2.0\). Score = \(\min((z - 2) / 2,\;1)\) for \(z > 2\), else \(0\).

8.0.0.7 Composite.

Signals are normalized to \([0,1]\): \(\tilde{s}_1 = \max(s_1 - 0.2, 0) / 0.8\), \(\tilde{s}_2 = \min(s_2/3, 1)\), \(\tilde{s}_3 = \min(s_3/3, 1)\), \(\tilde{s}_4 = \min(s_4/3, 1)\), \(\tilde{s}_5 = \min(s_5/5, 1)\). Composite \(\omega = (\tilde{s}_1 + \tilde{s}_2 + \tilde{s}_3 + \tilde{s}_4 + \tilde{s}_5) / 5\), clamped to \([0,1]\). Overthinking flag: \(\omega > 0.3\).

9 Signal Motivation and Prior Work↩︎

9.0.0.1 S1 (Repetition).

Reasoning models generate long chains of thought but often loop at low temperatures, repeating the same text [38]. [7] classify these as circular reasoning loops driven by self-reinforcing attention mechanisms that prevent escape from local minima.

9.0.0.2 S2 (Hedging) and S3 (Abandonment).

[8] examine negative flips—cases where extended reasoning changes correct answers to incorrect ones. They find that explicit reconsideration (hedging) precedes approach abandonment in over 67% of negative flip cases, and that abandonment rates increase monotonically with token count. The “alternatively” marker alone is 4.1–4.3\(\times\) more common in incorrect Nemotron traces across all scales tested.

9.0.0.3 S4 (Contradiction).

Self-contradiction is a prevalent LLM failure mode: [9] find contradictions in 17.7% of all ChatGPT sentences. [10] note that extended reasoning chains specifically increase opportunities for self-contradiction and degenerate outputs.

9.0.0.4 S5 (Recomputation).

[7] classify “numerical loops”—where models repeatedly derive the same constants—as a distinct loop category triggered by reasoning impasses. [11] show through large-scale analysis that models spend a substantial fraction of reasoning on confirmatory self-verification that rarely catches errors, reducing tokens by 20.3% when suppressed.

9.0.0.5 S6 (Length outlier).

Our analysis of Nemotron-4B across difficulty levels reveals a bimodal length distribution on hard problems: responses either solve in \(<\)​2K tokens (88% accuracy) or spiral into a 10K–25K token dead zone (0.6% accuracy). Within-group normalization captures this pathology adaptively without penalizing legitimately long solutions.

10 Full Overthinking Signals On Nemotron-4B↩︎

Figure 5 extends the main-text analysis by showing all four count-based linguistic signals on AIME 2024 and AIME 2025. The top row reports density-normalized counts within each length quintile, while the bottom row reports raw counts. This separation is important: if overthinking were only a length artifact, the density-normalized gaps between correct and incorrect traces should largely disappear once traces are compared within the same length bucket. Instead, incorrect traces retain higher signal density across most buckets, especially for approach abandonment and self-contradiction. These are precisely the behaviors most associated with harmful post-answer reflection: the model notices uncertainty, changes direction, and often fails to resolve the inconsistency before continuing.

The raw-count panels show the compounding effect of this behavior. Longer incorrect traces do not merely contain more words; they accumulate more abandonment, contradiction, hedging, and recomputation events in absolute terms. Thus the long-tail failures are both quantitatively longer and qualitatively different from successful traces. Hedging and recomputation are less individually diagnostic than abandonment, but their upward trend in incorrect traces suggests a common failure pattern: the model repeatedly re-opens decisions it has already made rather than converting verification into commitment. This supports the central motivation for DASH: reducing overthinking should not mean penalizing all long reasoning, but assigning credit according to whether reflection moves the answer trajectory toward resolution or back into search.

11 Contradiction Signal: Discrimination Analysis↩︎

The raw s4 count conflates two phenomena:

• Productive contradiction (predominant in DASH correct traces): detect inconsistency \(\to\) diagnose source \(\to\) resolve \(\to\) correct answer.

• Unproductive contradiction (predominant in GRPO incorrect traces): notice error \(\to\) fail to diagnose \(\to\) silently abandon \(\to\) repeat.

Table 4 shows that 33.1% of DASH correct traces contain contradictions (vs.% for GRPO), averaging 0.92 per trace—nearly double GRPO’s 0.47. This indicates DASH learned to use self-contradiction as a reasoning tool rather than being derailed by it.

11.1 Qualitative Example: Self-Correction vs.Spiraling↩︎

Figure 6 contrasts the two reasoning strategies on AIME25 Problem 20 (circle geometry). GRPO never identifies the root cause—a misunderstanding of internal tangency—and instead exhaustively substitutes values across 8 different parameterizations until truncation. DASH explicitly names the geometric impossibility (“that can’t be”), iteratively narrows the misunderstanding, resolves it at 59% of the trace, and proceeds linearly to the correct computation.

12 Training Dynamics↩︎

Figure 7 plots KL divergence and entropy over training for all methods. DASH maintains the lowest KL divergence throughout training, indicating that it achieves length reduction through targeted segment-level credit assignment rather than aggressive policy deviation, while GRPO + Brevity Bonus suffers from entropy collapse (\(\to\)​0.21), suggesting a loss of output diversity.

13 Training Configuration↩︎

All runs share:

• GRPO with group size \(n=16\), learning rate \(3 \times 10^{-6}\), grad clip 1.0

• KL penalty: \(\beta_\text{KL} = 10^{-4}\)

• Temperature 0.9, max response length 15,821 tokens

• 4 nodes \(\times\) 8 H100 GPUs, 1 epochs

13.0.0.1 DASH configuration.

• Advantage estimator: grpo_drift_hybrid

• Positive/negative scales: \(\alpha_+ = 1.0\), \(\alpha_- = 1.0\)

• Length penalty: \(\alpha = 3.0\), \(w_\text{max} = 3.0\)

• Neutral mode: conditional, \(\alpha_n = 0.1\)

• Segment length penalty: enabled

14 Extended Related Work↩︎

14.1 Adaptive Reasoning Depth↩︎

[39] approach overthinking from a model-selection perspective: their Thinkless framework trains a model to adaptively choose between short-form and long-form reasoning via control tokens and a Decoupled GRPO (DeGRPO) objective, learning when to engage in extended reasoning at all. Our work differs in granularity: rather than gating reasoning depth at the response level, we shape credit within a single reasoning trace.

14.2 Additional Length Control Methods↩︎

14.2.0.1 Global vs.local normalization.

REINFORCE++ [40] replaces GRPO’s group-level advantage normalization with global batch-level normalization, arguing that the former is a biased estimator that interacts poorly with length variation.

14.2.0.2 Value-based methods.

VAPO [41] addresses the length problem within a value-based PPO framework, introducing Decoupled GAE to handle heterogeneous sequence lengths and a length-adaptive discount to prevent long traces from dominating value estimates. While VAPO tackles length heterogeneity through the value function, our method instead operates directly on the policy gradient by reshaping advantages based on detected drift.

14.3 Token-Level Credit Assignment, Continued↩︎

TEMPO [42] builds prefix trees for nonparametric token-level credit. SPO [33] bridges token- and trajectory-level feedback through mid-grained, segment-level advantage estimation using flexible cutpoint- or tree-based partitions evaluated via Monte Carlo sampling without a critic model. These methods provide general-purpose credit assignment without specifically targeting drift; our approach is complementary in that we use the structure of intermediate answers to assign credit based on the model’s own reasoning trajectory, requiring no additional models, sampling, or learned components.

14.4 Inference-Time Early Exit, Continued↩︎

Certaindex [35] uses answer stability for early stopping, DEER [36] terminates based on confidence, and ROM [37] monitors for real-time overthinking indicators. [6] take a simpler approach with budget forcing, which controls test-time compute by forcefully terminating or extending the model’s thinking process via appended tokens.

15 Artifact Licenses↩︎

We list below the licenses of the scientific artifacts used in this work. Our use of all artifacts is restricted to non-commercial research on language-model reasoning, which is consistent with the intended use specified by each artifact’s authors.

• Llama-3.1-Nemotron-Nano-4B-v1.1 [12]: released by NVIDIA under the NVIDIA Open Model License, with additional terms from the Llama 3.1 Community License Agreement.

• veRL [43]: released by ByteDance Seed under the Apache License 2.0.

• OpenR1-Math-220K [14]: released by the Hugging Face Open-R1 team under the Apache License 2.0.

• OlympiadBench [16]: released by OpenBMB under the MIT License.

• AMC 2023, AIME 2024, AIME 2025: these are public mathematics competition problems originally published by the Mathematical Association of America (MAA). We use them solely as held-out evaluation benchmarks, consistent with established practice in the reasoning-LLM literature.

References↩︎