Fork-Think with Confidence
June 30, 2026
Abstract
Parallel thinking has enjoyed great success for boosting LLM performance on reasoning tasks without the need for any re-training. However, existing methods follow a think-first-then-decide paradigm, i.e., they first sample multiple reasoning paths, which inevitably leads to overgeneration, then prune or stop unnecessary paths to compensate. In contrast, decide-first-then-think—i.e., first identifying points that are likely to lead to desirable generations—has been underexplored so far. Following this paradigm, we propose Fork-think with confidence, that first identify forking points using model confidence in a single seeding path, then trigger thinking, sampling multiple continuations and aggregating them for the final response. Our experiments across three models and three reasoning benchmarks show that Fork-think reduces the token consumption by up to 30% and run-time by up to 57%, while performing comparable or better to parallel thinking. Our analysis reveals that Fork-think is able to identify forking points that are meaningful with respect to the downstream task and that sampling at later positions can lead to substantially better generations. Finally, we demonstrate how combining Fork-think with existing mechanisms such as early stopping and weighted voting can further boost the performance and perform comparably to existing state-of-the-art methods, without requiring any warm-up or offline training. Our results establish pre-determined forking as a promising research direction for efficient LLM reasoning.
1 Introduction↩︎
Despite the recent success of large language models (LLMs), they still struggle to reliably solve complex tasks, especially those that require multi-step reasoning and deduction [1]. To combat this, research has investigated various methods for building advanced reasoning models, for instance, by running supervised fine-tuning on additional reasoning data [2], [3] or using reinforcement learning [4]. However, such post-training methods are limited by the available data and can require substantial amounts of compute [5], [6].
In contrast, inference-time scaling avoids costly post-training (or can be used in a complementary manner) by allocating additional compute during inference [7], [8]. Over time, works have devised sequential methods that iteratively refine the quality of the generation [9] as well as search-based methods that sample (and aggregate) multiple continuations at each step in the generation [10]. However, sequential methods are bound by the maximum context size or deteriorate with overlong contexts [11] and search-based methods incur an exponential compute cost. To reduce compute cost, works have proposed parallel thinking as a lightweight alternative, where multiple reasoning paths were sampled only once—allowing compute cost to scale linearly with the number of samples—and then aggregated [12], [13]. Since then, researchers have studied increasingly sophisticated aggregation methods [14], [15] or even proposed to break up the reasoning process again into smaller steps [16], however, at the expense of an increased compute cost especially for verbose reasoning models [17]. Most recently, some works proposed to mitigate this using adaptive methods that dynamically identify and prune unpromising reasoning traces using a threshold found in a warm-up phase [18] or a scoring model trained offline [19], [20].
One common denominator of prior works is that they operate on a think-first-then-decide paradigm (i.e., sampling at the beginning), which is surprising as there exists no theoretical justification to do so. In this work, we propose to address parallel thinking in a decide-first-then-think manner which has not been studied so far. In other words, we propose to first identify forking points using a seed path from which we then sample thinking paths (Fork-think). To identify forking points, we utilize pivot tokens which have been found to substantially affect the generation [21] and which we identify using the model’s confidence. Our contributions are:
A new perspective on parallel thinking based on a decide-first-then-think paradigm and a first method (Fork-think) based on the novel paradigm.
A systematic evaluation across three models and three datasets showing that Fork-think is a simple, yet effective method reducing the overall token cost by up to 30% and the overall run-time by up-to \(\sim\)57% in comparison to parallel think.
Analysis and ablation studies investigating different factors and showing that later forking points can achieve a substantially higher performance compared to generating parallel samples at the beginning.
Our results indicate that the decide-first-then-think paradigm provides a promising new perspective on parallel scaling and that investigating more sophisticated methods to identify forking points could be a worthwhile endeavor for future research.
Figure 1: Left: Existing works commonly pursue a think-first-then-decide strategy which given a specific input (i.e., prompt & question), first generates multiple parallel traces which are pruned dynamically and aggregated afterwards. Right: Instead, we propose a decide-first-then-think strategy, which generates a greedy seed path first, and then identifies promising forking points to sample multiple parallel traces from. Note, that Fork-think can be used complementary to existing pruning or aggregation methods.. a — Think-first-then-Decide, b — Decide-first-then-Think
2 Related Work↩︎
2.1 LLM Reasoning↩︎
Evaluating and improving the reasoning capabilities of LLMs has been a long-sought subject of research [22]–[26]. Whereas early works were primarily concerned with improving the reasoning capabilities via in-context learning (ICL, [27]) or chain-of-thought prompting (CoT, [28]), recent works have also investigated the verification of reasoning chains [29]–[31] as well as how to directly instill reasoning capabilities into models [4], [32]. Others propose to break down a complex task into easier sub-tasks [33] or to minimize randomness (e.g., by falling back to greedy decoding) at critical reasoning steps to avoid wrong deductions [34].
2.1.0.1 The Importance of Diversity.
One recurring property to improve LLM reasoning is the diversity of the generated reasoning paths (for a single question). For instance, [35] show that aggregating responses with diverse reasoning paths (using different prompts) for a single question substantially boosts performance. Others propose to increase diversity by ensembling [36] or by sampling from a different distribution [37]. Finally, some methods directly mitigate the diversity collapse caused by post-training [38], by adaptively changing the temperature [39], [40] or adding regularizers during post-training [41], [42]. However these methods often rely upon repeatedly sampling the model at different temperatures (or require more offline training) which substantially increases compute cost. Instead, we propose to utilize pivot tokens [21] as our forking points.
2.2 Inference-time Scaling↩︎
Inference-time scaling (or test-time scaling) allows models to dynamically adjust their generation distribution during inference and has emerged as a powerful alternative to scaling model parameters [43], [44]. While scaling strategies such as sequential refinement [45] or search against a verifier [44] are effective, they are limited to single-path exploration or rely on expensive, difficult-to-train scoring functions. In contrast, parallel thinking (or repetitive sampling) bypasses the need for trained verifiers by independently sampling multiple, diverse reasoning paths via temperature scaling [12], [13]. These are then aggregated using voting strategies such as majority voting [12] and confidence-weighted voting [15], [46], or re-ranking techniques [47]. While parallel thinking yields substantial performance gains on complex reasoning tasks, it incurs a prohibitive token cost, especially for the long CoT generations of reasoning models [4].
2.2.0.1 Adaptive Methods.
To alleviate the computational overhead of parallel thinking using static generation or aggregation schemes, works have investigated methods that allocate compute more adaptively in a training-free manner [48]. Prior approaches typically follow pruning-based strategies that halt some traces based on a model’s confidence [18] or signals from hidden states [19]. Others halt the sampling process when a consensus is reached within a predefined window of samples [49] or the majority answer generated so far is statistically undeniable [50]. Some even investigate using offline training with additional objectives such as reducing redundancy [20]. Albeit increasing the efficiency, we argue that such a think-first-then-decide approach might lead to suboptimal branching points, and instead propose a decide-first-then-think view of parallel thinking. Finally, a very recent work which mixes both paradigms is EAGER [51]. Here, the authors propose to adaptively branch during generation based on a threshold that requires a warm-up phase with an extensive hyper-parameter search. In contrast, Fork-think does not require any warm-up phase or offline training.
3 Fork-think with Confidence↩︎
Existing methods primarily utilize a think-first-then-decide paradigm, i.e., they first sample multiple reasoning paths and then decide which to consider for the final response. This inherently leads to an overgeneration of reasoning paths, making pruning or early stopping essential and resulting in offline training or warm-up. We instead propose to shift the paradigm towards decide-first-then-think: we first generate a single seed path and then identify forking points from which we further sample multiple reasoning paths with a temperature \(\tau > 0\). Note that we do not require the seed path to be complete (i.e., a partial response suffices) which is distinctly different from sequential inference-time scaling methods that iteratively improve upon a single generated response.
3.1 Identifying Forking Points↩︎
Ideally, we would sample from forking points that maximize the chances of producing correct responses. However, this is not possible due to the absence of a scoring function which would require an offline training or warm-up phase. Instead, we use previous findings in LLM reasoning that show how aggregating a diverse set of responses generated for the same question can improve overall performance [35]. Hence, our objective is to identify the points where sampling is likely to generate diverse responses.
3.1.0.1 Pivot Tokens.
Multiple works report the presence of specific tokens or token windows that have a substantial impact on the future generation (positively or negatively). These so-called pivot tokens (sometimes also referred to as forking or critical tokens) typically correspond to a lower model confidence at a specific position in the generation process. Whereas previous work primarily utilizes these pivot tokens to improve LLM post-training [21], [52], we propose to utilize them as forking points where sampling with a temperature \(\tau > 0\) is likely to generate diverse responses. To quantify the confidence of the model \(c_i\) at the \(i\)-th position, we follow [18] and utilize the log-probability averaged over the \(k\) most likely tokens \(x_{i1},\ldots, x_{ik}\): \[c_i = \frac{1}{k}\sum_{j=1}^{k}\log P(x_{ij}), \label{eq:confidence}\tag{1}\] Where \(P(x_{ij})\) are the individual token probabilities of the original model normalized over \(k\). The best forking point \(c^*\) for a string \(X\) and the respective model confidences \(C\) is then described as: \[c^* = \mathop{\mathrm{arg\,min}}_{x_i \in X} C, \label{eq:least-confidence}\tag{2}\] i.e., the point where the model confidence is lowest.2 Note that 2 can easily be extended to multiple forking points by sorting all tokens \(x_i \in X\) according to their respective model confidences \(c_i\) and selecting the ones where the model is least confident.
3.2 Fork-think↩︎
Having defined the selection criteria to identify forking points, we now define Fork-think. We first specify the length \(l_0\) of the seed path, the number of top-\(k\) tokens to estimate the confidence of the model, a branching factor \(n\) that determines how many samples are generated at the forking point, the number of forking points \(m\) to sample from, and the temperature \(\tau\) to use for sampling (after seed path generation). Following [21], we utilize greedy sampling (i.e., \(\tau_0 = 0\)) to generate a seed path \(X = \{x_1, \ldots, x_l\}\) with length \(l_0\) and determine top-\(m\) forking points \(C^* = \{c_1, \ldots, c_m \}\) according to 2 . Next, we generate \(n\) samples for each forking point \(c_j \in C^*\) using a prefix \(\pi_j = \{x_1, \ldots, x_{<c_j}\}\). Finally, we aggregate all generated samples through majority voting. 2 provides an overview of the complete pipeline of Fork-think.
4 Evaluation↩︎
We conduct experiments on three LLM reasoning benchmarks and compare the performance of Fork-think against five baselines and state-of-the-art methods using three different models. Our evaluations further include Flex-Fork-think, a variation of Fork-think augmented with early stopping and weighted majority voting.
4.1 Experimental Setup↩︎
We quantify the performance of each method using majority-voting accuracy (in %), the number of tokens used (in million).
4.1.0.1 Models.
We evaluate Fork-think using three models from different model families, Qwen3-8B [54], DeepSeek-R1 distilled into a Qwen3-8B base model (referred to as DeepSeek-8B, [4]), and Phi-4 Reasoning Plus 14B (Phi-4-RP-14B, [55]). For each model, we set the hyper-parameters (maximum trace length, temperature \(\tau\), top-\(p\)) to the ones recommended in the respective model card. We provide all details of the model-specific hyper-parameters in appendix 7.2 and all prompt templates in appendix 8.
4.1.0.2 Benchmarks.
Following previous works, we use two widely-used benchmarks for evaluating LLM reasoning which are high-difficulty mathematical competition problems also featured in the MathArena leaderboard [56]. The problems stem from AIME (American Invitational Mathematics Examination), which are exams used to select the U.S. representative students at the international math olympiad. Each year consist of two rounds of exams (I & II) with each exam consisting of 15 challenging questions. We use the questions from the years 2024 (AIME24, [57], [58]) and 2025 (AIME25, [59], [60]). To benchmark Fork-think beyond mathematical reasoning, we also run experiments on GPQA [61], a dataset consisting of graduate-level scientific questions. More specifically, we use the diamond split, which consists of 198 questions answered correctly by experts and incorrectly by non-experts.
4.1.0.3 Baselines.
We evaluate Fork-think against several baselines as well as a very recent state-of-the-art method to assess its practical effectiveness.
- Greedy
-
Our first baseline reports the performance of greedy decoding (\(\tau = 0\)) without any parallel sampling. Instead, we continue generating from the greedy seed path with an extended length that matches the maximum trace length granted to the other methods. This serves as a lower bound on Fork-think’s performance.
- CoT
-
Our second baseline is a single reasoning trace that follows standard Chain-of-Thought prompting [28].
- Parallel Thinking
-
Our primary baseline was proposed by [12] and is a widely used baseline in LLM reasoning tasks. Starting from the given prompt (i.e., the instruction and a question), it first generates \(n\) parallel reasoning traces, then aggregates the responses using a self-consistency mechanism (i.e., majority voting) to obtain the final response. In our experiments, we refer to this method as parallel thinking.
- ASC
-
We also compare to Adaptive Self-Consistency (ASC, [50]) that generates reasoning traces one by one, and models the probability distribution over generated answers to quantify the confidence in the majority one, then stops the generation when the majority answer becomes statistically undeniable.
- DeepConf
-
Finally, we include a very recently published state-of-the-art method, namely, Deep Think with Confidence (DeepConf, [18]). DeepConf generates parallel reasoning traces which are pruned dynamically (i.e., during generation) using a threshold that is estimated during a warm-up phase. The generated traces are then aggregated using a confidence-weighted majority voting scheme. For a fair comparison, we report the performance for both aggregation schemata (confidence-weighted and non-weighted majority voting).3
4.1.0.4 Other Hyper-parameters.
For evaluation, we use a seed path length \(l_0=2,048\), a branching factor \(n=32\) (also for Parallel Thinking and DeepConf), and set the number of forking points \(m=1\). For the main experiments, we specifically use only a single forking point to better quantify the effect of branching later (Fork-think) compared to branching from the very beginning (Parallel Thinking and DeepConf). We further set top-\(k=10\) and select the branching temperature \(\tau\) as per recommendation in the respective model card (cf. appendix 7.2). Trace aggregation is done via simple majority voting for Parallel Thinking and Fork-think. For DeepConf, we perform a confidence-weighted majority voting using the best performing confidence metric reported by the authors.
4.1.0.5 Implementation Backend.
We implement Parallel Thinking, Greedy, and Fork-think using SGLang [62] as our backend. Fork-think heavily depends on prefixing, which is more compatible with SGLang, given its radix cache that skips massive amounts of prefill compute if a request shares a context history with prior requests. For DeepConf, we use the implementation provided by the authors that uses vLLM as the backend [63].
4.2 Results↩︎
lcccccccccc Model & & & &
(lr)2-3 (lr)4-5 (lr)6-7 (lr)8-9 & Token & Acc & Token & Acc & Token & Acc
& Token (\(\Delta\)%) & Acc
Qwen3-8B & 0.70 & 50.0 & 0.6 & 68.7 & 20.2 & 80.1 & 14.2 (-30%) & 81.8
DeepSeek-8B & 0.68 & 80.0 & 0.7 & 80.0 & 22.6 & 86.0 & 20.3 (-10%) & 86.0
Phi-4-RP-14B & 0.73 & 43.3 & 0.4 & 67.3 & 12.2 & 77.8 & 11.2 (-8%) & 77.0
Qwen3-8B & 0.62 & 60.0 & 0.7 & 64.0 & 22.0 & 75.4 & 15.9 (-27%) & 75.6
DeepSeek-8B & 0.75 & 80.0 & 0.8 & 75.3 & 25.2 & 83.3 & 21.4 (-14%) & 82.4
Phi-4-RP-14B & 0.66 & 46.6 & 0.4 & 58.7 & 13.5 & 62.6 & 12.5 (-7%) & 66.6
Qwen3-8B & 3.3 & 39.8 & 2.6 & 55.2 & 83.3 & 62.1 & 60.4 (-27%) & 63.2
DeepSeek-8B & 2.3 & 61.1 & 2.5 & 60.8 & 72.4 & 66.9 & 54.2 (-25%) & 62.1
Phi-4-RP-14B & 3.6 & 31.3 & 1.7 & 53.9 & 54.9 & 67.3 & 53.1(-3%) & 70.1
¿tbl:tab:main95res? shows the performance of Fork-think and baselines that do not utilize pruning or weighted aggregation methods beyond majority voting. We repeat each experiment across all models five times using different seeds for the pseudo-random number generator. We find that Fork-think uses substantially fewer tokens than Parallel Thinking, while achieving a comparable overall accuracy. Moreover, we can see that the improvements in terms of overall token usage as well as the performance are maintained across all models and datasets. We can also see that scaling the compute at inference time substantially outperforms the single-sample variants (greedy and CoT), highlighting the benefits of sampling multiple times instead of scaling model parameters.
4.2.0.1 Flex-Fork-think.
One advantage of Fork-think is that it is easily extensible with more sophisticated methods such as early stopping and other weight aggregation methods. To provide a fair comparison against methods that utilize such mechanisms, we evaluate Flex-Fork-think, a variant of Fork-think that includes confidence-weighted majority voting and early stopping. The early stopping strategy is similar to the one introduced in ASC, i.e., after forking, the samples are generated one by one until the majority answer is statistically prevalent. ¿tbl:tab:add95res? shows the performance of Flex-Fork-think against DeepConf and ASC. To provide a fair comparison against ASC which does not rely on confidence-weighted majority voting (WMV), we also report the majority voting (MV) scores for DeepConf and Flex-Fork-think. Overall, we find that Flex-Fork-think achieves a comparable (or slightly lower) performance compared to DeepConf; however, with substantially less tokens. Moreover, we observe that it performs on-par with ASC also in terms of accuracy. Interestingly, we find again that Flex-Fork-think consistently outperforms all other methods for the weaker (Qwen3-8B) model. Overall, our results suggest that selecting a later forking point can already save a substantial number of tokens while achieving a comparable performance, showing that decide-first-then-think is a promising paradigm.
lccccccc Model & & &
(lr)2-3 (lr)4-5 (lr)6-7 & Token & MV & Token & MV (WMV) & Token & MV (WMV)
Qwen3-8B & 10.4 & 80.1 & 11.1 & 81.8 (\(81.2\)) & 7.0 & 81.8 (81.8)
DeepSeek-8B & 10.1 & 86.0 & 16.3 & 89.2 (\(\textbf{90.0}\)) & 9.8 & 86.0 (86.6)
Phi-4-RP-14B & 6.4 & 77.8 & 8.9 & 80.0 (80.0) & 6.7 & 77.0 (76.6)
Qwen3-8B & 11.6 & 75.4 & 13.8 & 69.0 (\(71.0\)) & 9.1 & 75.6 (74.4)
DeepSeek-8B & 12.0 & 83.3 & 18.9 & 83.3(\(\textbf{83.6}\)) & 11.6 & 82.4 (83.3)
Phi-4-RP-14B & 13.5 & 62.6 & 11.2 & 76.3 (76.3) & 8.3 & 64.6 (67.3)
Qwen3-8B & 58.5 & 61.7 & 64.2 & 62.8 (62.3) & 35.0 & 63.1 (63.2)
DeepSeek-8B & 42.1 & 67.1 & 73.8 & 67.3 (66.3) & 34.5 & 62.1 (63.1)
Phi-4-RP-14B & 41.8 & 58.1 & 49.3 & 72.7 (72.3) & 37.8 & 70.7 (63.2)
4.2.0.2 Run-time.
We further compare the overall run-time of Fork-think and Parallel Thinking to illustrate the practical behavior of our implementation, and find that Fork-think leads to a substantial decrease in terms of overall run-time. As ¿tbl:tab:04-runtime? shows, the average run-times vary substantially between different models. We further find that the run-time reduction is substantially larger when using two GPUs although Fork-think reduces the token cost far less, namely, 14% compared to the 27% for Qwen3-8B. This is a very promising finding but remains to be investigated in future work using more models and GPUs.
lrccc Model & # GPUs & Parallel Thinking & & Time (\(\Delta\)%)
Qwen3-8B & 1 & 11.98 & 7.39 & -38%
DeepSeek-8B & 2 & 20.81 & 8.92 & -57%
Phi-4-RP-14B & 2 & 13.45 & 6.45 & -52%
5 Analysis↩︎
We analyze our results in terms of the forking position and nature of forking tokens, and conduct additional ablation studies for different model confidence estimates (Equation 1 ) as well as varying branching factors \(n\) and seed path length \(l_0\).
5.0.0.1 Forking Position.
For each AIME dataset, we study the effect of the forking position by binning each forking point with respect to its position in the greedy seed path. 3 shows the accuracy (averaged across each bin) for forking points attributed to one of five equidistant bins starting from 0 to 2,048 (\(l_0\)). In addition, we report the average performance for Parallel Thinking at position 0. Interestingly, we observe that some forking points (e.g., all points in the second or later bins for AIME25) exhibit a substantially higher performance than others, indicating the importance of choosing the right forking point. For AIME24, we can also observe model-specific differences as points from the third and fifth bins perform best for the Qwen3-8B model while for DeepSeek-8B points from the third and fourth bins perform best. Finally, we find that forking too early (in the first or second bins) often seems to hurt model performance, indicating that the commonly used think-first-then-decide paradigm might not always be optimal.
Figure 3: Accuracy (using majority voting), averaged over each bin of different forking point positions (for Fork-think) with 95% confidence intervals. Parallel Thinking always forks at position zero. We can see that in multiple instances, forking later in the seed path achieves a substantially higher performance compared to Parallel Thinking, providing evidence that identifying good forking points can substantially boost the performance (while reducing compute cost).. a — AIME24, b — AIME25
5.0.0.2 Forking Tokens.
We study the nature of forking tokens induced by Fork-think by analyzing the top-10 most prominent forking tokens across all models on the AIME datasets and find that these tokens correspond to compact mathematical syntax, including variables and operators such as {a, (x , b , =, -, +, 5, 2, 4, 1}. While these tokens may look syntactically small, they are functionally important in mathematical reasoning. A choice of equality, sign, operator, variable assignment, or algebraic transformation can determine the subsequent reasoning step and whether the solution remains correct or diverges. This is consistent with prior observations that pivotal or critical tokens in math reasoning do not always correspond to human-salient semantic words, and often include operators or numerical manipulation tokens [21], [64]. We further analyze the distribution of forking tokens relative to the solution steps in the reasoning trace on AIME24 and find that 83% of the forking points happen right after or in the middle of a key reasoning step, e.g., in the middle of an equation, while the rest of the points happen near text continuation that are not particularly key reasoning steps. This indicates that the forking points selected by Fork-think are important divergence points for the correctness of the solution.
5.1 Ablation Experiments↩︎
We conduct ablation experiments for varying the branching factor \(n\) as well as different model confidence estimation methods.
5.1.0.1 Impact of the Branching Factor.
Many existing methods often require a high branching factor \(n\).4 However, using large \(n\) is often not possible for resource-constrained, academic setups. We thus study the scaling behavior of Fork-think at small \(n \in \{4,8,16,32\}\). [{fig:Num-samples}] shows the averaged accuracy (using majority voting) and the number of required tokens (in millions) of Fork-think against Parallel Thinking. We can observe that Fork-think consumes substantially fewer tokens across all \(n\) and maintains a competitive performance to Parallel Thinking on AIME25 and even outperforms it on AIME24. We conclude that Fork-think offers a cheap alternative to Parallel Thinking even with a small branching factor.
Figure 4: Scaling behavior of Forkthink against baseline Parallel think. Models’ accuracy vs. tokens (\(\times 10^{6}\)) for different methods on different datasets.. a — AIME24, b — AIME25
5.1.0.2 Impact of the Seed Path Length.
To further understand the trade-off between the length of the seed path \(l_0\) and tokens spent in the branches, we study the impact of seed path length on overall token usage as well as performance. 1 shows the token usage and accuracy of Fork-think with Qwen3-8B on AIME24 when varying \(l_0 \in \{512, 1024, 2048, 4096\}\). We can see that Fork-Think is not sensitive with respect to accuracy. This is likely due to most forking tokens being centered earlier rather than later (cf. 3). Hence, increasing \(l_0\) does not affect the selected forking token. Regarding token usage, we do find differences, with a too-short \(l_0\) causing too-early branching for some of the questions, and a too-long \(l_0\) leading to overspending tokens. These results highlight the importance of \(l_0\) length on the overall token usage of Fork-think.
| \(l_0\) Length | 512 | 1024 | 2048 | 4096 |
|---|---|---|---|---|
| Accuracy (in %) | 81.1 | 81.1 | 81.8 | 81.8 |
| Tokens (in \(\times 10^{6}\)) | 16.0 | 15.2 | 14.2 | 15.2 |
5.1.0.3 Different Model Confidence Estimates.
Although we utilize the average log-probability (cf. Equation 1 ) in this work as a proxy for model confidence, various other metrics have been proposed in the past. We hence extend our evaluation to token entropy and select the forking point according to the highest entropy over the normalized top-\(k\) probabilities: \[H_i = -\sum_{j=1}^{k} \mathrm{P}(x_{j})\log \mathrm{P}(x_{j}), \label{eq:entropy}\tag{3}\] where \(\mathrm{P}(x_{j}) = \frac{P(x_j)}{\sum_{i=1}^{k} P(x_i)}\) is the normalized probability of the \(j\)-th token. We further conduct experiments with different \(k\) and evaluate a variant of Fork-think where we select the maximum instead of the minimum average confidence point (cf. 2 ).
2 shows the averaged accuracy (using majority voting) as well as the overall token consumption for all model confidence estimation metrics. We can see that using the model confidence estimate provided in Equation 1 uses fewer tokens overall. This result suggests that, although the average log-probability is a heuristic that depends strongly on the shape and sharpness of the top-\(k\) distribution at each position, it provides an effective and efficient signal for selecting forking points in our setting. In contrast, entropy over the normalized top-\(k\) distribution might be a more principled measure of uncertainty, but it does not lead to better performance in our experiments. We provide the results for all models and both AIME datasets in appendix 9. In future work, we will investigate more sophisticated model confidence estimation methods—for instance, ones based on previous decoding works such as locally typical sampling [65].
| Model Confidence | \(\arg\min_1\) | Entropy\(_{10}\) | Entropy\(_{1000}\) | \(\arg\min_{10}\) | \(\arg\max_{10}\) |
|---|---|---|---|---|---|
| Accuracy (in %) | 82.6 | 86.0 | 86.0 | 86.0 | 86.0 |
| Tokens (in \(\times 10^{6}\)) | 20.15 | 21.1 | 21.1 | 20.14 | 21.1 |
6 Conclusion↩︎
We proposed Fork-think, a method that shifts the perspective of Parallel Thinking from the predominantly used think-first-then-decide paradigm to a decide-first-then-think paradigm. In contrast to prior methods that by default sample parallel traces from the very beginning, Fork-think utilizes a greedy seed path to first identify forking points for sampling.
Our experiments across three benchmarks using three reasoning models show that Fork-think achieves a performance comparable to Parallel Thinking (both using majority voting) but using 7–30% less tokens and requiring 38–57% less run-time. The results further show that a variant of Fork-think, combined with early stopping techniques, is also competitive against other baselines, including a recent state-of-the-art method that utilizes adaptive pruning together with confidence-weighted majority voting. Our analysis reveals that using early forking points by default might not be a good choice, as later forking points often lead to substantially better parallel traces. We further show that Fork-think’s performance remains competitive to Parallel Thinking even in low-budget scenarios that use a small branching factor (4, 8, 16).
These results suggest multiple promising directions for future research. First, the high performance of a single forking point suggests that increasing the number of forking points (i.e., sampling a smaller number of parallel traces at multiple locations) could be a promising direction. Second, the dependence of token saving on the seed path \(l_0\) length motivates the need for an adaptive or budget-aware seed path that dynamically adjusts the length based on the problem at hand. Finally, we will explore other methods to estimate the model confidence (beyond entropy) which could substantially improve our understanding on selecting forking points.
Acknowledgments↩︎
We thank Michael Hahn, Dawei Zhu, and Sergey Trochin for their helpful discussions and feedback on this project. This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – SFB 1102 under project 232722074, and GRK 2853/1 “Neuroexplicit Models of Language, Vision, and Action” - under project 471607914.
Ethics Statement↩︎
6.0.0.1 Disclosure of LLM Use.
Generative AI tools were used solely to assist with debugging code, polish the manuscript (via paraphrasing), and for fixing grammar and typographical issues. These tools were not used to produce any scientific content, including the experimental design, analysis, citations, or conclusions. All authors have reviewed and approved the manuscript and assume full responsibility for its contents.
7 Evaluation↩︎
7.1 Preliminary Experiments using MATH500↩︎
The MATH dataset was originally published by [66] and consists of 12,5k problems (7,500 training and 5,000 test). Later, [29] utilized further 4,5k instaces from the original test set for model training, reserving 500 instances as an uncontaminated test set (MATH500). ¿tbl:tab:preliminary? shows our preliminary results using DeepSeek-8B for the MATH500 dataset using Greedy, Parallel Thinking, and Fork-think. As can be seen, the performance of the simple, greedy baseline is already very high, achieving an accuracy of \(\sim\)93%. We conjecture that there is a high chance of dataset contamination—i.e., the model might have seen the test set during training—which seems likely considering the dataset was already released in 2021. We thus decided to exclude this dataset from our evaluation. Regardless, we can see that Fork-think uses 33% less tokens compared to Parallel Thinking.
lcccccc Model & & &
(lr)2-3 (lr)4-5 (lr)6-7 & Token & Acc & Token & Acc & Token (\(\Delta\)%) & Acc
DeepSeek-8B & \(3.6\) & \(93\%\) & 119.6 & 93% & (-33%) & 94%
7.2 Hyper-parameters↩︎
For each model, we select the hyper-parameters for generation as per recommendation in the respective model card. We set following hyper-parameters: the maximum trace length \(l_{\text{max}}\), the temperature \(\tau\), and \(p\) for top-\(p\) (nucleus) sampling. ¿tbl:tab:04-parameters? provides a list of all model-specific hyper-parameters.
lccc Model & \(l_{\text{max}}\) & \(\tau\) & \(p\)
Qwen3-8B & 32,000 & 0.7 & 0.95
DeepSeek-8B & 64,000 & 0.7 & 0.95
Phi-4-RP-14B & 30,000 & 0.8 & 0.95
7.3 DeepConf Run-time↩︎
¿tbl:tab:04-runtime-deepconf? provides the averaged run-times for DeepConf. We can see that in general, DeepSeek-8B requires the longest amount of time despite having less parameters than Phi-4-RP-14B. The differences between Qwen3-8B and DeepSeek-8B can be attributed to the different number of GPUs used for the experiments. We note that DeepConf and Fork-think are inherently not comparable to each other due to the different backends.
lcr Model & # GPUs & Time
Qwen3-8B & 1 & 13.42
DeepSeek-8B & 1 & 27.43
Phi-4-RP-14B & 1 & 41.6
7.4 Compute Infrastructure↩︎
The experiments were conducted on a high-performance computing cluster equipped with NVIDIA H100 (80GB) or A100 (40GB/80GB) GPUs. GPU usage was limited to at most 8 GPUs being occupied simultaneously, with a single experiment using up to 2 GPUs at once.
8 Prompt Templates↩︎
We use the same prompts across all models and methods. To generate our greedy seed path (as well as for the greedy baseline), we use a basic CoT prompt:
Please reason about the problem step by step. {Question}
For Parallel Thinking, DeepConf, and Fork-think, we use the prompt that is stated in the usage recommendations for mathematical problems of the DeepSeek–R1 series:5
Please reason step by step, and put your final answer within boxed{}. {Question}
9 Additional Results on Forking Metrics↩︎
3 provides the results of all models across both AIME datasets for different model confidence estimates (cf. §5 for a description). We can see that choosing forking points corresponding to lowest average log-probability (\(\arg\min\)) can substantially save more tokens. At the same time, the mixed accuracy trends indicate that identifying optimal forking points remains a challenging problem and requires further investigation in future work.
| Dataset | Model | Parallel Think | \(\arg\min_{10}\) | \(\arg\max_{10}\) | Entropy\(_{1000}\) | ||||
|---|---|---|---|---|---|---|---|---|---|
| 3-4 (lr)5-6 (lr)7-8 (lr)9-10 | Tok. | Acc. | Tok. | Acc. | Tok. | Acc. | Tok. | Acc. | |
| AIME24 | Qwen3 | 20.21 | 80.67 | 14.23 | 81.80 | 17.89 | 81.11 | 18.77 | 84.44 |
| DeepSeek | 22.62 | 86.0 | 20.14 | 86.0 | 21.13 | 86.0 | 21.11 | 86.0 | |
| Phi | 12.22 | 77.80 | 11.38 | 77.33 | 11.54 | 67.78 | 11.61 | 74.44 | |
| AIME25 | Qwen3 | 22.08 | 75.40 | 15.96 | 75.60 | 20.77 | 75.66 | 21.47 | 74.44 |
| DeepSeek | 25.17 | 83.33 | 21.92 | 82.41 | 22.99 | 80.00 | 23.22 | 80.0 | |
| Phi | 13.59 | 62.60 | 12.87 | 66.67 | 12.59 | 65.56 | 12.91 | 65.56 | |
References↩︎
Co senior authors.↩︎
While we use model confidence, we note that there exist other metrics for quantifying a model’s uncertainty [53]. We provide respective ablation studies in 5.↩︎
Note, that between EAGER [51] and DeepConf, DeepConf is the stronger baseline as reported by [51].↩︎
For instance, [18] utilize 512 parallel traces in their experiments using an undisclosed compute infrastructure.↩︎
https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/README.md#usage-recommendations↩︎