1 Introduction↩︎

Recently, large language models (LLMs)[1], [2] have demonstrated remarkable capabilities across a wide range of natural language understanding and generation tasks. Despite these capabilities, LLMs still struggle with tasks that require complex and multi-step reasoning, such as mathematical problem solving, logical inference, and planning. To address this limitation, researchers have explored methods to elicit stronger reasoning behavior. Relevant methods[2], [3] mainly include prompt-based techniques, such as Chain-of-Thought (CoT)[4] prompting, which encourages models to generate intermediate reasoning steps, and training-based approaches, including Supervised Fine-tuning (SFT)[5] with reasoning traces and Reinforcement Learning (RL)[3], which explicitly incorporate reasoning processes into the model.

However, the introduction of CoT has also led to an emerging issue: overthinking[6]–[8]. Instead of allocating reasoning effort proportionally to task difficulty, CoT often induces LLMs to produce unnecessarily long reasoning traces even for simple queries, increasing both inference time and computational cost. To mitigate this, several research directions have emerged. For instance, SFT with preferred reasoning lengths[9]–[11] or reinforcement learning with length-based penalties[12], [13] encourages concise reasoning traces, while adaptive reasoning or selective thinking techniques[14]–[17] attempt to adjust the depth of reasoning based on query complexity.

While existing methods aim to balance reasoning depth and computational efficiency, current adaptive thinking strategies typically make a single global decision—whether to think or not—on a per-query basis. However, no prior work investigates adaptive interleaved thinking in multi-step reasoning settings, where decisions to reason, skip, or adjust thinking effort occur dynamically across intermediate steps.

This raises a key question: Is it necessary for an LLM to think at every step of multi-hop reasoning? If over-thinking exists within intermediate steps, how can an LLM learn to select Think* or No-Think adaptively based on the difficulty of each step?*

In this paper, we introduce AdaptR1, an RL-based adaptive interleaved thinking method for multi-hop question answering. AdaptR1 enables adaptive reasoning at each stage of a multi-step process, allowing models to allocate reasoning effort more efficiently. Motivated by AdaptThink[14], we use <think>no_think</think> to denote skipping explicit thinking in the current step. Under the Graph-R1 setting, our method reduces average think tokens by 69.71% while maintaining answer quality. Our analysis characterizes where efficient thinking emerges across multi-hop reasoning steps and how overthinking appears within the process. Our contributions are as follows:

• We study adaptive interleaved thinking for multi-hop QA and show that over-thinking appears in intermediate reasoning steps and can be reduced through learning.

• We propose an RL-only adaptive thinking method that avoids SFT cold-start trajectories, and we design a quality-gated efficiency reward for QA tasks with continuous answer rewards such as F1.

• AdaptR1 achieves comparable or better performance in multi-hop question answering with a 69.71% average think-token reduction under the Graph-R1 setting. Extensive analysis shows that over-thinking is concentrated in the early stages of multi-hop reasoning rather than the final synthesis stage.

2 Related Works↩︎

Efficient Reasoning in LRMs. Following recent observations regarding the “over-thinking” phenomenon in long Chain-of-Thought (CoT) reasoning, adaptive thinking strategies have garnered significant attention. Existing approaches to efficiency generally fall into two categories. The first involves intrinsic model modifications, achieved either through integrating length-based rewards in reinforcement learning (RL)[12], [13], [18]–[20], supervised fine-tuning (SFT) on concise responses[9]–[11], [21], [22], or by amalgamating reasoning and non-reasoning parameters [23]. The second category empowers LLMs to adaptively modulate their reasoning process based on query complexity. Prominent examples include AdaptThink [14], AutoThink [15], HiPO [24], and ARM [25]; notably, A\(^2\)FM [17] extends this framework to encompass instant, reasoning, and agentic modes. Our research aligns with this second paradigm. However, these methods usually make a single query-level routing decision, whereas interleaved multi-hop QA requires repeated decisions after each retrieval result. This step-wise setting also makes direct comparison with single-turn pruning methods less informative, because they do not support the reason-search-answer loop evaluated here. AdaptR1 therefore addresses the unexplored challenge of mitigating over-thinking within the granular steps of multi-hop reasoning, and it learns this behavior directly through RL without SFT cold-start trajectories.

Multi-hop Question Answering. Methodologies for multi-hop Question Answering (QA) can be broadly classified into training-free and training-based paradigms. Training-free methods employ prompting strategies such as Chain-of-Thought (CoT)[4] and various retrieval-augmented frameworks including IRCoT[26], ITER-RETGEN [27], WebGPT [28], ReAct [29], Self-RAG [30], Self-ask [31], and FLARE [32]. Conversely, training-based methods—such as R1-Searcher [33], DeepResearcher [34], R3-RAG [35], DeepRAG [36], Search-R1 [37], and Graph-R1 [38]—utilize SFT or RL to cultivate step-by-step reasoning capabilities. Despite these advancements, current literature lacks a mechanism to address over-thinking specifically within multi-hop QA contexts. Our proposed method fills this gap by implementing an RL-exclusive adaptive thinking strategy tailored for these scenarios.

3 Preliminaries↩︎

AdaptR1 adds one adaptive instruction to the native parent prompt: at each scheduled reasoning slot, the model can either generate explicit reasoning inside <think>...</think> or emit <think>no_think</think> to skip it. In the Graph-R1 setting, tool calls use <query>...</query> and retrieved evidence is returned inside <knowledge>...</knowledge>; Search-AdaptR1 preserves Search-R1’s native <search>/<information> interface. The full Graph-AdaptR1 prompt is provided in Appendix 8.

GRPO. Group Relative Policy Optimization (GRPO)[3] samples grouped rollouts for each question and updates the policy with group-normalized sequence-level advantages. In AdaptR1, the standard sequence reward is replaced by the adaptive reward in Eq. 2 ; the full GRPO objective is provided in Appendix 10.

Interleaved Thinking. R1-like methods employ an iterative process of reasoning and retrieval to synthesize a final output. This process is modeled as an action sequence \(\mathcal{A} = [a_0, a_1, \dots, a_t]\), initialized with \(a_0 = \boldsymbol{Think}\). For subsequent steps \(i > 0\), the transition logic dictates that if the preceding action \(a_{i-1}\) was a reasoning step (Think), the subsequent action \(a_i\) must be either Search or Answer. Conversely, if \(a_{i-1} \neq \boldsymbol{Think}\), the system defaults to Think. The action space is defined as follows:

• Think: Derives reasoning steps utilizing existing internal knowledge.

• Search: Queries an external knowledge base to retrieve supplementary information.

• Answer: Terminates the sequence by providing the final response once information sufficiency is achieved.

• No-Think: Introduced in AdaptR1, this operator permits the model to bypass the explicit reasoning phase during a scheduled No-Think step.

4 AdaptR1↩︎

AdaptR1 extends GRPO to train an interleaved QA policy that can decide at each scheduled reasoning slot whether to generate an explicit rationale or emit the No-Think token <think>no_think</think>. This step-wise decision is important for multi-hop QA: a trajectory may need explicit reasoning after some retrieval results, but not after every intermediate step. We train this behavior directly with RL rather than SFT, since constructing oracle trajectories that label exactly when reasoning should be skipped is ambiguous and dataset-dependent.

4.1 Quality-Gated Efficiency Objective↩︎

AdaptR1 is designed to optimize efficiency under an answer-quality constraint rather than as a pure length penalty. Let \(R_{\text{ans}}(o)\in[0,1]\) denote the answer reward of a generated trajectory \(o\), measured by F1, and let \(R_{\text{nt}}(o)\in[0,1]\) denote a bounded efficiency reward derived from the number of No-Think actions. The intended objective is to reward efficiency only inside the feasible region of sufficiently accurate answers: \[\max_{\pi_{\theta}} \mathbb{E}_{o\sim\pi_{\theta}}[R_{\text{nt}}(o)] \quad \text{s.t.}\quad R_{\text{ans}}(o)\geq\tau.\] Operationally, we implement this constraint with a threshold mask and scale the efficiency bonus by the answer reward: \[\begin{align} r_{\text{AdaptR1}}(o) &=R_{\text{ans}}(o)\bigl(1+\omega R_{\text{nt}}(o) \\ &\quad \cdot \mathbb{I}[R_{\text{ans}}(o)\geq\tau]\bigr). \end{align} \label{eq:constrainedreward}\tag{1}\] Eq. 1 is a compact view of the objective: the threshold mask prevents low-quality trajectories from receiving positive efficiency gradients, while the clipping and KL penalty in GRPO retain the trust-region-style stabilization of the base optimizer.

4.2 AdaptR1 Reward↩︎

As illustrated in Figure 2, our default implementation uses an absolute No-Think reward. Let \(n_{\text{nt}}(o)\) be the number of No-Think actions in trajectory \(o\). We first compute an uncapped efficiency bonus and then bound it by 1: \[\begin{align} r_{\text{nt}}(o) &= n_{\text{nt}}(o) \times r_0, \\ R_{\text{nt}}(o)&=\min(r_{\text{nt}}(o), 1), \end{align}\] where \(r_0\) is the unit reward for skipping one reasoning round. The actual training reward is the implementation form of Eq. 1 : \[\label{eq:AdaptR1reward} \begin{align} r_{\text{AdaptR1}}(o) &=r_{\text{answer}}(o) + \mathbb{I}[r_{\text{answer}}(o)\geq\tau] \\ &\quad \cdot \omega r_{\text{answer}}(o) R_{\text{nt}}(o). \end{align}\tag{2}\] Here, \(\tau\) is the answer-quality gate and \(\omega\) controls the strength of the efficiency bonus relative to the answer reward. The ceiling on \(R_{\text{nt}}\) keeps the No-Think signal auxiliary, so the model is rewarded for concise trajectories only when answer quality remains acceptable. The hyperparameters \(\tau\) and \(\omega\) are evaluated in the ablation studies.

4.3 Step-Wise Weighting and Reward Variants↩︎

The main experiments use the absolute No-Think reward in Eq. 2 . To make this reward position-aware, we replace \(R_{\text{nt}}(o)\) with a step-wise weighted efficiency reward. Let \(n_{\text{nt}}^{(j)}(o)\) count No-Think actions at step \(j\). Since the average trajectory length in our datasets is typically two to three rounds, we separate the first step from later steps: \[\label{eq:weighted-nothink} \begin{align} s_{\lambda}(o) &=\lambda n_{\text{nt}}^{(1)}(o) +(1-\lambda)\sum_{j>1} n_{\text{nt}}^{(j)}(o), \\ R_{\text{nt}}^{\lambda}(o) &=\min(r_0 s_{\lambda}(o), 1). \end{align}\tag{3}\] The coefficient \(\lambda\) controls whether the reward pressure favors early or later No-Think actions. We set \(\lambda=0.9\) in the main experiments and study its sensitivity in Section 6.

For reward-shape ablations, we additionally compare against a relative variant that normalizes the efficiency bonus by the total number of rounds: \[\begin{align} p_{\text{nt}}(o) = \frac{n_{\text{nt}}(o)}{n_{\text{all}}(o)}. \end{align}\] This variant uses the same answer-quality gate as AdaptR1: \[\label{eq:AdaptR1reward-relative} \begin{align} r_{\text{rel}}(o) &=r_{\text{answer}}(o) + \mathbb{I}[r_{\text{answer}}(o)\geq\tau] \\ &\quad \cdot \omega r_{\text{answer}}(o) p_{\text{nt}}(o). \end{align}\tag{4}\] This relative reward is not the default AdaptR1 objective; it is included to test whether ratio-based normalization encourages more stable exploration than the absolute bounded bonus.

5 Experiments↩︎

5.1 Setups↩︎

Datasets and Metrics. Following Graph-R1[38], we conduct experiments on six common QA datasets[39]: 2Wikihop[40], HotpotQA[41], Musique[42], NQ[43], PopQA[44], and TriviaQA[45]. To keep comparisons controlled, all training-based methods use the same fixed split of 5,120 training and 128 testing instances per dataset. We use EM and F1 to evaluate answer quality, and R-S to evaluate retrieval performance.

Baselines. We consider both training-free and training-based baselines. The training-free methods include NaiveGeneration, StandardRAG [46], GraphRAG [47], LightRAG [48], PathRAG [49], HippoRAG2 [50], and HyperGraphRAG [51]. The training-based baselines include SFT [5], R1 [3], R1-Searcher [33], Search-R1 [37], and Graph-R1 [38]. For the key RL comparisons, Search-R1, Graph-R1, Search-AdaptR1, and Graph-AdaptR1 are initialized from the same Qwen2.5-7B-Instruct[52] backbone and trained under the same data split and hyperparameter budget; AdaptR1 is not initialized from trained Search-R1 or Graph-R1 checkpoints. This isolates the effect of the adaptive reward from differences in data, initialization, and training overhead.

Implementation Details. We instantiate AdaptR1 on two baselines, Search-R1 and Graph-R1, yielding Search-AdaptR1 and Graph-AdaptR1. Unless otherwise stated, we use the absolute No-Think reward with weight \(\lambda=0.9\), threshold \(\tau=0.6\) and coefficient \(\omega=0.2\). Detailed implementations of baselines and our method are provided in Appendix 13.

Retriever. The retriever follows the corresponding backbone method. Search-R1 uses E5[53], while Graph-R1 uses hypergraph-based retrieval with bge-large-en-v1.5[54].

5.2 Main Experiments↩︎

We observe that Graph-AdaptR1 yields results comparable to Graph-R1, achieving an average F1 improvement of 1.5. Search-AdaptR1 outperforms Search-R1 by a larger margin, with an average F1 increase of 6.9. This consistent improvement across datasets and retrieval pipelines suggests that AdaptR1 is not tied to a single dataset or retrieval design. The improved R-S scores further indicate that adaptive skipping can preserve, and in some cases improve, the retrieval behavior needed for accurate answers.

Think Token Economy. To evaluate the think token economy of AdaptR1, we analyze the token consumption detailed in Table 2. Under the Graph-R1 setting, Graph-AdaptR1 lowers average think tokens from 102.07 to 30.92, corresponding to a 69.71% reduction, while maintaining or slightly improving average F1. The largest reduction appears on HotpotQA, where think tokens decrease from 103.68 to 10.00, a 90.35% reduction. This drastic decrease in token usage does not come at the cost of accuracy, as shown in the main experiments. Instead, it highlights that Graph-R1 contains substantial redundant explicit reasoning steps. By eliminating these superfluous reasoning steps, AdaptR1 lowers computational cost and latency while maintaining or improving performance.

5.3 Case Study↩︎

To provide a granular understanding of the model’s behavior, we present a comparison of generation trajectories with and without AdaptR1 in Section 12.4. The case study illustrates that standard LLMs can exhibit “over-thinking,” generating exhaustive and sometimes circular reasoning chains even in multi-hop QA. In contrast, AdaptR1 bypasses redundant thinking while preserving the key reasoning link needed for the answer. We additionally analyze cases where skipping thinking hurts answer quality in Section 12.5. These qualitative results corroborate our quantitative efficiency results while making clear that no_think is beneficial when used selectively rather than as an unconditional rule.

6 Ablations and Analysis↩︎

In this section, we validate the main design choices of AdaptR1 through a prompt-only control experiment, a temporal distribution analysis, and reward-design ablations. All analyses use Graph-R1 as the backbone.

6.1 Necessity of RL Training↩︎

To verify that adaptive skipping is learned through RL rather than triggered by the prompt alone, we compare the base model before training with the RL-trained Graph-AdaptR1 model under the same adaptive prompt. As shown in Table 3, the base model rarely uses no_think on complex datasets and performs poorly. RL training substantially improves both answer quality and adaptive skipping; for example, on Musique, the No-Think rate increases from 13.02% to 50.67%, while F1 improves from 8.41 to 53.42. This indicates that no_think becomes useful only after the model learns how to integrate it into the reason-search-answer trajectory.

6.2 Step-wise Adaptive Thinking and Temporal Distribution↩︎

We first evaluate the temporal dynamics of the No-Think mechanism to understand how the model allocates its reasoning budget across different stages of the problem-solving trajectory. Table 4 presents the average step-wise No-Think ratio with a balanced penalty factor (\(\lambda=0.5\)) for Musique. Other datasets are shown in 12.2. Compared with saturated datasets such as HotpotQA, NQ, PopQA, and TriviaQA, Musique exposes more complex reasoning dynamics and is therefore the most diagnostic setting for studying where explicit thinking remains necessary.

Balanced Strategy: The results reveal a distinct, emergent behavior: the model predominantly learns to bypass extensive reasoning during the initial steps (Steps 1–2), reserving its computational budget for the final stages of the trajectory (Steps 3–4). This observation is somewhat counter-intuitive; one might expect the initial planning phase to require significant cognitive load. However, the data suggests that for multi-hop QA tasks, the model adopts a “retrieve-then-reason” strategy. The early steps likely involve schema activation or direct information retrieval that can be handled heuristically, whereas the final steps require synthesis and deduction to formulate the answer. We therefore examine whether larger early-step weights better match this behavior.

Sensitivity to \(\lambda\): We further explore the impact of varying the step-wise reward weight \(\lambda\) from 0.1 to 0.9.

• Low \(\lambda\) (0.1–0.4): Assigning insufficient reward to early No-Think actions generally leads to lower F1, suggesting that the model may still over-reason on simple intermediate steps.

• High \(\lambda\) (0.6–0.9): Larger \(\lambda\) often improves over low \(\lambda\), and \(\lambda=0.9\) gives the best F1 on Musique. The model consistently skips the first step but still uses explicit thinking later when retrieved evidence must be synthesized, supporting a retrieve-then-reason strategy rather than indiscriminate skipping.

These findings support the step-wise reward design, so we use \(\lambda=0.9\) in the main experiments in Table 1.

6.3 Design of the Adaptive Reward Function↩︎

The efficacy of Reinforcement Learning (RL) is heavily contingent on reward shaping. In this subsection, we investigate the specific design components of the AdaptR1 reward structure, including formulation (absolute vs. relative), constraints (ceilings), and hyperparameter sensitivity.

6.3.0.1 Absolute vs. Relative Reward Formulation

We compare our absolute No-Think reward against a relative formulation (Table 5). A relative reward scales with the fraction of skipped steps, which appears smoother but is easier to exploit: the model can shorten the trajectory and skip a larger ratio of steps regardless of context. In contrast, the absolute formulation preserves the multi-turn reasoning structure and yields better average F1.

6.3.0.2 Impact of Reward Ceiling

We further test the necessity of a “top ceiling,” a hard limit on the accumulation of efficiency rewards. As shown in the “w/o top ceiling” row of Table 5, removing this constraint degrades performance significantly. Without a ceiling, the efficiency reward can dominate the optimization landscape and encourage reward hacking. The ceiling keeps efficiency as a secondary objective that should not override answer accuracy.

6.3.0.3 Threshold Sensitivity (\(\tau\))

The threshold \(\tau\) determines the confidence level required for the model to trigger a No-Think action. Table 6 illustrates the trade-off:

• Low \(\tau\) (\(<0.6\)): A permissive threshold results in frequent, unjustified skipping of reasoning steps, harming performance (Avg F1 \(\approx 55.3\)).

• High \(\tau\) (\(>0.8\)): An overly strict threshold renders the No-Think reward too sparse. The model rarely attempts to skip, negating the efficiency benefits of AdaptR1.

Our results identify \(\tau=0.6\) as the critical inflection point where the model reliably filters unnecessary reasoning without truncating valid cognitive processes.

6.3.0.4 Reward Coefficient (\(\omega\))

Finally, we analyze the magnitude of the efficiency reward relative to the correctness reward, controlled by coefficient \(\omega\) (Table 7). The data exhibits an inverted U-shaped curve. A small \(\omega\) (\(0.1\)) provides a weak adaptive signal, while a large \(\omega\) (\(>0.4\)) distracts optimization from the primary QA objective. We find that \(\omega=0.2\) provides the best trade-off, suggesting that the efficiency signal should remain auxiliary to answer correctness.

Taken together, the ablations provide practical guardrails for avoiding reward hacking: \(\tau\) should not be lower than 0.6, \(\omega\) is most reliable around 0.1–0.2, and \(\lambda\) is best treated as a step-wise pressure term rather than a universal instruction to skip. These ranges preserve the answer reward as the dominant objective while allowing the model to discover efficient trajectories.

6.4 Training Dynamics↩︎

To better understand the learning progression of AdaptR1, we visualize No-Think behavior on the difficult Musique dataset in Figure 3. The curve shows a phased rather than monotonic trade-off between accuracy and reasoning length. In early training (steps 0–20), \(F_1\) rises rapidly to about 0.4 while think tokens remain high (\(\sim 120\)), suggesting that the model first learns to solve the task with ample reasoning. During steps 20–80, No-Think behavior emerges: \(F_1\) peaks above 0.6 as think tokens decline toward \(\sim 80\), showing that AdaptR1 learns to prune redundant thinking after acquiring task competence. Past step 90, however, tokens fall further to \(\sim 60\) while \(F_1\) drops to about 0.45, indicating that excessive pruning can skip necessary synthesis. Appendix 12.3 shows the same trend on other datasets; most reduce think tokens early and then improve accuracy, while 2Wiki briefly recovers more thinking near step 80.

7 Conclusion↩︎

In this paper, we presented AdaptR1, an RL-based framework for mitigating “over-thinking” in multi-hop question answering. Unlike adaptive methods that rely on global routing decisions or SFT cold-start trajectories, AdaptR1 learns a fine-grained, step-wise policy that decides when to reason explicitly, when to query external knowledge, and when to skip redundant thinking at each intermediate stage. Empirically, AdaptR1 improves both Search-R1 and Graph-R1 settings: Search-AdaptR1 raises average F1 from 43.80 to 50.68, while Graph-AdaptR1 raises average F1 from 59.27 to 60.74. Under the Graph-R1 setting, it reduces average think tokens by 69.71%, with the largest per-dataset reduction reaching 90.35%, while maintaining or slightly improving answer performance. Our analyses further show that adaptive skipping is learned through RL, that overthinking is concentrated in the initial planning stages rather than the final synthesis stage, and that quality-gated rewards are important for avoiding reward hacking. These results suggest that RL-only adaptive interleaved thinking is a promising direction for efficient multi-hop reasoning.

Limitations↩︎

The limitations of AdaptR1 primarily stem from its sensitivity to hyperparameters and potential training instability, as the method relies on balancing the confidence threshold (\(\tau\)), reward coefficient (\(\omega\)), and step-wise weights (\(\lambda\)). Our ablations identify useful ranges, but overly aggressive settings can still cause the model to over-prune essential reasoning steps and prioritize brevity over correctness in later training epochs. In addition, our current scope is multi-hop QA, where trajectories are typically short to medium length. We do not claim that the same reward design directly transfers to DeepResearch-style tasks that require much longer planning and sustained reasoning; extending AdaptR1 to that setting remains future work.

8 Prompt Template↩︎

Table 8 shows the Graph-AdaptR1 prompt. The template preserves the original reason-search-answer format while adding one adaptive instruction: when explicit reasoning is unnecessary, the model may emit <think>no_think</think> before either a query or final answer. Search-AdaptR1 uses the same adaptive instruction but keeps Search-R1’s native tool tags.

9 Ethics Statement↩︎

This work utilizes publicly available datasets (2WikiMultiHopQA, HotpotQA, Musique, NQ, PopQA, and TriviaQA) that are widely used in the research community. We have adhered to the licenses and terms of use associated with these datasets. To the best of our knowledge, these datasets do not contain personally identifiable information (PII) or offensive content that would pose a risk to individuals. This study does not involve human subjects or human annotation, as all evaluations were conducted using automatic metrics.

A primary contribution of this work is the reduction of computational costs in Large Language Models (LLMs). By reducing the number of generated "think tokens" by up to 90% compared to standard reasoning methods, AdaptR1 significantly lowers the energy consumption and carbon footprint associated with model inference. This aligns with the goals of Green AI.

However, we acknowledge that our method relies on the pre-trained Qwen2.5-7B-Instruct model. Like all LLMs, this backbone model may carry inherent biases or the potential to generate toxic content derived from its training data. While our adaptive strategy aims to improve efficiency and does not explicitly introduce new biases, it does not actively mitigate existing ones. Users should exercise caution and implement appropriate safety guardrails when deploying such models in real-world applications.

AI assistants were used for language polishing, improving clarity and readability, and editorial integration of author-provided revision material. All scientific content, including the research ideas, methodology, experimental design, results, and conclusions, was conceived, implemented, and verified by the authors. The use of AI tools did not influence the technical decisions or the interpretation of experimental results.

10 GRPO Objective↩︎

For each question \(q \sim P(Q)\), GRPO samples a group of outputs \(\{o_1, o_2, \dots, o_G\}\) from the old policy \(\pi_{\theta_{\text{old}}}\) and updates the policy model \(\pi_{\theta}\) by optimizing: \[\begin{align} \mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q),\, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)} \\ \Bigg[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Big( \min\big(r_t(\theta)\hat{A}_{i,t},\\ \, \operatorname{clip}(r_t(\theta),\, 1-\varepsilon,\, 1+\varepsilon)\hat{A}_{i,t}\big) \\ - \beta\, \mathbb{D}_{\text{KL}}(\pi_{\theta} \,||\, \pi_{\text{ref}}) \Big) \Bigg]. \end{align} \label{eq:grpo}\tag{5}\] Here, \(r_t(\theta)=\frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}\) is the probability ratio, \(\varepsilon\) controls clipping, \(\beta\) regulates the KL penalty, and \(\hat{A}_{i,t}\) is computed using group-relative reward: \[\hat{A}_{i,t} = \frac{r_i - \mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}. \label{eq:advantage}\tag{6}\] where \(\mathbf{r} = \{r_1, r_2, \dots, r_G\}\) is the reward vector of \(G\) outputs. Since GRPO uses sequence-level rewards, \(\hat{A}_{i,t}\) is constant across all tokens within the same trajectory.

11 Algorithm↩︎

We show the algorithm for AdaptR1.

12 Additional Experiments↩︎

12.1 Robustness Across Random Seeds↩︎

We further evaluate robustness with 5 random seeds on three representative datasets. Table 9 reports mean and standard deviation for answer metrics and think-token usage. Accuracy variance remains moderate, and the average think-token count remains substantially below the corresponding baseline in every dataset.

12.2 No-Think Ratios↩︎

We show the No-Think ratios of left 5 datasets in Table 10,11,12,13,14.

12.3 Training Dynamics↩︎

We show the training dynamics of six datasets in Figure 4.

12.4 Case Study↩︎

We show the case of Graph-R1 in Table 15 and Graph-AdaptR1 in Table 16.

12.5 Failure Analysis↩︎

We analyze when no_think can hurt answer quality by comparing Graph-AdaptR1 with the full-thinking Graph-R1 baseline. Table 17 separates examples that only AdaptR1 answers correctly, examples both methods answer correctly, and cases where Graph-R1 is correct but Graph-AdaptR1 fails. The last category approximates failures where explicit reasoning was likely useful but was skipped or shortened too aggressively.

The failures are relatively infrequent, and in all datasets except NQ the number of examples fixed by AdaptR1 is larger than the number lost by AdaptR1. Qualitatively, the main failure mode appears in strict comparison questions that require precise extraction and comparison of attributes, such as dates or numerical values. For example, for the question “Which film has the director who died later, The Hellions or Hum Kaun Hai?” Graph-AdaptR1 answers The Hellions after skipping final synthesis, while Graph-R1 retrieves the directors separately, compares their death years, and answers Hum Kaun Hai. This suggests that no_think is most risky when the final step requires explicit symbolic comparison over retrieved evidence.

13 Experimental Settings↩︎

13.1 Baselines in the Graph-R1 Setting↩︎

We categorize our baselines into two distinct groups based on the underlying backbone model.

GPT-4o-mini Based Methods. The first group employs GPT-4o-mini as an inference-only generator. We evaluate the base model’s intrinsic capacity using NaiveGeneration, a zero-shot approach without retrieval. We also include StandardRAG [46], representing the conventional chunk-based retrieval-augmented generation paradigm. Furthermore, we assess a suite of graph-based retrieval strategies: GraphRAG [47], which constructs entity graphs for one-shot retrieval; LightRAG [48], a streamlined variant designing compact graphs for efficiency; PathRAG [49], which executes retrieval via path-based pruning on entity graphs; HippoRAG2 [50], utilizing a hierarchical path planner over knowledge graphs; and HyperGraphRAG [51], which leverages n-ary relational hypergraphs to facilitate single-step retrieval.

Qwen2.5-Instruct Based Methods. The second group utilizes the Qwen2.5-Instruct (7B) model. We establish foundational performance bounds using NaiveGeneration, the classic StandardRAG [46] pipeline, and SFT [5], which applies supervised fine-tuning on QA pairs. Additionally, we evaluate advanced methods optimized via reinforcement learning (RL): R1 [3], a policy trained with GRPO to generate answers directly without retrieval; Search-R1 [37], a multi-turn chunk-based retrieval approach trained via GRPO; R1-Searcher [33], a two-stage GRPO-based framework for chunk-based retrieval; and Graph-R1 [38], an agentic GraphRAG framework enhanced by end-to-end reinforcement learning.

13.2 Evaluation Metrics↩︎

We assess model performance using three primary metrics focusing on answer accuracy and retrieval quality.

Exact Match (EM). We employ Exact Match to strictly evaluate answer accuracy. This metric determines whether the generated answer \(y_i\) is identical to the ground-truth reference \(y_i^\star\) following a normalization process (i.e., lowercasing, punctuation removal, and whitespace standardization). The EM score is averaged over all \(N\) samples: \[\text{EM} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I} \left[ \text{norm}(y_i) = \text{norm}(y_i^\star) \right],\] where \(\mathbb{I}[\cdot]\) denotes the indicator function.

F1 Score. To provide a more granular assessment of generation quality beyond binary matching, we utilize the F1 score. This metric measures the token-level overlap between the prediction and the ground truth, defined as the harmonic mean of precision and recall: \[\text{F1} = \frac{1}{N} \sum_{i=1}^{N} \frac{2 \cdot |\mathcal{T}(y_i) \cap \mathcal{T}(y_i^\star)|}{|\mathcal{T}(y_i)| + |\mathcal{T}(y_i^\star)|},\] where \(\mathcal{T}(\cdot)\) represents the set of tokens in a given text.

Retrieval Similarity (R-S). To assess the efficacy of the retrieval module in isolation, we compute Retrieval Similarity. This metric measures the semantic alignment between the retrieved context \(k_{\text{retr}}^{(i)}\) and the ground-truth "gold" context \(k_{\text{gold}}^{(i)}\). We utilize a semantic embedding function, \(\text{Enc}(\cdot)\), to compute the cosine similarity between the vector representations: \[\text{R-S} = \frac{1}{N} \sum_{i=1}^{N} \cos\left(\text{Enc}(k_{\text{retr}}^{(i)}), \text{Enc}(k_{\text{gold}}^{(i)})\right).\]

13.3 Implementation Details↩︎

We outline the general hyperparameters in Table 18. While our AdaptR1 models largely inherit the configurations from their respective backbones, we introduce specific adjustments to the reinforcement learning setup to facilitate adaptive training. We configure the group size \(G\) (number of rollouts per query) to 8 for Graph-AdaptR1 and 5 for Search-AdaptR1. For Search-AdaptR1, we add the step-wise No-Think reward to the original answer and retrieval set rewards. For the AdaptR1-specific coefficients, we set the confidence threshold \(\tau=0.6\), the reward weight \(\omega=0.2\), and the step-wise penalty factor \(\lambda=0.9\). Main and ablation experiments were conducted on NVIDIA H200 GPUs; robustness statistics in Table 9 are repeated over 5 random seeds on three representative datasets. The total computational budget for all reported experiments (across all datasets and ablation studies) was approximately 1500 GPU hours.

14 Details of Research Artifacts and Licenses↩︎

14.1 Datasets and Licenses↩︎

In this work, we utilize six publicly available datasets to evaluate the multi-hop reasoning capabilities of our model. All datasets are widely used in the research community, and our use is consistent with their intended use for research and evaluation purposes.

• 2WikiMultiHopQA [40]: A multi-hop QA dataset using structured and unstructured data. It is distributed under the Apache-2.0 License.

• HotpotQA [41]: A dataset with question-answer pairs based on Wikipedia articles, distributed under the CC BY-SA 4.0 License.

• MuSiQue [42]: A dataset for multi-hop reasoning over connected paragraphs, distributed under the CC BY 4.0 License.

• Natural Questions (NQ) [43]: A dataset consisting of queries issued to the Google search engine, distributed under the Apache-2.0 License.

• PopQA [44]: A dataset focusing on long-tail knowledge retrieval using entity-centric questions, distributed under the MIT License.

• TriviaQA [45]: A reading comprehension dataset containing question-answer-evidence triples, distributed under the Apache-2.0 License.

These datasets are primarily in English and are derived from public sources such as Wikipedia or Web snippets. Aligned with the experimental setup of Graph-R1 [38], we standardize our data usage by uniformly sampling 5,120 instances for training and 128 instances for testing per dataset, thereby balancing computational workload and consistency.

14.2 Models, Frameworks and their licenses↩︎

We conduct our training and evaluation using the following models and frameworks:

• Language Model:
  We use Qwen2.5-7B-Instruct [52] as our backbone model. Qwen2.5 is open-sourced under the Apache-2.0 License, allowing for research and commercial use.

• Retrievers:
  The choice of retriever depends on the specific method employed. In Search-R1, we utilize E5 [53]. In Graph-R1, we employ hypergraph-based retrieval equipped with bge-large-en-v1.5 [54]. Both embedding models are open-sourced under the MIT License.

• Framework:
  We implement our methods using VeRL [55], a flexible framework for reinforcement learning with LLMs. The VeRL library is open-sourced under the Apache-2.0 License.

References↩︎