1 Introduction↩︎

Large reasoning models (LRMs) have recently achieved strong performance by generating explicit thinking traces before final answers [1]–[4]. This paradigm is especially effective for mathematics, coding, and symbolic reasoning, where intermediate steps help decompose problems, verify partial results, and derive final answers [5]–[8]. Explicit thinking can also benefit factuality-oriented question answering (QA), as reasoning traces may help models recall relevant knowledge, compare candidate facts, and refine final responses. As a result, thinking has become a widely used strategy for improving modern large language models (LLMs).

However, in factuality QA, the benefit of thinking is not uniformly positive at the instance level. To examine how thinking changes model predictions, we compare non-thinking and thinking modes on TriviaQA [9]. As shown in Table 1, on Qwen3-8B [3], thinking corrects 11.74% of examples that are originally incorrect under non-thinking, confirming its factual gains. At the same time, it overturns 7.50% of originally correct non-thinking answers into incorrect predictions. We call this failure mode thinking-induced hallucination, where explicit thinking introduces misleading or unsupported content and changes a correct direct answer into a hallucinated one. This is particularly concerning because the correct answer is already accessible to the same model under the non-thinking mode, but is lost after generating an explicit thinking trace.

This instability stems from a key difference between factual QA and derivation-based reasoning. In mathematical reasoning, intermediate steps can often derive the answer from given premises. In factual QA, however, the answer usually depends on retrieving specific knowledge [10]–[12]. When the model is uncertain about the relevant fact, longer thinking does not create new evidence; instead, it may introduce unsupported associations, amplify uncertainty, and cause factual drift [13]–[16]. Thus, the central challenge is to preserve the factual gains of thinking while preventing reasoning traces from corrupting correct direct-answer tendencies. These observations motivate a residual view of explicit thinking in factuality QA. The non-thinking response reflects the model’s direct factual tendency before an explicit reasoning trace is introduced, while the thinking response adds a reasoning residual on top of this tendency. This residual is helpful when it recovers the correct fact, but harmful when it introduces unsupported associations, entity confusion, or speculative intermediate claims. From this perspective, factuality optimization should account for how thinking changes the model’s direct-answer tendency, rather than rewarding final answers in isolation.

Building on this interpretation, we propose MARGO, Mixed-Mode Advantage Regularization for Grounded Optimization, a reinforcement learning framework for mitigating thinking-induced hallucination in factuality QA. Unlike standard GRPO with all-thinking rollout groups, MARGO constructs mixed rollout groups containing both thinking and non-thinking trajectories for each question. This design makes the non-thinking response a same-model reference in advantage estimation, so thinking trajectories are optimized relative to the model’s direct-answer behavior rather than only against other thinking samples. As a result, hallucination-prone reasoning that corrupts a correct non-thinking answer receives lower relative advantage, while reasoning that improves factual correctness remains reinforced. The overall architecture is shown in Figure 1.

We train MARGO and evaluate it on two models across six factuality-oriented QA benchmarks. Experimental results show that MARGO consistently improves factual reliability over fixed-mode inference and training-based baselines, including mode selection, supervised fine-tuning, and RL variants trained with the same curated data. Evaluation on mathematical benchmarks further shows that MARGO preserves the general reasoning ability of LRMs. The contributions of this paper are summarized as follows:

• We identify the thinking-induced hallucination issue in factuality-oriented QA, where explicit thinking turns correct non-thinking answers into incorrect ones, and show through instance-level transition analysis that this phenomenon appears consistently across different LRM scales.

• We formulate explicit thinking in factuality QA as a reasoning residual over the model’s direct-answer tendency, and propose MARGO, a mixed-mode advantage regularization framework that uses non-thinking rollouts as same-model references to suppress hallucination-prone thinking while preserving trajectories that provide factual value.

2 Related Work↩︎

2.1 Hallucination in Large Reasoning Models↩︎

Although large reasoning models (LRMs) have achieved strong performance on reasoning-intensive tasks, recent studies suggest that explicit reasoning may introduce additional factuality risks. In factuality-oriented settings, long reasoning traces can introduce unsupported intermediate claims, amplify early mistakes, or produce plausible but factually incorrect conclusions [17]–[22]. For example, [19] show that LRMs generate substantially more hallucinations on long-form factuality benchmarks and propose online RL with rewards balancing factual precision, response detail, and relevance. [18] further analyze hallucination caused by reasoning-oriented RL and propose factuality-aware step-wise policy optimization. Another line of work detects or characterizes hallucinations inside reasoning traces: [23] study reasoning hallucination from a mechanistic perspective, where logically coherent but factually incorrect traces lead to persuasive yet faulty conclusions, while [24] analyze LRM hallucinations by representing reasoning trajectories as structured graphs. Different from these studies, we focus on an instance-level mode-induced failure: the same model can answer correctly without thinking, but becomes incorrect after generating an explicit thinking trace.

2.2 Reasoning Optimization and Adaptive Thinking↩︎

Reinforcement learning has become a common approach for improving the reasoning ability of LLMs, with recent methods showing that RL can elicit long-form reasoning behaviors such as exploration, reflection, and self-correction [2], [25], [26]. GRPO [26] further provides a practical framework for reasoning-oriented training by estimating relative advantages within groups of sampled responses [26]. Meanwhile, adaptive thinking methods study how to decide whether, when, or how long a model should reason, often treating thinking as a mode-selection or budget-allocation problem [27]–[33]. Different from these approaches, MARGO does not learn a question-level policy for when to think, nor does it optimize only all-thinking rollout groups. Instead, it constructs mixed rollout groups containing both thinking and non-thinking trajectories, using non-thinking responses as same-model references to regularize reasoning traces that degrade factuality while preserving those that provide factual value.

3 Method↩︎

3.1 Preliminary: Thinking and Non-Thinking Inference Modes↩︎

Recent large reasoning models (LRMs) are typically deployed under two distinct inference modes, namely thinking and non-thinking. These modes are not selected adaptively by the model itself at inference time; instead, they are determined by the prompting template.

Given an input question \(x\), let \(o=(o_1,\dots,o_T)\) denote the output sequence generated by an LRM parameterized by \(\theta\). The generation process follows the standard autoregressive factorization: \[P_\theta(o \mid x) = \prod_{t=1}^{T} P_\theta(o_t \mid x, o_{<t}).\] The difference between the two modes lies in the prompting format and the resulting structure of the generated response. In the thinking mode, the model is prompted to generate an explicit thinking trace before producing the final answer. Denoting the reasoning trace by \(r\) and the final answer by \(y\), the generated response takes the form: \[o = [\texttt{<think>},\, r,\, \texttt{</think>},\, y],\] where \(r\) is a non-empty reasoning segment enclosed by the <think> and </think> tags. In the non-thinking mode, the prompting template already provides an empty reasoning block after the question, i.e., <think></think> is pre-filled as part of the input context. As a result, the model directly generates the final answer \(y\) without producing an explicit thinking trace.

Therefore, the distinction between thinking and non-thinking does not stem from any change in model parameters, but from the prompting template and the induced generation pattern. Although both modes operate on the same underlying model, they may lead to substantially different behaviors.

3.2 Empirical Observation: Thinking-Induced Hallucination↩︎

While LRMs often benefit from additional test-time computation, whether explicit thinking is uniformly beneficial for factuality tasks remains unclear. To investigate this, we conduct preliminary experiments comparing the thinking and non-thinking modes on TriviaQA [9], a representative factuality benchmark. Beyond overall performance, we further perform a fine-grained comparison at the instance level to examine how prediction correctness changes when switching from the non-thinking mode to the thinking mode.

l0.48 6pt

Formally, for each example \(x\), let \(c^{\text{N}}(x)\in\{0,1\}\) and \(c^{\text{T}}(x)\in\{0,1\}\) denote the correctness of the prediction under the non-thinking mode and the thinking mode, respectively, where \(1\) indicates a correct prediction and \(0\) otherwise. Based on the pair \(\big(c^{\text{N}}(x), c^{\text{T}}(x)\big)\), we partition all examples into four transition groups: \[\tau_{i,j}=\{x \mid c^{\text{N}}(x)=i,\;c^{\text{T}}(x)=j\}, \quad i,j\in\{0,1\}.\] Specifically, \(\tau_{0,0}\) denotes the examples that are incorrect under both modes, \(\tau_{0,1}\) denotes the examples that change from incorrect to correct, \(\tau_{1,0}\) denotes the examples that change from correct to incorrect, and \(\tau_{1,1}\) denotes the examples that remain correct under both modes. Among these four groups, \(\tau_{1,0}\) is particularly important, as it captures the cases where thinking overturns an originally correct non-thinking prediction and induces hallucination. We further define the transition ratio of each group as: \[r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}, \quad i,j\in\{0,1\},\] where \(\mathcal{D}\) denotes the evaluation set. As shown in Table 1, although thinking can correct a portion of non-thinking incorrect predictions (i.e., \(r_{0,1}\)), there also exists a non-negligible fraction of examples in \(r_{1,0}\), where thinking overturns an originally correct non-thinking answer and leads to an incorrect prediction. We refer to this failure mode as thinking-induced hallucination. These results reveal that explicit thinking plays a dual role in factuality QA: it can recover some missed answers, but it can also introduce harmful reasoning that drifts away from an originally correct direct answer. We provide additional transition analyses in Appendix 9.

We further inspect the thinking traces in \(\tau_{1,0}\) and find that they often contain uncertainty or revision markers, such as “But wait”, “Hmm, not sure”, “Wait, no, maybe that’s not right”, and “but I’m getting confused”. These traces suggest that thinking-induced hallucination is often not caused by a complete absence of the correct fact, but by instability introduced during the thinking process. When the relevant factual knowledge is uncertain, longer thinking does not create new evidence; instead, it may amplify uncertainty, introduce entity confusion, or over-elaborate unsupported associations. Since the two modes share the same model parameters and differ mainly by whether an explicit thinking trace is generated, the non-thinking response reveals the model’s direct factual tendency before this additional thinking process is introduced. This motivates our residual view of explicit thinking: in factuality QA, thinking can be seen as a reasoning residual on top of the model’s direct-answer behavior, and the value of this residual should be evaluated relative to the non-thinking reference.

3.3 Residual Value of Explicit Thinking↩︎

The transition analysis above motivates modeling thinking as a reasoning residual over the model’s non-thinking behavior. For the same question \(x\), the non-thinking response reflects the model’s direct factual tendency, while the thinking response adds an explicit thinking trace before producing the final answer. The key question is therefore whether this additional reasoning residual provides positive factual value over the non-thinking reference.

Formally, let \(\mu_m(x)\) denote the expected factual reward under mode \(m\in\{\mathrm{T},\mathrm{N}\}\): \[\mu_m(x) = \mathbb{E}_{y^{m}\sim \pi_\theta(\cdot|x,m)} \left[ R(x,y^{m}) \right].\] We define the residual value of explicit thinking as: \[\Delta_{\mathrm{res}}(x) = \mu_{\mathrm{T}}(x)-\mu_{\mathrm{N}}(x).\] A positive \(\Delta_{\mathrm{res}}(x)\) means that thinking provides factual value beyond direct answering, while a negative value means that thinking introduces harmful residual contamination.

Standard GRPO with all-thinking rollout groups cannot directly capture this residual value. When all trajectories in a group are sampled in the thinking mode, the reward baseline is determined only by other thinking trajectories. As a result, a hallucination-prone thinking trajectory may still receive a favorable relative advantage as long as it performs better than other thinking samples in the same group. This optimizes relative quality within the thinking distribution, but does not evaluate whether thinking itself adds factual value over the model’s direct-answer behavior.

To expose the residual value of thinking, we construct mixed rollout groups containing both thinking and non-thinking trajectories. Suppose a rollout group contains thinking and non-thinking trajectories with proportions \(\alpha\) and \(1-\alpha\), respectively. The expected mixed reward baseline is: \[\mu_{\mathrm{mix}}(x) = \alpha\mu_{\mathrm{T}}(x) + (1-\alpha)\mu_{\mathrm{N}}(x).\] Ignoring the normalization scale for clarity, the expected relative advantage of a thinking trajectory becomes: \[\begin{align} A_{\mathrm{mix}}^{\mathrm{T}}(x) &\propto R(x,y^{\mathrm{T}})-\mu_{\mathrm{mix}}(x) \\ &= \underbrace{R(x,y^{\mathrm{T}})-\mu_{\mathrm{T}}(x)}_{\text{within-mode advantage}} + (1-\alpha) \underbrace{\Delta_{\mathrm{res}}(x)}_{\text{residual value}} . \end{align}\] The first term measures the relative quality of a thinking trajectory within the thinking distribution, while the second term adjusts this advantage by the residual value of thinking over the non-thinking reference. Therefore, when \(\Delta_{\mathrm{res}}(x)<0\), thinking is expected to introduce harmful residual contamination, and thinking trajectories receive lower relative advantage under the mixed baseline. When \(\Delta_{\mathrm{res}}(x)>0\), thinking provides positive residual value and is reinforced. By symmetry, non-thinking trajectories receive the opposite residual adjustment, making them relatively preferred when thinking harms factuality. We present the detailed theoretical analysis in Appendix 6.

3.4 Mixed-Mode Advantage Regularization for Grounded Optimization↩︎

Following the residual-value formulation above, we instantiate MARGO with GRPO by constructing mixed rollout groups for each factual question. The key idea is to place thinking and non-thinking trajectories into the same group, so that the GRPO advantage reflects not only within-mode reward differences but also whether thinking adds factual value over the model’s direct-answer behavior.

Given an input question \(x\), we sample both thinking and non-thinking responses: \[\mathcal{G}(x) = \{y^{\mathrm{T}}_1,\ldots,y^{\mathrm{T}}_{K_T}\} \cup \{y^{\mathrm{N}}_1,\ldots,y^{\mathrm{N}}_{K_N}\},\] where \(y^{\mathrm{T}}_i\) is sampled under the thinking prompt and \(y^{\mathrm{N}}_j\) is sampled under the non-thinking prompt. Both modes share the same model parameters and differ only in the prompting template. Thus, the mixed rollout group provides both thinking trajectories and same-model non-thinking references for the same question.

For each response \(y_i \in \mathcal{G}(x)\), we compute a reward \(r_i\) based on factual correctness and response format. GRPO then normalizes rewards within the mixed group: \[A_i = \frac{r_i - \mathrm{mean}(\{r_j\}_{j=1}^{|\mathcal{G}(x)|})}{\mathrm{std}(\{r_j\}_{j=1}^{|\mathcal{G}(x)|})}, \qquad \rho_i(\theta)= \frac{\pi_\theta(y_i\mid x,m_i)}{\pi_{\theta_{\mathrm{old}}}(y_i\mid x,m_i)},\] where \(m_i\in\{\mathrm{T},\mathrm{N}\}\) denotes the prompting mode used to sample \(y_i\). This mixed-group normalization implements the residual-value principle in Section 3.3: thinking trajectories are evaluated not only against other thinking samples, but also against non-thinking references for the same question.

The clipped GRPO objective is defined as: \[\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{x,\mathcal{G}(x)} \left[ \frac{1}{|\mathcal{G}(x)|} \sum_{y_i\in \mathcal{G}(x)} \min\Big( \rho_i(\theta)A_i,\, \mathrm{clip}\big(\rho_i(\theta),1-\epsilon,1+\epsilon\big)A_i \Big) \right].\] In practice, we additionally apply KL regularization to stabilize training and constrain the updated policy from drifting too far from the reference model.

3.4.0.1 Reward for Factuality QA.

Unlike conventional rule-based RLVR tasks such as mathematical reasoning, factuality QA does not admit a simple deterministic verifier. In math problems, the final answer can often be extracted and matched against the ground truth. In contrast, factual answers may appear in different surface forms, aliases, abbreviations, or paraphrases, making exact string matching unreliable. Therefore, we use Qwen3-32B as an automatic judge to evaluate sampled rollouts, consistent with [17]. The judge is not asked to verify factual correctness from its own parametric knowledge. Instead, it is provided with the question, the ground-truth answer, and the model prediction, and is required to determine whether the prediction is semantically consistent with the provided ground truth. The detailed judging prompt is provided in Appendix 7.

Based on the judge result, we define the reward as: \[r = r_{\mathrm{corr}} + r_{\mathrm{fmt}}.\] Here, \(r_{\mathrm{corr}}\) measures factual correctness, while \(r_{\mathrm{fmt}}\) provides a lightweight bonus for valid response formatting. The correctness reward is defined as: \[r_{\mathrm{corr}} = \begin{cases} +1, & \text{if the response is judged correct},\\ -1, & \text{otherwise}. \end{cases}\] A response is regarded as format-valid if it contains a properly closed <think> and </think> blocks. We define the format reward as: \[r_{\mathrm{fmt}} = \begin{cases} +0.05, & \text{if the response format is valid},\\ 0, & \text{otherwise}. \end{cases}\]

Since GRPO computes relative advantages over both thinking and non-thinking trajectories for the same question, a hallucination-prone thinking trajectory receives lower advantage when the non-thinking reference achieves a higher reward. This makes the regularization effect emerge from group-wise comparison rather than from manually designed mode rewards.

4 Experiments↩︎

4.1 Experimental Setting↩︎

4.1.0.1 Models and Benchmarks.

We evaluate MARGO on two LRMs, Qwen3-4B and Qwen3-8B [3]. For evaluation, we use six representative factuality-oriented benchmarks: SimpleQA [34], SimpleQA-Verified [35] (denoted as SimpleQA-V in our paper), TriviaQA [9], NQ\(\_\)Open [36], PopQA [37], and HotpotQA [38]. We use Qwen3-32B as an automatic judge to determine whether the model prediction is semantically consistent with the ground-truth answer [17]. Detailed descriptions and the judging prompt are provided in Appendix 7 and Appendix 8.2.

4.1.0.2 Training Data Construction.

We construct model-specific training data from the TriviaQA training split. For each base model, we run both thinking and non-thinking modes on the training questions and retain examples where the two modes show a clear factuality gap, including cases where thinking performs much better and cases where non-thinking performs much better. This filtering is performed separately for Qwen3-4B and Qwen3-8B, since the effect of thinking is model-dependent. Details are provided in Appendix 8.1.

4.1.0.3 Baselines.

We compare MARGO with several baselines trained under the same data setting. Fixed\(_{{NoThink}}\) and Fixed\(_{{Think}}\) denote the original model evaluated under the non-thinking and thinking modes, respectively. Adaptive\(_{{CLS}}\) trains microsoft/deberta-v3-base [39] as a binary classifier to predict whether a given question should invoke thinking. We also include Adaptive\(_{{SFT}}\), which uses the same data to teach the model when to think and when not to think via SFT. Adaptive\(_{{RL}}\) [28] also allows the model to select between thinking and non-thinking modes via reinforcement learning. In addition, to isolate the effect of mixed-mode advantage regularization from reinforcement learning alone, we include two fixed-mode RL baselines. Fixed\(_{{NoThink+RL}}\) fine-tunes the model with GRPO using the non-thinking format for all training examples, while Fixed\(_{{Think+RL}}\) fine-tunes the model with GRPO using the thinking format for all training examples. All training-based baselines use the same curated training data as MARGO. The detailed descriptions are provided in Appendix 8.3.

4.1.0.4 Training and Evaluation Details.

We train MARGO using the verl [40] framework. All experiments are conducted on 8 NVIDIA H200 GPUs, where 4 GPUs are used for policy training and the remaining 4 GPUs are used to serve the reward model. We train the model for one epoch with a training batch size of 64, a learning rate of \(1\times10^{-6}\), and a maximum response length of 8192 tokens. For each prompt, we sample 8 rollouts, including 6 thinking rollouts and 2 non-thinking rollouts; thus, the thinking-ratio coefficient \(\alpha\) is set to 0.75. We also apply KL regularization to constrain the updated policy from drifting too far from the reference model, with the KL loss coefficient set to 0.001. We evaluate all benchmarks using vLLM [41]. For factuality evaluation, we use the same judge model, Qwen3-32B, and the same prompt strategy as in training. Following the official decoding recommendations, we set the temperature to 0.6, top-\(p\) to 0.95, and top-\(k\) to 20 for thinking mode generation. For non-thinking mode generation, we set the temperature to 0.7, top-\(p\) to 0.8, and top-\(k\) to 20. The maximum generation length is set to 8192 tokens.

4.2 Experimental Results↩︎

4.2.0.1 Main Results.

Table 2 reports the results on six factuality-oriented QA benchmarks across Qwen3-4B and Qwen3-8B. Overall, MARGO achieves the best average performance on both model scales. On Qwen3-4B, MARGO improves the average accuracy from 22.81% under the fixed thinking mode to 25.49%, yielding a gain of 2.68%. On Qwen3-8B, MARGO improves the average accuracy from 27.59% to 29.57%, yielding a gain of 1.98%. These results show that explicitly regularizing thinking behavior with non-thinking references improves factual reliability beyond simply enabling the thinking mode at inference time.

Compared with adaptive mode-selection baselines, MARGO also shows clear advantages. Adaptive\(_{{CLS}}\) and Adaptive\(_{{SFT}}\) do not consistently outperform the fixed thinking mode, suggesting that a simple question-level mode-selection strategy may be insufficient in this setting. This is consistent with our motivation that the usefulness of thinking is model-specific and trajectory-dependent. In contrast, MARGO avoids making a hard mode-selection decision and instead regularizes thinking through mixed-mode advantage estimation. We further compare MARGO with RL baselines trained on the same data. Although Fixed\(_{{Think+RL}}\) and Fixed\(_{{NoThink+RL}}\) improve over some fixed-mode baselines, they remain consistently below MARGO. For example, on Qwen3-4B, Fixed\(_{{Think+RL}}\) achieves an average accuracy of 23.31%, while MARGO reaches 25.49%. On Qwen3-8B, Fixed\(_{{Think+RL}}\) reaches 27.92%, compared with 29.57% for MARGO. This indicates that the improvement does not come from reinforcement learning alone, but from the mixed-mode advantage regularization that evaluates thinking trajectories against same-model non-thinking references.

Across individual benchmarks, MARGO achieves the best performance on all six datasets for both Qwen3-4B and Qwen3-8B. The gains are especially pronounced on SimpleQA and SimpleQA-Verified, where factual precision is critical and unsupported thinking can easily introduce incorrect claims. These consistent improvements demonstrate that MARGO effectively mitigates thinking-induced hallucination while preserving the useful factual benefits of explicit thinking.

4.2.0.2 Transition Analysis.

To further understand where the gains of MARGO come from, we analyze how training changes the instance-level transition ratios on Qwen3-4B. We focus on \(\tau_{1,0}\) and \(\tau_{1,1}\), where \(\tau_{1,0}\) captures cases in which thinking overturns an originally correct non-thinking answer, and \(\tau_{1,1}\) captures cases that remain correct under thinking. Figure 2 reports the change of these two ratios relative to Fixed\(_{{Think}}\). Fixed\(_{{Think+RL}}\) produces only marginal changes in both \(r_{1,0}\) and \(r_{1,1}\), indicating that applying RL with the same data does not substantially alter the harmful transition pattern. In contrast, MARGO consistently reduces \(r_{1,0}\) across TriviaQA, NQ\(\_\)Open, and PopQA, while increasing \(r_{1,1}\) on all three datasets. This shows that the improvement of MARGO is not merely due to RL training or the curated data itself. Instead, the gains come from mixed-mode advantage regularization, where non-thinking rollouts serve as same-model references to suppress hallucination-prone thinking and preserve correct direct-answer tendencies. This also suggests that MARGO does not simply bias the model toward non-thinking outputs, but instead changes the relative optimization signal according to whether thinking improves factual correctness for the same question.

4.2.0.3 Qualitative Analysis.

We present a factuality QA case study to illustrate how MARGO mitigates reasoning-induced hallucination. As shown in Figure 3, given the question “Christmas Island is administered by which country?”, Fixed\(_{{NoThink}}\) directly produces the correct answer, Australia, indicating that the model already contains the required factual knowledge in its direct-answer mode. However, when forced to generate an explicit reasoning trace, Fixed\(_{{Think}}\) gradually drifts away from this correct knowledge. It introduces unsupported intermediate claims, confuses Christmas Island with British territories, and eventually predicts the United Kingdom as the administering country. This example demonstrates that additional reasoning can sometimes overwrite a correct direct answer with a hallucinated conclusion, rather than improving factual reliability. In contrast, MARGO preserves the correct direct-answer tendency from the non-thinking rollout while still allowing a reasoning process. By using the non-thinking rollout as a same-model reference, MARGO suppresses the erroneous factual drift in the thinking trajectory and recovers the correct answer.

4.3 Ablation Studies↩︎

4.3.0.1 Mathematical Generalization.

Although MARGO is designed for mitigating hallucination in factuality-oriented QA, we further evaluate whether it preserves the general reasoning ability of LRMs on widely used mathematical benchmarks. Specifically, we compare the model trained with MARGO against the original model under both non-thinking and thinking modes on three math benchmarks: AMC23, AIME24, and AIME25. For all three benchmarks, we report Mean@16 and Pass@16, where Mean@16 denotes the average accuracy over 16 sampled responses and Pass@16 denotes whether at least one of the 16 responses gives the correct answer. We set the maximum generation length to 32,768 for all math evaluations. As shown in Figure 2 (c), MARGO preserves mathematical reasoning ability and even yields slight gains over the base model under the thinking mode. Specifically, MARGO achieves the best Mean@16 across all three benchmarks at both the 4B and 8B scales, improving the average accuracy by 0.5% on the 4B model and 1.5% on the 8B model. The detailed results are provided in Table 4.

4.3.0.2 Data Selection Evaluation.

Since MARGO uses non-thinking rollouts as references to regularize thinking rollouts, the informativeness of the training prompts is important for effective optimization. Ideally, training examples should exhibit a clear factuality gap between thinking and non-thinking modes, so that the mixed rollout group can provide meaningful reference signals. To evaluate the effect of our data selection strategy, we conduct an ablation study on Qwen3-4B. We compare MARGO trained on our selected training set with MARGO trained on a random subset of 5,660 examples sampled from the same TriviaQA training pool. All training and evaluation settings are kept identical. As shown in Table 3, random data selection yields limited improvement over the fixed thinking baseline, while our selected data leads to consistent gains across all benchmarks. This suggests that examples with clear thinking/non-thinking factuality gaps are more effective for mixed-mode advantage regularization.

5 Conclusion↩︎

In this paper, we studied factuality-oriented QA in large reasoning models (LRMs) and identified thinking-induced hallucination, where explicit thinking can overturn correct non-thinking answers and lead to factual drift. We interpreted explicit thinking as a reasoning residual over the model’s direct-answer behavior, which may either recover missing knowledge or introduce unsupported associations. Based on this view, we proposed MARGO, a mixed-mode reinforcement learning framework that uses non-thinking rollouts as same-model references to regularize hallucination-prone thinking. Experiments across multiple factuality-oriented QA benchmarks show that MARGO improves factual reliability over strong baselines, while preserving general reasoning ability on mathematical benchmarks. Overall, our findings suggest that improving LRMs requires not only encouraging more thinking, but also learning when explicit thinking provides factual value over direct answering.

Limitations↩︎

This work focuses on improving the factual reliability of LRMs in factuality-oriented QA. While our experiments cover multiple benchmarks and model scales, future work can further explore the applicability of mixed-mode advantage regularization to broader scenarios, such as open-ended generation, multi-turn interaction, retrieval-augmented systems, and multimodal reasoning.

Broader Impact↩︎

By reducing cases where explicit thinking turns correct direct answers into hallucinated predictions, MARGO can help improve the reliability of LRMs in information-seeking and educational applications. At the same time, the method is intended to complement, rather than replace, external verification mechanisms in high-stakes domains. We hope this work encourages future research on more reliable and mode-aware optimization for LRMs.

6 Advantage Decomposition of Mixed-Mode Regularization↩︎

In this section, we provide a formal derivation showing why mixed-mode rollout groups introduce a residual-value adjustment that is absent in standard all-thinking GRPO. This analysis characterizes the advantage signal induced by group normalization, rather than providing a convergence guarantee.

6.0.0.1 Setup.

For a factual question \(x\), let \(\mathrm{T}\) and \(\mathrm{N}\) denote the thinking and non-thinking modes, respectively. A response sampled under mode \(m\in\{\mathrm{T},\mathrm{N}\}\) is denoted by \(y^m\sim \pi_\theta(\cdot\mid x,m)\), and its factual reward is \(R(x,y^m)\). We define the expected reward of each mode as \[\mu_m(x) = \mathbb{E}_{y^m\sim \pi_\theta(\cdot\mid x,m)} \left[ R(x,y^m) \right], \qquad m\in\{\mathrm{T},\mathrm{N}\}.\] The residual value of explicit thinking is defined as \[\Delta_{\mathrm{res}}(x) = \mu_{\mathrm{T}}(x)-\mu_{\mathrm{N}}(x).\] A positive \(\Delta_{\mathrm{res}}(x)\) indicates that thinking provides higher expected factual reward than direct answering, while a negative value indicates that thinking harms factuality relative to the non-thinking reference.

6.0.0.2 All-Thinking GRPO.

Consider a standard all-thinking rollout group where all samples are generated in the thinking mode. Ignoring the standard-deviation normalization for clarity, the group-relative advantage of a thinking response \(y^{\mathrm{T}}\) is centered around the all-thinking reward baseline: \[A_{\mathrm{T}}(x,y^{\mathrm{T}}) \propto R(x,y^{\mathrm{T}})-\mu_{\mathrm{T}}(x).\] This advantage compares \(y^{\mathrm{T}}\) only against other thinking-mode samples. Therefore, it can rank thinking trajectories within the thinking distribution, but it does not reveal whether thinking itself improves factuality over the model’s direct non-thinking behavior.

6.0.0.3 Mixed-Mode GRPO.

Now consider a mixed rollout group containing thinking and non-thinking trajectories with proportions \(\alpha\) and \(1-\alpha\), respectively. The expected group reward baseline is \[\mu_{\mathrm{mix}}(x) = \alpha\mu_{\mathrm{T}}(x) + (1-\alpha)\mu_{\mathrm{N}}(x).\]

6.0.0.4 Implication.

Proposition 1 shows that mixed-mode advantage estimation introduces a residual-value adjustment that is absent from all-thinking GRPO. For thinking trajectories, this adjustment is proportional to \(\Delta_{\mathrm{res}}(x)\). When \(\Delta_{\mathrm{res}}(x)>0\), thinking has higher expected factual reward than non-thinking, and the mixed-mode advantage increases the relative preference for thinking trajectories. When \(\Delta_{\mathrm{res}}(x)<0\), thinking harms factuality relative to direct answering, and the mixed-mode advantage reduces the relative preference for thinking trajectories. Thus, mixed-mode GRPO does not merely reward correct final answers; it adjusts the optimization signal according to whether explicit thinking provides factual value beyond the model’s direct-answer behavior.

6.0.0.5 Relation to MARGO.

MARGO instantiates this principle by constructing each GRPO rollout group with both thinking and non-thinking trajectories for the same question. The reward function itself does not explicitly prefer either mode, nor does it impose a handcrafted penalty on thinking length. Instead, the regularization effect emerges from group-wise advantage normalization: non-thinking trajectories serve as same-model references that expose whether the reasoning residual introduced by thinking is beneficial or harmful for factuality.

7 Prompts↩︎

We present the prompt used for factuality evaluation. Specifically, the judge model takes as input the question, the ground-truth answer (denoted as the gold target in the prompt), and the model prediction (denoted as the predicted answer in the prompt). Based on the consistency between the prediction and the ground truth, the judge model outputs one of three labels: “A”, “B”, or “C”, representing correct, incorrect, and not attempted, respectively. For evaluation, we count only “A” as a correct prediction, while both “B” and “C” are treated as incorrect.

Importantly, this is different from using an LLM as an open-ended factual verifier. The judge is not asked to determine factual truth based on its own parametric knowledge. Instead, it is explicitly provided with the question, the ground-truth answer, and the model prediction, and is only required to compare whether the prediction matches the provided ground truth. This ground-truth-conditioned evaluation reduces reliance on the judge model’s internal knowledge and focuses the evaluation on semantic correctness rather than surface-form overlap.

Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"].
First, I will give examples of each grade, and then you will grade a new example.


The following are examples of CORRECT predicted answers.
```
Question: What are the names of Barack Obama's children?
Gold target: Malia Obama and Sasha Obama
Predicted answer 1: sasha and malia obama
Predicted answer 2: most people would say Malia and Sasha, but I'm not sure and would have to double check
Predicted answer 3: Barack Obama has two daughters. Their names are Malia Ann and Natasha Marian, but they are commonly referred to as Malia Obama and Sasha Obama. Malia was born on July 4, 1998, and Sasha was born on June 10, 2001.
```
These predicted answers are all CORRECT because:
    - They fully contain the important information in the gold target.
    - They do not contain any information that contradicts the gold target.
    - Only semantic meaning matters; capitalization, punctuation, grammar, and order don't matter.
    - Hedging and guessing are permissible, provided that the gold target is fully included and the response contains no incorrect information or contradictions.


The following are examples of INCORRECT predicted answers.
```
Question: What are the names of Barack Obama's children?
Gold target: Malia and Sasha
Predicted answer 1: Malia.
Predicted answer 2: Malia, Sasha, and Susan.
Predicted answer 3: Barack Obama does not have any children.
Predicted answer 4: I think it's either Malia and Sasha. Or it could be Malia and Jackie. Or it could be Joey and Malia.
Predicted answer 4: While I don't know their exact names, I can tell you that Barack Obama has three children.
Predicted answer 5: It's possible you may mean Betsy and Olivia. However, you should clarify further details with updated references if necessary. Is that the correct answer?
Predicted answer 6: It may be the case that Obama's child is named James. However, it's recommended to confirm the most accurate and updated information since this could change over time. This model may not always reflect the most current information.
```
These predicted answers are all INCORRECT because:
    - A factual statement in the answer contradicts the gold target. Incorrect statements that have some hedging (e.g., "it is possible that", "although i'm not sure, i think") are also considered incorrect.


The following are examples of NOT_ATTEMPTED predicted answers.
```
Question: What are the names of Barack Obama's children?
Gold target: Malia and Sasha
Predicted answer 1: I don't know.
Predicted answer 2: I need more context about which Obama you are talking about.
Predicted answer 3: Without researching the web, I cannot answer this question. However, I can tell you that Barack Obama has two children.
Predicted answer 4: Barack Obama has two children. I know that one of them is Malia, but I'm not sure about the other one.
```
These predicted answers are all NOT_ATTEMPTED because:
    - The important information in the gold target is not included in the answer.
    - No statements in the answer contradict the gold target.


Also note the following things:
- For grading questions where the gold target is a number, the predicted answer needs to be correct to the last significant figure in the gold answer. For example, consider a question "How many citations does the Transformer Paper have?" with gold target "120k". 
    - Predicted answers "120k", "124k", and 115k" are all CORRECT. 
    - Predicted answers "100k" and "113k" are INCORRECT. 
    - Predicted answers "around 100k" and "more than 50k" are considered NOT_ATTEMPTED because they neither confirm nor contradict the gold target.
- The gold target may contain more information than the question. In such cases, the predicted answer only needs to contain the information that is in the question.
    - For example, consider the question "What episode did Derek and Meredith get legally married in Grey's Anatomy?" with gold target "Season 7, Episode 20: White Wedding". Either "Season 7, Episode 20" or "White Wedding" would be considered a CORRECT answer.
- Do not punish predicted answers if they omit information that would be clearly inferred from the question.
    - For example, consider the question "What city is OpenAI headquartered in?" and the gold target "San Francisco, California". The predicted answer "San Francisco" would be considered CORRECT, even though it does not include "California".
    - Consider the question "What award did A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity win at NAACL '24?", the gold target is "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered CORRECT, because "award" is presumed in the question.
    - For the question "What is the height of Jason Wei in meters?", the gold target is "1.73 m". The predicted answer "1.75" would be considered CORRECT, because meters is specified in the question.
    - For the question "What is the name of Barack Obama's wife?", the gold target is "Michelle Obama". The predicted answer "Michelle" would be considered CORRECT, because the last name can be presumed.
- Do not punish for typos in people's name if it's clearly the same name. 
    - For example, if the gold target is "Hyung Won Chung", you can consider the following predicted answers as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung".


Here is a new example. Simply reply with either CORRECT, INCORRECT, NOT ATTEMPTED. Don't apologize or correct yourself if there was a mistake; we are just trying to grade the answer.
```
Question: {question}
Gold target: {target}
Predicted answer: {predicted_answer}
```

Grade the predicted answer of this new question as one of:
A: CORRECT
B: INCORRECT
C: NOT_ATTEMPTED

Just return the letters "A", "B", or "C", with no text around it.

8 Additional Experimental Details↩︎

8.1 Training Data Construction↩︎

We construct our training data from the TriviaQA training split. The original TriviaQA training split contains 138,384 question-answer pairs, but many questions are duplicated. After removing duplicated questions, we obtain 76,523 unique training examples.

For each base model, i.e., Qwen3-4B and Qwen3-8B, we independently construct a model-specific training set. Specifically, for each question \(x\), we sample \(N\) responses under the thinking mode and \(N\) responses under the non-thinking mode. We then use Qwen3-32B as the automatic judge to determine whether each sampled response is correct with respect to the ground-truth answer. In our setting, \(N\) is set to 6 by default.

Let \(c^{\mathrm{T}}_k(x)\in\{0,1\}\) and \(c^{\mathrm{N}}_k(x)\in\{0,1\}\) denote the correctness of the \(k\)-th sampled response under the thinking and non-thinking modes, respectively. We compute the empirical correctness ratios of the two modes as: \[S_{\mathrm{T}}(x) = \frac{1}{N} \sum_{k=1}^{N} c^{\mathrm{T}}_k(x), \qquad S_{\mathrm{N}}(x) = \frac{1}{N} \sum_{k=1}^{N} c^{\mathrm{N}}_k(x).\] Here, \(S_{\mathrm{T}}(x)\) and \(S_{\mathrm{N}}(x)\) estimate the model’s factual correctness under the thinking and non-thinking modes, respectively.

We further compute the average mode score: \[b(x) = \frac{S_{\mathrm{T}}(x)+S_{\mathrm{N}}(x)}{2},\] and define the mode-specific relative scores as: \[A_{\mathrm{T}}(x) = S_{\mathrm{T}}(x)-b(x), \qquad A_{\mathrm{N}}(x) = S_{\mathrm{N}}(x)-b(x).\] Equivalently, the factuality gap between the two modes is: \[\Delta(x) = S_{\mathrm{T}}(x)-S_{\mathrm{N}}(x).\]

We retain examples where the two modes show a clear factuality gap. For Qwen3-4B, an example is selected as a non-thinking-favored example if: \[\Delta(x) \leq -0.7 \quad \text{and} \quad S_{\mathrm{N}}(x) \geq 0.7.\] This condition indicates that the model is substantially more accurate under the non-thinking mode than under the thinking mode. Similarly, an example is selected as a thinking-favored example if: \[\Delta(x) \geq 1.0 \quad \text{and} \quad S_{\mathrm{T}}(x) \geq 1.0.\] This condition selects questions where the model consistently answers correctly under the thinking mode while performing worse under the non-thinking mode.

For Qwen3-8B, we use a slightly relaxed threshold, since the relative gap between thinking and non-thinking modes becomes less extreme as the base model becomes stronger. Specifically, we select non-thinking-favored examples using: \[\Delta(x) \leq -0.5 \quad \text{and} \quad S_{\mathrm{N}}(x) \geq 0.5,\] and thinking-favored examples using: \[\Delta(x) \geq 0.7 \quad \text{and} \quad S_{\mathrm{T}}(x) \geq 0.7.\]

The final training set is the union of the selected examples from both directions. This construction focuses training on questions where thinking and non-thinking exhibit clear mode-dependent factual differences, which provide informative signals for mixed-mode GRPO. All filtering is performed only on the TriviaQA training split, and no information from any test benchmark is used. This procedure yields 5,660 training examples for Qwen3-4B and 5,721 training examples for Qwen3-8B.

Although the resulting training sets are relatively small, they are sufficient for our setting because we start from existing reasoning models rather than training reasoning behavior from scratch. Our goal is not to teach the model new factual knowledge or build reasoning ability from a base model, but to regularize when explicit thinking is helpful or harmful relative to the model’s own non-thinking behavior. Therefore, a compact set of examples with clear mode-dependent factual differences provides targeted supervision for our mixed-mode optimization.

8.2 Models and Benchmarks.↩︎

We evaluate MARGO on two LRMs, Qwen3-4B and Qwen3-8B [3], both of which exhibit strong reasoning ability across diverse tasks. For evaluation, we use six representative factuality-oriented benchmarks: SimpleQA [34], SimpleQA-Verified [35] (denoted as SimpleQA-V in our paper), TriviaQA [9], NQ\(\_\)Open [36], PopQA [37], and HotpotQA [38]. These datasets cover a broad range of factual knowledge, from short-form fact retrieval to multi-hop open-domain question answering, allowing us to evaluate factual reliability across different types of factuality tasks. Unlike mathematics or multiple-choice tasks, factuality QA cannot always be reliably evaluated by exact string matching. The same correct answer may appear in different surface forms, aliases, abbreviations, or paraphrases. Therefore, we use Qwen3-32B as an automatic judge to determine whether the model prediction is semantically consistent with the ground-truth answer.

8.3 Baselines↩︎

We compare MARGO with several baselines trained or evaluated under the same data setting. Fixed\(_{{NoThink}}\) and Fixed\(_{{Think}}\) denote the original model evaluated under the non-thinking and thinking modes, respectively, measuring the default behavior of LRMs under fixed generation modes.

Adaptive\(_{{CLS}}\) trains microsoft/deberta-v3-base [39] as a binary classifier to predict whether a given question should invoke thinking. This baseline examines whether the decision of when to think can be learned as a question-level classification problem.

We also include Adaptive\(_{{SFT}}\), which uses the same curated data to teach the model when to think and when not to think through supervised fine-tuning. For examples where thinking is preferred, we construct the SFT target as the model-generated thinking process followed by the final answer. For examples where non-thinking is preferred, we use an empty thinking block followed by the final answer. All constructed targets follow the official response format, and all target answers are judged correct.

We further compare with Adaptive\(_{{RL}}\) [28], an adaptive thinking baseline originally designed to improve inference efficiency for mathematical reasoning. In our setting, we adapt it to factuality QA by training the model to decide when to skip thinking when thinking is unnecessary or potentially harmful.

In addition, to isolate the effect of mixed-mode advantage regularization from reinforcement learning alone, we include two fixed-mode RL baselines. Fixed\(_{{NoThink+RL}}\) fine-tunes the model with GRPO using the non-thinking format for all training examples, while Fixed\(_{{Think+RL}}\) fine-tunes the model with GRPO using the thinking format for all training examples. All training-based baselines use the same curated training data as MARGO.

8.4 Detailed Results↩︎

We provide the detailed results for Figure 2 (c) in Table 4. Specifically, we report Mean@16 and Pass@16 on AMC23, AIME24, and AIME25 for both Qwen3-4B and Qwen3-8B. Compared with Fixed\(_{NoThink}\) and Fixed\(_{Think}\), MARGO achieves the best Mean@16 on all three benchmarks across both model scales. These results show that MARGO improves factual reliability while preserving the general reasoning ability.

8.5 Generation Length Statistics↩︎

Table 5 reports the average generation length of Fixed\(_{{Think}}\) and MARGO on TriviaQA, NQ\(\_\)Open, and PopQA. The length includes both the thinking trace and the final response. On Qwen3-4B, MARGO produces slightly shorter generations than Fixed\(_{{Think}}\), with relative changes ranging from \(-5.24\%\) to \(-3.42\%\). On Qwen3-8B, however, MARGO produces slightly longer generations, with relative changes ranging from \(+0.09\%\) to \(+1.48\%\). These results show that MARGO does not impose a consistent shortening effect on the generated outputs, which is consistent with its design of regularizing reasoning through mixed-mode advantage estimation rather than an explicit length penalty.

9 Additional Transition Analysis↩︎

Beyond the TriviaQA analysis in Section 3.2, we further examine the transition behavior on additional benchmarks, including NQ\(\_\)Open and PopQA. As shown in Tables 6 and 7, thinking-induced hallucination also consistently appears on these factuality-oriented QA benchmarks. On NQ\(\_\)Open, the effect is particularly pronounced. For the 8B, 14B, and 32B models, \(r_{1,0}\) reaches 8.12%, 7.89%, and 7.62%, respectively, which is close to the corresponding \(r_{0,1}\) values of 9.42%, 8.95%, and 9.61%. This indicates that, on NQ\(\_\)Open, the fraction of examples harmed by thinking is comparable to the fraction of examples corrected by thinking. A similar trend also appears on PopQA, where \(r_{1,0}\) remains around 4%–5% for most model scales, showing that correctness-to-incorrectness transitions are not limited to a single factuality benchmark.

We further compare this trend with GSM8K [42], a mathematical reasoning benchmark. As shown in Table 8, the \(r_{1,0}\) ratios on GSM8K remain below 2% across all model scales, with values ranging from 0.88% to 1.84%. Meanwhile, the \(r_{1,1}\) ratios are consistently high, reaching over 86% for models from 4B to 32B. This suggests that, unlike factuality-oriented QA, explicit thinking in mathematical reasoning is much less likely to overturn an originally correct prediction. This contrast aligns with the nature of the tasks: factuality QA often requires retrieving specific knowledge, where additional thinking may introduce unsupported associations or entity confusion; in mathematical reasoning, intermediate steps are more likely to be grounded in the problem conditions and help derive the final answer.

Overall, these additional analyses further support the key observation of our paper: thinking-induced hallucination is a recurring issue in factuality-oriented QA, rather than an artifact of a single benchmark. They also show that the risk of harmful thinking is task-dependent, being more prominent in knowledge-intensive factual QA than in mathematical reasoning.

References↩︎