1 Introduction↩︎

Retrieval-Augmented Generation (RAG) extends Large Language Models (LLMs) with non-parametric access to external corpora and has become a standard framework for knowledge-intensive tasks [1], [2]. Early RAG systems primarily relied on single-turn pre-generation retrieval. However, this setting is often insufficient for complex queries that require decomposition or information synthesis from multiple sources. As a result, recent work has shifted toward agentic search, in which reasoning models interleave internal deliberation with iterative retrieval actions over multiple steps. Recent advances have shown that reinforcement learning (RL) can be used to optimize these agents directly from task rewards, leading to search-aware reasoning systems such as Search-R1 [3] and DeepResearcher [4].

Most existing agentic search approaches primarily optimize the reasoning agent while treating the retrieval model as a frozen black-box component. This design implicitly assumes that a sufficiently capable reasoning model can compensate for retrieval failures through improved query reformulation alone. This paper challenges this assumption by arguing that sub-optimal retrieval can be a bottleneck in agentic search performance. Recent studies such as Agentic-R [5] and CoSearch [6] attempt to address this issue by jointly optimizing retrievers and reasoning agents. In practice, however, these methods are difficult to apply in settings where the reasoning model cannot be further trained, the retriever is externally provided, or gold-passage supervision is unavailable.

To address this, we propose Critic-R, a framework that closes the feedback loop between the reasoning agent and retriever during both inference and training time. Instead of blindly accepting the retrieved documents provided by the retriever, Critic-R enables the agent to assess whether they satisfy its current information requirements before proceeding to subsequent retrieval or reasoning steps. We employ a separate critic model for this purpose for two reasons. First, it allows the framework to be applied to arbitrary reasoning agents without requiring built-in self-criticism or modifications to the underlying model. Second, in long multi-step trajectories, accumulated context and reasoning noise can cause the reasoning model to become overconfident or less sensitive to retrieval failures [7], motivating the use of a dedicated evaluation component. To achieve this, the critic analyzes the agent’s introspective reasoning trace—namely, the reasoning generated immediately after consuming the retrieved evidence—to determine whether the retrieved context is sufficient for the next reasoning step. This design uses the observation that the agent often explicitly indicates whether the retrieved documents contain the information required to continue reasoning.

This verification signal enables two complementary mechanisms: (1) Critic-R-Zero (Inference-Time Scaling): An iterative reasoning-evaluation loop that operates entirely at inference time without additional gradient updates. When the critic determines that the retrieved evidence is insufficient, it rewrites the retrieval query and instruction for another retrieval attempt. This process continues until the agent is satisfied with the retrieved evidence or a predefined refinement budget is exhausted. Importantly, the reasoning agent itself remains unchanged and only interacts with retrieved documents, while the refinement process is handled externally. This mechanism dynamically allocates additional inference-time computation to recover from retrieval failures. (2) Critic-Embed (Retriever Fine-Tuning): To amortize the computational overhead introduced by iterative refinement, we leverage the execution trajectories generated by Critic-R-Zero as a source of automatic supervision for retrieval model training. Documents that satisfy the reasoning agent are treated as positive examples, while documents rejected during unsuccessful refinement attempts are treated as hard intra-trajectory negatives. Using this intra-trajectory contrastive learning, we fine-tune the retrieval model without requiring relevance information (i.e., gold passages). These two components are complementary and can be combined within a unified system, where a trained retriever can benefit from the inference-time refinement capabilities.

We evaluate Critic-R on several challenging multi-hop question answering benchmarks, including HotpotQA [8], 2WikiMultihopQA [9], MuSiQue [10], and Bamboogle [11]. Our results demonstrate that explicitly modeling retrieval quality within the agentic reasoning loop leads to substantial improvements in downstream answer accuracy. First, we show that the inference-time refinement mechanism, Critic-R-Zero, significantly alleviates retrieval failures across reasoning agents of different scales by iteratively evaluating retrieved evidence and refining retrieval queries, yielding an overall relative improvement of 12.4%. We further demonstrate that the fine-tuned dense retriever, Critic-Embed, consistently outperforms both off-the-shelf retrievers and prior co-trained baselines, achieving up to an overall 7.5% relative improvement. Finally, combining the trained retriever with the inference-time refinement loop—i.e., integrating Critic-R-Zero and Critic-Embed into a unified system called Critic-R—yields the strongest overall performance, achieving a 10.9% relative improvement overall. To support future research on this topic, we release our code, data, and trained models.¹

2 Related Work↩︎

2.0.0.1 Retrieval-Augmented Generation and Agentic Search.

Retrieval-Augmented Generation (RAG) extends Large Language Models with non-parametric access to external corpora and has become the standard recipe for knowledge-intensive QA [12]–[14]. Early systems issued a single query at the start of generation, which is ill-suited to multi-hop questions whose information needs only become apparent partway through reasoning [15], [16]. Two lines of work have addressed this. The first uses prompting to interleave reasoning with retrieval, exemplified by IRCoT [16] and ReAct [17]. The second teaches models when and how to retrieve through supervised fine-tuning, including Self-RAG [18] and Toolformer [19]. More recently, reinforcement learning has enabled agents to acquire multi-turn search policies directly from task-outcome rewards: Search-R1 [3], R1-Searcher [20], ReSearch [21], and DeepResearcher [4] all train an LLM to alternate <think>, <search>, and <answer> turns, producing a search-aware reasoner. Our work builds directly on this paradigm. We use Search-R1 as our reasoning agent, but is orthogonal to its training objective: rather than modifying how the agent is trained, Critic-R intervenes at inference time to inspect and repair the agent’s individual retrieval calls. The gains from improving how the agent interacts with retrieval are largely orthogonal to gains from improving the agent itself.

2.0.0.2 Retrieval optimization for Agents.

A complementary line of work also rejects the frozen-retriever assumption, but addresses it through additional training of the retrieval side. REPLUG [22] and Atlas [23] optimize the retriever using generator likelihood as a signal; later approaches use task-level metrics or LLM-judged passage utility [24]–[26]. Two recent works extend this idea explicitly to the agentic-search setting and make the retrieval bottleneck their central claim. Agentic-R [5] trains a retriever tailored for multi-turn search by jointly modeling local query-passage relevance and global answer correctness, and iteratively co-optimizes the retriever with the agent. CoSearch [6] quantifies the bottleneck directly through an oracle-retrieval experiment. They show double-digit relative F1 gains when correct documents are guaranteed to appear and jointly train a generative reranker alongside the reasoning agent with GRPO, using a composite reward over ranking quality and final answer correctness. We share the diagnosis with these works but split the problem along a different seam. The first is Critic-R-Zero, a purely inference-time loop that requires no gradient updates anywhere: a separate critic model inspects each retrieval, judges whether the returned context is sufficient based on the reasoning agent’s feedback, and rewrites the search instruction and query when it is not. Critic-R-Zero treats the underlying retriever as a fixed black box and is therefore composable with any retriever, including those produced by Agentic-R or CoSearch. The second part, Critic-Embed, is a retriever trained on the trajectories that Critic-R-Zero collects on the train splits of two QA datasets, turning the critic’s free-form feedback into supervision without ever requiring gold passage annotations. The full system, Critic-R, is Critic-R-Zero running on top of Critic-Embed. This decomposition allows us to address both lines of prior work: for frozen-retriever agentic search, we introduce Critic-R-Zero; and for retriever-training approaches, we provide a training recipe supervised entirely by the inference loop’s own feedback.

2.0.0.3 Inference-Time Scaling for Reasoning.

Parallel to the work above, recent work shows that allocating more compute at inference through longer chain-of-thought, self-consistency, or process supervision [27]–[29] can rival the gains from scaling model parameters. OpenAI’s o1-style models and subsequent open-source reasoners take this further by training models to spend more tokens deliberating before answering. Critic-R-Zero can be viewed as inference-time scaling targeted at the retrieval bottleneck rather than the reasoning trace: each additional refinement attempt and each step-up in critic size is a controlled investment of compute aimed specifically at recovering from a bad retrieval.

3 The Critic-R Framework↩︎

This section presents the approaches that lead to the development of Critic-R. First, we describe Critic-R-Zero, an inference-time critic on retrieval results for query refinement that does not require any additional training. Second, we introduce Critic-Embed by explaining how Critic-R-Zero can be used to optimize retrieval models through a novel intra-trajectory contrastive learning approach. Next, we describe how these two approaches can complement each other to form Critic-R. Last, we describe our implementation details.

3.1 Critic-R-Zero for Inference-Time Query Refinement↩︎

As shown in Algorithm 2 and Figure 1, we assume access to a frozen reasoning agent \(\mathcal{M}_R\) operating under the ReAct framework [17], which is allowed to perform at most \(M\) actions to answer a question \(Q\) (Line [line:for95loop95max95iterations]). At each step \(i\), the agent \(\mathcal{M}_R\) produces a reasoning trace \(T_i\) and an action \(A_i\) (Line [line:sample95reasoning95and95action]), which are appended to the overall trajectory (Line [line:append95to95the95trajectory]). If \(A_i\) is a final answer, the trajectory terminates (Line [line:if95answer95break]). Otherwise, if \(A_i\) is a search action, the agent extracts an initial query \(q_i^{(1)}\), which is augmented with a default instruction² \(I_i^{(1)}\) for the retrieval model \(\mathcal{R}\) (Lines [line:extract95query]–[line:default95instruction]). Entering the search phase, an instruction-aware retrieval model \(\mathcal{R}\) returns the top \(k\) documents, \(D_i^{(t)} = \mathcal{R}(q^{(t)}_i, I^{(t)}_i, k)\) (Line [line:retriever95docs]). Note that these initial retrieved documents frequently fail to provide the necessary evidence, making single-turn retrieval a severe bottleneck for agentic search performance. This dissatisfaction is typically reflected in the model’s subsequent reasoning trace. To address this limitation, we introduce a speculative refinement loop (Line [line:refine95query95at95most95k95times]) that allows the agent to recover from an ineffective retrieval. At each refinement step, the retrieved documents \(D_i^{(t)}\) are speculatively provided to the reasoner to generate an introspective reasoning trace, without yet committing these documents to the persistent trajectory (Line [line:sample95next95thinking95and95action95based95on95retrieved95docs]). A separate critic model, denoted as \(\mathcal{M}_C\), then evaluates this trace to determine whether the retrieved evidence sufficiently resolves the reasoner’s information need (Line [line:check95if95reasoner95is95satisfied95with95documents]). If the critic produces positive feedback, \(\sigma_i^{(t)} = \text{yes}\), the retrieved documents are added to the candidate positive set \(\tilde{\mathcal{D}}^{+}\) as useful evidence, and the refinement loop terminates (Lines [line:add95retrieved95documents95as95positive]–[line:break95refinement95loop]). Otherwise, the documents are assigned to the candidate negative set \(\tilde{\mathcal{D}}^{-}\) (Line [line:add95retrieved95documents95as95negative]). The critic then leverages the reasoner’s explicit dissatisfaction in its thinking trace to generate a refined query and retrieval instruction for the next iteration (Line [line:generate95new95query95and95instruction95for95next95round]). This process repeats for at most \(K\) refinement steps, adaptively bridging retrieval failures before the final selected documents are committed to the reasoning history (Line [line:commit95final95documents95to95history]). The procedure concludes by returning the final extracted answer together with the collected positive and negative document sets (Lines [line:if95answer95final]–[line:return95final95outputs]). These document sets are used exclusively for retriever training and are not required during inference.

3.2 Critic-Embed: Intra-Trajectory Contrastive Learning for Training Instruction-Following Dense Retrieval↩︎

While the inference-time refinement loop effectively detects and repairs retrieval failures, repeated interaction with the critic introduces additional computational overhead. To amortize this cost and permanently improve the retrieval backbone without relying on expensive human-annotated gold passages, we leverage the execution trajectories generated by Critic-R-Zero as a source of automatically constructed supervision. Specifically, each refinement trajectory produces a natural training signal: documents that satisfy the reasoner, as verified by the critic, are treated as positive examples (\(\mathcal{D}^{+}\)), while documents rejected during earlier unsuccessful refinement attempts are treated as hard intra-trajectory negatives (\(\mathcal{D}^{-}\)). We use the collected supervision signals to fine-tune the retriever, resulting in Critic-Embed, a retriever trained to better align retrieved evidence with the information requirements of the reasoning agent for a given query. To ensure label quality, we retain only trajectories whose final prediction is correct according to the downstream task metric. The contrastive learning loss for each training instance is defined as: \[\mathcal{L} = - \log \frac{\exp\!\big(\mathrm{sim}(q_i, z_i^+)/\tau\big)}{\sum_{z \in \mathcal{Z}_i} \exp\!\big(\mathrm{sim}(q_i, z)/\tau\big)}\] where \(q_i\) is the query embedding, \(z_i^+\) is its paired positive document, \(\mathcal{Z}_i\) contains the positive, all in-batch negatives, and the query’s intra-trajectory hard negatives, \(\mathrm{sim}(\cdot,\cdot)\) denotes cosine similarity, and \(\tau\) is the temperature.

3.3 Critic-R: Improving Critic-Embed with Inference-Time Scaling↩︎

The complete Critic-R pipeline pairs the Critic-R-Zero inference loop with the trained Critic-Embed retriever. This composition ensures the agent starts with a highly capable, domain-aligned retrieval that minimizes initial search errors, while still maintaining the inference-time ability to introspect and recover from complex edge-case retrieval failures.

3.4 Implementation Details↩︎

The reasoning agent \(\mathcal{M}_R\) is an instruction-tuned LLM operating under the ReAct paradigm [17], alternating <think>, <search>, and <answer> actions, with retrieved documents injected within <information> tags. The critic \(\mathcal{M}_C\) is a separate LLM that operates in two sequential modes: a satisfaction judgment mode that emits a binary verdict \(\sigma_i^{(t)}\) and diagnostic reason \(r_i^{(t)}\) conditioned on \((Q, q_i^{(t)}, D_i^{(t)}, T_{i+1}^{(t)})\), and a query refinement mode invoked only when \(\sigma_i^{(t)}=\texttt{no}\) that rewrites the sub-query and retrieval instruction using \(r_i^{(t)}\). Full component descriptions and all prompts are deferred to Appendix 7.

4 Experiments↩︎

We structure our evaluation around four questions:

• RQ1. Can the retrieval bottleneck in agentic search be mitigated without modifying the retriever itself, and, how does the gain scale with the critic model’s parameters size?

• RQ2. Do the trajectories that Critic-R-Zero collects contain transferable retrieval supervision i.e., does fine-tuning a retriever with them (Critic-Embed) beat both an off-the-shelf dense retriever and a retriever co-trained end-to-end with the search agent?

• RQ3. Can combining inference-time query refinement loop and the trained retriever yield further gains?

• RQ4. Is the agent’s introspective feedback \(T_{i+1}\) a key source of supervisory signal that Critic-Embed inherits?

4.1 Datasets↩︎

Following previous work [3], we evaluate our method on four multi-hop QA datasets requiring synthesizing information across multiple documents: HotpotQA [8], 2WikiMultihopQA [9], MuSiQue [10], and Bamboogle [11]. To assess whether the critic loop also helps when a single-hop retrieval is sufficient, we additionally report results on three general-domain QA datasets: NQ [30], TriviaQA [31], and PopQA [32] for Critic-R-Zero experiments. Dataset statistics are reported in Table 6 in Appendix 8. For evaluation, we report Exact Match (EM) and token-level F1, following prior work [3].

4pt

4.2 Experimental Setup↩︎

We use the optimized checkpoints of Search-R1 [3] for reasoning models, which are instruction-tuned GRPO-trained variants of Qwen2.5-Instruct³. To study scale, we evaluate three sizes: SearchR1-Qwen2.5-3B,-7B, and -14B. For the critic, we use frozen instruction-tuned Qwen2.5 with 14B⁴, 32B ⁵, and 72B ⁶ parameters.

The frozen-retriever experiments use a dense retriever based on the Stella-400M embedding model [33], with the December 2018 Wikipedia dump [34] indexed as the retrieval corpus for all experiments. The instruction interface is essential to Critic-R-Zero: it is what allows the critic’s refined instruction to alter the retrieval behavior without re-indexing. We evaluate three retrieval depths, top \(k \in \{1, 3, 5\}\), to measure how the critic’s benefit varies as more raw recall is given to the reasoner.

To collect intra-trajectory hard negatives, we run Critic-R-Zero with a Search-R1 (14B) reasoner, a Qwen2.5-72B critic, the frozen Stella-400M backbone, on the train splits of HotpotQA and Musique. The resulting dataset consists of roughly 11K natural contrastive pairs (search calls that underwent at least one refinement, yielding both positives and intra-trajectory hard negatives) and 67K positive-only samples (search calls satisfied on the first attempt).

Critic-Embed is initialized from Stella-400M [33] and fine-tuned with an InfoNCE objective using intra-trajectory hard negatives combined with in-batch negatives, with natural contrastive pairs oversampled relative to positive-only samples. Full training hyperparameters are reported in Appendix 9.

For the full Critic-R experiments, the Stella-400M backbone is replaced by Critic-Embed, our fine-tuned retriever. Experiments were run on NVIDIA A100 (80GB) GPUs. We set the maximum number of refinements to \(K = 2\), as additional iterations do not yield more improvements.

4.2.0.1 Baselines.

We compare against two baseline families, depending on the question each table targets:

• No-critic ablation. The Critic is removed and Search-R1 is run with default setting. This isolates the contribution of the critic’s verdict and refinement (used for RQ1 and RQ3).

• Retriever baselines (Table 2). The same Search-R1 reasoner makes a single top \(k\) retrieval call against the off-the-shelf Stella-400M backbone and the Agentic-R [5] retriever, with no critic loop. This isolates the contribution of the retriever itself and lets us compare Critic-Embed against both an untrained dense backbone and a retriever that is co-trained end-to-end with the search agent.

4pt

4.3 Results↩︎

4.3.0.1 RQ1: Can inference-time refinement close the retrieval gap on a frozen retriever?

We hold reasoner family (Search-R1), retriever (frozen Stella-400M, top \(k{=}1\)), and refinement budget (\(K{=}2\)) fixed, and vary (i) the reasoner scale (3B / 7B / 14B) and (ii) the critic scale (14B / 32B / 72B). Results reported in Table 1.

(1) Any critic beats no critic. For every (reasoner, dataset, metric) cell, the smallest critic (14B) already strictly improves over the no-critic ablation. The lift is large even on the hardest datasets (Bamboogle, Musique), confirming that the gains are driven by the critic’s verdict + instruction rewrite rather than by the extra forward passes alone.

(2) Inference-time scaling is not strictly monotonic: a larger critic does not guarantee better generation. Rather than a linear improvement, scaling the critic from 32B to 72B yields sharply diminishing returns and occasional performance degradation on complex tasks. The 72B critic reliably boosts the weaker 3B reasoner across the board, but the 32B critic proves optimal for harder datasets like Musique, and on Bamboogle when paired with the 7B reasoner. Ultimately, averaged across the suite, the 32B critic edges out the 72B for the 7B reasoner (0.3293 / 0.4176 vs. 0.3192 / 0.4108 EM/F1), showing that beyond 32B parameters, injecting more evaluator compute may not overcome the limitations of the reasoner and retriever.

4pt

4.3.0.2 RQ2: Do Critic-R-Zero trajectories transfer into a better retriever?

We evaluate Critic-Embed as a drop-in replacement for the retriever, with no loop active. This isolates the supervision signal that the trained retriever absorbs from Critic-R-Zero trajectories. Table 2 reports three retrievers with identical agent and conditions: the off-the-shelf Stella-400M backbone, the Agentic-R retriever baseline, and our Critic-Embed.

Critic-Embed is the best-performing retriever in every setting. The largest absolute gains come at top \(k=1\), where retrieval errors are most costly: on Bamboogle, EM/F1 climbs from 0.3520 / 0.4963 (Stella-400M) and 0.4240 / 0.5260 (Agentic-R) to 0.4480 / 0.5872 with Critic-Embed; the multi-hop average rises from 0.3472 / 0.4470 to 0.3794 / 0.4806. As \(k\) increases the absolute gap narrows because of the higher recall, yet Critic-Embed retains the lead at \(k=3\) (average 0.4128 / 0.5144 vs. / 0.4990 for Stella-400M and 0.4036 / 0.4972 for Agentic-R) and \(k=5\) (average 0.4269 / 0.5272 vs. / 0.5119 and 0.4105 / 0.5104). Figure 3 visualizes the multi-hop average for the three retrievers across \(k\). The result establishes that the trajectories produced by Critic-R-Zero contain genuine, transferable retrieval supervision: even before any inference-time criticism is layered on top, training retriever on those trajectories outperforms a retriever that was co-trained end-to-end with an agent on the same task.

4.3.0.3 RQ3: Does combining the inference-time loop and the trained retriever yield further gains?

Having established that the inference-time query refinement loop and the trained retriever each close part of the retrieval gap on their own, we now ask whether combining them yields further gains. Table 3 reports four configurations on the same Search-R1 (14B) reasoner at top \(k\!=\!1\): the static Search-R1 baseline (Stella-400M backbone, no critic loop); Critic-Embed as a static retriever (no loop); Critic-R-Zero (the loop running on top of the frozen Stella-400M); and the full Critic-R system (the loop running on top of Critic-Embed).

Three observations emerge. (1) Critic-Embed alone, with no inference-time loop, already lifts the multi-hop average over the static Stella-400M baseline from 0.3472 / 0.4470 to 0.3794 / 0.4806 EM/F1, without any test-time refinement. (2) the Critic-R-Zero loop on the frozen backbone reaches 0.3903 / 0.4855, showing that inference-time refinement and retriever fine-tuning each close part of the same overall gap, with the loop modestly ahead on this configuration. (3) combining them, yields the best overall configuration: average 0.3957 EM / 0.4959 F1, exceeding both Critic-Embed alone and Critic-R-Zero alone. The per-dataset picture is mixed: Critic-R wins decisively on Bamboogle (0.4800 / 0.6200 vs. / 0.5627 for Critic-R-Zero) and on 2Wiki, while Critic-R-Zero edges ahead on HotpotQA and Musique. The two configurations are therefore not redundant: each repairs a different slice of retrieval failures, and the loop and the trained retriever are best read as complementary contributions rather than alternatives.

4.3.0.4 RQ4: Is the agent’s introspective feedback a key source of the supervisory signal?

A central design choice of Critic-R-Zero is that the critic does not judge retrievals from the query and documents alone: it is conditioned on the reasoning agent’s own introspective trace \(T_{i+1}\) over the retrieved passages. We claim that this conditioning is essential. We test that directly by re-collecting training trajectories with a modified Critic-R-Zero in which the speculative-feedback step is removed and the critic receives only the global question \(Q\), the generated query \(q_i\), and the retrieved documents \(D_i\). We then fine-tune a separate retriever, denoted “w/o Introspective Feedback (\(T_{i+1}\))”, on these alternative trajectories using the same recipe as Critic-Embed, and evaluate it under the same no-loop static-retriever protocol used in Table 2.

Table 4 reports the results. At every retrieval depth, removing the introspective feedback \(T_{i+1}\) strictly degrades the resulting retriever. The drop in multi-hop average is substantial \(-0.0180\) EM and \(-0.0285\) F1 at \(k\!=\!1\), \(-0.0225\) EM and \(-0.0257\) F1 at \(k\!=\!3\), and \(-0.0298\) EM and \(-0.0282\) F1 at \(k\!=\!5\) — and consistent across HotpotQA, 2Wiki, Musique, and (with a small exception on Bamboogle EM at \(k\!=\!1\)) Bamboogle. This indicates that the agent’s introspection contains signals of dissatisfaction of the agent about the retrieved documents and is not just a marginal input to the critic but a primary source of the supervisory signal that Critic-Embed inherits: without it, the critic’s verdicts are noisier, the trajectories are weaker, and the distilled retriever inherits the deficit. So, inference-time scaling alone doesn’t make agentic search better. The result supports our reading that the critic specializes over the reasoner’s introspection rather than independently of it.

5 Conclusion↩︎

In this work, we demonstrate that retriever remains a critical bottleneck in agentic search. We introduced Critic-R, a framework where a dedicated critic evaluates retrieved evidence against the reasoning agent’s introspective trace, with two complementary mechanisms: Critic-R-Zero, an inference-time procedure that iteratively refines queries and retrieval instructions, and Critic-Embed, a retriever fine-tuned on the contrastive trajectories produced by this procedure without manual relevance annotation. Evaluated across several challenging multi-hop QA benchmarks, the combined Critic-R system achieved substantial improvements in downstream task accuracy, proving that explicitly modeling and optimizing retrieval quality from within the agentic loop is a powerful path toward more robust agentic search.

Limitations↩︎

The success of the critic relies heavily on the reasoning agent’s capacity to give feedback about the retrieved documents or identify missing information. While state-of-the-art RL-tuned reasoning models (like Search-R1) naturally possess this ability, weaker or smaller language models may struggle to produce accurate introspective traces, thereby degrading the critic’s verification signal. Furthermore, our experiments primarily focus on multi-hop and general knowledge-intensive question answering using a static Wikipedia corpus. The behavior and efficacy of the Critic-R has not yet been evaluated in highly dynamic environments, such as real-time web search or private enterprise document systems, where corpus noise and distribution shifts are drastically more pronounced.

Acknowledgments↩︎

This work was supported in part by the Center for Intelligent Information Retrieval, in part by NSF grant #2402873, in part by the Office of Naval Research contract #N000142412612, and with support from Google.org. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

6 General-Domain QA Results↩︎

Table 5 reports the inference-time scaling results for Critic-R-Zero on the three general-domain QA benchmarks (NQ, TriviaQA, PopQA), complementing the multi-hop results in Table 1. The same trends observed on the multi-hop suite hold here: any critic reliably improves over the no-critic ablation across all reasoner scales, and the largest critic (Qwen2.5-72B) typically yields the strongest average performance.

4pt

7 Implementation Details↩︎

7.0.0.1 Reasoning Agent (\(\mathcal{M}_R\)):

The reasoning agent is an instruction-tuned LLM operating under the ReAct paradigm [17]. At each step, the agent first emits a reasoning trace enclosed in <think>…</think> tags, followed by an action drawn from two types: a search action <search>\(q\)</search>, which issues a sub-query \(q\) to the retriever when external evidence is required, or a final answer action <answer>\(\hat{y}\)</answer>, which terminates the trajectory. Retrieved documents returned by the retriever are injected back into the agent’s context within <information>…</information> tags, and the agent is explicitly instructed to use its subsequent thinking trace to verbalize which aspects of the retrieved evidence are missing or misaligned with its current sub-goal. This introspective feedback is what the critic subsequently exploits to detect retrieval failures. The full system prompt is provided in Figure 4 in Appendix 10.1.

7.0.0.2 Critic Model (\(\mathcal{M}_C\)):

The critic is a separate LLM, and it operates in two sequential modes that decouple the judgment of retrieval quality from the act of refining it. In the satisfaction judgment mode, the critic is prompted with the original question \(Q\), the current sub-query \(q_i^{(t)}\), the retrieved documents \(D_i^{(t)}\), and the reasoner’s introspective thinking trace \(T_{i+1}^{(t)}\), and is asked to emit a binary verdict \(\sigma_i^{(t)} \in \{\texttt{yes}, \texttt{no}\}\) within a <satisfactory> tag together with a concise diagnostic reason \(r_i^{(t)}\) within a <reason> tag that states precisely what evidence, if any, is missing. In the query refinement mode, the critic is invoked only when the previous verdict is negative; it is then prompted with the failed sub-query \(q_i^{(t)}\), the failed instruction \(I_i^{(t)}\), and the diagnostic reason \(r_i^{(t)}\), and is asked to produce a refined retrieval instruction inside an <instruction> tag and a refined sub-query inside a <query> tag for the next retrieval attempt. Splitting the critic into these two modes prevents premature commitment to refinements when retrieval is in fact adequate, and allows the refinement step to focus entirely on diagnosing and bridging the specific gap identified during judgment. The full satisfaction judgment prompt \(P_{\text{J}}\) (Figure 5) and the query refinement prompt \(P_{\text{R}}\) (Figure 6) are provided in Appendix 10.2.

8 Dataset Statistics↩︎

Table 6 reports the evaluation set sizes for the seven QA datasets used in our experiments. Following [3], we evaluate on the dev split when an official test split is not publicly available, and otherwise use the test split. The first four datasets are multi-hop QA benchmarks; the last three are general-domain (predominantly single-hop) QA benchmarks.

5pt

9 Critic-Embed Training Details↩︎

Critic-Embed is initialized from Stella-400M embedding model [33]⁷ and fine-tuned with InfoNCE (temperature \(\tau = 0.02\)). The effective batch size is 128 (per-device 32 with 4-step gradient accumulation), trained for 5 epochs at learning rate \(2{\times}10^{-5}\), weight decay 0.01, linear warmup over the first 10% of steps, and gradient clipping at 1.0. Mixed precision (FP16) is used throughout. Natural contrastive pairs are oversampled by a factor of 4 relative to positive-only samples, and up to 3 intra-trajectory hard negatives are retained per query. Hard negatives are combined with in-batch negatives.

10 Prompts↩︎

Throughout this section, we use {slot} to denote runtime-substituted variables and <tag> / </tag> to denote the structured output markers parsed from the model’s response.

10.1 Reasoning Agent Prompt↩︎

The reasoning agent \(\mathcal{M}_R\) is driven by a single user-turn prompt that establishes the ReAct interaction protocol. The full template is shown in Figure 4.

10.2 Critic Model Prompts↩︎

The critic \(\mathcal{M}_C\) is invoked with two distinct prompts corresponding to its two modes: the satisfaction judgment prompt \(P_{\text{J}}\) (Figure 5) and the query refinement prompt \(P_{\text{R}}\) (Figure 6).

11 Use of AI Assistants↩︎

We use Claude⁸ to improve the presentation of the paper.

None

None

None

References↩︎

---

1. Available at: https://github.com/zarif98sjs/Critic-R↩︎

2. The default instruction is: “Given a query, retrieve relevant passages that answer the query”.↩︎

3. https://hf.co/collections/PeterJinGo/search-r1-v03↩︎

4. https://hf.co/Qwen/Qwen2.5-14B-Instruct↩︎

5. https://hf.co/Qwen/Qwen2.5-32B-Instruct↩︎

6. https://hf.co/Qwen/Qwen2.5-72B-Instruct↩︎

7. https://hf.co/NovaSearch/stella_en_400M_v5↩︎

8. https://claude.ai/↩︎