1 Introduction↩︎

In recent years, search agents based on Large Language Models (LLMs) have made significant progress in the field of information retrieval [1]–[3]. The search paradigm is shifting from traditional static corpus retrieval to autonomous Web Search Agents capable of handling complex goals [4]–[6]. These agents can simulate human navigation and browsing behaviors, achieving multi-step reasoning and information integration in dynamic and complex internet environments [7], [8].

However, current paradigms often presuppose that the user’s initial query is complete and explicit. This assumption deviates significantly from real-world information-seeking behavior, where users often provide vague or fragmented queries due to blurred memories or cognitive load limitations [9], [10]. In deep search scenarios, the impact of this discrepancy is further amplified. Unlike traditional single-hop retrieval [11], deep search involves complex multi-step reasoning chains [12], meaning any subtle ambiguity in the initial query can lead to cascading errors in subsequent navigation and information integration, wasting expensive computational resources on the wrong path. As illustrated in Fig. 1, failing to proactively clarify ambiguous checkpoints further propagates errors throughout the remaining search process. Consequently, introducing interactive clarification mechanisms to resolve ambiguity has become particularly important. Meanwhile, search tasks naturally provide strong factual grounding, allowing both interaction quality and retrieval correctness to be objectively verified through external evidence. This property offers a reliable signal for evaluating an agent’s ability to identify and resolve ambiguity in complex interactive settings.

While the academic community has recognized the importance of query ambiguity and interaction, existing benchmarks still struggle to evaluate the disambiguation capabilities of search agents. Mainstream retrieval benchmarks (e.g., GAIA [1], BrowseComp [2], AgentBench [13]) mostly assume explicit queries and focus on multi-hop reasoning while neglecting proactive interaction. Ambiguity-focused datasets (e.g., AmbigQA [14], DEEPAMBIGQA [15]) primarily consist of static scenarios and lack dynamic interaction simulation, whereas interaction-based benchmarks (e.g., IN3 [16], UserBench [17]) are often confined to closed sandbox environments, falling short in the depth and breadth of Web-scale open-domain knowledge. The recent INTERACTCOMP [18] has begun to address search interaction, yet it remains limited in terms of task authenticity, the amplification effects of ambiguity within long-chain reasoning, and the naturalness of interaction modalities.

To bridge these gaps, we introduce DiscoBench (Deep Interactive Search with ClarificatiOn Benchmark)¹, a benchmark for evaluating whether search agents can proactively clarify and resolve ambiguity during multi-step search. Unlike prior benchmarks that mainly focus on static query understanding, DiscoBench models ambiguity as a dynamic phenomenon arising during multi-step search trajectories. At each ambiguous checkpoint, agents must proactively identify underspecified information and interact with the user to obtain discriminative clues, rather than relying on direct guessing or closed-form option selection strategies.

We conduct experiments on DiscoBench across a set of representative LLMs. The results show that current search agents still struggle to determine when clarification is needed: even stronger models often fail to recognize ambiguity during the search process or ask effective clarification questions. This suggests that deep interactive search requires not only stronger retrieval and reasoning abilities, but also better ambiguity awareness and proactive clarification strategies. Our main contributions are as follows:

• We construct DiscoBench, a benchmark that models ambiguity as a dynamic phenomenon propagating along multi-step reasoning chains rather than a static property of individual queries, covering 211 samples with 463 ambiguity instances across 11 real-world domains and four ambiguity types.

• We propose an ambiguity-aware evaluation framework for multi-turn interactive deep search, together with a user simulator that progressively reveals discriminative clues, enabling unified evaluation of ambiguity detection, clarification effectiveness, and interaction cost.

• Through extensive experiments, we reveal that ambiguity detection and effective clarification are distinct capabilities. We further identify a dominant failure mode in which models repeatedly continue searching instead of asking for clarification, leading to lower success rates than direct guessing. This finding highlights the need for mechanisms that explicitly bridge retrieval uncertainty and user interaction.

2 Related Work↩︎

2.1 Web Search Benchmark↩︎

Efforts to benchmark search agents often bifurcate into two dimensions. One branch focuses on reasoning depth, challenging agents to navigate complex web hierarchies for multi-hop tasks, such as GAIA [1] and the BrowseComp series [2], [19], [20]. In parallel, other benchmarks explore information width, necessitating the synthesis of vast horizontal data, as seen in PaSa [21], SPAR [22], and WideSearch [3]. Complementary agent benchmarks like WebArena [8], VisualWebArena [23], Mind2Web [24], and WebShop [25] further evaluate web navigation capabilities in realistic environments. Recent work, such as DeepWideSearch [26], has further begun to encompass both dimensions. Constrained by the assumption of complete queries, these benchmarks leave query ambiguity unexplored, which remains an essential aspect of autonomous search.

2.2 Ambiguity Benchmark↩︎

Research on query ambiguity has evolved into several taxonomies. Semantic and structural ambiguity benchmarks represented by AmbiEnt [27] investigate logical divergences arising from linguistic properties. In the realm of multi-answer factual ambiguity, datasets such as AmbigQA [14] and ASQA [28] focus on mapping single queries to multiple concurrent valid facts. Furthermore, conditional and contextual ambiguity benchmarks including TempAmbigQA [29], CondAmbigQA [30] and SituatedQA [31] address scenarios where answers depend on latent temporal or geographical backgrounds that remain unstated in the initial query. While valuable, these benchmarks rely on a static evaluation paradigm that prioritizes answer identification over the dynamic, interactive process required for agent-user collaboration to resolve uncertainty.

2.3 Interactive Clarification Benchmark↩︎

Interactive benchmarks evaluate agent performance in multi-turn collaborative environments. For instance, ColBench [32] and UserBench [17] target code generation and travel planning, respectively, while IN3 [16] and GAIA2 [33] investigate implicit intent understanding and conflicting requests in local environments. In conversational QA, Abg-CoQA [34] requires agents to clarify coreference or semantic vagueness. Although IDRbench [35] recently introduced interactive clarification into deep research, it is not designed for open-domain web search scenarios. Most closely related, InteractComp [18] pioneers the evaluation of interactive disambiguation for search agents. However, it still contains scenarios that rely more on internal knowledge than external retrieval and mainly focuses on initial entity ambiguity rather than cascading errors in deep reasoning. Moreover, its binary feedback setting cannot fully capture the descriptive nature of real user interactions.

3 Task Formulation↩︎

We formulate multi-turn interactive retrieval as a sequential question-answering task in which an agent resolves a complex question \(q\) through a series of structured checkpoints( \(CP\)), with the ability to interact with a user when ambiguity is encountered. As illustrated in Fig. 2, the agent must determine whether the current retrieval state is ambiguous and decide whether to continue retrieval or request clarification from the user.

3.0.0.1 Question and Checkpoints.

Each question \(q\) is decomposed into an ordered sequence of \(n\) checkpoints \(\{CP_1, CP_2, \ldots, CP_n\}\), each representing an intermediate retrieval sub-goal. A checkpoint \(CP_i\) is assigned one of two types:

• Unambi: an unambiguous checkpoint where the agent can answer directly via retrieval.

• Ambi: an ambiguous checkpoint containing one of four injected ambiguity types, which causes retrieval to return multiple candidates or no valid result.

3.0.0.2 Agent Actions and User Interaction.

At each checkpoint \(CP_i\), the agent may execute one of three actions: \[a_i \in \{\mathrm{\small Search},\;\mathrm{\small Ask},\;\mathrm{\small Answer}\}\] For unambiguous checkpoints, the agent directly issues \(\mathrm{\small Search}(q_i)\) and proceeds with \(\mathrm{\small Answer}(r_i)\). For an ambiguous checkpoint \(CP_k\), the agent should invoke \(\mathrm{\small Ask}(\cdot)\) to request supplementary information, upon which the user releases a pre-defined clue \(c\). The agent then refines its search and resolves the ambiguity before issuing \(\mathrm{\small Answer}(r^*)\).

3.0.0.3 Evaluation.

We evaluate the agent from four aspects: task utility, ambiguity detection, interaction strategy, and cost efficiency.

4 Methodology of Dataset Construction↩︎

We construct DiscoBench, an interactive ambiguous question answering (QA) benchmark designed to evaluate whether LLMs can identify ambiguity, proactively request clarification, and recover correct reasoning trajectories in multi-turn open-domain search tasks. As illustrated in Fig. 3, the construction pipeline consists of two phases: (1) Seed Data Preparation, which builds high-quality multi-hop reasoning chains, and (2) Ambiguous Data Construction, which injects ambiguity and generates discriminative facts for interactive disambiguation. The entire pipeline adopts a semi-automatic collaborative framework.

4.1 Seed Data Preparation↩︎

The goal of Phase 1 is to construct high-quality multi-hop seed questions that serve as the foundation for subsequent ambiguity injection.

4.1.0.1 Topic & Seed Collection.

We first manually collect seed topics from 11 diverse knowledge domains to ensure broad domain coverage and knowledge diversity. In terms of knowledge sources, we utilize encyclopedic resources (e.g., Wikipedia and Baidu Baike) together with search engine result pages from search engines (e.g., Google, Bing, and Baidu). DiscoBench is primarily constructed in Chinese to better reflect realistic ambiguity patterns and retrieval behaviors in Chinese web environments. To ensure realistic retrieval requirements, all questions are required to satisfy the following conditions: (1) the answer must be objectively verifiable; (2) the question cannot be solved purely through common sense reasoning; and (3) external retrieval is necessary for task completion.

4.1.0.2 Multi-hop QA Construction.

Inspired by existing multi-hop QA datasets [12], [36], we adopt a collaborative framework that combines LLM-based preliminary expansion with human verification and reconstruction. Specifically, the LLM first generates preliminary single-hop factual QA pairs based on manually collected seed topics, and further performs graph-structured expansion with external retrieval results to construct candidate multi-hop reasoning chains. After automatic generation, human annotators further review and reconstruct the reasoning chains. Finally, each Seed QA sample is organized into a structured multi-hop instance, serving as the foundation for subsequent ambiguity construction.

4.2 Ambiguous Data Construction↩︎

Phase 2 aims to inject ambiguity into existing multi-hop reasoning chains, transforming deterministic QA tasks into interactive reasoning tasks that require clarification.

4.2.0.1 Ambiguity Point Identification.

Given a deterministic multi-hop reasoning chain, we identify hops where ambiguity can be naturally introduced. Instead of injecting ambiguity randomly, we focus on nodes whose target entity has similar alternatives, such that relaxing the distinguishing constraint leads to multiple plausible candidates. A node is retained as a candidate ambiguity point if: (1) its target entity shares attributes with sibling entities; (2) the downstream reasoning chain remains executable under underspecification; and (3) the ambiguity can be resolved with a single user-provided clue. All candidate positions are further verified manually.

4.2.0.2 Ambiguity Construction.

After identifying ambiguity points, we inject ambiguity into the original reasoning chain by replacing strong constraints with shared attributes among multiple candidate entities. Specifically, the system retrieves candidate entities satisfying the current reasoning constraints and uses LLMs to identify shared characteristics, such as common authors, temporal ranges, or organizational relations. The original question is then rewritten using these shared features, allowing multiple candidates to satisfy the same description. Human annotators further verify the naturalness, solvability, and logical consistency of the constructed ambiguous questions.

4.2.0.3 Discriminative Facts Generation.

To support interactive disambiguation, we construct discriminative facts for each ambiguity point, which simulate supplementary clues provided by users and distinguish the target entity from distractors. Candidate facts are generated by retrieval-augmented LLMs from perspectives such as entity attributes, temporal information, relations, numerical facts, version differences, and organizational associations, and are then manually verified for factual correctness, distinguishability, and naturalness.

4.3 Data Statistics and Quality Control↩︎

Tab. 1 presents the overall statistics of DiscoBench, including domain distribution, task difficulty, and ambiguity types. Task difficulty is determined by the number of ambiguity checkpoints, where easy, medium, and hard correspond to 1, 2, and 3 ambiguity points, respectively. DiscoBench covers four ambiguity types: Entity (multiple entities satisfy the same description), Version (different temporal or version-specific states), Criteria (different evaluation standards or ranking criteria), and Factual Inaccuracy (descriptions inconsistent with objective facts).

DiscoBench construction process involved an expert annotation instructor, six undergraduate annotators, and two quality inspectors. Annotators are recruited from diverse academic backgrounds in multiple disciplines to ensure annotation diversity and broad domain coverage. During the construction process, all samples were further reviewed for factual correctness, retrieval feasibility, logical consistency, and ambiguity solvability.

1.8pt

5 Experiments↩︎

5.1 Experimental Setup↩︎

\(^{\dagger}\) GPT-5.4 failed on 37 neutral-prompt questions due to usage-policy filtering; guided results are omitted due to only 62 valid runs, so this model is excluded from subsequent analysis.

5.1.0.1 Models and Tools.

We evaluate Claude-Opus-4.7, GPT-5.4, Gemini-3.1-Pro-Preview, Doubao-Seed-2.0-Pro-High, DeepSeek-V4-Pro, Qwen-3.6-Max, MiniMax-M2.7, GLM-5.1, MiMo-v2.5-Pro, Kimi-K2.6, and Hunyuan-3.0-Preview under the same interactive retrieval framework and checkpoint-level evaluator. For models supporting configurable reasoning effort, we use the maximum available reasoning-effort setting in the main experiments. All Search calls are implemented using Tavily [37] as the backend search engine. We use Gemini-3-Flash-Medium as the simulated user model for multi-turn interaction and ambiguity clarification during evaluation.

5.1.0.2 Prompting Settings.

We consider two prompting settings. In the Neutral setting, the agent receives no explicit instruction that ambiguity may exist and must independently decide whether clarification is needed. This setting evaluates the model’s spontaneous ambiguity detection and proactive interaction ability. In the Guided setting, the prompt explicitly reminds the agent to be aware of potential ambiguity and to ask clarification questions when necessary, which provides an ambiguity-aware condition and reflects the model’s upper-bound performance when it is encouraged to interact.

5.1.0.3 Metrics.

We report metrics from four aspects. For task utility, we use end-to-end accuracy and checkpoint pass rate. For ambiguity detection, we report detection accuracy and detection F1. For interaction quality, we report the accuracy of the clarification question (CE-A) and the clarification-to-advance rate (CE-B). For cost efficiency, we report average ask turns, tool-use turns, and token consumption. Detailed definitions of all metrics and additional analysis of token consumption are provided in Appendix 8 and Appendix 13.

5.2 Main Results↩︎

5.2.0.1 Frontier models still struggle with clarification-aware deep search.

As shown in Tab. 2, current frontier models still show limited performance on DiscoBench. Under the Neutral setting, the best-performing model, Doubao-Seed-2.0-Pro, achieves only 43.1% end-to-end accuracy, while Gemini-3.1-Pro reaches 40.8%. Most other models remain below 40%, and weaker models such as MiniMax-M2.7 and Qwen3.6-Max achieve only 16.1% and 12.3%, respectively. At the same time, there is still a substantial gap between the pass rate of the checkpoint and end-to-end accuracy. For example, Claude-Opus-4.7 achieves a checkpoint pass rate of 57.0% but only 39.8% accuracy. This suggests that models may solve several intermediate retrieval steps while still failing to complete the full reasoning trajectory due to unresolved ambiguity. Therefore, deep search for clarity requires not only retrieval and reasoning ability, but also stable ambiguity recognition and interaction planning throughout the reasoning process.

5.2.0.2 Guided prompting improves performance but still reveals limited clarification ability.

Guided prompting generally improves model performance by explicitly encouraging the agent to identify ambiguity and ask clarification questions when necessary. Averaged over the 10 models with valid results under both settings, end-to-end accuracy increases from 28.6% to 33.7%, checkpoint pass rate rises from 50.1% to 57.6%, and detection F1 improves substantially from 45.3% to 64.9%. The improvement is mainly reflected in ambiguity detection rather than downstream reasoning, suggesting that Guided prompting primarily helps reduce missed ambiguity cases. However, additional interaction does not always translate into better end-to-end performance. For example, Claude-Opus-4.7 achieves a higher checkpoint pass rate under Guided prompting while slightly decreasing in final accuracy, indicating that stronger local interaction behavior may still fail to recover the complete reasoning trajectory. Overall, prompt engineering can partially activate ambiguity-aware behavior, but current models still lack robust and stable clarification ability. Additional analysis by reasoning effort is provided in Appendix 9, showing that higher reasoning effort improves performance.

5.2.0.3 Knowing when to ask and asking effectively are distinct capabilities.

Detection and clarification metrics capture different aspects of proactive interaction: recognizing when clarification is needed and asking questions that effectively resolve ambiguity. These abilities are not always aligned. Qwen3.6-Max has only 16.0% detection F1 and asks just 0.07 questions per task under the Neutral setting, but achieves 94.7% CE-A and 89.5% CE-B, indicating strong conditional question quality but weak proactive clarification. By contrast, MiniMax-M2.7 asks more often, with 0.61/1.10 asks under Neutral/Guided settings, yet its CE-B remains lower at 60.7%/66.5%. Thus, successful clarification-aware search requires both ambiguity detection and effective question asking.

5.2.0.4 More tool use does not necessarily lead to better performance.

Fig. 4 further reveals that higher retrieval intensity does not consistently translate into better task performance. Increasing search tool calls alone cannot reliably improve accuracy. For example, Claude-Opus-4.7 exhibits relatively high tool-use frequency among evaluated models, yet its accuracy still remains below Gemini-3.1-Pro and Doubao-Seed-2.0-Pro. Meanwhile, several models also perform frequent retrieval actions while still achieving poor end-to-end performance. These observations suggest that DiscoBench does not reward excessive or inefficient retrieval behavior. Successful clarification-aware deep search depends not on searching more, but on whether models can strategically allocate retrieval actions, identify ambiguity at the correct checkpoints, and effectively utilize retrieved evidence and user-provided clues to recover the reasoning trajectory.

5.3 Performance by Ambiguity Types↩︎

Fig. 5 shows that models exhibit clear differences in detection performance across ambiguity types. Stronger models, such as Gemini-3.1-Pro and Doubao-Seed-2.0-Pro, generally achieve more balanced performance, while mid-performing models, such as DeepSeek-V4-Pro and Claude-Opus-4.7, show a more type-dependent pattern. In particular, Factual Inaccuracy is often easier to detect, likely because factual errors tend to create explicit conflicts with retrieved evidence, helping models recognize that the current question cannot be directly resolved. In contrast, Entity and Criteria ambiguities are more challenging because they usually do not create explicit factual conflicts. Instead, they require models to distinguish among multiple plausible entities or identify missing decision criteria, making models more likely to follow one plausible path prematurely. This suggests that current search agents still struggle with ambiguities that require active clarification rather than direct fact checking. Additional analysis by ambiguity complexity is provided in Appendix 10.

5.4 Behavioral Profile Analysis↩︎

5pt

DG denotes DirectGuess; SHG denotes SearchHeavyGuess; STA denotes SearchThenAsk; \(^*\) denotes \(N<30\).

To better understand the behavioral differences behind model performance, we categorize ambiguous-checkpoint trajectories into four interaction profiles based on the ordering of Search and Ask actions: DirectGuess, SearchHeavyGuess, DirectAsk, and SearchThenAsk. Detailed definitions and profile distributions are provided in Appendix 11.

5.4.0.1 Clarification substantially improves success rates.

As shown in Tab. 3, SearchThenAsk consistently achieves the highest pass rate across all evaluated models, reaching an average of 93.4%, substantially outperforming DirectGuess (56.5%) and SearchHeavyGuess (51.9%). The gap remains stable within every model, indicating that proactive clarification is critical for successful ambiguity resolution in deep search.

5.4.0.2 Search-heavy guessing reveals a major failure mode.

Notably, SearchHeavyGuess even underperforms DirectGuess despite performing more retrieval steps. Repeated retrieval often indicates that the model is already aware of multiple candidate entities. Therefore, these failures arise not from completely missing ambiguity, but from failing to escalate retrieval uncertainty into clarification. This finding further explains why increased tool use alone does not reliably improve performance.

5.5 Ablation Study↩︎

5.5.0.1 Effect of Search Tool.

Tab. [tab:ablation] shows that the external search tool is crucial for DiscoBench. After removing the search tool, all models suffer substantial accuracy drops. For example, Doubao-Seed-2.0-Pro decreases from 43.1% to 2.4%, with a drop of 40.7 points; Gemini-3.1-Pro and DeepSeek-V4-Pro also drop by 20.9 and 25.7 points, respectively. This indicates that DiscoBench cannot be solved by relying solely on parametric knowledge. Models need external retrieval to gather evidence, verify intermediate constraints, and continuously revise the search trajectory.

5.5.0.2 Effect of Ambiguity.

The comparison with unambiguous questions further shows that ambiguity is a major source of task difficulty. After removing ambiguity, all models achieve significantly higher accuracy, with improvements ranging from 26.8% to 40.2%. This suggests that current search agents are better at answering well-specified questions, but still easily fail when facing ambiguous ones.

2.2pt max width=

6 Conclusion↩︎

We introduced DiscoBench, a benchmark for evaluating clarification-aware deep search. DiscoBench models ambiguity as a dynamic issue that emerges during multi-step search and uses structured checkpoints to evaluate whether search agents can detect ambiguity, ask for clarification, and recover correct reasoning paths with user-provided clues. Experiments show that current LLM-based search agents still struggle with interactive deep search. Guided prompting improves ambiguity detection, but end-to-end performance remains limited. Meanwhile, proactive clarification is substantially more effective than repeated search or direct guessing. These findings suggest that future search agents need not only stronger retrieval and reasoning abilities, but also better ambiguity awareness and interaction planning.

Limitations↩︎

DiscoBench primarily focuses on four representative ambiguity types grounded in objective question answering scenarios. More complex forms of ambiguity, such as subjective preference ambiguity, remain underexplored and are left for future work. In addition, although DiscoBench employs an ambiguity-aware multi-turn user simulator with progressive clue disclosure, the interaction behavior is still generated by LLMs rather than real human users. As a result, the current simulator may not fully capture the diversity, noisiness, and unpredictability of real-world clarification interactions.

Ethical Considerations↩︎

DiscoBench is constructed from publicly available web resources, including encyclopedic websites and search engine result pages, and does not involve private or personally identifiable information. The benchmark is designed solely for research purposes to evaluate ambiguity handling and clarification abilities in search agents. Although the user simulator is LLM-based rather than collected from real users, we acknowledge that simulated interactions may not fully reflect the diversity of real-world human behavior.

7 Author Contributions↩︎

7.0.0.1 Benchmark Design and Methodology.

Yiling Tao, Shihan Deng, Zhihao Zhu, and Pengzhi Wei jointly contributed to the design of DiscoBench, and the overall methodology was integrated and proposed by Shihan Deng and Yiling Tao.

7.0.0.2 Data Construction and Annotation.

Shihan Deng led the construction of the multi-hop seed data, while Yiling Tao led the ambiguity data construction pipeline. Shihan Deng, Zhihao Zhu, Yiling Tao, and Pengzhi Wei were responsible for the critical quality control and final verification of the constructed data.

7.0.0.3 Evaluation Framework and User Simulator.

Yiling Tao and Shihan Deng jointly designed the framework, and Shihan Deng was primarily responsible for its code construction and implementation.

7.0.0.4 Experiments and Analysis.

Shihan Deng conducted the main experiments. The result analysis was performed by Yiling Tao, Shihan Deng, and Zhihao Zhu.

7.0.0.5 Paper Writing.

Yiling Tao led the paper writing, and Meiling Tao produced all the figures and organized the key information. Zhihao Zhu and Shihan Deng participated in revising the manuscript.

7.0.0.6 Project Supervision.

Zhichao Hu and Zhihao Zhu supervised the project and served as the corresponding authors.

8 Evaluation Metrics↩︎

This section provides detailed definitions of the evaluation metrics used in our experiments. All metrics are computed at the question level or checkpoint level and then averaged over all valid questions for each model.

8.1 End-to-End Accuracy↩︎

End-to-end accuracy evaluates whether the agent’s final answer to the full question matches the ground-truth answer. Equivalence between the two answers is determined by an LLM-based answer-equivalence judge, which abstracts away surface-form variations such as transliterations, date formats, and list ordering.

For each question \(q\), let \(\hat{a}_q\) denote the agent’s final answer and \(a_q^{\star}\) the ground-truth answer, and let \(\mathrm{equiv}(\hat{a}_q, a_q^{\star}) \in \{0, 1\}\) denote the judge’s binary verdict, where \(1\) indicates that the two answers are judged equivalent and \(0\) otherwise. The per-question correctness indicator is defined as: \[\mathrm{Acc}(q) = \mathrm{equiv}\!\left(\hat{a}_q,\, a_q^{\star}\right).\]

The model-level end-to-end accuracy is computed by averaging over all valid questions: \[\mathrm{Acc} = \frac{1}{|Q|}\sum_{q \in Q} \mathrm{Acc}(q),\] where \(Q\) denotes the set of valid evaluated questions. This question-level normalization ensures that each question contributes equally to the final score.

8.2 Checkpoint Pass Rate↩︎

Each question in DiscoBench is decomposed into a sequence of checkpoints. A checkpoint is counted as successfully advanced if the agent either answers the checkpoint correctly and proceeds to the next checkpoint, or correctly completes the final checkpoint. In our evaluator logs, this corresponds to one of two outcomes: (1) correct_answer, where the agent correctly resolves a regular checkpoint; (2) missed_ambiguity_correct, where the agent misses the ambiguity at an ambiguous checkpoint but still happens to answer it correctly.

For each question \(q\), let \(N_q\) denote the ground-truth number of checkpoints, and let \(A_q\) denote the number of checkpoints that are successfully advanced. The checkpoint pass score for question \(q\) is defined as: \[\text{CP}(q) = \frac{A_q}{N_q}.\]

The model-level checkpoint pass rate is computed by averaging over all valid questions: \[\text{CP} = \frac{1}{|\mathcal{Q}|}\sum_{q \in \mathcal{Q}} \text{CP}(q),\] where \(\mathcal{Q}\) denotes the set of valid evaluated questions. This question-level normalization ensures that each question contributes equally to the final score, regardless of how many checkpoints it contains.

8.3 Ambiguity Detection Metrics↩︎

We evaluate ambiguity detection at the checkpoint level. For each reached checkpoint, we compare the ground-truth checkpoint type with the agent’s interaction behavior. A checkpoint is labeled as Ambi if it contains an injected ambiguity, and Non-Ambi otherwise. We define the four detection outcomes as follows:

• True Positive (TP): the checkpoint is Ambi, and the agent correctly asks a clarification question targeting the ambiguity.

• False Negative (FN): the checkpoint is Ambi, but the agent fails to ask or does not correctly target the ambiguity.

• False Positive (FP): the checkpoint is Non-Ambi, but the agent unnecessarily asks for clarification.

• True Negative (TN): the checkpoint is Non-Ambi, and the agent proceeds without asking for clarification.

Accordingly, \(TP+FN\) corresponds to all reached ambiguous checkpoints, while \(FP+TN\) corresponds to all reached non-ambiguous checkpoints. The total number of evaluated detection decisions is \(TP+TN+FP+FN\).

8.3.0.1 Detection Accuracy.

Detection accuracy measures the overall correctness of ambiguity detection decisions: \[\text{Detection Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.\]

8.3.0.2 Detection F1 Score.

We further report detection F1 to better account for the imbalance between ambiguous and non-ambiguous checkpoints. Precision and recall are defined as: \[P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}.\]

The detection F1 score is computed as: \[\text{Detection F1} = \frac{2 \cdot P \cdot R}{P + R}.\]

Here, recall measures the proportion of ambiguous checkpoints that are correctly detected, while precision measures how often the agent’s clarification decisions are correct. F1 provides a more robust measure when ambiguous and non-ambiguous checkpoints are unevenly distributed.

8.4 Clarification Effectiveness↩︎

Clarification effectiveness evaluates the quality and usefulness of the agent’s clarification behavior. Both metrics share the same denominator: the number of checkpoints where the agent actively invokes Ask, regardless of whether the question is correct.

Let \(\mathcal{C}\) denote the set of evaluated checkpoints, and let \(\mathcal{C}_{\texttt{asked}}\) denote the set of checkpoints where the agent invokes Ask: \[\mathcal{C}_{\texttt{asked}} = \{c \in \mathcal{C}: \texttt{asked}(c)\}.\]

We further define: \[\mathcal{C}_{\texttt{right}} = \{c \in \mathcal{C}_{\texttt{asked}} : \texttt{asked\_right}(c)\},\] \[\mathcal{C}_{\texttt{adv}} = \{c \in \mathcal{C}_{\texttt{asked}} : \texttt{cp\_advanced}(c)\}.\]

8.4.0.1 CE-A: Clarification Question Accuracy.

CE-A measures whether the agent asks the right clarification question when it decides to interact: \[\text{CE-A} = \frac{ \left|\mathcal{C}_{\texttt{right}}\right| }{ \left|\mathcal{C}_{\texttt{asked}}\right| }.\]

8.4.0.2 CE-B: Clarification-to-Advance Rate.

CE-B measures whether a correct clarification eventually helps the agent advance the current checkpoint: \[\text{CE-B} = \frac{ \left|\mathcal{C}_{\texttt{right}} \cap \mathcal{C}_{\texttt{adv}}\right| }{ \left|\mathcal{C}_{\texttt{asked}}\right| }.\]

CE-A reflects whether the agent asks in the correct direction, while CE-B further evaluates whether the agent can use the returned clue to successfully advance the checkpoint.

9 Additional Analysis by Reasoning Effort↩︎

Fig. 6 shows that increasing reasoning effort leads to consistent overall improvements. Taking Doubao-Seed-2.0-Pro as an example, when the reasoning effort is increased from medium to high, the average score rises from 45.7% to 54.0%, with an overall gain of 8.3 points. The improvements are particularly pronounced on ambiguity-related metrics: Det. F1 increases by 9.0 points, and Ambi. Rec. improves from 37.2% to 47.3%, yielding a 10.1-point gain, which is larger than the improvement on CP. This suggests that higher reasoning effort mainly helps models identify ambiguous search states, compare multiple candidate entities, and incorporate user clues into subsequent search refinement.

This result is consistent with the characteristics of DiscoBench. Clarification-aware deep search requires models not only to retrieve the final answer, but also to continuously judge whether the evidence is sufficient across multiple ambiguity checkpoints, while maintaining and revising the search trajectory. However, even under the high-effort setting, the accuracy remains below 45% and Ambi. Rec. remains below 50%, indicating that simply increasing reasoning effort is still insufficient for robust clarification-aware behavior. Models still need stronger mechanisms for ambiguity localization, evidence verification, and deciding when to ask rather than directly guess.

10 Additional Analysis by Ambiguity Complexity↩︎

Fig. 7 presents model performance across different levels of complexity of ambiguity under neutral prompting. Across nearly all evaluated models, accuracy consistently decreases from Easy to Hard, showing that ambiguity complexity introduces substantial additional difficulty beyond ordinary retrieval and reasoning. Although stronger models such as Doubao-Seed-2.0-Pro (High), Gemini-3.1-Pro, Claude-Opus-4.7, and DeepSeek-V4-Pro achieve relatively strong performance on Easy examples, their accuracy still drops markedly on Hard examples. This suggests that increasing ambiguity complexity challenges not only evidence retrieval, but also the model’s ability to recognize underspecified states and proactively request clarification during multi-step search.

The performance gap between Easy and Hard settings further indicates that stronger reasoning ability alone is insufficient for robust clarification-aware search. As ambiguity becomes more subtle and accumulates across multiple checkpoints, models are increasingly prone to following plausible but incorrect search trajectories without initiating clarification. Lower-performing models exhibit the same downward trend from a lower baseline, indicating simultaneous weaknesses in both basic task completion and ambiguity resolution.

11 Profile Classification Details↩︎

11.0.0.1 Behavioral profiles.

We classify ambiguous-checkpoint trajectories into four profiles:

• DirectGuess: no Ask with search count \(\leq K\).

• SearchHeavyGuess: no Ask with search count \(>K\).

• DirectAsk: ask before any retrieval.

• SearchThenAsk: retrieve before clarification.

The threshold \(K\!=\!3\) is determined data-driven as the median search count among successful no-ask trajectories.

11.0.0.2 DirectAsk rarity.

DirectAsk is extremely rare, accounting for only \(0\)–\(7\) trajectories per model. For 7 of 9 models, the count satisfies \(N\!\leq\!1\), indicating that current models almost never initiate clarification before retrieval.

11.0.0.3 Common Subset Robustness.

The common subset contains 146 ambi-CPs reached by all 9 models in Tab. 3. GPT-5.4 is excluded due to prompt filtering, while Qwen is excluded because of insufficient reach. Model rankings on the common subset remain highly consistent with the full dataset (Spearman \(\rho\!=\!0.95\)), indicating that the observed behavioral patterns are robust to reach-rate differences.

5pt

12 Evaluated Models and API Configurations↩︎

3pt max width=

Tab. [tab:model95details] lists the API configuration of every model used in our main experiments and ablation studies, restricted to the information needed to reproduce a call: the model name used in the paper, the provider, the API identifier used at invocation time, and the reasoning- or thinking-mode setting. To control for the confound of reasoning budget, models exposing a configurable effort level are run at the highest available setting in the main experiments; Doubao-Seed-2.0-Pro-Medium is additionally included only for the reasoning-effort analysis in Appendix 9. Frontier proprietary models (GPT-5.4, Gemini-3.1-Pro-Preview, and Claude-Opus-4.7) are restricted to a concurrency of \(5\) by provider-side rate limits, whereas the remaining agent endpoints permit a concurrency of \(300\); this asymmetry influences wall-clock evaluation cost but not per-question correctness. All Search calls are executed through Tavily regardless of the agent backbone. A single auxiliary model, Gemini-3-Flash-Medium running with thinking level set to medium, serves both as the simulated user that releases discriminative clues during clarification turns and as the checkpoint-level judge that scores each step; its configuration is listed in the bottom block of Tab. [tab:model95details] and uses a concurrency of \(100\).

13 Token Consumption↩︎

To provide a supplementary reference for inference cost, we report the token consumption of the evaluated models in the main experiments and ablation studies. Tab. [tab:token95consumption] summarizes the total input and output tokens under the Neutral and Guided prompting settings. The Guided entry of GPT-5.4 is left blank because its Guided runs were excluded from the main analysis due to insufficient valid results. Doubao-Seed-2.0-Pro-Medium is included only for the reasoning-effort analysis under the Neutral setting, and therefore does not have a corresponding Guided entry.

Tab. [tab:ablation95token95consumption] further reports token consumption for the two ablation settings used in the main text: removing the search tool and evaluating on unambiguous questions. These ablation runs are conducted under neutral prompting and are reported separately from the main Neutral/Guided comparison.

3pt max width=

3pt max width=

14 Reproducibility under a Black-Box Search Backend↩︎

All Search calls in DiscoBench are routed through Tavily [37], a hosted web-search API whose index, ranking model, and freshness policies are not publicly disclosed. From the agent’s point of view the backend therefore behaves as a black box: the same query issued on different days can return different snippet sets, different rankings, and even different source domains, both because the web itself is non-stationary (pages appear, are edited, or are de-indexed) and because Tavily’s own retrieval and re-ranking stack can be updated without notice. Strict bit-exact reproducibility of an end-to-end trajectory is therefore not achievable.

14.0.0.1 Solvability of individual instances.

This stochasticity does not, however, undermine the solvability of individual questions, because the gold answer in DiscoBench is by construction time-invariant. Every question is built on stable, verifiable factual knowledge, so the correct answer does not drift as the web changes. Tavily’s black-box behavior varies only the retrieval surface: which snippet from which source is returned, and in what order. Since the underlying evidence lives in well-established, widely indexed public sources, any reasonably comprehensive web index can be expected to surface at least one supporting snippet for a reasonable query. DiscoBench therefore remains solvable in principle on every run.

14.0.0.2 Variance across runs.

The black-box effect manifests not as a change in answerability but as variance in trajectory shape and retrieval efficiency: the specific snippets surfaced, their ranking, and therefore which queries are sufficient and how many Search calls an agent needs before the relevant evidence appears in its context window. Two runs of the same agent on the same question may consequently take different paths and consume different numbers of tool calls even when both ultimately succeed.

14.0.0.3 Implications for interpretation and replication.

A model’s reported score on DiscoBench should be read as an expectation over Tavily snapshots rather than as a per-run deterministic quantity, and an individual incorrect trajectory should be inspected for its underlying cause—retrieval ordering on a given day versus a genuine ambiguity-handling failure—before being attributed to model capability. To make replications maximally comparable, we recommend that future users of DiscoBench (i) run all compared agents within a short, contiguous evaluation window so they observe near-identical Tavily snapshots, and (ii) where feasible, cache and release the raw Tavily responses alongside model outputs, which converts a hard reproducibility problem (re-deriving an identical live web view) into a tractable one (replaying a fixed snippet log).

15 Examples↩︎

To illustrate the design of DiscoBench, we walk through three cases from the benchmark, each spotlighting a different ambiguity type. Case 1 (Tab. 5) shows cascading Entity ambiguity along a multi-hop chain; Case 2 (Tab. 6) pairs Factual Inaccuracy (CP\(_1\)) with Version ambiguity (CP\(_3\)) in a single trajectory, forcing the agent to switch detection strategies mid-chain; Case 3 (Tab. 7) presents Criteria ambiguity, where identical surface wording maps to two distinct ranking standards. Each case lists the original question, the ambiguity-injected rewrite shown to the agent, and the checkpoint trajectory with ground-truth types, targets, ambiguity logic, and the discriminative clue the user simulator releases upon a well-targeted Ask. In every table, English rows (gray tint) precede the original Chinese (cream tint). Colored text tracks each substitution thread: the same color appears on the original constraint, its weakened rewrite, and the clue that later restores it.

15.0.0.1 Case 1: cascading Entity ambiguity across a multi-hop chain.

Tab. 5 presents a four-checkpoint question from the Video Games domain. The original question fully specifies two distinguishing awards – the GWB Indie Game Awards Bronze Prize and the 2012 Top Ten Most Anticipated Web Games – which uniquely pin down the two target entities My Time at Sandrock (a Pathea Games title) and Jingtian Zhanshen Online (a ZQ Game title). The ambiguity-injection step replaces each of these strong constraints with the generic phrase “won an award”, producing two cascading Ambi checkpoints (CP\(_1\) and CP\(_3\)) of type Entity: at each, the rewritten constraint matches multiple notable award-winning products of the queried company, so an unaided Search returns more than one candidate. The agent should detect this and invoke Ask; the user simulator then releases the original award name as a discriminative clue, allowing a refined Search to lock onto the target. This case also exposes the benchmark’s central failure mode, silent cascading: an incorrect resolution at CP\(_1\) (e.g., Portia instead of Sandrock) still routes to a syntactically valid publisher at CP\(_2\), but every downstream checkpoint then targets the wrong entity, with no local indication of the upstream error.

15.0.0.2 Case 2: Factual Inaccuracy followed by Version ambiguity.

Tab. 6 presents a four-checkpoint question from the Sports domain. Two substitution threads run through the trajectory but follow different injection patterns. Thread A (CP\(_1\)) is a Factual Inaccuracy: the original country nickname “Land of Windmills” (the Netherlands) is replaced by “Land of Hajimi”, a fabricated term that does not refer to any real country. The agent cannot resolve this checkpoint through retrieval alone – a faithful search returns no match – and must invoke Ask rather than guess. Thread B (CP\(_3\)) is a Version ambiguity: the original match identifier “at the 60th minute of a 2018 CSL match” is weakened to the looser window “in a CSL match in March–April”. Within that window Wang Chu came on as a substitute in two different matches on different dates, replacing Wang Gang in one and Cao Yongjing in the other; only the precise date disambiguates which match is meant. The two threads together stress that DiscoBench requires the agent to switch detection mode within a single trajectory rather than apply a single clarification heuristic uniformly.

15.0.0.3 Case 3: Criteria ambiguity in a long bridging chain.

Tab. 7 presents a three-checkpoint question spanning the Technology domain. Unlike Case 1 and Case 2, here a single colored phrase in the original is deleted rather than replaced: the qualifier “also nicknamed the ‘Ice City’” is removed during ambiguity injection. After deletion, the surviving constraint – “a city whose Chinese name has three characters and lies above 40\(^{\circ}\)N, listed among the top three [beer-festival] cities” – can be satisfied under two distinct enumeration criteria: the world’s top three beer festivals (yielding Munich,

, \(\sim\)​48\(^{\circ}\)N) or China’s top three beer festivals (yielding Harbin,

, \(\sim\)​45\(^{\circ}\)N). The agent must recognize that a single description fits two rankings and clarify which ranking the user intends before proceeding.

3pt

3pt

3pt

16 Annotation Details↩︎

16.0.0.1 Recruitment and compensation.

Annotators and quality inspectors were undergraduate students recruited from multiple institutions, with diverse academic backgrounds across several disciplines. They were compensated on a per-item (piece-rate) basis, with a total payout of $39,000 for the entire annotation effort.

16.0.0.2 Annotator consent.

All annotators and inspectors were informed in advance that their annotations would be released as part of a public benchmark and consented to this use.

16.0.0.3 Ethics review.

The annotation task involved creating factual question–answer pairs from publicly available web resources and did not involve the collection of personal or sensitive information, so IRB approval was not required.

17 Quality Inspection↩︎

This section provides further details on the quality control (QC) process introduced in Section 4.3. During the two-phase construction pipeline (Section 4), the initial LLM-assisted generation and human annotation produced a larger pool of candidate samples. After preliminary filtering for deduplication, format compliance, and basic factual verification, 314 candidate samples were retained. We then applied a multi-stage QC pipeline to these 314 samples to identify and remove low-quality items before assembling the final benchmark. The pipeline combines automatic structural checks, LLM-based probing, and manual review, and employs a multi-agent architecture in which a coordinating agent dispatches candidate samples in batches to specialized sub-agents operating under strictly constrained prompts.

17.0.0.1 Stage 1: Structural Validation.

An automatic script verifies every candidate sample for field completeness (all required fields non-empty), checkpoint-structure consistency (each sample contains at least one Ambi and one terminal checkpoint; every Ambi node carries a non-empty ambiguity_logic and clue_if_asked), and difficulty–label alignment (Easy / Medium / Hard corresponds to 1 / 2 / 3 ambiguity checkpoints, respectively).

17.0.0.2 Stage 2: LLM-Based Probing.

Each sample is independently tested under two complementary conditions to assess whether the task genuinely requires multi-step retrieval and multi-turn clarification. In both cases, the sub-agent receives the complete rewritten question (i.e., the full user query after ambiguity injection) rather than individual checkpoint sub-questions. The prompt templates are provided in Box [box:qc-prompts].

• Closed-book probing. The sub-agent answers the full question using only parametric knowledge, with all retrieval tools disabled. A correct answer signals potential knowledge leakage. Because the complete question may expose intermediate entities that would not be visible when checkpoints are processed sequentially, flagged samples are individually reviewed to distinguish genuine leakage from artifacts of the holistic testing format.

• Open-book probing. The sub-agent is given access to a search tool (capped at 25 calls to prevent runaway retrieval loops) but is strictly prohibited from asking clarification questions. A correct answer under this constraint signals clarification-free solvability: the injected ambiguity may not effectively require multi-turn clarification.

Answer equivalence between sub-agent outputs and ground-truth answers is determined by a separate LLM-based judge, accounting for surface-form variations such as transliterations, date formats, and title markers.

17.0.0.3 Stage 3: Ambiguity and Factual-Error Assessment.

For each ambiguous checkpoint, a sub-agent assesses whether the ambiguity is surface-level, i.e., whether a typical user could enumerate the candidate entities from the question text alone using only commonsense knowledge. Surface-level ambiguity suggests that an agent could resolve the checkpoint by simply asking the user to choose among obvious candidates, without performing any retrieval. Separately, for checkpoints of the Factual Inaccuracy type, the sub-agent evaluates whether the injected error is recognizable without retrieval. Errors that a typical user could identify through commonsense alone (e.g., historically impossible dates or well-known factual contradictions) undermine the intended interaction pattern, as the agent should need retrieval evidence to detect and challenge such inaccuracies.

17.0.0.4 Stage 4: Manual Review and Answer Verification.

All automatically flagged samples undergo manual review covering three aspects: (1) question and ambiguity design, including whether the question text uniquely constrains the expected answer, whether the injected ambiguity is realistically triggerable during retrieval, and whether sub-questions are logically coherent with the overall reasoning chain; (2) clue and retrieval quality, including whether the discriminative clue is natural and sufficient for disambiguation, and whether the target answer is retrievable through mainstream search engines; and (3) answer correctness, where we cross-check ground-truth annotations against external sources. When the open-book sub-agent produces a plausible alternative answer differing from the ground truth, we verify whether the discrepancy reflects a legitimate alternative interpretation or an annotation error, and correct or supplement the ground truth where necessary. Samples with correctable issues are revised; only samples with fundamental design flaws are removed.

17.0.0.5 Stage 5: Final Verdict.

A rule-based decision tree aggregates the signals from the preceding stages to determine whether each sample effectively requires both deep retrieval and multi-turn clarification. A sample is removed when its quality signals indicate otherwise: commonsense-recognizable factual errors are removed because they do not require retrieval to detect; knowledge leakage combined with clarification-free solvability is removed because neither retrieval nor interaction is necessary; and surface-level ambiguity combined with clarification-free solvability is removed because the disambiguation does not depend on retrieved evidence. Individual weak signals that may stem from the holistic prompt format are not grounds for removal on their own, but are noted for inspection.

17.0.0.6 Results.

Tab. [tab:qc-results] summarizes the QC outcomes. Of the 314 candidate samples, 236 (75.2%) passed quality control and 78 (24.8%) were removed. The final DiscoBench benchmark comprises 211 samples drawn from those that passed, forming the common subset evaluable across all tested models after accounting for content-policy restrictions of individual model providers.

4pt

18 Prompt Templates↩︎

18.1 Multi-Turn Responder Prompt↩︎

18.2 Neutral System Prompt↩︎

18.3 Guided System Prompt↩︎

18.4 System Prompt without Search↩︎

18.5 System Prompt without Ask↩︎

References↩︎

---

1. Our code and data will be publicly released soon.↩︎