1 Introduction↩︎

Large Language Models (LLMs) with advanced reasoning capabilities [1]–[3] have driven substantial progress across a wide range of natural language tasks. Building on these advances, LLM-based agents that equipped with planning, tool use, and multi-step reasoning capabilities [4], [5]—have achieved strong performance on complex real-world challenges, including computer operation [6], deep research [7], and information seeking [8], [9].

5o far, existing benchmarks for evaluating agents can be systematically categorized along two critical dimensions (Figure 1): search width (measured by the number of information units to be searched) and search depth (measured by average search steps for each unit), revealing four distinct categories: (1) Low width, high depth benchmarks (e.g., GAIA [8], BrowseComp [9]), which focus on intricate deep reasoning over multi-hop retrieval for searching target answers; (2) Low width, low depth benchmarks (e.g., TriviaQA, HotpotQA), which address simple fact-finding tasks; (3) High width, low depth benchmarks (e.g., WideSearch [10] and PaSa [11]), which emphasize broad information collection about specific questions; and critically, (4) High width, high depth tasks, which collect extensive information that required deep reasoning—a critical capability for real-world applications like comprehensive market analysis and business development but entirely unaddressed by current benchmarks. For instance, as shown in Figure 2, the case “identifying the Top-10 EV maker in China by MoM sales growth (Aug 2025) and its Top-3 best-selling new EV cars (price and range)” exemplifies this challenge. It requires agent to gather a large volume of candidates, i.e., EV makers, to fill the result table through wide-scale search, and verify each candidate by performing deep reasoning, a combinatorial complexity that exceeds both the scope of width-focused evaluations and the scale of depth-focused benchmarks.

To address this critical evaluation gap, we introduce DeepWideSearch, the first benchmark explicitly designed to evaluate the capability of agents in deep and wide information seeking. Since it is challenging to construct deep and wide search instances even with human annotation, we develop two methods for conversing established datasets: (1) Deep2Wide Conversion, which extends deep search benchmarks (e.g., GAIA and BrowseComp) by augmenting their information scope through human-annotated table schemas; and (2) Wide2Deep Conversion, which enhances wide search queries by replacing explicit entities with synthesized complex sub-questions that necessitate multi-hop search steps. Both approaches integrate rigorous human validation protocols to ensure data quality while maintaining the combinatorial complexity inherent in real-world information-seeking scenarios. The final benchmark comprises 220 meticulously curated questions spanning 15 diverse domains, featuring both Chinese and English queries with human-verified ground truths, with 85 instances derived from Deep2Wide and 135 from Wide2Deep construction methods.

We conduct comprehensive experiments across state-of-the-art LLMs and agent systems on DeepWideSearch. Our results demonstrate that even the most advanced agent systems achieve only 2.39% average success rate on DeepWideSearch, highlighting the substantial difficulty of this kind of information seeking task. Notably, while agent frameworks consistently improve core entity identification (e.g., +15.91 absolute percentage points in Core Entity Accuracy), they exhibit limited efficacy in wide-scale information collection, frequently underperforming their LLMs counterparts using internal knowledge. Through systematic error analysis, we identify four fundamental failure modes: (1) lack of effective reflection mechanisms when encountering problematic search trajectories; (2) overreliance on parametric internal knowledge leading to outdated or inaccurate information; (3) insufficient retrieval despite accessing relevant webpages; and (4) context overflow exceeding current agent architectue limitations. These empirical findings expose key limitations of current agent architecture for the deep and wide information-seeking task. To facilitate further research in this critical domain, we have publicly released the DeepWideSearch benchmark, including datasets and evaluation codebase.

2 Related Work↩︎

2.1 LLM-based Search Agents↩︎

The emergence of LLM-based agent systems has enabled sophisticated information-seeking capabilities, with frameworks ranging from closed-source implementations (e.g., OpenAI Deep Research) to open-source platforms (e.g., WebAgent [12] and Cognitive Kernel-Pro [13]). These systems have demonstrated proficiency in numerous application domains, including computer-use agents, deep research for complex problem investigation [14], and multi-step information retrieval through tool use [4]. Among these applications, information-seeking agents represent a critical frontier impact real-world utility. Current research in this domain primarily addresses five technical challenges: (1) agentic system architecture design [15]–[18], (2) synthetic data generation for complex scenarios [19]–[21], (3) optimization techniques for retrieval efficiency [22]–[24], (4) knowledge management for multi-hop reasoning [15], [25], and (5) evaluation methodologies for performance assessment [26], [27].

2.2 Benchmarks for LLM-based Agents↩︎

Existing evaluation frameworks for information-seeking agents primarily target two distinct capabilities: (1) Depth in multi-hop reasoning, measured by benchmarks like GAIA [8] and BrowseComp [9] for general complex reasoning, and domain-specific variants in healthcare [28] and E-commerce [29]; (2) Width in information collection, assessed by WideSearch [10] for comprehensive retrieval of atomic information, and PaSa [11] and SPAR [30] for academic literature retrieval. Crucially, no existing benchmark captures the combinatorial complexity inherent in real-world information-seeking tasks that simultaneously demand extensive exploration (width) and intricate multi-step reasoning (depth). This fundamental gap in evaluation methodology has prevented meaningful progress toward agents capable of handling the complex real-world information-seeking. To address this limitation, we propose DeepWideSearch, the first benchmark explicitly designed to evaluate the capability of agents in the deep and wide information-seeking task.

3 Task Formulation↩︎

As shown in Figure 3, DeepWideSearch establishes an evaluation framework that explicitly captures the combinatorial complexity of real-world information-seeking tasks—requiring agents to perform deep reasoning and wide-scale information collection. The evaluation metrics (Column F1, Row F1, Item F1, and Success Rate) illustrated in Figure 3 will be formally described in Section 4.4.

3.0.0.1 Input

Formally, each task in DeepWideSearch is defined as a tuple (\(Q, C\)): (1) Question \(\boldsymbol{Q}\) represents a complex natural language query for deep and wide information seeking; and (2) Columns \(\boldsymbol{C=\{c_i\}_{i=1}^N}\) define the table schema as a set of \(N\) attributes and constraints need to be collected and verified, such as EV price and MoM scales growth in Figure 3 (right).

3.0.0.2 Output

As shown in Figure 3 (medium), agents are required to generate a structured tabular response \(R\) by performing wide search for gathering numerous candidates and deep search for the verification of each candidate.

4 Methodology of Dataset Construction↩︎

Constructing DeepWideSearch instances from scratch presents significant challenges due to the substantial human effort. To address this challenge while maintaining methodological rigor, we propose two methods to converse established datasets into deep and wide search questions: (1) Deep2Wide Conversion and (2) Wide2Deep Conversion. Both methodologies are complemented by human annotation procedures to ensure the quality.

4.1 Convert Deep Search Datasets (Deep2Wide)↩︎

Existing deep search benchmarks such as GAIA [8], BrowseComp [9] and BrowseComp-zh [31] require agents to employ multi-hop web browsing and deep reasoning to identify target answers. Building upon these resources, we develop the Deep2Wide conversion methodology by expanding the scope of searched information. As illustrated in Figure 4 (Top), our approach follows a three-stage pipeline inspired by WideSearch [10]: (1) Core Entity Filtering: We sample 80 Chinese questions from BrowseComp-zh and 20 English questions from BrowseComp, filtering out instances where answers are unsuitable as core entities (e.g., dates and numerical values). For example, as shown in Figure 5, Dan Lin is the core entity of the deep search question; (2) Table Schema Definition: Human annotators design structured table schemas by defining relevant information about the core entities; (3) Comprehensive Annotation: Annotators perform exhaustive web searches to populate the tables. Each instance requires approximately 30 minutes of human annotation time, ensuring high-quality and verified data. Following a design similar to that of the WideSearch benchmark [10], we incorporated timestamps into each question to ensure that the answers remain invariant over time.

4.2 Convert Wide Search Datasets (Wide2Deep)↩︎

Given that WideSearch [10] represents the publicly available dataset providing human-annotated tabular answers for wide-scale information-seeking, we develop the Wide2Deep conversion methodology to transform these wide search queries by introducing complexity in entity identification. This approach reuse the valuable human-annotated table in WideSearch while enhancing the deep reasoning requirements. Inspired by WebWalker [12], we implement a human-in-the-loop pipeline (Figure 3, bottom) comprising five stages: (1) Entity Extraction: Advanced LLMs identify core entities in 160 English and Chinese WideSearch questions, similar to the core entity in the deep search benchmark (Figure 5); (2) Deep Sub-Question Synthesis: Following prior work [20], [21], a web search agent are implemented to recursively traverse official websites about core entities and collecting their rich entity information. Then, a complex sub-question is generated based on these rich information, adhering to two critical constraints: (a) Uniqueness: The answer to the question must be a single, well-defined entity; (b) Complexity: Direct derivation of the entity from the question must require at least one additional web search step; (3) Question Fusion: Claude-sonnet-4 fuses the deep sub-question with the original wide search query; and (4) Human Annotation: A team of seven master’s-level annotators validates and refines the synthesized questions to ensure uniqueness, complexity, and linguistic naturalness. This process requires approximately 40 minutes of human annotation per instance, maintaining the high-quality standards essential for a rigorous benchmark. The prompts of core entity extraction, deep sub-question synthesis and question fusion are placed at Appendix 11.

4.3 Data Statistics↩︎

Table 1 provides a comprehensive comparison of our DeepWideSearch benchmark against existing datasets across multiple dimensions. Our benchmark demonstrates significantly higher search complexity compared to prior work, with an average table volume of 414.10 information units, substantially exceeding deep search benchmarks like GAIA and BrowseComp. Crucially, DeepWideSearch requires 4.21 average search steps to identify core entities—nearly 4× more complex than WideSearch (1.24). The dataset spans 15 diverse domains, covering both English and Chinese queries, with 220 carefully curated instances (85 from Deep2Wide, 135 from Wide2Deep). These statistics empirically validate the deep and wide attributes of our proposed DeepWideSearch. Cases and more details about the data in Table 1 can be found in Appendix 9.

4.4 Evaluation Metrics of DeepWideSearch↩︎

As shown in Figure 3, we evaluate agent performance on DeepWideSearch along three complementary axes: Depth, Width, and Efficiency.

4.4.0.1 Depth Evaluation

The depth dimension evaluate the capability of agents to correctly identify target entities through deep reasoning over multi-hop retrieval. Following previous works [8], [9], we introduce the Column-F1 metric. As shown in Figure 3, Column-F1 is computed as the F1 score over the unique columns in the table. These unique columns correspond to the core attributes of entities (i.e., rows) that uniquely identify them. Therefore, Column-F1 can be seen as the extension of the accuracy used in established deep search benchmarks, computing the precision of a group of entities (rows in the table). Higher Column-F1 scores indicate more precise entities identification across the entire table structure. Moreover, since our proposed two methods include the core entity of questions, we also introduce the Core Entity Accuracy (CE Acc.), serving as an additional indicator of deep reasoning capability.

4.4.0.2 Width Evaluation

The width dimension measures how comprehensively and accurately the agent retrieves all associated information units for entities (rows in the table). Building upon the evaluation framework of WideSearch [10], we assess performance at three granularities: (1) Success Rate: A binary metric indicating whether the agent’s output table exactly matches the human-annotated ground truth (all rows, columns, and values identical); (2) Row-level F1: Computes precision, recall, and F1 scores at the row level (i.e., for each entity and its associated attributes), capturing whether the agent retrieves complete contextual information per entity; (3) Item-level F1: The finest-grained metric evaluating accuracy at the individual cell level, reflecting fidelity in retrieving atomic information units within the structured table.

4.4.0.3 Efficiency Evaluation

To address the substantial computational costs inherent in web-scale tool usage (including search, browsing APIs), we further evaluate system efficiency through two metrics: (1) Input/Output Token: The total tokens consumed during reasoning and tool calls; (2) Cost: Estimated cost expenditure based on standard model inference API pricing during query resolution. These efficiency metrics are critical for real-world deployment considerations, particularly given the demanding requirements for extensive multi-round search and browsing.

To account for stochasticity in LLM-based agent behavior, we conduct four independent runs per question for each baseline system. For both depth and width metrics, we report three complementary statistics: (1) Avg@4: The mean performance across all four runs; (2) Max@4: The best performance observed across the four runs; and (3) Pass@4: The proportion of questions solved successfully in at least one run (only for Success Rate). This comprehensive evaluation protocol ensures robustness against sampling variance while also highlighting the system’s peak performance potential.

5 Experiments↩︎

5.1 Experimental Setup↩︎

We evaluate three kinds of baselines on our proposed DeepWideBenchmark: (1) Closed-source LLMs (without tool calls): OpenAI o3-mini, GPT-4o, GPT-5, Claude-sonnet 4, Gemini 2.5 Pro and Qwen-Max; (2) Open-source LLMs (without tool calls): DeepSeek-V3/R1 [2], [3], KIMI-K2 [34], Qwen3 series [35]; and (3) Open-source Agent Systems: WebSailor [20], Smolagents [36] and OWL [37] are equipped with advanced GPT-5, Claude-sonnet-4 and Gemini-2.5-Pro backbone models. All agent systems utilized identical tools: (1) Google Search API; and (2) Webpage Visit tool. Since webpages in HTML format are often very lengthy, we use the same LLM in the agents to summarize the HTML into a concise summarization. The cost of this summarization process is also counted into the efficiency metrics. We utilized the official API endpoints of these LLMs with their default decoding parameter settings.

5.2 Main Results↩︎

The complete results are presented in Table 2. It can be found that most baselines demonstrate near-zero success rates, with only WebSailor (Gemini 2.5 Pro) and WebSailor (Claude Sonnet 4) exceeding 1-2% in Success Rate (Avg@4), confirming the inherent complexity of simultaneously handling deep reasoning and wide-scale information collection. Notably, Gemini 2.5 Pro emerges as the top-performing LLM, achieving the highest Column F1 (45.27%, Avg@4), Core Entity Accuracy (73.98%, Avg@4), and Pass@4 Success Rate (1.82%), even outperforming several agent systems. This exceptional performance indicates that Gemini 2.5 Pro possesses advanced reasoning capabilities for entity identification and extensive internal knowledge for filling result tables without external search. Furthermore, we detail the performance of baselines from depth and width metrics as below.

5.2.0.1 Depth Metrics

Our analysis reveals that agent systems generally enhance the deep search capabilities of base LLMs, as evidenced by consistent improvements in Core Entity Accuracy (CE Acc.). For example, the CE Acc. (Avg@4) of GPT-5 increases from 58.41% (base LLM) to 74.32% when integrated into WebSailor, representing a +15.91 percentage point gain. Similarly, Claude Sonnet 4 improves from 57.95% to 70.91% under WebSailor, demonstrating the effectiveness of iterative tool calls and multi-step reasoning in complex information retrieval. However, Gemini 2.5 Pro represents a notable exception to this trend. Upon close inspection of generated outputs, we find that Gemini 2.5 Pro in agent systems frequently fails due to three critical issues: (a) producing invalid markdown-formatted tables; (b) executing incorrect tool call APIs; and (c) incomplete task solving due to inference errors, occurring in 24.24% of cases on average—substantially higher than GPT-5 (16.36%) and Claude Sonnet 4 (17.80%). This suggests that Gemini 2.5 Pro’s output formatting behavior becomes brittle when subjected to multi-step tool orchestration. Critically, while agent systems improve core entity identification, they fail to consistently enhance column-level precision. For instance, the Column F1 (Avg@4) of Claude Sonnet 4 model declines from 32.63% (base LLM) to 30.08% in OWL and 21.60% in Smolagents. This pattern highlights a fundamental limitation: even when agents successfully identify core entities through multi-hop reasoning, current agent architectures cannot reliably collect complete entities, with their effectiveness often falling below the usage of internal knowledge in base LLMs.

5.2.0.2 Width Metrics

When evaluating width metrics that measure comprehensive information collection, we observe that most agent frameworks do not significantly improve the base LLMs’ wide search capabilities. Only three combinations demonstrate consistent improvements across all width metrics: OWL (Claude Sonnet 4), WebSailor (Claude Sonnet 4), and WebSailor (GPT-5). The remaining agents show substantial performance degradation compared to their counterpart base LLMs. Beyond the issues specific to Gemini 2.5 Pro that described above, the Smolagents framework also consistently underperforms across nearly all metrics. Our investigation reveals that Smolagents employs minimal reasoning before tool calls, which restricts the effectiveness of subsequent tool calls. This architectural constraint prevents Smolagents from formulating precise search queries, resulting in inadequate information coverage and poor performance on width metrics.

6 Analysis↩︎

In this section, we conduct several detailed analysis on Efficiency (Section 6.1), Tool Calls (Section 6.2), Differences in Dataset Construction Methods (Section 6.3), Per-topic Performance (Section 6.4), and Error Analysis (Section 6.5).

6.1 Efficiency Analysis↩︎

r0.5

Compared to deep search or wide search, DeepWideSearch imposes significantly higher computational and operational overhead. As shown in Table 3, even state-of-the-art agents incur substantial resource costs per query. For instance, OWL (GPT-5) and WebSailor (Claude Sonnet 4) achieve average $2.75 and $1.40 per question — with many queries remaining unresolved despite this high cost. Due to unstable network conditions and tool call errors, agents often require multiple retry attempts to complete tasks such as search, significantly increasing computational overhead—for instance, OWL (GPT-5) incurs an average cost exceeding $6.8 under retry conditions. These results underscore a critical inefficiency in current agent architectures when tackling complex deep and wide information seeking tasks. This suggests that existing systems are not yet scalable for real-world deployment of DeepWideSearch, motivating future work on efficient planning, memory reuse, and adaptive resource allocation.

6.2 Tool Calls Analysis↩︎

r0.4

Table 4 shows the average number of tool calls (Search and Visit tools) per sample across different backbone LLMs in WebSailor. Notably, WebSailor (Claude Sonnet 4) exhibits a significantly higher Search tool calls (23.23) compared to Gemini 2.5 Pro (4.77) and GPT-5 (8.72). This aligns with its superior performance (Table 2), suggesting that scaling the search tool calls improves the performance.

6.3 Differences in Dataset Construction Methods↩︎

Table 5 demonstrates the average performance of advanced LLMs (GPT-5, Claude Sonnet 4 and Gemini 2.5 Pro) with their counterpart agent systems. It can be found that the Deep2Wide construction method produces substantially more challenging data than Wide2Deep method. For example, LLMs and agents achieves nearly 0.0% success rate on Deep2Wide (Avg. LLMs: 0.0% Avg@4; Avg. Agents: 0.15% Avg@4), compared to the Wide2Deep (Avg. LLMs: 1.17% Avg@4; Avg. Agents: 1.23% Avg@4). Critically, the overall Entity Accuracy on Deep2Wide is only 33.29% (vs. 88.84% on Wide2Deep). This observation indicates that the synthesized deep sub-question in the Wide2Deep method is easier for LLMs to solve. Nevertheless, the column-F1 of Wide2Deep remains below 51%, indicating that comprehensively collecting entities is still challenging.

6.4 Per-topic Performance Analysis↩︎

As shown in Figure 6, we analyze topic-wise performance through bidirectional bar charts evaluating depth metrics (Column-F1, CE Acc.) and width metrics (Item-F1, Row-F1), excluding domains with fewer than 5 samples. Four key patterns emerge: (1) The top-5 most frequent topics (sample count >20) are Film & Movies, Politics, Finance, Technology, and Sports; (2) Politics achieves the highest item- and row-level F1 scores (35% and 19%), indicating wide search are more tractable in this topic, while Politics and Finance attain the highest column F1 and CE accuracy, suggesting deep search are comparatively easier here; (3) Despite strong depth performance in Finance, Travel, and Education topics, the performance of baselines exhibit substantially lower width metrics on these three topics (e.g., Travel 20% item F1 and Finance 8% row F1), revealing that strong deep search capability does not guarantee effective wide search capability; and (4) History and Games consistently underperform across all metrics (e.g., 5% Column-F1 of History), establishing them as the most challenging topics. These findings highlight the heterogeneous nature of search complexity across topics.

6.5 Error Analysis↩︎

As shown in Tables 2, agent systems might underperform backbone LLMs on DeepWideSearch tasks. Our error analysis reveals four key failure patterns: (1) Lack of Reflection: agents often lack effective reflection mechanisms. When encountering wrong trajectories (Figures 13) or tool call errors (Figure 14), they prematurely conclude the task is unsolvable and output empty tables rather than analyzing failure causes and exploring alternative paths; (2) Overreliance on Internal Knowledge: agents frequently overrely on internal knowledge. Even when correctly identifying core entities (Figure 15), they often generate tables solely using their internal parametric knowledge rather than performing proper web queries, resulting in outdated or inaccurate information due to limited training data scope; (3) Insufficient Retrieval: information retrieval is often insufficient. For example, despite identifying relevant pages (Figure 17), agents frequently fail to properly access complete context through visit operations, leading to significant information omissions. Even when visit operations are executed correctly, summarized webpage data may still miss critical details. This limitation motivates the design of a question-aware, customized webpage summarization process in agent systems; and (4) Context Overflow: context overflow presents a fundamental challenge. Deep wide search requires extensive multi-step reasoning and numerous search tool calls, significantly expanding context length (Figure 16). This issue occurred in 24.96% of cases, exceeding the context management capabilities of current agent architectures; In summary, these four error patterns highlight that current agents face substantial limitations when addressing the challenges of depth and width in complex information-seeking tasks. Addressing these limitations requires specialized architecture for deep wide search scenarios.

7 Conclusion↩︎

This paper addresses the critical gap in information-seeking agent evaluation by introducing DeepWideSearch benchmark, the first benchmark designed to simultaneously assess deep reasoning and wide-scale information collection. Our experiments demonstrate that state-of-the-art agents achieve only 2.39% average success rate on this challenging benchmark, revealing fundamental limitations for current agents. These results underscore the combinatorial complexity of deep and wide search as a key frontier to guide future research toward more capable information-seeking agents.

8 Limitations and Future Work↩︎

Despite our established DeepWideSearch benchmark, there are three key limitations remain to be addressed in the future work: (1) As shown in Table 5, the Wide2Deep construction method produces significantly easier questions than Deep2Wide, as evidenced by the substantially higher CE Accuracy. We will iteratively refine sub-questions to increase question complexity while maintaining natural language quality; (2) Our current dataset exhibits slight differences with real-world deep and wide search questions in terms of solution paths (Cases in Appendix 10). In future work, we will iteratively refine the DeepWideSearch dataset to better align with real-world applications; and (3) Our dataset construction relies heavily on human annotation, limiting scalability. Future work should explore automated data generation techniques and develop reference-free evaluation metrics that avoid complex, human-verified tabular answers, enabling efficient dataset expansion and model optimization across diverse domains.

9 Details of Datasets↩︎

The table volume in Table 1 represents the number of the searched information in the DeepWideSearch questions, which is defined as the product of rows and columns of the table. The average steps of the search entities is counted as the number of the reasoning steps and tool calls. Specifically, the average steps of GAIA is counted by the reference trajectories in the dataset, and the average steps of WideSearch is annotated by our three human raters. Besides, Figure 7 and Figure 8 present two cases in our proposed DeepWideSearch dataset.

10 Differences between Our Dataset and Real-world Questions↩︎

Figure 9 illustrates two representative deep and wide search questions: the first is an example from our constructed DeepWideSearch dataset, and the second is drawn from a real-world e-commerce scenario. While our dataset captures the essential characteristics of deep and wide search, the primary difference from real-world settings lies in the solution path. In our dataset, the process emphasizes first performing a deep search to gather critical information, followed by a wide search to expand relevant attributes. In contrast, real-world tasks often begin with a wide search to collect a large pool of candidates, followed by a deep search over each candidate for verification. Nevertheless, it is important to emphasize that despite this procedural difference, our dataset still exhibits the traits of deep and wide search. Specifically, during the initial deep search phase, the model also need to list and reason over a set of candidates, systematically applying deep verification to determine which candidates satisfy the problem constraints and thereby identify the correct target entity. Consequently, even this first-stage deep search inherently incorporates the characteristic of the wide search.

11 Prompts for DeepWideSearch Data Construction↩︎

This section presents three prompts for Wide2Deep method: (1) Core Entity Extraction Prompt in Figure 10; (2) Deep Sub-Question Synthesis Prompt in Figure 11; (3) Question Fusion in Figure 12.

12 Error Cases in DeepWideSearch↩︎

This section provides the four kinds of representative errors of agents: (1) Lack of Reflection (Figure 13 and Figure 14); (2) Overreliance on Internal Knowledge (Figure 15); (3) Context Overflow (Figure 16); and (4) Insufficient Retrieval (Figure 17).

References↩︎