1 Introduction↩︎

Web agents—language models coupled with a browser and tool-use interfaces—have rapidly emerged as a core direction for building general-purpose assistants that can search, navigate, and act on the open web [1]–[5]. Despite striking demonstrations, progress is increasingly constrained by data. Existing benchmarks provide only human-annotated task descriptions paired with a curated end-to-end solution, offering no supervision over the agent’s latent decision process [3], [6]–[9]. This is problematic: prior work has highlighted that web navigation is naturally a hidden Markov process, where the critical uncertainty lies in the unobserved intermediate states and actions [4], [10]. Without reliable ground-truth trajectories, agents are overfitting to small hand-authored sets, and struggling to generalize to realistic multi-hop, cross-page use cases [11], [12].

To address the scarcity of large-scale data, recent efforts have begun to generate web tasks automatically. For instance, AgentTrek [13] converts online tutorial documents into executable trajectories and validates them with a vision-language agent, while WebWalker [14] and NNetNav [11] use an LLM-based policy model to explore websites and then label the resulting trajectories with intent. While promising, these approaches face three critical limitations. First, they are prohibitively expensive in both time and cost: constructing a single task requires repeated agent–environment rollouts and multiple LLM calls, driving costs upward and limiting throughput. Second, they induce strong exploration bias: LLM policies tend to overfit to “obvious” paths (e.g., repeatedly selecting product detail pages), while overlooking more nuanced or less salient site functionalities such as rewards programs, sales, or new arrivals. As a result, large portions of the site graph remain unexplored and untested. Third, they lack compositionality and utility: most tasks generated by existing approaches collapse to single-hop retrieval or form-filling that could be solved by a simple search query, failing to capture the multi-hop, cross-page dependencies of real-world navigation. As a result, such tasks offer little practical utility for users, since solving them does not require deeper reasoning, sustained interaction, or problem-solving skills that would actually test or benefit an intelligent web agent in realistic settings. We provide detailed comparison in 1.

We introduce GTA(Generate Long-Horizon Tasks for Web Agents at Scale), a scalable task-generation and benchmarking framework for web agents that integrates crawl, retrieve, in-context generation, and quality control to produce realistic, graded-difficulty tasks paired with executable ground-truth trajectories. Concretely, we first crawl publicly accessible sites to construct a content/link graph; next, we retrieve candidate pages and evidence snippets as seeds; we then generate tasks via in-context prompting that references on-site facts, UI elements, and workflows; we filter with an LLM-based quality control protocol that enforces clarity and solvability against the current crawl; and finally we record a minimal, programmatically executable navigation path (clicks, scrolls, inputs) enabling deterministic replays and step-level attribution.

This design directly targets the three challenges identified earlier. Efficiency and cost: we separate one-time crawling from lightweight, retrieval-seeded task generation, eliminating repeated policy rollouts and reducing both redundant LLM calls and wall-clock latency. Exploration coverage: by leveraging retrieval-based seeding rather than policy-driven exploration, our method avoids the strong biases of LLM navigators and systematically samples diverse, semantically rich regions of each website’s graph. Compositionality and utility: grounding generation in the site graph enables the construction of multi-hop tasks that require reasoning across interconnected pages, mirroring the dependencies and open-ended objectives characteristic of real user workflows.

To automatically generate diverse multi-hop web agent tasks, we distinguish between two primary settings: intra-website queries, where reasoning unfolds entirely within a single domain, and inter-website queries, which require coordination across multiple domains. Within intra-website tasks, our benchmark spans a broad range of languages, including English, Italian, Japanese, German, and Chinese, ensuring coverage beyond English-centric environments.

We instantiate the pipeline on over 50 public websites spanning e-commerce catalogs, entertainment/government pages, forums, and finances, etc. Beyond automatic filtering, we collect human annotations to assess answer correctness, complexity, utility and realism, confirming that tasks are reasonable for humans yet challenging for current agents. Our contributions are three-fold:

• Problem formulation. We introduce and formally define the new task of multi-hop web-agent task generation, where tasks require agents to reason across multiple pages or domains rather than collapsing to trivial information lookup.

• Efficient automatic data generation. We propose an efficient pipeline that markedly reduces per-task time and cost while producing challenging tasks; each task includes an executable minimal “gold path” for deterministic replay and step-level diagnostics, safeguarded via verifier checks and negative controls.

• Evolving benchmark. Rather than treating GTA as a fixed dataset, we frame it as a self-evolving benchmark ecosystem in which end users can continually generate tasks from current web content. Coupled with our verification pipeline, GTA preserves task quality and reproducibility while exposing agents to a changing, realistic task distribution, thereby reducing overfitting and enabling ongoing evaluation of generalization.

2 Benchmark Design↩︎

The goal of our benchmark is to create web-agent tasks that systematically capture the multi-hop nature of realistic online interactions. Unlike prior datasets that treat each task as a linearized trajectory, we explicitly ground task generation in the structural properties of the underlying web graph. This design choice serves two objectives: (i) it reflects the fact that solving many real-world queries requires reasoning across multiple, interlinked webpages, and (ii) it enables reproducible task construction decoupled from transient agent rollouts. In what follows, we describe the full pipeline in four stages: collecting observations by crawling and constructing a site graph, generating multi-hop tasks through retrieval-augmented prompting, controlling task quality via systematic verification, and analyzing coverage relative to existing benchmarks.

2.1 Benchmark Construction↩︎

We aim to construct tasks that require multi-hop reasoning over web environments. Formally, let \(\mathcal{G}=(V,E)\) denote a directed web graph, where each node \(v \in V\) corresponds to a crawled webpage and each edge \((v_i,v_j)\in E\) corresponds to a hyperlink from \(v_i\) to \(v_j\). A task is represented as \(T = \langle q, A, P^\star \rangle\), where \(q\) is the natural-language query, \(A\) is the answer, and \(P^\star = \langle v_1, v_2, \ldots, v_m \rangle\) is a minimal executable path within \(\mathcal{G}\) that leads to \(A\).

We define a task as multi-hop of order \(n\) if solving it requires aggregating information from at least \(n\) distinct webpages, i.e., \[| \{ v \in P^\star : \text{evidence}(v) \neq \emptyset \} | \geq n, \quad n > 1.\] Here, \(\text{evidence}(v)\) denotes the subset of page content from node \(v\) that is necessary to derive the correct answer. In contrast, a single-hop task admits \(n=1\), solvable by local reasoning from one node only.

To instantiate \(\mathcal{G}\), we perform breadth-first crawling over publicly accessible domains. Each hyperlink discovered from page \(v\) is initially added as a child node. However, paths to a given webpage are not unique, which risks cycles and inflated traversal lengths. To address this, we enforce a canonical shortest-path policy: whenever a node is reached via a shorter path, we update its parent assignment accordingly. Specifically, each node stores the accessibility tree of its corresponding webpage.

Once \(\mathcal{G}\) is collected, we embed each node \(v \in V\) into a dense representation \(h_v \in \mathbb{R}^d\) using an LLM-based encoder \(f_\theta\). Given a seed node \(v_s\), we retrieve a candidate set \(\mathcal{R}(v_s) = \{v_1,\ldots,v_k\}\) via dense retrieval. To induce compositionality, we randomly select a secondary node \(v_t \in \mathcal{R}(v_s)\) and provide \((\text{content}(v_s), \text{content}(v_t))\) as context to a generator LLM. The model is prompted to synthesize a query \(q\) whose answer \(A\) requires reasoning across both pages. This retrieval-and-pairing strategy enforces multi-hop dependencies by design, rather than relying on incidental cross-references.

All generated tasks are subjected to a multi-stage verification pipeline:

1. Multi-hop validation. We reject cases where the model produces a conjunction of two independent single-hop queries (e.g., linked by “and”), which do not constitute genuine compositional reasoning.

2. Answer correctness. For each candidate \((q, A)\), we condition a verifier LLM on the content of supporting nodes only, requiring it to reproduce or validate \(A\). Incorrect or unverifiable outputs are discarded.

3. Ambiguity detection. We assess whether \(q\) admits a unique answer. Queries missing critical qualifiers (e.g., temporal or geographic constraints) are filtered out.

4. Solvability check. We ensure the existence of at least one minimal executable path \(P^\star\) within \(\mathcal{G}\) such that traversal yields sufficient evidence to derive \(A\). This guarantees deterministic replayability and prevents spurious generations.

The resulting benchmark thus consists of well-formed, unambiguous multi-hop tasks, each paired with a cached graph \(\mathcal{G}\) and a minimal executable trajectory \(P^\star\), enabling reproducible training and evaluation.

2.2 Benchmark Details↩︎

A key feature of our pipeline is that it is fully automatic: once a website graph is constructed, tasks can be generated without any manual authoring or intervention. This design implies that, in principle, the pipeline can scale to produce arbitrarily many tasks across arbitrarily many domains.

In this paper, we instantiate the benchmark on over 50 publicly accessible websites spanning healthcare, finance, e-commerce, entertainment, government, and scientific resources. This breadth ensures heterogeneous evaluation settings, as summarized in 4.

In total, we generate 5,000 intra-website tasks across the covered domains. Within the Healthcare and Finance group of websites, we generate 600 cross-website inter-website tasks. These generated tasks are used for experiments in this paper.

We release the generation pipeline to allow end users to create up-to-date tasks directly from live websites. This supports continual refresh of evaluation data, reduces overfitting to a static benchmark, and reframes GTA as a reusable task-generation framework rather than a one-time dataset.

3 Analysis↩︎

In this section, we first compare query structures across datasets (3.1). We then quantitatively assess task difficulty by benchmarking performance against a simple search-based baseline, revealing that, unlike previous datasets, GTA tasks cannot be trivially solved by single-hop retrieval (3.2). We perform a page-coverage analysis to measure how broadly each benchmark explores the underlying website graph, showing that GTA achieves substantially higher structural diversity (3.3). We present human evaluation results confirming that GTA yields complex, verifiable, and realistic tasks for web-agent assessment (3.4). Finally, we present a time and cost efficiency analysis (3.5).

3.1 Qualitative Comparison of Queries↩︎

A closer examination of representative tasks highlights substantial differences between GTA and prior benchmarks. Unlike WebDS, which largely consists of factual lookups or pairwise comparisons (e.g., retrieving COVID-19 statistics or population growth rates), GTA tasks are explicitly compositional. They often require integrating multiple heterogeneous sources of evidence, such as combining numerical statistics (career strikeouts), product attributes (price and ratings), and domain-specific safety constraints (pharmacological dosage and drug interactions). This design ensures that tasks involve multi-hop reasoning rather than collapsing into atomic queries.

Another distinction lies in the nature of the required reasoning. GTA emphasizes context-dependent decision-making, where answers are not limited to factual retrieval but also demand comparative or evaluative judgments (e.g., identifying which product is superior, or whether a medical combination is safe). By contrast, AgentTrek and NNetNav focus on navigation-oriented instructions whose outcomes are often underspecified (e.g., “expand results on SeatGeek” or “find a restaurant”).

Finally, the answer space of GTA tasks is designed to be precise and verifiable, grounded in structured numerical, categorical, or medical evidence. This stands in contrast to the open-ended or underspecified nature of prior benchmarks, where the lack of explicit answers limits their utility for rigorous evaluation. Collectively, these properties position GTA as a benchmark that emphasizes long-horizon, evidence-grounded task construction, thereby enabling a more faithful assessment of reasoning and integration capabilities in web agents.

3.2 Difficulty Comparison Against a Simple Search Baseline↩︎

We first evaluate a simple baseline where an LLM retrieves results from the Google Search API and generates an answer based on the returned snippets. To enable a fair comparison, we filter information-retrieval style tasks from both AgentTrek and NNetNav. As shown in 1, the correctness rate is surprisingly high: 95% on AgentTrek and 100% on NNetNav. On GTA, however, the baseline only reaches 14%. This indicates that tasks in AgentTrek and NNetNav often reduce to single-hop fact lookup, which can be trivially solved by a search engine. The inflated correctness rates reveal that these datasets do not capture the core challenge of web agents: reasoning over dispersed information across multiple pages and interfaces.

3.3 Page-Coverage Analysis↩︎

We next turn to the page-coverage rate, defined as the fraction of first-level nodes in the crawled website graph (roughly corresponding to distinct site functionalities, such as “team roster,” “product details,” or “checkout page”) that are exercised by benchmark tasks. This metric is inspired by the line-coverage concept in programming languages, which measures how much of the codebase is actually exercised by a test suite. Analogously, high page coverage implies that benchmark tasks probe a diverse set of site functionalities, while low coverage indicates narrow task construction.

As shown in 1, the coverage is extremely limited: 20.2% for AgentTrek, 24.3% for NNetNav, and only 11.0% for WebDS, which are all far lower than GTA page coverage rate. This narrow span suggests that existing datasets focus disproportionately on a small subset of site functionality. The overlooked regions represent critical parts of the user experience, yet they remain underrepresented in benchmark tasks. Taken together, these analyses demonstrate that prior benchmarks (i) collapse to factoid-style queries easily solved by search engines, and (ii) induce strong exploration biases that leave large swaths of website functionality untested.

3.4 Human Evaluation↩︎

To further assess the reliability of our automatically constructed benchmark, we randomly sample 100 tasks and evaluate the correctness of the annotated answers through human judgment. For each sampled task, we provide annotators with the generated task description, the proposed answer, and the two source web pages from which the answer is derived. Annotators are asked to verify whether the annotated answer is indeed correct given the provided evidence. The resulting human agreement rate reaches 87%, demonstrating that our pipeline produces tasks and answers that are both interpretable and verifiable, with only a small fraction of cases requiring clarification.

We further conduct a controlled human study (20 tasks per dataset) to compare tasks generated by our approach (GTA), WebDS, and AgentTrek along four dimensions: complexity, utility, realism, and overall quality. Results are shown in Table 3.

4pt

Annotators judge GTA tasks to be substantially more complex (2.35) than those in WebDS (1.90) and AgentTrek (1.23), indicating that our pipeline successfully produces multi-hop navigation and reasoning scenarios rather than trivial, single-hop lookups. On utility, GTA achieves the highest rating (2.03), followed closely by AgentTrek (2.00) and WebDS (1.98), suggesting that GTA maintains comparable usefulness with added reasoning depth.

For realism, AgentTrek scores the highest (1.98), outperforming WebDS (1.68) and GTA (1.53). This gap may stem from our graph-grounded generation process, which can introduce queries that are less reflective of natural user intents. Finally, GTA attains the best overall quality score (1.82), surpassing both WebDS (1.77) and AgentTrek (1.73), highlighting that tasks generated by our approach are both coherent and challenging.

Taken together, these results demonstrate that GTA yields tasks that best balance realism, reasoning depth, and evaluation value—producing scenarios that are both practically useful and more faithful to the complexities of real-world web navigation.

3.5 Time and Cost Efficiency↩︎

A key advantage of GTA lies in its scalability and efficiency. We present detailed quantitative comparisons in 10. In summary, GTA achieves a dramatic reduction in both computational and financial overhead compared to prior task-generation pipelines.

4 Experiments↩︎

In this section, we first introduce the baseline agents evaluated in this paper (4.1). We then report their performance on GTA intra- and inter-website tasks (4.2). We also extend evaluation to multilingual web environments (4.3). Finally, we conduct an error analysis to identify key failure modes ([ssec:32error95analysis]). Specific implementation details are presented in 13.

4.1 Baselines↩︎

² Browser Use is an open-source framework that couples large language models with a real browser environment. The perceive–decide–act cycle enables robust handling of multi-step workflows such as navigation, form-filling, and information extraction across diverse sites. It is a widely adopted baseline, achieving strong results on benchmarks like WebVoyager.

[17] AgentOccam represents a complementary approach that prioritizes simplicity and alignment. It refines the interface between the LLM and the web environment by pruning rarely used actions and filtering page observations into concise textual representations. It has achieved state-of-the art performance on benchmarks like WebArena.

We evaluate a limited number of agent backbones by design, consistent with prior web-agent benchmarks, which typically focus on representative agent paradigms rather than exhaustive architectural sweeps, for example, WebDS[16] evaluates Browser Use and AgentOccam, while NNetNav[11] and WebWalker[14] primarily study ReAct-style agents. Our goal is to validate the benchmark under commonly used agent behaviors; accordingly, the primary contribution lies in the benchmark itself rather than in the breadth of agent architectures evaluated.

4.2 Benchmark Results↩︎

2 reports success rates of Browser Use and AgentOccam across domains. A clear gap emerges between our benchmark and prior ones such as WebVoyager and Mind2Web. While both agents achieve relatively high scores on these earlier datasets (up to 82% on WebVoyager and 45% on Mind2Web), their performance on our benchmark domains is markedly lower, often below 20–30%. This contrast suggests that existing benchmarks have become static and saturated, allowing agents to overfit to templated interfaces or repeated patterns, whereas our benchmark introduces dynamic, evolving sites that demand genuine adaptation and remain useful over a longer horizon.

Beyond the overall drop in success rate, we also observe sharp performance variations across domains. As shown in 8 musicbrainz.org or cvshealth.com admit moderate success, whereas shopping portals like underarmour.com and ticketcenter.com prove substantially harder. Such heterogeneity highlights that real-world navigation cannot be captured by single-domain evaluation: robustness requires agents to generalize across diverse layouts, workflows, and interaction styles.

To demonstrate the flexibility of our pipeline, we also generate inter-website multi-hop tasks, where solving a task requires integrating information from multiple websites rather than staying within a single domain. For these experiments, we select two domains with naturally complementary sources of information: healthcare (Mayo Clinic, WebMD, Drugs.com, CVS Health, and CDC) and finance (Yahoo Finance, markets.ft and Seeking Alpha).

We evaluate these tasks using the Browser Use agent, which executes tasks in a real browser environment. Experiments show that success rates for cross-website tasks are substantially lower than those observed in single-website settings. For example, Mayo Clinic and WebMD tasks achieve a 12.5% success rate, CDC and WebMD reach 8.2%, and Drugs.com & CVS Health drop to just 4.2%. These numbers are considerably lower than the success rate of same-website tasks reported earlier, confirming that cross-website integration poses unique challenges.

We attribute this difficulty to two factors. First, cross-website tasks require agents to reason about complementary but non-overlapping knowledge: one site may provide background definitions or guidelines, while the other presents instance-specific details (e.g., drug side effects vs. dosage instructions). Second, agents must manage distributional shifts across sites, such as inconsistent terminology, different UI structures, or divergent information formats. This stands in contrast to same-site navigation, where once an agent adapts to the local interface, it can generalize within that site.

4.3 Multi-lingual Task Performance↩︎

To further evaluate the flexibility and coverage of our pipeline, we extend task generation beyond English to multilingual web environments spanning the health, sports, and news domains. We crawled representative websites in four languages—English, Italian, German, and Japanese—and constructed tasks from each. We then evaluate these tasks using two representative web agents, Browser Use and AgentOccam. As shown in 9, the results reveal a pronounced degradation in performance on non-English websites. For example, the Browser Use agent attains a 0% success rate on the German site, 2.9% on the Japanese site, and 0% on the Italian site. In contrast, the same agent achieves markedly higher performance on English-language health websites, with success rates of 8.2% on CDC and 6.0% on NIH.gov. These findings highlight a strong bias in current web agents toward English-centric environments, while exposing severe limitations in their ability to operate on foreign-language websites. A similar trend is consistently observed with AgentOccam and across other domains, corroborating the language-specific performance disparity.

4.4 Error Analysis↩︎

We conduct a detailed error analysis on a random sample of 100 failed tasks to better understand the limitations of current web agents on our benchmark. Three major categories of failure modes emerge.

A common error pattern arises when the agent fails to correctly navigate to all webpages required by a multi-hop query. Since many tasks require synthesizing evidence from two or more distinct sites, failure to reach even one of them leads to incomplete reasoning chains and ultimately incorrect answers. We observe that this issue often stems from brittle link-following strategies, where the agent either misinterprets hyperlink semantics (e.g., clicking on visually similar but contextually irrelevant links) or becomes trapped in navigation loops.

Another prominent failure mode involves premature termination of the task. Agents sometimes generate a final answer after retrieving partial evidence, without fully verifying consistency across all required sources. This behavior is especially prevalent in tasks that involve numerical aggregation or comparisons, where one piece of evidence might seem sufficient but is actually incomplete. Such early stopping reflects the broader challenge of balancing efficiency with completeness in long-horizon reasoning.

We also find that agents frequently overuse on-site search boxes rather than exploiting the explicit structure of the web graph. While search boxes sometimes provide shortcuts, they are not guaranteed to expose the necessary evidence, and reliance on them can bypass important compositional navigation steps. In many cases, agents retrieve irrelevant or overly broad results through poorly formulated queries, which then propagate errors into the reasoning process.

5 Related Work↩︎

A number of benchmarks have been developed to evaluate web agents. Mind2Web [6] provides human-authored task–solution pairs for open-domain web navigation. WebVoyager [7], WebArena [3], BrowseComp [15] and WebDS [16] provide realistic environments and curated tasks for web-based decision making. However, these benchmarks mainly provide end-to-end trajectories, offering limited support for fine-grained diagnostics and compositional reasoning.

To reduce annotation costs, recent work has explored automatic task generation. AgentTrek [13] transforms web tutorials into executable trajectories, while WebWalker [14] and NNetNav [11] use LLM-driven exploration to synthesize tasks from raw environments. Despite their promise, these approaches can be expensive due to repeated rollouts, biased toward salient regions of a site, and often produce tasks that collapse into simple single-hop retrieval. GTA also differs in construction strategy. Unlike WebWalker, which discovers tasks through model-driven exploration, GTA first systematically crawls websites, builds explicit site graphs, and then generates tasks from enumerated graph structures. This design reduces exploration bias and enables controlled graph coverage and cross-page evidence composition. Compared with human-authored benchmarks such as BrowseComp, GTA is fully automated and scalable across 50+ websites, while explicitly guaranteeing multi-hop evidence distribution, cross-page dependency, and non-answerability from any single page.

6 Conclusion↩︎

We introduced GTA, a scalable benchmark for web agents. By decoupling crawling from task generation and anchoring tasks to the site graph, GTA produces compositional, verifiable tasks with executable trajectories across diverse domains and languages.

Our analyses show that GTA surfaces challenges overlooked by prior benchmarks—agents fail on cross-website and multilingual tasks, and simple search baselines cannot solve them. Human studies further confirm that tasks are more complex, useful, and diagnostic. We hope GTA will serve as a foundation for advancing robust, generalizable web agents.

Limitations↩︎

Although GTA spans over 50 publicly accessible websites across diverse domains and languages, the current selection remains limited relative to the vast heterogeneity of the modern web. Certain interaction types—such as authentication-gated services, dynamically rendered JavaScript interfaces, and transaction-based workflows—are excluded due to ethical and security constraints. Extending GTA to such settings will require robust sandboxing and privacy-preserving mechanisms for exploration.

The automatic verification process, though effective at filtering low-quality or ambiguous tasks, still depends on LLM-based validators. Subtle reasoning errors or underspecified compositional dependencies may occasionally persist. Incorporating human-in-the-loop validation or multi-model consensus could further strengthen quality assurance, especially for tasks involving nuanced domain knowledge.

Our pipeline relies on periodic crawls to construct site graphs, enabling scalable data collection and timely iteration. However, task difficulty may vary due to multiple factors, including structural updates in websites [12], the time interval between task creation and evaluation, and content drift or removal. These dynamics can affect task solvability and consistency across different crawl snapshots. Future work should explore more generalizable web agents that are robust to such temporal and structural variations.

GTA focuses on multi-hop information-seeking tasks rather than the full range of web agent behaviors. In particular, it does not cover procedural workflows such as form-filling, checkout, or other action-heavy interactions. This scope is intentional: prior information-seeking benchmarks often collapse into single-hop retrieval, leaving cross-page evidence integration underexplored. Procedural workflows are an important complementary challenge and are already represented in benchmarks such as WebArena.

Ethics Statement↩︎

We strictly adhere to ethical and legal standards in data collection and experimental design. All web data used in this study were obtained through automated scripts that fully comply with each website’s crawler and robots.txt policies. We only accessed publicly available webpages and refrained from interacting with or collecting any user-specific or private information. The crawling process respected server load limitations by introducing randomized delays and rate-limiting to minimize network impact.

Additionally, all data were used solely for research purposes and are released or analyzed in aggregate form to prevent any potential misuse or privacy risk. Our benchmark construction and experiments comply with institutional and community ethical guidelines for web data research.

Acknowledgments↩︎

Muhao Chen was supported in part by the National Science Foundation under grants OAC-2531126 and ITE-2333736.

7 Website Coverage↩︎

4 shows representative websites covered in GTA benchmark.

8 Prompt Details↩︎

3 shows detailed prompt of multi-hop web agent task creation. 4, 7, and 5 show detailed prompts of the quality control pipeline.

None

None

None

None

None

9 Performance Per Website↩︎

8 shows the success rates of browseruse and Agentoccam on a subset of websites.

10 Cost of Task Generation↩︎

An important consideration in designing GTA is the computational and monetary cost of generating large-scale tasks. Our pipeline is deliberately lightweight: the only substantive expense lies in producing webpage descriptions for retrieval indexing. Crawling itself incurs no cost, and in-context task generation requires only a small number of LLM calls.

Concretely, generating descriptions for approximately 2,000 webpages and generating 100 multi-hop tasks via in-context prompting costs around $10. Once the site graph \(\mathcal{G}\) is constructed and indexed, however, the marginal cost of additional tasks is effectively negligible: the graph enables arbitrarily many tasks to be generated without re-incurring indexing costs. By contrast, prior benchmarks such as AgentTrek report an average cost of $0.55 per task, as they rely on LLM-based exploration policies to traverse websites. This design makes the marginal cost of each new task significantly higher and limits scalability, whereas our graph-based approach amortizes indexing cost and supports large-scale task generation at roughly $0.10 per task in our experiments. This low overhead is particularly advantageous for dynamic test sets, where new tasks can be generated continuously with minimal additional expense.

In addition to monetary efficiency, our pipeline is designed to be lightweight in terms of time. The most time-consuming stage is crawling webpages and creating retrieval index. For a medium-scale setting of approximately 2,000 webpages, this process completes within 30–40 minutes on a single machine. Once the site graph is built, task generation itself is relatively fast, allowing us to rapidly scale the benchmark to thousands of multi-hop tasks without significant runtime bottlenecks. This property stands in contrast to LLM-exploration-based benchmarks, where each new task requires running an agent rollout through the website, incurring substantial time costs per task.

11 Multilingual Task Performances↩︎

9 shows web agents’ cross-lingual performances. Both BrowserUse and AgentOccam exhibit a consistent decline in success rate when moving from English to non-English sites. The results highlight a strong bias in current web agents toward English-centric environments, while exposing severe limitations in their ability to operate on foreign-language websites.

12 Annotation Details↩︎

We recruit 3 annotators with computer science background for our human evaluation. An example annotation interface is shown in 10. The inter-annotator agreements (Cohen’s Kappa) is 0.3, which indicates fair agreement.

13 Implementation Details↩︎

We use GPT-4o as the underlying large language model for agent exploration and GPT-5.1 for task generation. Our web interaction stack is built on Chromium, which is used to render webpages and capture visual snapshots for agent perception. This setup matches that of prior web-agent benchmarks, enabling consistent state replay and fair comparison. Each environment step corresponds to a deterministic browser action followed by a rendered page snapshot.

We inherit environment settings and site configurations from established benchmarks where applicable (e.g., using the same website instances and task distributions as WebArena and WebDS). To ensure safety and reproducibility, we restrict the agent to publicly accessible webpages and explicitly avoid login-gated content, personal accounts, or sensitive transactional flows.

Dynamic webpage elements such as pop-ups, cookie consent banners, and modal dialogs are handled using the same rules and heuristics defined in the corresponding benchmark environments. We ensures that agent behavior remains comparable to prior baselines and that observed improvements stem from reasoning and decision-making rather than environment customization.

References↩︎

---

1. Code available at https://github.com/tenghaohuang/GTA.↩︎

2. https://github.com/browser-use/browser-use.↩︎