1 Introduction↩︎

While leading frontier models from the US and China are shifting their evaluation focus from foundational capabilities (e.g., instruction following, reasoning, and tool calling) toward compositional agentic evaluation [1], [2], the Korean AI community remains largely anchored to static benchmarks [3]–[6]. Korean agentic benchmarks are still virtually nonexistent, leaving the community without a standardized way to measure progress. Developing such benchmarks is important for two reasons.

• For Korean developers and users, language usage and population size place Korea at a structural disadvantage relative to larger language communities, raising AI sovereignty concerns for queries requiring Korean local and cultural knowledge [7], [8].

• Second, Korean agentic benchmarks serve the broader research community. Frontier models increasingly saturate existing benchmarks, blurring the boundary between in-distribution and out-of-distribution evaluation [9], [10]. Agentic benchmarks grounded in linguistically and culturally distinct contexts can therefore provide a principled testbed for broader generalization [11], [12].

Based on this motivation, we take a first step toward building Korean agentic benchmarks by proposing K-BrowseComp, a browsing agent benchmark grounded in Korean contexts. We focus on browsing agents for two reasons. First, browsing agents are uniquely dependent on local and cultural knowledge: their core function is to retrieve region-specific information from the web, making the gap between Korean and English-centric contexts especially consequential for end users. Second, browsing agents are inherently compositional, jointly exercising instruction following, tool calling, and multi-turn interaction, and thus provide a comprehensive testbed for Korean agentic models.

K-BrowseComp consists of 400 problems in total, including the K-BrowseComp-Verified subset of 300 manually crafted problems, which are validated by native Korean speakers, and a 100-problem synthetic diagnostic split. As shown in Figure 1 and Table 2, the strongest evaluated models achieve only 30.00–45.67% on K-BrowseComp-Verified. Specifically, GPT-5.5 and DeepSeek-V4-Pro obtain 45.67% and 30.00%, respectively, which is a substantial drop from their reported performance on the original BrowseComp [13] (84.4% and 83.4%, respectively). Moreover, Korean LLMs released through the first round of government-funded “Proprietary AI Foundation Model Project” [3] obtain substantially lower scores ranging from 0.00% to 10.33%. Our trajectory analysis shows that these gaps do not reduce to a single bottleneck. Some models terminate search prematurely or fail to emit stable tool-call trajectories, while others retrieve relevant Korean web evidence but fail to preserve candidate sets, constraints, role bindings, or final-answer state across sources. This indicates that K-BrowseComp can serve as a diagnostic target for the Korean ecosystem to develop browsing agents tailored to Korean users.

Beyond K-BrowseComp-Verified subset, the core technical contribution of this paper lies in our methodology for constructing the remaining 100 problems with LLMs. A defining property of browsing tasks is their information asymmetry: solving a problem can be difficult, while verifying a candidate answer is comparatively easier once the relevant evidence path is known [13]. Motivated by this asymmetry, we examine an analogous question on the construction side: while solving such problems is hard, is it also hard to create them?

To this end, we employ a web browsing LLM agent as a proposer in the same format as our 17 human annotators and study whether it can produce challenging and well-defined problems. When naively instructed, the machine-generated problems were either solvable by frontier models or ill-defined. However, when we (i) provide hard human-written problems as few-shot exemplars or (ii) instruct the agent to target categorized failure modes identified in our analysis, the generated problems become substantially higher in quality and difficulty. Specifically, the models achieve 0.00–26.00% accuracy on the 100 machine-generated synthetic problems, compared with 0.00–45.67% on the corresponding subset of K-BrowseComp-Verified. These results indicate that our generation method produces a challenging diagnostic split.

2 Related Work↩︎

2.0.0.1 Web browsing agents.

Prior work established the foundation of evidence retrieval and reasoning, the core components of open-domain QA [14]–[17]. However, their evidence sources are typically predefined (e.g., Wikipedia), or LLMs operate over a fixed number of short turns, which became insufficient to evaluate frontier LLMs. Hence, recent web browsing benchmarks test LLMs as agents that use tools (e.g., web search) over a longer horizon across multiple websites. For instance, BrowseComp [13] proposed a set of challenging questions that even humans with access to a web browser can’t solve within 2 hours, and BrowseComp-ZH [18] extends the paradigm to utilizing multiple Chinese websites. In our work, we propose K-BrowseComp, which necessitates accessing multiple Korean websites. Beyond simply curating the question in Korean language, we consider a broad range of components that reflect the needs of Korean users (e.g., search conventions, local entities, semi-structured pages in Korean websites, and culturally grounded clues) that aren’t covered by existing benchmarks.

2.0.0.2 Korean and regional language evaluation.

Korean LLM evaluation has mainly focused on static language understanding, factual knowledge, and reasoning. KorQuAD and KoBEST measure reading comprehension and core NLU skills, while KMMLU scales multitask knowledge evaluation for Korean [8], [19], [20]. CLIcK [7] evaluates cultural and linguistic intelligence in Korean, and recent regional or multilingual benchmarks such as INCLUDE [11] and MENLO [12] emphasize locally grounded knowledge and native-like norms. These resources are important for Korean language evaluation, but they usually do not require agents to search the web, maintain evidence state, or synthesize information across pages. K-BrowseComp fills this gap by converting Korean cultural, institutional, educational, geographic, and media knowledge into agentic browsing tasks. This design evaluates whether a model can act as a useful Korean web agent, beyond answering questions from static benchmark distributions.

2.0.0.3 Synthetic task generation.

A growing body of work uses LLMs to scale instruction data and evaluation data, including self-instruction, synthetic task generation, and adversarial example construction [21]–[23]. At the same time, benchmark quality remains a central concern because model-generated tasks can be underspecified, too easy, or contaminated by highly indexed sources. Recent work on data contamination and benchmark generalization shows that evaluation results can overstate model ability when tasks are memorized or only require shallow generalization [9]. K-BrowseComp contributes a complementary construction strategy for web-browsing tasks. It uses human-verified Korean browsing questions to identify recurring failure modes, then asks a browsing agent to generate new questions that target those modes. This pipeline uses the verifiability structure of browsing questions to scale Korean agentic evaluation while keeping the answer unique and publicly supported.

4pt

3 K-BrowseComp↩︎

K-BrowseComp is a Korean browsing benchmark designed to evaluate whether web browsing agents can retrieve hard-to-find public information grounded in Korean contexts to solve user queries. As shown in Figure 2, each question is designed to have a single short answer that is stable over time and supported by publicly accessible web evidence. The same figure illustrates the benchmark’s two reasoning formats. Parallel-branching questions require intersecting multiple independent constraints to identify a unique answer, while multi-hop questions require using an intermediate finding to retrieve subsequent evidence. In the following sections, we describe the construction and validation of the K-BrowseComp-Verified subset (§3.1) and the 100 synthetic split (§3.2).

3.1 K-BrowseComp-Verified↩︎

3.1.0.1 Annotator guidelines.

We ask annotators to follow three rules when constructing K-BrowseComp-Verified. First, each question should be grounded in Korean contexts and supported by public textual web evidence. Second, each question should be difficult to answer through direct search, but easy to verify once the answer was found. Third, each question should require either multi-hop reasoning or parallel constraint satisfaction, with at least four steps or constraints. We prohibit annotators from using LLMs to create questions, and submitting questions that relies on private, paid, downloaded, or non-textual sources. Also, we require each item to have a unique answer (i.e., no questions that have multiple plausible answers) and temporally stable answer (i.e., answers that don’t change over time). Further details are explained in Appendix 8.

3.1.0.2 Validation procedure.

All finalized items are manually verified by the authors. For each item, we check whether the gold answer, intermediate entities, and cited sources are recoverable from public web evidence. Items with inaccessible, insufficient, or inconsistent evidence are returned to the original annotators for revision. We also review each item for natural wording, temporal stability, and answer uniqueness. When our baselines produce a concrete answer that differs from the gold answer, the case is manually inspected to determine whether the model finds a plausible alternative answer. An item is retained only when the problem statement, gold answer, expected trajectory, source URLs, and checklist values are mutually consistent, and when the answer is unique.

3.1.0.3 Dataset statistics.

The verified subset consists of 300 questions, where 160 are multi-hop questions (53.3%) and 140 are parallel-branching questions (46.7%). Figure 3 illustrates the category and reasoning-format distribution. The largest category is Entertainment and Media, with 109 questions (36.33%). The second largest category is Transportation, Places, and Regions, with 48 questions (16.00%). The remaining questions cover a range of categories, including education, sports, science, literature, products, history, and public policy.

3.1.0.4 Trajectory-level failure taxonomy.

Each K-BrowseComp item requires an extended information-seeking process: forming useful search queries, retrieving candidate evidence, selecting relevant results, interpreting semi-structured source metadata, combining information across multiple steps or parallel branches, and deriving the final answer. To identify where errors occur, we define a trajectory-level failure-mode taxonomy, summarized in Table 1. The taxonomy is constructed through manual inspection of recurring model errors. In Appendix 10, we provide examples of each failure mode.

3.2 Generating synthetic web-browsing problems with browsing agents↩︎

3.2.0.1 Motivation

While constructing K-BrowseComp-Verified, we identified which Korean websites web browsing agents fail to retrieve information from, as well as the behaviors they exhibit when they fail. However, asking humans to construct such problems is costly and time-consuming. Inspired by recent approaches to generating synthetic problems [24]–[26], we therefore ask whether we can leverage this failure taxonomy, together with the identified websites, to have an AI agent create equally challenging web browsing problems.

3.2.0.2 Generation pipeline.

To generate synthetic problems, K-BrowseComp-Verified provides both seed pages and a failure taxonomy. Its gold source pages are used to collect related Korean web pages, while observed baseline failures define the nine failure modes in Table 1. For generation, we target F1–F8 and exclude F0, since malformed or incomplete trajectories are not a reusable content-level failure for question construction. Each seed page is provided to Claude Code (claude-opus-4.7 at maximum effort) and is tasked with drafting a single adversarial question. Specifically, it opens the web page and constructs a Korean question backwards (i.e., as opposed to opening a web page after reading a question) so that reaching it requires multi-hop or parallel-constraint retrieval, as in the verified set. The question withholds the answer and any paraphrase of it, the source URL and domain, and the page’s most identifying entity name, and targets a designated failure mode. The agent refines the question over four iterations (draft \(\rightarrow\) test \(\rightarrow\) revise) until it satisfies the filtering rules.

3.2.0.3 Filtering rules.

Each candidate question passes three sequential filters, and failing any one returns the agent to revision. The first tests searchability: the agent issues several web queries and checks whether the gold answer already appears among the results; if so, the question is too easy and is rewritten to target a less exposed detail. The second tests well-formedness: a reference solver receives the full source page together with the question and must recover the answer, confirming that it is uniquely and faithfully extractable from the page rather than ambiguous or absent. The third tests adversarial difficulty: a search-only solver, granted web search but no direct access to the page, attempts the question, and the candidate is retained only if every target model (gpt-5.4-mini and gemini-3-flash-preview) fails, where failure includes both an incorrect answer and an explicit abstention. Acceptance further requires that each model’s failure be attributable to one of the eight failure modes, F1–F8; the agent labels the realized mode honestly, re-assigning it from the intended one when the observed failure differs, for instance recording an unreachable-page failure as F2 rather than disguising it as a parsing failure.

3.2.0.4 Dataset statistics.

Among the 268 candidate questions we generated, 100 problems were not filtered out (37.3% yield). Among the accepted questions, 55, 32, 10, and 3 are obtained on the first through fourth iteration, respectively. Each accepted question is labeled with the failure mode(s) it elicits, distributed as F4 (59), F7 (21), F2 (14), F8 (13), F3 (13), F6 (8), F5 (6), and F1 (1), with semi-structured parsing and constraint accumulation being the most frequently exploited weaknesses. Rejections (i.e., problems that were filtered out) are dominated by seed pages that are already fully indexed: for the majority of the 168 rejected items the gold answer surfaces directly in search (i.e., searchability). Also, 66 are solvable by either gemini-3-flash-preview or gpt-5.4-mini. In the subsequent sections, we use the 100 problems that were not filtered.

3.2.0.5 Synthetic split diagnostics.

We analyze the generated problems as a complementary stress split. The synthetic split preserves the reasoning-format balance of K-BrowseComp-Verified, with 53.0% multi-hop and 47.0% parallel examples. Its main differences appear in category composition and question length. The share of Entertainment and Media drops from 36.3% to 9.0%, while the share of Science, IT, and Academia rises from 6.7% to 33.0%. The questions are also longer, with mean length increasing from 174.46 to 248.40 characters. We further embed all questions with a multilingual sentence-transformer model and test whether the two splits can be separated from question text alone. A simple classifier separates the two splits well, with an ROC AUC of 0.8873. These results suggest that the synthetic split preserves the reasoning format, but differs in domain and surface profile. We therefore report it separately from K-BrowseComp-Verified in Section 5.2 and provide further analyses in Appendix 11.

4 Experimental Setup↩︎

4.0.0.1 Baselines.

We evaluate proprietary models and open-weight models on K-BrowseComp-Verified. The proprietary baselines include GPT-5.5, GPT-5.4-mini, and Gemini-3.1-Flash-Lite [27], [28]. The open-weight baselines include DeepSeek-V4-Pro, GLM-5.1, Qwen3.6-35B-A3B, and Gemma-4-31B-it [2], [29]–[31]. We also evaluate Korean open-weight models, including K-EXAONE-236B-A23B, A.X-4.0, HyperCLOVAX-SEED-Think-32B, and Kanana-2-30B-A3B-Thinking-2601 [6], [32]–[34].

4.0.0.2 Evaluation protocol.

We build our browsing-agent evaluation on the search_evals framework [35]. We use the deep-research agent and set Perplexity Search as the search backend for evaluation. For comparability across models, each agent is given a budget of 10 search calls per question, which is the default setting of the search_evals framework. Each model is evaluated once on each of the 300 questions in the verified subset. We use GPT-5.4-mini to extract the final answer from each model response and match it against the gold answer, following BrowseComp [13]. We report the single-run accuracy, which corresponds to pass@1 in this setting.

3.2pt

5 Experimental Results↩︎

5.1 Main results↩︎

Table 2 reports the performance of closed and open models on K-BrowseComp-Verified. GPT-5.5 achieves the highest accuracy, with 45.67%. Among the remaining models, GPT-5.4-mini and GLM-5.1 both reach 30.67%, while DeepSeek-V4-Pro obtains 30.00%. These results show that the benchmark remains challenging even for recent high-performing models.

Smaller open-weight models show a wide performance range. Gemma-4-31B-IT reaches 23.33%, outperforming Qwen3.6-35B-A3B at 12.00%. Korean open models score lower in this setting, with K-EXAONE-236B-A23B at 10.33%, A.X-4.0 at 5.33%, and HyperCLOVAX-SEED-Think-32B at 2.33%.

Following BrowseComp and BrowseComp-ZH [13], [18], we report expected calibration error over five equal-width confidence bins, computed as the weighted average gap between mean confidence and empirical accuracy. Lower values indicate better alignment between confidence and correctness.

5.2 Results on Synthetic Splits↩︎

The last column of Table 2 reports performance on the 100-question synthetic split. We evaluate this split using the same browsing-agent harness and grading protocol as in Section 4. Accuracy remains low across all evaluated models, ranging from 0.00% to 26.00%, and no model exceeds 30.00% on this split. Even the strongest models on the verified subset remain substantially below their verified-subset accuracy, with GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1 scoring 26.00%, 22.00%, and 19.00%, respectively. These results suggest that the synthetic split exposes persistent browsing failures under the same evaluation protocol. Because failure-based filtering is part of the construction process, we use the synthetic split as a diagnostic stress test and do not pool it with the verified score. We treat the GPT-5.4-mini result separately here because this model was used during adversarial filtering.

6 Analysis↩︎

6.1 Trajectory-Level Failure Patterns↩︎

Whereas Table 1 catalogs failure modes at the level of individual reasoning or search steps, here we examine how a subset of these modes manifest as recurring trajectory-level dynamics: patterns in which step-level errors compose across many steps after partially relevant evidence has already been retrieved. Many errors arise after models have already found partially relevant evidence, and these errors often come from losing track of candidates, constraints, and evidence across multiple steps. An illustration is shown in Figure 4.

6.1.0.1 F5+F7: Search-result selection & constraint-tracking failure (candidate capture).

This failure occurs when the model commits to a plausible entity before all upstream constraints have been verified. After this early commitment, later searches become confirmatory: the model searches within the local evidence space of the selected candidate and treats unresolved constraints as satisfied. The final answer can appear locally supported, while still violating the full question. In the left panel of Figure 4, the model should first identify the target drama before searching its OST lyrics, but instead starts from OST and lyric cues and anchors on My Lawyer, Mr. Jo. It then returns Trapped Heart, an OST Part 4 song from the wrong drama, illustrating how premature candidate commitment can mislead. See Appendix 10.1 for additional examples.

6.1.0.2 F7: Constraint-tracking failure (unmerged evidence branches).

In this pattern, the model searches for several relevant constraints, but each query creates a separate evidence branch. The branches are never converted into filters over a shared candidate set. The trajectory may appear systematic because it touches many clues, but the candidate table remains inconsistent across steps.

The middle panel of Figure 4 shows this pattern. The question asks for a K-pop group satisfying parallel constraints on pre-debut competition history, member nationality, and album release status. The gold state is the intersection of these constraints, which yields Ladies’ Code. However, the model searches these clues separately, applies them to different artists or groups, and returns Winner, even though it violates the album-release constraint. This error shows that the retrieved clues were not used to derive the answer consistently.

6.1.0.3 F3: Cross-source hopping failure (misbound evidence chains).

In this pattern, the model follows a plausible search sequence, but binds an intermediate result to the wrong role. This is most visible when the question changes entity type across steps. If the region, institution, song, and lyric roles are not preserved, a high-visibility entity can replace the intended target. The right panel of Figure 4 illustrates this failure. The gold path must preserve a chain from the drama to the actress’s award year, the political attack region, a university in that region, and the first phrase of verse 2 of its school song. The model identifies part of the drama and actress branch, but leaves the event region unresolved. It then folds the award, event, university, and cheer-song clues into a broad query, shifts to a visible university-song branch, and returns a plausible phrase from the wrong institution.

6.1.0.4 F0: Incomplete trajectory or malformed output (answer finalization).

Certain models begin a reasonable browsing trajectory but do not stabilize it into a well-formed final answer. This includes incomplete trajectories, unstable intermediate conclusions, excessive uncertainty, or final responses that do not follow the expected answer format. These cases show that browsing performance depends not only on retrieval and reasoning, but also on controlled finalization. The model must know when the evidence is sufficient, when more search is needed, and how to produce a concise exact answer.

6.1.0.5 Takeaway.

Together, these patterns show that many browsing failures emerge after the required evidence has been retrieved, and that F0, F3, F5, and F7 are the dominant failure modes once retrieval has succeeded. This suggests that progress on K-BrowseComp depends on stronger mechanisms for maintaining candidates, constraints, role bindings, and final-answer state across multiple turns.

2.5pt

6.2 Search Effort and Failure Persistence↩︎

One might question if the reason behind low performances is due to the insufficient budget allocated for interacting with the search API. In this subsection, we conduct an analysis that shows this is not the case. Table 3 provides a complementary trajectory-level view by comparing search-call usage on correct and incorrect trials. For most models, incorrect trials use more search calls than correct trials and often approach the 10-call budget. This pattern is especially clear for GPT-5.5 (7.08 vs. 9.30), DeepSeek-V4-Pro (7.47 vs. 9.80), Gemma-4-31B-it (5.20 vs. 8.10), and Qwen3.6-35B-A3B (8.11 vs. 9.69).

The higher search usage on failed trials suggests that many errors are not simply due to insufficient retrieval. Models fail even after evidence has been retrieved, because they do not reliably merge constraints, preserve entity roles, or verify candidates before committing to an answer. A.X-4.0 is an exception, using few searches on both correct and incorrect trials, which suggests shallow search or premature stopping. HyperCLOVAX-SEED-Think-32B shows little separation between correct and incorrect trials, consistent with failures in trajectory completion or final-answer stabilization. Overall, search volume is a weak predictor of success. A stronger signal is whether the model can maintain candidate, constraint, and role state across the searches it performs.

6.3 Failures in Korean Open-Weight Models↩︎

A benchmark should not merely report low scores but inform development. The Korean open-weight models we evaluate underperform global counterparts by a wide margin (Table 2); we provide a model-level diagnosis to guide future iterations.

6.3.0.1 A.X-4.0.

Built on Qwen2.5 [36], the model often commits to plausible Korean web snippets before all constraints are verified, consistent with its unusually low search usage in Table 3. Korean-targeted post-training has not, under this evaluation setting, translated into trajectory-level state maintenance (Appendix 12).

6.3.0.2 K-EXAONE-236B-A23B.

Despite its scale, the model frequently loses the intermediate entity chain after relevant retrieval, with later queries drifting from the target entity (Appendix 12). Parameter count alone does not appear to substitute for long-horizon state tracking.

6.3.0.3 HyperCLOVAX-SEED-Think-32B.

The model initiates reasonable browsing trajectories but rarely converges to a well-formed final answer. Its search counts on correct and incorrect trials are nearly identical (Table 3), suggesting the bottleneck lies in finalization, not retrieval effort.

6.3.0.4 Kanana-2-30B-A3B-Thinking-2601.

The model does not produce completed browsing runs under the search\(\_\)eval harness, often emitting tool-call objects that violate the protocol. We read this as incomplete adaptation to the tool-use paradigm instead of a content-level limitation.

6.3.0.5 Takeaway.

Across models, Korean open-weight LLMs show no clear advantage on K-BrowseComp despite its Korean-centric design, and the gap to closed frontier APIs remains substantial. The bottlenecks we observe are stage-specific (e.g., tool-call protocol, post-retrieval state, and answer finalization) rather than reducible to a single cause. We position K-BrowseComp as a shared diagnostic target to encourage prioritization of browsing-agent capabilities alongside static-benchmark performance.

6.4 Trajectory Diagnostics on the Synthetic Split↩︎

A trajectory review helps explain why the synthetic split remains difficult. As reported in Section 3.2, accepted synthetic questions concentrate on modes that the pipeline can instantiate reliably, especially semi-structured parsing (F4) and constraint tracking (F7). The completed runs show that these targeted operations also reappear as solving failures.

Models often reach the right source family or intermediate entity, but fail at the page-level or candidate-level state needed for the final answer. In a repository metadata question (Appendix 13 Figure 14), the model reaches the KOPRI repository source family, but extracts a nearby incorrect file-size value instead of the target PDF metadata field. In a KBO record question (Appendix 13 Figure 15), the model identifies 안우진 (An Woo-jin) from the 2022 Kiwoom Heroes pitching clues, but does not maintain a stable table of his 2026 appearances and opponent AVG values before selecting the final date. These cases show that the main bottleneck is not broad retrieval. The synthetic split instead demonstrates whether models can preserve the correct source or candidate state long enough to extract an exact value or enforce the final comparison.

7 Conclusion↩︎

We introduce K-BrowseComp, a Korean web-browsing agent benchmark with a 300-question human-verified subset and a 100-question synthetic split. Even strong frontier models achieve low scores, and Korean open-weight models lag substantially behind global counterparts. The synthetic split is similarly difficult, indicating that failure-mode-targeted generation, paired with verification and filtering, produces useful diagnostic items. Our analysis shows that many failures occur after models retrieve relevant Korean web evidence: they often fail to maintain candidates, constraints, source pointers, or final-answer state across the trajectory. Progress on Korean browsing therefore requires stronger trajectory-level state maintenance, not only broader language coverage or larger model scale. We release K-BrowseComp as a target for building reliable web-browsing agents that function for Korean web environments.

Limitations↩︎

While K-BrowseComp provides a first step toward evaluating Korean web-browsing agents, several limitations remain. First, the verified set is still modest in scale and uneven in domain coverage: it contains 300 human-written items, with a substantial share concentrated in entertainment/media and place-related queries. Thus, the benchmark may not fully represent the breadth of Korean web-search use cases. Second, model performance is measured under a single browsing harness, search backend, search-call budget, and pass@1 setting, which may conflate model capability with retrieval coverage, tool-interface reliability, and evaluation protocol choices. Third, although the synthetic split is useful as a diagnostic stress test, it differs from the verified set in surface form and domain composition. Finally, because web evidence and search rankings change over time, continued revalidation will be necessary to preserve answer uniqueness, accessibility, and reproducibility.

8 Dataset Construction↩︎

8.0.0.1 Contributor instructions.

Contributors were given a written guide that specified the goal, item format, and exclusion rules for K-BrowseComp-Verified. Figure 5 shows an excerpt of the instructions used during question construction. They were asked to write difficult fact-seeking questions that require web browsing and evidence synthesis. Each item had to be grounded in Korean contexts and had to have a short answer supported by public web evidence. Contributors were encouraged to test draft questions with web-enabled models to check difficulty, but they were instructed not to use LLMs to generate or revise the questions.

8.0.0.2 Korean grounding.

Each question had to contain Korean-specific information in the question, the answer, or the intended evidence path. This included Korean institutions, local places, public transportation, educational materials, cultural artifacts, media, historical facts, or expressions commonly used in Korea. The grounding requirement was intended to test browsing ability in Korean information environments, including sources whose structure, terminology, and search conventions differ from English web resources.

8.0.0.3 Reasoning formats.

Each question followed one of two formats. In multi-hop questions, the solver first identifies an intermediate entity or value, then uses it to retrieve later evidence and reach the final answer. In parallel-branching questions, the solver evaluates several independent constraints and identifies the unique entity satisfying all of them. Multi-hop items required at least four reasoning steps. Parallel-branching items required at least four independent constraints. Contributors were asked to balance the two formats across their submitted items.

8.0.0.4 Difficulty criteria.

Questions were designed to be difficult to answer through direct keyword search. Contributors typically began with a verified target fact, selected attributes with a large search space, and then wrote an inverted question that required recovering the target through search. The answer and intermediate steps had to be easy to verify once the correct path was found. At the same time, the answer keyword should not be the title of a standalone document that reveals the answer without following the intended reasoning path.

8.0.0.5 Evidence requirements.

All required evidence had to be available from public web pages. Sources requiring login, payment, private databases, or membership access were not allowed. Items were also excluded when solving required downloading or interpreting PDFs, spreadsheets, images, or other non-textual artifacts. Contributors were discouraged from relying on a single web platform as the sole basis for an item. When possible, intermediate facts were supported by multiple independent sources.

8.0.0.6 Answer constraints.

Each item had to have a unique final answer under the stated conditions. Contributors were asked to add constraints when another answer could plausibly satisfy the question. Items involving time-sensitive facts had to include a reference date in the question. During review, questions were checked for ambiguous wording, unstable clues, and plausible alternative answers.

8.0.0.7 Submission format.

Each item is submitted in a structured JSON format containing the problem statement, gold answer, expected solving trajectory, source URLs, checklist values for key intermediate slots, Korean-specific keywords, and a short rationale. The checklist is used to check whether the evidence path is valid in our validation procedure. The rationale explained why the item required Korean-context browsing and why direct search was unlikely to be sufficient.

8.0.0.8 Revision and acceptance.

Submitted items were manually reviewed for evidence accessibility, source consistency, wording clarity, answer uniqueness, and temporal stability. During this review, we also checked submissions for private personal information, sensitive personal data, and offensive content, and revised or removed items that failed this screen. Items were returned to the original contributor when the evidence was inaccessible, insufficient, inconsistent, or dependent on excluded source types. Items were also revised or removed when a baseline web-enabled model produced a different concrete answer that was judged to be a plausible alternative. An item was accepted only after the problem statement, gold answer, evidence path, source URLs, and checklist values were mutually consistent.

8.0.0.9 Recruitment and Payment

Non-researcher participants of this study were recruited via a social media post and the remaining participants were authors of this paper. Non-researcher participants were paid ₩100,000 (approximately USD 60–70) per 10–15 instances crafted, which corresponds to roughly 4 hours of work and exceeds South Korea’s 2025 minimum hourly wage of ₩10,030. Author-participants did not receive separate compensation.

8.0.0.10 Data Consent

All participants were aware that the annotations would be part of constructing the K-BrowseComp benchmark.

8.0.0.11 Artifact release and licensing.

We will release K-BrowseComp, including the verified and synthetic questions, source URLs, expected trajectories, checklist values, and evaluation code, to support research and evaluation. The K-BrowseComp dataset and evaluation code will both be released under the MIT License. The license applies only to benchmark items, metadata, and code created by the authors and contributors; it does not grant rights to third-party web pages linked as evidence, whose contents remain governed by their original providers’ terms. We provide source URLs for verification but do not redistribute the full contents of linked web pages.

9 Dataset Construction Details↩︎

K-BrowseComp-Verified consists of 300 human-written Korean browsing problems. The problems were written by Korean-speaking contributors, including both researchers and non-researchers, who were instructed to create questions that require web browsing. Each contributor submitted items in a structured JSON format containing the problem statement, gold answer, expected reasoning chain, source URLs, intermediate checklist values, and Korean-specific keywords. An example submission format is shown in Figure 6.

9.1 Category-wise Performance Analysis↩︎

Table 4 reports model accuracy across the major categories of K-BrowseComp-Verified. The categories are ordered by the number of questions. Performance varies substantially across domains, even for the strongest frontier models.

Across most models, Entertainment & Media and Sports & Games are among the highest-performing categories. These questions often involve public figures, entertainment programs, music releases, sports records, or widely discussed media content that are relatively well represented on the public web. By contrast, categories such as Science, IT & Academia, Products, Brands & Beauty, and Education, Colleges & Exams remain difficult for nearly all models. These questions frequently require linking sparse Korean entities across multiple sources, resolving institutional terminology, or navigating semi-structured local web pages.

The gap between global frontier models and Korean-specialized models is also substantial. GPT-5.5 achieves the strongest overall performance at 45.67%, while Korean-specialized models such as K-EXAONE-236B-A23B and A.X-4.0 remain below 11%. This gap persists even in categories that contain strongly Korean-specific cultural or institutional information. The results suggest that current Korean-specialized LLMs still struggle with long-horizon retrieval, multi-step evidence integration, and constraint tracking in realistic Korean web environments.

Several categories additionally exhibit high variance due to sparse but difficult entity structures. For example, History, Culture & Politics shows relatively high scores for some models despite its difficulty because the category contains only ten questions and several items involve recoverable public records once the correct evidence chain is identified. In contrast, Science, IT & Academia consistently remains difficult because many questions require cross-source reasoning over researcher profiles, academic affiliations, institutional announcements, and Korean-language metadata.

4.2pt

10 Failure Modes Details↩︎

This appendix provides additional details for the trajectory-level failure patterns discussed in Section 6.1. The main text identifies three representative patterns: candidate capture, unmerged evidence branches, and misbound evidence chains. Here, we describe how these patterns appear in concrete browsing trajectories and provide additional examples in Figure 11. The examples show that many failures occur after partial retrieval has already succeeded, when the model fails to preserve candidate, constraint, or role state across the trajectory.

10.1 Candidate Capture Failure Details↩︎

Candidate capture occurs when the model commits to a plausible candidate before the upstream constraints have been fully verified. After this early commitment, later searches are conducted inside the local evidence space of that candidate. The final answer may therefore appear supported by retrieved evidence, but the candidate itself is not licensed by the full question.

Figure 11 (b) shows an answer-side variant of this pattern. The question requires preserving the dependency chain from company to brand, H&B store, and award product. The model searches Olive Young award products before fixing the upstream company and brand. This search order leads the trajectory toward a plausible CLIO/Goodal/Peripera branch, and the model then collects award products from that branch. These products are locally plausible, but they are not supported by the company–brand constraint required by the question. The resulting output is therefore a set of plausible products, not the unique gold answer, 어성초 흔적 에센스 패드 (Abib Heartleaf Spot Pad Calming Touch).

11 Synthetic Split Diagnostics↩︎

We conduct additional diagnostics to characterize the 100 synthetic problems. The goal is to test whether the synthetic split follows the same distribution as K-BrowseComp-Verified, and to identify which factors explain any observed separation.

We first compare the two splits using question embeddings. All 400 questions are embedded with sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, producing 384-dimensional vectors. We then train a cross-validated domain classifier whose task is to predict whether a question comes from K-BrowseComp-Verified or from the synthetic split. This classifier reaches ROC AUC 0.8873 \(\pm\) 0.0281. This means that the two splits can be separated from question text alone.

We also run a maximum mean discrepancy test in PCA-reduced embedding space. Maximum mean discrepancy measures how far apart two sets of points are as distributions under a kernel. In this analysis, the two point sets are the verified question embeddings and the synthetic question embeddings. The permutation test randomly shuffles the split labels and recomputes the MMD statistic many times. The resulting \(p\)-value measures how often a random label split produces an MMD value at least as large as the observed one. We obtain RBF MMD\(^2=0.031967\) and \(p=0.004975\), which indicates a statistically clear embedding-level shift.

To interpret the source of this separation, we train additional classifiers using only simple metadata. Question length alone gives ROC AUC 0.7885 \(\pm\) 0.0558. Category alone gives ROC AUC 0.7773 \(\pm\) 0.0342. The multi-hop or parallel label alone gives ROC AUC 0.4250 \(\pm\) 0.0197. Using length, category, and type together gives ROC AUC 0.8277 \(\pm\) 0.0275. These results show that much of the embedding separability comes from longer synthetic questions and category rebalancing. The reasoning-format label contributes little, consistent with the near-identical multi-hop and parallel proportions across the two splits.

12 Trajectory-Level Failure Diagnostics for Korean Open-Weight Models↩︎

This appendix expands the analysis of Korean open-weight models in Section 6.1. The main results show that their low performance does not come from a single failure type. Instead, the trajectories reveal different bottlenecks in the browsing loop: some models fail to form a useful candidate state after retrieval, while others retrieve relevant evidence but lose the dependency chain needed for later search and final answer selection.

12.0.0.1 Shallow evidence control in A.X-4.0.

Figure 12 illustrates a failure in which A.X-4.0 reaches relevant Korean historical evidence, but does not convert the retrieved snippets into a maintained candidate ledger. The question requires one historical person to satisfy several constraints at once: civil-service examination records, surname-and-clan evidence linked to a modern athlete, inspection-related office history, and a memorial academy in the person’s hometown. The model issues a single broad query that concatenates most of these clues. This search surfaces locally relevant names, including Jeong Hwang, Min Yeong-hwan, and Kim Jong-seo, but the model evaluates them as separate snippet-level candidates. The same-origin modern-athlete clue is never used as a binding constraint, so the trajectory does not keep track of which historical candidates remain valid after each check. The final I don’t know response therefore occurs after partial retrieval has succeeded. The bottleneck is state formation: the model sees relevant evidence, but fails to organize it into a shared candidate table that can support elimination and verification.

12.0.0.2 Cross-source chain drift in K-EXAONE-236B-A23B.

Figure 13 shows a later-stage failure. Unlike the A.X-4.0 case, K-EXAONE-236B-A23B initially reaches the right web region. Its first search retrieves the KakaoTalk free-emoticon event and a plausible list of distributed character emoticons. At this point, the model should freeze the event-side state, identify the smaller-animal character, and then search for the creator and official YouTube channel metadata. The trajectory instead moves to creator search before stabilizing the target emoticon. Subsequent queries contain misspelled character names, malformed creator strings, and unrelated YouTube terms. Although the model later finds a plausible creator-like entity and a channel date, it cannot verify the event–emoticon–creator–channel link. This failure shows that successful initial retrieval is not enough. The intermediate entity must remain stable across source boundaries; otherwise, later searches may be locally plausible but no longer licensed by the original question.

12.0.0.3 Trajectory completion and protocol stability.

The remaining Korean open-weight models show different bottlenecks. HyperCLOVAX-SEED-Think-32B often begins a reasonable browsing process, but its trajectories do not reliably converge to an exact final answer. These failures are not limited to one content type. They include unresolved constraint checks, unstable candidate states, and final responses that remain uncertain or under-specified. Kanana-2-30B-A3B-Thinking-2601 shows an earlier bottleneck in the evaluation loop. Because it frequently fails to emit valid tool-call objects under our browsing harness, many runs do not become completed search trajectories. This indicates that tool-use protocol reliability remains a prerequisite for evaluating higher-level browsing skills such as evidence selection and cross-source reasoning.

12.0.0.4 Summary.

These examples clarify why Korean open-weight models score poorly on K-BrowseComp. The issue is not only whether a model can retrieve Korean web pages. A successful browsing agent must convert retrieved pages into a persistent trajectory state: candidate ledgers, constraint checks, entity-role bindings, and final-answer commitments. The Korean open-weight models fail at different points in this process. A.X-4.0 often stops at shallow snippet-level evidence, K-EXAONE-236B-A23B loses dependency chains after relevant retrieval, HyperCLOVAX-SEED-Think-32B struggles to stabilize final answers, and Kanana-2-30B-A3B-Thinking-2601 is limited by tool-call protocol failures.

13 Synthetic Split Trajectory Diagnostics↩︎

This appendix expands the trajectory-level analysis of the synthetic split. The synthetic split is constructed to target failure modes that can be instantiated reliably by the generation pipeline, with semi-structured parsing (F4) and constraint tracking (F7) appearing most frequently among accepted items. The examples in Figures 14 and 15 show how these targeted modes also appear in completed model trajectories. In both cases, the model does not fail because it searches in an unrelated region of the web. It reaches the correct source family or recovers the main intermediate entity, but loses the page-level or candidate-level state needed to produce the exact final answer.

13.0.0.1 Metadata-field extraction after successful source localization.

Figure 14 illustrates a semi-structured parsing failure in a repository metadata question. The question asks for the file size displayed for a PDF attached to a KOPRI repository record. The required state is narrow: the solver must preserve the target repository item and read the file-size field attached to that specific PDF. The model first retrieves the correct topical context, namely the 2015 KOPRI press release about the Araon Antarctic mid-ocean ridge study. It also reaches the relevant repository neighborhood. However, the trajectory does not keep the target item fixed when reading the file metadata. Because the search results contain nearby PDF records with similar filenames and related institutional context, the model selects a neighboring file-size value, 1.56 MB, while the gold value is 698.85 kB.

This case shows a page-level state failure. The model has enough evidence to enter the right source family, but it does not bind the final extraction step to the correct repository record. The error is therefore not a failure of broad topical retrieval. It is a failure to preserve an exact source pointer and extract the requested field from that source. This is the kind of operation that appears simple after the target page is known, but remains difficult for browsing agents because visually or textually adjacent records create plausible distractors.

13.0.0.2 Record-level ledger failure after entity recovery.

Figure 15 shows a constraint-tracking failure in a KBO record question. The first part of the question identifies the target player through 2022 pitching clues: a Kiwoom Heroes right-handed pitcher who led the league in ERA, strikeouts, and innings. The model correctly recovers 안우진 (An Woo-jin) as the target player. It then searches for his 2026 appearance records and reaches relevant baseball-record pages and reports about specific appearances. The remaining task is not entity identification, but comparison over a stable game-level ledger. The solver must maintain rows containing the date, appearance, and opponent AVG values for the player’s 2026 games, then select the unique maximum.

The model does not maintain this ledger. Instead, it selects a plausible later appearance date from the retrieved evidence without verifying the maximum opponent AVG condition across all appearances. Its final answer is 05.14, while the gold answer is 05.08. This demonstrates a candidate-level state failure. The model has already found the central entity and entered the right record space, but the final comparison requires a structured table over all relevant games. Without such a table, the trajectory can remain locally plausible while failing the global constraint.

13.0.0.3 Relation to the synthetic generation pipeline.

These two cases clarify what the synthetic split contributes beyond the human-written verified set. The generation pipeline does not merely produce questions with obscure answers. It creates items in which the answer is easy to verify once the correct state is fixed, but difficult to obtain unless the model preserves that state across the trajectory. For F4-style items, the key state is the exact page item and field to be read. For F7-style items, the key state is a candidate or record ledger that must survive later filtering and comparison. The accepted synthetic questions therefore stress operations that are common in real browsing: reading institutional metadata, selecting among neighboring records, maintaining candidate tables, and enforcing final constraints after the main entity has already been found.

13.0.0.4 Takeaway.

The synthetic split should be interpreted as a complementary diagnostic stress split, not as a replacement for K-BrowseComp-VERIFIED. Its category and surface profile differ from the verified set, but its trajectory failures reveal the same broader weakness: models often retrieve relevant Korean web evidence without converting it into a persistent state for exact extraction or final selection. Figures 14 and 15 show this at two granularities. The repository example fails at the source-field level, while the KBO example fails at the candidate-ledger level. Together, they explain why accuracy on the synthetic split remains low even when models reach relevant sources.

14 AI assistants in research/writing↩︎

In 3.2, we used claude-code to generate the synthetic subset of K-BrowseComp and in 4, we include multiple LLMs as our baselines in K-BrowseComp. For writing, we mainly used AI assistants for revising grammar and fluency.

15 Computational experiments↩︎

15.0.0.1 Model Size

Among the open-weight baselines we evaluate, several disclose their parameter counts. Three are dense models: Gemma-4-31B-it (31B parameters), HCX-SEED-Think-32B (32B parameters), and A.X-4.0 (72B parameters; continual pre-training of Qwen2.5-72B on Korean data). Five are Mixture-of-Experts (MoE) models, for which we report total / active parameter counts: Qwen3.6-35B-A3B (35B / 3B), Kanana-2-30B-A3B-Think (30B / 3B), K-EXAONE-236B-A23B (236B / 23B), GLM-5.1 (744B / 40B), and DeepSeek-V4-Pro (1.6T / 49B). For the closed-weight baselines (GPT-5.5, GPT-5.4-mini, and Gemini-3.1-Flash-Lite), parameter counts are not publicly disclosed by the providers.

15.0.0.2 Compute Budget

All experiments ran on openrouter API calls except for Korean models that were not supported on the platform. The total API cost for this project was approximately 320 USD.

None

None

None

None

References↩︎

---

1. https://github.com/prometheus-eval/K-BrowseComp↩︎

2.   Corresponding author↩︎