1 Introduction↩︎

Bugs are inevitable in software systems. When a bug is logged, various tasks take place, such as triaging, debugging, root cause analysis, and finally fixing. Across all these tasks, bug localization often remains the first step [1]–[3]. Bug localization (BL) is the process of identifying faulty code regions within a software repository. In large and evolving repositories, determining which files are most likely responsible for a reported issue can require substantial effort [3], [4]. Therefore, effective localization plays a broader role in software engineering by reducing the search space for both human and automated maintenance workflows.

Traditional localization approaches often operate at multiple granularities, including file-level, function-level, and statement-level localization [5], [6]. However, recent empirical evidence reveals that file-level localization is the most critical component in hierarchical bug localization pipelines [5]. The study demonstrated that removing file-level localization from a multi-granularity localization framework caused a catastrophic 94% drop in Top-5 accuracy and a 96% reduction in Mean Average Precision (MAP) at the statement level. This finding suggests that without accurate file-level localization, even sophisticated localization techniques may fail to identify buggy code.

Take SWE-bench [7] as an example, a widely used Automated Program Repair (APR) benchmark for large language model (LLM)-based coding assistants. SWE-bench averages over 11,000 functions and 168,000 statements—direct statement-level localization is computationally infeasible, particularly for LLM-based systems [5]. File-level localization, by contrast, reduces the search space by several orders of magnitude, enabling efficient downstream localization and repair. Consequently, improving file-level accuracy offers the most impactful path toward advancing end-to-end APR performance. Recent advances in LLMs have significantly improved APR capabilities on real-world bugs [8]–[10]. However, these systems remain fundamentally constrained by localization quality, as inaccurate file selection directly limits repair effectiveness [5], [6]. While bug localization also supports developer-centric activities such as manual debugging and root cause analysis, evaluating its effectiveness in such settings often requires subjective or task-specific studies. We therefore adopt APR as a representative downstream evaluation setting, where localization quality can be assessed objectively through patch correctness and issue resolution.

Consider the motivating example illustrated in Figure 1, which demonstrates how incorrect file localization directly leads to incorrect patch generation in an example (django-10924) from the Django project in the SWE-bench-Lite dataset. The bug report describes the need to allow FilePathField to accept a callable for its path parameter, thereby enabling runtime resolution of file system locations. An existing APR system (Agentless [10]) incorrectly localizes the fault to files such as serializer.py. This misdirection causes the system to generate a patch modifying the migration serializer to handle callable values but misses the core issue persisting in a different file. In contrast, when the correct file __init__.py is identified through more precise localization, the generated patch by the same APR resolves the callable path within the FilePathField.formfield() method, addressing the root cause of the issue.

This example shows that even sophisticated repair systems fail when bug localization provides incorrect context, highlighting the critical need for improved localization strategies that can bridge the semantic gap between natural language bug reports and code-level implementations. While providing more candidate files might seem to mitigate localization errors, this approach may face critical constraints. LLMs are constrained by limited context windows, and processing more files individually incurs substantial computational overhead. Moreover, LLMs are also sensitive to the order of the inputs, where their ability to extract relevant information deteriorates when details appear in the middle or later of lengthy inputs [11], [12]. For such reasons, APRs typically restrict consideration to only the top-k (e.g., Top-3) files to balance context length and computational feasibility [10], [13]. Retrieval-Augmented Generation (RAG) [14] offers a promising direction for addressing these localization challenges. RAG enhances LLMs by grounding their predictions in external knowledge retrieved from code repositories, thereby improving factuality and reducing hallucinations. Recent work has explored RAG for code-related tasks such as code completion and bug localization, demonstrating that retrieval-based approaches can effectively bridge the vocabulary gap between natural language bug descriptions and technical code artifacts [7], [15]–[17]. However, conventional RAG strategies for code often rely on holistic code-text embeddings and naive text-based chunking, which may fail to capture the structural intricacies of code and result in suboptimal retrieval quality [15].

Furthermore, purely embedding-based retrieval approaches, while efficient, lack the reasoning capabilities necessary to assess the functional relevance and appropriateness of retrieved code. Recent advances in LLM-based agents–where LLMs autonomously plan, reason, and act using external tools—present an opportunity to enhance RAG systems with explicit code-level reasoning [18], [19]. Agentic approaches have shown promise in various software engineering tasks by combining iterative reasoning with tool usage, enabling more informed decision-making compared to static retrieval methods [8], [9], [20], [21]. Recent agentic localization methods often employ graph-based repository traversal, which can incur prohibitive costs [6], [8], [21] or require extensive LLM-finetuning [5], [6], [21].

Motivated by these limitations, we propose BLAgent (Bug Localization using Agentic RAG), a novel agentic RAG framework for file-level bug localization that addresses the key limitations of typical RAG pipelines. First, BLAgent encodes a code repository using path-augmented, abstract syntax tree (AST)-aware chunking where source files are split at syntactic boundaries (e.g., functions, classes, logical units) via AST analysis, and each chunk is prepended with its relative file path. This preserves semantic integrity within chunks while creating a direct alignment channel between path-like signals in bug reports (e.g., tracebacks, module references) and the code they describe. Second, it applies dual-perspective query transformation, decomposing each bug report into a structural query that highlights code entities such as function names, modules, and identifiers, and a behavioral query that captures runtime symptoms and expected vs. observed behavior. Issuing both queries independently for retrieval and merging their candidate sets ensures that the retrieval pool covers faults described by lexical reference and by observable behavior. Third, BLAgent performs two-phase agentic reranking: a ReAct-based reasoning agent iteratively inspects structural skeletons of candidate files and assigns relevance scores, followed by a single-shot evidence-anchored reranking step that expands only retriever-highlighted code regions within top-ranked files for a final, implementation-level ranking decision. Unlike graph-based agentic approaches, BLAgent bounds its reasoning to a compact, retrieval-filtered candidate set, avoiding the costly unbounded traversal of existing approaches. We evaluate BLAgent through two research questions:

RQ1. How effective is BLAgent over baselines for bug localization?

BLAgent significantly outperforms both conventional RAG pipelines and state-of-the-art approaches — including complex, graph-based methods — achieving an MRR of 0.900 and a Top-1 accuracy of 86.7% with a closed-source model, surpassing LocAgent [21] at 18 times lower API cost, and an MRR of 0.851 with a Top-1 accuracy of 78.6% using an open-source model.

RQ2. How does such a pipeline influence end-to-end Automated Program Repair (APR)?

When integrated into Agentless [10], an established open-source APR framework, BLAgent improves the overall issue resolution rate by over 20%, demonstrating that more precise file-level localization directly translates into measurable gains in end-to-end patch synthesis.

2 BLAgent Methodology↩︎

We introduce BLAgent, an agentic RAG framework that identifies and ranks source files likely to require modification for a given bug report. As illustrated in Figure 2, BLAgent follows a structured localization pipeline that progressively narrows the repository search space through retrieval and agentic reranking. The framework operates in four major steps:

1. Repository Encoding (Section 2.1). BLAgent first preprocesses the target repository by extracting source files and segmenting them into syntactically meaningful code chunks using AST-aware splitting. Each chunk is augmented with its relative file path before being embedded and stored in a vector database, enabling retrieval that captures both code semantics and repository structure.

2. Query Transformation (Section 2.2). Given a bug report, BLAgent rewrites it into two complementary retrieval-oriented queries. The structural query emphasizes code entities such as modules, classes, functions, and traceback cues, while the behavioral query captures the observed failure and expected behavior.

3. Candidate File Retrieval (Section 2.3). The transformed queries are independently used to retrieve relevant code chunks from the vector database. Chunk-level similarity scores are aggregated into file-level scores, and the top files (ordered by scores) from both query perspectives are merged into a compact candidate set.

4. Agentic Reranking (Section 2.4). Finally, BLAgent reranks the retrieved candidate files through a two-phase agentic process. A ReAct-based agent first inspects file skeletons and assigns relevance scores, after which an evidence-anchored reranking step expands only retriever-highlighted code regions to produce the final ranked file list.

2.1 Repository Encoding↩︎

Given a target repository, we begin with repository-level preprocessing and encoding. All source files (e.g., .py) are extracted, while non-executable files (e.g., licenses, markdowns) are filtered out since they rarely contribute to bug fixing. The selected source files are then segmented into smaller, semantically meaningful units (i.e., chunks) for vector encoding and storage. Instead of relying on naive text-based chunking—where code is divided by arbitrary character or line limits—we employ a code chunking strategy using CodeSplitter¹. This approach leverages the abstract syntax tree (AST) of the target language to produce chunks that align with syntactic and semantic boundaries (e.g., functions, classes, or logical code blocks). Such segmentation aims to preserve the structural and semantic integrity of the chunks.

Semantic Preservation. We define semantic preservation as ensuring that no chunk cuts across an AST construct. For example, splitting a function body mid-expression or separating a decorator from its decorated function. Such fragmentation strips away local dependencies (e.g., parameter bindings, loop invariants, return contracts) that the embedding model requires to produce a meaningful vector representation. As shown in Figure 3, naive text splitting breaks the normalize() function mid-expression, merging incomplete fragments into a single incoherent chunk, whereas AST-aware splitting retains each function as a self-contained logical unit. When a function exceeds the maximum chunk size, the splitter recursively subdivides it at the finest available AST boundaries—such as nested blocks, conditionals, or statement-level nodes—rather than at an arbitrary character offset. Each resulting sub-chunk thus still represents a syntactically complete program fragment, preserving local variable scopes and control-flow context within that segment.

File Path Augmentation. To further enhance contextual precision, each chunk is augmented with its file path (e.g., src/utils/math.py). This augmentation creates a direct alignment channel between bug reports and code chunks. For example, bug reports frequently contain implicit or explicit path-like signals—module references (e.g., django.db.models.fields), traceback entries (e.g., File "django/db/migrations/writer.py"), or package-qualified identifiers (e.g., sklearn.linear_model.LogisticRegressionCV)—that correspond structurally to file system paths. Without path augmentation, these signals are present in the query embedding but absent from the chunk embedding, resulting in a systematic vocabulary mismatch. Prepending the relative file path to each chunk closes this gap: the embedding model jointly encodes content and location; therefore, path-bearing tokens in the bug report directly reinforce similarity with the chunks they describe. This is also useful for identically named entities across different project hierarchies (e.g., module/classA.method_x vs. module/classB.method_x), where content-only embeddings would otherwise assign near-identical similarity scores to structurally unrelated functions. Figure 3 illustrates the difference between naive text chunking and our code-aware, path-augmented chunking. The final chunks are embedded using a pre-trained model to obtain dense vector representations, stored in a vector database for efficient similarity-based retrieval. We adopt dense retrieval, as prior work demonstrates its superiority over traditional sparse methods [22], [23].

However, we did not consider LLM-based semantic chunking, where a language model identifies topically coherent boundaries and segments code accordingly. Recent work suggests that the advantages of semantic chunking are highly task-dependent and frequently fail to justify the added computational cost  [24], whereas AST-based chunking yields self-contained, semantically coherent units that improve retrieval on code-related tasks [15]. Given that our experimental setup spans the full SWE-bench-Lite benchmark—repositories averaging over 11,000 functions each—applying such chunking involving an LLM at indexing time would incur substantial cost that is difficult to justify.

2.2 Query Transformation↩︎

Directly using the raw text of a bug report for retrieval can lead to suboptimal results as lexical mismatches between natural language descriptions and code identifiers, as well as extraneous narrative or contextual information, can distort query embeddings and hinder retrieval of relevant code fragments [25]–[27]. In practice, relevant code entities may be described either structurally (e.g., via class or method names) or behaviorally (e.g., through observed runtime behavior), but these aspects are not always explicitly stated in the surface text.

To address this challenge, we employ a query transformation approach inspired by retrieval-augmented generation (RAG) methods (e.g., rewriting, decomposition, etc.) that improve document-query alignment [25], [28], [29]. The key idea is to reformulate a bug report into retrieval-oriented queries that emphasize complementary perspectives of the fault, bridging the semantic gap between natural language descriptions and the underlying code.

Specifically, we decompose every bug report into two complementary transformations: a structural transformation (\(T_0\)) and a behavioral transformation (\(T_1\)). We chose these two because bug reports typically contain both (1) explicit or implicit references to program entities (e.g., function names, APIs, parameters), and (2) descriptions of the erroneous behavior or execution context. The structural transformation (\(T_0\)) distills the report into its code-related components like identifiers, modules, and relevant structural cues, making it suitable for retrieving code elements that directly match these terms. In contrast, the behavioral transformation (\(T_1\)) reformulates the report, capturing how the system is expected to behave versus how it actually behaves.

Structural Query Transformation \((T_0)\). This transformation focuses on the static, syntactic aspects of the bug report, extracting signals related to identifiable code entities—such as modules, functions, and helper routines. We use prompt \((PT_0)\) in Listing [lst:structural_augmentation_prompt] to transform queries for structural augmentation. We provide one example to the LLM to show how this can be done. In Listing [lst:structural_augmentation_prompt], the example shows that when a specific configuration is set, initializing LogisticRegressionCV(...) ends up in an error. The transformed query also adds more structural cues like __init__, __repr__ since the issue happens during initialization. This is particularly beneficial for dense retrieval where reasoning is not available and its complete dependency on the embedding similarity of query and the documents.

Similarly, we provide one real example shown in Figure 5a, where the transformation focused on lexical and structural components. The transformed query explicitly names the relevant function (slugify), module (django.utils.text), possible helper routine (_slugify), and even points to specific syntactic constructs such as the regular expression and post-processing call (.strip()). This structural transformation of the original bug guides retrieval toward locations in the repository where the implementation logic or post-processing behavior may reside.

Behavioral Query Transformation \((T_1)\). The behavioral query transformation specifically focuses on the runtime behavior described in the bug report, capturing cues about faulty execution, unexpected outputs, or missing functionality. As shown in Figure 5b, this transformation reframes the bug in terms of observed program behavior and expected outcomes, allowing the retriever to better connect textual failure descriptions to the underlying runtime logic.

For example, in the slugify case, the raw bug report describes an undesired output string produced by the function (i.e., it fails to strip leading and trailing hyphens or underscores). The behavioral transformation reformulates this report to explicitly contrast the observed behavior (“returns strings with trailing underscores or dashes”) with the expected outcome (“should strip non-alphanumeric separators”). This reframing highlights the symptom in a way that points the retriever toward code regions responsible for normalization or post-processing. We use prompt \((PT_1)\) (Listing [lst:bhavioral_augmentation_prompt]) to elicit such transformations, guiding the LLM to emphasize runtime mismatches, incorrect outputs, and failed handling of edge cases.

2.3 Candidate File Retrieval↩︎

Let \(T_0\) and \(T_1\) denote the two query transformations (structural and behavioral, respectively). For a transformed query \(q^{(k)}\) with \(k\in{0,1}\), we embed the query using the same encoder applied to repository chunks. Each repository file \(f\) is partitioned into chunks, and each chunk \(c_{i,j}\) (the \(j\)-th chunk of file \(f_i\)) has an embedding \(\mathbf{c}_{i,j}\) produced by the same encoder. Similar code chunks are then retrieved from the database using nearest neighbour search [30]. As a code file typically contains multiple functions or logical units while a bug is most often localized to a single unit, we aggregate chunk-level similarity scores for each file by taking the maximum over its chunks. The file-level similarity under query transformation \(T_k\) is therefore \[\mathrm{sim}\big(q^{(k)}, f_i\big) = S^{(k)}*i = \max*{j \in C(f_i)} s^{(k)}_{i,j},\] where \(C(f_i)\) is the index set of chunks in file \(f_i\). Taking the maximum highlights the single region of a file that best matches the query and avoids diluting strong local matches by averaging with unrelated code.

For each transformation \((T_k)\) we retrieve a ranked list of files ordered by decreasing file-level similarity: \[{FT_k} = \big[ f^{(k)}*1, f^{(k)}*2, \dots, f^{(k)}*{n_k} \big], \qquad \mathrm{sim}\big(q^{(k)}, f^{(k)}*i\big) \ge \mathrm{sim}\big(q^{(k)}, f^{(k)}*{i+1}\big).\] To form the agent’s candidate input we take the top-(m) files from each list, concatenate these two prefixes, and remove duplicates while preserving first-occurrence order: \[{F_\text{candidate}} = \mathrm{unique}\Big( {FT_0}^{[:m]} \oplus {FT_1}^{[:m]} \Big),\] where \((\oplus)\) denotes concatenation, \({FT_k}^{[:m]}\) represents the first \(m\) files from the ranked list \(FT_k\), and \(\mathrm{unique}(\cdot)\) eliminates repeated file entries while keeping the ordering induced by the concatenation. By construction each file appears at most once in \({F}_{\text{candidate}}\). We limit the number of candidates to 15. Although this concatenation order places \(T_0\) candidates first by default, Section 3.5 shows that reversing the order produces negligible difference in localization accuracy, confirming that the ordering is an implementation convenience rather than a structural dependency.

Finally, the agent receives the candidate file set \({F}_{\text{candidate}}\) including the chunks in each candidate together with the original problem statement for downstream reasoning and reranking.

2.4 Agentic Reranking↩︎

Figure 7 illustrates how BLAgent reranks the candidate file set \(F_{\text{candidate}}\) obtained from the retrieval stage. The reranking proceeds in two phases. In Phase 1, a reasoning agent inspects structural skeletons of candidate files and assigns relevance scores. In Phase 2, a one-shot LLM inference reranks the top-scored files using pruned, chunk-grounded file contexts to resolve near-ties and verify implementation-level relevance.

Phase 1: Skeleton-Based Agent Scoring (SAS). We instantiate a ReAct-based [31] reasoning agent that iteratively assesses the retrieved candidates. The agent operates through an explicit reasoning loop (Thought \(\rightarrow\) Action \(\rightarrow\) Observation \(\rightarrow\) Answer), where each iteration involves (i) forming a hypothesis about which files are likely relevant, (ii) selectively inspecting file representations through the equipped tool ReadFileSkeleton, and (iii) updating its internal assessment before producing a final scored output. This reasoning pattern allows the agent to dynamically decide which files to examine more closely based on observed evidence, rather than relying solely on static similarity scores.

To facilitate symbolic inspection, each code file is represented as a structural skeleton capturing its classes, function signatures, and docstrings. Rather than processing full source code, the agent operates on these skeletal representations to identify files most relevant to the reported bug. For file-level localization, the most informative cues are typically structural and declarative—function names, signatures, and docstrings—rather than implementation details. This abstraction enables the agent to reason efficiently about code organization while staying within a manageable context budget.

The agent is prompted to assign a relevance score in the range of 0–10 (not relevant to highly relevant) to each candidate file (Listing [lst:reranking_prompt]), reflecting how likely it is to require modification to resolve the reported bug. By iteratively inspecting file skeletons, it refines its confidence estimates and produces a final mapping from file paths to scores.

Phase 2: Evidence-Anchored Reranking (EAR). Skeleton-based scoring provides an efficient, organization-level view of each candidate file. To further refine this ordering with implementation-level evidence—particularly when multiple candidate files implement similar interface or inherit from a common base class, making their skeletons structurally indistinguishable at the signature level—we introduce a one-shot evidence-anchored reranking step over the top-\(M\) Phase 1-scored files. We set \(M=5\) to restrict EAR to a small set of highly plausible candidates, enabling effective joint comparison while keeping the reasoning context bounded. This design choice is further supported by our ablation study (Section 5.5), which shows diminishing returns when increasing the number of candidates. Similarly, for each of the \(M\) files, we construct a pruned file context using the retriever’s top 5 chunks as evidence anchors: we preserve global file structure (imports, class definitions, and method/function signatures) while expanding only the methods that contain retrieved chunks to their full implementations; all other methods are kept as signatures. This exposes implementation-level detail precisely where the retriever considers it most relevant, without loading entire files. The LLM jointly compares these \(M\) pruned contexts in a single inference pass using the prompt defined in Listing [lst:reranking_prompt_phase2] and outputs the final ranked list of file paths, improving ranking precision while controlling context length and inference cost.

This two-phase design is deliberate. Pruned-context reranking over all 15 candidates would significantly inflate context size, increasing cost and risking mid-context attention degradation [11]; moreover, the ReAct agent contributes an independent localization signal through hypothesis-driven structural inspection that no single static inference pass can replicate. Phase 1 thus both reduces the candidate set to a tractable size and enriches the ranking signal that Phase 2 refines.

3 Accuracy Assessment of Bug Localization in BLAgent (RQ1)↩︎

To evaluate how effectively BLAgent localizes bugs at the file level, we decompose RQ1 into the following sub-questions:

• RQ1.1: Can BLAgent outperform state-of-the-art baselines?

  We investigate whether agentic reasoning in a RAG pipeline (i.e., BLAgent) can improve file-level localization beyond current state-of-the-art methods.

• RQ1.2: How do different retrieval and RAG configurations influence file-level localization accuracy?

  We evaluate different chunking and retrieval configurations in a traditional RAG pipeline to identify the most effective setup for file-level localization.

• RQ1.3: Can agentic reranking improve over traditional RAG?

  We examine if agentic reranking can improve over traditional RAG pipelines in localization by using a ReAct-based agent to rerank dense retrieval candidates.

• RQ1.4: Can query transformation further improve localization?

  We investigate whether reformulating bug reports into structured, retrieval-oriented queries further improves localization.

• RQ1.5: Does the choice of LLM variant affect the performance of the agent?

  We analyze whether the underlying LLM influences the agent’s ability to accurately localize faults. Specifically, we examine how different LLM variants—varying in size and type (e.g., closed or open-source) impact the overall effectiveness.

• RQ1.6: Can BLAgent be extended to function-level localization?

  We assess whether the proposed architecture can be generalized to function-level localization.

3.1 Experimental Setup↩︎

3.1.1 Localization Baselines.↩︎

We compare BLAgent against recent dedicated bug localization frameworks and localization components within APR systems. Among dedicated localization frameworks, LocAgent [21] adopts a graph-guided LLM agent framework that represents code repositories as directed heterogeneous graphs encoding structural elements and dependencies, enabling multi-hop reasoning to navigate and localize relevant code entities efficiently. CoSIL [6] introduces an LLM-driven, function-level localization framework that performs a two-phase code graph search — first exploring broadly at the file level through module call graphs, then refining at the function-level with pruning and reflection mechanisms to control search direction and context quality. BugCerberus [5] adopts a hierarchical localization design powered by specialized LLMs operating at the file, function, and statement levels, which progressively narrow down bug locations by leveraging intermediate program representations and contextual cues.

Among end-to-end APR frameworks, Agentless [10] combines both an embedding-based retrieval method and an LLM-based approach that reasons over the project’s file tree to rank relevant files. AutoCodeRover [32] is an agentic framework that utilizes multiple specialized agents alongside AST-based code search APIs and spectrum-based fault localization to iteratively localize buggy code and propose fixes. We also compare the localization accuracy with OpenHands [8], which uses its default CodeAct agent that localizes bugs by iteratively issuing shell and code execution actions to explore the repository.

3.1.2 Evaluation Metrics↩︎

Following prior work [5], [6], [10], we employ standard information retrieval metrics to assess localization performance.

Mean Reciprocal Rank (MRR). MRR measures the average of the reciprocal ranks of the first correctly localized file for each bug report. Formally, let \(Q\) denote the set of bug reports, and \(r_i\) the rank position of the first correctly localized file for query \(i\). Then, \[\mathrm{MRR} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{r_i}.\] A higher MRR indicates that the model ranks the correct file closer to the top.

Top-\(k\) Accuracy. Top-\(k\) Accuracy evaluates the proportion of bug reports for which at least one correct bug location appears within the top-\(k\) ranked predictions. Let \(\mathbb{1}[\cdot]\) denote the indicator function, which equals 1 if the condition is true and 0 otherwise. Then, \[\mathrm{Top\text{-}k} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \mathbb{1}[\text{correct file} \in \text{Top-}k_i].\] We report Top-\(\{1,3,5,10\}\) accuracy to evaluate the competitive methods.

3.1.3 Dataset↩︎

For our experiments, we use the SWE-bench-Lite dataset [7], which comprises 300 real-world bug instances collected from open-source Python projects. Each entry in the dataset contains a detailed bug report, the corresponding commit hash of the repository representing the HEAD of the repository before the issue was fixed, ground truth patch, etc. SWE-bench-Lite is specifically designed for evaluating bug localization and program repair techniques, providing a diverse set of bugs that vary in complexity, type, and affected code components.

3.1.4 Large Language Models.↩︎

We use the GPT-OSS-120B ² model from Ollama as the primary LLM for all agentic reasoning and text-generating components in our framework and the downstream APR framework. The model uses 128 mixture-of-experts (MoE) [33] and has a context length of 128,000. We particularly adopt this model due to its openness and proven reasoning capability compared to other proprietary models like GPT-o4-mini, or o3-mini on different benchmarks. Additionally, we also validate our localization results with SOTA proprietary LLM—Claude-4.6-Sonnet (Claude-4.6).

3.1.5 Embedding Model.↩︎

For text and code representation, we employ the nomic-ai/nomic-embed-text-v1 model³ to generate high-quality embeddings used in retrieval and similarity-based ranking. This model was selected for its strong empirical performance—comparable to proprietary alternatives such as OpenAI’s text-embedding-3-small—and its large context window of 8192 tokens, enabling effective encoding of longer code segments and detailed bug reports.

3.1.6 System Configuration↩︎

All experiments were conducted on a system equipped with 2\(\times\)NVIDIA L40S 48GB GPUs, an Intel Xeon Gold 6442Y\(\times\)​96 processor, and 250 GB RAM.

3.2 RQ1.1 Can BLAgent outperform state-of-the-art baselines?↩︎

3.2.1 Approach.↩︎

To assess the effectiveness of the proposed agentic RAG framework, we conduct a comparative evaluation against several state-of-the-art bug localization approaches discussed in Section 3.1.1. We evaluate BLAgent using GPT-OSS-120B as the base LLM and nomic-embed-text-v1 as the embedding model. Rather than reproducing all baseline results—which would incur prohibitive API costs—we report each baseline at its best published configuration on SWE-bench-Lite. For systems primarily designed as end-to-end APR pipelines that do not explicitly report file-level localization accuracy, we use results from published papers on the same benchmark. AutoCodeRover results are taken from Chang et al. [5], and OpenHands results are taken from Chen et al. [21].

To further disentangle the contribution of the BLAgent architecture from the choice of LLM, we conduct a controlled comparison where both BLAgent and LocAgent (the strongest Top-\(k\) baseline) are run under the same model, Claude-4.6-Sonnet. Due to the substantial API cost of running full benchmark evaluations with proprietary models, we restrict this controlled comparison only to these two systems. We allocate a fixed evaluation budget of $300 for LocAgent using Claude-4.6-Sonnet to control evaluation cost. Under this constraint, LocAgent processes 182 of the 300 instances before exhausting the budget, and results are reported over this subset (see Section 5.1 for detailed cost analysis).

3.2.2 Results↩︎

Table 1 reports the overall file-level localization accuracy for BLAgent (Phase 1 + Phase 2) compared to leading baselines on the SWE-bench-Lite dataset. The results indicate a consistent advantage for agentic RAG across all metrics.

Comparison against dedicated localization methods. BLAgent (GPT-OSS-120B) achieves MRR 0.851 with a Top-1 accuracy of 78.6%, outperforming all baselines at their best published configurations. LocAgent (Claude-3.5) is the closest competitor, achieving Top-1 77.7% and Top-3 91.9%, yet BLAgent surpasses it on Top-1 and Top-3 while relying on an open-source model and without requiring any specialized fine-tuning, pre-built repository dependency graphs, or static analysis infrastructure. These gains are particularly meaningful at lower \(k\) values, as downstream repair systems often restrict consideration to only the top-1 or top-3 ranked files due to cost and context window constraints [10].

Comparison against APR-based localization strategies. BLAgent consistently outperforms the localization components used within existing APR systems under standard ranked retrieval metrics. Agentless [10], combining embedding-based retrieval and LLM reasoning, achieves Top-1 63.0% with GPT-4o. AutoCodeRover [32], which performs localization through agentic code search using AST-based APIs and spectrum-based signals, achieves 55.7% Top-1 accuracy. Among APR systems, OpenHands [8] achieves the strongest file-level localization (76.2% Top-1 accuracy with Claude-3.5), yet remains below BLAgent (78.6%) using an open-source model. However, we note that not all APR systems report localization under comparable evaluation protocols. In particular, LingmaAgent [34] reports a file-localization accuracy of 67.7% based on a patch-level criterion—checking whether the generated patch targets the correct file—rather than ranked retrieval metrics (Top-\(k\), MRR). As such, this result is not directly comparable. Nevertheless, even under a conservative interpretation treating this figure as Top-1 accuracy, it remains below multiple baselines reported in Table 1, including BLAgent.

Controlled comparison under the same LLM. When both systems are run under Claude-4.6-Sonnet (Claude-4.6), BLAgent achieves an MRR of 0.900 and a Top-1 of 86.7%, Top-3 of 93.0%, and Top-5 of 94.7% across all 300 instances. The closest competitor, LocAgent (Claude-4.6), evaluated on 182 completed instances, achieves a Top-1 of 82.4% — higher than its Claude-3.5 result (77.7%), suggesting that the stronger model improves Top-1 predictions. However, its Top-3 and Top-5 accuracy both drop to 86.3%, falling well below its own Claude-3.5 results (91.9% and 94.1%, respectively). For a fair comparison, we evaluated the localization results of BLAgent on the same 182 instances, and it demonstrated a 90.6% Top-1 accuracy. Furthermore, we inspected the localization results of LocAgent and found that Claude-4.6 causes LocAgent to return fewer candidate files in most instances, often only one or two, reflecting overconfidence in its top prediction at the expense of recall. In contrast, BLAgent with Claude-4.6-Sonnet improves consistently across all thresholds, confirming that the architectural design — bounded agentic reasoning over a structured candidate set — is the primary driver of its gains rather than the underlying LLM.

3.2.3 Discussion↩︎

While BLAgent demonstrates superior performance compared to state-of-the-art methods across all Top-k, it is important to examine the cases where it fails to correctly localize faulty files. While using Claude-4.6, we find that in all 14 failed cases, the correct file was not retrieved in the top 15 candidates by any of the query transformation techniques.

Failure case analysis. A systematic analysis of the 14 retrieval failures reveals three distinct patterns, summarized in Table 2. Two cases involve feature requests, where no existing code snippet (e.g., function) was modified by the patch, and the fix resides in configuration or infrastructure code with no strong retrieval signal. Notably, our baseline LocAgent explicitly excludes these same instances from its own evaluation, stating they do not modify any existing function [21]. Five cases involve hidden dependencies, where retrieval correctly identifies the primary symptom site but the fix resides in an imported utility or base class that the retrieved file depends on. The remaining seven are vocabulary mismatch cases, where the bug report’s surface text strongly suggests one module while the fix resides in a behaviorally connected but lexically distant one. To investigate the recall ceiling, we extended retrieval up to Top-50 per query transformation for the 12 non-feature-request failures. The correct file appears at ranks 18–50 in 8 instances, while the remaining 4 are absent from the top-50 of both transformations, asserting that an embedding-level semantic gap that merely increasing \(k\) may not resolve. However, extending \(k\) to 50 inflates the candidate pool (\(T_0 + T_1\)) from 15 to \(\sim\)​63 unique files on average, incurring substantial cost and context overflow in the agentic reranking pipeline.

Repository-level concentration. Ten of the 14 failures originate from the sympy repository, representing a 13.0% repository failure rate (10 out of 77 instances), compared to only 3.5% for django (4 out of 114 instances). This concentration reflects a systematic mismatch between how sympy bug reports are written and how its implementation is organized. In 9 of 10 failed sympy cases, BLAgent retrieves at least one file from the same directory as the true patch, indicating that the relevant neighborhood is usually identified correctly. Bug reports describe the user-visible mathematical symptom, which consistently points retrieval toward the observable behavior rather than the underlying implementation site. For example, in sympy-21612 the report describes a “LaTeX parsing” error, so retrieval surfaces parsing/latex/ and printing/latex.py; yet none of the dominant report terms appear in the actual patch file printing/str.py. In sympy-18087, the terms trigsimp, cos, and sqrt are absent from the patch file core/exprtools.py, making it invisible to embedding-based retrieval. This is compounded by sympy’s mathematical naming conventions: files such as exprtools.py and operations.py implement low-level algebraic primitives that are silently invoked by the higher-level operations users interact with and are described mostly in abstract algebraic terms that bear no lexical resemblance to user-reported symptoms. By contrast, django’s domain-descriptive identifiers align naturally with the terms users write in bug reports, giving the retriever a strong discriminative signal.

These observations highlight two critical insights: first, dense retrieval remains a crucial component of the pipeline, as it determines the initial pool of files available for reranking; second, improving recall in the retrieval stage may directly translate into improved localization accuracy.

3.3 RQ1.2 How do different retrieval and RAG configurations influence file-level localization accuracy?↩︎

3.3.1 Approach.↩︎

We design and evaluate RAG pipelines using different chunking strategies under two ranking settings: (1) Direct Dense Retrieval (Query \(\rightarrow\) Retrieval), where candidate files are ranked purely by embedding similarity with no LLM involvement, and (2) LLM Reranking (Basic RAG: Query \(\rightarrow\) Retrieval \(\rightarrow\) Generation), where the top retrieved chunks are passed as context to an LLM that reranks the candidate files based on their relevance to the bug report. We extract all source code files (e.g., .py files) from each repository and apply three distinct chunking strategies:

• Text-based chunking, which splits files into fixed-size textual segments without code structure awareness;

• Code-aware chunking, which preserves syntactic boundaries (e.g., functions and classes) to retain structural coherence; and

• Code-aware with file path context, which further embeds each chunk with its file path to provide hierarchical grounding during retrieval (discussed in Section 2).

These chunks are then encoded into dense vector representations and stored in a vector database (chromadb⁴) for efficient retrieval. Next, we assess the retrieval accuracy of each chunking strategy independently to measure how effectively relevant code files can be identified given a bug report. Finally, we integrate the retrieval component into a full RAG pipeline (Figure 10), where the retrieved code chunks serve as contextual input to an LLM. We provide Top-15 files to the LLM for ranking to keep the context length reasonable. The prompt used for the RAG pipeline is provided in Listing [lst:rag_prompt]. This allows us to find the suitable configuration (e.g., chunking strategy, whether LLM integration is useful or not) that provides the best result.

3.3.2 Results.↩︎

Table 3 presents the localization performance across different dense retrieval and RAG configurations on the SWE-bench-Lite dataset. The results reveal several important observations.

First, direct dense retrieval methods show limited capability in accurately ranking faulty files in top positions. When standard text-based chunking is applied using conventional text-splitters designed for natural language, the performance is notably low. This confirms that code structure and semantics are poorly preserved under text-oriented segmentation. In contrast, code-aware chunking slightly improves the MRR and Top-1 accuracy, demonstrating that structural cues such as function boundaries and class definitions contribute to more meaningful embeddings and, consequently, more accurate retrieval. However, we observed that when we embed file paths (e.g., module/src/file.py) along with each chunk, the retrieval performance increases significantly (16.9% MRR improvement compared to straightforward Code-Aware Chunking).

Integrating these retrieval configurations into a RAG framework (i.e., when we use an LLM to rerank the dense retrieval candidates) leads to even greater gains, with MRR improving by up to 53% across chunking strategies. However, the best results are achieved with path-embedded, code-aware chunking. This suggests that while dense retrieval alone may struggle to surface the faulty file in the top positions (e.g., Top-1 or Top-3), ensuring that the correct file appears within the Top-10 or Top-15 candidates allows the LLM to leverage its contextual reasoning capabilities to rerank the files more effectively. However, for larger values of k (e.g., Top-10), direct dense retrieval with code-aware chunking and file path augmentation outperforms LLM reranking across all chunking strategies.

3.3.3 Discussion.↩︎

While LLM reranking in the RAG pipeline substantially improves ranking accuracy compared to direct dense retrieval, the effectiveness of the underlying dense retrieval remains a critical factor. It determines whether the relevant file appears within the candidate list at all—since an LLM or agent can only rerank files that have already been retrieved. We observe that path-augmented code-aware chunking yields the most effective retrieval results. For instance, this direct retrieval technique achieves a Top-10 accuracy of 87.3%, compared to only 70.3% for standard code-aware chunking—a 24% improvement. This suggests that incorporating hierarchical path information helps embeddings better capture repository structure and contextual dependencies, improving semantic recall at the file level that is consistent with the design rationale in Section 2.1.

Impact of Path Augmentation. We further analyze how path augmentation improves localization by examining cases where path-augmented chunks correctly surface the faulty file in the top-10 while naive code-aware chunks fail. We identify two prevalent patterns by which bug reports carry path-like signals that augmented chunks can exploit: implicit path matches, where package-qualified identifiers in the bug report structurally mirror the repository path hierarchy, and explicit traceback matches, where stack trace entries directly reference the file path. In both cases, naive chunks—containing only the function body—lack these signals and rank substantially lower, while path-augmented chunks align with query tokens and achieve higher similarity scores (Figure 12).

Case 1 (Implicit Path Match). In django__django-12308, the bug report references django.contrib.admin.utils. Although not a verbatim file path, the dot-separated package hierarchy structurally mirrors the repository path django/contrib/admin/utils.py. A path-augmented chunk prepending this path aligns with these tokens in the query embedding, yielding a high similarity score and surfacing the correct file. The naive chunk, containing only the function body, lacks this signal and ranks substantially lower.

Case 2 (Explicit Traceback Match). In matplotlib__matplotlib-26020, the bug report contains a direct stack trace entry referencing mpl_toolkits/axes_grid1/axes_grid.py. The path-augmented chunk prepended with the relative path of the code matches these tokens directly, producing a high similarity score. The naive chunk, containing only the _tick_only function body, misses this signal entirely and ranks lower.

RAG vs.Direct Retrieval. While the traditional RAG pipeline performs well in Top-{\(1-3\)} accuracy, its performance declines for larger \(k\). Specifically, for code-aware chunking with file paths, LLM-based reranking achieves a Top-10 accuracy of 82.3%, lower than the 87.3% achieved by direct dense retrieval alone (Table 3). We find that the LLM often returns fewer than ten ranked files despite explicit prompt instructions to return 10 files. This behavior likely arises from the model’s tendency to prioritize high-confidence predictions and truncate low-confidence candidates to conserve context length, leading to incomplete Top-\(k\) outputs. Overall, while a basic RAG pipeline may significantly improve Top-1 or 3 positions, it may not be as useful for later positions.

3.4 RQ1.3 Can agentic reranking improve over traditional RAG?↩︎

3.4.1 Approach.↩︎

In this RQ, we explore whether adding an agent-based reranker can improve over the traditional RAG pipeline. To do this, we adopt the best chunking method (\(\star\)) identified in 3.3 and compare four setups under the same configuration–(1) a baseline setup without any reranking, where the dense retriever’s initial ranking is used directly; (2) a traditional RAG pipeline that applies LLM-based contextual reasoning over the top-ranked files to select the most relevant one; (3) BLAgent with agentic scoring only (Phase 1), which uses a ReAct agent to iteratively inspect file skeletons and assign relevance scores; and (4) the full BLAgent pipeline (Phase 1 + Phase 2), which further applies evidence-anchored reranking.

3.4.2 Results.↩︎

Table 4 summarizes the comparative results. The traditional RAG pipeline already improves substantially over direct dense retrieval (MRR: 0.734 vs.), confirming that contextual reasoning over retrieved code snippets enhances relevance estimation. However, introducing agentic reranking yields further improvements across all metrics—raising MRR by 11.5%. This gain indicates that the agentic reranking can better identify and promote truly relevant files that would otherwise remain buried in the candidate list.

3.4.3 Discussion.↩︎

The results demonstrate that each phase of the reranking pipeline contributes meaningfully and for distinct reasons. Basic RAG with LLM reranking already captures substantial gains through contextual reasoning over retrieved candidates. Skeleton-based Agentic Scoring (Phase 1) extends this through iterative, hypothesis-driven skeleton inspection, but produces marginal Top-1 gains (+1.8% over Basic RAG) when structural signatures alone are insufficient to distinguish files that implement similar interfaces or inherit from a common base class. Evidence-anchored reranking (Phase 2) further improves accuracy by constructing pruned file contexts that expand only the method bodies the retriever considers most relevant, grounding the final ranking in implementation-level evidence. The substantial Top-1 improvement from Phase 2 (+11.2% over Phase 1) confirms that richer, targeted context yields better ranking decisions. Together, the two phases deliver a +11.5% MRR gain over Basic RAG, with dense retrieval establishing the candidate pool, agentic scoring narrowing it through structural reasoning, and evidence-anchored reranking refining the final ordering with implementation-level precision.

3.5 RQ1.4 Can query transformation further improve localization effectiveness?↩︎

3.5.1 Approach.↩︎

In Section 3.3, we discussed that the overall performance of RAG pipelines heavily depends on the quality of dense retrieval. While chunking strategies play a crucial role in preserving code semantics, another key factor influencing retrieval effectiveness is the formulation of the input query used to search the vector database. We investigate whether systematic transformations of bug reports can lead to more effective retrieval. To this end, we experiment with two distinct query transformation strategies (see Section 2.2). We apply both transformations on a bug report and then use the transformed reports (i.e., query) as input queries for dense retrieval. The retrieved candidates are then passed to the BLAgent pipeline to evaluate their impact on localization accuracy across both setups.

3.5.2 Results.↩︎

Table 5 summarizes the impact of query transformation on localization accuracy across both dense retrieval and BLAgent. In the dense retrieval setup, both transformation strategies substantially improve performance over using the base bug report as query (i.e., No Transformation), indicating that enriching the input with syntactic or behavioral cues enhances embedding alignment with relevant code fragments. Among the two, the syntactic transformation (\(T_0\)) proves more effective (22.9% MRR improvement compared to using base bug report as query) during direct dense retrieval, suggesting that structural elements—such as module and function identifiers—provide stronger retrieval signals than behavioral descriptions alone.

When these transformations are used within BLAgent, smaller but consistent improvement is realized. Once a relevant file is retrieved within the candidate pool (e.g., Top-15), the agent can typically recover it through reasoning, regardless of its initial rank. Nevertheless, combining both transformations (\(T_0 + T_1\)) enables the agent to reason over candidates retrieved by complementary signals resulting in a consistent gain. These transformations are particularly valuable when using smaller-context LLMs or when faster reasoning is desired by reducing the number of candidates passed to the model. In such settings, the number of candidates passed to the model is reduced, making it more dependent on the quality of the initial query to ensure that relevant files are retrieved.

Sensitivity to Candidate Ordering. To assess whether the concatenation order of \(T_0\) and \(T_1\) introduces positional bias when combining candidates from both transformations in the agentic scoring stage, we evaluate two orderings: (1) \(T_0\)-first (default), where candidates retrieved by the structural query are prepended before behavioral-query candidates, and (2) \(T_1\)-first, where this order is reversed. Table 6 shows that both orderings produce nearly identical results across all metrics, with MRR differing by only 0.2% (0.795 vs.) and Top-1 by 0.8% (0.710 vs.). This robustness follows from two properties of the pipeline. First, the candidate pool is constructed by appending only unique files from the second transformation that are not already retrieved by the first — so both orderings yield largely overlapping candidate sets that differ only in which files occupy the marginal positions. Second, the ReAct agent selectively decides which files to inspect based on its evolving reasoning state, rather than processing the list sequentially. Together, these properties ensure that the \(T_0\)-first default is an implementation convenience rather than a structural dependency of the pipeline.

3.5.3 Discussion.↩︎

We further analyze how different transformations impact dense retrieval by examining the overlap and exclusivity of cases where the correct file is ranked highly. Figure 13 illustrates the complementary nature of the three approaches: the actual bug report, structural query transformation (\(T_0\)), and behavioral query transformation (\(T_1\)).

In the Top-1 scenario, both query transformations identify numerous faulty files that the base bug report alone fails to find. Specifically, \(T_0\) and \(T_1\) uniquely contribute to 29 and 12 Top-1 localizations, respectively, that were not captured by the base report. Additionally, 73 instances are covered by at least one transformed query but missed by the base, demonstrating that transformations are not merely overlapping but in many cases essential for recall. When the window is extended to Top-10, while most of the cases are covered by at least one transformation, there are still 28 cases combined where only query-transformed inputs succeed. However, behavioral transformation becomes less effective on such ranks.

Complementarity. Across 300 instances, structural queries (\(T_0\)) retrieve the correct file within Top-10 in 92.0% of cases and behavioral queries (\(T_1\)) in 88.7%, but their union achieves 94.3%—a gain neither type reaches alone. In 17 cases, only \(T_0\) retrieves the correct file; in 7 cases, only \(T_1\) does. This improvement results from a systematic divergence in retrieval behavior: \(T_0\) anchors the candidate set to implementation files by matching on identifiers, traceback tokens, and module references, while \(T_1\) broadens it toward observable entry points by aligning with symptom descriptions and runtime behavior. Furthermore, 57.7% of instances exhibit high complementarity, with at least two unique files in each query’s Top-5, providing the downstream agent with a broader and more diverse candidate pool. Figure 14 illustrates this complementarity with two representative cases.

When Structural Queries Outperform Behavioral Ones. Structural queries are more effective when the bug report contains an explicit class, method, or traceback references that uniquely identify the implementation site. In seaborn-3190 (Figure 14 (a), patch: seaborn/_core/scales.py), the structural query encodes the precise traceback location (ContinuousBase._setup) and the specific file path, retrieving the correct file at Rank 1. The behavioral query, despite naming the correct module, disperses similarity mass across color-handling files (e.g., categorical.py, palettes.py) due to symptom-level tokens, pushing scales.py outside the Top-10. The structural query wins because low-frequency, high-specificity tokens from the traceback uniquely ground the embedding toward the implementation module.

When Behavioral Queries Outperform Structural Ones. Behavioral queries work best in cases where structural cues are weak or misleading—specifically, when the bug manifests as a runtime output mismatch rather than a named API failure. In sympy-11897 (Figure 14 (b), patch: sympy/printing/latex.py), the structural query targets _print_Mul and _print_Pow, focusing on the mathematical equations provided in the bug report and retrieves files responsible for such operations (e.g., bench_solvers.py) which are actually unrelated to this issue. The behavioral query, describing the observed output discrepancy between latex() and pprint() for the same expression, retrieves the correct file at Rank 2 alongside closely related printing modules. The behavioral framing succeeds because it aligns with the module’s functional responsibility (output formatting) rather than its internal method names.

3.6 RQ1.5 Does the choice of LLM variant affect the performance of the agent?↩︎

3.6.1 Approach.↩︎

To evaluate the robustness of BLAgent across different LLM backbones, we integrate the agentic framework with multiple large language models spanning open-source and closed-source families: Qwen3-32B, GPT-OSS-120B, and Claude-4.6-Sonnet. All models operate on the same query-transformed retrieval process and reasoning protocol with the same temperature (0.7), allowing us to isolate the effect of model scale and architecture on localization performance. We assess file-level accuracy on the SWE-bench-Lite dataset and compare metrics such as MRR and Top-k accuracy to determine whether model scale and type—open-source versus closed-source—provide meaningful gains in the context of agentic reasoning.

3.6.2 Results.↩︎

Table 7 reports localization performance across all three LLM backbones. Among open-source models, despite GPT-OSS-120B being nearly \(3.8\times\) larger than Qwen3-32B, Phase 1 MRR differs by only 0.6% (0.794 vs.), and Top-\(k\) accuracies are nearly identical across all thresholds. This indicates that model scale provides only marginal gains within the agentic reasoning framework when operating at the skeleton-inspection level.

The addition of Phase 2 (evidence-anchored reranking) yields consistent improvements across all three models. Notably, Qwen3-32B benefits most from Phase 2, with MRR improving by 7.3% and Top-1 by 12.8% (0.706 \(\rightarrow\) 0.797). GPT-OSS-120B shows a similar pattern with a 7.1% MRR gain. However, the performance for both models in both stages remains almost identical. In contrast, Claude-4.6-Sonnet already achieves strong Phase 1 performance (MRR 0.889, Top-1 0.843), and Phase 2 provides a marginal improvement. This model-dependent pattern suggests that larger, more capable models produce better-separated relevance scores in Phase 1, reducing the marginal contribution of implementation-level evidence in Phase 2, while smaller open-source models rely more heavily on Phase 2 to resolve ambiguous rankings.

3.6.3 Discussion.↩︎

These findings reveal a consistent two-tier pattern across model families. For open-source models, agentic reasoning and retrieval coordination are the dominant performance drivers — the LLM’s role is primarily to follow structured reasoning protocols over retrieved skeletal evidence rather than to draw on parametric knowledge, which explains why Qwen3-32B and GPT-OSS-120B perform near-identically in Phase 1 despite a \(3.8\times\) size difference. Phase 2 then contributes meaningfully for both, confirming that implementation-level context bridges the gap that skeleton-level scoring leaves open.

For closed-source frontier models, Claude-4.6-Sonnet demonstrates that stronger reasoning capability translates into more precise skeleton-based scoring, achieving Top-1 84.3% in Phase 1 alone — substantially above both open-source models. Phase 2 still improves MRR and Top-1 accuracy, but the gain is smaller, implying that well-separated Phase 1 scores leave less room for consolidation to refine. Practically, this presents a clear deployment trade-off: open-source models with Phase 2 achieve competitive accuracy at lower cost, while frontier models with Phase 1 alone may suffice for a cost-effective solution.

3.7 RQ1.6 Can BLAgent be extended to function-level localization?↩︎

3.7.1 Approach.↩︎

While BLAgent is primarily designed around file-level localization motivated by the requirements of LLM-based APR pipelines, its core components, such as query transformation, AST-aware retrieval, and agentic reranking, are not inherently file-level specific. To evaluate whether the architecture generalizes, we extend BLAgent to function-level localization with a single modification: we replace the final consolidated reranking prompt (Listing [lst:reranking_prompt_phase2]) in Phase 2 (EAR) of our agentic reranking stage (Section 2.4) to instruct the LLM to identify relevant Class::method entities rather than files (Listing [lst:function_loc_prompt]), keeping all upstream components unchanged. We report Top-\(k\) accuracy following prior work [5], [6], where a prediction is counted as correct if any predicted method name appears within the method part of any ground-truth file::method string in the Top-\(k\) predictions. To support a same-model comparison with the strongest baseline, we additionally evaluate BLAgent under Claude-4.6 and compare with LocAgent under the same LLM, mirroring the controlled comparison setup used in Section 3.2.

3.7.2 Results.↩︎

BLAgent generalizes effectively to function-level localization with a single prompt change, achieving state-of-the-art accuracy under both open-source and frontier LLM settings (Table 8). With the open-source GPT-OSS model, BLAgent is already competitive with the best prior system, LocAgent (Claude-3.5). Under the same-model comparison, BLAgent (Claude-4.6) establishes a new state of the art across all Top-\(k\) thresholds, with the largest margin at Top-1, where it improves over LocAgent (Claude-4.6) by 88%. Interestingly, LocAgent (Claude-4.6) remains competitive at Top-5 and Top-10 but suffers a sharp Top-1 drop relative to its Claude-3.5 configuration, suggesting that the stronger model surfaces the correct function within its candidate list but ranks it lower within the LocAgent framework. This could be due to the training of the models, of which we cannot speculate. BLAgent, in contrast, improves consistently across all thresholds when moving to Claude-4.6, similar to the trend observed at the file level (Section 3.2). Hence, BLAgent offers a more stable (besides improved) performance over LocAgent across the models.

Importantly, this result is achieved through the single-prompt modification described in Section 3.7—all upstream pipeline components remain unchanged—meaning the function-level extension introduces minimal architectural overhead beyond the file-level pipeline. The EAR consolidation step averages only \(\approx\)​15,300 tokens per instance in a single inference pass, similar to the cost reported for file-level localization in BLAgent (Section 5.1). BLAgent’s ability to use an open-source model without extensive fine-tuning further removes dependence on proprietary APIs entirely, making the function-level extension practical for deployment at scale.

3.7.3 Discussion.↩︎

The strong function-level result is best understood as a direct consequence of BLAgent’s file-level precision rather than the prompt modification alone. BLAgent retrieves the correct patch file within its Top-5 candidates in over 90% of all instances, and once the correct file is present, the problem of identifying relevant methods becomes a tightly constrained search. When the candidate file set is accurate and compact, function-level—and by extension statement-level—prediction tasks benefit from a reduced search space, lower context pressure on the LLM, and a cleaner signal-to-noise ratio in the retrieved evidence. The further gains from GPT-OSS to Claude-4.6 (35.4% improvement at Top-1) reinforce that, given a high-precision candidate set, function-level accuracy scales primarily with the underlying LLM’s reasoning capability rather than with additional architectural complexity.

This points to a natural integration opportunity. Rather than treating file-level and function-level localization as competing tasks, a cascaded design—in which BLAgent’s ranked files serve as input to a graph-guided function-level agent such as LocAgent or Agentless—would allow each component to operate in its strongest regime: BLAgent contributes high-precision, low-cost file discovery, while the downstream agent focuses its traversal within a pre-filtered, high-confidence candidate set. Such a cascade could outperform either system in isolation while remaining cost-efficient, and is a promising direction for future hierarchical repair pipelines.

4 Impact Assessment of BLAgent Bug Location on Automated Bug Repair (RQ2)↩︎

To better understand how BLAgent localization influences downstream repair effectiveness, we break down RQ2 into the following sub-RQs:

• RQ2.1: Does localizing with BLAgent improve program repair?

  We investigate whether replacing the baseline localization module with BLAgent leads to a higher proportion of correct repairs in Agentless, thereby assessing the causal impact of improved localization on overall APR effectiveness.

• RQ2.2: At which stage does the repair process fail even with correct localization?

  We decompose the end-to-end APR pipeline to identify where failures arise—across different levels of localization (e.g., file, function, line) and patch generation—and analyze why these issues occur despite accurately identifying the faulty file.

4.1 Experimental Setup↩︎

4.1.1 Agentless APR Framework.↩︎

We adopt Agentless [10] as our downstream APR baseline due to several reasons. First, it is widely used both in the industry (e.g., OpenAI) and academia [5], [6], [21] to create and compare bug repair patches. Second, it is highly modular with a clear separation between localization and repair components. Third, Agentless has demonstrated competitive performance on the SWE-bench dataset, making it a representative baseline for modern LLM-based APR systems. While Agentless itself is non-agentic, we use it purely as a downstream APR framework to evaluate the impact of BLAgent’s enhanced file-level localization. Since the original framework does not natively support Ollama-based LLMs or the embedding models used in our setup, we extend it to interface with the Ollama runtime and the HuggingFace embedding library. All other components of the Agentless pipeline, including patch generation, validation, and evaluation, remain unchanged to ensure comparability. We use the same GPT-OSS-120B model as the base LLM and nomic-embed-text-v1 as the embedding model (similar to Section 3.1.1).

For this evaluation, we first execute Agentless using its native localization module as a baseline. Then, we replace Agentless’ native file-localization approach with BLAgent. Given a bug report and a target repository, our framework produces a ranked list of candidate files, assigning each file a relevance score via agentic reranking. The top-3 files (matching Agentless’s default) are passed to the Agentless repair pipeline. From this stage onward, the workflow follows the standard Agentless protocol: line-level localization is performed within the selected files using targeted LLM prompting, after which 40 candidate patches are generated using an LLM through a high temperature of 0.8 (default configuration). The final patch is then selected via majority voting, with regression and reproduction tests employed to validate the correctness of the repair. The overall setup is shown in Figure 16.

4.1.2 Evaluation Metrics↩︎

Following the evaluation protocol of SWE-bench [7], we assess the effectiveness of program repair using the Resolved Rate. This metric represents the proportion of bug reports for which the generated patches successfully pass all the validation tests. Let \(N_{\text{resolved}}\) denote the number of resolved bugs and \(N_{\text{total}}\) the total number of bugs. Then, \[\mathrm{Resolved\;Rate} = \frac{N_{\text{resolved}}}{N_{\text{total}}} \times 100\%.\] A higher resolved rate indicates a stronger end-to-end repair capability of the system.

4.2 RQ2.1 Does localizing with BLAgent improve program repair?↩︎

4.2.1 Approach.↩︎

We integrate BLAgent into the Agentless framework [10] and evaluate it on the SWE-bench-Lite dataset [7]. Similar to the original work, we conduct experiments under three configurations: (i) Majority Voting, which generates 40 patch candidates and selects the final one via majority voting; (ii) +Regression Test, where available regression tests are included; and (iii) +Reproduction Test, which further generates and incorporates reproduction tests to select a generated patch.

4.2.2 Results.↩︎

Table 9 summarizes the results. Integrating BLAgent consistently improves the end-to-end repair rate across all settings. We run the framework with both localization approaches three times and report the best result for both methods in Table 9. Under the majority voting setup, our approach resolves up to 102 bugs versus 83 by the baseline (comparable to the 79 originally reported by Agentless with GPT-4o), representing a relative improvement of over 20%. When regression and reproduction tests are included, a similar performance gap persists.

A closer examination of the generated patches reveals another important benefit. When using GPT-OSS-120B as the base LLM, the baseline framework produced 38 empty patches—instances where no valid patch was ultimately selected—compared to 14 when we used the BLAgent as the file-level localization method. Empty patches often indicate that the LLM received insufficient or irrelevant context, either due to inaccurate localization or because the generated patch could not be parsed as a valid fix. The reduction in such cases suggests that the proposed method provides more semantically grounded input, allowing the LLM to reason more effectively about the underlying fault and produce syntactically valid, contextually coherent patches.

We note that a subset of issues (e.g., 23 issues from matplotlib) could not execute regression or reproduction tests due to the reliance of Agentless on an obsolete SWE-bench 2.1 interface, which is now outdated. Attempts to migrate to the latest version revealed compatibility issues requiring nontrivial engineering effort, which we leave for future work. Therefore, the true repair potential is likely higher than reported, as these missing cases could result in additional successful patches.

Although BLAgent achieves a Top-1 file-level localization accuracy close to 80% with GPT-OSS, the corresponding end-to-end resolution rate remains around 38%. This disparity highlights a key insight: accurate file-level localization, while necessary, is not sufficient for successful repair. Downstream steps such as line-level localization and semantic patch generation in the APR pipeline remain major limiting factors.

4.2.3 Discussion.↩︎

While BLAgent consistently improves repair performance, we next investigate the stability of these improvements. Figure 17 compares repair overlaps across multiple runs using both the native and BLAgent localizations. We observe some divergence between runs, with each resolving partially disjoint sets of issues. This instability can largely be attributed to the stochasticity inherent in high-temperature decoding during both patch and test generation, suggesting that overall resolved counts alone may obscure deeper behavioral patterns of APR systems. To quantify whether the observed improvements are statistically meaningful, we apply a paired Wilcoxon signed-rank test over per-bug solve frequencies across the three runs, confirming that the observed improvement is statistically significant (\(p = 0.014\)) with \(p < 0.05\). Across all 3 runs, using BLAgent as the localization module leads to 136 unique total bug fixes (45.3%) in the SWE-bench-Lite dataset, where it solves 35 unique bugs, while the native localization method only solved 106 (35.3%).

Furthermore, we compare the results from multiple runs of the Agentless APR framework—once using its native localization method and once using BLAgent for file-level localization. Specifically, we focus on the uniquely repaired issues, cases successfully fixed when using one localization method but not the other. For these issues, we examine whether the alternative method failed to correctly identify the faulty file. Table 10 summarizes this comparison by showing how often the other method failed to localize the correct file for these uniquely repaired cases. When our approach is used during localization, and it exclusively repairs an issue, the baseline localization method (Agentless) often fails to correctly localize the faulty file (e.g., 50% of such cases in Run 3). In contrast, for issues uniquely repaired by Agentless when using its own localization method, BLAgent localized the correct file in nearly all instances. This pattern shows that unique repairs resulting from the BLAgent’s localization frequently arise from scenarios where the baseline’s localization was insufficient, suggesting that improved file-level grounding directly enables successful patch synthesis. Conversely, the few cases where Agentless succeeded despite our correct localization indicate failures in later APR stages—such as patch generation—rather than in file-level localization itself.

Overall, these findings highlight that accurate localization is not a peripheral factor but a foundational determinant of downstream repair success. While improved localization does not guarantee a correct patch, it substantially increases the probability of success by ensuring the LLM operates on the correct fault context. When the faulty file is missing from the retrieved candidates, even advanced reasoning cannot compensate.

4.3 RQ2.2 At which stage does the repair process fail even with correct localization?↩︎

4.3.1 Approach.↩︎

To better understand where and how the program repair framework fails despite high file-level localization accuracy, we conduct a detailed error analysis across Agentless’s hierarchical pipeline. Specifically, we aim to identify whether unresolved cases stem from failures in localization (at file, function, or line level) or from the subsequent patch generation step. For this experiment, we analyze the artifacts from Run 2 (Section 4.2), where BLAgent was used as the localization method and the lowest number of bugs (102) were resolved across the three runs. Focusing on the worst-performing run provides a conservative view of the failure modes, ensuring that the identified bottlenecks are not artifacts of a particularly favorable run.

We begin by selecting 177 unresolved bug instances—cases where Agentless failed to produce a correct patch—and systematically trace their progression through each stage of the framework. Using Agentless’s default configuration, the model considers the top-\(k\) ranked outputs from each previous stage as context for the next (with \(k=3\) in our experiments). For example, during function-level localization, the model only explores functions from the top-3 retrieved files; for line- or statement-level localization, it only considers the top-3 functions, and so on. This hierarchical dependency makes error propagation particularly important to diagnose.

For each stage, we compare model predictions with ground-truth locations extracted from the actual developer patches. At the file level, localization is considered correct if the ground-truth file appears among the top-3 retrieved candidates. The same criterion is applied to function-level localization. However, evaluating line-level accuracy is more nuanced, since small semantic differences can lead to valid fixes at slightly different line numbers or equivalent statements. To address this, we employ a lightweight heuristic approximation to estimate line-level localization accuracy.

Line-level Approximation. The approximation algorithm compares the first changed statement in the model-generated patch with that in the ground-truth patch. It proceeds as follows:

1. Extract the first code line (excluding comments and docstrings) that was added or removed in both patches.

2. If both edits occur within the same file and their line numbers differ by at most 5 lines, line-level localization is considered correct.

3. Otherwise, compute the normalized sequence similarity between the two modified lines. We straightforwardly calculate the similarity using SequenceMatcher ⁵.

4. If the edits belong to the same file, exhibit textual similarity above a threshold (similarity \(\geq 0.6\)), and occur within a small positional window (e.g., within 5–10 lines), we treat the result as an approximate match at the line level. The threshold of 0.6 was determined by manually inspecting a small sample of cases, confirming that it reliably captures genuine near-miss localizations.

This heuristic allows us to capture cases where the framework localizes the correct logical region but introduces the modification slightly above or below the true location—an issue common in real repair settings. While this approximation does not provide an exact measure of line-level accuracy, it offers a pragmatic way to distinguish between near-miss localizations and genuine mislocalizations. Similar strategies have been adopted in prior studies to approximate line-level localization accuracy [10].

4.3.2 Results.↩︎

As shown in Figure  18a, approximately one-third of all instances (34%) were successfully resolved by the system. However, a majority of the cases remain unresolved, including a smaller fraction (7%) of empty patches (i.e., no patch was selected at the end), suggesting that the APR framework often fails to produce a syntactically and logically valid modification. The unresolved cases (almost 60%) where a patch was generated but could not fix the bug motivate a deeper look into the repair pipeline to understand where failures originate. Out of the 177 unresolved cases, we find that 158 issues were localized correctly at the file-level (Figure 18b) with BLAgent but failed in a later stage in the program repair process.

Failures Across Localization Stages. We decompose unresolved cases into four categories in Figure 18 (b), reflecting the specific stages at which the program repair pipeline fails. Most of these issues occurred in one of the stages of the hierarchical localization pipeline.

File-level localization failures constitute 10.7% of the unresolved cases, indicating that a subset of bug reports still leads the retrieval module toward incorrect source files. These failures predominantly occur when the correct file does not appear among the Top-3 retrieved candidates. While expanding the retrieval depth could mitigate some of these errors, it would also increase the computational overhead and complexity of the Agentless framework, creating a trade-off between coverage and efficiency.

In our observation, the most frequent failures occur at the function-level localization (44.6%), highlighting persistent challenges in accurately ranking the correct function among the Top-3 candidates even when the faulty file itself is correctly retrieved. A notable source of these errors arises from formatting inconsistencies in LLM outputs that prevent the APR framework from correctly parsing the generated function names. For instance, in sympy__sympy-18057, although the correct function name appeared in the LLM’s response, the framework failed to interpret it due to a structural mismatch, leading to no valid function being selected (see Listing [lst:loc-failure-func-level]).

The second most common failures occur at the line level localization (26.6%), where the model successfully identifies the faulty function but struggles to locate the precise statement or edit region within it. These near-miss cases indicate that while the model captures the correct semantic region, it lacks sufficient fine-grained reasoning about program behavior and fix placement. The underlying causes of such line-level failures can be diverse—ranging from intrinsic LLM limitations in reasoning about code execution to occasional hallucinations [35] or incomplete code understanding when the LLM outputs completely unrelated line numbers. In our investigation, we also observed cases such as pallets__flask-5063, where the ground-truth patch modifies multiple methods, but the LLM focused only on a single one (lines 1021–1034), resulting in a partial and ultimately incorrect localization (see Listing [lst:loc-failure-line-level]).

Finally, patch-generation failures (18.1%) occur when localization is successful at all steps, but the code synthesis stage either produces invalid or semantically incorrect edits. This points to the limitations of current generation models in maintaining compilation consistency and applying contextually coherent changes, even with accurate localization cues. Figure 21 provides a concrete example from sympy__sympy-22005. Here, the localization is accurate down to the line level, and the generated patch targets the same location as the ground-truth patch. However, the generated patch (Figure 21a) adds a conditional check to restrict univariate polynomials to systems with a single generator. While syntactically correct, it does not fully encode the intended semantics. In contrast, the ground-truth patch (Figure 21b) raises a NotImplementedError for underdetermined systems, enforcing the zero-dimensional constraint and preserving correct program behavior.

Overall, our evaluation indicates that while end-to-end APR failures arise at multiple stages, localization constitutes a major bottleneck. File-level, function-level, and line-level localization collectively account for the majority of unresolved cases. Notably, in our analysis, approximately 82% of unresolved issues could be traced to failures at some stage of hierarchical localization. These findings highlight that precise bug-context localization is a necessary prerequisite for effective LLM-based program repair, and that improvements in both fine-grained localization and code synthesis are essential to substantially increase repair success.

4.3.3 Discussion.↩︎

Given that most unresolved cases stem from function- or line-level localization errors, we investigate whether these failures are purely localization-related and if providing correct statement-level context could lead to successful repair.

We analyze the case of django__django-11133, where file-level localization was correct but line-level localization failed in one of the experimental runs. During the failed run, the APR system incorrectly targeted line 309 (Figure 22a) instead of the correct region near line 229 (Figure 22b). As a result, the generated patch was syntactically valid but semantically irrelevant, and the repair failed.

To test whether precise localization alone could resolve the issue, we re-executed the repair process by replacing the model’s predicted line number with the correct one. With accurate line-level context, the APR successfully generated the correct patch (Figure 23), demonstrating that the previous failure was not due to deficiencies in patch synthesis but rather to mislocalized edit positions. This case study highlights a key insight: deviations in statement- or line-level localization can derail the repair process entirely.

5 Impact of Key Design Decisions in BLAgent↩︎

5.1 Cost Analysis↩︎

Table 11 reports the average token consumption and estimated API cost per bug instance in USD for BLAgent across two model backends. For the Claude-4.6-Sonnet model, we calculate the cost using its official pricing documentation⁶, and for the GPT-OSS-120B model, we estimate the cost using a third-party provider ⁷. With GPT-OSS-120B, the full agentic reranking pipeline consumes 24,523 prompt tokens and 1,939 completion tokens per bug, corresponding to an estimated cost of $0.0017 per instance. With Claude-4.6-Sonnet, the total consumption is 26,180 prompt tokens and 582 completion tokens, at an estimated $0.09 per instance. Across all 300 SWE-bench-Lite instances, the total localization cost amounts to less than $1 with GPT-OSS-120B and $27 with Claude-4.6-Sonnet.

Comparison with LocAgent. To contextualize BLAgent’s cost efficiency, we ran LocAgent under identical conditions using Claude-4.6-Sonnet with a fixed budget of $300 for the 300 instances in the SWE-bench-Lite benchmark. LocAgent exhausted the full $300 budget after processing only 182 instances, producing valid results for 158 of them. The remaining instances either timed out or exceeded 800,000 tokens/minute per instance due to unbounded graph traversal loops before the budget was depleted. Among the 158 completed instances, LocAgent consumed an average of 282,720 prompt tokens and 2,634 completion tokens per bug, at an average cost of $0.89 per instance. However, this figure understates the true cost, as failed instances consumed disproportionately more tokens—bringing the observed cost across all 182 attempted instances to $1.65 per instance ($300 \(\div\) 182). In contrast, BLAgent completed all 300 instances at $0.09 per instance ($27 total), representing a cost reduction of more than 18\(\times\) relative to LocAgent’s observed rate.

5.2 ReAct Agents for Iterative Reasoning↩︎

While the ReAct agent employed in this study demonstrated clear improvements in bug localization compared to traditional RAG pipelines, its architecture introduces several inherent limitations. ReAct operates through a strictly sequential reasoning loop (Thought \(\rightarrow\) Action \(\rightarrow\) Observation \(\rightarrow\) Answer/Repeat), which constrains both scalability and responsiveness. Each reasoning step appends additional traces to the prompt, causing rapid context growth over multiple iterations. This iterative accumulation increases inference latency and memory overhead. For instance, the traditional RAG pipeline required approximately 10–25 seconds per issue on our local GPT-OSS-120B setup with Ollama, whereas the agentic RAG pipeline often required 15–40 seconds—particularly when the agent performed more reasoning steps before reaching a final decision. However, such issues were not noticed when using the Claude-4.6 model due to its fast inference.

To mitigate this overhead, our proposed pipeline incorporates two strategies. First, the agent operates on a limited candidate set (e.g., Top-15 retrieved files) to reduce unnecessary reasoning scope. Second, it is explicitly instructed to analyze detailed file contents (i.e., code skeletons) only when deemed necessary by its reasoning policy. Nonetheless, the ReAct architecture remains inherently sequential and lacks parallelism. Consequently, the agent’s decision-making becomes tightly coupled with the linear order of its reasoning: an incorrect hypothesis formed in an early reasoning step—for example, prematurely concluding that a structurally similar file is the fault location—can bias subsequent file inspections and persist uncorrected through the final scoring output. This strict sequentiality, while advantageous for interpretability and modular reasoning, imposes trade-offs in scenarios requiring long candidate pool exploration or frequent context switching.

5.3 Code Skeletons for Efficient Context Management↩︎

To support effective agentic reasoning, we employed abstract code skeletons—structural representations containing only class and function signatures—instead of full source code. Prior work has shown that using code skeletons improves localization accuracy in LLM-based repair systems [10]. In our setting, however, this design choice is primarily motivated by the context sensitivity of ReAct agents, whose internal state grows cumulatively across reasoning turns. Specifically, each iteration in a ReAct agent’s cycle adds reasoning traces, action descriptions, and intermediate results to the conversation history. This cumulative expansion rapidly depletes the available context window and escalates computational cost, making ReAct agents expensive when many reasoning steps are required or the input is large.

During our experiments, we observed that limiting the agent’s view to the code skeleton substantially improved both reasoning efficiency and overall stability. When restricted to skeletons, the agent typically converged within five reasoning steps, as it was guided to reach a conclusion as soon as sufficient evidence was gathered. This behavior eliminates the need to sequentially rank all related files, allowing the agent to focus only on those it considers most relevant. In contrast, when provided with full source code files—often spanning thousands of tokens—the agent’s prompt grew rapidly, causing significant latency (sometimes exceeding ten minutes per action) and frequent failures due to context overflow.

Consequently, supplying structural representations rather than full implementations mitigates prompt saturation, reduces token redundancy, and allows deeper inspection of candidate files within a fixed computational budget. These observations indicate that for ReAct-style agentic bug localization systems, the use of code skeletons is not merely an optimization but a functional necessity for maintaining both efficiency and reliability.

5.4 Query Transformation for Retrieval in Agentic Pipeline↩︎

Bug localization often suffers from a severe lexical gap between bug reports and source code [36]. While neural retrieval models (e.g., transformer-based encoders [37]) partially mitigate this issue through semantic embeddings, they still struggle to capture the structural and behavioral cues essential for accurate localization. To address this, BLAgent reformulates each bug report into two complementary queries. The structural query extracts code-centric identifiers, while the behavioral query abstracts the described symptoms—an approach shown to improve retrieval effectiveness [38] (Observation [obs:query-transform], [obs:query-transform-2]).

The candidates retrieved by these queries provide BLAgent with complementary, multi‑perspective evidence for file‑level localization. Each query generates its own ranked list of candidate files, allowing the agent to cross‑validate results based on structural reasoning (i.e., viewing the code structure). Prior studies confirm that exploiting structural identifiers like class and method names improves retrieval precision in bug localization [39], and that bug reports that clearly articulate observed and expected behavior provide richer localization cues [40]. However, while using multiple transformed queries is beneficial, expanding the candidate pool with too many queries or a very large candidate pool can result in increased latency, agent reasoning overhead, and token budget in the ReAct loop (Observations [obs:react-limitation], [obs:code-skeleton]). Hence, to balance coverage and efficiency, we limit the transformations to two complementary queries, ensuring that we capture the most important cues without overloading the agent. We then merge only the unique top-ranked files, limiting the candidate pool to 15 files (see Section 2.4), and the agent subsequently ranks the top-10 files among this pool.

5.5 Effect of Context Size on EAR Performance↩︎

To validate the design choices underlying Phase 2, we ablate two key hyperparameters: the number of candidate files passed to EAR (#Files) and the number of retriever-highlighted chunks used to construct each file’s pruned context (#Chunks). Table 12 reports Top-1, Top-3, and prompt token counts across four configurations. Increasing the number of files from 5 to 10 raises token usage by over 86% without Top-1 improvement, suggesting that lower-ranked candidates carry little additional signal and only inflate the context size. Similarly, an increasing number of chunks from 5 to 10 adds 26–27% token overhead with no noticeable accuracy gain, indicating that the retriever already surfaces the most relevant method bodies within the top-5 chunks and further expansion recovers no new evidence. Based on these insights, we find the Top-5 files and chunks optimal for Phase 2 reranking.

Overall, these results highlight that accurate file localization with a RAG pipeline depends less on how much context is provided and more on how well that context is selected, a principle reflected across both reranking phases and the ablation results.

6 Recommendations↩︎

Based on the observations (O\(\star\)) from our study of agentic RAG for bug localization and its impact on program repair, we provide targeted recommendations (R\(\star\)) organized by three overarching themes: (1) Retrieval Quality and Pipeline Architecture, (2) Hierarchical Localization and Fine-Grained Reasoning, and (3) Practical Utility. Each recommendation is grounded in specific findings and paired with actionable implementation strategies. Table 13 summarizes the relationships among our findings, recommendations, and supporting observations.

6.1 Theme 1. Retrieval Quality and Pipeline Architecture↩︎

Recommendation (R). Dense retrieval recall fundamentally bounds a RAG or agentic RAG pipeline—when the correct file is absent from the candidates provided to the pipeline, no subsequent reasoning can compensate. Instead of scaling to larger models, practitioners may focus on improving embedding quality and retrieval mechanisms. Path-aware, code-structured chunking yields 20.4% improvement over text-based chunking, and incorporating relative file paths adds another 16.9% gain to retrieval accuracy in our experiments. Similar approaches may be explored for code-aware embeddings that preserve syntactic boundaries and augment chunks with hierarchical repository context (module/file paths) to enhance semantic recall at the file level.

Recommendation (R). Retrieval pipelines should treat bug reports as multi-perspective queries rather than single text inputs. Refining queries has been found to be useful in RAG pipelines [28], [41]. Similarly, we also showed how transformations that disentangle structural (syntactic) and behavioral (semantic) aspects of a bug report allow the system to retrieve complementary code regions. This dual-channel formulation promotes balanced recall and precision, ensuring robust localization even when individual query views are incomplete. Similar approaches should be explored for better retrieval.

Recommendation (R). While RAG pipelines improve over dense retrieval, agentic reranking allows for further improvement in file-level localization. Such pipelines can generalize to function-level with minimal adaptation, and cascading them with graph-guided function-level agents represents a promising direction toward fully hierarchical, cost-efficient localization for LLM-based patch generation.

Recommendation (R). We discussed in Section 3.6 that models with substantially different sizes can still yield competitive performance within an agentic pipeline when the right context is provided to the agent. This observation enables a practical approach for cost-conscious organizations: adopt tiered LLM utilization where medium-sized models (e.g., Qwen3-32B) handle localization through agentic reasoning, reserving larger or proprietary models (e.g., GPT-4) only for computationally demanding downstream tasks like patch generation, where semantic complexity may justify the additional expense.

6.2 Theme 2. Hierarchical Localization and Fine-Grained Reasoning↩︎

Recommendation (R). As shown in Figure 18b, function- and line-level localization comprise the majority of residual failures—even when file-level BLAgent succeeds. To address this critical bottleneck, we recommend extending reasoning methods to every hierarchical stage of localization. In particular, frameworks should (1) implement multi-step, structured reasoning to iteratively assess and rank function and line candidates, (2) deploy confidence-aware candidate expansion strategies—such as dynamically retrieving additional functions or statements when top candidates have low discriminatory signal, and (3) integrate program structure-aware validation checks that verify proposed localizations and edits maintain code correctness and semantic intent. Pursuing these reasoning-based context-rich approaches at all bug localization stages will close the gap between file-level success and end-to-end automated repair performance.

Recommendation (R). APR frameworks should incorporate robust and tolerant output parsing strategies that leverage fuzzy matching or schema-driven validation to handle minor variations and formatting inconsistencies in LLM-generated outputs. Such techniques may reduce false negatives from strict parsing failures and enhance system reliability by enabling automatic corrections or fallback prompts. Empirical studies in LLM-based log parsing and structured data extraction highlight the effectiveness of these approaches in improving parsing robustness and downstream task performance [42].

Recommendation (R). Given that manual provision of correct line-level localization can improve automated repair success (see Figure 23), APR systems may implement iterative refinement mechanisms for line-level localization. Specifically, these systems can dynamically generate, evaluate, and update candidate faulty lines through multi-pass reasoning or feedback loops, enabling correction of initial mislocalizations.

6.3 Theme 3. Practical Utility↩︎

Recommendation (R). To improve APR success rates amidst semantic patch generation errors despite correct localization (Observation [obs:patch-failure]), we strongly recommend tightly integrating testing and semantic validation within the patch selection pipeline. Specifically, regression and reproduction tests should be employed wherever available to filter generated patches before acceptance. When test oracles are unavailable, lightweight semantic validations—such as early syntax compilation checks, static analysis for invariant preservation, and heuristic differential testing against available reference implementations—should be incorporated.

Recommendation (R). Given the inherent stochasticity of APR frameworks employing high-temperature decoding and sampling, as observed in our study with notably divergent repair sets across runs (Observation [obs:repair-diff]), evaluations should incorporate multiple independent execution runs to capture variability. Rather than reporting only point estimates, practitioners are encouraged to aggregate results across multiple trials, analyze overlaps among uniquely repaired issues, and provide nuanced metrics that reflect both mean effectiveness and coverage diversity.

Recommendation (R). The improvement in repair rates (Observation [obs:repair-performance]) when integrating BLAgent into Agentless demonstrates that modular localization components can enhance existing APR systems. Framework developers may expose localization as a pluggable interface with standardized API contracts, allowing researchers and practitioners to quickly swap improved localization modules without modifying downstream components. This reduces integration friction and accelerates adoption of localization advances across the broader APR ecosystem.

Recommendation (R). Agentic reasoning with ReAct architectures may face scalability challenges due to cumulative context growth and sequential reasoning constraints (Observations [obs:react-limitation], [obs:code-skeleton]). To address this, if practitioners want to adopt an agentic approach, they may (1) limit input context by providing lightweight structural abstractions such as code skeletons when possible, (2) implement adaptive iteration controls to encourage early convergence of reasoning, and (3) crucially incorporate explicit agent memory mechanisms [43] that persist and selectively reuse past reasoning states, observations, and decisions. The integration of persistent memory allows the agent to flexibly update its internal state without reiterating the entire reasoning history, mitigating prompt saturation and error propagation.

Recommendation (R). Unbounded graph-guided localization achieves strong accuracy but at substantial inference cost. Agentic RAG instead constrains reasoning to a compact, retrieval-filtered candidate set, avoiding unnecessary context expansion while achieving comparable accuracy. Practitioners may adopt agentic RAG as a low-cost upstream filter and reserve graph-guided traversal for low-confidence cases, enabling scalable hierarchical localization without sacrificing precision.

7 Threats to Validity↩︎

Internal Validity. Internal validity concerns whether the observed improvements can be confidently attributed to the proposed agentic RAG framework rather than uncontrolled factors. To mitigate potential confounding effects, we maintained consistent experimental conditions across all pipelines, including identical retrieval databases, model configurations, and prompt templates. The ReAct agent’s reasoning cycles were constrained by a fixed step limit and identical prompting protocol to improve reproducibility. Nevertheless, since LLM inference remains stochastic under the selected temperature settings, intermediate reasoning behaviors may vary across runs. Future work could further reduce this variance by incorporating repeated trials and statistical aggregation of results.

External Validity. Our evaluation relied primarily on the SWE-bench-Lite benchmark, which, while representative of Python-based GitHub projects and issues, may not fully reflect the diversity of industrial-scale software development. Projects with highly domain-specific APIs, sparse documentation, or unconventional code organization could produce different outcomes. Moreover, the agent’s effectiveness was assessed under specific model and retrieval settings (e.g., medium-sized LLMs with static retrieval backends). Extending the evaluation to other programming languages, larger repositories, and alternative retrieval or agent architectures would enhance the generalizability of our findings. Furthermore, our evaluation uses APR as a representative downstream task. While APR provides an objective and scalable setting, bug localization is also used in developer-centric tasks such as debugging and root-cause analysis, where evaluation typically requires user studies. Thus, our findings may not fully generalize to all usage scenarios. We leave broader evaluations as future work.

8 Related Work↩︎

Our work bridges two active research areas: bug localization and automated program repair, with a particular emphasis on leveraging retrieval-augmented generation and agentic reasoning for repository-level bug localization. We review representative work in these areas and position our contributions.

8.1 Bug Localization↩︎

Bug localization techniques aim to identify buggy code locations to assist debugging. These approaches can be categorized into spectrum-based, information retrieval-based, and learning-based methods.

Spectrum-Based Fault Localization (SBFL). SBFL techniques [1], [44], [45] leverage program execution spectra—coverage information collected from passing and failing test cases—to compute a suspiciousness score for each program entity. Classic formulas such as Tarantula [44], Ochiai [45], and Dstar [46] quantify the likelihood that a statement or function is faulty. To improve on these traditional methods, Zhang et al. [47] proposed PRFL, which incorporates a PageRank-based reweighting mechanism to better capture the relative contribution of individual tests, achieving notable accuracy gains over conventional SBFL formulations.

Information Retrieval (IR)-Based Localization. IR-based approaches [39], [48]–[51] model localization as a retrieval problem, ranking source files according to their textual similarity to bug reports. BugLocator [48] pioneered this direction by introducing a revised Vector Space Model (rVSM) that integrates historical bug-fix information to enhance ranking. Subsequent work, such as Pathidea [52], augmented IR-based retrieval with execution path reconstruction from log data, demonstrating that runtime information can significantly complement textual similarity in identifying faulty files.

Learning-Based Localization. More recently, deep learning methods have become increasingly prevalent [53]–[55]. Zhang et al. [53] conducted a comprehensive empirical study comparing convolutional neural networks (CNNs), recurrent neural networks (RNNs), and multilayer perceptrons (MLPs) for real-world bug localization, finding CNN-based approaches most effective. Lam et al. [49] integrated deep neural representations with VSM to mitigate lexical mismatch between bug reports and code artifacts.

The emergence of LLMs has further advanced this field by enabling models to jointly reason over natural language and source code [5], [6], [56]–[58]. Recent studies have explored both agentic [57] and agent-less [5], [7], [10] paradigms for repository-level bug localization, consistently outperforming traditional approaches. AutoFL [59] is an early LLM-based approach that uses ChatGPT with tool calls to navigate Java projects and localize faults on Defects4J [60], given failing tests and their execution outcomes. AgentFL [57] scales this setting to project-level context by coordinating multiple LLM-based agents for fault comprehension, code navigation, and confirmation, but assumes a test-driven Defects4J workflow. LocAgent [21] constructs a heterogeneous code graph over files, classes, and functions, and uses an LLM agent to navigate it iteratively via BM25-based entity search, graph traversal, and code retrieval, progressively narrowing the fault location from file to function-level on SWE-bench-Lite and LocBench. While effective, its multi-step graph traversal incurs substantially higher cost per instance than retrieval-based approaches according to our evaluation and literature [6].

To the best of our knowledge, no prior work has examined how RAG pipelines perform when coupled with an agentic reasoning process for file-level localization. Moreover, existing studies have not explored whether transforming bug reports into retrieval-friendly queries can further improve ranking accuracy in agentic pipelines.

8.2 Automated Program Repair↩︎

Automated Program Repair (APR) techniques attempt to automatically generate patches to fix software bugs. While we do not propose a new APR technique or improve upon existing APR techniques, one effective way to measure the impact of an FL technique is to see how the downstream APR gets impacted by a localization method. Different approaches like template-based [61], heuristic-based [62], or LLM-based [8], [10], [63] are utilized for automated program repair. Recent works have shown that learning-based frameworks adopting LLMs that generate multiple candidate patches for each bug outperforms all other traditional approaches, including neural machine translation (NMT) approaches [64].

LLM-based APR has evolved through four main paradigms–fine-tuning, prompting, procedural, and agentic [65]. Fine-tuning approaches [66], [67] adapt LLM weights using bug-fix data for task alignment, achieving strong performance but demanding high computational cost. Prompting methods [68], [69] rely on carefully designed prompts to elicit repairs from pre-trained models without retraining, offering flexibility but limited by prompt quality and context length. Procedural frameworks [7], [10], [16] decompose repair into explicit stages such as localization, patch generation, and validation, enabling reproducibility with moderate overhead. Agentic systems [8], [9], [20], [32] instead delegate control to the LLM, allowing dynamic planning and reasoning for multi-file bugs at the cost of higher latency. Despite the impressive success of LLM-based APR frameworks, their accuracy in generating correct patches highly depends on correct FL [5], [70]. While existing LLM-based APR research has primarily focused on enhancing patch generation mechanisms and prompt engineering strategies, our work addresses the upstream bottleneck–file-level localization, which may limit all these approaches.

9 Conclusion & Future Work↩︎

In this paper, we presented BLAgent, a novel agentic retrieval-augmented generation (RAG) framework for file-level bug localization that combines structure-aware retrieval, dual-perspective query transformation, and bounded agentic reasoning. By explicitly modeling both structural and behavioral signals from bug reports and grounding reasoning over retrieval-filtered code contexts, BLAgent achieves state-of-the-art localization performance on SWE-bench-Lite while maintaining substantially lower cost compared to existing graph-based or heavily instrumented approaches. Our findings further demonstrate that improvements at the file level translate directly into meaningful gains in downstream automated program repair. Furthermore, the proposed framework generalizes across localization granularities, as our results show that the same agentic RAG design naturally extends to finer-grained tasks such as function-level localization, highlighting its flexibility as a unified reasoning framework over code.

Looking forward, a key direction is to evolve BLAgent into a unified, multi-granularity localization framework that seamlessly integrates file-, function-, and line-level reasoning within a single pipeline. While the current design naturally extends across these levels, further optimization is needed to enable precise fine-grained localization without compromising efficiency. In particular, we envision a hybrid retrieval strategy where dense retrieval is complemented with lightweight graph-based exploration in cases where semantic signals are insufficient, enabling more robust candidate discovery in complex dependency structures. Coupled with tighter integration between retrieval and reasoning, this would allow the agent to progressively refine localization from coarse to fine granularity, moving toward an end-to-end localization system that directly supports debugging and automated repair.

10 Data Availability↩︎

The replication package for this study, including the implementation, experimental scripts, and supporting materials, is publicly available at: https://github.com/afifaniks/BLAgent.

References↩︎

---

1. https://developers.llamaindex.ai/python/framework-api-reference/node_parsers/code/#llama_index.core.node_parser.CodeSplitter↩︎

2. https://openai.com/index/introducing-gpt-oss/↩︎

3. https://huggingface.co/nomic-ai/nomic-embed-text-v1↩︎

4. https://www.trychroma.com/↩︎

5. https://docs.python.org/3/library/difflib.html#difflib.SequenceMatcher↩︎

6. https://platform.claude.com/docs/en/about-claude/pricing (Input $3/M Tokens; Output $15/M Tokens)↩︎

7. https://novita.ai/models/model-detail/openai-gpt-oss-120b (Input $0.05/M Tokens; Output $0.25/M Tokens)↩︎