1 Introduction↩︎

Automating issue resolution is a longstanding goal in software maintenance, and recent LLM-based tools have renewed interest in practical automation [1], [2]. A central step is issue localization, which occupies approximately two-thirds of the total debugging time according to empirical studies [3]. Given an issue description and a repository (with its accompanying test suite), this task aims to identify the code locations that should be modified to implement the intended behavior.

Recent solutions typically follow a direct issue–code alignment paradigm [4]–[6]. In practice, they either (i) treat localization as a retrieval task and rank code elements by similarity to the issue description [7], [8], or (ii) use a structured search procedure, often powered by an LLM agent, to traverse the repository hierarchy, invoke tools, and iteratively narrow down candidate files and functions [9]–[11]. Despite their progress, these approaches still hinge on matching requirement-level language in issues to implementation-oriented identifiers in code, making them brittle when the issue describes behavior while the relevant code is organized by technical concerns [12], [13].

Concretely, issues express behavioral expectations while code identifiers reflect technical organization. As illustrated in 1 (a), the issue requests support for “ServerlessV2” (a functional capability), yet the target function _get_db_cluster_kwargs is named after its implementation role in parameter extraction, with no indication of the feature it serves. A natural alternative would be requirement-aware localization, but explicit requirement documentation drifts from implementation and rarely maps to code precisely [14]. Tests, however, must remain synchronized with implementation to pass [15]. As 1 (b) shows, the test test_create_serverless_db_cluster shares behavioral terminology with the issue (both reference “serverless” and “db_cluster”), providing a natural intermediate target for retrieval. Executing the retrieved tests further yields execution traces that connect the test intent to the exercised implementation, including the target function. This two-hop pathway (issue \(\rightarrow\) tests \(\rightarrow\) code) can provide a more reliable evidence chain than direct issue–code matching alone. This is not an isolated case: our theoretical analysis suggests that test-driven localization reduces uncertainty by 7.73 bits on average compared to direct retrieval (2). To validate this premise, our empirical study on 18 high-quality open-source Python repositories (3) shows that existing tests cover 96.98% of ground-truth files and 66.70% of functions, while the two-hop pathway yields stronger semantic connectivity than direct matching in 82.4% of cases.

This effectiveness stems from tests’ unique position in software development: they are both human-readable specifications and machine-executable validators [16]–[18]. Each test verifies specific functionality, with identifiers that encode requirement-level semantics [19], [20], while execution traces establish deterministic links to implementation [13], [21], [22]. This duality motivates our key insight: test suites can serve as executable requirements, as test names and assertions provide a requirement-level semantic channel, while execution traces provide a concrete dynamic link from tests to the exercised code.

Leveraging test coverage for issue localization introduces two technical challenges:

• C1. Domain-Specific Semantic Gap: Retrieving relevant tests from issues requires domain knowledge absent from generic embeddings, e.g., retrieving test_tz_aware_datetime from “connecting wrong time displayed for users in different countries” requires recognizing that “tz” denotes timezone [23]–[25];

• C2. Infrastructure Noise and Trace Dilution: Localizing target code from test coverage is non-trivial, as a single test may execute thousands of functions, including infrastructure code irrelevant to the requirement. Moreover, existing tests are all passing before issue resolution, leaving no failing-test information required by spectrum-based fault localization techniques [26], [27].

To address these challenges, we propose IssueExec(Issue Executable), a test-mediated localization framework that treats test suites as executable realizations of issue requirements. In particular, IssueExec tackles C1 by retrieving candidate tests that reflect the issue intent using test representations enhanced with project-specific domain knowledge (e.g., abbreviations and API aliases) mined from historical commits. It then tackles C2 by leveraging the runtime execution traces of the retrieved tests and modeling their execution hierarchy as a trace graph, which helps filter incidental infrastructure and highlight requirement-central code locations. IssueExec combines (1) domain knowledge enhancement via historical commit mining for robust issue–test alignment; and (2) dynamic trace graph modeling for hierarchy-aware trace denoising and localization.

Together, these components construct a focused, requirement-centric search space that maintains high recall while substantially reducing context size. On SWE-bench Lite [28], IssueExec boosts the Recall@1 localization performance at the file, module, and function levels by 17.78%, 25.98%, and 41.57%, respectively. When integrated into the Agentless pipeline [9], our approach resolves 17.72% more issues, indicating benefits for downstream patch generation.

Our contributions are as follows:

• We formalize and operationalize the notion of tests as executable requirements for issue localization, showing how semantic alignment at the test level and execution grounding at runtime jointly bridge the issue–code gap.

• We propose IssueExec, a procedure-based framework that enhances test representations with domain knowledge for effective issue-test alignment and leverages execution trace hierarchy to pinpoint suspicious locations.

• We demonstrate state-of-the-art issue localization performance on SWE-bench Lite, with 17.72% improvement in end-to-end resolution when integrated into Agentless.

The source code, prompts, and additional materials are available on the artifact page [29].

2 Theoretical Motivation↩︎

We conduct a preliminary theoretical analysis and empirical validation to establish whether the test-driven localization paradigm yields meaningful uncertainty reduction over direct issue-to-code matching. We model how uncertainty evolves along the hierarchical retrieval process (issue \(\rightarrow\) tests \(\rightarrow\) trace-guided code) and validate the predicted gain under method-agnostic conditions [30]. Our goal is to quantify the uncertainty reduction achievable by the test-driven paradigm itself, independent of specific implementation choices, thereby providing principled justification for the subsequent framework design.

2.1 Formal Preliminaries↩︎

Setup and Notation. Given an issue description \(d\) and repository \(R\), let \(L\) be the universe of candidate locations (e.g., all functions/methods), with \(N = |L|\). Let \(\mathcal{G}\subseteq L\) denote the ground-truth edit set for \(d\), with \(|\mathcal{G}|>0\). For any issue localization system that returns a ranked list of locations under a predefined budget \(k_{\text{ret}}\); the resulting set is denoted as \(L_{\text{ret}}\), where \(|L_{\text{ret}}| = k_{\text{ret}}\).

For the test-driven pipeline, we retrieve a fixed-size set of tests \(\mathcal{T}_d\), based on the given issue description \(d\), with a predefined budget \(k_{\text{test}}\), i.e., \(|\mathcal{T}_d| = k_{\text{test}}\). Each retrieved test \(t \in {\mathcal{T}}_d\) has a dynamic coverage set \(\text{Cov}(t) \subseteq L\). The trace-constrained candidate subspace is the union coverage: \[L_{\text{cov}} = \bigcup_{t\in {\mathcal{T}}_d}\text{Cov}(t), \qquad N_{\text{cov}} = |L_{\text{cov}}|. \label{eq:subspace95union}\tag{1}\]

Entropy Proxy. We approximate uncertainty with Hartley entropy \(H(\mathcal{C}) \approx \log_2 |\mathcal{C}|\) [31], [32], where \(\mathcal{C}\) denotes a candidate set and \(|\mathcal{C}|\) represents the size of the search space. In our issue localization scenario, \(\mathcal{C}\) corresponds to the candidate code locations under consideration. Our objective is not to estimate the intractable distribution \(P(L\mid d)\), but to quantify the relative uncertainty reduction induced by hierarchical retrieval and trace constraints.

2.2 Quantifying Localization Uncertainty↩︎

We now formulate the localization uncertainty for two different paradigms to quantify the theoretical advantage of test-driven localization. Specifically, we define \(H_{\text{direct}}\) for the baseline approach that localizes code locations by directly matching the issue description to code, and \(H_{\text{indirect}}\) for our proposed two-stage hierarchical process that leverages tests as executable requirements to perform indirect localization via an issue-to-test and test-to-code mapping.

Direct Retrieval Uncertainty. A direct method ranks the entire space \(L\) and returns \(L_{\text{ret}}\). Let \(g_{\text{ret}} = |\mathcal{G}\cap L_{\text{ret}}|\) be the number of ground-truth locations already included in the returned set. We define a partition-based entropy proxy that accounts for whether ground-truth locations fall inside or outside the returned set:

\[\begin{align} H_{\text{direct}} ={}& \mathbb{1}[g_{\text{ret}} > 0]\cdot \Big(-\log_2 \frac{g_{\text{ret}}}{|L_{\text{ret}}|}\Big) + \mathbb{1}[|\mathcal{G}|-g_{\text{ret}} > 0]\cdot \Big(-\log_2 \frac{|\mathcal{G}|-g_{\text{ret}}}{N-|L_{\text{ret}}|}\Big) \end{align} \label{eq:direct95entropy}\tag{2}\]

Intuitively, the first term measures the ambiguity among returned candidates that contain (some of) the true edits, while the second term captures the residual uncertainty when some ground-truth edits are not retrieved.

Test-Driven Retrieval Uncertainty. Test-driven localization decomposes the search into two stages: (i) retrieving requirement-relevant tests, and (ii) localizing within a trace-constrained subspace. We model uncertainty as the sum of stage-wise entropies, accounting for realistic failure modes.

Stage 1: Test selection entropy. Among the retrieved tests \(\mathcal{T}_d\), we call a test effective if it covers at least one ground-truth location, i.e., \(\text{Cov}(t)\cap \mathcal{G}\neq\emptyset\). Let \(n\) be the number of effective tests in \({\mathcal{T}}_d\). We define: \[H_{\text{stage1}} = \mathbb{1}[n>0]\cdot \Big(-\log_2 \frac{n}{k_{\text{test}}}\Big). \label{eq:stage195entropy}\tag{3}\]

This term captures how confidently the pipeline selects tests that are functionally linked to the true edits. When \(n=0\), the pipeline fails to retrieve any effective test, and the second stage will necessarily incur a large penalty (defined below).

Stage 2: Trace-constrained localization entropy. We partition the global space \(L\) into three disjoint regions induced by the trace subspace and the returned top-\(k_{\text{ret}}\) locations. Critically, in the test-driven pipeline, the final retrieved set \(L_{\text{ret}}\) is a subset of the trace-covered space, i.e., \(L_{\text{ret}} \subseteq L_{\text{cov}}\). We define the partitions as: \[S_{\text{ret}} = L_{\text{ret}}, \qquad S_{\text{cov}} = L_{\text{cov}} \setminus L_{\text{ret}}, \qquad S_{\text{excl}} = L\setminus L_{\text{cov}}.\] Here, \(S_{\text{excl}}\) represents the excluded regions that are not covered by any retrieved test traces, as shown in 2. Let \(N_i = |S_i|\) and \(g_i = |\mathcal{G}\cap S_i|\) for \(i\in\{\text{ret},\text{cov},\text{excl}\}\). We then define: \[H_{\text{stage2}} = \sum_{i\in\{\text{ret},\text{cov},\text{excl}\},\, g_i>0} \Big(-\log_2 \frac{g_i}{N_i}\Big). \label{eq:stage295entropy}\tag{4}\] This formulation makes two realistic phenomena explicit: (i) Space compression [33]: when \(L_{\text{cov}}\) is much smaller than \(L\), uncertainty shrinks because \(N_{\text{cov}} \ll N\); (ii) Coverage/retrieval penalty: if \(g_{\text{excl}}>0\) (i.e., some true edits fall outside trace coverage), the term \(-\log_2(g_{\text{excl}}/N_{\text{excl}})\) can dominate, reflecting the high uncertainty caused by incomplete trace constraints. We refer to this condition as external failure hereafter.

Finally, the hierarchical uncertainty is: \[H_{\text{indirect}} = H_{\text{stage1}} + H_{\text{stage2}}. \label{eq:indirect95entropy}\tag{5}\]

2.3 Entropy Gain Analysis↩︎

For each issue, we define the empirical uncertainty reduction: \[\Delta H = H_{\text{direct}} - H_{\text{indirect}}. \label{eq:entropy95gain}\tag{6}\] A positive \(\Delta H\) indicates that introducing tests and traces reduces the ambiguity of edit localization compared to direct matching, under the same budget of returning top-\(k_{\text{ret}}\) locations. The decomposition further reveals two complementary directions for maximizing entropy gain: (1) improving test retrieval precision to increase \(n/k_{\text{test}}\) and reduce the external failure rate (\(g_{\text{excl}}>0\)), thereby lowering \(H_{\text{stage1}}\); (2) refining trace analysis to shrink \(N_{\text{cov}}\) while preserving \(g_{\text{ret}}\), thereby lowering \(H_{\text{stage2}}\). These insights directly inform our framework design in subsequent sections.

2.4 Theoretical Validation↩︎

We compute the above entropy proxies on 340 real-world resolved issues. To ensure a fair comparison that isolates the effect of the paradigm itself, we employ a unified embedding model for both retrieval pathways: (i) direct retrieval, which ranks code locations in \(L\) by similarity to the issue description, and (ii) indirect retrieval, which first retrieves tests by the same similarity measure, then constrains the candidate space via execution coverage. Test coverage is obtained from standard execution traces without task-specific filtering.

r0.5

The validation results are shown in ¿tbl:tab:entropy95bounds?. A consistently positive \(\Delta H\) confirms that the two-hop pathway (issue \(\rightarrow\) tests \(\rightarrow\) trace-constrained code) reduces localization uncertainty on average. Notably, even with a high external failure rate (24.41%), the overall \(\Delta H\) remains strongly positive. For issues without external failure, trace constraints yield substantial gains (\(\Delta H = 9.21\) bits) by compressing the search space. For issues with external failure, direct retrieval often fails to assign high priority to the ground-truth locations (\(H_{\text{direct}}=11.90\) bits), while indirect retrieval still achieves partial hits within the trace-constrained subspace, resulting in a positive \(\Delta H\) of 3.15 bits. These findings validate the theoretical advantage of test-driven localization and motivate the design of IssueExec, which realizes this paradigm through enhanced test retrieval and hierarchical trace analysis.

3 Empirical Study↩︎

Theoretical arguments in 2 suggest that tests can serve as executable requirements for issue localization by providing (i) requirement-level semantics and (ii) a dynamic link to exercised code. In this section, we validate this premise empirically on a large-scale dataset, and quantify when and how tests provide actionable signals for localization. We test the following hypotheses:

• H1 (Coverage feasibility). Existing tests execute the ground-truth edit locations.

• H2 (Retrievability). Tests that cover ground-truth locations are more semantically aligned with the issue than non-covering tests.

• H3 (Bridging effect). The two-hop pathway Issue\(\rightarrow\)Test\(\rightarrow\)Location provides stronger semantic connectivity than direct Issue\(\rightarrow\)Location matching.

3.1 Setup↩︎

The empirical investigation is conducted on a collection of 929 issues sourced from SWE-bench [28] and SWE-bench Gym [34], covering 18 widely-adopted Python repositories (each with \(\geq\)​500 stars). These projects represent diverse domains including web frameworks, data processing, and scientific computing. To ensure the reliability of dynamic analysis, we employ a Docker-based execution framework. Each repository-issue pair is isolated within a container where the environment is reset to its pre-patch state. We retain only those instances where the test environment is sufficiently robust, defined as having at least 50% of test functions execute successfully, to prevent environment configuration errors from confounding the findings.

Function-level coverage traces are collected via instrumentation using the Python sys.settrace mechanism. During the execution of the full test suite, the tracer records call and return events for every function invocation. We apply a filtering layer to exclude standard library calls and third-party dependencies, retaining only repository-internal functions identified by their fully-qualified paths. Ground-truth edit locations are extracted by parsing the pull request diffs associated with each issue, identifying the specific functions modified by developers to resolve the reported problem.

For semantic analysis, the bge-large-en-v1.5 [35] model is utilized to generate dense vector embeddings. Each test case is transformed into a textual representation by concatenating its docstring, its fully-qualified module path, and its complete function body. These components are truncated to 512 tokens to fit the model context window. Code locations are similarly represented by their fully-qualified signatures. Semantic similarity is then quantified using the cosine similarity between the resulting embeddings.

3.2 Findings↩︎

H1. (Coverage feasibility). 3 demonstrates the structural feasibility of using existing tests to reach target locations. By intersecting the dynamic traces of the entire test suite with the ground-truth edit sets, we observe that existing tests provide broad coverage, reaching 96.98% of files and 66.70% of functions requiring modification. This confirms that for the vast majority of issues, the necessary execution signals already exist within the repository infrastructure. Furthermore, the search space reduction is quantified by comparing the cardinality of the union of all test-covered functions against the total functions in the repository. This restriction narrows the candidate pool to 58.39% of the repository on average, achieving high recall while providing effective search space compression for subsequent localization stages.

H2. (Retrievability). For test-based localization to succeed, covering tests must be distinguishable from non-covering tests via semantic signals. Our experimental setup compares the similarity of tests that cover ground-truth locations against a baseline of randomly sampled tests from the same repository. After verifying the normality of the similarity distribution via the Shapiro-Wilk test, a paired t-test is conducted [36]. As shown in [fig:empirical95h2], covering tests exhibit significantly higher similarity to issue descriptions in 90.9% of instances (\(p < 0.001\); Cohen’s \(d = 0.56\)), indicating a medium-to-large statistical effect. Additionally, Spearman’s rank correlation analysis between the number of ground-truth locations covered and the issue similarity yields \(\rho = 0.18\) (\(p < 0.001\)). This suggests that semantic relevance can effectively prioritize tests with broader functional coverage of the requirement.

H3. (Bridging effect). Our central hypothesis is that tests serve as semantic bridges between abstract issues and technical code. For each (issue, location) pair where coverage exists, we compare the direct similarity \(s_{\text{direct}} = \text{sim}(\text{issue}, \text{location})\) against a test-mediated pathway strength. We formalize this mediated connectivity using a two-hop geometric mean formulation [30]: \[s_{\text{mediated}} = \max_{t \in \mathcal{T}_{\text{cover}}} \sqrt{\text{sim}(\text{issue}, t) \cdot \text{sim}(t, \text{location})}\] This formulation ensures that a strong bridge requires balanced semantic associations across both the requirement-to-test and test-to-implementation links. As illustrated in the results, the mediated pathway is stronger in 82.4% of cases (paired \(t\)-test, \(p < 0.001\); Cohen’s \(d = 0.93\)). The substantial effect size confirms that tests act as effective semantic relays, leveraging both requirement-level language and deterministic execution grounding to achieve connectivity that direct issue-to-code matching cannot.

3.3 Implications for IssueExec↩︎

These findings directly inform IssueExec’s design. H1 establishes that test coverage provides a high-recall, reduced-entropy search space. H2 confirms that semantic retrieval can identify
requirement-relevant tests, though the moderate effect size (\(d = 0.56\)) motivates our domain-knowledge enhancement (4.2) to strengthen Issue\(\rightarrow\)Test alignment. H3 validates the two-hop localization strategy, while the gap between covering and non-covering code within traces motivates our hierarchical trace analysis (4.4) to filter infrastructure noise.

4 The IssueExec Framework↩︎

The previous two sections provide complementary support for our central premise that tests can serve as executable requirements for issue localization: 2 motivates the premise from a theoretical perspective, and 3 validates it empirically at scale. Building on this premise, we introduce IssueExec, which operationalizes tests as an intermediate layer that connects issue descriptions to candidate code locations.

4.1 Problem Formulation↩︎

Given a natural-language issue description \(d\) and a repository \(R\) with code entities \(V\), issue localization aims to identify the code locations \(L = \{l_1, l_2, \ldots, l_k\} \subseteq V\) that need to be modified to resolve the issue [23], [37]. Formally, we seek the optimal \(L^*\) that minimizes \(|L|\) such that \(\text{Patch}(R, L) \models d\).

Guided by the preliminary study in 2, which confirms the uncertainty reduction achievable via test-driven localization, we instantiate a two-stage formulation. First, we retrieve tests \(\mathcal{T}_d\) semantically aligned with \(d\) to maximize the effective test ratio \(n/k_{\text{test}}\). Second, we leverage the execution coverage \(\text{Cov}(\mathcal{T}_d)\) to localize \(L^*\) within a trace-constrained, low-entropy search space.

Framework Overview. 4 illustrates the framework. IssueExec first performs offline preprocessing to collect test execution traces and mine commit history for enriching test representations with domain knowledge (4.2). At inference time, given an issue description, the localization pipeline proceeds as following stages:  retrieve relevant tests \(\mathcal{T}_d\) through two-phase filtering (4.3), implementing the first stage of our formulation;  analyze execution traces to identify suspicious locations within \(\text{Cov}(\mathcal{T}_d)\) (4.4);  refine and rerank candidates to produce the final \(L^*\) (4.5).

4.2 Domain Knowledge Enhancement↩︎

The semantic distance between the generic embeddings of the issue description and the tests remains substantial, as default test representations (e.g., function signatures and docstrings) fail to capture domain-specific associations and project-specific terminology. By incorporating domain knowledge \(D_t\), we bridge the terminology gap between issue descriptions and test identifiers, effectively reducing the semantic gap for requirement-relevant tests.

To this end, we extract domain knowledge from the repository’s commit history, which captures the semantic context under which tests were introduced and evolved. For each test function \(t\), we extract two types of signals.

First, we identify the commit that first introduced \(t\) via AST-based diff analysis, filtering out low-quality commits (e.g., merge commits, bulk refactoring). The associated commit message provides requirement-level semantic context: \[m_t = \text{msg}(\mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}} \{ \text{time}(h) \mid t \in \text{added}(h) \}) \label{eq:commit95msg}\tag{7}\] where \(\mathcal{H}\) denotes the commit history and \(\text{added}(h)\) returns test functions introduced in commit \(h\).

Second, we identify co-changed entities, i.e., code locations frequently modified together with \(t\) across commits: \[\mathcal{A}_t = \{ l \in V \mid \text{cochange}(t, l) \geq \tau \} \label{eq:cochange}\tag{8}\] where \(V\) is the set of all code entities, \(\text{cochange}(t, l)\) counts the number of commits in which both \(t\) and \(l\) were modified, and \(\tau\) is a frequency threshold. These entities typically represent requirement-related code that \(t\) implicitly covers.

Finally, we distill domain-specific tokens from these extracted signals to form the test-specific knowledge set \(D_t\):

\[D_t = \text{tok}(m_t) \cup \text{ident}(\mathcal{A}_t) \setminus \text{tok}(\text{sig}(t)) \label{eq:domain95tokens}\tag{9}\] where \(\text{tok}(\cdot)\) extracts semantic tokens from the commit message and \(\text{ident}(\mathcal{A}_t)\) extracts function and class names from co-changed entities. The set difference removes tokens already present in the test signature to avoid redundancy.

We construct the enriched representation by concatenating three components for each test: \[\text{repr}(t) = [\text{sig}(t); \text{doc}(t); D_t] \label{eq:repr}\tag{10}\] where \(\text{sig}(t)\) denotes the fully qualified test identifier and \(\text{doc}(t)\) is the docstring if present. By incorporating \(D_t\), domain-specific tokens provide additional semantic alignment between issues and tests beyond what generic embeddings can capture from \(\text{sig}(t)\) alone, thereby more effectively bridging the terminology gap and reducing the semantic gap for requirement-relevant tests.

4.3 Relevant Test Retrieval↩︎

This section implements the first stage of our localization pipeline. Given an issue description \(d\) and the enriched test representations constructed in 4.2, IssueExec employs a two-phase filtering process that retrieves a subset \(\mathcal{T}_d \subseteq \mathcal{T}\) with \(|\mathcal{T}_d| \ll |\mathcal{T}|\) while maximizing information extraction for downstream localization from the issue.

In the lexical filtering phase, we apply BM25 [38] to efficiently reduce the search space from potentially thousands of tests to a manageable candidate set: \[\mathcal{T}_{\text{cand}} = \text{Filter}(d, \mathcal{T}, N) \label{eq:filter}\tag{11}\] where \(\text{Filter}\) returns the top-\(N\) tests ranked by lexical similarity to the issue \(d\). This phase prioritizes recall, ensuring that requirement-relevant tests are retained.

In the semantic selection phase, we employ an LLM to carefully identify the most relevant subset from the candidates. Given the enriched representations \(\text{repr}(t)\) that incorporate domain-specific tokens \(D_t\), the LLM explicitly reasons about the issue’s intent and the tests’ purposes: \[\mathcal{T}_d = \text{Select}(d, \{\text{repr}(t) \mid t \in \mathcal{T}_{\text{cand}}\}, k) \label{eq:select}\tag{12}\] where \(k \ll N\) controls the size of the final retrieved set. The LLM is prompted to analyze which components are likely responsible for the reported issue, which test modules exercise those components, and which specific tests are most likely to cover the root cause. This produces a small set of relevant tests \(\mathcal{T}_d\) whose execution traces will guide subsequent localization. The LLM is guided by manually designed examples illustrating common pitfalls and selection criteria; the full prompt is provided on the artifact page [29].

This two-phase design strikes a balance between efficiency and quality: BM25 filtering reduces the search space from \(|\mathcal{T}|\) to \(N\) candidates efficiently, while LLM selection further refines to \(k \ll N\) high-quality tests, avoiding the prohibitive cost of applying LLM reasoning to all tests.

4.4 Trace-Guided Localization↩︎

The retrieved tests \(\mathcal{T}_d\) narrow the search space from the entire test suite to a small, requirement-relevant subset. However, test retrieval alone is insufficient for precise localization: the execution traces of these tests typically cover a large portion of the codebase, where the true edit locations constitute only a small fraction (\(|L^*| \ll |\text{Cov}(\mathcal{T}_d)|\)). While a single test may execute hundreds of functions, its execution trace preserves caller-callee relationships that encode causal structure. We leverage this hierarchical information to distinguish requirement-central code from incidental infrastructure and utility functions.

For each retrieved test \(t \in \mathcal{T}_d\), we extract its dynamic execution trace and construct a directed graph \(G_t = (V_t, E_t)\) rooted at \(t\). The nodes are functions covered by the test: \[V_t = \text{Cov}(t) \label{eq:trace95nodes}\tag{13}\] The edge set captures observed caller-callee relationships: \[E_t = \{(u, v) \mid u \xrightarrow{t} v\} \label{eq:trace95edges}\tag{14}\] where \(u \xrightarrow{t} v\) denotes that \(u\) directly calls \(v\) during execution of \(t\). Since \(G_t\) may exceed LLM context limits, we apply BFS-based pruning to retain only shallow layers of the call hierarchy. To ensure trace quality, we collect complete execution paths only from tests that execute and pass successfully, discarding traces from tests that fail or raise errors to avoid noise from engineering failures. When the issue’s root cause lies outside the coverage of retrieved tests, localization may fail, and we discuss such cases in 6.1.

We then analyze these execution traces in conjunction with the issue description \(d\) and the enriched test representations \(\text{repr}(t)\): \[S_t = \text{Analyze}(d, \text{repr}(t), G_t) \label{eq:analyze}\tag{15}\] where \(S_t \subseteq V_t\) denotes the set of suspicious locations identified from trace \(G_t\). Guided by step-by-step instructions, the LLM generates diagnostic reports that analyze potential assertion deviations, hypothesize root causes, trace error propagation paths through the call chain, and identify systematic risks across related functions.

Based on these analyses, each location \(l \in S_t\) is assigned a confidence level \(\text{conf}(l) \in \{\text{critical}, \text{high}, \text{medium}\}\). The aggregated suspicious set is: \[S = \bigcup_{t \in \mathcal{T}_d} S_t \label{eq:suspicious95set}\tag{16}\]

By analyzing execution traces with diagnostic reasoning, we aim to acquire the suspicious set \(S\), which is designed to be sparse (\(|S| \ll |\text{Cov}(\mathcal{T}_d)|\)) while maintaining high recall of ground-truth locations, filtering out infrastructure code that contributes little information about the issue’s root cause.

4.5 Refinement and Reranking↩︎

Building on the trace-guided suspicious set \(S\) from 4.4, we further refine and rerank candidates by enriching each location with additional contextual information.

The coverage-based candidate set may miss relevant locations that are structurally related but not directly covered. To address this, for each file \(f\) containing at least one suspicious location, we retrieve its structure as context. For modules exceeding context limits, we provide a compressed skeleton that preserves function signatures and class hierarchies while omitting implementation details. Here \(S'\) denotes the expanded candidate set after contextual refinement, obtained via step-by-step guided reasoning over module structure. \[S' = S \cup \text{Refine}(\{f \mid f \cap S \neq \emptyset\}, d) \label{eq:refine}\tag{17}\]

To prioritize candidates and filter false positives, we retrieve the full source code for each candidate location and employ the LLM to perform code-aware reranking, guided by a designed example showing how to filter non-fix locations and retain coupled locations requiring joint changes, reasoning about root-cause distinctions, inter-candidate causalities, and the architectural appropriateness of each contextual refinement: \[L^* = \text{Rerank}(S', \{\text{code}(l) \mid l \in S'\}, d) \label{eq:rerank}\tag{18}\] where \(\text{code}(l)\) denotes the full source code of location \(l\). The output \(L^*\) is the final ranked list of edit locations for downstream patch generation.

This refinement and reranking process ensures that \(L^*\) includes not only directly covered locations but also structurally and semantically related entities, improving recall beyond execution coverage. The reranking stage then prioritizes the most plausible edit locations, improving top-\(k\) precision.

5 Experiments↩︎

We evaluate IssueExec on SWE-bench Lite to answer the following research questions:

• RQ1 (Localization performance, 5.2) How effective is IssueExec for issue localization?

• RQ2 (Issue resolution performance, 5.3) Can improved localization benefit downstream issue resolution?

• RQ3 (Cost Analysis, 5.4) What is the cost-efficiency of IssueExec compared to existing baselines?

• RQ4 (Ablation study, 5.5) How does each component contribute to the overall performance of IssueExec?

5.1 Experimental Setup↩︎

Benchmarks. We evaluate on a widely-adopted benchmark for automated issue resolution. SWE-bench Lite [28] contains 300 issues sampled from 11 popular Python repositories, filtered for self-contained problems solvable without extensive codebase knowledge.

Baselines. We compare against three categories of methods. Retrieval-based approaches include BM25 [38], mGTE [39], CodeSage [40], and CodeRankEmbed [41]. Procedure-based methods include Agentless [9] and PatchPilot [10]. Agent-based approaches include LocAgent [4], SWE-Agent [11], OrcaLoca [42], OpenHands [43], and MoatlessTools [44].

Metrics. We evaluate localization at three granularities: file, module (class or top-level function), and function. We report Precision (Prec.) and Recall (Rec.) at cutoffs @1, @3 and @5, measuring the ability to identify ground-truth edit locations within the top-ranked predictions. We use resolved rate as the metric for the downstream repair task, and report F1-score for the ablation study.

Implementation Details. We use GPT4o-2024-05-13 and Claude-sonnet-20241022 as the base models for all methods to ensure fair comparison. For IssueExec, we set the co-change frequency threshold \(\tau\) to 3, the candidate test set size \(|\mathcal{T}_{\text{cand}}|\) to 200, and the final selected test set size \(|\mathcal{T}_d|\) to 5.

5.2 RQ1. Localization Performance↩︎

1 summarizes localization performance on SWE-bench Lite at file, module, and function granularities, reporting Precision and Recall at @1/3?/5?.

IssueExec achieves state-of-the-art performance across all granularities and metrics. Specifically, IssueExec with GPT-4o attains 70.07% file-level Recall@1, outperforming the best baseline Agentless by 17.78%. The improvements are more pronounced at finer granularities: IssueExec achieves 60.58% / 46.72% Precision@1 and 58.39% / 41.07% Recall@1 at module and function levels respectively, representing a notable boost of 25.98% and 41.57% in Recall@1 over the strongest baselines (Agentless at module level and MoatlessTools at function level); it further improves function-level Recall@3 by 23.38% over the strongest baseline. Similar gains hold with Claude, indicating that IssueExec is robust across backbone models, as test-mediated localization (issue\(\rightarrow\)tests\(\rightarrow\)code) reduces reliance on direct issue–code matching.

5.3 RQ2. Issue Resolution Performance↩︎

To evaluate whether improved localization translates to better end-to-end performance, we integrate IssueExec into the Agentless pipeline [9], replacing its original localization module while keeping the rest unchanged.

2 presents the results on SWE-bench Lite. IssueExec-augmented Agentless achieves a resolution rate of 37.67%, compared to 32.00% for the original Agentless, representing 17 additional resolved issues. This improvement demonstrates that superior @1 localization performance, particularly the advantages of fine-grained localization at the function levels, directly benefits downstream patch generation by providing the repair model with a highly focused and relevant context. Unlike agentic search, which must discover relevant tests and infer their connections to implementation through heuristic exploration, IssueExec explicitly decomposes localization into the issue\(\rightarrow\)tests\(\rightarrow\)code pathway and uses execution traces as structured intermediate evidence. By pinpointing exact edit locations rather than providing entire files, IssueExec effectively avoids overwhelming the model with extraneous code, thereby reducing potential distractions during the reasoning process; detailed examples and cases illustrating this effect are provided on the artifact page [29].

5.4 RQ3. Cost Analysis↩︎

We evaluate the cost-efficiency of IssueExec by measuring the average monetary cost per issue on SWE-bench Lite using GPT-4o, with results compared against representative baselines in ¿tbl:tab:efficiency?. Compared with representative agent-based methods, IssueExec achieves significant cost savings by reducing the average expense per issue by approximately 17.54%, 41.98%, and 46.89% relative to OpenHands, SWE-Agent, and OrcaLoca, respectively, averaging 35.47% across these baselines. Although IssueExec incurs a slightly higher cost than the procedural baseline Agentless due to the overhead of dynamic trace analysis, this modest investment is justified by its superior performance. IssueExec significantly outperforms Agentless in localization accuracy (5.2) and enables the resolution of 17.72% more issues in downstream tasks (5.3). In summary, IssueExec strikes an optimal balance between resource consumption and diagnostic effectiveness, providing state-of-the-art capabilities with much higher cost-efficiency than complex agents.

5.5 RQ4. Ablation Study↩︎

We conduct ablation experiments to understand the contribution of each component, reporting F1@3 scores across three granularities in Table [tbl:tab:ablation]. Removing the test-based intermediate entirely (w/o Tests), i.e., replacing Relevant Test Retrieval and Trace-Guided Localization with direct retrieval, causes significant degradation, with F1@3 dropping by 12.72, 9.80, and 12.94 percentage points at the file, module, and function levels. This confirms that leveraging tests as structured intermediate evidence is central to IssueExec’s effectiveness. Removing trace analysis results in a performance decrease across all metrics, with an average drop of 8.94% and specific declines of 9.25%, 9.04%, and 8.53% at the file, module, and function levels. These results emphasize that execution-based call hierarchies are essential for establishing causal links between tests and code, effectively mitigating the noise inherent in static analysis. Excluding domain knowledge leads to an average 7.37% reduction in performance. This component facilitates the alignment of issue requirements with project-specific test identifiers by leveraging historical commit data to bridge terminology gaps. The removal of the refinement stage produces a smaller average decline of 4.37%, suggesting that in the majority, the IssueExec achieves robust localization performance within the space covered by existing tests alone. Replacing the full refinement process with module-level refinement limits the search space while preserving performance near the full IssueExec configuration. This observation suggests that when operating under restricted budgets, narrowing the refinement scope can further improve system efficiency with minimal impact on localization accuracy. The substantial performance loss in the absence of reranking is attributed to the F1@3 experimental setting, as this module provides more accurate importance-based ordering of entities. Furthermore, prioritizing the most probable edit locations at the top of the recommendation list reflects a method design that aligns with the requirements of real-world software maintenance tasks.

6 Failure Analysis↩︎

While IssueExec achieves promising performance in issue localization, it still fails to identify the correct edit locations in several recurring scenarios. This section focuses specifically on these failure modes. For each failure mode, we qualitatively analyze the underlying design choice or assumption that causes the failure, illustrate it with a representative example, and discuss potential mitigation directions for future work.

6.1 Localization Failure due to Missing Test Coverage↩︎

While IssueExec’s test-driven localization paradigm achieves notable effectiveness in bridging issue descriptions with target code locations, a specific failure mode arises when the repository’s existing test suite fails to adequately cover the buggy code regions. In such scenarios, the hierarchical trace graph \(G_t\) constructed from retrieved test execution cannot propagate localization signals to uncovered code regions, manifesting as \(\text{Cov}(\mathcal{T}_d) \cap \mathcal{G} = \emptyset\), where the union of dynamic coverage from all retrieved tests \(\mathcal{T}_d\) exhibits no intersection with the ground-truth edit set \(\mathcal{G}\). This represents a boundary condition for the H1 (Coverage Feasibility) hypothesis established in 3. Even after filtering for repositories with at least 50% test function execution success (3.1), incomplete coverage persists as a structural challenge. This reflects the varying test maturity levels found in real-world codebases, an inherent constraint of the test-driven localization paradigm when applied at scale.

Specifically, as depicted in 5, although the retrieval stage surfaces tests that are semantically related to the target functionality, the selected tests \(\mathcal{T}_d\) collectively cover only auxiliary or adjacent locations, leaving the ground-truth edit method entirely absent from \(\text{Cov}(\mathcal{T}_d)\). Consequently, the trace-guided analysis can only reason over the covered locations and receives no direct execution evidence for the uncovered ground-truth method, causing it to be excluded from the suspicious set and the final localization output. This case thus exemplifies how semantic relevance at the test-retrieval stage does not guarantee execution reachability at the method level, a distinction that is invisible to retrieval-only approaches but becomes a hard boundary when trace-guided localization is constrained by incomplete coverage.

The scope of this limitation is empirically grounded in the H1 analysis of 3, which establishes that 33.30% of ground-truth functions fall outside the full test suite’s coverage, indicating that coverage gaps are a non-negligible structural risk in real-world instances. Critically, this boundary cannot be overcome by improving the semantic alignment between issues and tests (H2) or by strengthening the two-hop bridging pathway (H3), since all downstream stages are fundamentally constrained by what the selected tests execute at runtime. A natural direction for future work is to combine IssueExec with issue-to-test generation techniques [45] that synthesize bug-triggering tests directly from issue descriptions, thereby relaxing the dependency on pre-existing test coverage and extending the reach of trace-guided localization to repositories with lower test maturity.

6.2 Localization Failure under Long Call Chains↩︎

Even when the ground-truth edit location is covered by the retrieved tests’ execution traces, localization can still fail when the ground-truth node is reachable only through a deep and complex call chain. The execution path from the selected tests to the ground-truth location passes through a large number of intermediate orchestration layers, whose high invocation frequency causes them to dominate the trace graph \(G_t\) and obscure the true fix site. As a result, the localization pipeline fails to recover \(\mathcal{G}\) in \(L^*\) not due to an absence of execution evidence, but because the signal-to-noise ratio within \(G_t\) degrades with increasing chain depth, rendering the ground-truth node statistically indistinguishable from higher-level infrastructure components. This constitutes a qualitatively distinct failure mode from the coverage gap discussed in 6.1.

As illustrated in 6, the selected tests are semantically well-aligned with the issue and their execution traces do reach the ground-truth edit location through the full build pipeline. However, the trace is dominated by high-level orchestration components that are invoked far more frequently and appear structurally more prominent within \(G_t\). As a result, the contextual refinement stage retrieves class-level context around these upstream nodes, and the subsequent reranking elevates pipeline-level candidates over the precise domain-level resolver where the actual fix resides. This exposes a characteristic difficulty in trace-guided localization, where high-level pipeline components are both highly visible in the trace and semantically plausible given the issue description, making it difficult to distinguish them from the true root-cause location without reasoning over the depth and role of the covered call path.

The prevalence of this failure mode is closely tied to the architectural complexity of the target codebase. Projects with deeply layered execution pipelines, such as build systems, compilers, or documentation frameworks, are particularly susceptible, as their runtime behavior is inherently dominated by high-traffic orchestration components that sit far above the actual logic being modified. A promising mitigation direction is to incorporate call-chain depth as an explicit signal during candidate ranking, discounting locations that appear primarily as transitive callers rather than direct implementors of the relevant behavior. Complementarily, future work could leverage static call graph analysis to prune orchestration-layer noise from the trace prior to suspicious set construction, thereby improving the ability to surface deep but precise edit locations.

7 Threats to Validity↩︎

Internal Validity The effectiveness of IssueExec’s domain knowledge enhancement depends on the availability and quality of historical commit data. If a repository has insufficient commit history or maintains messy, non-atomic commit logs, the AST-based mining process may fail to extract meaningful domain tokens, potentially weakening the semantic alignment between issues and tests. While our framework implements filtering mechanisms to exclude low-quality commits, the reliance on historical signals remains a potential internal threat. Future work could involve incorporating external documentation or API references to supplement missing local history.

External Validity A primary external threat is the generalizability of IssueExec to issues that cannot be connected to runnable tests or partially executable behaviors. IssueExec is designed for the runnable and test-accessible issue localization setting, where tests can serve as executable proxies for requirements. For non-runnable issues, or cases where the issue behavior is not exercised by existing tests, the ground-truth edit locations may not be captured within execution traces. However, our empirical study suggests that even in large-scale repositories, existing tests often exercise core functional paths, providing a high-recall starting point. Future work could combine IssueExec with issue-to-test generation techniques to synthesize bug-triggering tests from issue descriptions, thereby relaxing the dependency on existing test coverage.

Another external threat is language and framework support. Currently, our implementation and evaluation focus exclusively on Python repositories and the sys.settrace instrumentation. The characteristics of dynamic tracing and caller-callee relationship extraction may differ in statically typed languages like Java or C++. While the core methodology of test-mediated localization is language-agnostic, the current lack of cross-language validation remains a threat to external validity. Generalizing IssueExec to other languages mainly requires adapting language-specific components, including test collection, execution tracing, and the language frontend, while the core issue-to-test-to-code pipeline remains unchanged. We plan to implement support for additional programming languages in future iterations.

Construct Validity The process of collecting full execution traces and constructing trace graphs can be computationally expensive for massive software systems. This overhead might raise concerns regarding the practical utility of the tool in rapid CI/CD environments. To mitigate this threat, we emphasize that trace collection is primarily an offline preprocessing step. Furthermore, this process can be optimized through incremental trace updates where only affected tests are re-executed and traced. Such engineering optimizations will ensure that the system remains scalable as the project evolves.

8 Related Work↩︎

Information Retrieval and Bug Localization. Retrieval-based approaches compute similarity between issues and code to bridge the gap between natural language and formal syntax [46], [47]. Early works relied on sparse retrievers like BM25 [38] utilizing lexical overlap. To improve accuracy, subsequent studies introduced query reformulation with contextual cues [48] or composed richer evidence sources such as code change histories, metadata, and project-specific edit patterns [49]–[52]. While effective, these enhancements introduce sensitivity to modeling choices, where configuration substantially impacts localization performance [53]. More recently, dense retrievers such as CodeSage [40] and CodeRankEmbed [41] have emerged to capture semantic relationships beyond keyword matching. This paradigm has been further adapted to specialized scenarios: CoRet [7] incorporates call graph dependencies, BLAZE [8] addresses cross-project settings, and other models target concurrent programs [54]. However, as noted in classic studies, IR-based methods remain largely static proxies for relevance and continue to struggle with the "vocabulary mismatch" problem where informal issue descriptions and formal code syntax lack sufficient overlap [55].

Traditional Fault Localization and Traceability. Our work is fundamentally grounded in the long-standing challenges of Spectrum-based Fault Localization (SBFL) and Traceability Link Recovery (TLR). SBFL techniques, such as Tarantula [26] and Ochiai [27], utilize program spectra from passing and failing tests to rank suspicious code entities. While foundational, empirical studies show that SBFL effectiveness varies significantly across scales and often requires combination with additional signals [56]–[58]. Crucially, these methods rely on pre-existing failing tests, limiting their applicability in general issue resolution where such tests are often absent. Complementary research has explored project-specific test generation for requirement validation and test generalization for broader scenario coverage [59], [60]. Traceability Link Recovery (TLR) aims to link requirements to code but remains costly to maintain and recover automatically [47], [61]. Related research in Feature Location [12] and dynamic analysis has demonstrated that execution scenarios and traces can provide direct evidence of functional behavior [62], [63]. However, interpreting these dynamic traces has historically relied on manual effort or complex static analysis. Bridging this gap by automating diagnostic reasoning on execution traces—potentially via LLMs—remains an open challenge for recovering dynamic traceability links at scale.

Procedure-based Approaches employ hierarchical pipelines to narrow the search space. Agentless [9] pioneered a three-phase paradigm, while SWE-Fixer [64] streamlines this with BM25 retrieval and reranking. BugCerberus [65] extends to statement-level localization with program slicing, and PatchPilot [10] adds reproduction and refinement stages. Related work also uses formal refinement to guide and verify LLM-generated code [66]. While efficient, these rigid pipelines are prone to irreversible error propagation, particularly when the initial retrieval step relies on suboptimal queries or configurations [48], [53].

Agent-based Approaches frame localization as sequential decision-making. SWE-Agent [11] designs an Agent-Computer Interface for navigation, while OpenHands [43] provides Docker-sandboxed environments. Multi-agent systems like MASAI [67] and SWE-Search [44] distribute tasks across sub-agents. Graph-based methods have also emerged: LocAgent [4] constructs heterogeneous graphs with multiple edge types; RepoGraph [5] supports k-hop retrieval; CodexGraph [6] enables Cypher-based querying. Despite their sophistication, these methods navigate structural rather than functional relationships, missing disconnected targets.

9 Conclusion↩︎

This paper revisits issue localization from a requirement-centric perspective and argues that test suites can serve as executable requirements that bridge the abstraction gap between natural-language issues and code. Motivated by both theoretical analysis and large-scale empirical evidence, we propose IssueExec, a test-driven localization framework that leverages domain-knowledge-enhanced test representations for effective issue–test alignment and hierarchical execution trace analysis to filter infrastructure noise and pinpoint requirement-central code.

Extensive experiments on SWE-bench Lite demonstrate that IssueExec achieves state-of-the-art localization performance across file, module, and function levels. When integrated into an end-to-end issue resolution pipeline, the improved localization quality translates into tangible gains in downstream patch generation, confirming that precise, requirement-aware localization is a critical enabler for automated software maintenance. Our in-depth analysis further reveals that the advantages of test-driven localization are most pronounced for complex, multi-file issues, while also highlighting limitations arising from incomplete test coverage and noisy execution traces.

Data Availability↩︎

The source code of IssueExec, LLM prompts, detailed analysis from the theoretical motivation and empirical study, illustrative case studies, and all other supplementary materials are available at [29].

References↩︎