1 Introduction↩︎

Code localization within large repositories is a fundamental capability for autonomous coding agents [1]–[4]. Localization accuracy has further been shown to correlate strongly with downstream task resolution rates [5], [6], making it one of the key contributors to end-to-end agent performance. More broadly, information retrieval serves as the cornerstone of agentic systems and has consequently been the subject of extensive research [7]–[9]. These agentic systems generally operate on three interacting pillars: the underlying foundation model, the agent design (prompt engineering and tool use), and the environment itself (repository contents, search indices, and documentation).

To improve an agent’s ability to navigate a specific environment, a natural approach is to fine-tune the underlying foundation model using repository-derived data [10]–[13]. While this allows the model to internalize environmental knowledge into its weights using mature optimization techniques, it inherently locks the system to a specific model. Consequently, the agent may lose access to the superior reasoning and semantic capabilities of state-of-the-art commercial LLMs, rendering this approach sub-optimal for practical, evolving applications.

Alternatively, researchers have sought to augment the agent’s environment by constructing static or reverse indices [5], [6], [8], [9] or hierarchical summaries [14], [15] and equipping the agent with sophisticated tools to interact with these structures. While these environment-centric approaches are model-agnostic by design, they remain static, lacking a mechanism to adapt dynamically to the agent’s specific retrieval challenges or to leverage data—such as human feedback or synthetic interaction traces—to continuously improve the system.

A separate line of work leverages interaction trace data, allowing agents to learn from their own failures through adversarial fine-tuning [16] or by maintaining persistent memory artifacts within the environment [17], [18]. Despite these advancements, the field still lacks a model-agnostic, data-driven framework that can actively evolve an agent’s information retrieval capabilities for a specific repository.

To bridge this gap, we introduce Libra, representing a paradigm shift in repository indexing: moving from static, external embeddings to an actively evolving, plain-text index natively integrated into the repository. Libra constructs a hierarchy of Markdown files, termed catalogs, that acts as a navigation map. Instead of relying on predefined heuristics, these catalogs are iteratively optimized through an adversarial LLM-driven training loop. A Prompter agent generates synthetic queries from random code chunks, while a Solver attempts to answer them using only the catalogs for guidance. The Solver’s navigation failures provide a rich, targeted training signal for a Healer agent to continuously refine the catalogs, aligning the index structure with how language models actually search for information.²

We evaluate Libra across the 12 Python repositories of SWE-bench Lite. Our experiments demonstrate the efficacy of this LLM optimization approach: training on synthetic queries yields significant improvements in navigation accuracy that robustly transfer to resolving real-world SWE-bench Lite issues. Furthermore, we show that the fully-trained catalogs act as a persistent, model-agnostic resource, consistently enhancing the localization performance of various downstream models. Compared to state-of-the-art index-based approaches like LocAgent and RepoMem, Libra achieves competitive performance while offering the distinct advantages of interpretability and portability.

2 Related work↩︎

2.0.0.1 Code localization and repository indices.

Repository-scale localization on benchmarks like SWE-bench is dominated by two families: agentic search over the source tree (SWE-agent [3], Agentless [19], LocAgent [5], RepoMem [6]) and fine-tuned retrievers or models trained on repository data [10], [12], [13], [20], [21]. Hierarchical or graph-structured retrievers (RAPTOR [14], GraphRAG [15], HippoRAG [22]) impose multi-resolution structure on the corpus by clustering or LLM extraction. All of these construct their index, graph, or model weights once and never repair them from query-time failures; Libra keeps the agentic-search interface but turns the index into a state variable updated by Solver failures.

2.0.0.2 Synthetic Q/A generation.

LLM-generated queries have a long history as training data for dense retrievers (Doc2Query [23], InPars [24], Promptagator [25], GPL [26]) and for bootstrapping instruction corpora (Self-Instruct [27], Evol-Instruct [28]). The closest precedent for our information-asymmetry framing is Beat-the-AI [29], where annotators exploit oracle access to write hard questions. Libra’s Prompter follows this lineage.

2.0.0.3 Self-improving agents and persistent memory.

Many systems improve an agent in place by editing its scratchpad (Reflexion [30], Self-Refine [31], STaR [32]), by growing per-agent libraries or memory stores (Voyager [33], MemGPT [34], A-MEM [35]), or by routing self-generated trajectories back into model weights (AgentTuning [36], Dr. Zero [16]); for software engineering specifically, ExpeRepair [18] and SWE-Adept [17] attach dual-memory or git-like stores to the issue-resolution agent. In all of these the memory lives inside the agent and is read-write by the agent exclusively; Libra externalizes the memory to a plain-text environment sharable across agents.

2.0.0.4 LLMs as optimizers of textual artifacts.

A growing line of work uses one LLM to optimize a textual artifact that another LLM later consumes: prompts via direct search (APE [37], PromptBreeder [38]) or via critique-as-gradient (ProTeGi [39], TextGrad [40]); solutions conditioned on score history (OPRO [41]); and entire LLM pipelines (DSPy [42]). Libra’s Healer fits this template and applies it to the realm of repository information retrieval.

3 Method↩︎

The Libra system consists of three independent, frozen LLM agents—Prompter, Solver, and Healer—and an orchestration mechanism that composes them into an LLM optimization loop. Let \(R\) denote a source repository. The system produces three artifacts:

1. Synthetic Q/A pairs. Generated by the Prompter and split into disjoint train and test sets. Training pairs drive the LLM optimization loop; test pairs evaluate the learned catalogs.

2. Failure set \(\mathcal{F}\). The subset of training queries on which the Solver fails to retrieve the correct answer. This intermediate artifact serves as the learning signal for the Healer.

3. Libra catalogs \(K\). A set of mutable, plain-text Markdown files that encode repository routing information. Once optimized, these catalogs serve as a persistent index to assist any coding agent in navigating \(R\).

In this paper, we evaluate the system on fixed repository snapshots. While the plain-text format of the catalogs naturally supports incremental updates as the repository evolves, we leave this extension to future work.

3.1 Libra catalogs↩︎

The Libra catalogs are a set of Markdown files placed at the repository root and in each submodule directory. Our hypothesis—validated empirically in Section 4—is that through iterative refinement, an LLM optimizer can encode repository routing information into these files, and that this information transfers to other coding agents, improving their localization performance. The Libra catalogs are human-readable, human-editable, and typically range from 100–1000 KB when fully populated.

3.2 Three frozen agents↩︎

We now define the three agents that constitute the Libra system. All three are frozen: their system prompts, tool configurations, and underlying model weights remain fixed across all rounds. Only the catalogs \(K\) are mutable.

All three agents are tool-using LLM agents that operate over the source repository \(R\) as a shared environment. Following standard practice, we write \(R\) as an input to each agent’s signature to denote tool-mediated read access; the role-specific tool sets (and, for the Healer, additional write access to \(K\)) are detailed in Appendix 12.

3.2.0.1 Prompter (Data Synthesizer).

The Prompter generates synthetic question–answer pairs that probe the Solver’s retrieval capability. Let \(\mathcal{C}(R) = \{c_1, \ldots, c_N\}\) denote the set of code chunks in repository \(R\), and let \(\pi(c)\) denote the unique file path containing chunk \(c\). Formally, \[\mathrm{\small Prompter}:\; (c, R) \;\mapsto\; (q, p^*),\] where \(c \sim \text{Uniform}(\mathcal{C}(R))\) is a uniformly sampled chunk, \(q\) is a natural-language question simulating a realistic user query answerable from \(c\), and \(p^* = \pi(c)\) is the gold file path.

The Prompter has full access to the sampled chunk \(c\) and may further explore \(R\) via its tools to gather surrounding context. Because the question is crafted with the answer in hand, correctness of the gold label is guaranteed by construction.³

3.2.0.2 Solver (Query Resolver).

The Solver is the coding agent responsible for resolving queries by navigating the repository. It represents the downstream consumer of the catalogs. Formally, \[\mathrm{\small Solver}:\; (q, K, R) \;\mapsto\; \hat{p},\] where \(q\) is a natural-language query, \(K\) is the current catalog state, and \(\hat{p}\) is the predicted file path.

While the Solver can freely traverse \(R\) via its tools, it has no access to the gold chunk or file path; it must locate the answer solely through repository exploration guided by the catalogs. A hypothesis which we later validate empirically is that this asymmetry in information and difficulty in task creates a gap in performance between the Prompter and the Solver, from which the Healer can harvest reliable signals to refine the catalogs.

3.2.0.3 Healer (LLM Optimizer).

The Healer refines the catalogs based on Solver failures, functioning as the optimizer in the Libra system. Formally, \[\mathrm{\small Healer}:\; (\mathcal{F}, K, R) \;\mapsto\; K',\] where \(\mathcal{F} = \{(q_i, p^*_i, \hat{p}_i)\}_{i=1}^{|\mathcal{F}|}\) is the failure set—each entry contains the query \(q_i\), the gold path \(p^*_i\), and the Solver’s predicted path \(\hat{p}_i\)—and \(K'\) is the updated catalog state. Beyond read access to \(R\), the Healer also holds write access to \(K\), which it edits in place.

3.3 The LLM optimization loop↩︎

The LLM optimization loop (also referred to as the training loop) orchestrates the three agents over \(T\) rounds. Each round consists of a testing phase, in which the Prompter and Solver generate a batch of queries and expose failures, followed by a Healing phase, in which the Healer updates the catalogs. The procedure is formalized in Algorithm 2.

4 Experiments↩︎

We evaluate Libra along five axes. First, we run an extended training trajectory on sympy to establish our core result: catalog training continuously improves code localization accuracy, yielding logarithmic returns over time (§4.2). We select sympy as our primary testbed because it is the largest repository in the SWE-bench Lite dataset, and its well-structured, evenly distributed submodules make it ideal for evaluating hierarchical catalogs. Second, we verify that this improvement pattern holds across all 12 SWE-bench Lite repositories (§4.3), which vary significantly in size and structure. Third, we test whether catalogs trained on synthetic Prompter data transfer to real-world problem distributions by replaying on a subset of SWE-bench Lite (§4.4).³ Fourth, we assess the model-agnosticism of the learned catalogs by replaying the training trajectory with three different Solver setups (§4.5). Finally, we demonstrate how individual test queries transition from failure to success as the catalog evolves (§4.6).

4.1 Setup↩︎

4.1.0.1 Data.

For catalog training, we use synthetic data generated offline by the Prompter. We evaluate in-distribution learning on a held-out test split of this synthetic data. To assess cross-problem generalizability, we additionally evaluate on a subset of the real SWE-bench Lite problem set. Because the original SWE-bench Lite instances (300 in total) each assume a different base commit, we filter the set down to 199 instances that can be evaluated on a single base commit per repository.⁴ The exact number of training, testing, and SWE-bench Lite instances chosen for each repository, along with the base commit used to rebase the SWE-bench Lite instances, is detailed in Appendix 10 (Table 5). Both the synthetic Prompter splits and the re-anchored SWE-bench Lite subset are released; see Appendix 9 for schema and license details.

4.1.0.2 Agents and Catalog.

The Solver is powered by GPT-5-mini and operates under a strict budget of max_turns=5 during training. We constrain the Solver to ensure that routing failures are plentiful, thereby providing the Healer with adequate learning signals (see Appendix 7 for a saturation analysis on frontier-tier Solvers that motivates this choice). By design, the Solver follows a deliberately minimalist recipe: it is equipped with only two general-purpose tools—Bash (restricted to ls, grep, and find) and Read (file read with line-range selection)—alongside a short system prompt that instructs it to consult the Libra catalogs. We intentionally avoid bespoke retrieval tooling, repository graphs, or commit-history indices used by prior work (cf.the 3-tool LocAgent and 7-tool RepoMem setups in the Baselines paragraph), so that any performance gain can be attributed to the catalogs themselves rather than to richer agent scaffolding. Initially, empty catalogs are created under the repository root and each Python submodule. The root catalog is always provided as context. For our cross-model generalizability evaluation, we maintain this minimalist 2-tool design but swap the underlying model to GPT-5, Gemini-2.5-Flash, or GPT-5-mini with an expanded budget of max_turns=12.

The Prompter and Healer both utilize Claude Opus 4.6. They are granted full access to the ClaudeAgentSDK tool suite⁵ and operate without any turn budget.⁶

A small set of soft heuristics, all communicated via system prompts rather than programmatically enforced, shape agent behavior. The Healer is given structural guidelines that keep catalogs terse and discriminative (a table-like, hierarchical layout with bounded bullet counts and lengths), and the Prompter is instructed to produce harder queries by avoiding exact identifiers and favoring questions that plausibly match sibling chunks. A detailed prompt-engineering study is outside the scope of this paper; the verbatim prompts are reproduced in Appendix 12.

Catalogs are initialized empty for this study. Their content and structural conventions are indirectly shaped by the Healer design.

4.1.0.3 Baselines.

We compare against two agentic localization methods: LocAgent [5] and RepoMem [6]. Like Libra, both are agentic Solvers that explore the repository through tool calls; unlike Libra, neither rewrites the environment as it runs. LocAgent replaces our Bash/Read tools with three structure-aware tools backed by a heterogeneous repository graph (directories, files, classes, functions, and their contain/import/invoke/inherit edges) plus a sparse, hierarchical entity index that falls back from exact-ID/exact-name lookups to BM25 [43] over entity IDs and code chunks. RepoMem extends LocAgent with four frozen commit-history tools that surface episodic and semantic memory mined offline from prior commits and edit frequencies. Both baselines use the same Solver model (GPT-5-mini) and turn budgets as Libra to ensure a controlled comparison; full implementation details are deferred to Appendix 8. All baseline results are averaged over multiple independent runs (5 runs for the sympy test set, 10 runs for the SWE-bench Lite evaluation), and we report the mean \(\pm\) SEM.

4.1.0.4 Metrics.

We measure accuracy by the exact match between the Solver’s prediction and the ground-truth file (File ACC) or function (Func ACC). Because the Solver is constrained to output exactly one prediction, these metrics correspond to the Top-1 accuracy (ACC@1) used in prior work [5], [20]. As file and function accuracies are highly correlated (see Appendix 6, Figure 7), we report file accuracy as our primary metric.

4.1.0.5 Replay methodology.

Each Healer step commits the updated catalog to version control, producing a sequence of snapshots that we term the training trajectory. To evaluate performance at different stages of training, we replay the held-out test set against these catalog snapshots. This approach decouples evaluation from the training loop, allowing us to assess any combination of Solver model, setup, and problem set against any historical catalog state without retraining.

4.2 Main result: extended training on sympy↩︎

We run an 85-step training trajectory on the sympy repository (the largest repository in SWE-bench Lite, with \({\sim}430\)k lines of Python code). Each step processes a batch of 400 training instances, totaling 34,000 instances over the full run. The held-out test set contains 300 instances at the same base commit. We also evaluate the baselines LocAgent [5] and RepoMem [6] on the same test set using the same model and turn budget.

Table 1 and Figure 3 summarize the end-to-end results and the full training trajectory. We observe several key dynamics. First, while the Libra Solver initially underperforms the baselines due to its lack of sophisticated tools and turn-budget efficiency, its performance quickly catches up as the catalogs build up, ultimately surpassing both baselines by a significant margin. Second, the trajectory exhibits rapid early gains: the first 5 steps (2,000 instances) lift test file accuracy from \(60.3\%\) to \(72.0\%\) (\(+11.7\) pp), capturing roughly half of the total improvement. Third, we observe logarithmic gains thereafter, with accuracy climbing in \(1\)–\(3\) pp increments from step 5 onward and plateauing near step 60 (\(80.7\%\)). Finally, train and test track each other closely; the gap between train and test accuracy stays within \({\sim}2\) pp throughout. This confirms our hypothesis that the routing information encoded in the catalog is transferable to problems unseen during training.⁷

4.3 Training on all SWE-bench Lite repositories↩︎

To assess Libra’s effectiveness across a wider spectrum of repositories varying in size, quality, and structure, we train Libra independently on all 12 repositories selected for SWE-bench Lite. Each repository uses the same configuration (GPT-5-mini, 5 turns), with batch sizes and step counts adapted to the repository’s size (see Table 5 in Appendix 10 for exact parameters).

Figure 4 confirms that every repository consistently benefits from training. The training curves exhibit the same logarithmic-gains pattern observed on sympy: the first few hundred instances capture the steepest improvement as the Healer bootstraps the empty catalogs. As training progresses, the gains continue and eventually plateau. For some repositories, we observe a slight regression during the final steps, suggesting that the catalogs begin to "overfit" to the training data. This likely occurs because chunk overlaps cause similar questions to appear repeatedly despite single-epoch training, and variable-length catalogs accumulate noise alongside useful information as training progresses.

4.4 Replay on SWE-bench Lite problem set↩︎

To verify that catalogs trained on synthetic data improve the Solver’s performance on real bug-localization queries, we evaluate the Solver across three catalog healing stages using the SWE-bench Lite problem set: Empty (step 0), Bootstrapped (step 1), and Trained (final step). We compare these results against our baselines and replicate all experiments with an extended turn budget of 10 to analyze the impact of turn limits across methods. Note that RepoMem is evaluated exclusively at 10 turns, as the LLM requires more than 5 turns to effectively utilize its 7 tools. To account for the limited number of instances in the SWE-bench Lite dataset, we average all measurements across 10 independent runs and report the mean \(\pm\) SEM.

Table 2 shows consistent gains on real SWE-bench Lite instances as the catalogs are bootstrapped and fully trained. Furthermore, once the catalogs are fully trained, the Libra Solver achieves the best performance among all methods, surpassing both LocAgent and RepoMem in most repositories despite being trained entirely on synthetic data.

4.5 Replay on different Solver setup↩︎

A central claim of our work is that training the environment is orthogonal to the choice of Solver model and design. To test this, we take the first 60 steps of the catalog trajectory produced by GPT-5-mini training and replay the test set with three different Solver setups: GPT-5-mini (the training Solver) with extended 12-turn budget, GPT-5 (a stronger sibling), and Gemini-2.5-Flash (a different model family). The catalogs are never re-trained for the new Solvers. Performance of the original trained Solver is also included as reference.

All three Solvers improve along the catalog trajectory (Figure 5). The weaker the no-catalog baseline, the larger the absolute lift: Gemini-2.5-Flash gains \(+31.0\) pp (from \(46.0\%\) to \(77.0\%\)), GPT-5 gains \(+12.0\) pp (from \(77.3\%\) to \(89.3\%\)), and GPT-5-mini gains \(+10.3\) pp (from \(81.7\%\) to \(92.0\%\)). This supports the claim that the catalog is a model-agnostic routing artifact whose value compounds with improvements in model capabilities and model cost efficiency.

4.6 Healing effect on individual instances↩︎

To isolate the steady-state gain, we trace each of the 300 test instances across all evaluation checkpoints and compute two windowed pass rates: an early window (steps 0–10, 11 evaluations) and a later window (steps 40–85, 10 evaluations).

Figure 6 (left) shows the per-instance view: \(163\) of \(300\) instances (\(54.3\%\)) improve from the early to later windows, while only \(50\) (\(16.7\%\)) worsen, yielding a \(3.3{:}1\) ratio of improved to degraded performance. The heatmap (Figure 6, right) visualizes the pass/fail trajectory of each individual instance over the course of training. It highlights a subset of instances (healed) that transition to a consistent pass rate (\(>66.7\%\)) in the later window, demonstrating the system’s ability to systematically correct its routing failures.

5 Conclusion↩︎

We introduced Libra, a framework that shifts the paradigm of agentic information retrieval from static environment augmentation and model-specific fine-tuning toward dynamic, self-evolving environments. By encoding repository routing knowledge into plain-text catalogs through an adversarial LLM optimization loop, Libra creates a persistent, model-agnostic artifact that continuously improves based on interaction failures. Our findings establish that a system trained entirely on self-generated, synthetic queries can successfully transfer its learned routing capabilities to solve real-world software engineering tasks. Ultimately, Libra demonstrates that optimizing text-based indices via interaction feedback offers a viable and adaptable alternative to traditional retrieval methods.

5.1 Limitations and future work↩︎

While our results are promising, several limitations present opportunities for future work. First, the LLM optimization loop is computationally intensive; exploring more efficient Healer designs and catalog structures could reduce this overhead. Second, our evaluation assumes static repositories. Because real-world codebases evolve, developing mechanisms for versioning and incremental catalog updates is crucial for practical deployment. Third, the current Prompter primarily generates single-hop queries; synthesizing high-quality, multi-hop queries would better reflect real-world complexity and provide a stronger training signal. Finally, while we focused on software engineering benchmarks using a specific three-agent architecture, extending Libra to other domains (e.g., general documentation) and exploring the broader design space of agent interactions remain important directions for future research.

6 File vs.function accuracy correlation↩︎

Figure 7 reports test-set file and function accuracy along the sympy training trajectory of §4.2 (GPT-5-mini Solver, \(\texttt{max\_turns}=5\)); the two curves move together across every step. Note that the sympy train and test splits were generated by two slightly different Prompter variants (footnote 5), so the file/function gap visible here is a property of the test-time gold annotations and not of the training signal. Identical correlation behaviour holds on the other \(11\) repositories, where train and test are produced by the same Prompter; we report file accuracy as our primary metric throughout the paper.

7 Saturation analysis on recent models↩︎

We chose GPT-5-mini at \(\texttt{max\_turns}=5\) as the training-time Solver after observing that frontier models saturate the benchmark even with no catalog. Table 3 reports the no-catalog baseline for several recent models on the same \(300\)-instance test set, all run through the Libra Solver agent at \(\texttt{max\_turns}=20\). Above \(\sim 85\%\) file-level accuracy, methodological gains from training become hard to distinguish from noise, and the portion of failures that are due to the model giving an alternative valid fix becomes significant. We therefore picked a model for which a tightly-budgeted (\(5\)-turn) configuration still leaves substantial headroom on this test set.

8 Baseline implementation details↩︎

This appendix expands on the LocAgent [5] and RepoMem [6] baselines summarized in §4.1. Both share Libra’s overall agentic-localization setup (same Solver model, same turn budgets, same gold targets) but differ in the retrieval substrate the Solver is given: LocAgent exposes a structure-aware repository graph in place of Bash/Read, while RepoMem layers an additional set of frozen commit-history tools on top of LocAgent. Crucially, neither substrate is rewritten during the run, which is the orthogonal axis Libra adds.

8.0.0.1 LocAgent.

LocAgent builds a directed heterogeneous graph over the repository (nodes: directories, files, classes, functions; edges: contain, import, invoke, inherit) and layers a sparse, hierarchical entity index on top of it: (1) an entity-ID lookup keyed by fully qualified name, (2) an entity-name dictionary keyed by short name (with case-insensitive and Class.method splitting fallbacks), (3) a BM25 inverted index [43] over entity IDs as a fuzzy fallback when (1)/(2) miss, and (4) a BM25 inverted index over each entity’s code chunks for keywords that never appear in any ID (e.g.global variables). The Solver receives three tools in place of Libra’s Bash/Read:

• SearchEntity — keyword search through the four-layer cascade above; returns matching entities with their source at one of three detail levels (fold / preview / full) chosen by match count.

• TraverseGraph — type-aware multi-hop BFS along contain/import/invoke/inherit edges.

• RetrieveEntity — full source retrieval for a chosen node.

We use LocAgent’s reference repository defaults: each BM25 index is queried at similarity_top_k=10, with display further capped at the top 5 hits per query (and at top 3 for non-file/-directory entities in the entity-ID layer). When the entity-ID BM25 layer returns nothing, a small rapidfuzz token-set retrieval (top-3) is consulted before falling through to the content-BM25 layer.

8.0.0.2 RepoMem.

RepoMem extends LocAgent with four commit-history tools that expose frozen memory mined offline from the repository’s git log:

• SearchCommit, ExamineCommit — episodic memory, BM25-matched against commit messages from a window of \(7{,}000\) prior commits.

• ViewSummary, SearchSummary — semantic memory, LLM-generated summaries of the \(K{=}200\) most frequently edited files.

The memory itself is built once before evaluation and never updated by the Solver; failures cannot feed back into the index. This contrasts with Libra, where the catalog is the artifact under training and is continuously rewritten by the Healer in response to Solver failures.

9 Release artifacts↩︎

We release code and Prompter data to facilitate reproducibility. The top-level README.md contains paste-ready commands for every workflow. The code and data are released under the CC BY NC 4.0 License, see README and LICENSE for details.

9.0.0.1 Code.

• agents/ — the three frozen agents (Prompter, Solver, Healer) and their shared Read/Bash tools. Each role has an OpenAI-compatible backend (*_plain.py) and a ClaudeAgentSDK backend (*_ant.py).

• orchestrator/ — the training loop (train.py, implementing Algorithm 2) and the single-shot replay evaluator (evaluate.py), together with their YAML configs (train_config.yaml, eval_config.yaml) which expose every hyperparameter used in the experiments.

• scripts/init_catalogs.py (seeds the empty catalog.md files the Healer rewrites) and scripts/replay_test_eval.py (replays the test set against any historical catalog snapshot).

• llm.py (LLM client), pyproject.toml + uv.lock (pinned environment), .env.example, README.md.

9.0.0.2 Data (\(13\) configs, \(\approx\!86\) MB Parquet).

• prompter_<repo> (\(12\) configs, one per SWE-bench Lite repository) — synthetic Q/A pairs from the Prompter at the per-repo base commit listed in Table 5. Each config has a train and a test split . See Table 5 for the per-repository instance counts (\(90{,}736\) train and \(3{,}287\) test in total). Schema: (instance_id, problem_statement, gold_files, gold_functions, gold_reasoning, chunk_content, line_numbers, is_valid_chunk).

• SWE-bench_Lite_Libra (\(1\) config, \(199\) instances) — the \(199\) SWE-bench Lite bug reports that re-anchor on a single base commit per repository (Table 5), augmented with gold_files/gold_functions derived from the ground-truth patches.

10 Training cost breakdown↩︎

Table 5 reports the one-time Prompter data-generation cost and the per-epoch training cost for each of the 12 SWE-bench Lite repositories. The Prompter and Healer both run Claude Opus 4.6; the Solver runs GPT-5-mini at 5 turns. We split the training cost into Train-Eval (Solver evaluation on each training batch) and Healer (catalog-rewrite proposer + writer agents); their sum is the total Training cost reported in the rightmost column. Test-set evaluation cost (the periodic test-eval that runs every \(N\) batches during training, plus any standalone test-eval) is excluded. Every repository trains for exactly one epoch. All dollar figures in this section are computed from the per-token list prices in Table 4.

^\(*\)Sympy Prompter data generation predates per-call cost instrumentation; the value is estimated by extrapolating from the other repos.

10.0.0.1 Zoom: per-step cost on django.

Figure 8 unpacks the django row of Table 5 step-by-step. The root catalog is always provided in the system prompt, so as it grows, we notice that the Solver’s input token grows accordingly. However, the cache-hit rate also climbs as the root catalog stays the same across queries within a batch. As a result, the Train-Eval cost stays relatively flat. Healer cost dominates the bill as we’re using the higher-end Claude Opus 4.6 model to rewrite the catalogs.

11 Training dynamics: additional case studies↩︎

For reference, we walk through two healed-stable test instances from the top-left cluster of Figure 6. For each instance the trajectory string records the evaluation outcome on that instance every \(5\) training steps (at steps \(\{0,5,\ldots,85\}\)), T for a file-level pass and F for a fail, with bars separating the early (\(0{-}10\)), transition (\(15{-}35\)) and late (\(40{-}85\)) windows. The trigger and diff we list are the training-batch failure and the resulting catalog.md edit that we believe fixed the instance stably going forward.

1.prompter_173 — _solve_system (nonlinear path vs.linear-path trigger)↩︎

Test. Gold solvers/solvers.py::_solve_system; trajectory FFF|FFTFF|TTTTTTTTTT; question: “When iteratively solving a system of equations symbol by symbol, what happens if a candidate solution for one variable contains references to variables that were already determined in earlier iterations?” (the nonlinear branch).

Trigger (heal step 20). prompter_8387; same gold function but a different sub-path: “How does the system of simultaneous first-degree symbolic equations get converted into an augmented coefficient table before being dispatched to a linear-system resolution routine?” (the linear branch). Solver predicted solvers/solveset.py::linear_eq_to_matrix.

Diff (solvers/catalog.md).

- `_solve_system(exprs, symbols, **flags)` (L1631-L1829) -
-   Solves systems of equations. For nonlinear polynomial systems
-   with more unknowns than equations, enumerates subsets of free
-   symbols (sized to match equation count), calls
-   `solve_poly_system` on each subset, ...
+ `_solve_system(exprs, symbols, **flags)` (L1631-L1829) -
+   Solves systems of equations.
+   - Linear path: when all poly expressions are first-degree,
+     constructs an augmented coefficient matrix (n x (m+1)) by
+     iterating monomial terms, placing each degree-1 coefficient at
+     the corresponding slot ...
+   - Nonlinear path: for polynomial systems with more unknowns than
+     equations, enumerates subsets of free symbols (sized to match
+     equation count), calls `solve_poly_system` on each subset,
+     and discards candidate solutions that re-introduce already-
+     eliminated symbols.

The trigger was about the linear branch; the test query lands on the nonlinear branch via the same now-richer entry.

2.prompter_157 — lib_interval.cos via wholesale interval-math enumeration↩︎

Test. Gold plotting/intervalmath/lib_interval.py::cos; trajectory FFT|TTTTT|TTTTTTTTTT (a clean flip from no useful prediction at step 5 to steady-state from step 10); question: “I’m getting incorrect results when evaluating the cosine of a plain numeric value (not a range) through the interval math plotting module. The output looks like it’s returning the sine value instead. Where is this likely implemented?”

Trigger (heal step 5). prompter_2097 and the near-duplicate prompter_2279; both have gold plotting/intervalmath/lib_interval.py::cosh (different function, same file): “When computing the hyperbolic cosine over a range that crosses zero (e.g., from a negative to a positive value), how does the interval arithmetic library determine the lower bound of the result?” Solver predicted core/expr.py::_eval_interval for the first and functions/elementary/hyperbolic.py::Cosh._eval_interval for the second — both wrong files in the wrong submodule.

Diff (sympy/catalog.md, root catalog).

- `intervalmath/` - Interval arithmetic engine (`interval()`)
    for adaptive implicit-plot sampling; used by `plot_implicit`.
+   - `lib_interval.py`: interval-aware math functions (`sin`,
+     `cos`, `cosh` with zero-crossing minimum detection, `exp`,
+     `log`, `atan`, etc.).

Even though both trigger queries asked specifically about cosh’s zero-crossing minimum logic, the Healer’s edit was written at the file level: a single bullet enumerating the interval-aware math functions in lib_interval.py, including cos verbatim. The test query about cos sin/cos confusion in the interval-math module then flips on the very next evaluation step (heal step \(5 \to\) test step \(10\)) purely because the catalog now mentions lib_interval.py::cos by name.

12 Agent prompts↩︎

This appendix reproduces verbatim the prompts used for the three frozen agents in our main experiments.

12.1 Prompter↩︎

The Prompter receives a \(\sim\!100\)-line code chunk together with a randomly-sampled additional rule and emits a JSON object containing a developer-style question, a file::function answer, and a boolean is_valid_chunk flag (used to drop pure-boilerplate chunks).

12.1.0.1 System prompt (prompter_system).

You generate evaluation Q/A pairs from code chunks. Each pair tests whether an agentic RAG system can locate the correct file and function for a developer's query based purely on semantic understanding, NOT keyword matching.

The chunk you receive is the ground-truth location. Your job: write a question a real developer would ask that this chunk answers, then give the answer as a file::function locator.

### QUESTION STYLES (vary across these)
- "How does the system handle [concept]?" - targets a specific mechanism
- "I got [error/symptom] when [operation]" - bug report / debugging
- "How to support [use case] in [area]?" - feature extension
- "Where is [behavior] implemented?" - localization

### RULES
1. **No self-reference.** Never say "in this snippet", "the provided code", etc. Write as if you're a developer querying a codebase.
2. **Ground the question in the enclosing file.** You may explore beyond the provided chunk - read surrounding code in the same file, or read other parts of the repo for context. However, the final question must be answerable by the file that contains the provided chunk.
3. **Target core logic, not names.** Don't just ask "what does this code do?" - ask about a specific detail, edge case, or mechanism inside it.
4. **NO EXACT IDENTIFIERS (CRITICAL).** You must NOT use the exact names of classes, functions, variables, or files in your question. Instead of asking "How does the Wavefunction class determine limits...", ask "How does the system represent quantum states when determining coordinate boundaries...". Force the evaluating agent to use semantic search.
5. **Dead chunks.** If the chunk is pure boilerplate (only imports, whitespace, closing brackets) with no meaningful logic, set is_valid_chunk to false.
6. Strictly follow the additional rules provided by the user if any.

### ANSWER FORMAT
Use the most specific locator visible in the chunk:
- Top-level function: `filepath::function_name`
- Method on a class: `filepath::ClassName.method_name`
- If multiple functions are relevant, pick the primary one.

### REASONING (think step by step before generating)
1. Explore the chunk's enclosing file and module and write a brief summary of the functions of the chunk, file and module in the repo.
2. Identify the key elements in the chunk: functions, classes, logic branches, comments, edge cases.
3. **List Forbidden Words:** Explicitly list the exact class names, function names, and highly specific variable names found in the chunk. You are banned from using these in the question.
4. Pick the most interesting or non-obvious aspect - a bug, an edge case, a design choice, a specific behavior.
5. Write a question targeting that aspect using conceptual synonyms instead of your forbidden words.

### TOOLS
When using tools (Read, Grep, Bash, etc.) to explore the codebase, always use
**relative paths** (e.g. `core/expr.py`), never absolute paths. The working
directory is already set to the repository root.

12.1.0.2 User template (prompter_user_template).

Filepath: $filepath
Lines: $start_line-$end_line

--- CHUNK ---
$chunk_content
--- END CHUNK ---
--- ADDITIONAL RULES ---
$additional_rules
--- END ADDITIONAL RULES ---

Generate an evaluation Q/A pair for this chunk. Output raw JSON only - no markdown fences, no extra text.

{
  "reasoning": "Step-by-step reasoning following the system prompt guidelines.",
  "is_valid_chunk": true,
  "question": "A realistic developer query.",
  "answer": "filepath::Class.method or filepath::function",
}

12.1.0.3 Additional rules.

At each call we sample one rule uniformly at random and substitute it into $additional_rules. The two rules used in our runs are:

# rule_banDomainVocab
**NO MODULE/DOMAIN VOCABULARY (CRITICAL).** Do NOT use words that map directly to module or directory names in the codebase. Banned terms include (but are not limited to): $ban_domain_vocabulary. Any word that would return hits if you `grep`-ed it against the codebase's docstrings or comments is BANNED. Replace concrete library jargon with its abstract formal equivalent.
**Examples:**
Instead, describe the concept abstractly or from a user-behavior perspective.
**Examples:**
- BAD: "How does the geometry module compute triangle area?"
  GOOD: "How does the system compute the area of a three-sided planar figure?"
- BAD: "Where is polynomial GCD implemented?"
  GOOD: "Where is the greatest-common-divisor algorithm for symbolic algebraic expressions implemented?"
- BAD: "How does the pretty printer handle matrix display?"
  GOOD: "How does the human-readable output formatter render two-dimensional tabular data structures?"

# rule_preferEdgeCase
**PREFER EDGE CASES OVER PRIMARY PURPOSE (CRITICAL).** Your question should target a specific conditional branch, edge case, error-handling path, or special-case logic visible in the chunk - NOT the main/obvious purpose of the code. Focus on if/else branches, try/except blocks, boundary checks, type-specific dispatches, or fallback behaviors.
**Examples:**
- BAD: "How does the matrix constructor work?"
  GOOD: "What happens when the builder receives an entry that is itself a two-dimensional array rather than a scalar?"
- BAD: "How does the parser tokenize input?"
  GOOD: "How does the tokenizer recover when it encounters an unterminated string literal at end-of-file?"
- BAD: "Where is the caching layer implemented?"
  GOOD: "What fallback behavior triggers when the cache store reports a connection failure mid-read?"

12.2 Solver↩︎

The Solver receives only the problem statement as the user message; the root catalog.md is inlined into the system prompt. We report results with \(\texttt{top\_k}=1\) in the main paper; the \(\texttt{top\_k}=5\) variant differs only in output schema.⁸

12.2.0.1 System prompt (locator_plain_system_template).

You are a bug-localization agent. Given a problem statement, explore the
repository to find the single most relevant file and function that would
need to be edited to fix the issue.

The repository contains catalog.md files at the project root and within
each module/sub-package. These catalogs summarize the contents and purpose
of their respective directories. The root catalog.md is provided below. Use it to orient yourself, **then drill into module-level catalog.md files as needed**.

## Root catalog.md
$catalog

## Tools available
  - read: read a file by its path (relative to the repo root). Optionally
specify start_line and end_line to read only a portion of the file.
  - bash: run a shell command. ONLY ls, grep, and find are permitted.

## Output format

When you are confident, respond with ONLY a JSON object (no extra text):
{"file": "<relative path>", "function": "<Class.method or function_name>", "reasoning": "<one sentence>"}

12.3 Healer↩︎

The Healer receives one target catalog file and a batch of Solver failure reports whose target_md was assigned to that file by a deterministic path-prefix match. It diagnoses each failure, optionally drops it, and applies all surviving edits in-place via Read/Edit/Grep/Bash tools.

12.3.0.1 System prompt (healer_system_template).

You are a catalog-healing agent. You will receive a batch of failure reports
from a code-localization system - cases where the Solver predicted the wrong
file for a given **Problem Statement**. Each failure also includes a **Gold
Reasoning** - the correct chain of thought that would have led to the right
answer. Your job is to diagnose each failure, identify those that pinpoint a
gap in the catalog, and improve the catalog in-place to close that gap.

You'll be working under the directory $cwd.

# Catalog Spec (the artifact you are healing)
A catalog file is a navigation aid for the Solver: terse, definitive prose
that lets it pick the right file/module without reading the source.

Structure:
- **Module catalog**: top-level sections group `.py` files by role. Each
  `.py` file (linking to source) is a sub-header inside its section.
- **Root catalog**: same shape, but each **submodule** plays the role of a
  `.py` file - one sub-header per submodule with a summary and link to its
  module catalog.
- Optional **Glossary**, **Architecture Overview**, and **Notes** sections
  at the top.
- Test files are excluded.

Per-entry content (file or submodule sub-header):
1. A one-line summary of what the file/submodule does.
2. Functions/classes with **adaptive verbosity** - match detail to complexity:
   - Trivial/obvious - omit entirely.
   - Simple helpers/utils - list name or declaration only.
   - Moderate complexity - name + short summary.
   - High complexity - bullet list of key functionalities, logic flow,
     components, etc.
   Each listed section (class, function, method, etc.) MUST be annotated
   with its source line range in the form `(L<start>-L<end>)` immediately
   after the name/declaration (e.g. `process_batch (L42-L88)`). Verify ranges by reading the source file - do not guess.
3. (Optional) brief "Caveats" note for surprises or easy-to-misuse behavior.

Fixes must **strictly** follow these constraints:
- Be concise.
- Be definitive, not descriptive (assert what something IS, not how it works).
- Respect adaptive verbosity - don't promote trivial entries to verbose ones.
- Only edit the target catalog file specified in the user message.
- Never create new .md files or delete existing ones.
- Any added or modified content must conform to the Catalog Spec above
  (required structure + per-entry content). Do not invent new structural
  conventions that deviate from the spec.
- Do NOT cross-reference entries - perform Sharpen Entry instead.
- Do NOT include implementation details.
- Do NOT include examples.

Maintain these catalog invariants:
- Each bullet/line <= 250 characters; if longer, split into multiple bullets.
- Each section <= 20 bullet points; if more, merge and purge to <= 15 bullets
  keeping only the most discriminating information.

If a failure can only be fixed by violating the above, drop it. Generalizability beats completeness.

You will be given ONE catalog file to focus on. Follow these steps IN ORDER.
Use tools (Read, Edit, Grep, Bash, etc.) freely.

# Steps
## Step 1 - Gather evidence
For each failure in the batch:
- Study **Gold Chunk** - the code the Solver should have found. Optionally
  read the full **Gold File** for more context.
- Read **Predicted File** - what the Solver chose, to see why it looked
  plausible.
- Compare **Gold Reasoning** to **Prediction Reasoning**. Identify precisely
  where the Solver diverged.

## Step 2 - Triage: is the catalog actually to blame?
Before diagnosing, decide whether the catalog is responsible. Drop the
failure (no edit, count as dropped) if ANY of the following hold - these are
**not** catalog issues:
- **Solver reasoning error** - the catalog context was adequate, but the
  Solver failed to reason about or break down the problem correctly.
- **Ambiguous problem statement** - the question is underspecified or
  misleading; no catalog change could have deterministically routed the
  Solver to the gold target.
- **Intra-file confusion** - the problem hinges on implementation details
  inside the file that aren't tied to its high-level role.
- **Out-of-scope target** - the gold file is a test file or otherwise
  excluded from the catalog spec.
- **Solver gives correct alternative** - the predicted file can fix/answer the problem
  just as well as the gold file; it's essentially an alternative answer.
Skip the rest of the steps if the failure is not catalog-attributable.

## Step 3 - Diagnose root cause
For each remaining (catalog-attributable) failure, pick one or more of:
- **Information Gap** - the catalog entry for the gold target is missing the
  key signal (function/class/role) that would have anchored the Solver.
- **Catalog Noise** - the entry is too verbose/over-detailed; the relevant
  signal is buried. Often means verbosity is set too high for the complexity.
- **Narrow Separation** (common) - the gold entry and a confusable sibling
  entry don't distinguish their functionalities clearly.
- **Structural Issue** - entries are mis-grouped, mis-sectioned, or the
  catalog lacks the sections/glossary/caveats the Solver needed.

## Step 4 - Propose fixes
Choose fixes that target the diagnosed root cause. Examples (not exhaustive):
- **Enrich Entry** - add the missing signal so the gold target is
  distinguishable from commonly confused alternatives.
- **Compact Entry** - drop the entry to a lower verbosity tier (e.g. moderate
  -> simple, or high -> moderate); strip implementation noise.
- **Sharpen Entry** - rewrite definitions of similar entries so each one
  asserts what it uniquely owns.
- **Reorganize** - regroup entries into sections that better reflect role.
- **Meta Change** - add/modify glossary, caveats, or appendix entries when
  the disambiguating signal is cross-cutting.

## Step 5 - Apply fixes
Apply all necessary changes to the target catalog file in-place using your
editing tools. Combine insights from multiple failures - if several failures
point to the same gap, make one coherent fix rather than redundant edits. If
there are >5 fixes required, use the agent tool to parallelize fixes with
subagents.

12.3.0.2 User template (healer_user_template).

Target Catalog File: $target_md

Below are $n_failures localization failure(s) where Target Catalog File may be
relevant. Diagnose each one and apply any necessary fixes to the target file.

$failures_block

12.3.0.3 Per-failure template (failure_template).

The user message embeds one block per failure, formatted as:

--- Failure $failure_idx ---
Instance: $instance_id

Problem Statement:
$problem_statement

Gold File:      $gold_files
Gold Function:  $gold_functions

Gold Chunk (lines $line_numbers):
$chunk_content

Gold Reasoning:
$gold_reasoning

Predicted File:     $pred_file
Predicted Function: $pred_func

Prediction Reasoning:
$reasoning

References↩︎

---

1. Corresponding author.↩︎

2. In our implementation, we execute the Prompter separately to generate the complete training dataset upfront. The Solver then processes these questions in batches, and the Healer analyzes the aggregated failure signals from each batch to update the catalogs.↩︎

3. Strictly, correctness holds to the extent that the Prompter LLM generates a question that is genuinely answerable from the given chunk and not from other locations in the repository. In practice, we observe this assumption to be reliable enough for the precision of this research.↩︎

4. Gold files and functions are derived from each instance’s ground-truth patch and validated against the base commit’s file tree. We exclude instances where (1) the referenced file or function does not exist at the base commit, (2) the referenced file or module has been moved or renamed between the instance’s original commit and the base commit, or (3) the instance requires multi-hop patching spanning multiple files.↩︎

5. Across all training runs, the active tools observed in the agents’ trajectories are Bash, Read, Write, Edit, Glob, Grep, Agent (subagent dispatch), and StructuredOutput (final-answer capture). See the Claude Agent SDK documentation at https://docs.claude.com/en/agent-sdk/overview.↩︎

6. The Prompter used to generate the sympy training set is a slightly earlier variant: it also runs Claude Opus 4.6 but is equipped only with Bash and Read tools rather than the full ClaudeAgentSDK.↩︎

7. Because we train for only one epoch, every training batch is itself new to the catalog, so even training accuracy measures generalization. We nonetheless maintain a fixed held-out test set for additional clarity.↩︎

8. In the released code, the Solver is named Locator (visible in the template identifier locator_plain_system_template below). The two names refer to the same agent.↩︎