Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity
July 15, 2026
Abstract
An LLM agent’s real-task performance is shaped as much by the harness around its model as by the frozen model itself: its prompts, injected knowledge, runtime control, and configuration. In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights. The hard part is not generating changes but knowing which one truly helped. Self-generated feedback is noisy, and an apparent gain can be a measurement artifact or an edit that merely overfits the tasks it was tuned on. We present a self-evolving agent-harness framework that separates proposing changes from crediting them: a language model diagnoses failures and proposes patches, while all sampling, measurement, and significance testing are owned by deterministic code, so every credited improvement is trustworthy by construction. Patches populate a gated, categorical quality-diversity archive (GSME) keyed on the (where \(\times\) why) pathology an edit addresses rather than the tasks it fixes, an anti-overfitting inductive bias; generalization is measured on a sealed test scored only after evolution. Across seven domains with a frozen open-weight model, the harness is train-selected and scored once on a sealed test; its credited gains there are \(+9\) to \(+15.5\) pp and retain 86–147% of the training gain, evidence they generalize rather than overfit. The winning patch tracks the model’s dominant pathology, not its size or family: changing the model can change the pathology and the patch, while the same pathology-to-patch match recurs across two model families. What transfers is the diagnose-and-credit loop, not any specific harness.
1 Introduction↩︎
An LLM agent’s performance on real tasks is shaped as much by the harness around its model as by the model itself: the same frozen model, well- or poorly-harnessed, differs sharply in what it accomplishes. By harness we mean the full runtime scaffolding, not only the system prompt targeted by prompt optimization [1]–[4], but the injected knowledge, the control loop with its hooks and recovery, the tools, and the configuration. This harness is the lever that elicits a given model’s best performance, and in deployment it is often the only lever available, since the weights are fixed (rented, closed, or too costly to retrain). This scaffolding is nonetheless hand-engineered by experts, which scales poorly as models proliferate and turn over. So it is natural to ask whether an agent can improve its own harness automatically, by diagnosing its own failures and editing its own harness in a closed loop.
A 2026 line of work does exactly this, auto-evolving coding-agent harnesses and reporting gains [5]–[8]. But these systems typically credit an improvement by the peak pass rate over many rounds (which inflates gains by the noise it maximizes over), usually on a single task set with no held-out split [9], [10]. Such protocols cannot tell a harness that genuinely improved from one that got luckier or overfit; the reported numbers are not auditable.
Two difficulties make this hard. First, the hard part is not generating changes, which a capable model does endlessly, but knowing which one truly helped: self-generated feedback is noisy, since a single evaluation swings by several points on borderline tasks, an apparent gain can be a measurement artifact (a flaky sandbox, a verifier timeout), and a training-set gain may merely overfit, so a loop that trusts it drifts rather than compounds. Second, even trustworthy feedback leaves what to change open: the edit space is large, and a greedy loop degrades two ways, collapsing onto a few safe edits (usually prompt tweaks) once bolder changes fail [9] and overfitting patches to the specific tasks they were tuned on rather than to a failure mode that generalizes. We therefore ask: under what conditions is an agent’s self-generated feedback reliable enough, and its search structured enough, that harness improvements compound and generalize rather than drift or overfit?
Against these two difficulties we make two innovations, realized in a system that pairs a frozen task agent with a separate evolver (a stronger model) that diagnoses the agent’s failures and edits its harness while the agent stays frozen (Figure 1). First, to make self-generated feedback trustworthy, we separate proposing from crediting: the evolver proposes semantic (where \(\times\) why) patches, but every credit decision is made by deterministic code, a validity gate (re-run infrastructure failures), an activation gate (the patch actually fired), and a paired-\(2\sigma\) significance test, on a sealed test withheld from all decisions, so a credited gain is trustworthy by construction; because every scored comparison runs on the frozen agent and is paired, the split does not confound the gains (§3.2.0.5). Second, to search what to change without overfitting or collapsing, we organize edits as Gated Semantic MAP-Elites (GSME): a categorical quality-diversity archive keyed on the (where \(\times\) why) pathology an edit addresses (not the tasks it fixes), with one gated elite per cell and cross-cell recombination, so keying on pathology is an anti-overfitting bias and per-cell diversity resists collapse.
Across seven domains from three sources (terminal-bench-2, the EvoAgentBench suite, and AppWorld), the train-selected harness’s credited sealed-test gains are \(+9\) to \(+15.5\) pp, retaining 86–147% of the training gain; the same paired test also flags and rejects harmful candidates. On AppWorld, evolving across three models spanning two families shows that the winning patch tracks each model’s dominant pathology, not its size or family. Our contributions:
An auditable self-evolution loop that separates proposing from crediting. A complete, reproducible loop over a git-tracked evolution tree: the evolver owns semantics, while deterministic code owns all sampling, measurement, and significance testing, so every credited improvement is reproducible and attributable.
Gated Semantic MAP-Elites (GSME), and evidence it earns its keep. The pathology-keyed quality-diversity archive above, plus its payoff: on four of six domains the credited harness is a cross-cell recombination the archive makes available, and accepted edits span all four levers rather than collapsing to prompt edits alone.
Held-out evidence that the gains generalize. Credited sealed-test gains of \(+9\) to \(+15.5\) pp retain 86–147% of the training gain on every credited benchmark, a positive result where the harness-optimization literature documents overfitting [9]–[11].
Evolved harnesses are model-specific; the loop is what transfers. The dominant diagnosed pathology and the winning patch change with the model, a pathology-to-patch matching law: engagement\(\to\)verify-finalize on one model, careless\(\to\)submit-verify on two others, the latter replicating across the Qwen and Google families (§4). This reframes harness self-evolution as fitting a corrective patch to a specific model’s failure distribution rather than discovering universally good scaffolding, and locates the transferable contribution in the diagnose-and-credit loop itself.
2 Related Work↩︎
Prior work bears on the three questions our loop must answer: what to evolve around a frozen model, how to know a proposed change truly helped, and how to search the space of changes. We take them in turn.
2.0.0.1 What to evolve.
A frozen model can be improved by editing the scaffolding around it: prompts, tools, memory, control loop, configuration [12], [13]. Prior methods either search prompts and workflows [1]–[4], [14]–[16] or let an agent rewrite its own code against an open-ended archive [17]–[20]. A parallel line evolves the agent’s context or memory rather than its code (agentic context engineering, reasoning/experience memory, and self-evolving memory architectures [21]–[25]), which our knowledge lever subsumes as one editable surface among four. GEPA [4] is the closest on efficiency, evolving prompts on a Pareto front via natural-language reflection with up to \(35\times\) fewer rollouts than RL; like the rest of this line, it neither gates edits on statistical significance nor reports held-out generalization.
At production scale, a systems line updates weights from production workloads [26], treating harness evolution as one control-plane axis but, like the optimization line, not saying how a change is credibly credited. Our frozen-weight, sealed-test measurement is complementary: the credit judgment a control plane needs, established offline.
2.0.0.2 Closest to us: coding-agent harness evolution.
The nearest answers to what to evolve come from a concurrent 2026 line auto-evolving coding-agent harnesses, several on our benchmark. Self-Harness [5] clusters failed traces by a (cause, status, mechanism) signature, akin to our where/why split, and promotes edits by a non-regression rule; HarnessFix [6] maps failures to repair operators; AHE [7] verifies each edit against a self-declared prediction (terminal-bench-2 \(69.7\!\to\!77.0\)); Meta-Harness [27] searches harness code with an agentic proposer; HarnessX [8] reports multi-benchmark gains; and EvoTrainer [28] co-evolves weights and harness, whereas we freeze weights to isolate the harness. We build on this line and add three elements it has not combined: a categorical quality-diversity archive keyed on the diagnosed pathology, with cross-cell recombination; a paired-\(2\sigma\) significance gate alongside a measurement-validity gate; and a retention scalar on a test withheld from all decisions and scored once.
2.0.0.3 How to know a change helped.
All of the above accept edits without statistical validation. VeRO [9] is the closest: access-controlled train/val/test splits documenting how agent self-optimization overfits; it establishes the held-out discipline we adopt but is a measurement substrate, not a method, and reports no retention scalar (related: validation-gated skill edits [29] and direct optimizer ranking [30]). AIRA_2 [10] argues some reported overfitting is evaluation noise, not memorization, motivating exactly our paired-significance and measurement-validity gates, while benchmark-gaming and contamination work [11], [31] motivates withholding the test set. Against this, our result is positive: in a frozen-27B, long-horizon regime the evolved harness’s gains largely survive on a sealed test.
2.0.0.4 How to search.
Quality-diversity supplies our search template. MAP-Elites [32], [33] illuminates a behavior space by keeping an elite per descriptor cell. Recent work applies it to LLMs with hand-defined structural prompt descriptors [34] or learned/embedding descriptors [35]. Our descriptor is neither measured nor learned but an LLM-assigned (where \(\times\) why) pathology of the harness; combined with quality-biased selection and cross-cell recombination, this categorical, pathology-keyed archive is, to our knowledge, absent from prior QD-for-LLM work.
2.0.0.5 Proposals and measurements.
The search is only as good as its proposals and measurements. Failure attribution in (multi-)agent systems is active [36]–[38]; our why labeling is a single-agent instance, and reflection studies [39], [40] find root-cause diagnosis where even strong models fail most (end-to-end below \(30\%\); stronger models’ errors resist self-correction [41]), the fallibility our propose-but-never-credit split survives. [42] find the capacity to produce harness updates model-tier-flat; we revisit this for diagnosis quality. Our anchor screening (\({\approx}0.5\)-pass tasks) is the dual of discarding all-pass/all-fail prompts [43], [44].
3 Method↩︎
We fix what may change (mutable surface vs.kernel), then give the loop that changes it, the GSME archive that organizes what it finds, and the gates that decide what is credited.
3.1 Problem setup: harness vs.kernel↩︎
We fix a model \(M\) (weights frozen) and define an agent as \(M\) wrapped in a harness \(H\), the deterministic scaffolding that turns \(M\) into a task-solving system: the system/role prompt, injected knowledge and tool definitions, the control loop (iteration, recovery, finalization), and runtime configuration. Self-evolution searches over \(H\) with \(M\) held constant.
Not all of \(H\) is mutable. We partition it into a kernel that evolution must not touch (measurement and evaluation code, the self-evolution machinery, and the product/interface contract) and the mutable surface \(\mathcal{X}\). A kernel-touching patch is rejected at apply time; every accepted patch is env-gated and defaults off, so the unevolved harness is byte-identical to vanilla.
Each edit carries a two-axis coordinate. where names the surface it touches, coarsened to four levers: prompt, knowledge (retrieved context, docs, demonstrations, skills, memory), runtime (control loop, hooks, recovery), and config (numeric knobs); the prompt/knowledge split mirrors the instruction-vs-demonstration distinction in prompt optimization [3]. why names the failure pathology it addresses (a small, evolvable set: method lock-in, thinking-runaway, premature finalization, …). where is bound mechanically from the edited file; why is assigned by the diagnosing model, concentrating modeling error on that axis.
3.2 The self-evolution loop↩︎
A running example. We illustrate with the BrowseComp+ lineage (Figure 2). The frozen model shows two failures: it exhausts its token budget mid-reasoning and returns an empty turn (thinking-runaway), and it finalizes without checking its answer. The loop diagnoses two (runtime) pathologies and credits a patch for each on train, selective recovery (\(+6.1\) pp) and a verify-finalize self-check (\(+9.8\) pp); their cross-cell recombination beats either alone and generalizes (\(16.9\to30.8\) sealed, \(+13.9\) pp, credited). We return to it at the archive and the results.
3.2.0.1 Cold start (once per model \(\times\) domain).
We score vanilla on the full train set at \(K{=}3\) for a per-task thick ledger (the baseline lock and source of all variance estimates, §3.2.0.4); the diagnosing model reads vanilla’s failing trajectories into a failure map of {pathology (why), tasks, a soft where hint}.
3.2.0.2 Each round
follows the funnel of Figure 1 (precise protocol in Algorithm [alg:loop]). The evolver reads the previous round’s trajectories (train only) and designs 3–4 env-gated candidates; a zero-inference preflight drops any that declare no activation beacon or whose trigger never fired; the rest are screened on the anchor subset at \(K{=}1\), and survivors are confirmed on the full set at \(K{=}3\) and sent through the three gates (§3.4). The best harness becomes the next round’s parent, or the parent is kept and re-diagnosed if none beats vanilla.
3.2.0.3 Termination.
Stop when no candidate beats vanilla on train for \(P\) consecutive rounds or a round cap \(R\) is reached (\(P{=}10\), \(R{=}20\)); termination never consults the test set.
3.2.0.4 Sampling.
Evaluation cost is the binding constraint, so candidates pass through three layers of increasing cost. Because a paired comparison needs matched per-task outcomes (unlike methods that collapse a benchmark to one score [45], [46]), subsets only screen and navigate; the credited effect is always the full set at \(K{=}3\) under a per-task paired test. A probe (\({\sim}15\)–\(30\) anchor tasks at \(K{=}1\), drawn by per-task Bernoulli variance) culls only egregious losers at a data-driven margin (\(1.5\,\sigma_{\text{screen}}\); App. 9); an estimator (a stratified \(K{=}3\) subset, the full set on our small benchmarks) drives round-to-round navigation; and the full set at \(K{=}3\) gives the cold-start baseline and the final credited number, never a subset headline. Running full \(K{=}3\) throughout yields a complete ground truth, and from it a reusable finding: tree-aware score borrowing reproduces every credited decision at \(67\%\) fewer child trials (Theorem 1; validated offline \(14/14\) and live \(7/7\); App. 9), a recipe for cutting evaluation cost as benchmarks grow.
3.2.0.5 Division of labor.
The loop splits cleanly: the evolver owns the semantic steps (reading trajectories, assigning why, designing and writing patches), while deterministic code owns everything reproducible (sampling, the gates, aggregation, the paired test, bookkeeping). The model never performs the quantitative judgments; they are computed, not estimated, for reproducibility and because arithmetic is error-prone for an LLM. The loop runs end-to-end: an LLM agent (Claude Opus 4.8 through Claude Code) executes a fixed operating procedure for the full cycle, reading trajectories, diagnosing the why pathology, authoring and applying patches, launching scoring, and advancing rounds. The measurement layer (sampling, the gates, the paired test, aggregation) is deterministic scripted code the agent invokes but never computes itself, keeping every credited number reproducible and out of the model’s hands. The operator launches the run; the content of every edit is the model’s.
3.3 Gated Semantic MAP-Elites (GSME)↩︎
Figure 2: Evolution tree (running example, BrowseComp+). Nodes are git commits (RBR \(=\) reasoning-budget recovery, VF \(=\) verify-finalize, QD \(=\) query-decompose). Of the three round-1 recombinations only VF\(+\)RBR is synergistic (\(\star\); sealed \(+13.9\) pp, credited); the others match their stronger component, and four follow-ups fail to beat it..
The loop generates and measures candidates; the archive decides what is kept, and on what axis diversity is preserved.
3.3.0.1 Descriptor.
Each edit is placed in a cell of a categorical archive indexed by (where \(\times\) why). The descriptor is semantic, assigned by the diagnosing model from the failure trajectory rather than measured from the rollout, and keyed on the pathology an edit addresses, not the tasks it fixes. This is the anti-overfitting bias: search is organized by failure type, not training-task identity. The pathology comes free from the diagnosis step (no descriptor to learn or hand-engineer) and yields an interpretable coverage map (App.Fig. 4). Since the label only steers exploration and never decides credit, the archive degrades gracefully under label noise (App. 9).
3.3.0.2 Archive.
One gated elite per cell (a patch enters only after clearing the gates of §3.4). The archive is categorical and coarse (a handful of pathology cells), not a dense grid. A separate elite per pathology counters a reported failure mode where naive optimizers collapse toward safe prompt edits once ambitious changes fail [9]; our accepted edits span prompt, runtime, config, and hook levers, not prompts alone (App.Fig. 4).
3.3.0.3 Selection & variation.
Canonical MAP-Elites selects cells uniformly to illuminate the whole behavior space; our goal is narrower, to find the single best harness under a tight budget. So we use quality-biased selection (each round extends the global-best harness) with cross-cell recombination (stacking it with elites from other cells): the budget goes to the champion while recombination still exploits the archive’s diversity, keeping the anti-overfitting benefit without uniform illumination’s cost. On four of six domains the selected harness is such a recombination (Figure 2). We do not claim uniform-selection or dense-grid MAP-Elites.
3.3.0.4 Evolution tree.
The archive’s substrate is a git tree: each candidate is a branch \(+\) commit (root \(=\) vanilla) with a per-node ledger (parent, status, activation spec, where/why, touched files), an auditable record for structure and de-duplication while git holds the code.
3.4 Three gates↩︎
A patch is credited only after clearing three gates, applied in increasing order of cost.
3.4.0.1 The validity gate.
Before scoring, an environment health pre-check (sandbox, network, dependencies, verifier output) rejects a dirty environment. After scoring, infrastructure failures (missing verifier output, sandbox crash, verifier timeout) are not counted as agent failures: the trial is re-run up to twice and the environment repaired, and a task that still fails infra is scored 0 but kept in the denominator. pass@1 uses the total task count throughout, since excluding infra-failed tasks inflates the score (they skew hard) while counting them as agent failures unfairly penalizes; re-running to a real measurement is the unbiased middle.
3.4.0.2 The activation gate.
A patch is credited for an effect only if it fired. Each mechanism emits an activation beacon and a ledger records, per trial, whether it triggered; effects on trials where the mechanism never fired are not attributed to it.
3.4.0.3 The significance gate.
Single-run pass@1 swings by several points on borderline tasks (\({\approx}5\) pp), so a deterministic rule (“improves a split” or “does not regress”) cannot separate a real gain from a lucky draw, and accepting such gains accumulates noise. We keep two thresholds distinct. Navigation (promotion, parent selection) uses \(K{=}3\) mean pass@1 \(>\) vanilla; a credited gain (what we report) requires a paired \(2\sigma\) test on the same task set (\(\sigma\) from per-task paired differences, \(z \ge 1.96\)). Pairing removes between-task difficulty variance, so the paired threshold is well below the unpaired one; we publish the per-task differences so \(\sigma\) is auditable.
The gates decide what is credited; what we report is measured on a sealed test withheld from the evolving agent throughout and scored once after evolution (no test signal informs diagnosis, design, parent selection, or termination), and summarized by retention, the held-out lift as a fraction of the training lift. Enforcing the withholding mechanically is future work.
4 Experiments↩︎
4.1 Setup↩︎
Harness-only evolution, with the frozen model never updated. The seven-domain main results use qwen3.6-27B; the cross-model study (Table 3) additionally evolves two further frozen models on AppWorld,
Qwen3.5-397B-A17B-GPTQ-Int4 and Gemini 3 Flash (a non-Qwen API model). The evolver’s diagnoser \(D\) (a separate, stronger model, \(D\neq M\)) is Claude
Opus 4.8, driven through Claude Code to run the loop end-to-end (§3.2.0.5); \(D\) is reported for reproducibility, not required. Seven domains across three sources:
terminal-bench-2 (TB2; terminal), the EvoAgentBench suite (LiveCode / Omni-MATH / BrowseComp+ / GDPval / SWE-bench; code / math / web / professional-deliverable / repository), and AppWorld (app/API). Six are credited on disjoint train/sealed-test
splits (Table 1, top six); the seventh (SWE-bench) is preliminary (bottom row) and analyzed at the end of this section. \(K{=}3\) attempts per task; per-task success (pass@1, except GDPval’s
graded rubric \(\ge0.6\)) with denominator \(=\) total tasks (§3.4); env-gated patches default off \(=\) vanilla
byte-identical. The train-selected harness is scored once on the sealed test, never consulted during evolution (§[sec:protocol]). Verifiers: TB2 harbor
(timeout-fair overlay); LiveCode lcb_runner; Omni-MATH / BrowseComp+ LLM judge (pass \(=\) reward \(1.0\)); GDPval graded rubric (pass \(=\)
reward \(\ge 0.6\)); AppWorld task-success check.
4.2 Main results↩︎
The train-selected harness improves over vanilla on the sealed test of every benchmark, and every sealed-test gain clears the paired \(2\sigma\) credit bar (Table 1, Figure 3). Gains are \(+9\) to \(+15.5\) pp; the recurring cross-domain win is harness-level recovery of the frozen model’s dominant failure mode (empty “thinking-runaway” turns), on several domains stacked with a finalize self-check (VF). Our running example, BrowseComp+, is one such recombination (recovery \(+\) verify-finalize), credited at \(+13.9\) pp on the sealed test.
The seventh domain, SWE-bench (repo-level bug fixing), runs under the same loop on the suite’s canonical \(101/26\) train/test split. The train-selected harness, a repeated-tool-call breaker (nudge to change approach after three identical calls) stacked on a verify-falsify finalize check, is strongly credited on training (\(+13.9\) pp, \(n{=}101\)) and lifts the held-out set in the same direction (\(+5.1\) pp). We credit only what the sealed test confirms, and at \(n{=}26\) even a real \(+5\) pp effect sits below the paired-\(2\sigma\) bar (\(z{=}0.78\)); we report SWE-bench as preliminary rather than stretch the bar to fit. Train and test are disjoint by construction, so its low apparent retention is noise at \(n{=}26\), not overfitting; a larger held-out set is the natural follow-up, bounded only by SWE-bench’s rollout cost (the highest in our suite).
| Benchmark | train \(\Delta\) | sealed \(\Delta\) | \(z\) | retention |
|---|---|---|---|---|
| TB2 | \(+6.3\) | \(\mathbf{+9.3}\) | \(2.71\) | \(147\%\) |
| LiveCode | \(+15.8\) | \(\mathbf{+13.7}\) | \(2.65\) | \(86\%\) |
| Omni-MATH | \(+10.1\) | \(\mathbf{+11.7}\) | \(3.66\) | \(115\%\) |
| BrowseComp+ | \(+15.8\) | \(\mathbf{+13.9}\) | \(3.96\) | \(88\%\) |
| GDPval | \(+8.4\) | \(\mathbf{+9.2}\) | \(2.13\) | \(109\%\) |
| AppWorld | \(+17.8\) | \(\mathbf{+15.5}\) | \(6.44\) | \(87\%\) |
| SWE-bench | \(+13.9\) | \(+5.1\) | \(0.78\) | \(37\%\) |
5pt
Figure 3: Sealed-test success across seven domains: vanilla vs.evolved harness. The six credited domains (solid) clear paired \(2\sigma\); hatched: not credited. Details in Table 1..
4.3 Significance and generalization↩︎
All six sealed-test gains are credited: each per-task paired difference clears \(z\!\ge\!1.96\) on tasks the evolving agent never saw (\(p\) from \(<10^{-4}\) to \(0.033\); AppWorld strongest at \(z{=}6.44\), \(n{=}168\)). Each credited number is a single sealed comparison, the train-selected winner scored once on held-out tasks, so the many candidate comparisons during evolution cannot inflate it (researcher degrees of freedom are spent on train) and no multiple-comparison correction is needed. On TB2 the train-set gain (\(+6.3\) pp, \(n{=}53\)) sits just under its train \(2\sigma\) threshold (\(z{=}1.60\), high at small \(n\)), so crediting happens on the sealed test. The machinery is two-sided: on TB2 it flagged and culled a significantly negative candidate (\(-6.4\) pp). AppWorld carries higher run-to-run variance (a vanilla self-rerun drifted \({\sim}9\) pp); the VF gain (\(+15.5\) pp) far exceeds this floor, but smaller candidate effects were not resolvable there and we do not claim them.
Sealed lift retains \(86\)–\(147\%\) of train lift: three benchmarks exceed their training gain and three retain \(86\)–\(88\%\), with no overfitting collapse. Retention is a ratio, so near the significance floor it is noisy; the load-bearing claim is the credited held-out gain (Table 1), with retention as corroboration.
The gains are not mere variance reduction: best-of-3 (pass@3) also rises on every credited domain (\(+5.6\) to \(+15.4\) pp, App.Table 6), so the evolved harness expands the set of solvable tasks, not just the per-attempt hit rate. pass@1 remains our primary, deployment-relevant metric.
4.4 Diagnosis, and the cost of a wrong hypothesis↩︎
The credited wins trace back to a small set of diagnosed pathologies (per-domain diagnosis-to-mechanism map in Appendix 10). Two why pathologies recur, each fixed by a distinct runtime mechanism: (i) thinking-runaway empty turns (the model exhausts its token budget on reasoning and emits nothing), fixed by selective recovery; and (ii) premature/unverified finalization (it stops without a checked final answer), fixed by a verify-finalize self-check. Domain-specific pathologies add one-off mechanisms (iteration-budget for turn-cap exhaustion, code-/deliverable-finalize for un-emitted outputs). That the loop converges on the same dominant bottleneck across six independent sealed tests is evidence it identifies the true top failure mode, not a per-benchmark artifact.
why is an LLM-assigned hypothesis, not ground truth: on AppWorld the loop diagnosed a knowledge gap (a PDF-bill task), proposed a knowledge injection, and the gate rejected it (the real bottleneck was capability, not missing knowledge). why only steers which candidates to try, while credit comes solely from the paired-\(2\sigma\) gate, so a wrong label costs at most a rejected candidate and never credits a bad harness: “diagnosis proposes, measurement disposes.”
4.5 Cross-model dissociation↩︎
Thinking-runaway is dominant on four of six domains, so does the loop re-fix one qwen3.6-27B quirk? Three observations say no. (i) It is a model property (reasoning-budget overrun), not a harness defect: the vanilla harness already retries empty turns, and the credited recovery is a selective, sticky policy disabling thinking only after a runaway. A single-variable ablation (Table 2) shows this specificity is what matters: on LiveCode a \(16\times\) larger token budget lifts train pass@1 only \(+6.5\) pp and a blanket thinking-off toggle \(+8.6\) pp, while selective recovery reaches \(+14.8\) pp; neither a bigger budget nor a one-line global toggle reproduces it. (ii) The credited wins span several mechanisms and lever cells, not one patch. (iii) Cold-started on other models the loop self-evolves their own credited harnesses, not the 27B’s, each targeting that model’s distinct pathology (Table 3).
| arm | max_tokens | thinking | pass@1 |
|---|---|---|---|
| vanilla | \(8192\) | on | \(64.6\) |
| \(+\) budget | \(32768\) | on | \(67.0\) |
| \(+\) budget | \(131072\) | on | \(71.1\) |
| thinking-off | \(8192\) | off | \(73.2\) |
| recovery | \(8192\) | on\(^{\dagger}\) | \(\mathbf{79.4}\) |
4pt
These patches follow a pathology\(\to\)patch matching law (Table 3): each model’s dominant pathology draws a matched patch, credited on the sealed test, that helps it but not models with a different pathology. On AppWorld, where this cross-model comparison runs, the 27B’s dominant failure is empty “engagement” turns (not the thinking-runaway of the reasoning-heavy domains above), fixed by verify-finalize; the 397B and Gemini 3 Flash instead make careless wrong answers, fixed by a submit-verify checklist. Gemini, a non-Qwen model, reproduces the careless\(\to\)checklist match (\(+13.5\) pp, \(z{=}4.15\)), extending the law across model families, not Qwen sizes alone. Off the diagonal the mismatched patch is near-zero on the Qwen pair (\(+0.2\) and \(+1.2\)), a near-clean dissociation; on Gemini the mismatched verify-finalize still gains \(+5.8\) (\(z{=}2.14\)), a gradient rather than a switch because it shares a verify step with the checklist. A datetime patch, credited on both Qwen models (\(p{=}0.003/0.009\)), is a model-agnostic positive control; cross-model absolute rates are per-model, so we read the pattern, not a ranking.
| model | family | VF | SV | retention |
|---|---|---|---|---|
| 27B | Qwen 3.6 | \(\mathbf{+15.5}\) | \(+1.2\) | \(0.87\) |
| 397B | Qwen 3.5 | \(+0.2\) | \(\mathbf{+13.6}\) | \(1.18\) |
| Gemini | \(+5.8\) | \(\mathbf{+13.5}\) | \(1.07\) |
6pt
4.6 Ablations: gates and archive↩︎
GDPval illustrates the train-then-seal discipline. The train-selected harness (recovery\(+\)deliverable-finalize) is credited on the sealed test (\(+9.2\) pp); separately, a recovery\(+\)VF variant reached the highest sealed score (\(+11.5\) pp) but ranked below on train, so train selection missed it. Train selection is a lower bound on what generalizes: train-only crediting can mis-rank candidates, which is why we credit on a sealed set.
Does the paired-\(2\sigma\) gate matter, or would a deterministic “mean improves” rule (as used by prior work) suffice? On TB2’s ten candidate mechanisms a deterministic \(\Delta>0\) rule credits four, including context-compact, whose activation beacon fired zero times (only the activation gate rejects it). A single-run gate credits a truly-neutral mechanism as a win \({\sim}60\%\) of the time (\({\sim}25\%\) as a \(\ge3\) pp “win”), quantifying how non-statistical loops manufacture phantom gains; of the mechanisms it credits on train, only the genuinely significant one survives paired-\(2\sigma\) on the sealed test.
Two signs the archive earns its keep: on most benchmarks the strongest harness is a cross-cell recombination (only per-cell elites make it available), and accepted edits span all four levers, not prompts alone. Diversity-preserving selection beats greedy hill-climbing for prompts [4]; our question is whether the diversity axis should be semantic: a categorical (where\(\times\)why) pathology archive rather than a per-task Pareto frontier. We do not compare against a naive task-keyed baseline, which overfits by construction.
5 Discussion↩︎
The cross-model matching law licenses a reading of what an evolved harness is. It is less a better scaffold than a corrective patch fitted to a model’s failure distribution: the recurring credited wins manage a specific failure mode (reasoning-budget overrun, unverified finalization) rather than adding general capability, and the patch is keyed to that distribution, changing when the pathology changes and recurring when it recurs, even across model families. This also delimits where the loop pays off: because it fits a patch to a failure distribution, harness evolution has the most headroom, and the most economic value, on mid-capability models one can deploy cheaply; frontier models fail less and lie outside our scope.
Two consequences follow. First, a harness is portable exactly as far as the pathology it targets is shared: the careless\(\to\)checklist patch carries across two model families, while the 27B’s engagement patch helps little on a model lacking that failure. Because one cannot assume a new model shares a given pathology, a new model still needs its own diagnosis. Second, what reliably transfers is the diagnose-and-credit loop, not any specific harness, though the patch it produces recurs wherever the pathology does. More broadly, this reframes how “self-improving agent” results should be read: an apparent capability gain from harness evolution may be a model-specific correction, and only a sealed, paired test separates a genuine, transferable improvement from one fitted to the evaluation at hand.
Two directions would make the loop more like ongoing practice than a one-off study. On cost, full \(K{=}3\) re-evaluation of every candidate is affordable at our scale but not at production scale; the tree-aware borrowing estimator (App. 9) is a validated step, reproducing every credited decision at a third of the cost (\(67\%\) fewer trials) by reusing what a candidate shares with its ancestors, much as a practitioner reuses experience instead of re-testing from scratch. Harder is supervision: crediting here relies on a per-task verifier, so moving toward weaker or self-generated signals, closer to unsupervised reflection, without losing the trustworthiness guarantee is the central open problem. We view cheaper, well-credited evaluation as a prerequisite for that lower-supervision, lifelong regime, not a substitute for its crediting challenge.
6 Limitations and Future Work↩︎
6.0.0.1 Verifier dependence.
Our guarantee is relative to the reward signal: paired-\(2\sigma\) crediting certifies that a gain is real on the measured metric, not that the metric is valid. The method therefore applies only to domains with a reliable per-task verifier and inherits any gameability of that verifier: a harness that legitimately exploits a weak checker would still be credited. Sealing the test set and gating on significance guard against overfitting and noise, not against an invalid reward.
6.0.0.2 Proxy objective and short horizon.
We optimize per-task benchmark success, a verifiable but short-horizon proxy. Fuzzy or unverifiable objectives (research taste, novelty) fall outside the method, and long-term qualities such as robustness, maintainability, and sustained utility are not captured. A credited gain is a gain on this proxy, and we claim no more.
6.0.0.3 Improvement driven by repeated scoring.
The loop credits improvement only through repeated benchmark scoring: every candidate is re-evaluated on graded tasks. This makes evolution expensive and binds it to domains and cadences where frequent scored evaluation is available. Human competence, by contrast, develops largely through doing real work under sparse, indirect feedback, with formal graded exams occasional rather than the primary driver. A self-improvement paradigm that leans this heavily on dense scoring may not transfer to open-ended settings where a grader is unavailable or the meaningful signal is diffuse and long-delayed; tree-aware score borrowing (Theorem 1) cuts the cost but not the dependence.
7 Conclusion↩︎
We presented a self-evolving agent-harness framework built on separating proposing from crediting: a stronger evolver diagnoses a frozen model’s failures into (where \(\times\) why) pathologies and writes patches, while deterministic code owns measurement, significance, and the sealed test, with candidates held in the GSME pathology archive. Across seven domains the credited held-out gains are \(+9\) to \(+15.5\) pp and retain \(86\)–\(147\%\) of training, the same gate rejects harmful and inert edits, and changing the model changes both the diagnosed pathology and the winning patch. What transfers is not any harness but the loop that grows one.
8 The self-evolution loop↩︎
Algorithm [alg:loop] gives the precise per-round protocol summarized in the main text by Figure 1. Every quantitative step (task sampling, the gates, the paired significance test) is deterministic code; the evolver supplies only the semantic steps (diagnosis, patch design).
frozen model \(M\); vanilla harness \(H_0\); train \(\mathcal{T}\), sealed test \(\mathcal{T}^\star\); attempts per task \(K{=}3\); diagnoser \(D\) (stronger, with \(D{\neq}M\)); round cap \(R\); patience \(P\) Cold start (once per model\({\times}\)domain): \(L \gets \Call{Score}{M,H_0,\mathcal{T},K}\);\(v \gets \mathrm{pass@1}(L)\) \((\mathcal{T}_A,v_A,\tau) \gets \Call{SelectAnchor}{L}\) \(\mathcal{F} \gets D.\Call{Diagnose}{\textproc{Fails}(L)}\);\(\mathcal{A} \gets \{H_0\}\);\(\mathrm{stale}\gets 0\) Evolution: if \(r>1\): \(\mathcal{F} \gets D.\Call{Diagnose}{\text{round-}(r{-}1)\text{ trajectories}}\) \(\{C_i\} \gets \Call{Preflight}{D.\textproc{Design}(\mathcal{F})}\) \(H_i \gets \Call{Apply}{\mathrm{parent},C_i}\) continue \(\rho_i \gets \Call{Score}{M,H_i,\mathcal{T},K}\) continue insert \(H_i\) into cell \((\mathrm{\small where}_i,\mathrm{\small why}_i)\) of \(\mathcal{A}\) if a better elite \(\mathrm{parent} \gets \arg\max_{H\in\mathcal{A}} \mathrm{pass@1}(H,\mathcal{T})\), recombined with elites of other cells \(\mathrm{stale} \gets 0\) if some \(C_i\) beat \(v\), else \(\mathrm{stale}\gets\mathrm{stale}+1\) if \(\mathrm{stale}\ge P\): break Unseal (after evolution; \(\mathcal{T}^\star\) untouched above): \(H^\star \gets \arg\max_{H\in\mathcal{A}} \mathrm{pass@1}(H,\mathcal{T})\);score \(H^\star,H_0\) on \(\mathcal{T}^\star\) once \(H^\star\); retention \(=\Delta_{\mathrm{test}}/\Delta_{\mathrm{train}}\) (\(\Delta\): lift over vanilla); credit gains passing paired \(2\sigma\) on \(\mathcal{T}^\star\)
9 Formal analysis↩︎
The loop’s design choices rest on component-level guarantees, each with an explicit assumption. We do not prove end-to-end monotonic improvement, which depends on the model’s improvability (the 27B/397B results show model-conditional ceilings); each result below certifies one component, turning missing or mislabeled data into a bounded cost term rather than a bias.
Proposition 1 (Measurement validity). If infrastructure failures are heterogeneous across tasks and scored as agent failures, the archive’s per-cell \(\arg\max\) can flip; admitting only trials with a valid verifier receipt yields an unbiased success estimate on the runnable subset.
Proof sketch. Scoring an infra-failed trial as \(0\) is a missing-not-at-random error [47]. Two candidates over two tasks suffice: with true per-task pass rates \(p_x=(0.9,0.1)\) (score \(0.50\)) and \(p_y=(0.4,0.4)\) (score \(0.40\)) and heterogeneous infra-survival \(q=(0.2,1.0)\), the naive estimates become \(\hat{f}(x){=}0.14<\hat{f}(y){=}0.24\), so the worse candidate wins. Restricting to admitted trials estimates \(\Pr[\text{pass}\mid\text{ran}]\), unbiased on runnable tasks; unrunnable tasks are reported as coverage, not scored \(0\). ◻
Proposition 2 (Graceful degradation under label noise). A judge mislabel rate \(\varepsilon\) on the why axis inflates the samples needed to fill a cell by \(1/(1-\varepsilon)\) (a slope, not a cliff) and leaves the deployed harness unchanged, since deployment selects \(\arg\max\hat{f}\), whose definition contains no label.
Proof sketch. Each arrival is correctly archived with probability \(1-\varepsilon\), so accumulating \(m\) correct entries takes \(m/(1-\varepsilon)\) arrivals in expectation (binomial thinning). Label noise perturbs only the archive’s bookkeeping, not \(\hat{f}\). ◻
Lemma 1 (Anchor information and screen noise). For a task with posterior mean \(\hat{p}\) and effective mass \(m\), one further trial reduces the posterior variance by \(\hat{p}(1-\hat{p})/(m+1)^2\), so selecting anchors by per-task Bernoulli variance is maximally informative. The \(K{=}1\) anchor-mean sampling deviation is \(\sigma_{\text{screen}}=\sqrt{(1/n^2)\sum_i \hat{p}_i(1-\hat{p}_i)}\), and the screen culls at \(1.5\,\sigma_{\text{screen}}\).
Proof sketch. The total-variance identity for a Beta posterior under one Bernoulli observation gives the per-task reduction; summing independent per-task Bernoulli variances gives the anchor-mean \(\sigma\). ◻
Theorem 1 (Tree-aware score borrowing, worst-case-no-harm). Let a child reuse ancestor scores with tree-distance discount \(w(d)=e^{-\lambda d}\) and borrowed mass \(\alpha_0=\sum_i w_i k_{a_i}\), giving \(\hat{\theta}_c=(p_c+\sum_i w_i p_{a_i})/(k_c+\alpha_0)\). With probability \(\ge 1-\delta\), \[|\hat{\theta}_c-\theta_c|\;\le\;\underbrace{\tfrac{\alpha_0\,\eta}{k_c+\alpha_0}}_{\text{bias}} +\underbrace{\sqrt{\tfrac{\log(2/\delta)}{2(k_c+\alpha_0)}}}_{\text{noise}},\] where \(\eta\) bounds the true ancestor–child gap. Capping \(\alpha_0\le \Delta k_c/(4\hat{\eta}-\Delta)\) holds the bias \(\le\Delta/4\) for any* \(\eta\), so a candidate with true margin \(\ge\Delta\) is never mis-ranked and the worst case reduces to the no-borrowing cost.*
Proof sketch. Split the error into bias (\(|\mathbb{E}\hat{\theta}_c-\theta_c|\), expectation algebra bounded by \(\alpha_0\eta/(k_c+\alpha_0)\)) and noise (weighted Hoeffding, using \(\sum_\ell c_\ell^2\le\sum_\ell c_\ell=k_c+\alpha_0\) since weights are \(\le 1\)). The cap solves the bias constraint \(\alpha_0\eta/(k_c+\alpha_0)\le\Delta/4\). ◻
9.0.0.1 Empirical validation.
We validate Theorem 1 against full re-evaluation on two real evolution trees, BrowseComp+ (a \(2^3\) mechanism factorial, \(8\) nodes \(\times 154\) tasks) and LiveCodeBench (\(97\) tasks), each at \(K{=}3\). Offline (replaying existing ledgers), the borrowing decision reproduces the full-\(K{=}3\) promote/prune decision on all \(14/14\) nodes at \(-67\%\) child-trial cost. Live on BrowseComp+ (a fresh, independent \(K{=}1\) pass on all \(7\) non-root nodes plus borrowed ancestor mass) it reproduces \(7/7\) decisions with zero flips at \(33.4\%\) of the child cost (\(1078\) vs.\(3231\) trials). The cap engaged on every edge, and estimates shrink conservatively toward the parent, never enough to flip a decision in either direction; assumption A3 (tree smoothness) holds distributionally (per-edge \(|\Delta p|\) median \(0\), mean \(\approx 0.18\)). The live run removes the subsampling caveat; the remaining scope is the conservative uniform-cap route (not activation stratification). Because the main results use full re-evaluation (the conservative gold standard), borrowing here is validated against ground truth rather than relied upon to produce it, which is what lets us recommend it: a validated recipe for cutting evaluation cost as harness evolution scales to larger benchmarks.
10 Per-domain diagnoses and patches↩︎
Table 4 maps each domain to the dominant pathology the loop diagnosed and the credited mechanism that fixed it; sealed gains and significance for these mechanisms are in Table 1. Every credited mechanism is a runtime edit, and two pathologies (thinking-runaway and unverified finalization) recur across domains.
| Benchmark | dominant diagnosed | credited mechanism (\(=\)runtime) |
|---|---|---|
| TB2 | unverified finalize \(\to\) wrong | verify-finalize |
| LiveCode | thinking-runaway empty | recovery (\(+\)iter \(+\)CF) |
| Omni-MATH | runaway empty \(+\) wrong-answer | recovery \(+\) verify-finalize |
| BrowseComp+ | thinking-runaway empty | recovery \(+\) verify-finalize |
| GDPval | runaway empty \(+\) no deliverable | recovery \(+\) deliverable-finalize |
| AppWorld | empty / no finalize | verify-finalize |
| SWE-bench | repeated-call stall \(+\) false finalize | repeat-break \(+\) verify-falsify |
11 The WHERE taxonomy: the harness’s editable surface↩︎
Figure 4: GSME archive, aggregated across the six credited domains. Rows are the four where levers; columns are open, LLM-diagnosed why classes (the last marks model-capability limits no harness edit can fix); each cell prints a mechanism’s status and one-line reason (verdicts from training splits; sealed scores in Table 1). Takeaways: broad exploration with strict promotion; credited gains concentrate in runtime; and the boxed wrong-output column is attacked by all four levers but moved only, and only locally, by prompt: the where axis matters. Recombination is shown in Figure 2..
Figure 4 aggregates the GSME archive across the six credited domains: broad exploration under strict promotion, with credited gains concentrated in runtime. The where descriptor has two layers. The lever (four values: prompt, knowledge, runtime, config) is the coordinate the GSME archive is keyed on (§3.3); beneath it sits an operational layer of concrete edit surfaces: the specific files a patch may touch (Table 5). Unlike why (LLM-diagnosed and open/evolvable), where is a fixed, closed engineering surface: it enumerates exactly the parts of the harness an edit is allowed to change, which is what makes each patch’s coordinate mechanically bindable from its artifact rather than self-declared. An edit’s where lever is read off which surface its diff touches; the control arm (no patch surface) exists only for bookkeeping.
| Lever | Edit surface | Touches | Example patch (this paper) |
|---|---|---|---|
| prompt | system_prompt_template | agent system prompt (templates/*.md) | — |
| task_wrapper_prompt | per-task instruction (domains/*/prompt.md) | TWP (TB2); datetime-inject (AppWorld) | |
| judge_prompt | evaluation-internal prompt | — | |
| tool_description | a tool’s description field | — | |
| knowledge | skill | injected skill (skills/*) | knowledge-base arm (AppWorld; not credited) |
| memory | injected memory (everos/*) | — | |
| runtime | hook_new | new agent hook (agent/hook/*.py) | verify-finalize; code-/deliverable-finalize |
| hook_modify | modify an existing hook | — | |
| tool_new | new tool (agent/tools/*.py) | — | |
| loop_override | scoped loop override | — | |
| context_override | scoped context/provider override | thinking-runaway recovery | |
| tool_override | scoped tool override | — | |
| config | config | numeric YAML/JSON defaults | iteration budget; max-iter; temperature |
4pt
On the frozen qwen3.6-27B, credited gains concentrated in runtime and config; the two in-context levers were limited (the prompt lever’s one clean node, TWP, fired on \(158/159\) TB2 tasks yet produced no gain, and a knowledge injection on AppWorld did not credit). We report this as a model-conditional finding, not a property of the levers in general.
12 pass@1 vs.pass@3↩︎
Our primary metric is pass@1 (per-task success, averaged over \(K{=}3\) attempts), the deployment-relevant single-attempt quantity, and the one the paired-\(2\sigma\) credit gate is built on. Table 6 reports pass@3 (best of the three attempts) alongside it on the sealed test. The gains are not an artifact of variance reduction: pass@3 rises on every credited domain, and since vanilla’s pass@3 is far below ceiling, the evolved harness expands the set of solvable tasks rather than merely stabilizing per-attempt outcomes.
| pass@1 (%) | pass@3 (%) | |||||
|---|---|---|---|---|---|---|
| 2-4(lr)5-7 Benchmark | van | ev | \(\Delta\) | van | ev | \(\Delta\) |
| TB2 | \(36.1\) | \(45.4\) | \(+9.3\) | \(52.8\) | \(58.3\) | \(+5.6\) |
| LiveCode | \(58.1\) | \(71.8\) | \(+13.7\) | \(66.7\) | \(79.5\) | \(+12.8\) |
| Omni-MATH | \(54.3\) | \(66.0\) | \(+11.7\) | \(65.0\) | \(75.0\) | \(+10.0\) |
| BrowseComp+ | \(16.9\) | \(30.8\) | \(+13.9\) | \(33.8\) | \(49.2\) | \(+15.4\) |
| GDPval | \(43.7\) | \(52.9\) | \(+9.2\) | \(58.6\) | \(67.2\) | \(+8.6\) |
| AppWorld | \(41.3\) | \(56.7\) | \(+15.5\) | \(67.3\) | \(75.6\) | \(+8.3\) |
| SWE-bench | \(47.4\) | \(52.6\) | \(+5.1\) | \(61.5\) | \(69.2\) | \(+7.7\) |
13 Vendor-reported numbers and our reproducible baseline↩︎
Our vanilla is a reproducible minimal harness (the fixed baseline over which every credited lift is measured), not a vendor’s tuned evaluation harness. On terminal-bench-2 it reproduces an independent minimal-harness reference (\({\approx}36\%\)), confirming it is a sane baseline rather than an artificially depressed strawman.
Vendor-reported agentic scores for the same open model are substantially higher: the official qwen3.6-27B model card1 lists Terminal-Bench 2.0 \(59.3\) (Harbor/Terminus-2 harness, 3 h timeout, 32 CPU / 48 GB per task), LiveCodeBench v6 \(83.9\), and SWE-bench Verified \(77.2\). These figures rely on undisclosed, heavily-engineered harnesses, large per-task compute, and specific benchmark versions/subsets, and independent reproduction routinely lands well below them. We therefore do not target or compare against such vendor upper bounds: our absolute numbers reflect a deliberately minimal harness, and our contribution is the lift a self-evolution loop discovers on top of it, certified on a sealed test by paired significance. That self-reported numbers are themselves hard to reproduce is an instance of the measurement-trust problem this paper sets out to address.