1 Introduction↩︎

Structured environments such as electronic health records, databases, and web interfaces impose strict operational rules on LLM agents. Violations of these rules produce recurring errors that often go uncorrected. A clinical assistant working with an EHR, for instance, must retrieve the right resources, apply temporal eligibility criteria, avoid duplicate medication orders, and verify that changes were written [1], [2]. These failures reflect the structure of the environment rather than the particular case, so reliability depends on procedural knowledge of how to act in a specific environment, not on general language ability [3], [4].

To acquire this knowledge, prior self-improvement methods accumulate natural-language guidance at inference, either as reflections on individual failures [5] or as rules extracted across many [6], admitting each item as soon as it is produced (Appendix 7). Without a check against existing behavior, a note that fixes one trajectory can silently regress another, and over time monotonic accumulation dilutes task-relevant content and degrades the agent’s context [7], [8]. Avoiding both failures requires held-out validation of each candidate against previously correct behavior, together with a bounded library that supports modification and removal alongside addition.

We introduce Grasp (Gated Regression-Aware Skill Proposer), which treats agent self-improvement as a sequence of validated edits to a small library of behavioral instructions, or skills, injected into the agent’s context at inference [9]. After each batch of episodes, failed trajectories are grouped by failure mode, and candidate edits are proposed for the most frequent modes, each adding, modifying, or removing a skill. Each candidate is re-run on a balanced probe of previously failing and previously passing examples, and accepted only if it fixes more failures than it causes regressions, with no new regression beyond the existing baseline. The mechanism requires no parameter updates and the library remains small, versioned, and reversible.

We evaluate Grasp across five language models (gpt-oss-120b, DeepSeek V4 Flash, Gemini 3.1 Flash Lite, GPT-4.1, GPT-5.4) on two FHIR-based clinical agent benchmarks, MedAgentBench [1] and MedAgentBench-v2 [10], with supporting evaluation on FHIR-AgentBench [11] and four non-clinical environments [12]–[14]. Our main results are as follows.

• Grasp substantially outperforms a no-skills agent and five self-improvement baselines spanning memory, rule-extraction, and skill-library methods on MedAgentBench, an advantage that holds across all five base models.

• Ablations attribute the gain to comparative proposal generation, the acceptance gate, and the hard regression budget, showing that without them the skill-writing pipeline is no better than using no skills, which localizes the improvement to validated editing rather than skill writing.

• The mechanism generalizes beyond the clinical domain, helping in non-clinical environments where tasks recur with verifiable structure and remaining flat only where the action space is open-ended.

• Frozen skill libraries transfer across models and related benchmarks. A stronger writer improves weaker executors beyond what they learn for themselves while the reverse does not, an asymmetry that no ungated baseline reproduces, indicating that the libraries encode environment-specific procedural knowledge rather than model-specific patterns.

All code, learned skill libraries, and per-seed results are available at https://github.com/jomoll/GRASP.

2 Grasp: Gated Regression-Aware Skill Proposer↩︎

Grasp improves an LLM agent through repeated validated edits to a library of behavioral instructions, with no parameter updates. After each batch of training episodes, failed trajectories are grouped by failure mode, a skill-writing model proposes candidate edits for the most frequent modes, and each candidate is evaluated on a held-out probe before acceptance (Algorithm [alg:grasp]).

2.1 Skills↩︎

A skill is a structured behavioral instruction injected into the agent’s context at inference [9], specifying a trigger condition, a behavioral rule, an optional verification step for actions with side effects, and a contrastive example. Skills are Markdown documents with YAML frontmatter, making them auditable and reversible. The library starts empty and is populated by Grasp from training trajectories (Appendix 8.4 and 8.5).

2.2 Failure-Driven Skill Proposal↩︎

After each batch, a skill-writing model observes the failed traces and proposes edits, building on the verbal-feedback paradigm [5], [6] but operating on a structured library rather than a flat memory.

2.2.0.1 Mechanism-specific failure classification.

An LLM classifier assigns each failing trace an open-vocabulary label from the action sequence, the reported answer, and the expected answer. Labels must be mechanism-specific, meaning two labels imply two different corrective actions. A trace omitting a required date filter is labeled date_filter_omitted rather than wrong_answer, since the former points to a corrective skill and the latter does not. Labels from prior epochs are presented as candidates, and new ones are minted when none fits (classifier dynamics in Appendix 11.1). Failures are grouped by label with no minimum group size, so a single failure can prompt a proposal if its mechanism is distinct.

2.2.0.2 Grouped proposal generation.

The framework generates \(K\) proposals per update by cycling over failure-mode groups from largest to smallest. Each call to the skill writer provides the failing traces from one group, a sample of passing traces, active skill summaries with provenance and effectiveness statistics, and the labels of other active failure modes to discourage regressions on unrelated tasks. The skill writer returns an ADD, MODIFY, or REMOVE edit. To limit the number of skills, ADD is blocked at capacity unless a paired REMOVE frees a slot.

2.3 Regression-Aware Selection↩︎

Each candidate is evaluated against a balanced probe before being committed, following the held-out validation strategy familiar from prompt-optimization frameworks [15]. Probe examples are drawn exclusively from the development split, with validation and test examples never entering training (full per-batch loop in Appendix 8.1).

2.3.0.1 Probe construction.

The probe contains up to \(N/2\) previously-failing and \(N/2\) previously-passing samples from episodes completed earlier in the current epoch, stratified by task type, where previously-failing and previously-passing refer to per-sample correctness recorded under the library state in effect when each sample was originally run. For the first batch of each epoch, the previous epoch’s samples serve as the baseline. The probe is always out-of-sample relative to the batch that generated the proposals.

2.3.0.2 Acceptance criterion.

Probe labels from earlier batches reflect library states that may differ from the current library \(S\), so we re-run \(S\) on the probe for a fresh causal baseline. Let \(\mathcal{P}_{\mathrm{fail}}\) and \(\mathcal{P}_{\mathrm{pass}}\) denote the previously-failing and previously-passing subsets. Running \(S\) yields baseline counts \(F_0 = |\{x \in \mathcal{P}_{\mathrm{fail}} : x \text{ passes under } S\}|\) and \(R_0 = |\{x \in \mathcal{P}_{\mathrm{pass}} : x \text{ fails under } S\}|\). For a candidate library \(S^c\), the analogous counts \(F(c)\) and \(R(c)\) are computed on the same probe. A candidate is accepted only if \[\bigl(F(c) - F_0\bigr) - \bigl(R(c) - R_0\bigr) > 0 \quad \text{and} \quad R(c) \leq R_0. \label{eq:acceptance}\tag{1}\] The first condition requires a net improvement in fixes over regressions, the second is a hard regression budget. The highest-scoring candidate satisfying both is applied, otherwise the library is left unchanged. If the winning candidate introduces any regression, a contrastive revision step prompts the skill writer to narrow the trigger or add a guard clause, and the revision replaces the original only if its adjusted score is strictly higher and still satisfies the regression budget (Appendix 8.2).

3 Experimental Setup↩︎

3.1 Benchmarks↩︎

Our primary evaluation uses three FHIR-based clinical agent benchmarks. MedAgentBench [1] requires an agent to query a live FHIR server, reconcile medication orders, and verify writes through follow-up reads. MedAgentBench-v2 [10] extends it with multi-step decision tasks, coordinated writes, and clinical safety protocols. FHIR-AgentBench [11] evaluates structured clinical QA and tool use on an independent FHIR environment. Each is split into disjoint development, validation, and held-out test sets (Table 4), with additional out-of-distribution (OOD) splits for the two MedAgentBench variants. We report exact-match accuracy against ground-truth answers and FHIR-server state (scoring details in Appendix 10). For evidence of generality, we also evaluate on four AgentBench [14] environments, ALFWorld [12], WebShop [13], OS Interaction, and DBBench.

3.2 Models↩︎

We evaluate Grasp with five base models spanning open and proprietary families and a range of capability. The open-source models are gpt-oss-120b, an open model that fits on a single H200 GPU, and DeepSeek V4 Flash, a larger long-context alternative, both self-hosted. The proprietary models are Gemini 3.1 Flash Lite, GPT-5.4 (low reasoning effort), and GPT-4.1, three frontier models accessed via official provider APIs. Exact handles, sizes, precision settings, and access dates appear in Appendix 9. Within a run, the same model serves as both executing agent and skill-writer.

3.3 Baselines↩︎

We compare Grasp against a no-skills baseline and five self-improvement methods, all using the same base agent, decoding settings, and training episodes. For every method, the same model serves as executing agent and as the LLM that writes notes, rules, or skills.

3.3.0.1 Sequential memory [10]

appends a natural-language correction note to a flat memory block after each failing episode. Notes are task-specific conditional instructions, and the memory grows unbounded with no acceptance gate.

3.3.0.2 Batch memory

is identical but updated once per batch, matching Grasp’s cadence and isolating the acceptance gate from update frequency.

3.3.0.3 ExpeL [6]

extracts contrastive natural-language rules by pairing each failing trace with its nearest successful trace and prompting an LLM to propose AGREE/EDIT/REMOVE/ADD operations on a bounded rule list. Rules are scored by accumulated operation counts and capped at 20.

3.3.0.4 Evo-MedAgent [16]

maintains episodic and semantic memory stores with top-\(k\) retrieval and per-rule utility tracking, but no probe-based acceptance gate.

3.3.0.5 SkillX [17]

extracts skills from successful trajectories via LLM decomposition and retrieves them by token overlap, with no regression control.

All methods inject into the same field of the task prompt, isolating the learned content and update rule from placement effects (Appendix 8.5).

3.4 Training and Evaluation Protocol↩︎

3.4.0.1 Training.

Each method trains for 5 epochs over the dev split with a batch size of 48 episodes. After every epoch, validation accuracy is computed, the learned state is checkpointed when it improves, and the best checkpoint is restored after the final epoch. The test and OOD splits are evaluated once on this checkpoint and never participate in training, checkpointing, or hyperparameter selection. For Grasp, the probe contains 36 examples from episodes completed earlier in the epoch, and the skill writer generates 4 candidate edits per batch.

3.4.0.2 Cross-model transfer.

For each transfer cell, the source model \(M_s\) runs a complete Grasp training cycle as both executing agent and skill-writer. The resulting library is selected by best-validation on \(M_s\), frozen, and applied to the target model \(M_t\) at test time, with no further training, retrieval re-fitting, or library editing.

3.4.0.3 Compute and context budget.

Methods differ in where they spend training compute and how much context they inject at inference. Grasp uses roughly \(3\times\) more training-time LLM calls per batch than the simplest memory baselines, almost entirely from probe-based validation, since each batch evaluates \(K+1\) library variants on a 36-episode probe. At inference the picture inverts. Grasp libraries average 5 skills and 5.6k tokens, while the memory and skill-library baselines grow to 34–53k (Appendix Table 2). Only Grasp and ExpeL stay below 10k inference tokens.

3.4.0.4 Reporting and statistics.

Headline numbers are test accuracy unless otherwise indicated, reported as mean \(\pm\) standard deviation across seeds. Main results and benchmark transfer use five seeds for open-source models and three for proprietary, while ablations, sensitivity analyses, and model transfer experiments use three seeds. For every major claim in Section 4, Appendix 11.4 reports the mean difference \(\Delta\) with a 95% bootstrap CI from 10,000 resamples, Cohen’s \(d\), and permutation \(p\)-values.

3.5 Ablations↩︎

We ablate eight design choices of Grasp on gpt-oss-120b on MedAgentBench with three seeds. No failure grouping passes all failed traces to the skill-writer at once instead of grouping them by mechanism. No regression budget accepts edits on positive net fixes without enforcing \(R(c) \le R_0\). Fixes-only scores candidates on previously failing examples alone, omitting the passing examples that detect regressions. Append-only restricts the skill-writer to ADD, removing modification and deletion. Two no acceptance gate variants drop probe validation entirely and apply every edit unconditionally with FIFO eviction, at \(K{=}4\) and \(K{=}1\). Finally, two matched-compute variants separate the probe’s validation signal from the cost of running it, scoring all \(K{+}1\) candidates on the full probe as Grasp does but selecting the applied edit without the scores, one by the skill-writer’s preference and one at random. One-at-a-time sweeps over \(B\), \(N\), \(K\), and the invalid-action weight \(\lambda\) appear in Appendix 11.2.

4 Results↩︎

4.1 Benchmark Results↩︎

Table [tab:main] reports test accuracy on MedAgentBench and MedAgentBench-v2. On MedAgentBench, Grasp is the strongest method on every model and on both the in-domain and OOD splits. In-domain, it lifts gpt-oss-120b from 40.6% to 88.8%, a gain of 48.2 points over the no-skills baseline and 21.0 over the strongest baseline (Evo-MedAgent, 67.8%), and the pattern replicates across families, GPT-5.4 (low) from 45.1% to 85.4% (+40.3), GPT-4.1 from 45.5% to 84.9% (+39.4), DeepSeek V4 Flash from 47.7% to 70.0% (+22.3), and Gemini 3.1 Flash Lite from 54.2% to 71.4% (+17.2). On the OOD split the lead is larger, reaching 56.3% on gpt-oss-120b against 31.3% for the best baseline (a 25.0-point gain), with similarly large margins on DeepSeek (52.3% vs 20.0%), GPT-4.1 (27.2% vs 2.2%), and GPT-5.4 (80.6% vs 16.7%), and a narrower lead on Gemini (41.7% vs 38.3%). On MedAgentBench-v2 the picture is more mixed. Grasp leads the in-domain test split on three of the five models, improving gpt-oss-120b from 61.1% to 76.9% (+15.8) and exceeding the best baseline (ExpeL, 67.8%) by 9.1 points, with smaller leads on DeepSeek (30.9% vs 25.3%) and Gemini (67.2% vs 65.1%). On GPT-4.1 and GPT-5.4 it is within noise of the strongest baseline, where the dominant failure is exhausting the 8-action budget on two paginated-search task types and no method beats no-skills (Appendix 11.6). On the OOD split the methods are closely matched and Grasp does not lead overall, scoring 86.0% on gpt-oss-120b against 87.1% for the best baseline (SkillX) with overlapping standard deviations. We make no superiority claim on v2 OOD.

4.2 Learning Stability↩︎

Figure 1 shows validation accuracy across training epochs on MedAgentBench for gpt-oss-120b. Grasp improves almost monotonically and stabilizes above 85% from epoch 3. Sequential and Batch memory regress below the no-skills baseline as memory accumulates, ExpeL and SkillX stagnate near baseline, and Evo-MedAgent reaches competitive validation accuracy on some epochs but oscillates by more than 20 points between adjacent checkpoints. Seed variance corroborates the dynamic, with Evo’s standard deviation across five seeds of 14.1 against Grasp’s 5.8 at a 21.0-point higher mean.

4.3 Cross-Model Transfer↩︎

Table [tab:cross-model-transfer] reports cross-model transfer on MedAgentBench. Each column fixes an executor model and each row identifies the source model that produced the frozen skill library applied at inference. Diagonal entries reproduce the in-domain results of Table [tab:main], and off-diagonal entries apply a library to a different executor without further adaptation. The clearest effect is on OOD task types, where libraries written by GPT-5.4 (low) improve every executor over its own self-trained library. Applied to gpt-oss-120b, GPT-5.4 skills reach 77.8% OOD against 56.3% from gpt-oss-120b’s own library, a 21.5-point gain, while trading 12.8 points in-domain (88.8% to 76.0%). Applied to Gemini 3.1 Flash Lite, they lift OOD from 41.7% to 71.1% (+29.4) and in-domain test from 71.4% to 76.6% (+5.2). GPT-5.4 is the strongest cross-model source on every transfer cell except Gemini in-domain test, where the gpt-oss library is marginally stronger. Libraries from weaker source models applied to stronger executors are consistently worse than self-training. Off-diagonal transfer with Gemini as the source is high-variance (OOD standard deviations above 30 points) and we draw no conclusions from those cells.

4pt

4pt

4.4 Cross-Benchmark Transfer↩︎

Table [tab:cross-benchmark-transfer] reports cross-benchmark transfer on gpt-oss-120b. Each row fixes a target benchmark and split, and each column applies a frozen library trained on a different source. Libraries trained on the DBBench environment [14] serve as a non-clinical control.

Transfer between MedAgentBench and MedAgentBench-v2 is positive in both directions. MedAgentBench skills applied to MedAgentBench-v2 reach 78.1% test and 86.7% OOD, exceeding v2 self-training (76.9% and 86.0%). In reverse, v2 skills applied to MedAgentBench reach 51.6% test against a 40.6% baseline, with positive but high-variance OOD transfer (34.4% against an 8.7% baseline, \(\pm\)​22.8). On FHIR-AgentBench, self-training improves gpt-oss-120b from 51.5% to 58.2% across five seeds (\(\Delta = +6.8\), 95% CI \([3.7, 10.0]\)), while every library transferred into it degrades the baseline (MedAgentBench 47.2%, MedAgentBench-v2 45.7%, DBBench 48.3%). Transfer in the reverse direction, out of FHIR-AgentBench onto the MedAgentBench family, does not show this degradation. The DBBench control transfers negatively or neutrally to every clinical target.

4.5 Ablations↩︎

Table [tab:ablation] ablates the Grasp design choices defined in Section 3.5 on MedAgentBench with gpt-oss-120b. The results separate into three tiers. The full method reaches 88.8% test accuracy. Removing any single component while keeping the acceptance gate stays above 80%, above every comparator in Table [tab:main]. Removing failure grouping drops to 84.4% (\(-\)​4.4), the regression budget to 81.8% (\(-\)​7.0), and fixes-only and append-only both reach 80.2% (\(-\)​8.6), with fixes-only the least stable at \(\pm\)​13.7.

Removing the acceptance gate entirely is more damaging. Applying every proposed edit unconditionally falls to 63.5% (\(-\)​25.3) at \(K{=}4\) and to 40.1% (\(-\)​48.7) at \(K{=}1\), the latter matching the no-skills baseline (40.6%) within seed noise. The matched-compute variants spend Grasp’s full per-batch probe budget but select the applied edit without the probe scores, reaching 70.8% (proposer) and 67.2% (random), within the variance of the no-gate \(K{=}4\) variant (63.5%).

4.6 Non-Medical Environments↩︎

Table 1 reports test accuracy on four non-clinical AgentBench environments using gpt-oss-120b across three seeds. Grasp produces a large gain on ALFWorld (+28.4) and a substantial gain on WebShop (+20.6), where tasks recur with verifiable structure, a smaller gain on DBBench (+5.0), and no gain beyond seed noise on OS Interaction (+0.9), where the action space is open-ended.

5 Discussion↩︎

Grasp treats agent self-improvement as a sequence of validated edits to a bounded library of behavioral skills, each candidate gated by a regression-aware probe of previously failing and passing examples. The design responds to a specific failure mode of prior verbal-feedback methods, which accumulate guidance monotonically without checking that each item preserves existing behavior. Our results indicate that the gain over those methods comes from validated skill editing rather than from better skill writing. The skill-writer without validation at \(K=1\) reaches \(40.1\%\), matching the no-skills baseline, so skill writing alone is no better than no skills. Generating \(K=4\) candidates without validation reaches \(63.5\%\), so comparative generation contributes substantially even with no gate, and adding the acceptance gate brings the full method to \(88.8\%\). This gain is not an artifact of the probe’s compute, since spending the full probe budget but discarding the verdict stays within the variance of the no-gate variant that never runs the probe. Sequential and Batch memory update more often than Grasp yet regress below the no-skills baseline as context accumulates, illustrating that prompt-based agents forget through dilution rather than parameter drift, which a held-out probe naturally detects.

Cross-model and cross-benchmark transfer together indicate that the learned skills encode environment-specific procedural knowledge rather than model-specific in-context patterns or general clinical competence. The clearest signal is on held-out task types, where skills written by GPT-5.4 (low) improve every executor’s OOD accuracy over what it learns for itself, while the in-domain effect is smaller and can cost a few points on the strongest executors. The direction is asymmetric. A stronger writer improves a weaker executor, but libraries from weaker writers applied to stronger executors are consistently worse than self-training. This asymmetry is specific to Grasp. The same protocol applied to all five baselines leaves every one near or below 35% OOD after transfer (Appendix 11.5), indicating that the gate, not writer strength, makes a library transferable. A natural reading is that a stronger writer articulates operational facts about the environment that a weaker executor can follow even when it cannot generate them, a distillation-like effect without parameter updates, and the gain concentrating on held-out task types fits this account. Transfer across benchmarks bounds the same knowledge from the other side. Skills move between MedAgentBench and MedAgentBench-v2, which share a tool-calling convention, but degrade FHIR-AgentBench below its own baseline, because a library written against one convention emits malformed calls under FHIR-AgentBench’s Python-tool and free-text interface. What transfers is procedural knowledge tied to an environment and its interface, not clinical FHIR competence in general.

The non-medical results delineate where the mechanism applies. Grasp helps most where tasks recur with predictable structure and side effects are directly verifiable (ALFWorld, WebShop), and least where the action space is open-ended and consequences are diffuse, as on OS Interaction, where the gain does not exceed seed noise. DBBench falls between, since only a subset of its query types exhibits the recurring, checkable patterns a skill can target. A separate boundary is set by resource limits, not task structure. On MedAgentBench-v2 the proprietary models fail most often by exhausting the fixed 8-action budget on two paginated-search task types, which no method learned to avoid (Appendix 11.6). The common factor across domains is not subject matter but task structure, recurring failure modes paired with a verifiable signal the probe can use to admit or reject an edit.

Two properties make the mechanism practical to deploy. First, bounded libraries outperform unbounded memories at a fraction of the inference cost. Grasp reaches higher test accuracy than every memory baseline while injecting roughly an order of magnitude fewer tokens (Section 3.4), consistent with prior findings that LLMs degrade under irrelevant context [7], [8]. Because the library stays capacity-bounded, this overhead does not grow with training length. Second, the library is auditable by design. Each skill is a versioned Markdown document recording which failure mode prompted it, which probe score justified its acceptance, and which earlier skill it replaced (Appendix 12), so a reviewer can trace why an agent behaves as it does and revert a specific behavior without retraining. Both properties matter most where an agent must be inspected and corrected after deployment rather than retrained, as in a clinical setting.

6 Conclusion↩︎

Grasp converts agent failures into validated edits to a bounded library of behavioral skills, each candidate gated by a regression-aware probe of previously failing and previously passing examples. Against the strongest of five memory-based baselines, Grasp adds 21.0 points on MedAgentBench, and the gain holds across five base models on two FHIR clinical benchmarks. Ablations localize it to the validation gate rather than the skill-writing process, which alone closes most of the gap against the no-skills baseline. Frozen libraries written by a stronger model improve weaker executors beyond what they learn for themselves while the reverse does not, an asymmetry that no ungated baseline reproduces and that, with the way skills break across interface boundaries, indicates the libraries encode environment-procedural knowledge rather than model-specific patterns. The same mechanism helps across clinical and non-clinical environments where tasks recur with verifiable structure, and is flat only where the action space is open-ended. Treating agent self-improvement as gated editing of a bounded library, rather than monotonic accumulation of guidance, offers a path toward iterative refinement of LLM agents in environments where procedural reliability matters.

6.0.0.1 Code and Data Availability.

We release a unified codebase that implements Grasp and all five comparators against a shared agent and evaluation interface, with benchmark adapters for seven structured-agent environments spanning clinical FHIR and AgentBench tasks (https://github.com/jomoll/GRASP). A new self-improvement method can be added through a single update interface and run on any of these benchmarks without further integration. To make every reported result inspectable and reproducible, the repository also includes the learned skill libraries, the frozen transfer libraries, all prompts, and per-seed validation, test, and OOD accuracies for every cell of Tables [tab:main]–1.

6.0.0.2 Acknowledgments.

This work was supported by Bayern Innovativ (Bavarian State Ministry of Economics) under Grant KK6052501. We also thank François Beaulieu and Paul Vozila of Microsoft Healthcare & Life Sciences for their support of this project.

Limitations↩︎

Grasp is evaluated on benchmarks that score procedural reliability against ground-truth FHIR state rather than clinical correctness or patient outcomes. MedAgentBench and its variants use curated patient records, a containerized FHIR server, and exact-match scoring against pre-specified answers. Real clinical environments might differ in ways the benchmarks do not capture, including incomplete and contradictory records, large free-text note volumes, non-standard local codes, multi-system integration, and workflows that depend on context outside the EHR. All clinical evaluation is on English-language FHIR data, and generalization to non-English content and to proprietary EHR systems that do not expose FHIR is open. The skills Grasp learns are policies tuned to a benchmark rather than medical expertise, and translating the mechanism to a clinical setting would require clinician review of each skill, prospective comparison against current practice, and monitoring under live workflow conditions, none of which are reported here.

The validation step is the source of Grasp’s stability but also its dominant cost. Each batch evaluates \(K+1\) candidate libraries on a 36-episode probe, contributing roughly 440 agent calls per batch and accounting for the majority of training-time compute. The “no parameter updates” framing should not be read as “no cost”. The cost has moved from gradient updates to LLM inference. In settings where dev-split episodes are expensive, for example live clinical environments or slow tool interfaces, the probe size that delivered our reported results may need to be reduced, with the trade-off between probe size and gate reliability characterized in Appendix 11.2. Grasp’s probe compute is not budget-matched to the baselines, so its advantage over them could in principle reflect the extra compute the gate consumes rather than the gate’s decisions themselves. We separate the two within Grasp itself. Our matched-compute ablations (Section 4.5) hold the compute fixed and vary only the gate, spending the full per-batch probe budget but discarding its verdict when selecting an edit. Accuracy stays at the no-gate level, attributing the gain to the validation decision rather than to the probe compute. Whether the gate would also help the baselines if added to them is a separate question, one of generality across methods rather than of Grasp’s own mechanism, and we do not address it here.

The cross-model transfer result is asymmetric. Stronger writers produce libraries that improve weaker executors beyond self-training, but the reverse direction is consistently worse than self-training. A frozen library is therefore not fully model-agnostic in practice, and deploying a library written by one model on a different one requires verifying that the new executor is not weaker on the procedural axes the writer assumed. We do not study how to detect such mismatches automatically, and the present results give no guidance on choosing a writer model for a given target executor beyond “use the strongest available”. The asymmetry itself is specific to Grasp. We ran the cross-writer protocol on all five baselines in the two configurations where Grasp shows its strongest effect (GPT-5.4 to Gemini and to gpt-oss-120b), and none reproduces it (Appendix 11.5). We do not extend this comparison to the full writer-executor matrix, which we compute only for Grasp.

Three of the five base models we evaluate (GPT-5.4, GPT-4.1, Gemini 3.1 Flash Lite) are accessed through provider APIs that may change without notice. Exact model handles and access dates are reported in Appendix 9, and the proprietary rows of our results tables reflect access conditions specified there.

7 Related Work↩︎

7.0.0.1 Verbal-feedback self-improvement.

Reflexion [5] introduced the paradigm in which an agent writes natural-language critiques of its own failed trajectories and conditions on those critiques on subsequent attempts. ExpeL [6] extends this to cross-trajectory rule extraction with bounded AGREE, EDIT, REMOVE, or ADD operations. A critical survey [18] documents that LLMs often cannot reliably correct their own reasoning without external feedback. These approaches accumulate or revise guidance monotonically with no held-out check that a new piece of feedback preserves previously correct behavior. Grasp retains the verbal-feedback paradigm but gates every candidate edit on a regression-aware probe of previously failing and passing examples.

7.0.0.2 Learning from agent failures.

AutoGuide [19] creates context-aware guidelines from paired trajectories with different outcomes and retrieves them at inference. AgentDebug [20] introduces a modular taxonomy of agent failures and a debugging framework that traces failures to their root cause. ECHO [21] adapts hindsight experience replay to LM agents, generating counterfactual successful trajectories from failed ones. Experiential Reflective Learning [22] extracts heuristics from single-attempt trajectories and retrieves them at test time. Grasp shares the failure-driven orientation but differs on three axes. The unit of learning is a structured behavioral skill rather than a retrieved trajectory or heuristic, every edit passes through an explicit regression-aware gate rather than relying on accumulation or retrieval, and modification and removal are first-class operations, so the library evolves rather than grows.

7.0.0.3 Skill-library agents.

Voyager [9] established skill libraries as accumulated behavioral primitives conditioned on at inference. SkillX [17] extends this to general agent tasks by decomposing successful trajectories into named skills retrieved by token overlap, and SAGE [23] integrates a skill library into RL-based agent training. These methods grow their library without validating that each addition preserves prior behavior, and draw skills only from successful trajectories. Grasp retains the structured-library design but is failure-driven, supports modification and removal, gates every edit on probe-based net improvement, and requires no parameter updates.

7.0.0.4 Memory-augmented LLM agents.

MemGPT [24] introduced managed long-term memory with explicit page-in and page-out operations, and A-Mem [25] organizes agent memories into dynamic interconnected knowledge networks with agent-driven indexing decisions. In our experimental comparison, sequential and batch memory [10] append correction notes after each failing episode or batch, and Evo-MedAgent [16] maintains episodic and semantic memories with top-\(k\) retrieval. None of these methods evaluate updates against a held-out probe and the memory grows monotonically over training. In our experiments this leads memory baselines to either regress below the no-skills baseline as context dilutes (Sequential, Batch) or oscillate sharply between epochs (Evo-MedAgent), confirming that update frequency and retrieval are not substitutes for an explicit acceptance criterion.

7.0.0.5 Prompt and program optimization.

DSPy [15] treats the natural-language portion of an LLM pipeline as parameters optimized against a development set, using held-out validation to compare candidates. Broader prompt-optimization methods [26], [27] cast prompt design as search guided by held-out performance. Grasp shares the held-out validation idea but applies it at a different granularity. The optimized object is a structured library of behavioral skills with add, modify, and remove operations rather than a flat prompt or fixed program template, and the acceptance criterion explicitly bounds regressions on previously correct behavior. This connects Grasp to the broader continual-learning literature on catastrophic forgetting [28].

7.0.0.6 Clinical agent benchmarks.

LLM evaluation in clinical settings has historically focused on question answering. Med-PaLM [29] established MultiMedQA as a benchmark combining medical licensing exam questions, biomedical literature, and consumer health queries. These benchmarks test medical knowledge but not the procedural reliability required to operate in a clinical environment. MedAgentBench [1] introduced FHIR-based tasks requiring resource retrieval, medication reconciliation, and verified writes. MedAgentBench-v2 [10] extends the original with multi-step decision tasks, coordinated writes, and clinical safety protocols. FHIR-AgentBench [11] evaluates structured clinical question answering and tool use on an independent FHIR environment.

7.0.0.7 Structured-environment agents beyond clinical tasks.

ReAct [30] introduced the thought-action-observation loop that underlies most modern tool-using agents, and Toolformer [31] demonstrated that LLMs can learn to invoke external APIs without explicit supervision. AgentBench [14] provides a standard suite for operating systems, databases, and web interfaces, while Mind2Web [32] and WebArena [33] target the web and SWE-bench [34] targets software engineering. The non-medical experiments in Section 4.6 draw from the AgentBench environments.

7.0.0.8 Context degradation under long inputs.

A consequence of monotonic memory accumulation is that task-relevant guidance is diluted by outdated or redundant entries. [7] showed that LLMs are sensitive to irrelevant context inserted into their inputs, and [8] documented systematic position-based degradation in long-context retrieval. These findings motivate the bounded-library design of Grasp, which keeps the active skill set small by allowing removal and modification rather than only addition.

8 Method Details↩︎

8.1 Per-Batch Algorithm↩︎

Algorithm [alg:grasp] gives the full per-batch GRASP update referenced in Section 2.3.

8.2 Contrastive Revision↩︎

When the highest-scoring candidate \(c^\star\) satisfies Equation 1 but causes at least one regression on the probe, the skill-writer is invoked a second time with the original proposal and the regressing traces, and asked to produce a narrower version that preserves the original fixes while exempting the regression patterns. The revision is evaluated against the same probe and the same \(F_0, R_0, \mathcal{E}_0\) as the original candidate. It replaces \(c^\star\) only if its adjusted score \((F - F_0) - (R - R_0)\) is strictly higher and it satisfies \(R \leq R_0\). If the revision fails either condition, the original candidate is applied unchanged.

8.3 Invalid-Action Regression Weighting↩︎

Regressions from invalid agent actions (malformed tool calls, parse failures, or other agent-side errors the environment rejects) are weighted by \(\lambda = 2\) in the score, while the hard budget \(R(c) \leq R_0\) uses the unweighted count. The asymmetry reflects cost, not tuning. An invalid action fails the episode at the interface level, and under transfer a single rejected call ends the trajectory, so trading a wrong answer for a malformed call is a worse regression than the raw count indicates. The score penalizes these edits more heavily to discourage them without forbidding them, while the budget stays an unweighted count to keep the constraint interpretable. The result is insensitive to \(\lambda\) (Appendix 11.2), since invalid-action regressions are rare on the probe.

8.4 Skill Template↩︎

GRASP-authored skills follow the structure below, mirroring the Agent Skills SKILL.md convention. Selected examples appear in Appendix 12.

---
name: skill_name
description: one-line description
tags: [tag1, tag2]
version: 1
provenance:
  epoch: 0
  update_cycle: 2
  action: ADD
  probe_score: 4
---

## Trigger
When [specific condition]...

## Rule
You must [behavioral instruction]...

## Verification
After [action with side effect], confirm...

## Example
Failing: [failing trajectory]
Corrected: [corrected trajectory]

8.5 Skill and Memory Injection↩︎

Methods differ in how learned content reaches the agent at inference, in where it is placed, how much is injected, and how often it is refreshed. We hold the placement fixed across methods so that it cannot account for the accuracy differences, and report the remaining differences in injected volume in Appendix Table 2. On the first decision turn, all methods insert their learned content into the same structured field of the task prompt, the behavioral_skills field that Grasp uses, falling back to a prepend before the task description only when the prompt is not structured as JSON. Grasp injects its skill library, Sequential and Batch memory inject their accumulated correction-note block, Evo-MedAgent injects its retrieved episodic and semantic memory, ExpeL injects its numbered rule list, and SkillX injects its top-k retrieved skills. Each method keeps its own internal formatting of that content, but the injection point is common to all. On turns after the first, the memory and retrieval methods refresh their content per query in their native manner, appending the current block to the running context, while Grasp’s library is fixed for the episode and is not re-injected.

5pt

3pt

Holding placement fixed isolates two things that still vary across methods, the learned content itself and the update or selection rule that produces it, which is what the comparison is meant to test. The differences that remain do not favor Grasp. End-of-training injected-token counts in Appendix Table 2 show that Grasp injects roughly an order of magnitude fewer tokens than the unbounded memory baselines and than SkillX, so the gain cannot come from injecting more text. Batch memory shares Grasp’s injection field and its per-batch update cadence yet regresses below the no-skills baseline (34.4 versus 40.6 on gpt-oss-120b), indicating that injecting content into this field without an acceptance check is not by itself helpful. The no-gate ablation in Section 4.5 removes only the acceptance gate while leaving Grasp’s injection and proposal machinery unchanged, and test accuracy drops by 25.3 points, attributing the gain to the gate rather than to placement.

9 Model Specifications↩︎

Table 3 lists the exact model identifiers, providers, and access details for the five base models used throughout this work. Open-source models (gpt-oss-120b, DeepSeek V4 Flash) are self-hosted using vLLM on an NVIDIA H200 GPU. Proprietary models are accessed via official provider APIs. All agent execution uses temperature \(0.0\) with top-\(p = 1.0\). The skill-writer uses temperature \(0.7\) with top-\(p = 1.0\) for proposal diversity (see Section 3.2). Output is capped at 32,768 tokens per call across all methods and models, input prompts use each model’s default context window without manual truncation.

10 Benchmark and Split Details↩︎

4pt

Table 4 reports split sizes for all seven benchmarks. The three clinical benchmarks share a common training and evaluation protocol. The four AgentBench environments are used as exploratory non-clinical probes.

10.0.0.1 MedAgentBench.

Ten clinical task types (300 samples, 30 per type) executed against a live FHIR server hosted in Docker [1]. Tasks include resource retrieval, medication reconciliation, and verified writes. Two task types (tasks 6 and 7) are held out as the OOD split. The remaining eight task types are partitioned per task type into dev (12), val (10), and in-domain test (8) with a fixed seed. Scoring is exact-match against ground-truth answers and FHIR-server state after the agent completes its trajectory.

10.0.0.2 MedAgentBench-v2.

A redesigned variant with ten new task types covering imaging follow-up, anticoagulation reconciliation, vital-sign aggregation, catheter-removal protocols, oncology referrals, hypothyroidism management, QTc-prolongation protocols, opioid and naloxone pairing, and vaccination refresh [10]. The task types incorporate multi-step decision logic and clinical safety protocols absent from v1. OOD tasks (5 and 7) differ from v1, so the OOD split exercises an independent set of held-out clinical workflows. Split sizes, scoring, and FHIR backend match v1.

10.0.0.3 FHIR-AgentBench.

A read-only structured clinical question-answering benchmark on MIMIC-IV FHIR data hosted on the Google Cloud Healthcare API [11]. Tasks require retrieval and reasoning over FHIR resources but do not write. Evaluation differs from the MedAgentBench variants. Answer correctness is assessed by an LLM judge against a ground-truth answer with prompt-level normalisation, and retrieval is scored by precision and recall against ground-truth resource IDs. The single headline number reported in this paper is LLM-judged answer accuracy. The benchmark is partitioned into dev (120), val (80), and test (80) with no OOD split.

10.0.0.4 AgentBench environments.

Four environments from the AgentBench suite [14] are evaluated as exploratory non-clinical probes. These are Database (DBBench), Operating System (OS Interaction), ALFWorld [12], and WebShop [13]. Split sizes are listed in Table 4. DBBench combines real queries from the standard split (partitioned 60/40 by query type) with synthetic aggregation queries. OS Interaction uses worlds 1 to 5 and 7 split 60/40 stratified per world, with world 6 held out. ALFWorld partitions its standard split 60/40 stratified by task type. WebShop uses a round-robin partition of the standard webshop-std index range (instances 0–199) into dev (100), validation (50), and test (50). Scoring uses each environment’s native success criterion (task completion, query correctness, or attribute-matching reward).

11 Extended Results↩︎

11.1 Failure-Mode Classifier: Within-Run Vocabulary Dynamics↩︎

The failure-mode classifier (Section 2.2) is open-vocabulary. At each batch it assigns each failing trace a label, preferring prior labels and minting new ones only when no prior label fits. We characterize two within-run properties of the resulting vocabulary on MedAgentBench and MedAgentBench-v2 with gpt-oss-120b, across the five seeds used in the main results.

4pt

Table [tab:classifier-vocab] reports the mean cumulative number of distinct labels seen by batch index. The first batch discovers 12 to 13 labels, and the marginal addition per batch drops to 1 or 2 by mid-training. The vocabulary converges rather than expanding indefinitely.

4pt

Table [tab:classifier-reuse] reports the fraction of failing traces per batch whose label was already present in this seed’s vocabulary. Reuse climbs from 0% at batch 0 (by construction) to 80–90% by batch 4 or 5 and above 90% by batch 7. After the first epoch, the classifier overwhelmingly reuses prior labels rather than minting new ones.

Together, these two properties show that the classifier converges on a stable label set within a single training run. This is the property the skill-writer needs. By mid-training, failures within a batch are grouped under labels the skill-writer has already seen, allowing edits to target recurring mechanisms rather than novel ones.

11.2 Hyperparameter Sensitivity↩︎

6pt

Table [tab:sensitivity] reports a one-at-a-time sensitivity sweep over the parameters governing the per-batch update loop. These are the batch size \(B\) (dev episodes per update), the probe size \(N\) (episodes used to evaluate candidate libraries), the number of candidate edits \(K\) per batch, and the invalid-action regression weight \(\lambda\) (Appendix 8.3). Each block varies one parameter around the defaults \(B{=}48\), \(N{=}36\), \(K{=}4\), \(\lambda{=}2\) used throughout the paper, with the default run reused as the reference point in every block.

Grasp is robust to \(B\), \(N\), and \(\lambda\) but sensitive to \(K\). Varying \(B\) across \(\{24, 48, 96\}\) moves test accuracy by at most 2.1 points (87.0 to 89.1), all within one standard deviation of each other. Probe size \(N\) has a floor at 16 (82.3 test) and reaches its full value by 36 (88.8), with no further gain at 72, which the gate-selectivity analysis (Appendix 11.3) attributes to the gate already resolving its decisions at \(N{=}36\) rather than to a probe too small to discriminate. Varying \(\lambda\) across \(\{1, 2, 4\}\) leaves in-domain test accuracy in an 84 to 89 band, with all three settings overlapping in standard deviation. \(K\) is the exception. At \(K{=}1\) test accuracy falls to 73.4 (\(\pm\)​23.1), since a single candidate leaves the gate with a binary accept-or-reject decision and removes the comparative selection that sets Grasp apart from naive verbal-feedback methods, and \(K{=}8\) trails \(K{=}4\) (84.4 vs 88.8) as more candidates dilute the fixed probe budget. We set \(K{=}4\) to give the gate real comparative work without thinning per-candidate evaluation.

4pt

11.3 Gate Selectivity↩︎

Across five seeds on MedAgentBench with gpt-oss-120b, Grasp applies an edit in 64% of batches and rejects all four candidates in the other 36%. Across all candidates it admits 16%, and on average 1.3 of the 4 per batch clear the regression budget. The accepted candidate’s score is a median of 3 and a mean of 4 fixed examples above the acceptance threshold, with a tail reaching 15. These margins are several examples wide on an 18-positive probe, so the gate resolves its decisions well above the one-example floor, which is why a larger probe does not improve accuracy (Table [tab:sensitivity]).

11.4 Statistical Significance↩︎

Table 5 reports per-comparison statistics for the primary results. For each comparison we report the mean difference \(\Delta\), a percentile bootstrap 95% confidence interval on \(\Delta\) from 10,000 resamples, a two-sided unpaired permutation test (exact when \(\binom{n_a+n_b}{n_a} \le 50{,}000\), else Monte Carlo with \(10^5\) permutations), and Cohen’s \(d\) with pooled standard deviation. We treat \(\Delta\) and its bootstrap CI as the primary statistical evidence and report permutation \(p\)-values as supporting detail. Three-seed rows are reported as effect-size evidence rather than as significance claims, and we do not apply multiple-comparisons correction.

The seed counts in the main tables, five for open-source models and three for proprietary models and all transfer experiments, imply a minimum achievable two-sided permutation \(p\) of \(0.008\) at \(n_a = n_b = 5\) and \(0.10\) at \(n_a = n_b = 3\). The latter floor limits the power of permutation testing on three-seed rows but not the interpretation of \(\Delta\) and its CI computed on the same data.

The primary comparisons show large and reliably positive differences in every cell. On the five-seed open-source comparisons, gpt-oss-120b reaches \(\Delta = +21.0\) with 95% CI \([10.3, 33.8]\) and DeepSeek V4 Flash \(\Delta = +14.1\) with CI \([9.4, 18.8]\), both at \(p = 0.008\). The three-seed comparisons are of similar or larger magnitude with CIs that exclude zero. Gemini 3.1 Flash Lite reaches \(\Delta = +16.2\) with CI \([14.1, 18.2]\), GPT-4.1 \(\Delta = +37.5\) with CI \([27.1, 45.8]\), and GPT-5.4 (low) \(\Delta = +38.0\) with CI \([26.6, 47.4]\), each with the permutation \(p\) capped at \(0.10\) by the three-seed protocol. The corresponding Cohen’s \(d\) values appear in Table 5, where the Gemini-cell \(d\) of \(8.95\) is inflated by an unusually low pooled standard deviation in that row and should not be read as an estimate of typical effect magnitude. The MedAgentBench-v2 in-domain comparison on gpt-oss-120b is also strong, \(\Delta = +9.1\) with CI \([5.3, 13.1]\) at \(p = 0.008\).

The MedAgentBench OOD comparison on gpt-oss-120b shows \(\Delta = +25.0\) with a bootstrap 95% CI of \([3.7, 44.0]\) that excludes zero. Evo-MedAgent’s seed-to-seed standard deviation of \(21.7\) points on this split widens the permutation null to \(p = 0.087\) but does not shift the underlying gap.

On MedAgentBench-v2 OOD, Grasp scores \(1.1\) points below SkillX with a bootstrap CI that includes zero, and we make no claim of superiority on this split. The cross-model transfer experiments use three seeds per cell, so we do not assign \(p\)-values to individual cells and interpret those effects through \(\Delta\) and the standard deviations reported in the main tables. Cross-benchmark transfer and the FHIR-AgentBench in-domain comparison use five seeds per cell, with bootstrap CIs reported alongside the point estimates.

11.5 Cross-Writer Transfer Across Methods↩︎

Table [tab:cross-writer-baselines] tests whether the cross-model asymmetry in Section 4.3 reflects Grasp’s gate or any structured guidance from a stronger writer, by running the cross-writer protocol on all five baselines. For each method, GPT-5.4 (low) acts as both agent and writer during training, the resulting library is frozen, and a weaker executor (Gemini 3.1 Flash Lite or gpt-oss-120b) runs at test time.

4pt

The asymmetry does not replicate on any baseline. Grasp is the only method whose GPT-5.4 library generalizes OOD when transferred, reaching \(71.1\) on Gemini and \(77.8\) on gpt-oss, while no baseline’s transferred library exceeds \(35.0\) (SkillX\(\rightarrow\)Gemini) and most stay below \(20\) on both executors. The same frozen GPT-5.4 library reaches comparable OOD on two different executors, and in both cases the transfer exceeds the target’s own self-trained library (\(41.7\) for Gemini, \(56.3\) for gpt-oss), so the gains track the source library rather than the executor. Writer strength alone does not explain this, since the baselines’ GPT-5.4 libraries are weak even at the source (\(1.7\) to \(16.7\) OOD under self-training), leaving little procedural knowledge for transfer to carry. gpt-oss is a capable executor, yet baseline transfers to it stay near or below its self-train OOD rather than benefiting from the GPT-5.4 source. What survives transfer to a weaker executor is the procedural knowledge the gate selects for, not writer strength alone.

11.6 Action-Budget Failures on MedAgentBench-v2↩︎

On MedAgentBench-v2, the proprietary models GPT-5.4 (low) and GPT-4.1 most often fail by exhausting the fixed 8-action budget rather than by producing an incorrect answer. The effect concentrates on two task types involving paginated FHIR search (tasks 8 and 10), where the agent spends actions paging through search results and reaches the budget before completing the task. These two tasks reach the action limit on roughly 37–40% of attempts for GPT-5.4 (low), independent of the adaptation method.

Table 6 reports the average number of limit-hit episodes per run on MedAgentBench-v2 for GPT-5.4 (low), out of 64 evaluation samples, comparing the bare base model under each method’s evaluation conditions against the method’s best-validation checkpoint. The base-model column varies only by stochastic decoding, and its spread (5.7–7.0) is within Poisson noise for events at this rate. After adaptation, three methods stay at the base rate within noise and three sit modestly above it, but this split does not correspond to whether a method injects skills. Batch Memory, which injects only a flat memory block, shows the same increase as the skill-based methods, while Sequential Memory, which injects the same kind of memory, does not. With two to three runs per method these differences of one to two additional limit hits are directional rather than statistically resolved. None of the methods reduces the underlying failure, which is set by the base model’s behavior on paginated search rather than by any adaptation.

12 Selected Skills Learned by Grasp↩︎

The four skills below were selected by the authors, not sampled at random, to illustrate the format of Grasp-authored skills across a range of conditions. They span four benchmarks (ALFWorld, DBBench, MedAgentBench, MedAgentBench-v2), two authoring models (gpt-oss-120b and GPT-5.4 low effort), and two edit types (ADD and MODIFY). The selection is intended to show what Grasp can produce, not what it typically produces. We do not claim these four are representative of average skill quality, length, or specificity across runs. We include these particular four because they make the structural elements of a skill (trigger, rule, verification, contrastive example, provenance) inspectable across the procedural domains the paper evaluates.

The clinical skills (F.3 and F.4) encode procedures for operating MedAgentBench and MedAgentBench-v2 as software systems. They contain medical-domain vocabulary (drug names, RxNorm-style identifiers, FHIR resource types) because the benchmarks do, but the operational content is about the benchmark’s API, identifier scheme, and scoring criteria, not about real-world clinical care. The RxNorm codes shown in F.3, for example, are the codes used by MedAgentBench-v2’s FHIR fixture, the skill captures the benchmark’s coding scheme rather than the production RxNorm vocabulary. Reading these skills as drafts of clinical decision rules would misidentify what Grasp has learned. They are policies for completing benchmark episodes under exact-match scoring. The clinical scope of these experiments is discussed further in the Limitations section.

Enforce Cleaning Before Placement (non-clinical, recurring workflow constraint; gpt-oss-120b, AgentBench / ALFWorld; ADD)↩︎

---
name: enforce_clean_before_placement
description: Clean target object before placing into final receptacle
tags: [alfworld, workflow-order]
version: 1
provenance:
  epoch: 3
  action: ADD
---

## Pattern Description

The target object must never be placed into its final receptacle before
it has been cleaned. The constraint applies to any task whose
instruction mentions "clean" (e.g. "pick_clean_then_place", "clean the
X before placing") or whose query type is pick_clean_then_place.
Enforcing a cleaning action immediately after pickup and before any
placement guarantees that the environment state satisfies the
cleanliness predicate; otherwise the episode silently fails.

## Triggers

- Instruction contains the word "clean" (case-insensitive) or the
  phrase "clean the <object>".
- Query type is pick_clean_then_place.
- A `take <obj> from <loc>` action has just executed and no prior
  `clean` observation exists for the carried object.

## Example

### Wrong trajectory

Think: The task is "pick_clean_then_place-Spatula-None-Drawer-10".
Act:   take spatula 1 from drawer 2
Obs:   You are now carrying spatula 1.
Think: I have the spatula, place it.
Act:   put spatula 1 in/on drawer 1
Obs:   Spatula 1 is now in drawer 1.
       [Episode ends; goal NOT satisfied.]

### Correct trajectory

Think: Instruction mentions "clean" -- enforce cleaning before
       placement.
Act:   take spatula 1 from drawer 2
Obs:   You are now carrying spatula 1.
Think: Before placing, I must clean the spatula.
Act:   clean spatula 1 with sinkbasin 1
Obs:   Spatula 1 is now clean.
Act:   put spatula 1 in/on drawer 1
Obs:   Spatula 1 is now in drawer 1.
       [Goal satisfied.]

## Pitfalls

- Do not clean after placement.
- If the object is already clean, skip the redundant action.
- Choose a valid cleaning device (sinkbasin, dishwasher); other
  receptacles raise an invalid-action error.
- Do not trigger on tasks that do not mention cleaning; this wastes
  the step budget.

Parse Percentage Strings for Aggregation (non-clinical, low-gain benchmark; gpt-oss-120b, AgentBench / DBBench; ADD)↩︎

---
name: parse_percentage_strings_for_aggregation
description: Strip percent signs and normalise commas before numeric ops
tags: [dbbench, sql, locale-aware]
version: 1
provenance:
  epoch: 2
  action: ADD
---

## Pattern Description

When a task asks for a numeric aggregation (SUM, AVG, MIN, MAX) and
the table stores percentages as text with a comma decimal separator
and a trailing compared or summed. The agent must strip the the comma with a dot before casting to a numeric type. The same rule
applies whenever a WHERE clause filters on such a column.

## Triggers

- Instruction contains a percentage written with a comma and trailing
  - Schema includes a column whose name contains   `Foreign nationals in - Query type is an aggregation and the task mentions a percentage
  filter.

## Recommended Patterns

### Filter on a percentage column

-- WRONG
SELECT SUM(pop) FROM tbl
WHERE `Foreign nationals in 
-- CORRECT
SELECT SUM(pop) AS total
FROM   tbl
WHERE  CAST(REPLACE(REPLACE(`Foreign nationals in                             '            AS DECIMAL(10,2)) = 23.3;

### Aggregate a numeric column that contains commas

-- WRONG (keeps commas)
SELECT SUM(`Population`) FROM tbl;

-- CORRECT
SELECT SUM(CAST(REPLACE(`Population`, ',', '') AS UNSIGNED))
       AS total_population
FROM   tbl;

## Failure Indicators

- Observation shows 0 rows or NULL when a non-zero result is expected.
- SQL error: "Incorrect decimal value: '23,3- Final answer is a string that still includes the 

Opioid–Naloxone Verification (benchmark-procedural with medical vocabulary; gpt-oss-120b, MedAgentBench-v2; MODIFY)↩︎

---
name: opioid_naloxone_verification
description: Ensure active opioid orders have matching naloxone prescription
tags: [medagentbench-v2, medication-safety, rxnorm]
version: 2
provenance:
  epoch: 2
  action: MODIFY
---

## Pattern Description

Every active opioid analgesic order must have a matching naloxone
prescription. The skill encodes a four-step procedure: locate opioids
by explicit RxNorm codes, confirm status == active, search for a
naloxone counterpart, and only if absent, create a naloxone
MedicationRequest.

## Common Failure Patterns

- Missing code filter, treating the whole bundle as if it contained
  only opioids.
- Ignoring status, counting completed or entered-in-error orders as
  active.
- Incorrect naloxone code, matching free-text instead of an exact
  CPT/RxNorm code.
- Empty bundle != no opioids; a bundle may carry non-opioid meds and
  each entry must be checked.

## Procedure

### Step 1. Identify active opioid orders

GET /MedicationRequest?patient={patientId}

opioid_codes = [
  "860975",   # hydromorphone (RxNorm)
  "860976",   # oxycodone
  "860977",   # fentanyl
  "860978",   # hydrocodone
  "860979",   # morphine
]
# Keep entries whose medicationCodeableConcept.coding[].code is in
# opioid_codes AND whose status in {active, draft, on-hold}.

### Step 2. Search the same bundle for naloxone (RxNorm 860980)

If absent and the opioid list is non-empty, POST:

POST /MedicationRequest
{
  "resourceType": "MedicationRequest",
  "status": "active",
  "intent": "order",
  "medicationCodeableConcept": {
    "coding": [{
      "system": "http://www.nlm.nih.gov/research/umls/rxnorm",
      "code":   "860980",
      "display":"Naloxone"
    }]
  },
  "subject": { "reference": "Patient/{patientId}" },
  "authoredOn": "{today_iso}"
}

### Step 3. Report exactly one of three outcomes

FINISH(["No active opioid analgesic orders found ..."])      # case A
FINISH(["Active opioid orders have matching naloxone ..."])  # case B
FINISH(["Naloxone order created for patient ..."])           # case C

## Success Indicators

- Filters coding[].code against the opioid list and checks
  status == active before deciding.
- POSTs naloxone with the exact RxNorm code only when truly missing.
- FINISH message accurately reflects whether a naloxone order was
  created or not.

Patient Identifier to FHIR Reference Resolution (benchmark-procedural, cross-model writer contrast; GPT-5.4 low effort, MedAgentBench; MODIFY)↩︎

---
name: patient_identifier_to_fhir_reference_resolution
description: Resolve MRN to Patient.id before any dependent FHIR request
tags: [medagentbench, fhir, identifier-resolution]
version: 3
provenance:
  epoch: 4
  action: MODIFY
---

## Pattern Description

When a task supplies an MRN it must be treated as a search key, not a
guaranteed downstream resource reference. The agent must first issue
GET /Patient?identifier=..., read Patient.id from the response, and
only then issue dependent searches or writes that need patient= or
subject.reference.

The skill prevents two distinct failure modes: (i) firing dependent
requests before the patient lookup response arrives, and (ii) reusing
the raw input MRN, stripped digits, placeholders, or template text
where a resolved Patient.id is required.

## Common Failure Patterns

- GET /Observation?patient=S0789363&code=A1C without first parsing
  entry[0].resource.id.
- GET /Observation?patient=6550627&code=A1C after stripping the
  leading 'S'.
- "subject":{"reference":"Patient/"} or "Patient/UNKNOWN" in POST
  bodies.
- Concatenating actions on one line, e.g. GET /Patient... GET
  /Observation... FINISH(...) before the first response arrives.
- Repeating the patient lookup multiple times after one successful
  match instead of caching the resolved id.

## Recommended Pattern

# 1. Resolve once.
GET /fhir/Patient?identifier={input_identifier}

# 2. Read the bundle.
resolved_id = bundle.entry[0].resource.id

# 3. Use that resolved id in every later step:
#    - searches:  patient={resolved_id}
#    - writes:    "subject": {"reference": "Patient/{resolved_id}"}

## Example

### Correct trajectory

GET /fhir/Patient?identifier=S0789363
   -> entry[0].resource.id = "S0789363"
GET /fhir/Observation?patient=S0789363&code=A1C
POST /fhir/ServiceRequest  {
  ...,
  "subject":{"reference":"Patient/S0789363"}
}
FINISH([5.2, "2022-08-09T15:33:00+00:00"])

### Wrong trajectory

GET /fhir/Observation?patient=6550627&code=A1C   # stripped 'S'
POST /fhir/ServiceRequest {
  "subject":{"reference":"Patient/UNKNOWN"}
}
FINISH([-1])                                     # never resolved

## Success Indicators

- Bundle is read before any dependent action.
- Exactly one patient lookup per task.
- Every later patient= parameter and Patient/{id} reference matches
  entry[0].resource.id; no transformed, blank, or placeholder
  references appear.

References↩︎