SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents
May 28, 2026
Abstract
Retrieval-augmented LLM agents increasingly rely on curated skill banks: collections of reusable textual principles that guide decision making on complex tasks. Existing approaches typically expand these banks in an append-only fashion, continuously adding new skills without removing redundant, outdated, or harmful ones, resulting in inefficient and poorly curated repositories. In this paper, we formulate skill bank curation as a constrained multi-objective problem: a desirable bank must be useful for the agent, diverse in its content, and provide good coverage of the query distribution. To this end, we introduce SkillBrew, a multi-objective curation framework that formalizes skill bank curation as Pareto-aware optimization under a utility constraint, and solves it via a bi-level propose-then-verify loop. We evaluate our approach on two public benchmarks. Our findings suggest that treating skill banks as objects of principled curation, rather than ever-growing append-only logs, is an important step toward building self-improving LLM agents.
1 Introduction↩︎
Large language model (LLM) agents [1], [2] are increasingly deployed on long-horizon, open-ended tasks where the ability to accumulate and reuse procedural knowledge across episodes determines downstream performance. A widely adopted abstraction is the skill bank: a retrievable collection of distilled, reusable textual or executable artifacts that encode how the agent performs recurring tasks [3]–[5]. This abstraction has been standardized by Anthropic’s Agent Skills specification [6] and is now adopted by modern agent harnesses such as Claude Code. However, the skill bank is not a one-shot artifact: a bank that is simultaneously useful, diverse, and well-covered cannot be obtained by a single distillation pass, and requires iterative refinement that balances these competing properties [3], [7]. Yet how to build, refine, and prune skills jointly as a bank, rather than judging each skill in isolation, remains largely unexamined.
A growing line of work tackles skill bank construction, which we organize into two paradigms (Figure 1). The first, accumulative skill expansion, continuously adds new skills from experience and lets the bank grow over time, as in Voyager’s self-verified skill library [3], Trace2Skill’s hierarchical consolidation of trajectory-local lessons [5], and SkillRL’s failure-driven skill evolution coupled with RL policy updates [8]. The second, local skill-level optimization, retains, edits, or removes skills based on a single scalar curation signal, such as ExpeL’s vote-based Add/Edit operations on cross-task insights [4], top-\(k\) frontier selection on held-out accuracy [9], and non-parametric scoring gates [10]. While these methods advance skill bank construction along different axes, their curation reduces to a single scalar criterion, and bank-level properties such as diversity and coverage remain a less explored dimension.
Principled skill bank curation is challenging for three interrelated reasons. First, the quality of a bank is inherently multi-dimensional: a good bank must be useful, internally diverse, and broadly cover the query distribution. These criteria are mutually constraining: adding skills to broaden coverage tends to introduce redundancy that lowers diversity, while pruning to preserve diversity reduces coverage; they cannot be collapsed into a single scalar without introducing pathologies. Second, credit assignment is difficult: the contribution of any individual skill is observable only through expensive rollouts, and is further confounded by interactions with co-retrieved skills, making it hard to attribute outcomes to specific entries. Third, the objectives are non-differentiable and expensive to query, while the space of admissible edits is open-ended, since new skills are synthesized from trajectory evidence rather than chosen from a fixed pool. This rules out gradient-based optimization and exhaustive search, so curation requires a mechanism that can both propose plausible edits and verify them under the multi-objective criterion. These challenges explain why prior work has defaulted to accumulation without pruning or single-scalar curation rather than principled multi-objective curation.
In this paper, we propose SkillBrew, a training-free framework for iterative skill bank refinement that introduces a third paradigm, global bank-level optimization (Figure 1C), and addresses the above challenges through three design choices. First, we formalize skill bank curation as constrained multi-objective optimization over utility, diversity, and coverage, with utility as an explicit constraint and diversity/coverage as structural regularizers, reflecting the asymmetry that a bank must first be useful before its organization matters. Second, we attribute credit through per-skill counterfactual replay: leave-one-out rollouts isolate the marginal contribution of each skill, providing fine-grained Keep/Rewrite/Remove evidence for bank-level optimization. Third, we navigate the non-differentiable and open-ended edit space via a bi-level propose-then-verify loop, which uses trajectory evidence on a support split to propose candidate edits and Pareto-aware selection on a held-out query split to verify them, preventing overfitting to the trajectories that motivated each edit.
Across two public benchmarks, SkillBrew substantially outperforms strong baselines and transfers across worker LLMs of different scales, suggesting that treating skill banks as objects of principled curation, rather than judging skills in isolation, yields more effective procedural knowledge for retrieval-augmented agents.
2 Related Work↩︎
2.1 From Agent Memory to Agent Skill↩︎
A central challenge in deploying LLM agents [1] on long-horizon tasks is that each task is typically treated as an isolated episode, leaving no straightforward mechanism to accumulate transferable knowledge across tasks. One line of work equips the agent with a persistent memory of past interaction traces [7], [11], retrieved at inference time as references for similar future tasks.
A distinct line abstracts raw experience into skills: compact and reusable procedural artifacts that capture how agents perform recurring tasks. Voyager [3] first brought this paradigm to LLM agents by maintaining a growing library of executable code skills in Minecraft, and subsequent work generalizes the formulation to textual forms, including cross-task insights distilled from trajectories [4] and transferable skills consolidated from trajectory pools via parallel hierarchical analysis [5]. This paradigm has further been standardized by Anthropic’s Agent Skills specification [6]. Across these designs, the skill bank serves as a compact retrievable representation of procedural knowledge, but how to curate this bank iteratively, balancing the utility of individual skills against the diversity and coverage of the bank as a whole, remains largely open.
2.2 Skill Bank Curation↩︎
Early work on skill libraries, exemplified by Voyager [3], treats the bank as an append-only artifact that grows as the agent succeeds, without mechanisms to refine or prune stored skills. More recent work introduces explicit curation over individual skills. ExpeL [4] updates cross-task insights through vote-based ADD/EDIT operations, CoEvoSkills [12] uses a surrogate verifier for failure feedback, Skill-Pro [10] admits and prunes skills through online scoring, and SkillForge [13] closes a failure-driven refinement loop. However, the curation criterion in these methods collapses to a single scalar signal, folding utility, diversity, and coverage of the bank into one ordering, so the trade-offs that arise during curation cannot in principle be resolved.
3 Methodology↩︎
In this section, we present SkillBrew, a training-free framework for principled skill bank curation in LLM agents, as illustrated in Figure 2. We first introduce the core principles underlying effective skill bank construction, formalizing the desiderata of a curated bank in terms of utility, diversity, and coverage. We then cast skill bank curation as a constrained multi-objective optimization problem, where utility serves as the primary constraint while diversity and coverage regularize the bank structure. To solve this problem, we develop a bi-level propose-then-verify loop that iteratively generates candidate edits from trajectory evidence and selects non-dominated banks under a utility constraint.
3.1 Problem Definition↩︎
Let \(\mathcal{S}\) denote the space of admissible skills, where each skill \(s \in \mathcal{S}\) is a piece of reusable procedural knowledge that can be retrieved to guide agent behavior. A skill bank is defined as a subset \(\mathcal{B} \subseteq \mathcal{S}\) serving as the retrieval pool for a frozen agent \(\pi_{\text{agent}}\).
Given a task query \(t\), a retriever \(\mathrm{\small ret}\) returns a subset of relevant skills \(R_t = \mathrm{\small ret}(\mathcal{B}, t) \subseteq \mathcal{B}\), and the agent rolls out a trajectory \[\tau_t = \pi_{\text{agent}}(t \mid R_t)\] conditioned on \(R_t\).
A desirable skill bank should satisfy three complementary principles. First, retrieved skills should be useful, meaning they improve the agent’s task performance. Second, the bank should be diverse, avoiding redundant or overly similar skills. Third, the bank should provide broad coverage over the query distribution so that useful skills can be retrieved across a wide range of tasks. These principles are in tension: a bank of mutually orthogonal skills attains high diversity but offers little utility if the skills are off-task; aggressively pruning to a few high-utility skills sacrifices coverage on the long tail. We therefore cast skill bank curation as a constrained multi-objective optimization: \[\label{eq:cmoo} \begin{gather} \mathcal{B}^{\star} \in \operatorname*{Pareto\text{-}arg\,max}_{\mathcal{B} \in \mathcal{F}} \Big( J_{\mathrm{util}}(\mathcal{B}), J_{\mathrm{div}}(\mathcal{B}), J_{\mathrm{cov}}(\mathcal{B}) \Big), \\ \mathcal{F} = \left\{ \mathcal{B} \subseteq \mathcal{S} \,\middle|\, J_{\mathrm{util}}(\mathcal{B}) \ge \eta \right\}. \end{gather}\tag{1}\]
where \(J_{\mathrm{util}}\), \(J_{\mathrm{div}}\), and \(J_{\mathrm{cov}}\) respectively measure utility, diversity, and coverage, and \(\eta\) is a minimum utility threshold. The solution \(\mathcal{B}^{\star}\) is Pareto-optimal [14]: no feasible bank improves one objective without sacrificing another.
Importantly, we treat utility as the primary constraint, while diversity and coverage act as structural regularizers over the bank. This reflects the asymmetry of the three principles: a skill bank must first be useful to the agent, while diversity and coverage improve its organization, robustness, and generalization. In the next section, we describe the computation of the three objectives in detail.
We partition the task distribution into a support set \(\mathcal{D}_{\text{support}}\) and a query set \(\mathcal{D}_{\text{query}}\). In this paper, all three objectives \(\Big(J_{\text{util}}, J_{\text{div}}, J_{\text{cov}}\Big)\) are computed from agent rollouts on \(\mathcal{D}_{\text{query}}\), while \(\mathcal{D}_{\text{support}}\) is reserved for proposing candidate edits in Section 3.3. This separation prevents the bank from overfitting to the trajectories that motivated its edits.
3.2 Principles of Skill Bank Curation↩︎
We consider three principles that capture complementary and potentially conflicting aspects of skill bank quality: utility measures whether retrieved skills improve agent performance, diversity measures redundancy within the bank, and coverage measures how broadly the bank’s skills are retrieved and exercised across tasks.
3.2.0.1 Utility.
We measure utility through per-skill counterfactual contribution. For a trajectory \(\tau \in \mathcal{D}_{\text{query}}\) retrieving skill \(s\), we replay the agent with \(s\) removed under the same environment, yielding \[\Delta(\tau, s) = r(\tau, s) - r(\tau, \emptyset),\] where \(r(\tau, \cdot) \in [0, 1]\) is the task-level reward (success indicator or normalized score) obtained under the specified retrieval. The utility of \(s\) is computed by averaging over all trajectories retrieving it, \[J_{\text{util}}(s) = \frac{1}{|\mathcal{T}_s|} \sum_{\tau \in \mathcal{T}_s} \Delta(\tau, s),\] where \(\mathcal{T}_s = \{\tau : s \in \mathrm{ret}(\mathcal{B}, \tau)\}\). We then aggregate skill utilities into a bank-level objective, \[J_{\text{util}}(\mathcal{B}) = \sum_{s \in \mathcal{B}} w_s \cdot J_{\text{util}}(s),\] where \(w_s = |\mathcal{T}_s| / N_R\) and \(N_R\) is the number of trajectories with non-empty retrieval.
3.2.0.2 Diversity.
We measure diversity using the geometric volume spanned by skill embeddings. Let \(G\) denote the Gram matrix of L2-normalized skill embeddings in \(\mathcal{B}\). This objective is defined as, \[J_{\text{div}}(\mathcal{B}) = \det(G + \varepsilon I)^{1/|\mathcal{B}|} \in (0,1],\] where \(\varepsilon I\) is a small diagonal regularizer for numerical stability. This formulation penalizes redundant or highly correlated skills, assigning higher scores to banks with structurally diverse skills.
3.2.0.3 Coverage.
We measure coverage as the product of retrieval density and skill usage: \[J_{\text{cov}}(\mathcal{B}) = \mathrm{density} \cdot \mathrm{usage} \in [0, 1].\] The density term measures the average fraction of retrieval slots filled across tasks in \(\mathcal{D}_{\text{query}}\): \[\mathrm{density} = \frac{1}{|\mathcal{D}_{\text{query}}|} \sum_{t \in \mathcal{D}_{\text{query}}} \frac{|\mathrm{ret}(\mathcal{B}, t)|}{k_{\text{top}}}.\]
The usage term measures the fraction of skills retrieved at least once: \[\mathrm{usage} = \frac{ |\{s \in \mathcal{B} : s \text{ retrieved at least once}\}| }{ |\mathcal{B}| }.\]
Together, the two terms encourage banks that not only retrieve skills frequently across tasks, but also utilize a broad portion of the bank rather than relying on a small subset of dominant skills.
3.3 Pareto-Aware Skill Bank Curation↩︎
Directly solving Eq. 1 is intractable for two reasons: \(\mathcal{S}\) itself is open-ended, with new skills synthesized from trajectory evidence rather than drawn from a fixed pool; and all objectives are only observable through expensive agent rollouts. We therefore propose a bi-level propose-then-verify framework that iteratively refines the skill bank through candidate generation and Pareto-aware selection.
3.3.0.1 Bi-level propose-then-verify loop.
As shown in Figure 2, at iteration \(i\), given the current bank \(\mathcal{B}^{(i)}\), the framework alternates between an inner loop that proposes candidate edits and an outer loop that verifies candidate banks: \[\label{eq:bilevel} \begin{gather} \underbrace{ \mathcal{K}^{(i)} = \mathrm{Plan}\!\big( \texttt{Distill}, \texttt{Diagnose}; \mathcal{B}^{(i)}, \mathcal{D}_{\text{support}} \big) }_{\boldsymbol{inner loop: candidate generation}} \\[4pt] \underbrace{ \mathcal{B}^{(i+1)} = \mathrm{Select}\!\big( \mathcal{K}^{(i)}; \eta, \mathcal{D}_{\text{query}} \big) }_{\boldsymbol{outer loop: Pareto-aware selection}}. \end{gather}\tag{2}\]
The support/query split introduced in Section 3.1 aligns naturally with this bi-level structure: \(\mathcal{D}_{\text{support}}\) supplies trajectory evidence to the inner loop, while \(\mathcal{D}_{\text{query}}\) is reserved for the outer loop’s Pareto-aware selection.
3.3.0.2 Cold-start initialization.
As illustrated in the cold-start stage of Figure 2, we first construct an initial bank \(\mathcal{B}^{(0)}\) from trajectories collected without retrieval. Following [4], successful and failed trajectories are passed to a Skill Distiller, which synthesizes an initial set of Add-type skills from failure patterns in conjunction with successful trajectories.
3.3.0.3 Inner loop: candidate generation.
As illustrated in the inner loop of Figure 2 and summarized in Algorithm 4, at each iteration the inner loop transforms \(\mathcal{B}^{(i)}\) into \(K\) candidate banks through three skill curation operations: Add, Rewrite, and Remove.
Rolling out \(\pi_{\text{agent}}\) on \(\mathcal{D}_{\text{support}}\) with retrieval from \(\mathcal{B}^{(i)}\) yields four trajectory quadrants defined by outcome (success/failure) and retrieval (empty/non-empty): \[\label{eq:quadrants} \begin{align} \mathcal{T}^{+}_{\bar{R}} &= \{\tau_t : \mathrm{succ}(\tau_t),\; R_t = \emptyset\}, \\ \mathcal{T}^{-}_{\bar{R}} &= \{\tau_t : \mathrm{fail}(\tau_t),\; R_t = \emptyset\}, \\ \mathcal{T}^{+}_{R} &= \{\tau_t : \mathrm{succ}(\tau_t),\; R_t \neq \emptyset\}, \\ \mathcal{T}^{-}_{R} &= \{\tau_t : \mathrm{fail}(\tau_t),\; R_t \neq \emptyset\}. \end{align}\tag{3}\]
The no-retrieval quadrants \(\mathcal{T}^{\pm}_{\bar{R}}\) expose missing capabilities in the bank and are consumed by the Skill Distiller, which synthesizes candidate Add skills by clustering failure patterns in \(\mathcal{T}^{-}_{\bar{R}}\) and using \(\mathcal{T}^{+}_{\bar{R}}\) as positive references.
The retrieval quadrants \(\mathcal{T}^{\pm}_{R}\) provide counterfactual evidence on existing skills and are consumed by a Skill Diagnoser. For each retrieved skill \(s \in R_t\), we perform a leave-one-out replay \[\tilde{\tau}_{t \setminus s} = \pi_{\text{agent}} \!\left( t \mid R_t \setminus\{s\} \right),\] which isolates the marginal contribution of \(s\). Based on the factual outcome of \(\tau_t\) and the counterfactual outcome of \(\tilde{\tau}_{t \setminus s}\) aggregated over all trajectories retrieving \(s\), the Skill Diagnoser assigns one of three verdicts: Keep, Rewrite, or Remove.
Finally, an Edit Planner composes the three edit pools \((\mathcal{E}_{\text{add}}, \mathcal{E}_{\text{rewrite}}, \mathcal{E}_{\text{remove}})\) into \(K\) candidate banks \(\mathcal{K}^{(i)} = \{\mathcal{B}^{(i+1)}_k\}_{k=1}^{K}\), while preserving the Keep skills as protected slots.
The full prompts used to implement Skill Distiller, Skill Diagnoser, and Edit Planner are listed in Appendix 8.
3.3.0.4 Outer loop: Pareto-aware selection.
Given the candidate set \(\mathcal{K}^{(i)}\), the outer loop evaluates each candidate by its objective profile \[\Phi(\mathcal{B}) = \big( J_{\text{util}}(\mathcal{B}), J_{\text{div}}(\mathcal{B}), J_{\text{cov}}(\mathcal{B}) \big).\] We also include a null candidate, which represents keeping the current bank unchanged. This allows the selector to make no edit when none of the proposed candidates improves the bank. Over \(\mathcal{K}^{(i)} \cup \{\text{null}\}\), we first compute the non-dominated Pareto front [15] under the three objectives, then resolve the front lexicographically with \(J_{\text{util}}\) as the highest priority.
Since the absolute utility threshold \(\eta\) in Eq. 1 cannot be specified a priori over an open-ended skill space, we replace it with a front-adaptive threshold \(u_{\max}-\epsilon\). Let \(u_{\max}\) denote the largest \(J_{\text{util}}\) on the front, and let \[\mathrm{Tied} = \{\mathcal{B} \in \mathrm{front} : J_{\text{util}}(\mathcal{B}) \ge u_{\max} - \epsilon\}\] denote the candidates within tolerance \(\epsilon\) of the lex-optimal utility. If \(|\mathrm{Tied}| = 1\), the unique tied candidate wins; otherwise we break the tie inside \(\mathrm{Tied}\) by hypervolume contribution over the two regularizers \(J_{\text{div}}\) and \(J_{\text{cov}}\). This yields a per-round non-degradation guarantee, \(J_{\text{util}}(\mathcal{B}^{(i+1)}) \ge J_{\text{util}}(\mathcal{B}^{(i)}) - \epsilon\), by construction.
If the null candidate wins the tie-break, the bank carries forward unchanged into the next round; the loop terminates when the maximum number of rounds \(T\) is reached. We refer the reader to Algorithm 5 for the full procedure.
4 Experiments↩︎
| ALFWorld (Success %) | WebShop | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 2-8 (lr)9-10 Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. | Score | Succ. |
| Zero-Shot | 33.4 | 21.6 | 19.3 | 6.90 | 2.80 | 3.20 | 14.8 | 26.4 | 7.80 |
| ReAct [1] | 48.5 | 35.4 | 34.3 | 13.2 | 18.2 | 17.6 | 31.2 | 46.2 | 19.5 |
| Reflexion [2] | 62.0 | 41.6 | 44.9 | 30.9 | 36.3 | 23.8 | 42.7 | 58.1 | 28.8 |
| Mem0 [11] | 54.0 | 55.0 | 26.9 | 36.4 | 20.8 | 7.69 | 33.6 | 23.9 | 2.00 |
| MemP [7] | 54.3 | 38.5 | 48.1 | 56.2 | 32.0 | 16.7 | 41.4 | 25.3 | 6.40 |
| SimpleMem [16] | 64.5 | 33.3 | 20.0 | 12.5 | 33.3 | 3.84 | 29.7 | 33.2 | 8.59 |
| Voyager [3] | 62.5 | 61.1 | 51.6 | 52.2 | 28.6 | 17.6 | 47.0 | 50.0 | 26.4 |
| ExpeL [4] | 21.0 | 67.0 | 55.0 | 52.0 | 71.0 | 6.00 | 46.3 | 30.9 | 11.2 |
| EvoSkill [9] | 70.8 | 27.8 | 41.9 | 39.1 | 19.0 | 11.8 | 37.3 | 42.2 | 12.6 |
| Skill-Pro [10] | 66.7 | 66.7 | 45.2 | 43.5 | 47.6 | 23.5 | 49.3 | 38.7 | 23.2 |
| 70.8 | 72.2 | 58.1 | 60.9 | 57.1 | 29.4 | 59.0 | 59.3 | 38.4 | |
4pt
| ALFWorld (Success %) | WebShop | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 3-9 (lr)10-11 Method | Worker | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. | Score | Succ. |
| ReAct | Qwen2.5-7B-Instruct | 48.5 | 35.4 | 34.3 | 13.2 | 18.2 | 17.6 | 31.2 | 46.2 | 19.5 |
| Qwen3-4B-Instruct-2507 | 70.8 | 55.6 | 19.4 | 39.1 | 23.8 | 29.4 | 38.8 | 49.3 | 16.8 | |
| Qwen3-30B-A3B-Instruct-2507 | 83.3 | 61.1 | 41.9 | 52.2 | 33.3 | 64.7 | 55.2 | 43.8 | 14.0 | |
| GPT-4o | 81.2 | 63.1 | 23.2 | 56.1 | 20.2 | 41.0 | 46.4 | 33.3 | 27.2 | |
| Gemini-2.5-Pro | 85.2 | 53.1 | 67.3 | 68.2 | 29.1 | 59.4 | 61.8 | 52.3 | 38.1 | |
| Qwen2.5-7B-Instruct | 70.8 | 72.2 | 58.1 | 60.9 | 57.1 | 29.4 | 59.0 | 59.3 | 38.4 | |
| Qwen3-4B-Instruct-2507 | 79.2 | 61.1 | 54.8 | 56.5 | 66.7 | 41.2 | 60.4 | 51.0 | 36.6 | |
| Qwen3-30B-A3B-Instruct-2507 | 83.3 | 77.8 | 80.6 | 82.6 | 61.9 | 58.8 | 75.4 | 58.4 | 56.2 | |
| GPT-4o | 91.7 | 88.9 | 87.1 | 91.3 | 81.0 | 88.2 | 88.1 | 75.0 | 65.0 | |
| Gemini-2.5-Pro | 95.8 | 94.4 | 90.3 | 87.0 | 76.2 | 94.1 | 89.6 | 82.4 | 72.0 | |
4pt
4.1 Experimental Setup↩︎
4.1.0.1 Environments.
We evaluate on two agent benchmarks. ALFWorld [17] is a textualized embodied benchmark spanning six household task families that require compositional action sequences; we report per-task and overall success rate (%) on its test split. WebShop [18] is a simulated online shopping benchmark requiring multi-step navigation, option customization, and purchase decisions. We report two metrics on its test split: Score, the average normalized reward measuring attribute matching between the purchased and target product, and Succ., the fraction of episodes where every required attribute is matched.
4.1.0.2 Baselines.
We compare SkillBrew against ten baselines in Table 1, all using Qwen2.5-7B-Instruct as the frozen agent worker: (1) vanilla baselines (Zero-Shot and ReAct [1]); (2) memory-based agents (Reflexion [2], Mem0 [11], MemP [7], and SimpleMem [16]); and (3) skill-based methods (Voyager [3], ExpeL [4], EvoSkill [9], and Skill-Pro [10]). These training-free baselines are evaluated under frozen-worker setting. We do not compare with training-based methods such as SkillRL, which jointly updates the worker policy and the bank, since this changes the underlying agent and is not applicable to closed-source workers. We leave joint agent-bank optimization to future work.
4.1.0.3 Implementation details.
We use GPT-5.4 [19] for the Skill Distiller, Skill Diagnoser, and Edit Planner. For ALFWorld, we sample 200
train episodes as \(\mathcal{D}_{\text{support}}\), and use the full valid_seen split as \(\mathcal{D}_{\text{query}}\), and report results on the full
valid_unseen split as the test set. For WebShop we follow its standard test split with 500 episodes. Each skill in our bank is structured with: a concise title, a principle describing the strategy, and when_to_apply conditions specifying
applicability. The agent worker adopts ReAct-style prompting [1] and retrieves the top-3 skills from the bank via a hybrid scorer that combines
BM25 [20] with dense cosine similarity over the title, principle, and when_to_apply fields. We run the propose-verify
loop for a fixed budget of 10 rounds. At each round, the Edit Planner produces \(K\) candidate banks, and the Pareto-aware selector uses a utility tolerance \(\epsilon=0.03\) and
breaks ties by hypervolume contribution over diversity and coverage. To reduce the cost of repeated counterfactual leave-one-out rollouts required by both Skill Diagnoser and the \(J_{\text{util}}\) computation, we
employ a content-addressed counterfactual replay cache that is shared across stages and rounds; see Appendix 6.5 for details. All experiments run on a single 8\(\times\)A800 node.
4.2 Main Results↩︎
4.2.0.1 Comparison with baselines.
Table 1 compares SkillBrew against ten training-free baselines on ALFWorld and WebShop, all running on the same frozen Qwen2.5-7B-Instruct worker. SkillBrew achieves the highest overall average on ALFWorld at 59.0%, surpassing the strongest skill-based method Skill-Pro by 9.7% and ReAct by 27.8%. Notably, it also outperforms the append-only Voyager by 12.0%. Relative to ReAct, SkillBrew improves on all six task types, with the largest gains of 47.7% on Heat and 38.9% on Cool. On WebShop, SkillBrew achieves the best results on both metrics, with a Score of 59.3% and a Succ.of 38.4%. The substantial gain on Succ.indicates that the curated skills translate into precise purchases matching every required attribute, rather than near matches that earn only partial reward.
4.2.0.2 Cross-worker generalization.
Table 2 extends the evaluation to different worker backbones, including three Qwen open-source models [21] and two closed-source models, GPT-4o [22] and Gemini-2.5-Pro [23]. SkillBrew improves over the ReAct baseline on every backbone, with ALFWorld Avg.gains ranging from 20.2% on Qwen3-30B-A3B-Instruct-2507 to 41.7% on GPT-4o. The improvements are particularly pronounced on closed-source workers: on GPT-4o, ALFWorld Avg.rises from 46.4 to 88.1 and WebShop Succ.from 27.2 to 65.0, a gain of 37.8%. Even after curation, the Qwen3-4B-Instruct-2507 worker reaches 60.4%, approaching the 61.8% performance of Gemini-2.5-Pro without curation, suggesting that a well-curated skill bank can partially compensate for model size.
4.3 Analysis↩︎
4.3.1 Curation Loop Dynamics↩︎
Figure 3 traces the dynamics of skill bank over the curation loop across five worker backbones on ALFWorld. The bank size contracts from larger cold-start values to compact banks within 9–10 rounds and stabilizes thereafter, indicating that the curation process converges across all workers rather than growing unboundedly. The test success rate rises throughout the loop on every worker, with small round-to-round fluctuations but clear overall gains. Even at round 1, where the bank only uses the cold-start output of the Skill Distiller, the test success rate is already substantially above the ReAct baseline of each worker, confirming that the cold-start design provides a strong starting point that the subsequent curation loop further improves upon. Representative skills from the final curated bank for each benchmark are shown in Appendix 7.
4.3.2 Ablation Study↩︎
| Agent Worker | |||
|---|---|---|---|
| 2-4 Bank Source | |||
| Instruct-2507 | |||
| Instruct-2507 | GPT-4o | ||
| no bank (ReAct) | 38.8 | 55.2 | 46.4 |
| Instruct-2507 | 60.4 | 71.6 | 78.4 |
| Instruct-2507 | 52.2 | 75.4 | 82.8 |
| GPT-4o | 53.7 | 73.9 | 88.1 |
3pt
4.3.2.1 Bank transferability.
Table 3 examines whether a curated bank for one worker remains effective when applied to a different worker at inference time. We curate three skill banks with SkillBrew, each under a different source worker, and apply each bank to all three inference workers. Every cell substantially exceeds the ReAct baseline without retrieval. Diagonal cells, where each worker retrieves from its own bank, yield the best result for that worker at 60.4%, 75.4%, and 88.1%, indicating that curation specializes the bank to its source worker. Cells away from the diagonal remain close behind in both directions: a bank curated by the smallest Qwen3-4B-Instruct-2507 worker lifts GPT-4o from 46.4% to 78.4%, only 9.7% below GPT-4o’s own result; conversely, a bank curated by GPT-4o lifts Qwen3-4B-Instruct-2507 worker from 38.8% to 53.7%, only 6.7% below its own result. This suggests that the curated skills encode transferable procedural knowledge rather than artifacts of a particular worker, so a skill bank can be reused across workers in either direction with little loss.
| ALFWorld | WebShop | ||
|---|---|---|---|
| 2-2 (lr)3-4 Objectives | Avg. | Score | Succ. |
| \(J_{\text{util}}\) | 45.8 | 48.2 | 28.4 |
| \(J_{\text{util}} + J_{\text{div}}\) | 51.4 | 47.6 | 36.5 |
| \(J_{\text{util}} + J_{\text{cov}}\) | 52.6 | 51.0 | 37.6 |
| \(J_{\text{util}} + J_{\text{div}} + J_{\text{cov}}\) | 59.0 | 59.3 | 38.4 |
5pt
4.3.2.2 Effect of objective.
Table 4 ablates the contribution of each objective by varying which terms enter the Pareto-aware selector while keeping curation pipeline fixed. Using \(J_{\text{util}}\) alone yields a success rate of 45.8% on ALFWorld and 28.4% on WebShop, trailing the full objective by 13.2 and 10.0% respectively. Compared to using \(J_{\text{util}}\) alone, adding \(J_{\text{div}}\) or adding \(J_{\text{cov}}\) improves the success rate. The two objectives are complementary: \(J_{\text{div}}\) promotes diversity and prevents redundancy in skill bank, while \(J_{\text{cov}}\) penalizes banks that bloat with dormant or rarely retrieved skills. Combining both reaches the best results on both benchmarks.
| Edit Operation | ALFWorld | WebShop | |||
|---|---|---|---|---|---|
| 1-3 (lr)4-4 (lr)5-6 Add | Rewrite | Remove | Avg. | Score | Succ. |
| ✔ | 47.0 | 50.0 | 26.4 | ||
| ✔ | ✔ | 53.5 | 51.0 | 37.6 | |
| ✔ | ✔ | 48.3 | 49.5 | 34.5 | |
| ✔ | ✔ | ✔ | 59.0 | 59.3 | 38.4 |
4pt
4.3.2.3 Effect of edit operations.
Table 5 ablates the three edit operations available to the Edit Planner. With Add alone, the bank grows monotonically without ever pruning or refining, leaving dormant or harmful skills to crowd out retrieval slots; the success rate reaches only 47.0% on ALFWorld and 26.4% on WebShop. The Add+Rewrite setting lifts ALFWorld to 53.5% because the majority of skills the Diagnoser flags are salvageable, and refinement recovers their value rather than discarding it. Add+Remove setting lifts ALFWorld only to 48.3%, lower than Add+Rewrite because the operation is too coarse: every flagged skill must be discarded directly, including partially correct ones whose strategy could have been salvaged and refined by a small edit. Combining all three reaches 59.0% on ALFWorld and 38.4% on WebShop, indicating that the two operations play complementary roles: Rewrite is the default rescue path that handles the bulk of flagged skills, while Remove is reserved for the minority whose underlying strategy cannot be repaired by any edit.
5 Conclusion↩︎
We present SkillBrew, a training-free framework that curates skill banks for LLM agents through bank-level optimization. Instead of judging each skill independently or continuously adding new skills, SkillBrew treats the entire skill bank as the optimization object and balances utility, diversity, and coverage under a Pareto-aware selection process. Experiments on ALFWorld and WebShop show that this bank-level curation leads to more diverse, transferable, and effective skill repositories, improving agent performance across both open-source and closed-source workers. These results suggest that future self-improving agents should optimize skill banks as global knowledge systems, rather than manage them as isolated skills.
Limitations↩︎
One limitation of our framework is that the agent remains frozen during skill bank optimization, leaving joint co-adaptation between the agent and the evolving bank unexplored. Future work could investigate recursive co-training or reinforcement learning frameworks that jointly optimize both components. In addition, although our content-addressed cache substantially reduces the cost of counterfactual rollouts, the method still relies on repeated rollouts and leave-one-out replay, which may become expensive at larger scales; a more efficient method could further improve scalability. Finally, the current formulation assumes relatively static task distributions, and extending bank curation to open-world and continually shifting environments remains an important direction. Nevertheless, we believe this work introduces a new perspective: skill banks should be treated not as append-only memory repositories, but as globally optimized and continually refined objects, opening new directions for scalable and lifelong agent systems.
6 Implementation Details↩︎
6.1 Datasets↩︎
6.1.0.1 ALFWorld.
ALFWorld [17] is a text-based embodied benchmark that spans six household task families. We randomly sample 200 episodes from
the official train split as the support set \(\mathcal{D}_{\text{support}}\), and use the full valid_seen split as the query set \(\mathcal{D}_{\text{query}}\) and
the full valid_unseen split as the test set. Sampling uses random seed 42.
6.1.0.2 WebShop.
WebShop [18] is a simulated online shopping environment with a fixed product catalog and one-million-scale natural language goals; the standard test split contains 500 goals, which we use as the test set. We additionally sample 500 and 250 goals from WebShop training and evaluation pool (goal ids \(\geq 500\)) as \(\mathcal{D}_{\text{support}}\) and \(\mathcal{D}_{\text{query}}\) respectively, with seed 42. The support, query, and test sets are pairwise disjoint.
6.2 LLM Settings↩︎
The agent worker is prompted in ReAct style [1]. On ALFWorld, the worker is given a budget of 50 steps per episode and conditions on the full history at every generation step. On WebShop, the worker is given 15 steps per episode and conditions on the most recent one step as history. The maximum generation length per step is 2,048 tokens for all agent workers. For Skill Distiller, Skill Diagnoser, and Edit Planner, we all use GPT-5.4 [19]. Their prompts are shown in Appendix 8.
6.3 Skill Retrieval↩︎
The agent retrieves the top-3 most relevant skills from skill bank using a hybrid scorer that combines BM25 [20]
with dense cosine similarity. Concretely, the score of a skill \(s\) relative to a query \(q\) is \[\text{score}(q, s) = 0.30 \cdot \overline{\text{BM25}}(q, s) +
0.70 \cdot \cos(\mathbf{e}_q, \mathbf{e}_s),\] where \(\overline{\text{BM25}}\) denotes min-max normalized BM25 scores. The skill text used for retrieval is the combination of three fields: title,
principle, and when_to_apply. Skills with a combined score below 0.30 are dropped from the retrieval result. We use text-embedding-3-small for generating skill embeddings.
6.4 Curation Loop and Pareto-Aware Selector↩︎
We run the propose-then-verify loop for a fixed budget of 10 rounds. Within each round, the Skill Distiller, Skill Diagnoser, and Edit Planner are invoked in this order, and each candidate bank proposed by the Edit Planner is evaluated by rolling out the worker on \(\mathcal{D}_{\text{query}}\) to estimate \(J_{\text{util}}\), \(J_{\text{div}}\), and \(J_{\text{cov}}\). The Pareto-aware selector uses a hypervolume contribution criterion with lex priority on \(J_{\text{util}}\): candidates whose \(J_{\text{util}}\) falls within a tolerance of \(\epsilon = 0.03\) of the highest feasible \(J_{\text{util}}\) are admitted to the tie pool, and ties are broken by hypervolume contribution over \(J_{\text{div}}\) and \(J_{\text{cov}}\). The previous bank is augmented to the candidate set as a null candidate. If it wins the lex-Pareto tie-break by hypervolume contribution, the bank carries forward unchanged to the next round.
6.5 Counterfactual Replay Cache↩︎
The propose-then-verify loop requires counterfactual leave-one-out rollouts in two places: on support set \(\mathcal{D}_{\text{support}}\), where the Skill Diagnoser compares with and without each retrieved skill to issue Keep / Rewrite / Remove verdicts, and on query set \(\mathcal{D}_{\text{query}}\), where each candidate bank’s \(J_{\text{util}}\) is computed by the same leave-one-out procedure.
To improve efficiency across both stages, we maintain a single content-addressed cache shared by all rollouts. Each entry is keyed by a SHA-256 hash of the task id and the canonical content (title, principle,
when_to_apply) of every retrieved skill, ordered by retrieval rank, so any change visible to the worker is also visible to the key. Entries also carry a version tag derived from the worker configuration; whenever the worker model or its prompt
template changes, all cached entries are silently invalidated, preventing stale rollouts from polluting either the Diagnoser verdicts or the \(J_{\text{util}}\) signal.
Because the \(K\) candidate banks proposed by the Edit Planner within a round differ by only a few edits, and the winning bank further carries forward to the next round, most leave-one-out retrievals coincide across candidates and across rounds. We observe a cache hit rate above 60% from the second round onwards, reducing the cost of counterfactual rollouts.
7 Skill Examples↩︎
Figure 6 and Figure 7 present representative skills from the final curated bank on ALFWorld and WebShop, respectively; Figure 8 additionally traces a single skill before and after a Rewrite verdict from the Skill Diagnoser.
8 Prompts↩︎
Figures 9 and 10 show the two-stage prompt used by the Skill Distiller: the first analyzes failure trajectories to identify missing capabilities, while the second synthesizes candidate Add skills targeting those gaps. Figure 11 shows the prompt of the Skill Diagnoser, which assigns Keep, Rewrite, or Remove verdicts to each retrieved skill from its factual and counterfactual outcomes. Figure 12 shows the prompt of the Edit Planner. Rather than mechanically applying each edit verdict, the planner conditions on the full current bank together with all candidate edit pools, reasoning at the bank level about which combinations of edits compose into \(K\) candidate banks.
