More Skills, Worse Agents? Skill Shadowing
Degrades Performance When Expanding Skill Libraries
May 21, 2026
Abstract
Skill libraries allow LLM agents to load task-specific instructions on demand, letting non-expert users solve domain-specific tasks through natural language without knowing which skills exist or how they work. However, performance degrades as libraries grow—by up to 21% when scaling from a small set of helpful skills to a 202-skill library. In this work, we formulate this performance degradation as the pass rate drop between loading a library of known-helpful skills and the full library. Moreover, we propose to decompose the pass rate drop by conditioning on the skill(s) invocation—which skills the agent selects during a trajectory—into two effects: skill shadowing, where the agent selects wrong skills more often as the library expands, and context overhead, where the enlarged context degrades execution even when selection is correct. We derive upper bounds on both effects to characterize their magnitudes of impacts to the pass rate drop. Our empirical estimates of the effects and their upper bounds both show that the skill shadowing effect grows with library size and significantly contributes to the performance degradation, whereas the context overhead effect remains small and indistinguishable from zero. This observed asymmetry establishes that the skill selection failure, not the enlarged context, is the primary bottleneck when expanding the skill libraries.
Keywords: Agent skills, Agent degradation, Skill libraries, Skill selection, Skill shadowing, Context overhead.
1 ↩︎
1.0.1 Introduction↩︎
LLM agents are increasingly extended not only by adding tool calls but also by loading skills—self-contained packages of instructions, scripts, and reference documents that inject domain expertise into the agent’s context [1]. The abstraction has rapidly become an industry standard: Anthropic shipped Agent Skills for Claude and released the specification as an open format, with
partner-built skills from Atlassian, Stripe, and Figma entering a curated directory [2]; OpenAI adopted the same
SKILL.md convention for Codex shortly after [3]. The appeal is compositional: practitioners assemble dozens of
skills into a single agent, and foundational systems demonstrated that capability libraries can be composed and selected at scale through language-based interfaces—Voyager [4] incrementally builds a skill library retrieved by embedding similarity, while HuggingGPT [5]
selects among thousands of expert models using their natural-language descriptions. The premise is that a user can describe a task in plain language (“check why this customer’s cluster is slow”) and the agent autonomously selects the right skill, without
the user knowing which skills exist or how they work. In our experiments, a 202-skill library cuts this premise short: pass rate drops by 21%, averaged across both models.
Yet most prior work evaluates skills one at a time: how to author a skill, retrieve one, or measure its marginal contribution over a no-skill baseline [6]–[8]. Evidence from the more mature tool-use literature suggests this single-skill view is dangerously incomplete. Tool selection accuracy drops from above 90% with fewer than 30 candidates to 13.6% with 11,100 [9], and in-context selection “could completely fail” with numerous options [8]. Skills amplify these failure modes: they rely on natural-language descriptions with broader, more overlapping scope than typed tool schemas, and a wrong skill injects plausible but incorrect reasoning into the agent’s context where it is treated as authoritative [1]—whereas a wrong tool call is more likely to produce a detectable error (type mismatch, missing parameter, API error code) due to its structured schema.
Motivations. Despite growing awareness—recent work has documented degradation under progressively realistic skill settings [10] and shown that compressing skill content can yield a “less-is-more” effect [11]. As shown in Figure 1, we evaluate on SkillsBench [12] and observe monotonic pass rate drop when we load progressively larger skill libraries compared to only loading known-helpful skills (i.e., oracle skills). Our leading hypothesis is that certain skills in the library shadow the oracle skill. For example:
In the mario-coin-counting task, the library includes a skill named video-frame-extraction, whose description superficially matches the query better than the provided helpful skill. As a result, the agent invokes this wrong one
in ALL \(26\) trajectories.
However, no prior work has formalized the library-induced degradation, nor has tried to decompose it into separable causal mechanisms through controlled experimentation. The challenge is that an agent trajectory—from receiving the task query, through skill selection and invocation, to producing a final output—is largely a black box: we observe the task outcome but not the intermediate decisions that produced it. Without observing full intermediate states, we cannot attribute degradation to specific failure mechanisms.
Contributions. We propose to use skill invocations as one such observable intermediate state. We formalize the trajectory structure to define invocation patterns, which enables an exact decomposition of library-induced degradation into two separable effects—skill shadowing and context overhead—with upper bounds derived on each individual effect (§1.0.3). Our experiments show that skill shadowing dominates, accounting for up to 68% of degradation and serving as the only statistically significant effect, while context overhead yields consistently reasonable but insignificant point estimates, remaining indistinguishable from noise (§1.0.4).
1.0.2 Formulation↩︎
We study skill-augmented agents—LLM agents that select and load task-specific instruction sets from a shared library. Unlike tool-augmented agents [13], [14], where invoking a tool executes code and returns a computed result, invoking a skill injects instructional text into the agent’s context window; the agent then drives task execution itself using base tools, guided by the injected instructions.
1.0.2.1 Skill-Augmented Agents
Definition 1 (Skill). A skill is a triple \(S_i = (n_i, d_i, b_i)\) consisting of a name \(n_i\), a natural-language description \(d_i\), and a body \(b_i\) (the full instructions and examples that guide task execution). The pair \((n_i, d_i)\) is the skill’s interface—the only information visible to the agent at selection time. The body \(b_i\) is withheld until the skill is invoked.
Definition 2 (Skill library). A skill library is a finite set \(\mathcal{S} = \{S_1, \ldots, S_\ell\}\).
Our formulation adapts the agentic-skill tuple of [15], who define a skill by its applicability condition, policy, termination condition, and callable interface. Context-injection skills delegate execution entirely to the host agent, so the applicability condition, policy, and termination condition reduce to properties of the agent rather than the skill; What remains is the callable interface (our \((n_i, d_i)\)) and the instructional payload (our \(b_i\)).
Definition 3 (Agent). An agent \(\mathcal{A}_\theta\) is a large language model with parameters \(\theta\) that, at each turn, selects an action conditioned on the current observation, prior interaction history, task \(q\), and the skill description \(\{(n_i, d_i)\}_{i=1}^{\ell}\). An action is one of: a skill invocation \(\texttt{Skill}(n_i)\), a base-tool call* (e.g., file read, shell command), or a termination signal.*
Definition 4 (Agent context). The agent context* is the complete input the agent receives before acting: \[\label{eq:context} \mathcal{C}(q, \mathcal{S}) = \bigl(q,\; \mathcal{D}(\mathcal{S})\bigr), \quad \mathcal{D}(\mathcal{S}) = \{(n_i, d_i)\}_{i=1}^{\ell}.\tag{1}\] The agent context is fully determined by the task and library, and is the only input that varies across experiments (§1.0.4).*
This adapts the controller formulation of [13], who decompose the LLM’s decision as \(p_\theta(a_t \mid x_t, H_t, q)\) over perceiver output \(x_t\), history \(H_t\), and query \(q\). In our setting, the perceiver output is replaced by the skill description \(\mathcal{D}(\mathcal{S})\), and the history \(H_t\) is unobservable—we treat the trajectory as a black box from which only the invocation set \(\mathcal{I}\) and the final output are recorded.
1.0.2.2 Formalization of the Trajectory
Definition 5 (Task). A task is a triple \(\mathcal{T} = (q, r, v)\) consisting of a query \(q\) (a natural-language description of the objective), a reference solution \(r\) (a known-correct implementation that demonstrates solvability), and a verifier \(v\) (a deterministic test suite of independently evaluable subtasks).
This follows the task structure of [12], simplified by folding the environment into \(v\). The agent observes only \(q\); both \(r\) and \(v\) are withheld. Since each task is uniquely identified by its query, we index by \(q\) for brevity throughout. Given \(\mathcal{T} = (q, r, v)\) and \(\mathcal{S}\), a trajectory proceeds in three phases:
Phase 1: Context loading. The agent receives \(q\) and \(\mathcal{D}(\mathcal{S})\). Skill bodies are withheld.
Phase 2: Skill invocation to fulfill the task query. The agent acts for a bounded number of turns (or time). At each turn, it emits an action: 2-(1) If the action is \(\texttt{Skill}(n_i)\), the body \(b_i\) is appended to the agent’s context (i.e., skill invocation); 2-(2) If the action is a base-tool call, the system executes it and returns the result.
The agent may invoke multiple skills during a single trajectory. Let \(\mathcal{I} \subseteq \mathcal{S}\) denote the set of distinct skills invoked during the trajectory. The intermediate actions, observations, and internal context accumulated during Phase 2 are difficult to observe. We observe only the agent context \(\mathcal{C}(q, \mathcal{S})\), the invocation set \(\mathcal{I}\), and the final output evaluated by \(v\).
Phase 3: Output. The trajectory terminates when the agent signals completion or reaches the turn (or time) limit. The agent produces a final artifact (e.g., modified files, generated code) and the trajectory ends.
1.0.2.3 Output Verification
The verifier \(v\), external to the agent trajectory, evaluates the final environment state; the agent cannot observe or interact with \(v\). We define the task completion random variable \[\label{eq:task-completion} T_q = \frac{\#\{\text{passed subtasks verified by } v\}}{m} \in \left\{0,\, \tfrac{1}{m},\, \ldots,\, 1\right\}.\tag{2}\] The binary pass rate is the probability that all subtasks pass: \[\label{eq:pass-rate} p(q, \mathcal{A}_\theta, \mathcal{S}) = \Pr\!\left[T_q = 1 \mid q,\, \mathcal{A}_\theta,\, \mathcal{S}\right].\tag{3}\] The fractional pass rate is: \[\label{eq:fractional-pass-rate} \bar{p}(q, \mathcal{A}_\theta, \mathcal{S}) = \mathbb{E}\!\left[T_q \mid q,\, \mathcal{A}_\theta,\, \mathcal{S}\right].\tag{4}\] When \(m = 1\), \(T_q \in \{0, 1\}\) and the two metrics coincide. Since the agent \(\mathcal{A}_\theta\) is fixed throughout our experiments, we write \(p(q, \mathcal{S})\) for brevity.
1.0.3 Library-Induced Degradation: Decomposition and Upper Bounds↩︎
We formalize the two degradation mechanisms studied in this paper. Skill shadowing occurs when distractor skills displace oracle skills during selection, reducing the probability that the agent invokes a correct skill despite one being available. Context overhead occurs when the enlarged agent context degrades execution quality even when the agent selects correctly. We define oracle and distractor skills, classify trajectories by their invocation pattern, and derive an exact decomposition of the pass-rate drop into these two effects.
Definition 6 (Oracle skill set). For a task \(q\) and a skill-delta threshold \(\tau > 0\), the oracle skill set is the subset of skills whose marginal contribution over the no-skill baseline meets or exceeds \(\tau\): \[\label{eq:oracle} \mathcal{S}^\star(q) = \left\{S_i \in \mathcal{S} \;\middle|\; p(q, \{S_i\}) - p(q, \varnothing) \geq \tau\right\}.\qquad{(1)}\] The oracle set is determined empirically: each skill is tested in isolation against \(q\), and only those whose pass-rate improvement over the no-skill baseline meets the threshold are included. The set may contain multiple skills (when several skills independently improve performance on \(q\)) or be empty (when no single skill provides sufficient benefit). The value of \(\tau\) and the empirical oracle-set sizes are reported in §1.0.4.
Definition 7 (Distractor skills). For a task \(q\), all non-oracle skills \(\mathcal{S} \setminus \mathcal{S}^\star(q)\) are distractors. Distractors are not adversarially constructed; they are simply the skills in the library that do not meet the oracle threshold for \(q\).
Definition 8 (Invocation events). For a task \(q\) with \(\mathcal{S}^\star(q) \neq \varnothing\), recall that a trajectory produces an invocation set \(\mathcal{I}\). Every trajectory falls into one of three mutually exclusive events:
No skill invoked (\(\mathcal{I} = \varnothing\)). The agent attempts the task with base tools alone, or fails to act.
Mixed invocation (\(\mathcal{I} \neq \varnothing\) and \(\mathcal{I} \not\subseteq \mathcal{S}^\star(q)\)). The agent invoked at least one skill, but not exclusively oracle skills—either no oracle was invoked at all, or an oracle was invoked alongside distractors.
Oracle-only invocation (\(\varnothing \neq \mathcal{I} \subseteq \mathcal{S}^\star(q)\)). Every skill the agent invoked was an oracle skill.
Library-induced pass-rate drop. To quantify the cost of library expansion, we compare the pass rate when only the oracle skill set is available (selection is trivial) against the pass rate under the full library:
Definition 9 (Library-induced pass-rate drop). For a task \(q\) with \(\mathcal{S}^\star(q) \neq \varnothing\), the library-induced pass-rate drop is: \[\label{eq:delta} \Delta(q, \mathcal{S}) = p\!\left(q,\, \mathcal{S}^\star(q)\right) \;-\; p(q,\, \mathcal{S}).\qquad{(2)}\]
Decomposition by invocation event. For tasks with \(\mathcal{S}^\star(q) \neq \varnothing\), the three invocation events (Definition 8) partition the trajectory space. By the law of total probability: \[\label{eq:three-way} p(q, \mathcal{S}) = \pi_{\mathrm{\small n}}\,\rho_{\mathrm{\small n}} + \pi_{\mathrm{\small m}}\,\rho_{\mathrm{\small m}} + \pi_{\mathrm{\small o}}\,\rho_{\mathrm{\small o}},\tag{5}\] where \(\pi_{\mathrm{\small n}}, \pi_{\mathrm{\small m}}, \pi_{\mathrm{\small o}}\) are the probabilities of events n, m, o given \((q, \mathcal{S})\), and \(\rho_{\mathrm{\small n}}, \rho_{\mathrm{\small m}}, \rho_{\mathrm{\small o}}\) are the corresponding conditional pass rates. The probabilities of the three events sum to one, i.e., \(\pi_{\mathrm{\small n}} + \pi_{\mathrm{\small m}} + \pi_{\mathrm{\small o}} = 1\).
The same decomposition applies when the library contains only oracle skills, i.e., \(\mathcal{S} = \mathcal{S}^\star(q)\). Since no distractors are present, m is impossible, leaving only n and o. We denote the event probabilities and conditional pass rates under \(\mathcal{S} = \mathcal{S}^\star(q)\) with a star: \[\label{eq:oracle-decomp} p(q, \mathcal{S}^\star) = \pi_{\mathrm{\small n}}^\star\,\rho_{\mathrm{\small n}}^\star + \pi_{\mathrm{\small o}}^\star\,\rho_{\mathrm{\small o}}^\star,\tag{6}\] where \(\pi_{\mathrm{\small n}}^\star + \pi_{\mathrm{\small o}}^\star = 1\). Substituting Eqs. 5 and 6 into Eq. ?? : \[\label{eq:delta-decomp} \Delta(q, \mathcal{S}) = \bigl(\pi_{\mathrm{\small n}}^\star\,\rho_{\mathrm{\small n}}^\star + \pi_{\mathrm{\small o}}^\star\,\rho_{\mathrm{\small o}}^\star\bigr) \;-\; \bigl(\pi_{\mathrm{\small n}}\,\rho_{\mathrm{\small n}} + \pi_{\mathrm{\small m}}\,\rho_{\mathrm{\small m}} + \pi_{\mathrm{\small o}}\,\rho_{\mathrm{\small o}}\bigr).\tag{7}\] Adding and subtracting \(\pi_{\mathrm{\small n}}^\star\,\rho_{\mathrm{\small n}} + \pi_{\mathrm{\small o}}^\star\,\rho_{\mathrm{\small o}}\) yields: \[\begin{align} \label{eq:two-effect} \Delta(q, \mathcal{S}) &= \underbrace{ \pi_{\mathrm{\small n}}^\star\bigl(\rho_{\mathrm{\small n}}^\star - \rho_{\mathrm{\small n}}\bigr) + \pi_{\mathrm{\small o}}^\star\bigl(\rho_{\mathrm{\small o}}^\star - \rho_{\mathrm{\small o}}\bigr) }_{\displaystyle \Delta_{\mathrm{ctx}}} \quad+\quad \underbrace{ \bigl(\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}\bigr)\,\rho_{\mathrm{\small n}} + \bigl(\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}}\bigr)\,\rho_{\mathrm{\small o}} - \pi_{\mathrm{\small m}}\,\rho_{\mathrm{\small m}} }_{\displaystyle \Delta_{\mathrm{shd}}}. \end{align}\tag{8}\]
Context overhead effect (\(\Delta_{\mathrm{ctx}}\)). Event probabilities are held at their \(\mathcal{S}^\star\) values; only the conditional pass rates change. A positive \(\rho_{\mathrm{\small o}}^\star - \rho_{\mathrm{\small o}}\) indicates that the full-library context degrades execution even when the agent selects the same skills. The sole difference between conditions is the agent context (Definition 4): \(\mathcal{D}(\mathcal{S}^\star)\) contains \(|\mathcal{S}^\star(q)|\) entries, while \(\mathcal{D}(\mathcal{S})\) contains \(\ell\).
Skill shadowing effect (\(\Delta_{\mathrm{shd}}\)). By analogy with variable shadowing in programming, distractor skills whose descriptions overlap with oracle skills can hide the correct choice, diverting the agent toward wrong or no skills. Conditional pass rates are held at their full-library values; only the event probabilities change. Under the full library, probability mass shifts from o into m and n, reducing performance. Since \(\pi_{\mathrm{\small n}}^\star + \pi_{\mathrm{\small o}}^\star = 1 = \pi_{\mathrm{\small n}} + \pi_{\mathrm{\small m}} + \pi_{\mathrm{\small o}}\), we have \(\pi_{\mathrm{\small m}} = (\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}) + (\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}})\), and the skill shadowing effect simplifies to: \[\label{eq:delta-shd-simplified} \Delta_{\mathrm{shd}} = \bigl(\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}\bigr) \bigl(\rho_{\mathrm{\small n}} - \rho_{\mathrm{\small m}}\bigr) + \bigl(\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}}\bigr) \bigl(\rho_{\mathrm{\small o}} - \rho_{\mathrm{\small m}}\bigr).\tag{9}\]
Assumption 1. For each invocation event \(E \in \{\mathrm{\small n}, \mathrm{\small o}\}\), the conditional pass rate under \(\mathcal{S} = \mathcal{S}^\star(q)\) is at least as high as under the full library: \[\rho_E^\star \geq \rho_E.\]
Lemma 1 (Upper bound on context overhead). Under Assumption 1, \(\Delta_{\mathrm{ctx}} \geq 0\) and \[\label{eq:ctx-bound} \Delta_{\mathrm{ctx}} \leq \max\!\bigl(\rho_{\mathrm{\small n}}^\star - \rho_{\mathrm{\small n}},\; \rho_{\mathrm{\small o}}^\star - \rho_{\mathrm{\small o}}\bigr).\qquad{(3)}\]
Proof. Both terms in \(\Delta_{\mathrm{ctx}}\) are non-negative by Assumption 1. Since \(\pi_{\mathrm{\small n}}^\star + \pi_{\mathrm{\small o}}^\star = 1\), \(\Delta_{\mathrm{ctx}}\) is a convex combination, bounded by the maximum of its components. ◻
Lemma 2 (Upper bound on skill shadowing). The skill shadowing effect satisfies \[\begin{align} \label{eq:shd-bound} \bigl|\Delta_{\mathrm{shd}}\bigr| &\leq \bigl(|\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}| + |\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}}|\bigr) \;\cdot\; \max\!\bigl(|\rho_{\mathrm{\small n}} - \rho_{\mathrm{\small m}}|,\; |\rho_{\mathrm{\small o}} - \rho_{\mathrm{\small m}}|\bigr) \notag\\ &\leq |\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}| + |\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}}|. \end{align}\qquad{(4)}\]
Proof. Apply the triangle inequality to Eq. 9 and factor out the maximum absolute conditional pass-rate difference. The second inequality follows because \(|\rho_E - \rho_{\mathrm{\small m}}| \leq 1\). ◻
Interpretation. The two bounds serve as structural diagnostics: if the full-library context barely changes execution quality within each invocation event, the bound—and therefore \(\Delta_{\mathrm{ctx}}\) itself—must be small, regardless of sample size. A small bound thus disambiguates “undetectable due to noise” from “genuinely small.” The shadowing bound factors into two interpretable components: selection instability (how much the invocation-event distribution shifts under the full library) and selection stakes (how much pass rate differs across events). Shadowing can be large only when both factors are large: the agent’s selection must shift and the shift must move mass toward lower-performing events. We evaluate these bounds numerically in Appendix 2.0.2.2.
1.0.4 Experiments↩︎
We design controlled experiments to answer the remaining question: what drives the library-induced degradation?
Dataset. We evaluate on SkillsBench [12], a benchmark of \(88\) tasks spanning document processing, finance, scientific computing, data analysis, etc. Each task has a corresponding skill that could be helpful. We include only tasks whose corresponding skill yields at least a \(4\%\) pass-rate uplift over the no-skill baseline, giving us \(38\) (task, model) pairs with oracle skills (\(21\) tasks from Haiku 4.5, and \(17\) from Sonnet 4.6). For each pair we evaluate under a baseline where the library contains only oracle skills (\(\mathcal{S}{=}\mathcal{S}^\star\)) and three expanded libraries of total size \(|\mathcal{S}| \in \{52, 102, 202\}\). The final dataset comprises \(2{,}545\) trajectories. Further details, e.g., task selection, full trajectory counts, library construction, and bootstrap, are deferred to Appendix 2.0.1.
Decomposition and scaling of the pass-rate drop. For each library size we classify trajectories into the three invocation events (Definition 8) and report \(\Delta_{\mathrm{ctx}}\), \(\Delta_{\mathrm{shd}}\), and \(\Delta\) (Eq. 8 ) in Table 1: Both \(\Delta_{\mathrm{ctx}}\) and \(\Delta_{\mathrm{shd}}\) grow with library size, with \(\Delta_{\mathrm{shd}}\) accounting for an increasing share of \(\Delta\). The behavioral shifts underlying this scaling are visible in Table 2, which reports the fraction of trajectories in each of four invocation patterns—“oracle only” (event o), “mixed – oracle invoked” and “mixed – oracle not invoked” (sub-cases of event m), and “no skill invoked” (event n). As the library grows, the event o share collapses, with mass migrating into events n and m. Two complementary failure modes are visible: abandonment (mass into event n) and wrong selection (mass into event m). Additional shadowing examples appear in Appendix 2.0.1.4 and empirical studies of invocation pattern shift can be found in Appendix 2.0.2.4.
| \(|\mathcal{S}|\) | \(\Delta_{\mathrm{ctx}}\) | \(\Delta_{\mathrm{shd}}\) | \(\Delta\) |
|---|---|---|---|
| \(52\) | \(.06\;[-.11, .22]\) | \(.03\;[-.01, .07]\) | \(\mathbf{.08\;[.02, .15]}\) |
| \(102\) | \(.06\;[-.11, .23]\) | \(\mathbf{.08\;[.02, .15]}\) | \(\mathbf{.14\;[.09, .19]}\) |
| \(202\) | \(.07\;[-.13, .25]\) | \(\mathbf{.14\;[.06, .26]}\) | \(\mathbf{.21\;[.15, .27]}\) |
10pt
Which effect dominates the drop? \(\Delta_{\mathrm{shd}}\) is the only component whose CI excludes zero and accounts for the majority of \(\Delta\) at the largest library. \(\Delta_{\mathrm{ctx}}\) does not separate from zero at any library size, although its point estimate is consistently positive. Within event o—where the agent selected correctly—the binary pass rate still falls as the library grows (see Tables ¿tbl:tab:rq4-2-binary? and ¿tbl:tab:rq4-2-fraction? in Appendix 2.0.2.3), suggesting context overhead is real but our sample size is too small to confirm it. Based on the empirical observation, we conclude that shadowing is the dominant effect of the drop, and the context-overhead point estimates as suggestive only. The theoretical bounds of Lemmas 1 and 2 corroborate this conclusion: the context overhead bound remains small and indistinguishable from zero, while the shadowing bound grows sharply and is significantly positive throughout; see Appendix 2.0.2.2 for complete details.
| only | |||||
| oracle invoked | |||||
| oracle not invoked | |||||
| invoked | \(n\) | ||||
| \(\mathcal{S} = \mathcal{S}^\star\) | \(88.0\%\) | \(0.0\%\) | \(0.0\%\) | \(12.0\%\) | \(552\) |
| \(|\mathcal{S}| = 52\) | \(80.3\%\) | \(0.9\%\) | \(2.8\%\) | \(16.1\%\) | \(542\) |
| \(|\mathcal{S}| = 102\) | \(64.4\%\) | \(1.4\%\) | \(5.9\%\) | \(28.3\%\) | \(988\) |
| \(|\mathcal{S}| = 202\) | \(52.6\%\) | \(1.6\%\) | \(7.3\%\) | \(38.5\%\) | \(494\) |
4pt
1.0.5 Discussion↩︎
Our results point to skill shadowing as the primary bottleneck when scaling a skill library: once semantically similar skills compete for selection, accuracy degrades regardless of the underlying model. A natural next step is to attack this bottleneck directly through retrieval-based pre-filtering [9], [16], description disambiguation [11], learned routing [17], or other techniques. In addition, our study covers \(202\) skills on two models from a single provider, a scale at which context overhead remains modest. Whether these patterns—and the mitigations above—hold as libraries grow by an order of magnitude or span diverse architectures remains an open question.
Acknowledgment↩︎
We thank Zubair Anwar, Juanyan Li and Martin Valdez-Vivas for their valuable feedback and discussions throughout this work. Moreover, we truly appreciate the Data Science Leaders Sam Shah, Feng Pan, Zubair Anwar, and Divy Menghani for their support.
2 Appendix↩︎
2.0.1 Experiment Configurations↩︎
Definition 6 defines the oracle set at the individual-skill level, but in practice skills are often bundled together: multiple skills may jointly serve as the oracle set for
a given task. In Table 3, trend-anomaly-causal-inference is a good example.
| Primary task | Oracle bundle | distractor skill | Origin task(s) of distractor skill | \(\#_{\text{shd traj}}\) |
|---|---|---|---|---|
| mario-coin-counting | ||||
| image_editing, | ||||
| object_counter | video-frame-extraction | |||
| pedestrian-traffic-counting | \(51\) | |||
| trend-anomaly- causal-inference | ||||
| did_causal_analysis, | ||||
| feature_engineering, | ||||
| time_series_anomaly_detection | senior-data-scientist | powerlifting-coef-calc | \(37\) | |
| manufacturing- fjsp-optimization | ||||
| with-downtime-and-policy | ||||
| maintenance_guidance | manufacturing-equipment- maintenance | \(14\) | ||
| econ-detrending- correlation | timeseries-detrending | csv-processing | adaptive-cruise-control | \(4\) |
| energy-market-pricing | ||||
| economic-dispatch, | ||||
| locational-marginal-prices, | ||||
| power-flow-data | casadi-ipopt-nlp | energy-ac-optimal-power-flow | \(3\) |
0.1pt
In SkillsBench [12], we have skill bundle (or just bundle for brevity) authored by task creators—they are hand-crafted set of skills that the task author considers helpful for solving the given task. We call this set the task’s authored primary skill bundle, or just authored (skill) bundle for brevity. Bundles range from \(1\) to \(7\) skills (median \(2\), mean \(2.7\)); they are provided in the benchmark, not by this work.
2.0.1.1 Tasks included in our experiments
We only include a (task, model) pair into our experiment if every individual skill in the authored bundle clears the empirical threshold \(\tau = 0.04\) per our Definition 6, so that we can “break the bundle structure” and ensure each individual skill helps solve the task. The rationale is simple: if the skill does not help, one probably could not observe performance degradation even with more skills in the library. This filtering leaves us \(38\) (task, model) pairs: \(21\) Haiku and \(17\) Sonnet. From now on, the authored bundle serves as the oracle skill set \(\mathcal{S}^\star(q)\), and we use “oracle skill set” and “oracle bundle” interchangeably below. Our experiments consist of \(2{,}545\) trajectories: with \(303/297/544/273\) from Haiku and \(233/233/441/221\) from Sonnet for \(\mathcal{S} = \mathcal{S}^\star\), and \(|\mathcal{S}| \in \{52, 102, 202\}\), respectively.
2.0.1.2 Library construction
The largest library (\(|\mathcal{S}|{=}202\)) is the union of all \(88\) authored bundles after byte-identical de-duplication and numeric-suffix disambiguation for same-name skills with divergent bodies.
2.0.1.3 Bootstrap
Confidence intervals come from \(95\%\) percentile of the bootstrap distribution. For the decomposition (Eq. 8 ): at each bootstrap trial, we first resample (task, model) pairs with replacement, then resample trajectories within each sampled pair.
2.0.1.4 Examples with skill shadowing effect
Table 3 lists the most frequent empirical examples with skill shadowing effect at \(|\mathcal{S}|{=}202\), ranked by the number of trajectories in which the distractor skill is
the agent’s first pick. The first row is the mario-coin-counting example in our introduction §1.0.1.
2.0.2 Additional Experiments↩︎
2.0.2.1 Per-model pass rate drops
Table 4 presents the overall pass rate drops with a per-model breakdown, where we can observe both models degrade as the library grows. Haiku’s drop separates from zero already at \(|\mathcal{S}|{=}52\); Sonnet’s only at \(|\mathcal{S}|{=}202\).
| \(|\mathcal{S}|\) | Haiku 4.5 | Sonnet 4.6 |
|---|---|---|
| \(52\) | \(\mathbf{.10\;[.03, .18]}\) | \(.06\;[-.03, .15]\) |
| \(102\) | \(\mathbf{.22\;[.16, .29]}\) | \(.05\;[-.03, .13]\) |
| \(202\) | \(\mathbf{.26\;[.20, .33]}\) | \(\mathbf{.15\;[.06, .24]}\) |
20pt
| View | \(|\mathcal{S}|\) | \(\Delta_{\mathrm{ctx}}\) | \(\Delta_{\mathrm{shd}}\) | \(\Delta\) |
|---|---|---|---|---|
| Haiku 4.5 | \(52\) | \(.09\;[-.10, .27]\) | \(.02\;[-.04, .07]\) | \(.10\;[-.08, .28]\) |
| \(102\) | \(.15\;[-.04, .34]\) | \(.07\;[.00, .16]\) | \(\mathbf{.22\;[.05, .39]}\) | |
| \(202\) | \(.09\;[-.18, .36]\) | \(\mathbf{.17\;[.02, .37]}\) | \(\mathbf{.26\;[.09, .43]}\) | |
| \(52\) | \(.05\;[-.22, .33]\) | \(.01\;[-.07, .10]\) | \(.06\;[-.21, .33]\) | |
| \(102\) | \(.03\;[-.23, .30]\) | \(.02\;[-.06, .10]\) | \(.05\;[-.21, .31]\) | |
| \(202\) | \(.11\;[-.16, .40]\) | \(.04\;[-.04, .15]\) | \(.15\;[-.11, .41]\) |
10pt
Table 5 presents the estimates on both effects with a per-model breakdown: For Haiku, \(\Delta_{\mathrm{shd}}\) dominates and its CI excludes zero at the largest library, whereas for Sonnet, \(\Delta_{\mathrm{ctx}}\) matches or exceeds \(\Delta_{\mathrm{shd}}\) at every size, but we do not have necessary statistical power to distinguish from zero.
2.0.2.2 Theoretical upper bounds
We substitute the empirical \((\pi^\star_E, \rho^\star_E, \pi_E, \rho_E)\) from the decomposition into the right-hand sides of Lemma 1 (Eq. ?? ) and Lemma 2 (tighter form of Eq. ?? ). CIs are obtained by recomputing each bound on every bootstrap trial and taking the proper percentiles.
| Haiku 4.5 | Sonnet 4.6 | Pooled | ||||
|---|---|---|---|---|---|---|
| 2-3(lr)4-5(lr)6-7 \(|\mathcal{S}|\) | \(\Delta_{\mathrm{ctx}}^{\sup}\) | \(\Delta_{\mathrm{shd}}^{\sup}\) | \(\Delta_{\mathrm{ctx}}^{\sup}\) | \(\Delta_{\mathrm{shd}}^{\sup}\) | \(\Delta_{\mathrm{ctx}}^{\sup}\) | \(\Delta_{\mathrm{shd}}^{\sup}\) |
| \(52\) | \(.10\;[-.06, .36]\) | \(\mathbf{.04\;[.01, .19]}\) | \(.06\;[-.20, .33]\) | \(.16\;[.00, .44]\) | \(.08\;[-.08, .29]\) | \(\mathbf{.06\;[.01, .18]}\) |
| \(102\) | \(.17\;[.00, .41]\) | \(\mathbf{.09\;[.02, .32]}\) | \(.04\;[-.21, .30]\) | \(\mathbf{.17\;[.01, .43]}\) | \(.07\;[-.08, .31]\) | \(\mathbf{.14\;[.04, .29]}\) |
| \(202\) | \(.10\;[-.04, .43]\) | \(\mathbf{.35\;[.08, .74]}\) | \(.12\;[-.14, .41]\) | \(\mathbf{.15\;[.01, .37]}\) | \(.07\;[-.06, .34]\) | \(\mathbf{.24\;[.09, .46]}\) |
10pt
Table 6 reports the two upper bounds, where we denote \(\Delta_{\mathrm{ctx}}^{\sup} = \max\!\bigl(\rho_{\mathrm{\small n}}^\star - \rho_{\mathrm{\small n}},\; \rho_{\mathrm{\small o}}^\star - \rho_{\mathrm{\small o}}\bigr)\), and \(\Delta_{\mathrm{shd}}^{\sup} = |\pi_{\mathrm{\small n}}^\star - \pi_{\mathrm{\small n}}| + |\pi_{\mathrm{\small o}}^\star - \pi_{\mathrm{\small o}}|\). We observe \(\Delta_{\mathrm{ctx}}^{\sup}\) remains small across all conditions, and no CI separates from zero, confirming our finding that context overhead is not distinguishable from noise. \(\Delta_{\mathrm{shd}}^{\sup}\) grows sharply with library size and excludes zero under nearly all scenarios.
2.0.2.3 Pass rates for different invocation events
A finer partition of the invocation events. Definition 8’s mixed event m admits two types of events: the agent invoked at least one oracle skill alongside distractors, or it invoked only distractors. Combined with n and o, this gives a four-way partition:
“oracle only” (o);
“mixed – oracle invoked”;
“mixed – oracle not invoked”;
“no skill invoked” (n).
We report the binary pass rate for all four types of invocation events in Table ¿tbl:tab:rq4-2-binary? and the fractional pass rate in Table ¿tbl:tab:rq4-2-fraction?. CIs are Wilson \(95\%\) for binary and \(95\%\) percentile bootstrap for fractional.
10pt
@l l c c c c@ Invocation pattern & View & \(p\;(\mathcal{S}^\star)\) & \(p\;(|\mathcal{S}|{=}52)\) & \(p\;(|\mathcal{S}|{=}102)\) & \(p\;(|\mathcal{S}|{=}202)\)
& Haiku 4.5 & \(.43\;[.37, .50]\) & \(.33\;[.28, .40]\) & \(.26\;[.21, .32]\) & \(.34\;[.24,
.45]\)
& Sonnet 4.6 & \(.62\;[.56, .68]\) & \(.57\;[.50, .63]\) & \(.58\;[.53, .63]\) & \(.51\;[.43,
.58]\)
& Pooled & \(.52\;[.48, .57]\) & \(.44\;[.40, .49]\) & \(.45\;[.41, .49]\) & \(.45\;[.39, .52]\)
& Haiku 4.5 & — & — & \(.33\;[.12, .65]\) & —
& Sonnet 4.6 & — & \(.00\;[.00, .43]\) & \(.20\;[.04, .62]\) & \(.13\;[.02, .47]\)
& Pooled & — & \(.00\;[.00, .43]\) & \(.29\;[.12, .55]\) & \(.13\;[.02, .47]\)
& Haiku 4.5 & — & \(.00\;[.00, .35]\) & \(.05\;[.01, .22]\) & \(.00\;[.00, .23]\)
& Sonnet 4.6 & — & \(.00\;[.00, .32]\) & \(.14\;[.06, .29]\) & \(.13\;[.05, .32]\)
& Pooled & — & \(.00\;[.00, .20]\) & \(.10\;[.05, .21]\) & \(.08\;[.03, .22]\)
& Haiku 4.5 & \(.12\;[.06, .24]\) & \(.11\;[.06, .22]\) & \(.07\;[.04, .11]\) & \(.03\;[.02,
.07]\)
& Sonnet 4.6 & — & \(1.00\;[.77, 1.00]\) & \(1.00\;[.87, 1.00]\) & \(.80\;[.49, .94]\)
& Pooled & \(.12\;[.06, .24]\) & \(.27\;[.18, .38]\) & \(.16\;[.12, .20]\) & \(.07\;[.04, .12]\)
10pt
@l l c c c c@ Invocation pattern & View & \(\bar p\;(\mathcal{S}^\star)\) & \(\bar p\;(|\mathcal{S}|{=}52)\) & \(\bar
p\;(|\mathcal{S}|{=}102)\) & \(\bar p\;(|\mathcal{S}|{=}202)\)
& Haiku 4.5 & \(.83\;[.81, .86]\) & \(.77\;[.73, .80]\) & \(.77\;[.73, .80]\) & \(.80\;[.73,
.85]\)
& Sonnet 4.6 & \(.91\;[.88, .93]\) & \(.89\;[.87, .92]\) & \(.91\;[.89, .93]\) & \(.80\;[.74,
.84]\)
& Pooled & \(.87\;[.85, .89]\) & \(.83\;[.80, .85]\) & \(.85\;[.83, .86]\) & \(.80\;[.75, .83]\)
& Haiku 4.5 & — & — & \(.92\;[.87, .98]\) & —
& Sonnet 4.6 & — & — & — & \(.89\;[.81, .94]\)
& Pooled & — & — & \(.91\;[.85, .97]\) & \(.89\;[.81, .94]\)
& Haiku 4.5 & — & — & \(.70\;[.60, .79]\) & —
& Sonnet 4.6 & — & — & \(.85\;[.79, .90]\) & \(.64\;[.40, .87]\)
& Pooled & — & — & \(.80\;[.75, .85]\) & \(.64\;[.40, .87]\)
& Haiku 4.5 & \(.48\;[.36, .59]\) & \(.54\;[.45, .64]\) & \(.45\;[.40, .49]\) & \(.42\;[.37,
.48]\)
& Sonnet 4.6 & — & \(1.00\;[1.00, 1.00]\) & \(1.00\;[1.00, 1.00]\) & \(.80\;[.50, 1.00]\)
& Pooled & \(.47\;[.36, .59]\) & \(.62\;[.53, .71]\) & \(.50\;[.45, .55]\) & \(.44\;[.39, .50]\)
Context overhead is visible within the oracle-only bucket. For “oracle only” (event o)—trajectories where the agent invoked only oracle skills—the binary pass rate falls monotonically with library size on both models. This is the empirical signature of context overhead: the same invocation pattern produces a lower pass rate under a larger in-context library.
Fractional pass rate falls less than binary. Within “oracle only”, the fractional pass rate drops by less than the binary pass rate at every library size (e.g.pooled \(\bar p\): \(.87\!\to\!.80\) vs.pooled \(p\): \(.52\!\to\!.45\) at \(|\mathcal{S}|{=}202\)). Context overhead removes whole-task correctness more than partial progress.
Mixed and no-skill buckets are sample-limited. The two “mixed” buckets and “no skill invoked” draw on small samples in many conditions; their high-variance pass rates should be read alongside the cell counts in Table 8.
2.0.2.4 Invocation event distributions
We call \(\pi_{\mathrm{\small m}}\) (Definition 8) the empirical shadowing rate: the fraction of trajectories in which the agent invokes at least one distractor skill.
Table 7 reports \(\pi_{\mathrm{\small m}}\) (with Wilson \(95\%\) CIs) on the selected (task, model) pairs, and we find that Sonnet accumulates more wrong-selection mass than Haiku at every library size; Haiku’s \(\pi_{\mathrm{\small m}}\) stays low because its mass shifts to “no skill invoked” events instead of selecting wrong skills as shown in Table 8 later on.
| \(|\mathcal{S}|\) | Haiku 4.5 | Sonnet 4.6 | Pooled |
|---|---|---|---|
| \(52\) | \(.02\;[.01, .05]\) | \(.06\;[.03, .09]\) | \(.04\;[.02, .06]\) |
| \(102\) | \(.06\;[.04, .08]\) | \(.09\;[.07, .12]\) | \(.07\;[.06, .09]\) |
| \(202\) | \(.05\;[.03, .08]\) | \(.14\;[.10, .19]\) | \(.09\;[.07, .12]\) |
15pt
| \(|\mathcal{S}|\) | ||||||
| only | ||||||
| oracle | ||||||
| invoked | ||||||
| oracle not | ||||||
| invoked | ||||||
| invoked | \(n\) | |||||
| Haiku 4.5 | \(\mathcal{S}^\star\) | \(80.9\%\) | \(0.0\%\) | \(0.0\%\) | \(19.1\%\) | \(314\) |
| \(52\) | \(74.5\%\) | \(0.0\%\) | \(2.3\%\) | \(23.2\%\) | \(306\) | |
| \(102\) | \(47.8\%\) | \(1.6\%\) | \(4.0\%\) | \(46.5\%\) | \(546\) | |
| \(202\) | \(29.3\%\) | \(0.0\%\) | \(4.8\%\) | \(65.9\%\) | \(273\) | |
| \(\mathcal{S}^\star\) | \(97.5\%\) | \(0.0\%\) | \(0.0\%\) | \(\phantom{0}2.5\%\) | \(238\) | |
| \(52\) | \(87.7\%\) | \(2.1\%\) | \(3.4\%\) | \(\phantom{0}6.8\%\) | \(236\) | |
| \(102\) | \(84.8\%\) | \(1.1\%\) | \(8.1\%\) | \(\phantom{0}5.9\%\) | \(442\) | |
| \(202\) | \(81.4\%\) | \(3.6\%\) | \(10.4\%\) | \(\phantom{0}4.5\%\) | \(221\) | |
| \(\mathcal{S}^\star\) | \(88.0\%\) | \(0.0\%\) | \(0.0\%\) | \(12.0\%\) | \(552\) | |
| \(52\) | \(80.3\%\) | \(0.9\%\) | \(2.8\%\) | \(16.1\%\) | \(542\) | |
| \(102\) | \(64.4\%\) | \(1.4\%\) | \(5.9\%\) | \(28.3\%\) | \(988\) | |
| \(202\) | \(52.6\%\) | \(1.6\%\) | \(7.3\%\) | \(38.5\%\) | \(494\) |
22pt
Table 8 reports the per-bucket trajectory shares. The two models exhibit qualitatively different transitions as the library grows. On Haiku, “oracle only” (event o) collapses and the displaced mass goes almost entirely into “no skill invoked” (event n); the two mixed buckets stay small at every size. Haiku’s failure mode under expansion is therefore abandonment: rather than picking the wrong skill, the agent picks no skill. On Sonnet, event o declines more gently, event n stays small, and the displaced mass goes into the two “mixed” sub-cases of event m. Sonnet’s failure mode is wrong selection: the agent keeps invoking skills, but increasingly the wrong ones. This per-model contrast underlies the decomposition pattern in Table 1.
References↩︎
Author correspondence to Song Wei, e-mail:
vinson.wei@databricks.com↩︎