¹

1 Introduction↩︎

To add a new capability to an AI coding agent, developers used to browse a registry. Most of the time they just ask the agent which skill to use, and install whatever it names. A skill is a small bundle of natural-language instructions, kept in a SKILL.md file, that teaches an agent to do one task. The Agent Skills ecosystem is new, but it has grown quickly. Anthropic published the Agent Skills standard in December 2025, and OpenAI adopted it within weeks [1]–[13]. Analysts expect agent-enabled enterprise applications to rise from under 5% in 2025 to 40% the following year [14]. The registries kept pace, and SkillsMP alone lists more than 1.7 million skills [15]. Nobody can read through a catalog that large, so the agent decides and the user installs what it suggests. This habit is visible online. On Stack Overflow, Reddit, and Hacker News, developers routinely ask for skill and tool recommendations [16]–[18].

This delegation rests on a quiet assumption. The skill the agent names should exist. Often it does not. Language models hallucinate, and they state invented facts with full confidence [19]. Inside an agent, this shows up as a recommended skill whose name matches no entry in any registry. We call this a skill name hallucination. This is not a rare error. On real user questions it is the common case, and it persists even when the agent has a web-search tool and is told to verify the name first. These hallucinated skill names are not random. The model falls back on capabilities and tools it already knows. Some are generic capability phrases, such as file-system-operations or text-editor-tool. Others borrow names of familiar tools and products, such as visual-studio-code or ublock-origin. A web search finds results for all of these, because they name real capabilities or tools. The search confirms that the name means something, but not that any published skill has that name. Web search runs the wrong check.

A hallucinated name is harmless only while nothing exists behind it. An attacker removes that condition in three steps. The attacker first queries public agents and records the names they invent again and again. The attacker then registers one of those names on an open registry, with malicious instructions hidden in the SKILL.md. This step is cheap, because these registries barely check who publishes. Later, a victim who asks a similar question receives the same name and installs it. Installation is not a real barrier. Many agents auto-approve it, and even when a prompt appears, users tend to approve a recommendation from their own agent [20]. The same agent that recommends the skill is the one that later runs it (Section 2.2). No existing supply-chain or injection defense stops the attack, because a hallucinated name gives a defender nothing to work with. There is no real name to match, and the payload can hide in plain language rather than code [21]. This is what separates the threat from package squatting [22], and Section 4.3 measures the resulting detection failure.

To measure how real this threat is, we run the first systematic study of skill name hallucination. We send 15,000 prompts to 12 evaluated configurations and check every name they recommend against the live registries. The prompts are of two kinds, community prompts drawn from developer forums and skill description prompts built from real published skills. The 12 evaluated configurations comprise 4 standalone LLM configurations and 8 agent configurations, most of them given a web-search tool. We organize the study around four questions that follow the attacker’s path:

• RQ1 (Prevalence). How often do agents hallucinate skill names, and how robust is that rate to the model, its temperature, its size, and the way users ask?

• RQ2 (Targetability). How much attack coverage is possible by registering recurring hallucinated skill names?

• RQ3 (Detectability). Why are hallucinated skill names hard to detect by the model itself and by edit-distance filters?

• RQ4 (Mitigation). How well do retrieval-augmented generation (RAG), self-refinement, and cross-model voting reduce hallucination, and at what cost to the agent’s usefulness?

Every evaluated configuration hallucinates, at rates from 6.5% to 62.0%. The surface is widest where it matters most. On synthetic prompts, where a correct skill is known to exist, the mean rate is 29.4%, but on real developer questions it rises to 43.1%. No model, temperature, size, or phrasing removes the problem, and a web-search tool does not either. The names are also easy to target. In the current live-validated persistence run, Claude Sonnet 4.6 repeats its dominant hallucinated skill name 7.8 times in 10. Most names are specific to one configuration, but 410 are shared by two or more configurations, and these shared names account for 15.3% of all hallucinated recommendation events. The same naming prior also reaches neighboring software ecosystems, where 851 hallucinated skill names are real PyPI or npm packages.

The names are also hard to detect. A model asked to verify a name accepts real and hallucinated skill names at nearly the same rate, yielding 51% balanced accuracy, no better than a coin flip. They sit a median of six edits from any real skill, far beyond the reach of typosquatting filters. Defenses narrow the surface but do not close it. RAG cuts confirmed hallucination by 37.6 percentage points on average, from 40.8% to 3.2%, while self-refinement proves far less effective from 40.8% to 33.2%. Even the best-defended agent names the right skill only about one time in six, so it is safe mostly because it returns no output. The ecosystem stays hard to use safely.

We make the following contributions:

• A new attack on the agent skill supply chain. We identify skill name hallucination as the entry point of a skill name squatting attack, in which an agent’s own persistent hallucinations let an attacker pre-register the names that the agent will later recommend, and we show why this falls outside the reach of existing supply-chain and prompt-injection defenses (Section 2.2).

• A measurement of the attack surface and its targetability. Across 15,000 prompts and 12 evaluated configurations, we measure how prevalent, how reusable, and how stealthy hallucinated skill names are, including a new skill name squatting coverage analysis and a cross-ecosystem overlap between skills and Python and npm packages (Section 4).

• An evaluation of defenses and their cost. We test four defenses and report both their effect on the attack surface and their cost to the agent’s usefulness (Section 4.4).

2 Background↩︎

2.1 Skill Name Hallucination↩︎

Agents load skills progressively. The agent first sees only a skill’s name and description, and reads the full SKILL.md instructions only after it selects the skill. The name is therefore both a discovery signal and a security-sensitive input. Skill selection fails when the agent names a skill that does not exist. We call such a recommendation a skill name hallucination. The failure is similar to a known one in code generation, where models recommend packages that do not exist on PyPI or npm [22], [23].

2.2 Threat Model↩︎

Participants. We model four parties: the agent, which recommends and later loads a skill; the registry, such as ClawHub or SkillsMP, which accepts community skills; the victim, who installs the returned name; and the attacker, an ordinary user of the same public agents and registries. The attack relies on misplaced trust: the victim trusts the agent, the registry trusts publishers, and the agent cannot distinguish a real skill from one it invented (Section 4.3). Unlike package hallucination, one agent can invent, recommend, and later execute the installed skill.

Attack flow. The attack has three steps (Figure 1). First, the attacker queries public agents and records skill names that recur. Second, the attacker publishes a skill under one such name, with malicious instructions or scripts in its SKILL.md package. This requires little privilege: ClawHub accepts week-old accounts, and SkillsMP indexes public repositories. Third, a later victim receives the same hallucinated name and installs it. Even when installation is not automatic, prompts often show only the name, not the hidden instructions.

3 Experiment Design↩︎

To measure skill name hallucination across models and agents, we use a three-stage pipeline. (1) Prompt Construction builds a dataset of skill-seeking prompts. (2) Recommendation Generation sends each prompt to every model and agent and records the skill names they return. (3) Skill Name Validation checks each returned name against the live registries and counts any name no registry contains as a hallucination.

3.1 Prompt Construction↩︎

Our dataset has 15,000 prompts of two types, each measuring a different aspect of hallucination. Community Prompts are real questions drawn from developer forums. They measure how often an agent hallucinates on the questions users actually ask. Many of these questions have no matching skill, so the model must either state that none exists or invent a name. Skill Description Prompts are built from the description field of a real published skill’s SKILL.md, the short text that states what the skill does. Because each prompt comes from a real skill, a correct skill is known to exist for every one. These prompts measure whether a model can output a correct skill name it should be able to find. Together, the two types separate hallucination caused by a missing skill from hallucination that remains even when the right skill exists.

3.1.1 Community Prompts↩︎

To make the hallucination rate reflect a real attack surface rather than an artificial one, we collect prompts that match the questions real users ask. We mine real questions from three developer communities: Stack Exchange technical sites, AI and developer subreddits, and Hacker News (retrieved through the Algolia API). In each source, we search for skill-seeking questions with queries such as “what tool can I use,” “best plugin,” and similar skill recommendation phrases.

Most forum posts are irrelevant to our task, so before building prompts we score each post with a set of relevance rules. The rules use the post title, body, and source community. They add evidence for a software skill question when a post contains a target term (skill, tool, plugin, extension, mcp, agent), software or AI context (e.g., github, codex, claude, llm), a technical community (e.g., Stack Overflow, r/vscode), or a question form. They subtract evidence when a post uses skill in the human sense (e.g., soft skills, cooking skills), comes from a nontechnical community (e.g., r/AskReddit), or mentions a target term without technical context, which usually refers to a physical tool or human ability. We keep only posts whose score exceeds a threshold, so genuine software skill questions pass while irrelevant posts fall below the cutoff. The result is 5,000 community prompts covering skill name inquiries (2,395), skill recommendations (1,842), and usage descriptions (763), drawn from Stack Exchange (2,484), Reddit (1,878), and Hacker News (638).

3.1.2 Skill Description Prompts↩︎

Community prompts show how often agents hallucinate in the wild, but for many of them no real skill fits. We therefore cannot tell whether a hallucination comes from a missing skill or happens even when a correct skill exists. Skill description prompts resolve this ambiguity, because each one is built from a real published skill, so a correct answer is known to exist. We take the 10,000 most popular skills on SkillsMP and turn each skill’s description field into one prompt that asks the model to return the matching skill’s name. Because each prompt comes from a real skill, a correct answer always exists, so any hallucination here cannot be attributed to a missing skill.

3.2 Recommendation Generation↩︎

We evaluate skill name hallucination on two kinds of system, standalone LLMs and LLM-powered agents. The main design challenge is the knowledge cutoff. Anthropic released the Agent Skills standard on December 18, 2025 [1], after the training cutoff of most models we test. These models could then hallucinate because they never saw a real skill during training, not because of any deeper tendency. To rule out this explanation, we give every pre-cutoff model and agent a web-search tool [24], require it to search before it recommends a skill, and confirm that search works for each one. The one exception is Claude Opus 4.7 (cutoff January 2026), which already knows the standard. We run it without search to approximate the best case for training-time knowledge.

Table 1 lists the models and agents we evaluate. We use four standalone LLM configurations and eight agent configurations spanning four agent frameworks. Claude Code and Codex search the web natively, while OpenClaw and OpenCode use a self-hosted SearXNG instance through MCP. We run OpenCode with five backends (GPT-5.4-mini and four open-source models). Together, the 4 standalone LLM configurations and 8 agent configurations give the 12 configurations we report throughout the paper. Before running the skill prompts, we checked that web search worked for every search-enabled system. We asked each one to fetch a current-news item on the test day. This check only confirmed that search was reachable, and we excluded it from the hallucination measurements.

4pt

The OpenCode row with four Ollama models expands to four agent configurations: Ministral-3 8B, Granite 3.3 8B, Qwen 3.5 9B, and Nemotron-Cascade-2 30B. Later figures shorten these names to Min3-8B, Gra3.3-8B, Qwen3.5-9B, and Nem2-30B.

Figure 2 shows the two prompt templates. For community prompts, the system message instructs the model to act as a skill recommender and requires kebab-case names. The user message adds the forum post, the required output format, and an instruction to search the web first. For skill description prompts, the system message asks the model to identify the one real skill that matches, and to return an empty skill_names list rather than guess when it is unsure. Both templates require a JSON reply of the form {"skill_names": [...], "answer": "..."}. We cap each reply at 1,200 tokens and leave all sampling parameters at provider defaults, except in RQ2, where we vary them on purpose.

None

3.3 Skill Name Validation↩︎

We treat a recommended skill as a hallucination when it exists on no skill-hosting registry and in no open-source repository. To verify whether a skill is real, we use a two-stage validation. Registry verification. We match the name against a broad reference index built from multiple public sources of real skill names, including the SkillsMP and ClawHub registries, the curated awesome-openclaw list, Claude and Codex skill repositories, historical SKILL.md records, and our prior registry lookups. SkillsMP in particular is built by periodically harvesting skill from public GitHub repositories and now indexes over 1.7 million skills, so a name missing even from this large catalog is already very likely fake. Expanded verification. For every name the registries do not resolve, we expand the search to GitHub. We query GitHub Code Search with "name: <canonical-name>" filename:SKILL.md, fetch the candidate SKILL.md files, and accept a match only when a candidate’s front-matter name: field canonicalizes to the generated name. This stage is slow and costly, but it ensures that a real skill too new to appear in the registries is never miscounted. We label a name a hallucination only when both stages fail.

We take the names from the skill_names field of each response and match them against this validation pipeline. We first normalize both generated and reference names by trimming wrappers and markup, splitting camel case, converting & to and, replacing non-alphanumeric runs with hyphens, and lowercasing. A name counts as real on an exact match, a unique canonical-alias match, or a matching name: field in a SKILL.md found by either stage. We discard clear extraction artifacts, such as URLs, paths, schema tokens, trace IDs, and sentence-like strings. None of these can be a skill name, which is a short kebab-case identifier, so they are noise from parsing the response rather than genuine recommendations. We exclude them instead of counting them as hallucinations. We cache all lookups to avoid repeated searches.

Our primary metric is the skill name hallucination rate. We compute this metric at the unique-name level, not over repeated recommendation events. Let \(R\) be the set of distinct canonical names a system recommends, after we discard the extraction artifacts above. For each name \(r \in R\), the indicator \(\mathbb{1}[,r \text{ is hallucinated},]\) equals \(1\) when \(r\) fails both verification stages and \(0\) otherwise. The hallucination rate is the share of these unique recommended names that are hallucinated:

\[\label{eq:hallucination95rate} \text{Hallucination Rate} = \frac{\sum_{r \in R}\mathbb{1}[,r \text{ is hallucinated},]}{|R|}\tag{1}\]

A higher rate means a wider attack surface, because more of an agent’s recommendations point to skills that do not exist. On the skill description prompts, where a correct skill is known to exist, we additionally report recall, the fraction of prompts for which the model returns the correct name.

4 Empirical Results↩︎

We investigate four research questions, each on a distinct dimension of skill name hallucination. RQ1 measures prevalence, how often agents hallucinate and whether the rate holds across model family, temperature, size, and prompt domain. RQ2 measures targetability, how often hallucinated names repeat, how widely they are shared across configurations, how often they overlap with real software packages, and how few names cover most cases. RQ3 measures detectability, whether self-auditing or lexical distance to real skill names can identify hallucinated names. RQ4 measures mitigation, how well RAG, self-refinement, and cross-model voting reduce hallucination.

4.1 RQ1: Prevalence↩︎

To systematically measure the prevalence of this attack surface, we evaluated all 12 configurations on the full 15,000-prompt corpus. Every recommended name was validated using the two-stage pipeline from Section 3.3. To avoid duplicate counting, hallucination rates are computed over unique skill names rather than repeated recommendation events. 2 reports these statistics, separating name-level metrics and prompt-level outcomes to illustrate the varying attack surfaces under different external grounding methods. The results demonstrate that hallucination is ubiquitous, with rates ranging from 6.5% for GPT-5.4-mini to 62.0% for OpenCode with Ministral-3 8B, averaging 36.0% across standalone LLMs and 36.9% across agents.

3.5pt

Unique equals Matched, Halluc., and Other, which represent verified real skills, unverified hallucinations, and extraction artifacts, respectively. H. Rate is the ratio of Halluc. to Unique. No output indicates the fraction of prompts yielding no skill recommendation.

Rather than generating arbitrary noise, the models predominantly output semantically plausible but non-existent skill identifiers. Qualitative analysis reveals two dominant categories. The first comprises abstract capability descriptions (e.g., file-system-operations or text-editor-tool), where the model attempts to describe the required action instead of recalling a verified skill. The second involves adopting the names of real-world software or ecosystem objects that have not been officially registered as skills (e.g., visual-studio-code or ublock-origin).

External grounding via web search fails to mitigate this vulnerability. Despite enabling three of the four standalone LLMs with built-in search tools and explicitly prompting them to verify names prior to output, hallucination rates remain critically high (e.g., 49.4% for Claude Sonnet 4.6 and 51.4% for GLM-4.5-Air). This failure stems from the semantic plausibility of the hallucinated names, because searching for file-system-operations returns abundant relevant results, thereby tricking the agent into confirming the concept’s existence rather than checking registry status.

Furthermore, conservative output generation artificially deflates hallucination rates without addressing the underlying threat. For instance, GPT-5.4-mini achieves a deceptively low rate of 6.5% primarily because it returns no skill on 52.5% of the prompts. This behavior is more extreme in agent configurations, with Claude Code returning no output for 90.9% of prompts and yielding the fewest names overall, yet 34.6% of its outputs are still hallucinated. Consequently, higher no-output rates simply limit output volume, leaving the output skill names highly vulnerable to hallucination.

The evaluation of Claude Opus 4.7 provides insights into whether recent training data inherently neutralizes the attack surface. As its January 2026 knowledge cutoff postdates the December 2025 Agent Skills release [25], this configuration isolates the impact of training-time knowledge updates. Operating without web search, the model adopts a highly restrictive generation strategy, returning no skills for 56.8% of prompts. Although its hallucination rate drops to 36.7%, falling below that of Claude Sonnet 4.6, it remains higher than several search-enabled agents. For example, OpenCode with GPT-5.4-mini and OpenClaw with GPT-5.4-mini achieve lower rates of 23.3%, 25.5%, respectively. Thus, updated training corpora reduce output volume but do not remove the attack surface.

This vulnerability is particularly pronounced under real-world usage conditions. As 3 shows, forum queries pose a consistently higher risk than synthesized skill description prompts. The median hallucination rate reaches 38.8% on Hacker News and Reddit, and peaks at 50.1% on Stack Exchange. Because forum questions closely resemble the queries an attacker’s victim would issue in practice, this gap expands the real-world attack surface.

To assess whether this attack surface can be mitigated through hyperparameter tuning or model scaling, we evaluated various decoding temperatures and parameter sizes using prompt subsets of 4,000 prompts. 4 (a) shows that changing the temperature for search-enabled LLMs from 0 to 1.0 produces little improvement. The per-prompt traces show surface-level variation without recovery. For the Retell AI troubleshooting skill, whose real name is , GLM-4.5 produced , , , and across temperatures, but never the real name. Similarly, for the CPU monitoring skill , the same model alternated between and . These examples illustrate that temperature mainly changes which plausible paraphrase is emitted, rather than forcing the model to verify the registered identifier.

4 (b) shows a similar pattern for model scaling across Granite 3.3, Ministral-3, Qwen 3.5, and Nemotron-Cascade-2. Larger variants sometimes reduce the aggregate rate, such as the drop in Ministral-3 from 51% at 8B to 30% at 14B, but they still produce semantically close false names. In the Ministral-3 14B run, the model mapped to and mapped to both and . Qwen 3.5 27B similarly mapped to , to , and to . Even the largest evaluated model, Nemotron-Cascade-2 30B, retained an 11% hallucination rate over emitted names, corresponding to 390 hallucinated names among 3,588 recommendations on the shared 4,000-prompt subset. Thus, while scaling reduces the frequency of hallucinations across some model families, it fails to eliminate the underlying mechanism, as models continue to translate descriptions into plausible aliases similar to skills rather than recovering the exact published names.

4.2 RQ2: Targetability↩︎

The exploitability of this attack surface depends on predictability. To successfully execute a skill name squatting attack ahead of time, adversaries must anticipate which hallucinated names agents will generate before victims query them. Consequently, a hallucinated name becomes a high value target under two conditions: (1) high intra-model persistence, where a specific model reliably repeats the name to ensure consistent traps for victims, and (2) high inter-model convergence, where multiple models independently generate the same name to maximize the impact scope of a single registered payload. We evaluate both dimensions to quantify the targeting budget required for an attacker to achieve broad victim coverage.

Our experiment reveals high intra-model determinism for hallucinated names. By querying seven configurations 10 times each with 500 random prompts from RQ1, we define the dominant hallucination for each configuration and prompt as the confirmed hallucinated name that appears most often across the 10 runs, and then track how often it recurs. As shown in 5, Claude Sonnet 4.6 exhibits striking persistence, repeating its dominant hallucination 7.8 times out of 10 on average, and returning the exact same name across all 10 runs in 53% of cases. This stickiness stems from the model’s adherence to instructions rather than local caching. Across all 10 trials, it generated instead of the real , instead of , and instead of . While open-weight models are more volatile, repeating 1.5 to 2.0 times, highly persistent models allow attackers to reliably predict future outputs based on a single observed recommendation.

Beyond intra-model persistence, hallucinated names demonstrate significant inter-model convergence. Among the 1,728 prompt and model pairs with confirmed hallucinations, 219 dominant names were generated by at least two distinct models. For example, six independent models produced classification-model-builder, and five converged on names like exa-incident-response, firebase-
genkit and debugging-guidelines. This indicates that models share underlying naming heuristics. When presented with a plausible capability, they predictably converge on identical hallucinated names. For an attacker, this shared behavior means a single hallucinated name can compromise users across entirely different LLM ecosystems.

A measurable sign of this shared naming prior is overlap with the broader developer software ecosystem. To test whether hallucinated skill names reflect neighboring package namespaces, we queried PyPI and npm for the 5,669 unique confirmed hallucinated names. As shown in 3, 851 names, or 15.0%, exactly match an existing package, even though the prompts requested agent skills rather than software packages. These matches are not valid skills. They indicate that models sometimes reuse familiar package or tool identifiers as names for skills. The overlap is concentrated in single configurations, but 126 names recur across two or more configurations, supporting the hypothesis that neighboring ecosystems shape predictable cross-model names.

^*Percentage out of 5,669 unique, confirmed hallucinated names.

While the absolute number of unique hallucinated names is vast and highly fragmented across configurations, the actual event frequency is heavily concentrated in shared names. 6 illustrates the full set of names recovered from the 15,000-prompt corpus. Counted purely by distinct identifiers, 92.8% of hallucinated names are exclusive to a single configuration. However, weighting these names by recommendation frequency reveals a different threat landscape. The 410 names that appear in at least two configurations account for 15.3% of all 6,212 hallucination events, which is double their 7.2% share by distinct count. Thus, most hallucinated names appear only once, but the names that recur are recommended often enough to become highly reusable targets.

This concentration allows attackers to achieve high impact with a minimal targeting budget. 7 ranks hallucinated names by recurrence frequency to calculate the coverage an attacker would achieve by squatting the top-\(k\) names. Each cell reports the percentage of hallucinated recommendations covered at that budget, and the “All” row pools events across the seven configurations in the persistence experiment. For Claude Sonnet, registering merely the top 25 names intercepts 63% of all its hallucinated recommendations, and 50 names cover 96%. Even aggregated across all seven configurations, squatting the top 500 names covers 57% of all hallucinated recommendations. An attacker does not need to predict every rare hallucination, and a cheaply harvested dictionary of a few hundred predictable names sits exactly in the execution path of a majority of victims.

4.3 RQ3: Detectability↩︎

While RQ1 and RQ2 demonstrate the prevalence and predictability of this attack surface, an effective defense depends on whether defenders can identify hallucinated names prior to execution. We systematically evaluate two detection strategies: LLM self-auditing and typosquatting filters based on edit-distance analysis.

We first evaluate whether models can identify hallucinations in generated skill names. We asked seven models a yes/no question to check if a given skill name exists. Each model judged real skills against confirmed hallucinations under two conditions. The first is same-model testing for its own hallucinations. The second is cross-model testing for hallucinations pooled from the other six models. After removing unclear non-binary replies, we retain 3,707 of 4,448 model responses as valid binary judgments.

8 illustrates why this defense is fundamentally ineffective. As shown in 8 (a), the models overwhelmingly default to “no” and reject almost everything, specifically 90.6% of names in the same-model condition and 88.8% in the cross-model condition. While this extreme bias safely blocks 91% of hallucinations, it destroys the agent’s usefulness. The models recognize only 10.0% of real skills among their own outputs and 13.6% among those of other models. Even when a model does answer “yes” to its own generated names, it is still wrong 22.7% of the time. Consequently, the per-model separation shown in 8 (b) yields an aggregate balanced accuracy of merely 51% for same-model tests and 53% for cross-model tests. Averaged across the two classes, the performance is no better than guessing.

The same-model and cross-model comparison helps explain this failure. A model performs worse at judging its own hallucinations than those of its peers because it uses the same internal logic to review the name as it did to generate it. GPT-5.4-mini is the only partial exception. It shows a less extreme rejection bias of 72.7% and the highest real-skill recall of 28.0%, yet it still fails to audit reliably. Ultimately, a model cannot filter a mistake it is built to make.

Traditional typosquatting defenses are inadequate because hallucinated names are novel semantic constructs rather than minor misspellings. Standard typosquatting filters typically flag variations within a Levenshtein distance of one or two from known real skills. However, comparing 5,669 confirmed hallucinated names against a catalog of 22,805 real skills reveals a mean edit distance of 6.85, as shown in 9. While 9.3% of hallucinations represent near-misses, such as single substitutions, that such filters can catch, the vast majority, 56.1% at \(\ge6\) edits, bypass them entirely. These distant outliers are not typos, but rather long, newly synthesized capability descriptions, rendering structural similarity checks useless.

4.4 RQ4: Mitigation↩︎

Finally, we investigate whether defenders can effectively mitigate this attack surface and evaluate the corresponding trade-offs regarding agent utility. To this end, we systematically evaluate four defensive strategies. Three operate within a single-model context, including RAG, self-refinement, and their combination. The fourth is a cross-model defense that requires consensus among multiple independent models before accepting a generated name. We applied the single-model defenses to the four open-weight models from RQ1 using a standardized subset of 2,000 prompts. 4 summarizes the resulting hallucination and no-output rates, where no output denotes cases producing no skill recommendation.

We first evaluate whether RAG can neutralize the attack surface by grounding models in real skill names. Using RAG [26], we build a vector index over real skill names and their available descriptions with all-MiniLM-L6-v2 sentence embeddings and the Chroma backend. For each user query, we embed the original prompt, retrieve the ten nearest skills by semantic similarity, format their names as a plain-text reference list, and prepend this list to the query under a system instruction that permits recommendations only from the retrieved names. We use ten retrieved skills to provide enough candidate coverage while keeping the context short. As shown in 4, this defense yields the largest drop in hallucinations. For example, Qwen 3.5 9B falls from 32.5% to 0.9%, and Ministral-3 8B from 61.3% to 1.6%. On average, RAG cuts the hallucination rate by 37.6 percentage points. However, this defense is not free. It requires index maintenance, adds token costs, and creates retrieval-poisoning exposure [27], [28].

We next test whether the detection failure observed in RQ3 carries over to an end-to-end mitigation pipeline. In the self-refinement setup [29], the model first generates candidate skill names and then receives its own output in a second turn, where it is instructed to remove any name it cannot verify. This internal validation reduces the average hallucination rate from 40.8% to 33.2%, but the reduction comes mainly from suppressing outputs rather than from reliable verification. The no-output rate rises from 73.9% to 87.4% on average, and becomes extreme for Qwen 3.5 9B and Nemotron-Cascade-2 30B, reaching 97.5% and 96.7%, respectively. Therefore, the RQ3 detection failure is not merely an artifact of binary yes/no testing: when used as an operational defense, self-refinement trades hallucinations for no output and still leaves a high hallucination rate among the few names it retains.

Given the isolated outcomes of RAG and self-refinement, we also examined whether combining them could offer compounding benefits. Running self-refinement after RAG only marginally changes the average hallucination rate, from 3.2% under RAG to 2.7%, while increasing the no-output rate from 68.0% to 79.4%. Once RAG restricts the model to real skills, the extra self-refinement pass mostly removes additional outputs instead of adding a meaningful safety benefit.

Shifting away from single-model strategies, we evaluated a cross-model voting defense. Building on our insight from RQ2 that hallucinations are usually specific to one model while real skills are shared, we hypothesize that requiring multiple models to agree could act as a natural filter. Indeed, as shown in 10 (a), if we accept a name only when two independent models produce it, corresponding to \(N=2\), we eliminate 92.8% of hallucinations while keeping 44.8% of real skills. Raising the threshold to \(N=3\) removes over 98% of hallucinations, though real-skill preservation drops to 21.9%. This sharp drop confirms that hallucinated names are isolated to single models. This approach requires no external database or model updates, needing only multiple independent agents.

While the aforementioned filters successfully suppress hallucinations, they introduce a severe penalty to the agent’s overall utility. Because a defense that simply returns no output is not a practical fix, we also measure recall, defined as the rate at which the agent successfully recommends the correct skill when a real one exists. Under RAG, safety and usefulness fundamentally trade off against each other. For the two weaker models, RAG improves recall by handing them correct answers they would not have guessed, with Ministral-3 rising from 1.7% to 12.9% and Granite 3.3 increasing from 2.2% to 15.4%. But for the two stronger models, recall still drops, as seen when Qwen 3.5 falls from 16.5% to 10.0% and Nemotron-Cascade-2 decreases from 20.7% to 16.0%, showing that retrieval constraints can suppress hallucinated names without reliably preserving the exact correct skill. Ultimately, even under RAG, no model recommends the correct skill more than 16.0% of the time. 10 (b) shows that RAG shifts outputs toward verified real names without making real-name recall reliably high. The agent becomes safer mainly by returning fewer hallucinated names, not by becoming reliably correct.

5 Discussion and Limitations↩︎

Discussion. Our hallucination rates of 6.5% to 62.0% exceed the 5.2% to 21.7% reported for traditional packages [22]. This gap comes from a lack of knowledge about skills. The Agent Skills standard appeared in December 2025 [1], after the training cutoff of most models we test, so models often fall back on familiar package names from ecosystems such as npm [30] and PyPI [31]. As a result, a name like eslint-config-prettier is returned as a skill (Section 4.2), making the same hallucinated name useful for attacks across multiple ecosystems.

This explains the inadequacy of simple defenses. Web search verifies a tool’s existence, but not its validity as a skill. Self-refinement has the same problem because the second pass evaluates the answer using the same prior that produced the hallucination. RQ4 shows the cost of suppressing these names. Even the agent with the strongest defense returns the correct skill at most one time in six, while several configurations reduce hallucination mainly by returning no output. This trades false positives for false negatives, producing safer recommendations but fewer useful ones. The core challenge is therefore not silence, but retrieval of the right skill without admitting hallucinated skill names.

This trade-off also suggests that skill registries need defenses that are more explicit than package registries. Because a skill is primarily a natural-language instruction bundle, artifact-level checks provide weaker signals than they do for ordinary packages. A practical defense should therefore treat recommendation as a registry lookup. The agent should retrieve from an authenticated skill index and refuse names that cannot be resolved to a concrete SKILL.md. Namespace reservation for recurring hallucinated names may reduce ahead-of-time squatting, but it cannot replace exact lookup because our results show a long tail of model-specific names.

Limitations. Our study measures the attack precondition, in which agents hallucinate reusable skill names and defenses often miss them. We also verify the registration and installation path with a benign placeholder skill in an isolated sandbox, but we omit a full malicious compromise. Ethically, we did not register malicious skills, attack any public registry, or expose real users to the workflow. The exploitation step itself follows a known mechanism. Once a poisoned SKILL.md is installed, it enters the agent context and can act as instruction level prompt injection, as shown in prior work [21], [32], [33]. We establish this delivery channel is plausible because attackers can predict attractive names and register them in advance. Measuring whether real users install such recommendations requires a controlled user study. Finally, our recall analysis covers only the four open weight models used for mitigation, so we defer the full false-negative study across all 12 configurations.

6 Related Work↩︎

6.1 Supply Chain Attacks and Package Hallucination↩︎

While traditional supply chain attacks [34]–[37] exploit mistyped legitimate packages, LLMs introduce novel threats via hallucinated recommendations. Following discoveries of package hallucinations [22], recent research categorized code generation errors [38] and proposed API-level mitigations [39]. However, extending these insights to the Agent Skills ecosystem demands a fundamental threat model shift. Unlike mature code registries supporting artifact comparison against established references, skill hallucinations lack legitimate counterparts. Furthermore, skills execute natural language instead of compiled code, rendering static analysis ineffective.

Our work is closest to package hallucination, but differs in both the object being named and the user’s trust path. A hallucinated package must usually survive dependency resolution, build tooling, and code review before it affects an application. A hallucinated skill can be installed because the same agent that recommends it also helps fetch and activate it. The payload is also not limited to executable source code. Adversarial instructions can be embedded directly in a SKILL.md, where they become part of the agent’s future context. We therefore study not only whether names are false, but whether false names are persistent, reusable, and attractive enough for attackers to register before victims ask for them.

6.2 Agent Security and Prompt Injection↩︎

Installed malicious skills inherently threaten the execution integrity of LLM agents. Recent studies have surveyed widespread vulnerabilities in agent applications [40]–[44], demonstrating how malicious modules or online interactions amplify risks like data exfiltration and impersonation [45], [46]. A critical consequence of such compromises is indirect prompt injection, extensively benchmarked in recent literature [32], [33], [47]–[49]. Contemporary defenses utilize structured query separation [50], preference optimization [51], execution isolation [52], and guardrail agents [53] to sanitize adversarial inputs during inference. Yet, these methods implicitly assume the malicious payload arrives alongside the user query. Skill name hallucination circumvents these boundaries entirely. By tricking the user into installing a poisoned skill, the adversary injects adversarial instructions directly into the persistent configuration of the agent, bypassing standard alignment and runtime protections [54]–[56]. This makes installation a persistent and reusable trust boundary.

6.3 Threats in the Agent Skills Ecosystem↩︎

Consequently, the Agent Skills ecosystem presents a unique and largely unaddressed attack surface. Liu et al. [57] identify widespread vulnerabilities within published skills, while Qu et al. [58] demonstrate that malicious logic embedded in skill documentation can successfully manipulate agent behavior. Both works assume the attacker compromises existing skills or publishes payloads hoping for manual discovery. Our research complements these studies by formalizing vulnerabilities at the recommendation phase. This shifts the attack earlier than prior work considers. We demonstrate that attackers do not need to wait for organic discovery, but rather can weaponize the persistent hallucinations of LLMs in advance to actively drive victims toward malicious skills.

7 Conclusion↩︎

We studied skill name hallucination as a supply-chain attack across 12 configurations and 15,000 prompts. Every one hallucinated, at 6.5–62.0% overall and 43.1% on real questions. The names are targetable: models repeat a name up to 7.8 in 10, 410 shared names cover 15.3% of hallucinations, and 851 are real PyPI or npm packages. They also evade detection: self-checks match chance, and names sit a median of six edits from any real skill. RAG cuts hallucination by 37.6 percentage points, yet the best-defended agent recommends the right skill at most one in six. It is therefore a real, low-cost attack surface, and closing it requires registry-level reservation and verified recommendation, not tuning alone.

8 Data Availability↩︎

We release all prompts, skills, and responses at https://figshare.com/s/4e408bce9b2f64224f79.

References↩︎

---

1. Wenbo Guo and Feng Dong are corresponding authors.↩︎