1 Introduction↩︎

With the recent growth of autonomous software agents, such as Claude Code [1] and GitHub Copilot [2], that use Large Language Models (LLMs) to perform Software Engineering (SE) tasks, the ability to provide agents with task-specific and project-specific knowledge has become increasingly important. Agent Skills¹ [3] address this need by packaging procedural knowledge together with organization-, team-, and user-specific context into portable, version-controlled artifacts that agents can retrieve on demand. As evidenced by the emergence of public skill marketplaces containing more than 100,000 skills [4], [5], skills have become a widely adopted mechanism for adapting general-purpose agents to specialized tasks.

Originally introduced by Anthropic [3], the Agent Skill Specification defines a skill as a directory containing a required SKILL.md file and optional scripts, references, and assets [6]. The SKILL.md file consists of frontmatter, a metadata header in YAML format [7] with a set of predefined supported fields [6], which helps agents identify and load relevant skills, and a Markdown body containing skill instructions. Figure 2 shows an excerpt from a SKILL.md body alongside two excerpts from frontmatter sections. Beyond these structural requirements, skill authors are free to include any content that helps agents perform the target task effectively.

The largely unconstrained nature of SKILL.md body and content create the potential for substantial variation in skill quality. Because skills directly contribute to the context provided to agents, poorly written skills may introduce irrelevant information, omit important guidance, or present instructions in ineffective ways, ultimately resulting in suboptimal agent behavior. Such effects may impact not only task performance but also resource consumption by increasing context length and reducing the usefulness of retrieved information. Despite the rapid adoption of agent skills, prior research has primarily focused on their effectiveness [8] and security implications [9]–[12], leaving their authoring practices largely unexplored. Understanding this gap is particularly important because skills may increasingly be used as training data for future models and may themselves be generated by LLMs, creating the potential for poor authoring practices to propagate through both training and generation pipelines.

The history of SE shows that widely used textual artifacts tend to accumulate quality problems when they evolve without clear quality guidelines. For example, prior work found that commit messages, which are also free-form text describing a code change, were often incomplete, uninformative, or even empty [13], and that poor-quality commit messages were associated with software defects [14]. Because Agent Skills are likewise largely unconstrained in both structure and content, we posit that they represent an emerging software quality concern for agentic systems. Since Agent Skills are still in the early stages of adoption, there is a unique opportunity to understand current authoring practices and identify problematic patterns before they become widespread and entrenched. Hence, we conduct the first large-scale study of SKILL.md quality, best practices, and quality deficiencies in real-world agent skills to understand what information they contain, characterize the semantic components of their bodies.

RQ1: What are the semantic components of SKILL.md files in practice?

The next step toward assessing SKILL.md quality is understanding how their semantic components should be authored. Although practitioners and organizations have proposed best practices for writing SKILL.md files best-practices-google?, this guidance is scattered across disparate sources, making it unclear which semantic components are covered and which lack guidance. We therefore systematically collect and map existing best practices to the semantic components they govern. This establishes a notion of what constitutes well-authored SKILL.md files and provides the foundation for identifying quality deficiencies through violations of these practices.

Prior work in SE has shown that violations of established best practices often manifest as software quality problems. For example, violations of coding best practices, commonly referred to as code smells, have been associated with increased fault-proneness, reduced maintainability, and lower software quality [15], [16]. Motivated by this line of work, we define skill smells as violations of best practices for authoring SKILL.md files. Therefore, in our second research question, we investigate the best practices for authoring SKILL.md files, the semantic components they govern, and the resulting skill smells that arise when these practices are violated.

RQ2: What best practices exist for authoring SKILL.md, and what skill smells emerge from their violation?

Because agents rely on skills to obtain task-specific context and procedural guidance, the quality of a SKILL.md file directly affects the quality of the information supplied to the agent. Similar to code smells, skill smells may introduce unnecessary complexity, reduce the effectiveness of the guidance provided, and degrade the quality of the context on which agents rely for reasoning and decision-making. For example, unnecessary content in a skill may consume valuable context-window capacity and distract the agent from the task at hand [17]. However, before developing techniques to detect or mitigate skill smells, it is first necessary to understand how prevalent they are in real-world Agent Skills. Measuring their prevalence provides the first empirical evidence of the scale of the problem and helps prioritize future quality-improvement efforts. This leads to our next research question.

RQ3: How prevalent are skill smells in real-world SKILL.md files?

Identifying skill smells is only the first step toward understanding their impact on the agent-skill ecosystem. An important follow-up question is whether they are corrected over time or instead persist as skills evolve. Prior SE research has shown that quality characteristics often deteriorate throughout software evolution. For example, commit-message quality decline as software evolves [14], the prevalence of code smells tends to increase in open-source repositories [18]. Such longitudinal analyses help determine whether quality issues are naturally mitigated through maintenance activities or become increasingly entrenched over time. Understanding this evolution is particularly important for skill smells because persistent quality issues may continue to affect the context and guidance provided to agents. Moreover, if skill smells are rarely corrected once introduced, they strengthen the case for automated quality-assurance tools and improved authoring practices. We therefore investigate how the prevalence of skill smells evolve throughout the lifetime of real-world skills.

RQ4: How does the prevalence of skill smells evolve over a skill’s lifetime?

Overall, our study contributes in the following ways:

• We are the first to characterize the anatomy of SKILL.md files beyond frontmatter and body, providing a taxonomy of semantic components present in real-world agent skills.

• We systematically identify best practices proposed for SKILL.md effectiveness and map each practice to the component(s) it governs.

• We introduce the concept of skill smells in SKILL.md files that could impair agent performance in the agentic paradigm and propose Skill Smell Detector (SSD) to automatically identify them.

• We empirically analyze how the prevalence of skill smells evolves over time and demonstrate that most skill smells have become increasingly prevalent across our dataset.

The remainder of this paper is structured as follows. In Section II, we review related research relevant to our study. Section III describes our methodology for addressing all research questions. Section IV presents the results of our experiments. Section V discusses the potential effects of skill smells on agents according to the literature. In Section VI, we outline potential threats to the validity and describe the steps taken to mitigate them. Finally, Section VII concludes the paper and suggests directions for future work.

2 Related Work↩︎

Automated Construction and Evolution of Agent Skills↩︎

Recent work has explored methods for automatically constructing and evolving agent skills. CoEvoSkills [19] enables agents to autonomously create and refine multi-file skill packages, while SkillClaw [20] continuously updates skills by mining agent trajectories and converting recurring behaviors into skill revisions. These studies focus on how skills can be generated and evolved automatically. In contrast, we focus on the quality of SKILL.md files, the best practices that govern them, and how quality deficiencies persist over time.

Empirical Studies of Agent Skills↩︎

Several studies have examined agent skills, characteristics, and usage. Ling et al. [21] conducted a large-scale analysis of 40,285 skills from skills.sh [4], reporting that SKILL.md files are typically compact enough to fit alongside planning context and tool schemas within an agent’s context window. They further found that SE was the most common application domain, accounting for 54.7% of the skills analyzed. Gao et al. [17] analyzed paragraph-level items from 90 SKILL.md files and classified them into five pre-defined categories: core rules, background, examples, templates, and redundant content to optimize skill compression. In contrast, our goal is not to categorize paragraphs for optimization, but to understand SKILL.md files as a software artifact. Hence, rather than relying on predefined categories, we inductively derive a taxonomy of semantic components from real-world skills as the foundation for identifying best practices and defining skill smells.

Effectiveness and Optimization of Agent Skills↩︎

Researchers have investigated the efficiency of agent skills. SkillsBench [8] evaluated agent skills across 86 tasks spanning 11 domains and found that curated skills can provide substantial, but highly variable, performance improvements. SWE-Skills-Bench [22] examined the effectiveness of skills in SE tasks and found that many skills provide limited benefit, while some can even degrade performance when their guidance conflicts with project-specific context. Other work has focused on optimizing skills after they are created. SkillReducer [17] improves token efficiency by reducing unnecessary skill content, while SkillMOO [23] formulates skill optimization as a multi-objective search problem balancing effectiveness and cost.

Collectively, prior work has studied how agent skills are used, generated, evolved, and optimized. However, little attention has been paid to understanding the semantic structure of SKILL.md files, identifying best practices for their authoring, characterizing quality deficiencies, and studying how such deficiencies persist over time. Our work addresses these gaps through a large-scale empirical study of SKILL.md content, best practices, and skill smells.

3 Methodology↩︎

We begin by describing the skills sample dataset used throughout our analyses, and then describe our methodology, as illustrated in Figure 1.

Dataset Curation. Following previous work in agent skills studies [21], [24], [25], we used publicly available skills from the skills.sh marketplace [4]. Specifically, we used a public dump of agent skills from this marketplace [26], which contains 133,149 skills from 8,808 publishers across 13,460 repositories. From this dataset, we applied a multi-step filtering and sampling procedure following mining best practices [27], based on language [11], adoption signals [11], [27], and repository diversity. Specifically, we first removed all non-English skills using fast–langdetect full model [28], [29] consistent with prior work [11], retaining 123,319 English skills and discarding 9,830 non-English ones. Second, to focus on skills with meaningful real-world adoption, we retained skills with at least 10 weekly downloads (58,311 skills), then further restricted the sample to the top 10th percentile by repository star count [27], corresponding to a minimum of 8,000 stars (5,410 skills). Third, we deduplicated skills by name, retaining the version with the highest star count, resulting in 4,061 unique skills. Next, since multiple skills can belong to the same repository, to avoid over-representing any single repository [30], we grouped skills by repository and owner and randomly sampled one skill per repository, producing a set of 241 skills [27]. We sampled one skill per repository because it represented the minimum number of skills present across all repositories; sampling a larger fixed number would have left some repositories underrepresented or excluded entirely, resulting in an unbalanced sample. Out of the 241 sampled skills, we were unable to retrieve 3 due to issues such as mismatched skill names, non-default branch hosting, or post-publication deletion. Our final sample therefore consists of 238 SKILL.md files.

3.1 RQ1: Semantic Components of SKILL.md Body↩︎

In order to identify the semantic components that construct the body of SKILL.md files, we conducted a qualitative analysis on the SKILL.md files. Following prior work that uses headings as the primary unit of analysis in Markdown files [31], we analyzed SKILL.md bodies at the H2 heading level. We chose this granularity because H1 conventionally captures the document title and only occurs once per Markdown document [32], making H2 the coarsest meaningful structural unit for content analysis.

We conducted a two-stage qualitative content analysis following established practices [33]. In the first stage, codebook development, two authors independently analyzed a randomly selected sample of 128 SKILL.md files from a population of 238 files (90% confidence level, 5% margin of error) at the H2-heading level to identify fine-grained semantic components. The authors then compared their codes and resolved disagreements through negotiated agreement [34], [35]. Components identified with different terminology were reconciled, and components proposed by a single author were discussed and retained or discarded by consensus. To address semantic overlap among the resulting components, the authors reorganized them into hierarchical categories: lower-level components, and a higher-level category that groups the lower-level components. This produced a hierarchical taxonomy of SKILL.md semantic components. In the second stage, both authors independently applied the finalized codebook to the remaining 110 SKILL.md files. The resulting taxonomy and component distribution are presented in Section IV.

3.2 RQ2: SKILL.md Best Practices & Skill Smells↩︎

To the best of our knowledge, no systematic literature review or substantial body of scientific work exists on best practices for SKILL.md authoring. Therefore, we conducted a Multivocal Literature Review (MLR) [36]. We followed an approach similar to prior work [37] by using Google Search best-practices-google? to retrieve the top 50 online sources containing the terms “agent skills” or “SKILL.md” in combination with “best practices.” Two of the authors independently manually reviewed all retrieved sources for relevance, discarding those that were not text-based, such as video tutorials, and those that contained the query keywords but did not discuss best practices for writing SKILL.md files, such as a blog post by AGENSI [38] and then through negotiated agreement [34], [35] finalized 29 sources.

Next, two authors independently extracted and consolidated candidate best practices from the selected sources using a card-sorting process [39]. Similar recommendations expressed using different wording were grouped together, and disagreements regarding grouping or interpretation were resolved through negotiated agreement [34], [35]. We observed cases where recommendations conflicted across sources. To resolve such conflicts, we applied majority voting among the sources that discussed the recommendation: a best practice was retained if it was supported by a strict majority and discarded in the event of a tie. For instance, if a best practice was discouraged in two sources but encouraged in three, it was retained as a best practice.

After compiling the final set of best practices, we derived skill smells by inverting each best practice, defining a skill smell as an anti-pattern in SKILL.md files that violates the corresponding best practice. Two authors performed this inversion jointly, discussing the precise wording and scope of each resulting smell to ensure it captured a clear and detectable violation of its corresponding best practice. For example, the best practice recommending that SKILL.md content remain high-level, with context-specific guidance delegated to reference documents, was inverted into the Undelegated Details skill smell, defined as containing low-level information and specific guidance that is not delegated to reference documents or scripts. This process resulted in defining 26 skill smells, which we present in Section ¿sec:results:rq2?.

3.3 RQ3: Automated Detection of Skill Smells↩︎

In RQ2, we introduced 26 skill smells corresponding to violations of SKILL.md authoring best practices. However, the extent to which each skill smell occurs in practice remains unknown. RQ3 therefore investigates their prevalence across our sample of 238 SKILL.md files. Since we are the first to introduce skill smells, we sought a scalable means of identifying them in practice. To this end, we developed an automated skill smell detection tool, Skill Smell Detector (SSD), that serves a dual purpose: measuring the prevalence of skill smells in our sample, and assisting skill authors in detecting smells as they write SKILL.md files, thereby mitigating the potential impact of skill smells on downstream tasks discussed in Section [discussion].

We automated the detection process, beginning with the construction of a ground truth dataset. To this end, we randomly sampled 53 SKILL.md files (confidence level 90%, margin of error 10%), and two authors independently reviewed each file for occurrences of any of the 26 skill smells. Following the same procedure as RQ1, the authors then held a negotiated agreement session to resolve disagreements, discussing mismatched cases and reaching consensus. This yielded a labeled ground truth dataset of skill smell occurrences across the 53 files. In Section ¿sec:results:rq3?, we present Table [table:skill-smell-freq] that presents the distribution of skill smells identified across the 53 manually labeled files (Validation Dataset).

Skill smells fall into two categories based on the type of reasoning required for their detection. The first category, which we term Statically Detectable smells, can be identified using rule-based scripts without any semantic reasoning. For example, the Oversized SKILL.md smell, defined as a SKILL.md file exceeding 5,000 words, can be verified with a simple word count script. Therefore, we developed python scripts to detect skill smells within this category. In total, five smells were Statically Detectable. The second category, which we term Semantically Detectable smells, cannot be identified through static analysis alone and instead require contextual reasoning over the content of the file. In total, there were 21 Semantically Detectable smells. We present the skill smells that fall into each of these two detection types in Table [table:skill-smell-freq].

Given the demonstrated strength of LLMs in simulating human judgment on software engineering artifacts [40], [41], we employed LLMs to detect Semantically Detectable skill smells. Since open-source LLMs have demonstrated competitive performance compared with proprietary models on software engineering tasks [42], [43], we opted for an open-source LLM to detect semantically detectable skill smells. Inference was performed on a Linux server equipped with an NVIDIA A6000 GPU with 48GB of VRAM. We selected Qwen3.6-27B, which has achieved superior performance across several reasoning benchmarks [44], as our base model. However, deploying it in full precision exceeded our hardware constraints. To address this, we leveraged Activation-aware Weight Quantization (AWQ) [45], which has been shown to introduce minimal degradation in model performance and has demonstrated strong results on software engineering tasks such as commit message generation [42]. Specifically, we used a 4-bit AWQ quantized variant of Qwen3.6-27B available on HuggingFace [46].

Prior work has shown that structured output formats such as JSON improve LLM-based classification [47], and that providing examples enhances LLM annotation performance on software engineering artifacts [41]. Drawing on these findings, we prompted the model in thinking mode to enable extended reasoning capabilities [44]. We chose the sampling parameters per the recommendation of the model card for “Thinking mode for general tasks” [48]. The prompt consisted of an introduction followed by a per-smell section for each of the 21 Semantically Detectable smells. The introduction informed the model about the nature of SKILL.md files, the concept of skill smells, and its role as a skill smell detector. For each skill smell, we provided its definition, a positive example showing a SKILL.md excerpt that exhibits the smell, and a matched negative example showing the same excerpt without the smell, to help the model distinguish between the presence and absence of each smell. We provide the prompt in our supplementary material [49]. We evaluated the effectiveness of our LLM-based semantic skill smell detector by measuring precision, recall, and F1-score on the manually curated ground truth dataset, with results reported in Section ¿sec:results:rq3?. Finally, by combining the static and semantic detectors, we built an end-to-end pipeline that takes a SKILL.md file as input and outputs the list of skill smells it contains. We then applied SSD to the remaining 238 SKILL.md files in our sample to measure the prevalence of each skill smell.

3.4 RQ4: How Skill Smell Prevalence Evolves Over Time↩︎

To understand how skill smells persist over a skill’s life-cycle, we mined the complete commit history of each of the 238 SKILL.md files in our sample, yielding 1,295 commit records. As the purpose of this RQ is to understand the persistence of skill smells, we excluded all skills with a single commit (did not change after the initial commit), leaving 142 SKILL.md files, consisted of 1,199 commit records. We treated each commit as a distinct version of the file and applied our automated Skill Smell detector (introduced earlier in Section 3.3) to every version to identify the skill smells it contained. For each file, we then ordered its versions chronologically and, for each skill smell category, calculated the version-by-version sequence of the smell’s presence or absence.

Then, we projected these version sequences onto a common weekly timeline to characterize how smell prevalence evolves over time. We defined the observation window as the interval spanning the earliest and latest commit dates across all 142 files (October 16, 2025 to June 13, 2026) and partitioned it into 35 consecutive, non-overlapping weeks. Each observation window contained the earliest commit for each file. A file is considered to exist from the week of its initial commit onward. For each week, we determined the state of every existing file using a last-observation-carried-forward scheme: a file’s smell profile in a given week is taken from its most recent commit on or before the end of that week. This reflects that a SKILL.md file retains the smells of its latest version until a subsequent commit modifies it, so that weeks without a new commit inherit the preceding state. Given these weekly file states, we computed, for each smell category and each week, its prevalence: the proportion of files existing in that week whose current version exhibited the smell.

4 Results↩︎

RQ1: Semantic Components of SKILL.md Body↩︎

We identified a hierarchical taxonomy comprising 13 higher-level and 44 lower-level semantic components from 224 SKILL.md files containing 1,615 H2-level sections. Table [table:body-semantic-component] presents the identified components, their definitions, and their frequency of occurrence. The three most common higher-level components are Task (74.4%), Introduction (63.5%), and References (57.1%). Task primarily captures procedural guidance, such as step-by-step instructions, subtasks, and error handling, enabling agents to accomplish the intended objective. Introduction provides contextual information, including skill summaries, prerequisites, and triggering conditions that help agents determine when and how to apply the skill. References complements these components by pointing agents to supporting resources, such as reference documents, utility scripts, commands, and domain knowledge. Together, these components form the core structure of most real-world SKILL.md files.

RQ2: SKILL.md Best Practices & Skill Smells↩︎

Our MLR of 29 online resources identified 26 best practices for authoring SKILL.md files. Mapping these best practices to the semantic components identified in RQ1 (Section ¿sec:results:rq1?) revealed that only 7 of the 13 higher-level semantic components are governed by explicit authoring guidance. The remaining six components Introduction, Evaluation, MCP Integration, Agent Architecture, Tools, and Other are not covered by any identified best practice. Additionally, we found that 8 of the 26 best practices apply to SKILL.md at a level above individual semantic components: 6 target the frontmatter, which follows a structured schema [6], and 2 apply to the SKILL.md body as a whole. For each component, Table [table:best-practices-semantic-components] reports the number of associated best practices and the total number of sources highlighting those best practices (summed across the component’s best practices). Together, these results show that existing guidance covers only part of the semantic structure of SKILL.md files, leaving several commonly occurring components without explicit authoring recommendations. The complete list of best practices and their supporting sources is available in the supplementary material [49].

Building on these 26 best practices, we derived a taxonomy of corresponding violations for SKILL.md files, which we term skill smells. Table [table:skill-smells] presents the resulting Skill Smells, their acronyms, definitions, and grouping into categories based on the nature of the underlying deficiency.

Figure 2 presents three representative examples of skill smells identified in SKILL.md files. We present the three most prevalent presence-based smells among the six identified presence-based skill smells. Absence-based smells (e.g., No Validation Step) are excluded from this illustrative analysis, as their defining characteristic (omission of expected information) renders them unsuitable for representation through discrete code excerpts. In our results we have 20 absence-based smells. The Series of Commands smell is shown in a body excerpt that prescribes a rigid, line-by-line sequence of shell commands, including hardcoded file paths and arguments, rather than describing the underlying objective and allowing the agent to determine the appropriate invocation. The Unclear Skill Name smell is shown in a frontmatter excerpt where the skill is named dig, a name that conveys no information about the skill’s purpose and could plausibly refer to many unrelated tasks. Finally, the Confusing Skill Description smell is shown in a description that states only that the skill runs a benchmark and analyzes performance, without indicating when the skill should be invoked or what kind of input it expects, leaving the agent unable to reliably determine its relevance to a given task. Due to space limitations, examples for the remaining skill smells can be found in the supplementary material [49].

RQ3: Prevalence of Skill Smells↩︎

Due to the large number of skill smells in our catalog, manually inspecting each SKILL.md file for the presence of every smell was infeasible. We therefore developed an automated approach to detect skill smells in a given SKILL.md file. We adopted a hybrid detection strategy. Smells that can be identified without semantic reasoning, such as those defined by the count of words or the presence of specific characters, are detected through static analysis. The smells that require contextual understanding, on the other hand, are detected using an LLM-based approach. The LLM-based detector frames each smell as a binary classification task, returning Yes when the smell is present and No otherwise. As justified in Section 3.3, we ran our tool with a 4-bit AWQ quantized variant of Qwen3.6-27B and conducted an evaluation on the manually labeled dataset (Table [table:skill-smell-freq]). Our evaluation reports a weighted F1 score of \(0.78\), a weighted precision score of \(0.79\), and a weighted recall score of \(0.78\), providing sufficient accuracy for large-scale prevalence analysis.

We next applied the detector to all 238 SKILL.md files to quantify skill smell prevalence. Specifically, we measured both the prevalence of each individual skill smell across the corpus and the number of co-occurring skill smells within each SKILL.md file.

Table [table:skill-smell-freq] summarizes the prevalence of individual skill smells. Overall, skill smells are highly pervasive: 11 of the 26 identified skill smells occur in more than 50% of the analyzed SKILL.md files. The most prevalent smell is Rationalization Loophole (RL), which appears in 94% of skills, whereas Lengthy Skill Name (LSN) and Lengthy Skill Description (LSD) are the least common. Figure 3 further shows that skill smells rarely occur in isolation. Only one of the 238 analyzed SKILL.md files are entirely free of skill smells, and skills contain an average of \(10.5\) skill smells each.

RQ4: How Skill Smell Prevalence Evolves Over Time↩︎

To examine whether skill smells are corrected or persist over time, we conducted a longitudinal analysis of their weekly prevalence using the methodology described in Section 3.4. For each week, we computed the proportion of existing skills exhibiting a given smell, where a skill’s state is taken from its most recent commit (last-observation carried forward).

We present the results into a smell-by-week matrix and rendered it as the heatmap in Figure 4, in which each row corresponds to a skill smell, each column to a week, and each cell is colored on a continuous scale from green (a prevalence of 0; no existing file exhibits the smell) to red (a prevalence of 1; every existing file exhibits it). Because SKILL.md files enter the corpus over time, the number of files underlying each weekly column is not constant, growing from a single file in the first week to the full set of 142 (Week 29 onwards). The specific number of SKILL.md included, and the prevalence of each skill smell per week, is available in the supplementary material [49].

Two smells that had zero prevalence across all weeks are omitted from the figure. Lengthy Skill Name never appears in our dataset. Lengthy Skill Description does appear (Table [table:skill-smell-freq]), but only in a single SKILL.md with one commit; per our RQ4 methodology (Section 3.4), we exclude all single-commit SKILL.md files.

Across all analyzed skill smells, we observe no evidence that their prevalence decreases over time. Instead, the prevalence of each skill smell tends to worsen as time passes.

We acknowledge that the full 35-week window does not apply uniformly to every skill smell, as 51% of the skills did not yet exist within the first \(18\) weeks. To account for this, we outline the final block of weeks with a blue rectangle on our figure, by which every SKILL.md has been included. Within this region, we observed that the prevalence of each smell remains stable; skill smells are neither addressed nor newly introduced, indicating that skill smells are seldom corrected once introduced and instead tend to persist throughout the development cycle.

To validate that this trend is not driven solely by aggregate statistics, we also examined the evolution of each repository individually. Due to space constraints, we could not include all 142 individual skill-level figures in the paper; instead, they are provided in our supplementary material [49]. Across these figures, we observed a similar pattern of once introduced, skill smells are seldom removed during subsequent development.

5 Discussion↩︎

Our results show that skill smells are pervasive in real-world SKILL.md files and persist throughout repository evolution rather than being corrected over time. The direct impact of individual skill smells on agent performance remains an open question. Since SKILL.md files become part of an agent’s context window [11], we therefore draw on empirical findings from the broader LLM and agent literature, arguing that evidence on analogous context-level deficiencies provides a reasonable proxy for understanding how skill smells may influence agent behavior, efficiency, robustness, and safety.

Authoring high-quality SKILL.md files are fundamentally a balancing act between three complementary design principles: providing the right amount of procedural guidance, enabling effective verification, and supplying high-quality context. First, Under-Specified Guidance and Over-Prescribed Guidance represent opposite ends of the same spectrum. Skills should provide sufficient procedural structure to help agents decompose tasks and make informed decisions, yet avoid prescribing execution so rigidly that agents lose the flexibility to adapt when circumstances change. Prior work similarly shows that explicit workflow decomposition and decision-support mechanisms improve agent performance, whereas excessive prompt specification reduces an LLM’s ability to follow instructions reliably [50]–[53].

Second, high-quality skills should support the execution process through planning, validation, feedback, progress tracking, and operational safeguards. These mechanisms enable agents to verify intermediate results, recognize infeasible tasks, and prevent errors from propagating to the final output. Prior work consistently demonstrates that planning, self-refinement, structured checklists, and runtime guardrails improve reliability, reduce unsafe behavior, and enhance task performance [54]–[59]. Missing Verification & Feedback Loops, Missing Follow-Through Guards, and Missing Safeguards represent skill smells in this category.

Finally, effective skills should provide context that is both sufficient and focused. Context Bloat and Inadequate Contextual Grounding again represent opposing failure modes: unnecessary information increases reasoning burden and competes for limited context-window capacity, while insufficient grounding forces agents to compensate through inference. Prior work shows that both irrelevant context and missing contextual information degrade reasoning quality, highlighting that effective SKILL.md files should provide the right information at the appropriate level of detail rather than simply more or less information [60]–[63]. Collectively, these findings suggest that many skill smells arise not because a skill contains too much or too little of a particular element, but because it fails to strike the balance required for reliable and efficient agent execution.

Skill smells can also have security implications. Security Hazard skill smells expose agents to prompt-injection attacks by embedding content that becomes part of the agent’s context. Prior work shows that LLMs are susceptible to prompt injection through untrusted retrieved content [64], making structured tags in SKILL.md files a potential attack vector. These smells, therefore, increase the attack surface of Agent Skills and may compromise the security of agentic systems.

Seemingly superficial skill smells, such as those in the Convention & Style Violation and Unstructured Output categories, primarily affect naming conventions, metadata, or output templates, yet can substantially affect agent behavior. Tool names, descriptions, and parameter schemas are among the primary signals LLMs use when selecting tools [65], while explicit output templates improve adherence to user-defined output requirements [66]. Consequently, inconsistent metadata or missing output templates can hinder skill discovery, reduce correct skill invocation, and produce inconsistent outputs. These findings suggest that authoring conventions are more than stylistic guidelines; they directly shape how agents discover, invoke, and interact with skills.

Our findings have implications for researchers, tool builders, and practitioners. We introduce skill smells as a new software quality concern for agentic systems, opening several research directions. Future work should quantify the causal impact of individual skill smells on agent performance, investigate interactions among multiple smells, evaluate their effects across different foundation models and agent architectures, and study how skill smells influence increasingly complex multi-agent workflows.

Our findings also suggest that the high prevalence and persistence of skill smells make manual quality control unlikely to be sufficient. Our taxonomy provides a foundation for automated quality-assurance tools for Agent Skills. Similar to linters and static analyzers for source code, future tools could utilize the results of SSD to recommend refactorings, automatically repair common issues, and be integrated into IDEs, CI pipelines, and Agent Skill marketplaces.

Finally, developers should treat SKILL.md files as executable software artifacts rather than passive documentation. As Agent Skills become increasingly shared across organizations and marketplaces, establishing and adhering to high-quality authoring practices will be essential to building trustworthy and maintainable agent ecosystems.

Taken together, our findings suggest that skill smells should not be viewed as isolated documentation issues, but as an emerging software quality concern for agentic systems. This combination of high prevalence, long-term persistence, and plausible downstream consequences suggests that the community has an opportunity to intervene while Agent Skills are still in their early stages of adoption. Software engineering has repeatedly witnessed similar trajectories with software artifacts such as source code, tests, and security configurations, in which quality concerns were initially underestimated, prompting decades of research on code smells, test smells, security smells, automated detection, and refactoring techniques. Our findings suggest that Agent Skills present a similar opportunity for early intervention before poor authoring practices become deeply embedded in the ecosystem.

6 Threats to Validity↩︎

Construct Validity. One threat concerns the subjectivity of manually identifying semantic components and labeling skill smells. We mitigated this through using independent coding by two authors, followed by negotiated agreement to resolve all disagreements [34], [35]. Another unavoidable threat is that our best practices are derived from grey literature rather than peer-reviewed guidelines. To reduce potential bias, we independently reviewed the top 50 retrieved sources and retained only practices supported by a strict majority of the selected sources. A further threat concerns the reliability of our LLM-based detector for semantically detectable skill smells, which we mitigated by validating it on a manually curated ground-truth dataset.

External Validity. A key threat to external validity is that our sample was drawn exclusively from the skills.sh marketplace [4]. Furthermore, restricting the sample to popular English-language skills may limit generalizability and could underestimate the prevalence of skill smells in the broader ecosystem. Another threat concerns our use of a single LLM, Qwen3.6-27B (4-bit AWQ), for detecting semantically detectable skill smells; prevalence estimates may differ with other models or quantization schemes.

7 Conclusion↩︎

Our study presents the first systematic characterization of SKILL.md files as a software artifact. We identify a taxonomy of 13 higher-level and 44 lower-level semantic components, derive 26 best practices, and introduce skill smells as violations of these practices. Our empirical study shows that skill smells are highly prevalent and become more common as skills evolve over time. These findings establish SKILL.md quality as an important SE concern and motivate automated support for refactoring skill smells during skill authoring. All code, data, and replication materials are available in the supplementary material [49].

References↩︎