1 Introduction↩︎

The increasing adoption of Artificial Intelligence (AI) tools in software engineering has contributed to the emergence of Software Engineering 3.0 (SE 3.0), a paradigm characterized by AI systems that can perform a range of software development tasks with limited human intervention[1], [2]. Systems such as GitHub Copilot[3], OpenAI Codex[4], Claude Code[5], Cursor[6], and Devin[7] can execute multi-step development workflows, including task planning, code generation, testing, and the creation of Pull Requests (PRs) for integration into existing codebases. Unfortunately, this acceleration often compromises system trustworthiness and code quality[8]: the scale of agent-generated PRs outpaces human review capacity, leaving maintainers fatigued and over-reliant on automated outputs[9], [10].

The mismatch between code production velocity and reviewer capacity has demonstrably compromised repository-level security integrity; nascent empirical analyses, albeit derived from limited datasets, suggest that AI-generated code exhibits a vulnerability introduction rate approximately 2.7 times that observed in human-written code[11], [12]. However, current software engineering literature predominantly evaluates Large Language Model (LLM) code generation with benchmarks focused on functional correctness rather than the security posture of the specific files and paths agents modify[13], [14]. Although emerging empirical studies have begun to characterize agentic pull requests by analyzing their general modification patterns[15], test generation behaviors[16], [17], and commit quality[18], there remains a critical research gap with respect to the introduction of structural security flaws within the specific code paths that agents touch. Conducting a large-scale, scoped empirical study on real-world data is imperative to document and understand the distribution, types, and severity of security code smells introduced by the use of autonomous agents.

To address this gap, we present an empirical study leveraging the AIDev dataset: a large-scale repository of PRs generated by autonomous coding agents to systematically characterize their security implications. In this paper, we focus on ‘security smells’ (structural patterns that indicate potential security risks), rather than confirmed exploitable vulnerabilities, which require context-specific exploitability analysis. We utilize a combination of LLM-as-a-judge evaluation, and manual qualitative analysis to investigate the following Research Questions (RQs):

RQ1: Which security smells appear most in agentic PRs, and how do they distribute across categories and severity?

RQ2: How many flagged hard-coded secrets are genuine, who introduced them, and are they identified during review before integration?

This paper makes the following core contributions:

• A six-category taxonomy of security smells in agent-generated PRs, grounded in OWASP secure-coding guidance, the Center for Internet Security Benchmarks, and GitHub hardening documentation.

• An LLM-as-a-judge detection pipeline that categorizes security smells in agentic diffs with quantized open-source models, validated against a human-annotated gold standard using precision, recall, F1, and Cohen’s kappa.

• A large-scale empirical characterization of the prevalence, category distribution, and severity of security smells across agentic PRs in the AIDev dataset[1].

• A qualitative analysis of the most critical smells that traces each genuine credential to its committer, the agent or the human collaborator, and assesses whether automated bots and human reviewers intercept it before integration.

2 Methodology↩︎

We use AIDev, a public dataset of PRs from five coding agents on GitHub. We draw on three tables: pull_request holds per-request metadata, pr_commit_details holds one row per changed file per commit with the unified-diff patch, and pr_review_comments holds inline review comments. Our methodology is shown in Figure 1.

2.1 Detection and Quantification of Security Smells↩︎

Corpus construction. We keep files whose path matches a high-risk pattern set covering Continuous Integration (CI) definitions, container files, infrastructure-as-code, secret-bearing files, configuration files, shell scripts, and computational notebooks. We exclude lock files, minified assets, vendor directories, and binary formats. We assign each retained file to the file categories to support stratified analysis. We then reduce each patch to the lines added, retaining only + lines and discarding diff metadata and removed lines, so that every counted smell reflects introduced content. The corpus contains 16,370 file changes across 4,064 PRs. Because we filter for high-risk patterns and analyze only added lines, this work constitutes a scoped security-smell analysis rather than a comprehensive repository audit.

Taxonomy. We define six smell categories, grounded in the OWASP guidance, the Center for Internet Security Benchmarks, and the GitHub hardening documentation[19], shown in Table 1.

Detection. We label the corpus with a large language model acting as a judge. We prompt two open-source models, Qwen3.6-35B-A3B-FP8[20] and Gemma-4-26B-A4B-IT-FP8[21], with zero-shot prompting, at FP8 quantization through a local end-point at sampling temperature 0.1 (see Appendix 7.1). The judge performs independent categorization over the whole corpus: for each file change, it receives the patch, file path, language, and PR title. The judge marks the file clean or flags each smell with one of the six categories, a line number and snippet, a severity on a three-level scale, and a one-sentence rationale.

Reliability and Validation. We merge the runs of the models as a union, marking a file dirty when at least one run flags it, and deduplicating smells on category, line number, and line snippet. We report inter-rater agreement with Cohen’s Kappa and validate the judge against a random sample of 376 file changes with 95% Confidence Level and 5% Margin of Error. Two authors independently assign a smell label to each file change, and a third author adjudicates disagreements to a gold label matched to the judge. We report precision, recall, the F1 score, and Cohen’s kappa against the gold label, together with inter-annotator agreement.

Stratified Analysis. To examine which factors drive smell prevalence, we disaggregate the flagging results along four dimensions: agent type, repository primary language, file category and PR size. We take the agent and the repository identifier from the pull request table and the primary language from the repository table. We assign the file category during corpus construction. We compute the PR size as the sum of changed lines over the distinct commits of a PR and bin it into five tiers: XS (1-9), S (10-49), M (50-199), L (200-999), and XL (1000+). We report a file-level flagging rate as the share of files marked dirty, a PR-level flagging rate as the share of PRs with at least one dirty file, and the mean smells per file.

2.2 Validating and Reviewing Hard-Coded Secrets↩︎

A hard-coded secret is the highest-impact security smell, because a live credential committed to a public repository is exploitable the moment it is pushed. RQ2 examines the secrets_identity category in depth. We determine whether the secret is genuine, who committed it, and whether the existing reviewers catch it or not. We pay particular attention to automated security bots, since they are the layer expected to intercept a leaked credential before it causes harm. We extract the complete set of smells that the categorization judge flags under secrets_identity and manually examine each one. We carry no judge label into this stage so that the coders judge each secret on its own evidence. Two authors independently code each secret in two dimensions. Validity records whether the flagged value is a genuine credential rather than a placeholder, an example, or a misread. Reviewer detection records whether an automated security bot commented on the secret. The coders inspect the live repository, the pull-request discussion, and the commit history that follows the introducing commit.

3 Results↩︎

To support reproducibility, we provide a replication package containing the data, scripts, and documentation required to reproduce the analyses reported in this paper. The replication package is available in the following link[22].

3.1 RQ1: Distribution of Security Smells↩︎

The judge returned valid labels for 16,112 of the 16,370 file changes, which span 4,022 PRs. The judge failed to return parseable output for 258 file changes. The judge flagged 3,807 file changes as containing at least one smell, a rate of 23.6%, and recorded 8,701 distinct smells after deduplication. At the PR level, 1,563 of 4,022 requests contained at least one smell, a rate of 38.9%, and the remaining 2,459 requests stayed clean. The six categories differed by an order of magnitude in frequency, as Table 2 reports. As shown in Figure 2, the supply_chain_integrity category alone accounted for 7,160 of the 8,701 smells, a share of 82.3%, followed by over_privilege_execution at 9.6% and secrets_identity at 3.4%. The severity distribution placed 6,292 smells at the major level (72.3%), 2,156 at the minor level (24.7%), and 253 at the critical level (3.0%). Critical smells concentrated in one category: 252 of the 253 critical smells, a share of 99.6%, fell under secrets_identity. The judge thus reserved its highest severity almost entirely for hard-coded credentials. The human validation set comprised 376 file changes where the two coders manually labeled each item and reached a Cohen’s kappa of 0.929. The judge reached a precision of 0.908, a recall of 0.775, an F1 score of 0.836, and a Cohen’s kappa of 0.789 against the gold labels. The judge therefore identified smells at high precision while missing 22.5% of smells.

To identify the factors that drive smell prevalence, we disaggregated the PR-level flagging rate by agent, repository primary language, file category and PR size. Figure 4 compares the flagging rate against the 38.9% corpus average by agent and language. Copilot PRs carried a smell most often at 45.5%, and OpenAI Codex PRs least often at 34.9%, yet Codex’s flagged files carried the most smells per file at 0.62. JavaScript repositories showed the highest prevalence at 55.3% and Python the lowest among the major languages at 31.8%, a spread of 23.5 percentage points. TypeScript sat at the corpus average of 38.9%.

Table 3 disaggregates the file-level rate by file category. GitHub Actions workflows and Docker files together held 87.6% of all smells, consistent with the concentration of supply_chain_integrity smells. The application configuration files remained clean at a 2.2% dirty rate. Figure 5 groups the PRs by the total lines changed and shows the flagging rate increasing with the size of the PR, from 16.2% for the smallest PRs (1-9 lines) to 53.6% for the largest (1000+ lines), a spread of 37.4 percentage points.

3.2 RQ2: Genuine Secrets and Review Outcomes↩︎

The coders assigned a validity label to 272 of the 294 secret smells. The remaining 22 could not be labeled because the PRs no longer exist. The coders confirmed 74 genuine credentials, a rate of 27.2% of the labeled set. The commit author of each genuine secret fell into one of two classes. A human author committed 50 of the 74 genuine secrets, a rate of 67.6%, and an AI agent committed the remaining 24. The human collaborator therefore introduces genuine secrets into agent pull requests more often than the agent does.

A security bot or a human reviewer commented on the secret in 14 of the 74 genuine cases, a rate of 18.9%, and the reviewers removed the other 60 secrets without a comment. The commenting actors comprised seven distinct tools and reviewers: GitHub Advanced Security, GitGuardian, Qodo, Copilot, Greptile, Gemini Code Assist, and Cursor bot. A summary of these results is depicted in Figure 3.

4 Threats to Validity↩︎

The study relies on a manually annotated gold standard of 376 file changes, making results susceptible to annotation bias and human error. Similar risks apply to the manual validation of hard-coded secrets in repositories and commit histories. To reduce these threats, we used independent double-blind coding and third-author adjudication.

The use of quantized LLMs (Qwen3.6 and Gemma-4) introduces limitations such as non-determinism and incomplete detection. With a recall of 0.775, the LLM judge missed 22.5% of security smells, meaning reported frequencies likely underestimate the true prevalence of security smells.

By intentionally restricting the analyzed corpus to high-risk file paths and focusing exclusively on added lines, the findings characterize specific security smells introduced within targeted boundaries rather than providing a comprehensive audit of total repository security health. Security smells or misconfigurations present in excluded files or introduced through complex interactions across multiple existing files are not captured in this analysis.

The findings are based on the AIDev dataset, which focuses on selected coding agents in open-source GitHub repositories. As a result, the observed security smells and review patterns may not generalize to other agents, enterprise environments, or development platforms.

5 Related Work↩︎

The advent of autonomous coding agents has prompted extensive empirical investigation into the differences between AI-generated and human-authored PRs. Ogenrwot[15] found substantial differences in commit counts, merge outcomes, lifecycle dynamics, and description-to-diff alignment compared to human baselines. Yoshioka et al.[23] identified that while submitter attributes dominate merge success in both groups, review-related features exert contrasting effects on agentic PRs. This builds upon Watanabe et al.[24], who established the distinct architectural footprint of AI contributors. Rahman et al.[18] conducted task-level evaluations demonstrating that while certain models achieve high PR acceptance rates, others trigger significantly higher volumes of human and automated review discussions. Haque et al.[16] and Yoshimoto et al.[17] observed that agent-generated test methods present distinct structural patterns. The human-AI interaction layer is equally critical; Haider et al.[25] taxonomized reviewer responses to AI-authored code, revealing that inline comments predominantly address logical and functional correctness rather than security posture.

While AI applications provide a foundation for open-source software sustainability[26], system trustworthiness remains an underexplored bottleneck[27]. Prior security literature on PRs has primarily focused on human-authored vulnerabilities or automated patching mechanisms utilizing datasets like TREEVUL[28] or architectures like GraphSPD[29]. Wang et al.[30] recently addressed AI-specific security dynamics by examining automated versus human security patching patterns within the AIDev dataset. However, while these studies analyze patching efficacy, functional correctness, or general modification metrics, our work distinctly isolates the introduction of structural security smells (e.g., hard-coded secrets, supply chain misconfigurations) during autonomous generation.

6 Conclusion and Future Work↩︎

This study provides a large-scale empirical analysis of security code smells introduced within autonomous agent-generated pull requests. By analyzing the AIDev dataset using a combination of validated LLM-as-a-judge methodology and manual qualitative analysis, we established that agentic coding workflows carry significant security debt. Code smells were present in 38.9% of agent-authored PRs, with supply chain integrity misconfigurations constituting the vast majority (82.3%) of identified security smells. Furthermore, our investigation into critical-severity smells revealed that 99.6% are hard-coded credentials. Crucially, our findings show that human collaborators were mostly responsible, injecting 67.6% of genuine leaked secrets within agentic PRs. Compounding this risk, the current safety net of automated security bots and human reviewers failed to intercept 81.1% of these leaked credentials prior to integration.

These results reveal a critical operational insight: the velocity of agentic code generation outpaces and compromises traditional human-in-the-loop review pipelines. The high rate of human-introduced secrets suggests potential review fatigue or cognitive offloading, though further behavioral studies are required to determine why developers bypass standard security hygiene when interacting with autonomous workflows.

Future research must focus on mitigating developer cognitive load by designing context-aware security interfaces that evaluate agentic workflows without exacerbating review fatigue. Investigations should also explore novel mechanisms for actively enforcing security hygiene at the point of human-AI collaboration, specifically targeting preventative measures against the initial injection of live credentials. Finally, expanding this empirical analysis to enterprise environments is necessary to determine if proprietary guardrails alter security smell distributions or better sustain developer vigilance.

7 Appendix↩︎

7.1 LLM Setup Information↩︎

A sample of the prompt provided to the LLM-as-a-judge system is shown in Figure 6. A sample response sent to LLM is shown in Figure 7.

The LLMs were hosted locally using VLLM backend with the command shown in Figure 8. An NVIDIA RTX PRO 6000 Blackwell Workstation Edition Graphics Card with 96GB VRAM was used to host the LLMs.

7.2 Sample Secrets Found During Manual Verification↩︎

Figures 9 and 10 show two sample secrets found while manually verifying the results produced by the LLM-as-a-judge system.

References↩︎