1 Introduction↩︎

A developer who needs to implement a login flow, hash a password, or set up an OAuth exchange has, for over a decade, most often consulted a human answer on a site such as Stack Overflow. That practice has a measurable security cost. Copying from Stack Overflow has been shown to move insecure patterns into shipped applications [@fischer2017stackoverflow], and the information source a developer consults measurably changes the security of the code they write [@acar2016you]. The practice is now being displaced by large language models, which developers query in the editor rather than the browser, and their output carries its own security cost. Language-model code assistants produce insecure code at a meaningful rate [@pearce2022asleep], and controlled user studies find that developers with an assistant write less secure code while believing the opposite [@perry2023users; @sandoval2023lost].

This paper studies the two sources side by side. We treat the question as one of provenance and pattern rather than of overall quality. Two questions drive the work.

RQ1, provenance. Given only a code snippet for a common task, can we tell whether it was written by a human on Stack Overflow or generated by a language model, and can we tell which model. Code stylometry has long shown that human authors leave recoverable fingerprints [@caliskan2015deanonymizing; @abuhamad2018large]. We ask whether the human-versus-machine boundary, and the boundary between models, is just as recoverable from lightweight, interpretable features.

RQ2, security divergence. For the same task, do the two sources adopt the same security patterns. Does a model reach for PKCE, a bound query, or a strong password hash more or less often than the human corpus, and does it carry the human corpus’s insecure habits forward or leave them behind.

We answer both with open sources only, which is what makes the study an exercise in open-source intelligence. The human side is the public Stack Overflow API. The model side is a local gateway that exposes open-weight models through an OpenAI-compatible interface, from which we use 9 models spanning several families and sizes. For 31 security-sensitive tasks we collect real implementations from both sides, 528 samples in total, and score each with two families of deterministic detectors, per-task security-pattern checks and language-agnostic style metrics. Nothing is simulated. A task with no sample from a given model simply has no sample, and every generation carries the model, latency, and token counts that produced it.

Our contributions are the following.

• A reproducible, open-source pipeline for cross-provenance code analysis. Every task is a single specification entry carrying its prompt, its Stack Overflow query, and its security-pattern detectors, so the study generalises to any subject without new code.

• A provenance-attribution result. A cross-validated classifier separates human from model code at 93 percent against a 78 percent baseline, and attributes a model-written sample to its specific model of origin at 48 percent in a 7-way task.

• A security-divergence analysis. We report, per task and per pattern, how often the human corpus and the models adopt each secure and insecure practice, and which insecure human patterns the models reproduce.

• A released dataset and a fail-closed numeric checker that re-derives every figure in this paper from the stored samples.

2 Background and Related Work↩︎

Insecure code from human sources. A line of security research has measured the cost of the copy-and-paste development style. Stack Overflow contains both secure and insecure answers, and the insecure ones propagate into real software [@fischer2017stackoverflow]. The information source a developer is steered to, official documentation versus a question-and-answer site, changes the security of the resulting code [@acar2016you]. Our human corpus is drawn from the same source these studies examined, and we treat the age of an answer as a first-order confound, since a highly-voted answer written years ago predates the defaults that are now considered secure.

Insecure code from language models. As models trained on code became capable of solving programming tasks [@chen2021evaluating], the same security questions were asked of them. An assessment of GitHub Copilot found a substantial fraction of security-relevant completions were vulnerable [@pearce2022asleep]. User studies then showed the effect on developers, who write less secure code with an assistant and misjudge its safety [@perry2023users; @sandoval2023lost]. These studies mostly evaluate one commercial assistant. We instead compare many open-weight models to each other and to the human baseline on identical tasks, and we organise the tasks around recurring web-application weakness classes in the spirit of the OWASP Top Ten [@owasptop10].

Code authorship and provenance. Code stylometry recovers the human author of a program from stylistic features [@caliskan2015deanonymizing], and the approach scales to thousands of authors and across languages [@abuhamad2018large]. The naturalness of software, its predictability under a language model [@hindle2012naturalness], is the property such attribution exploits. Recent work extends stylometry directly to the machine-authorship question, recognising AI-written programs with multilingual code stylometry [@gurioli2024isthisyou]. Our provenance result is close in spirit, and we add two elements that work does not address, a per-model attribution rather than a binary human-or-machine label, and a security-divergence analysis on the same samples. We apply the same idea to a coarser but timely boundary, human versus model and model versus model, using interpretable features rather than a deep model, so that the attribution result comes with an explanation.

Naturalness and machine-generated code detection. The property that makes code authorship recoverable is that code is repetitive and predictable, its naturalness under a statistical language model [@hindle2012naturalness]. Human authors and, as we find, individual models occupy distinguishable regions of that predictable space. A growing body of work asks the specific question of whether machine-generated code can be told apart from human code, motivated by plagiarism, academic integrity, and supply-chain provenance. We contribute a lightweight, interpretable point on that spectrum, using a handful of surface features rather than a learned detector, and we pair the detection question with a security question that detection work usually leaves aside.

Position of this work. We connect the two literatures. The security studies ask whether a source is safe, the stylometry studies ask who wrote a piece of code. We ask both questions about the same samples, so that a security difference between sources can be read alongside the features that make the sources distinguishable in the first place. Two further choices set the work apart from its neighbours. We compare many open-weight models rather than a single commercial assistant, which lets us separate what is common to models from what is specific to one, and we run the same tasks in several languages, which lets us separate what is a property of the source from what is a property of a language.

3 Threat Model and Motivation↩︎

The provenance question is not academic. Two settings make it concrete. First, in review and audit, a team that knows a change was machine-generated can route it to the checks that machine code most often fails, and the security-divergence profile of §8 tells the reviewer which checks those are. Second, in supply-chain and plagiarism analysis, the ability to attribute a snippet to a source, or to a specific model, is an open-source-intelligence capability that operates on the artifact alone, without access to the author’s environment.

We assume the analyst has only the code, no metadata, no editor telemetry, and no account information. This is the open-source-intelligence setting. The material that feeds the study is likewise all public, the Stack Overflow API and a gateway of open-weight models, so the entire result is reproducible by a third party with no private access.

4 Data Collection↩︎

Subjects. A subject is one security-sensitive programming task. Each subject is a single specification entry that carries a natural-language prompt, a Stack Overflow search query, and a list of pattern detectors. Table 1 lists the 31 subjects, which span authentication and authorization (OAuth with PKCE, JWT verification, session cookies), data protection (password hashing, parameterized database access), input handling (file upload), transport policy (CORS), and a front-end design task (a responsive landing page). The design is deliberately data driven. Adding a subject is adding one entry, and the whole pipeline, generation, retrieval, extraction, and analysis, runs off the specification with no new code.

Human corpus. For each subject we query the public Stack Overflow API with the subject’s search string, retrieve the answers to the most relevant answered questions, and keep the largest code block from each positively-scored answer. We record the answer identifier, its score, and its creation year, so the corpus is auditable and re-fetchable and so the paper can report the age of the human material. We keep only answers whose code block is substantive.

Model corpus. The model side is a local gateway exposing open-weight models through an OpenAI-compatible interface. We first probe every model the gateway advertises with a trivial request and record whether it answers, is rate limited, or is unavailable, so that only models that genuinely respond enter the study. From the responding set we select 9 models chosen for diversity of family and size, listed in Table 2. For each model and subject we send the subject’s prompt verbatim, with a fixed system instruction asking for a single complete code solution, and store the returned code with the model, the resolved model name, the latency, and the token counts. Generation is repeated so that each model contributes more than one sample per subject, and the whole run is resumable and paced to respect the gateway’s rate limits. No output is edited or synthesised. The gateway throttles models independently and unpredictably, so coverage is filled by repeated resumable passes rather than forced, and a missing model-subject cell is reported as missing.

Pipeline. Listing [lst:pipeline] states the collection and analysis pipeline in full. It is a deterministic sequence of small steps, each writing an artifact to disk, and the fail-closed checker of Appendix 22 re-derives every reported number from those artifacts. Adding a subject is a data change, a single specification entry rather than a code change, so the study extends to a new task without modifying the pipeline.

5 Feature Extraction↩︎

Every sample, human or model, is reduced to two families of features by deterministic functions of the stored code, so that the reduction is reproducible and interpretable.

Security-pattern checks. Each subject carries a list of checks, and each check is a regular expression tagged as secure or insecure. A check fires when its pattern is present in the code. For OAuth with PKCE, for example, the secure checks include the presence of a code challenge and verifier following PKCE [@rfc7636], an anti-forgery state parameter, and validation of the redirect target, while an insecure check fires on a hardcoded client secret. For password hashing, a secure check fires on a strong key derivation function such as bcrypt, scrypt, argon2, or pbkdf2, and an insecure check fires on a bare md5 or sha1. From the checks we derive, for each sample, the fraction of secure patterns adopted, the fraction of insecure patterns present, and a single security score, \[\text{sec} = \frac{\#\text{secure adopted}}{\#\text{secure defined}} - \frac{\#\text{insecure present}}{\#\text{insecure defined}},\] which lies in \([-1, 1]\) and is higher for code that adopts the secure practices and avoids the insecure ones.

Style metrics. Independently of the task, we compute language-agnostic style metrics that capture the shape of the code, its length in characters and lines, the comment density, the number of import statements, the blank-line ratio, the average line length, and indicators for the presence of functions and of error handling. These are the features that code stylometry uses, in a lightweight and interpretable form, and they carry most of the provenance signal of §7.

6 Dataset↩︎

The collected corpus contains 528 real code samples, 117 from Stack Overflow and 411 generated by 9 open-weight models, spread across 31 subjects. The subjects reduce to 24 task families expressed across 5 languages, namely Python, JavaScript, Go, and a markup task in HTML, which is what makes the cross-language analysis of §12 possible. Table 1 lists the subjects with the number of security and style checks each carries and the count of human and model samples retained. Table 2 lists the models with their family, approximate size, and sample count. The model set spans several families and a wide size range, from small general models to very large mixture-of-experts systems, and includes both code-specialised and general models. Because the gateway throttles each backend independently, coverage is not uniform, and we report the true per-model counts rather than forcing an equal cell count.

7 Provenance Attribution↩︎

Human versus model. We first ask whether a sample’s provenance can be recovered from its features alone. We train a random forest [@scikit] on the style metrics of §5 together with the two aggregate security fractions, and evaluate it with stratified five-fold cross-validation against a majority-class baseline. The classifier separates human from model code with 93 percent accuracy against a 78 percent baseline, at an F1 of 0.955 for the model class. Provenance is therefore highly recoverable from lightweight, interpretable features, without any learned representation of the code.

Figure 2 reports the permutation importance of each feature for this task. The signal is dominated by the shape of the code rather than its security content. The strongest discriminators are the raw size of the sample, the presence of functions, and the number of imports, which reflects a consistent difference in how much scaffolding each source produces for the same request. The security fractions contribute, but they are secondary to style, so the human-versus-model boundary is primarily stylistic rather than security-based.

Which model. We then restrict to the model samples and ask a harder question, which specific model wrote a given sample. A 7-way random forest over the same features reaches 48 percent accuracy against a 17 percent majority baseline. Individual models therefore leave a recoverable fingerprint in their code, well above chance, though the boundary between models is far less sharp than the boundary between human and machine. Model attribution from code alone is a real but bounded capability, consistent with the view that models within a shared training regime converge on overlapping styles.

Figure 3 shows the row-normalised confusion matrix of the model classifier. The structure is informative. Models from the same family are confused with each other more than with models from other families, and the models that produce the most elaborate scaffolding are the most cleanly separated. The confusions follow family and size rather than being uniform noise, which is what a genuine per-model fingerprint predicts.

8 Security Divergence↩︎

The second question is whether the two sources adopt the same security patterns for the same task. Table 3 reports, for each detector, the adoption rate in the human corpus and in the model corpus, aggregated across subjects and weighted by sample count. Figure 4 plots the same contrast.

Figure 5 gives a per-family, per-pattern view of the same contrast as the signed difference in adoption between the models and the human corpus, so the blue cells are patterns the models adopt more and the red cells are patterns the human corpus adopts more. The figure makes the structure of the divergence visible at a glance, a broad band of blue on the defensive patterns and a small number of red cells on the patterns where the human corpus leads.

Models adopt defensive patterns more often. The clearest result is that the model corpus adopts the secure patterns at a substantially higher rate than the human corpus. The aggregate security score, the fraction of secure patterns adopted minus the fraction of insecure patterns present, is 0.40 for the models against 0.12 for the human corpus. The gap is largest on exactly the patterns that modern secure-coding guidance emphasises. The models reach for proof-key-for-code exchange in the OAuth task, a strong key-derivation function for password storage, a parameterized query for database access, signature verification and algorithm pinning and expiry checking for tokens, and the secure, http-only, and same-site flags for session cookies, all at markedly higher rates than the human answers, which frequently omit them. This is consistent with the models having absorbed the more recent consensus on these tasks, while the highly-voted human answers include material old enough to predate it.

Where the human corpus carries insecure legacy patterns. The same table shows the human corpus retaining insecure habits that the models have largely shed. Human answers concatenate user input into SQL, use bare or fast hash functions for passwords, and configure a wildcard cross-origin policy more often than the models do. These are the classic copy-and-paste hazards, and their lower rate in model code is the mirror image of the previous finding.

The exception, hardcoded secrets. The models are not uniformly safer. On one family of patterns they are worse. Model code introduces a hardcoded application secret key and a hardcoded client secret more often than the human corpus, typically as an inline placeholder value in an otherwise complete program. This is a real and recurring hazard. A placeholder secret that ships unchanged is a live vulnerability, and it is a pattern the human answers, which more often leave configuration to the reader, produce less. The security picture is therefore not that models are safe and humans are not, but that the two sources fail in different places, the human corpus on legacy defensive omissions and the models on inlined secrets and completeness that invites unedited reuse.

Security score by origin. Figure 6 breaks the security score down by origin, with the human corpus alongside each model. The human corpus sits at the bottom, and the models spread above it, so the aggregate gap is not driven by a single model but is a property of the model corpus as a whole. The spread among models is itself informative, and §8 discusses which models sit highest.

9 Style Divergence↩︎

The provenance result of §7 is driven by style, and Table 4 makes the style contrast concrete. For the same task, model code is far larger than the human answer, 1801 characters on average against 552, roughly a threefold difference. The models wrap their solution in functions, add error handling, and import more supporting modules, where the human answers are terser fragments that assume a surrounding context. Error handling is present in 31 percent of model samples against 8 percent of human samples. Human answers use longer individual lines, a density that comes from writing a minimal snippet rather than a complete program. These differences are the substance of the attribution result, and they also explain the completeness hazard of §8, since a longer, self-contained, runnable program is exactly the kind of artifact a developer is tempted to paste and ship with its placeholder secret intact.

Figure 8 normalises each style feature across the two sources for a direct visual comparison, and shows the same pattern, models higher on size, structure, and error handling, humans higher on line density. Figure 7 shows the distribution of code length behind the mean, and the two distributions barely overlap, which is why length alone is a strong provenance signal.

10 Per-Subject Analysis↩︎

The aggregate result of §8 hides real per-subject structure. The models are not uniformly safer than the human corpus. On some tasks they are far safer, on others they are worse, and the reasons differ by task. Figure 9 gives the security score of each source on each subject, and this section walks through the subjects that carry the clearest lesson, showing a real human answer and a real model generation for each.

10.1 Password Hashing↩︎

Password hashing is the subject with the sharpest divide. The human corpus scores at the floor, and the models near the ceiling. The reason is visible in the code. The representative human answer, Listing [lst:password_hash_h], reaches for a fast, general-purpose hash, the exact pattern that decades of guidance warn against for passwords, and it is a highly-voted answer precisely because it is old. The representative model answer, Listing [lst:password_hash_l], uses a purpose-built password key-derivation function with a per-user salt and a constant-time verify. This is the cleanest case of the models carrying the current consensus while the human archive preserves the practice that consensus replaced.

10.2 Token Verification↩︎

The JWT subject repeats the pattern. The human answers frequently decode a token without pinning the accepted algorithms or checking expiry, the ingredients of the classic algorithm-confusion and replay problems. The model answers, one of which is Listing [lst:jwt_verify_l], verify the signature, pin the algorithm list, and check expiry by default. The human example, Listing [lst:jwt_verify_h], shows the terser style that omits these steps because it answers a narrower question.

10.3 Cross-Origin Policy↩︎

CORS is the subject where the human corpus scores below zero, its insecure patterns outnumbering its secure ones. The common human answer opens the API to any origin with a wildcard, the fastest way to make a cross-origin call succeed and a standing security hazard. The model answers more often restrict the allowed origins to an explicit list. Listings [lst:cors_config_h] and [lst:cors_config_l] show the contrast.

10.4 OAuth and Session Cookies, Where the Models Slip↩︎

The two subjects where the models do not lead are the ones that expose their own failure mode. On OAuth the models reliably add the proof-key-for-code-exchange steps that the human answers omit, yet their overall score is dragged down because they also inline a client secret as a placeholder in the same file. On session cookies the same thing happens with the application secret key. The models produce a complete, runnable configuration, and completeness here means filling the secret slot with a literal value, which the terser human answers leave to the reader. Listings [lst:oauth_pkce_h] and [lst:oauth_pkce_l] show the OAuth case, where the model code is longer and more correct on the protocol yet carries the inlined secret that the audit of §8 flags.

10.5 Parameterized Queries↩︎

The database subject sits between the extremes. The human corpus is split. Some answers use a parameterized query, the safe form, while a substantial fraction build the query by concatenating or formatting the user input directly into the SQL string, the textbook injection hazard. The models almost always parameterize. Listings [lst:sql_query_h] and [lst:sql_query_l] show a human and a model answer. The human corpus also reaches for an object-relational mapper more often than the models, which hand-write the parameterized query, a difference of idiom that does not change the security outcome but does change the style.

10.6 File Upload↩︎

File upload is a subject where neither source scores well, and the reason is the number of separate defenses the task needs. A safe upload validates the file type, sanitizes the filename against path traversal, and limits the size, and few samples from either source do all three. The models do somewhat more of the validation than the human answers, as Listings [lst:file_upload_h] and [lst:file_upload_l] show, but both sources leave gaps, which makes upload a subject where provenance is a weaker guide to safety than hashing or tokens.

10.7 Injection and the Limits of the Trend↩︎

The injection-style subjects, added to probe tasks where the danger is subtler than a missing library call, complicate the clean picture of the earlier subjects and are the most interesting for it. On some of them the models keep their lead. For rendering user input into a page, the models reach for an auto-escaping template almost always, while the human answers more often build the HTML by hand and leave the escaping out, so the cross-site-scripting hazard is markedly lower in model code. For loading a secret, the models read it from an environment variable or a secret store, where the human answers more often inline a literal.

On other injection subjects the models lose their lead or fall behind, and the reason is again their objective. Asked to run a shell command with a user-supplied argument, the models invoke the shell directly, the convenient and dangerous form, more often than the human answers do, because it is the shortest complete solution. Asked to fetch a user-supplied URL, the models frequently fetch it with no validation of the scheme or the host, missing the server-side-request-forgery defense entirely, because nothing in the prompt named it and the happy-path solution does not need it. These are the same completeness hazard seen with secrets, in a more dangerous form. The model writes the whole feature and omits the defense that a security-aware developer would add against a threat the prompt did not mention. The lesson of the injection subjects is that the model advantage holds where the safe choice is a well-known library and evaporates, or reverses, where the safe choice is to anticipate an unmentioned threat. Listings [lst:xss_escape_l] and [lst:ssrf_fetch_l] put the two directions side by side, a model auto-escaping the cross-site-scripting task and a model fetching a user URL with no server-side-request-forgery check in the same corpus.

10.8 Summary of the Per-Subject Picture↩︎

Across the subjects the lesson is consistent with the aggregate but more precise. The models lead by a wide margin wherever the secure practice is a well-known library call that the human archive predates, hashing and token verification most of all. They lose their lead, and occasionally fall behind, wherever the secure practice is to leave a value unset, because their objective pushes them to produce a complete program and a complete program fills every slot. Provenance therefore predicts not just whether code is likely safe but which specific hazard to look for, a library-omission hazard in human code and a completeness hazard in model code.

11 Framework-Level Comparison↩︎

The subjects so far use a minimal library so that the security choice is isolated. A developer, however, asks a framework-level question, for example how to set up OAuth 2.0 login in a Java Spring Boot application, or how to add social login to Django, or how to wire up Passport in Express. We add four such subjects, phrased as the natural question a developer would type, and send the identical prompt to every model. Because every model answers the same question, we can place their implementations side by side and compare which security practices each one adopts.

Table 5 does this for the Spring Security OAuth subject. Each row is one model’s answer to the same prompt, and each column is a security or configuration check, marked secure or insecure. The comparison makes the differences between models concrete. All of the models produce a working Spring Security configuration, but they diverge on the details that matter. Some enable proof-key-for-code exchange and read the client secret from externalised configuration, while others inline the secret or disable the cross-site-request-forgery protection that Spring enables by default, the kind of convenience shortcut that is common in tutorials and dangerous in production. The same question therefore yields materially different security postures depending on which model answers it, and the table names exactly where.

10pt

The framework subjects also confirm the completeness hazard of §8 in a realistic setting. The framework answers are longer and more complete than the minimal-library answers, and the extra completeness again carries risk, since a model that produces a full runnable Spring configuration is more likely to fill the client secret with a literal placeholder than one that leaves the wiring to the reader. The practical implication for a developer is direct. The convenience of a complete, copy-ready framework answer is exactly what makes it worth re-checking for an inlined secret and for a disabled default protection before it is used.

12 Cross-Language Analysis↩︎

The core tasks are defined in three programming languages, Python, JavaScript, and Go, for the same underlying task, which lets us ask whether the two findings of the paper are properties of a language or of the source. We treat each task as a family with a variant per language, and compare within and across them. The security comparison uses all three languages. The provenance-transfer experiment, which needs a dense paired sample, uses the Python and JavaScript variants where coverage is deepest.

The security gap holds in every language. The central security result does not depend on the language. In each of the three the model corpus sits well above the human corpus, a mean security score of 0.35 against the human corpus in Python and 0.34 in JavaScript, with Go higher for both sources but the same gap between them. The absolute level differs by ecosystem, Go scoring higher because its standard library steers even the human answers toward safer defaults, but the direction and the size of the human-to-model gap are consistent across the three. Table 6 gives the per-family breakdown, and the pattern from §8 repeats. The models lead on the library-backed defensive tasks, hashing and token verification, and slip on the tasks where the safe choice is to leave a secret unset. That the same structure appears in three languages is strong evidence that it reflects how the two sources approach a task rather than a quirk of one ecosystem.

Figure 26 plots the same per-family scores as grouped bars, one bar per language, and the visual impression is of two profiles that rise and fall together. Where the models are strong in one language, on hashing and token verification, they are strong in the other, and where they slip, on the tasks that reward leaving a secret unset, they slip in both. The correlation of the per-family scores across languages is the quantitative form of this observation, and it is high, which is the core evidence that the security divergence is a property of the source rather than of a single language ecosystem.

Style varies by language. Table 7 reports the model style per language, and the reason the provenance boundary does not transfer becomes concrete. The models write a different amount of scaffolding in each language, driven by the conventions of the ecosystem, so the very features that carry the provenance signal, length and import count and structure, take different baseline values from one language to the next. A classifier that learned the model style of one language is therefore reading a different scale when it sees another, which is the mechanism behind the transfer failure below.

Provenance signal is partly language-specific. The attribution result behaves differently across languages, and the asymmetry is informative. A human-versus- model classifier trained on one language and tested on the other does not transfer symmetrically. Trained on JavaScript and tested on Python it reaches 89 percent, close to its within-language accuracy, but trained on Python and tested on JavaScript it falls to 93 percent, near chance. The provenance signal therefore has a language-general component, the gross size and structure difference that survives the transfer in one direction, and a language-specific component that does not. The asymmetry suggests that the model style learned on Python is narrower than the style learned on JavaScript, so a classifier calibrated on the richer JavaScript signal still recognises the Python one, while the reverse does not hold. This is a caution for anyone deploying a provenance classifier. It must be calibrated on the language it will see, because the human-versus-model boundary does not sit in the same place in every language.

The same task in three languages. The concreteness of the cross-language result is easiest to see in the code. Listings [lst:password_hash_l], [lst:password_hash_js_l], and [lst:password_hash_go_l] show a model solving the password-hashing task in Python, JavaScript, and Go. The three differ in every surface detail, the imports, the idioms, the shape, which is why a provenance classifier does not transfer freely between them. All three, however, use the same class of defense, a purpose-built password hash with a salt and a safe verify, which is why the security score is the same across the languages. The listings show both parts of the result, that the style is language-specific and the security posture is not.

Summary. The languages agree on the substantive result and disagree on the incidental one. The security divergence between human and model code is a property of the source and appears in both languages, while the exact decision boundary that separates human from model code is partly language-specific and does not transfer freely. A study in a single language would have reported the first and silently assumed the second.

13 Case Study, Vulnerability Repair↩︎

The generation results of §8 measure what a model writes when asked for a task from scratch. A second, more practical question is what a model does when it is handed insecure code and asked to fix it. This is the common real workflow of a developer pasting a flagged snippet into an assistant, and it is a different capability from clean-slate generation. We test it directly.

Setup. We assemble 21 short programs, each containing one well-known vulnerability drawn from 12 distinct Common Weakness Enumeration classes, among them a weak password hash (CWE-327), SQL injection (CWE-89), OS command injection (CWE-78), a disabled signature check (CWE-347), a wildcard cross-origin policy (CWE-942), a hardcoded credential (CWE-798), reflected cross-site scripting (CWE-79), path traversal (CWE-22), a non-cryptographic random source (CWE-330), server-side request forgery (CWE-918), unsafe deserialization (CWE-502), and an open redirect (CWE-601). The seeds span four languages, Python, JavaScript, Go, and Java. Each seed carries two regular expressions, one for the insecure pattern that a fix must remove and one for the secure pattern a correct fix must introduce. We send each seed to every model with a neutral instruction to fix the security vulnerability and return the corrected code, and we score the result deterministically. A repair counts as successful only when the insecure pattern is gone and the expected secure pattern is present, so a model that deletes the dangerous call without adding the correct defense is not credited. This yields 131 scored repairs.

Models repair most but not all vulnerabilities. The overall repair success rate is 77 percent. The models are competent at the task, and Table 8 shows the per-model breakdown, with the strongest model reaching 90 percent. The spread across models is real and follows capability, with the larger models repairing more reliably than the smaller ones, which matches the intuition that repair, like generation, rewards a model that knows the idiomatic secure replacement.

The partial-fix hazard. The most useful result of the case study is the failure mode. In 16 percent of all repairs the model removed the insecure pattern but did not add the secure one. This is not a cosmetic miss. A repair that deletes an md5 call but stores the password in plain text, or that removes a wildcard origin but sets no origin at all, produces code that no longer matches the insecure signature yet is not secure. A regex or signature-based scanner would mark such a repair as clean. The partial fix is the repair-time analogue of the completeness hazard of §8, and it is a concrete caution against trusting an assistant’s fix without re-checking that the defense it was supposed to add is actually there.

Some vulnerability classes are harder than others. Table 9 and Figure 29 give the success rate per vulnerability. The classes with a single canonical library fix, a parameterized query, a strong password hash, or a cryptographic random source, are repaired most reliably, often at or near a perfect rate, because the correct replacement is unambiguous and the models know it. The classes that require adding a check the original code did not have, validating a redirect target, a fetched URL, or a file path, are repaired least reliably, the open redirect and the path traversal being the hardest in our set. The reason is the same one that makes these classes written insecurely in the first place. The safe form is an addition the model must think to make, not a substitution it can pattern-match, and that is where both generation and repair are weakest. The pattern holds across all four languages, with the ranking of easy and hard classes preserved, so the effect is a property of the vulnerability class rather than of one language.

14 Discussion↩︎

Provenance is a stylistic boundary. The attribution result of §7 is strong, and its explanation matters as much as its accuracy. The human-versus-model boundary is recovered mainly from the shape of the code, its size, its structure, and its scaffolding, and only secondarily from its security content. A reviewer or an audit tool can therefore flag likely machine-generated code cheaply, from surface features alone, and the same features that make the code distinguishable are the ones that make it feel finished. This connects the provenance question to the security question. Machine code is recognisable because it is complete, and its completeness is exactly what makes its embedded flaws dangerous to paste unedited.

The security comparison is against a human baseline, not a gold standard. The finding that model code adopts secure patterns more often than the human corpus must be read carefully. It is a comparison between two real sources a developer might copy from, not a claim that model code is secure in absolute terms. The prior literature, which compares model output against a correct reference, finds that assistants still emit vulnerable code at a meaningful rate [@pearce2022asleep; @perry2023users]. Our result is compatible with that. Against the highly-voted but often dated human answers on Stack Overflow, the models have shed many legacy omissions, yet they introduce their own failure mode. Both readings hold, and the accurate summary is that the two sources fail differently rather than that one is safe.

Why the sources fail where they do. The pattern of divergence has a plausible cause. The human corpus is an archive. A highly-voted answer accumulates its score over years, and much of it predates the current defaults for hashing, token handling, and cross-origin policy, which is why the human corpus under-adopts exactly those defensive patterns. The models are trained on a snapshot that includes the more recent guidance, so they reproduce the current consensus on those tasks. The models’ own failure, the inlined placeholder secret, follows from their objective. They are asked for a single complete program, so they fill every slot, including the ones a human answer would leave to the reader, and a filled secret slot is a hazard the human fragment never creates.

Implications for review. The two results combine into concrete guidance. Code that a lightweight classifier flags as machine-generated should be routed to the checks that model code most often fails, which the divergence profile of §8 names, foremost the scan for hardcoded secrets and placeholder configuration. Code identified as copied from a human source should instead be routed to the legacy-omission checks, the hashing, token, and cross-origin defaults that the human corpus most often misses. Provenance is thus not an end in itself but a router that sends each artifact to the review it most needs.

Deploying a provenance classifier. The cross-language result of §12 is a direct warning for anyone who would deploy the attribution classifier as a tool. The boundary between human and model code does not sit in the same place in every language, and a classifier calibrated on one language can perform below chance on another, producing confident but incorrect labels. A deployable detector must therefore be calibrated per language, and its confidence must be discounted on any language it was not trained on. The interpretable features we use make this tractable, because the small feature set can be re-fit on a modest per-language sample rather than requiring a full retraining of a deep detector. It is therefore more accurate to treat provenance attribution as a language-scoped capability than as a universal one.

The moving-target caveat. Both of our results are snapshots. The human corpus ages in one direction, its highly-voted answers drifting further from current practice as time passes, which will tend to widen the security gap we measure. The models move in the other direction, retrained on newer data and tuned by their providers, which may narrow the gap or shift the failure mode. The value of the released pipeline is that it makes the measurement repeatable. The same specification re-run in a year will report where each source has moved, and the fail-closed checker guarantees that the new numbers are the ones the new data supports.

15 Limitations and Threats to Validity↩︎

Pattern detectors are lexical. The security checks are regular expressions, not a semantic analysis. They detect the presence of a pattern, for example a call to a strong key-derivation function, but they do not prove the pattern is used correctly. A sample can adopt bcrypt and still store the result badly. The detectors are a transparent, reproducible first cut, not a verifier, and the paper’s claims are therefore about pattern adoption rather than about proven security. We treat the lexical nature of the detectors as the principal threat to construct validity and avoid any claim that a high security score means a program is safe.

The human corpus is a snapshot of one site. The human material is drawn from Stack Overflow, retrieved through its public API with a fixed query per subject. A different query or a different site would surface different code, and the age distribution of highly-voted answers biases the corpus toward older practice, which is part of what the security-divergence result measures rather than a nuisance to remove. We record each answer’s score and year so this bias is visible and auditable.

The model set is open-weight and gateway-limited. The models are those a public open-weight gateway makes available, and the widely-used closed commercial assistants are not among them. The results therefore describe open-weight models, which is a real and growing population, but not the specific commercial products the earlier user studies examined. The gateway also throttles models unevenly, so per-model sample counts differ, and a model with few samples contributes a noisier security profile. We report true counts and weight aggregates by them.

Attribution is within a fixed pool. The model-attribution result is a closed-world classification among the models in the study. It does not address the open-world question of recognising an unseen model, and its accuracy would fall as the pool grows and model styles converge. We present it as evidence that a per-model fingerprint exists at this scale, not as a deployable detector.

Prompt sensitivity. Each subject uses a single fixed prompt. Model output is sensitive to phrasing, and a prompt that explicitly asked for security would raise the adoption of defensive patterns. Our prompts are deliberately neutral, phrased as an ordinary developer request, so the measured adoption reflects the default behaviour of each source rather than its best behaviour under prompting.

16 Ethics and Responsible Use↩︎

The two capabilities this paper demonstrates, attributing code to a source and profiling the security of that source, are dual use, and we state the intended and the misuse cases plainly.

Intended use. The provenance classifier is meant to route code to the review it needs, as §8 describes, and to support consented uses such as a team understanding the composition of its own codebase or an educator discussing the difference between hand-written and generated solutions. The security-divergence profile is meant to make secure-coding review more efficient by naming, per source, the failures most worth looking for.

Misuse and why the risk is limited. A provenance classifier could in principle be used to penalise the use of an assistant, for example to detect and sanction machine help where it is disallowed. We note two limits that reduce this risk. The accuracy is bounded and, as §12 shows, does not transfer across languages, so a naive deployment would produce confident errors that make it unsuitable as a sole basis for any consequential decision. And the signal is dominated by surface style, which a user who wished to evade detection could alter trivially by reformatting, so the classifier is not a robust adversarial detector and should not be presented as one.

Data handling. The human corpus is public Stack Overflow content retrieved through the official API within its terms, and we store only the code block, its public identifier, and its public score, no personal data. The model corpus is generated by us through a gateway we are entitled to use, and no third-party private code is collected. The released dataset therefore contains only public human answers and our own generations.

No claim of absolute security. The security scores are pattern-adoption measures, not verification, as §15 states. We are careful throughout not to certify any sample as secure, and we frame every result as a comparison between sources rather than as an audit of any individual program.

17 Conclusion↩︎

We studied two sources of developer code side by side, the human answers of Stack Overflow and the output of 9 open-weight language models, on 31 security-sensitive tasks, using only open sources and a fully reproducible pipeline. Two results stand out. Provenance is recoverable. A lightweight classifier separates human from model code at 93 percent and attributes a sample to its specific model at 48 percent, and the boundary is stylistic before it is a matter of security. Security diverges. Against the human corpus, the models adopt modern defensive patterns far more often, with an aggregate security score of 0.40 against 0.12, yet they introduce their own hazard by inlining placeholder secrets into otherwise complete programs. The two sources do not rank as safe and unsafe. They fail in different places, and knowing which source a snippet came from tells a reviewer which failure to look for. The pipeline is data driven and released, so any new task is one specification entry away, and a fail-closed checker re-derives every number here from the stored samples.

18 Pattern Detector Catalogue↩︎

Table 1 in Appendix 23 lists every security and style detector used, its subject, its polarity (secure or insecure), and a short description. Each detector is a regular expression evaluated against the sample text. The catalogue is the complete, reproducible definition of what the security-divergence analysis measures. A detector firing means its pattern is lexically present, which the paper treats as adoption of the pattern, not as proof of its correct use.

19 Per-Model Security Profile↩︎

Table 10 gives the mean security score and sample count for each model and for the human corpus, the per-origin detail behind Figure 6. The human corpus is included as a row so the models can be read against it directly.

5pt

20 Per-Model Security by Language↩︎

Table 11 gives each model’s mean security score in each language, the per-model detail behind the cross-language result of §12. The scores are close across languages within a model, which is the model-level form of the finding that the security posture travels with the source rather than the language.

21 Full Adoption Table↩︎

Table 2 in Appendix 23 gives the complete per-subject, per-pattern adoption rate for the human corpus and the model corpus, the detail behind the aggregate of §8 and the per-subject discussion of §10. H adopt and L adopt are the human and model adoption rates, and the paired counts give the sample size behind each cell so that a rate resting on few samples can be identified.

22 Reproducibility↩︎

The study is reproducible end to end from public sources. The pipeline is a sequence of small scripts, each writing an intermediate artifact to disk, and every number in the paper is re-derived from those artifacts by a fail-closed checker.

1. probe_fast.py probes every model the gateway advertises and records, in model_availability.csv, whether each answers, is rate limited, or is unavailable.

2. fetch_stackoverflow.py retrieves the human corpus for each subject through the public Stack Overflow API, storing each answer with its identifier, score, and year.

3. generate.py sends each subject prompt to each usable model and stores the returned code with its provenance metadata. It is resumable and paced, and it is run as repeated passes so that models throttled during one pass are collected in a later one.

4. extract_features.py reduces every sample to the security and style features of §5, writing features.csv.

5. benchmark.py computes the attribution classifiers and the security-pattern prevalence, writing the analysis tables and numbers.json.

6. gen_tables.py and gen_figures.py emit the paper’s tables and figures, and verify_numbers.py re-derives every macro from the data and fails if any disagrees.

The subject specification, the model list, and every collected sample are released with the code, so a third party can re-run the pipeline or extend it to a new subject by adding a single specification entry. The classifiers use scikit-learn [@scikit] with a fixed random seed, so the reported accuracies are deterministic given the same samples.

23 Reference Tables↩︎

The two reference tables below are the full detector catalogue and the complete per-subject adoption data. They are set in a single column so they can run across pages without truncation.