1 Introduction↩︎

The study of Code Authorship Attribution (CAA) centers around the analysis of coding styles, including coding structure, naming conventions, and formatting styles, to identify the authors of code samples [1], [2]. CAA plays a vital role in assisting with the automation of key software engineering (SE) tasks related to maintenance and software quality. For example, in automated bug triaging past research has aimed to use code authorship as means to build profiles of expertise for developers to more effectively triage bugs [3]. However, authorship information in software repositories may often be absent or inaccurate due to processes such as code review or pair programming [4], [5]. As such, CAA can assist in these tasks by disambiguating and accurately providing more reliable authorship information. Similar benefits of CAA hold true across other SE tasks such as fault localization, developer profiling and workload balancing [6], Security and Compliance Audits [7], [8], and test prioritization [9]. More specifically CAA can aid with Bug Triaging and Fault Localization by identifying the most likely author of a buggy code segment accelerates the debugging process and reduces maintenance overhead [5], [10]; Developer Profiling and Workload Balancing by analyzing developer coding styles can help assign tasks based on expertise levels, optimizing team productivity and reducing risk [6]; Security and Compliance Audits by attributing code authorship can detect unauthorized contributions or potential insider threats, supporting secure software practices [7], [8]; and finally Testing Efficiency: wherein stylometric patterns can inform targeted testing strategies, such as prioritizing tests for modules written by less experienced developers. This can reduce test suite size and accelerate validation cycles [11]. Given the role that CAA and the styolmetric identification of coding patterns more broadly have across various stages of the software development lifecycle its importance is clear.

Beyond development, CAA is also critical in application domains such as cybersecurity and software forensics for addressing copyright disputes, detecting plagiarism, and supporting criminal prosecutions [12], [13]. Consequently, this area of research is becoming increasingly popular [13]. However, manually verifying or identifying authors is tedious and time-consuming; these tasks are susceptible to human errors and inconsistencies, especially in large software systems [14]. These difficulties tend to arise because such systems often involve numerous authors, who often exhibit nuanced overlaps in coding styles (e.g., due to the presence of style guidelines or linters) that can make differentiation among contributors difficult [15].

Considering the above challenges, researchers have proposed approaches using Machine Learning (ML) and Deep Learning (DL) techniques to automate tasks related to CAA [1], [16], [17]. However, these techniques suffer from two key limitations. Namely, they typically rely on hand-crafted feature engineering and they are susceptible to adversarial attacks, wherein authors attempt to avoid identification through purposeful code perturbations. Hand-crafted features are problematic because there are many different aspects that can define coding style (e.g., length, whitespace, declaration usage, identifier style) and manually defining features to properly capture a range of different coding styles is challenging. Prior models have also been shown to be susceptible to adversarial attacks  [18]. Given these limitations, further research is needed to understand how to best design practical, robust techniques for carrying out CAA.

Recently, transformer-based Language Models (LMs) for code have shown remarkable effectiveness across several Software Engineering (SE) tasks including but not limited to Code Clone Detection, Vulnerability Detection, and Software Documentation [19], [20]. Much of the work that adapts LMs to SE tasks originated in the field of Natural Language Processing (NLP). In fact, stylometry for natural language is well studied within the field of NLP, where LMs have been illustrated to excel at the task [21], [22]. However, LMs have not been well studied for tasks related to code stylometry. We argue that it is worthwhile to understand the application of LMs to code styolmetry for two main reasons. First, LMs are typically trained on massive datasets, spanning both code and natural language, meaning that they have likely observed a wide array of coding styles, and hence may perform well at picking up on the nuanced patterns that define authorship. Second, models fine-tuned for CAA can take advantage of the automated feature engineering capabilities of LMs, making use of rich code embeddings for CAA.

As such, in this paper, we carry out the first comprehensive, large-scale empirical study applying LMs to the task of CAA, with the intent of examining in what settings LMs excel at the task, their limitations across datasets of varying code composition, and their robustness to adversarial code transformations. Additionally, we aim to gain a better understanding of why we observe the model performance reported in our study through an in-depth analysis of LM behavior using machine learning interpretability techniques.

Our study context encompasses 12K code snippets gathered from six different sources, one of which is a novel dataset not explored in prior work. These datasets are highly diverse in terms of coding styles, with contributions from 463 developers. We fine-tune two larger code LMs, Code Llama [23] and DeepSeek-Coder [24], and five smaller code LMs including CodeBERT [25], ContraBERT_C [26], ContraBERT_G [26], GraphCodeBERT [27] and UniXcoder [28], and evaluate their effectiveness in CAA across our studied datasets. More importantly, we also study the robustness of these models to adversarial attacks, by applying carefully crafted code perturbations to our dataset samples that have been shown effective at deceiving prior work. Finally, we conduct both qualitative and quantitative analysis by adapting well-known interpretability technique to our model and problem context in order to better understand why we observe the model performance captured in our study. Our analysis captures several significant findings that further our understanding of how LMs for code perform when tasked with tasks related to code stylometry, and point toward important directions for future investigation. The key findings from our study are as follows:

1. Larger and Smaller LMs are Effective in Different Settings: Larger LMs like Code Llama have significant capacity, and when properly fine-tuned for the CAA task, tend to perform well on multilingual, imbalanced datasets with shorter code snippets, provided that they undergo adequate fine-tuning. Smaller fine-tuned models tend to perform well on balanced, monolingual datasets.

2. LMs Outperform Prior Techniques by Learning More Separable Author Embeddings: The advantage of LMs over PbNN [4] is explained by embedding-space geometry rather than dataset size. Fine-tuned LMs separate intra- and inter-author cosine-similarity distributions with substantially higher Jensen-Shannon (JS) [29] than PbNN (e.g., 0.992 vs. 0.924 on GitHub-Java; 0.666 vs. 0.181 on LeetCode), and this separability predicts attribution accuracy across datasets. Pre-trained code embeddings encode authorship signal that path-based embeddings learned from scratch fail to capture.

3. Stylometric Variation Influences Attribution Performance: A larger number of training samples does not always translate to better performance. For instance, in a dataset we derive from LeetCode [30], C++ (62.19%) had the most samples but achieved only 64% accuracy, whereas C# (2.01%) had fewer samples but attained 94% accuracy. This discrepancy arises due to the reduced stylometric variation in C++, where many developers solve the same problem in a similar style, making authorship attribution harder.

4. Misattribution is Concentrated on Style-Overlap and Short Snippets: Across all models, misclassified samples have intra- and inter-author similarity distributions that overlap almost completely (JS divergence = 0.016)—different authors converge on identical idioms in algorithmically constrained problems. Independently, snippets below 200 tokens carry insufficient stylometric signal.

5. LMs Exhibit Orthogonal Stylometric Understanding: Different LMs are able to learn distinct, non-overlapping stylistic features to attribute authorship. The top 30% most important features (identified using Integrated Gradients [31], validated via necessity and sufficiency) play a crucial role in this process, and different models tend to learn different “important” features, motivating future work on combining different models.

6. LMs Are Less Vulnerable to Adversarial Attacks Than Prior ML/DL Techniques: Although adversarial success rates vary across models, LMs are uniformly more robust against adversarial attacks than PbNN, which has the highest adversarial success rate (42.18%) and the highest category-wise success rate across every transformation family.

7. Adversarial Robustness Depends on Feature-Utilization Strategy, Not Just Model Size: Using Integrated Gradients over 14 AST token categories, we find smaller LMs (CodeBERT, GraphCodeBERT, ContraBERT_C, ContraBERT_G—each 125M parameters) distribute attribution roughly uniformly across many categories, while larger LMs (UniXcoder at 223M, DeepSeek-Coder at 1.3B) and PbNN concentrate attribution on a few categories (e.g., PbNN on preprocessor and comments; UniXcoder on identifiers). Since each adversarial transformation perturbs only a subset of categories, distributed-attention models retain discriminative signal in untouched categories and degrade less. Robustness is therefore governed by what features the model relies on, not parameter count alone.

8. Contrastive Pre-Training Forces Structural Feature Learning: ContraBERT_C and ContraBERT_G—pre-trained with an InfoNCE objective over semantic-preserving augmentations (variable/function rename, statement reorder, dead-code insertion)—show substantially lower adversarial success rates than their non-contrastive base models (CodeBERT, GraphCodeBERT) on exactly the attacks that mirror those augmentations. The mechanism: contrastive training pulls semantically equivalent variants together in embedding space, pushing the model to rely on structural and syntactic features rather than surface identifiers.

9. Adversarial Changes that Modify Higher Number of Tokens are More Successful: The most effective adversarial transformation rules involve removing comments, eliminating unused code, adding print/log statements, modifying functions, and altering control flow. These transformations change a larger portion of the code, making them more effective in deceiving LMs.

2 Problem Statement and Research Questions↩︎

CAA involves analyzing a piece of code to determine who wrote it when the author is unknown. In this paper, as in prior work [4] we formulate this problem as a multi-class classification problem, wherein a model takes a snippet of code as input and, predicts one of a known set of authors. In this paper, we evaluate the effectiveness of LMs on CAA using the following research questions (RQs):

RQ\(_{1}\): How effective are LMs for CAA? This RQ aims to evaluate the effectiveness of LMs in identifying the author of a given code snippet across diverse scenarios. Specifically, we assess their performance across different programming languages, code snippet sizes, and number of potential unique authors. To provide a comprehensive analysis, we compare the performance of LMs with an existing DL-based technique, examining their relative strengths and limitations in CAA.

RQ\(_{2}\): What stylistic features of code do LMs learn when applied to CAA? CAA relies on the assumption that each developer has a unique coding style that models can learn to distinguish. While LMs may achieve high performance in CAA, it remains unclear whether they genuinely capture distinct stylistic features for each author or rely on spurious correlations. If LMs identify orthogonal stylistic features for each author, it would indicate that they are learning author-specific patterns. However, if the important features overlap significantly across authors, it would suggest that the models are making attributions based on shared structures rather than individual stylistic traits. Understanding this distinction is essential to ensure that LMs provide reliable and interpretable authorship predictions rather than merely optimizing for high accuracy through non-meaningful signals. This RQ aims to investigate whether LMs truly capture orthogonal stylistic features in CAA, by assessing their interpretability and robustness.

RQ\(_{3}\): How robust are LMs against adversarial attacks for the CAA task? This RQ investigates the vulnerability of LMs in CAA when exposed to adversarial modifications. Since LLMs may rely on superficial stylistic patterns, they can be susceptible to attacks such as variable renaming, code refactoring, and statement reordering, which preserve original program logic while altering surface-level features. We aim to assess whether LMs capture deep, meaningful stylistic patterns or merely rely on brittle heuristics. Understanding their robustness will provide insights into potential weaknesses and inform the development of more resilient attribution models.

3 Studied Datasets↩︎

2pt

To answer the above RQs, we utilized six datasets collected from GitHub, Google Code Jam (GCJ), and LeetCode, whose details are provided in Table 1. The first five datasets are employed in existing studies [12], [32], while we collected the last one ourselves. The datasets GCJ-Java [12], GCJ-CPP [12], and GCJ-Python [32] are originated from the GCJ challenge. The other two datasets, GitHub-C [12] and GitHub-Java [12], are sourced from GitHub. We compiled the last dataset from LeetCode. For this dataset, we scraped public solutions to various LeetCode problems. For instance, there are public solutions [33] to the Two Sum [34] problem. For each LeetCode problem, we collected all public solutions and submitted them to the platform to check for correctness. If the solution was accepted, it was included it in our dataset. This method ensured the validity of the solutions, as not all public solutions were correct. Note that we have included Table 9 in our paper appendix that presents a comprehensive breakdown of the LeetCode dataset across six programming languages. The dataset contains 4753 total samples from 198 authors. C++ dominates with 2,956 samples (62.19%) from 142 authors, followed by Java with 1180 samples (24.83%) from 87 authors, and Python with 322 samples (6.78%) from 29 authors. The remaining languages—JavaScript, C#, and Ruby—constitute 6.20% of the dataset combined. The samples-per-author (S/A) statistics reveal significant imbalance within each language: the maximum S/A ranges from 13 (Ruby) to 150 (C++), while standard deviations of S/A span from 4.99 (Ruby) to 23.86 (C#), indicating highly skewed author contributions. Notably, C# shows the highest relative variability with only 4 authors contributing between 1 and 56 samples each, while C++ exhibits the largest absolute spread with authors contributing between 1 and 150 samples. Finally, We contributed this dataset to evaluate LMs in following scenarios:

(1) Multilingual Scenario: As displayed in Table 1, existing datasets are monolingual, containing samples from a single programming language which often does not replicate the real-world. In the industries and open-source projects, developers often use multiple programming languages. For instance, a project may be written in Java but utilize Python for testing or JavaScript for web interfaces. This multilingual aspect is crucial for real-world CAA applications.

(2) Smaller Code Size and Imbalanced Data: As shown in Figure 1, the length of the code samples in the existing datasets is relatively larger than the code samples in the LeetCode dataset. As such, we can not evaluate the performance of LMs on small code samples using the existing datasets. However, in real-world scenarios, it is common to encounter small code snippets, such as those found in bug reports or code reviews. Therefore, we included the LeetCode dataset to evaluate the performance of LMs on small code samples. Additionally, existing datasets are mostly balanced, with each author contributing a similar number of samples. However, in real-world scenarios, the number of samples per author can vary significantly. For instance, some authors may contribute only a few lines of code, while others may contribute entire modules or libraries. Therefore, we included the LeetCode dataset to evaluate the performance of LMs on datasets with a large number of authors and varying numbers of samples per author.

(3) Less Stylometric Variation among Authors: In open-source or industry projects, developers often follow similar coding styles and conventions, making it challenging to distinguish between authors based on stylistic features. This is particularly true for large projects with many contributors, where the codebase is often subject to strict style guidelines and code reviews. While solving LeetCode problems, users often need to use the best data structure and algorithm to avoid time and space complexity issues. Therefore, there is less stylometric variation among authors.

(4) Adversarial Attack Scenario: While attacking LMs or any model, there are two main constraints needed to be considered: (1) the attack should not change the original logic of the code, and (2) adversarial samples should be plausible [18]. Prior CAA datasets do not include test cases, which makes it challenging to fulfill the first constraint. In our LeetCode dataset, we can test the first constraint by submitting the adversarial samples to LeetCode and checking if they are accepted. We discuss the second constraint for LeetCode datasets in threats to validity section.

4 Empirical Study Methodology↩︎

4.1 RQ\(_{1}\): LM Effectiveness in CAA?↩︎

To answer RQ\(_{1}\) and RQ\(_{2}\), we fine-tune six pre-trained LMs for code including CodeBERT [25], ContraBERT_C [26], ContraBERT_G [26], GraphCodeBERT [27], UniXcoder [27], DeepSeek-Coder [24], and Code Llama [23] on the CAA task. LMs for code fall into several categories based on their architecture, pretraining objectives and number of parameters. This study focuses on evaluating open-source LMs for CAA tasks. GPT-4/4o like models are closed models, and their weights are not publicly accessible, making them difficult/impossible to fine-tune for CAA and unsuitable for interpretability analyses. Our interpretability methods require access to internal model representations, which is not possible with GPT-4-based APIs or other commercial models.

Due to computational resource constraints, it was not reasonably practical to include all open-source models in our experiments. Instead, we selected representative models from all major transformer architectures—including encoder-only, decoder-only, and encoder-decoder—to ensure broad coverage. Specifically, we included DeepSeek-Coder (1.3B) and Code Llama (7B) as large-scale models to assess how model capacity influences performance. Finally, our selection considered varied pre-training objectives (e.g., masked language modeling, causal language modeling, contrastive learning), as these strategies are known to impact downstream performance. Therefore, our experimental design therefore balances architectural diversity, scale, and pre-training strategies, providing insights into how these factors influence CAA performance. The properties and attributes of these LMs are summarized in Table 2. In the evaluation section, we try to understand the effectiveness and generalization of different types of LMs for CAA by comparing their performance on different datasets.

Figure 1 and Table 2 show that most samples exceed the input size of our smaller studied LMs. As such, we split larger code samples into smaller chunks, ensuring that each chunk fit within the model’s input limit while retaining author information. This approach applied to all LMs except Code Llama and DeepSeek-Coder. Due to GPU memory constraints, we limited chunks to 2500 tokens rather than the maximum input size.

To fine-tune all encoder-only LMs, we adopt the standard sequence classification approach [35], which involves adding a classification layer to the embedding of the [CLS] token. For Code Llama and DeepSeek-Coder, we use the embedding of the last token as input to the classification layer. Fine-tuning is performed using the cross-entropy loss function, with the Adam optimizer [36] for weight updates. For all models, we set the maximum number of epochs to 20 and apply early stopping with a patience of 3. All experiments are conducted using the Hugging Face library [37].

In addition to this setup, we conduct hyperparameter tuning for the learning rate and batch size using a grid search approach, following an existing CAA study [4]. We evaluate three learning rates (5e-05, 3e-05, and 2e-05) and two batch sizes (16 and 32), as these values have been shown to be effective for fine-tuning LMs on downstream tasks [35]. This results in six configurations, each evaluated using 10-fold cross-validation for all datasets except GCJ-CPP, where we use 8-fold cross-validation due to 8 samples per author. We select the best hyperparameter set based on the average F1 score across all folds. In order to bolster the reproducibility of our work, we have included Table 8 in our appendix that provides a list of the best hyperparameters for each model.

Our experiments reveal that the optimal hyperparameters vary across models and datasets. However, a learning rate of 3e-05 with a batch size of 16 performs best for the majority of models and datasets. Fine-tuning is conducted on a single NVIDIA H100 GPU with 80GB of memory. Due to memory constraints, we do not apply grid search to Code Llama and DeepSeek-Coder. Instead, we tune only the learning rate while fixing the batch size at 2. For DeepSeek-Coder, learning rates of 5e-05 and 3e-05 yield the best results depending on the dataset, while for Code Llama, a learning rate of 8e-5 performs best across all datasets. For Code Llama, we could not fine-tune all 7B parameters due to GPU memory constraints. Therefore, we employed LoRA [38] to fine-tune the projection layers in the self-attention mechanism. Following the empirical recommendations from Hu et al. [38], we applied LoRA specifically to the query (Wq) and value (Wv) projection layers, as their analysis demonstrated that this combination achieves optimal performance compared to other configurations (e.g., adapting individual matrices or all four attention matrices). We set rank r=32 and alpha \(\alpha=64\), resulting in approximately 16.78M trainable parameters (0.24% of the model). While LoRA has been shown to match or exceed full fine-tuning performance across various tasks [38], we note that our approach is particularly effective on balanced datasets with high stylometric variation, as evidenced by the results in Table 3. Additionally, we conducted experiments with two other configurations: (rank=\(\left\{8,16\right\}\) and alpha=\(\left\{16,32\right\}\)). As shown in the Table 12, all three configurations yield nearly identical performance, with accuracy differences within \(\pm\)​1 percentage point.

For evaluating the effectiveness of LMs on the CAA task, we use Accuracy, Precision, Recall, and F1-Score as metrics. Accuracy is widely used in the literature [12], [32] to evaluate the effectiveness of baselines on the CAA task. However, Accuracy is biased towards majority classes, meaning if a model predicts the majority author correctly while failing on minority authors, it may still report a high accuracy. Therefore, we consider the other metrics as well to provide a more balanced evaluation. To ensure a robust assessment of model performance, we employ K-fold cross-validation, which mitigates biases from a single train-test split and provides a more stable performance estimate across different subsets of the dataset. This approach helps assess the generalization of LMs by ensuring that all authors contribute to both training and testing phases. As we mentioned above, we split the code samples into smaller chunks to fit the input size of LMs. To obtain the final prediction for a given code sample, we aggregate the predictions of its chunks by summing up their scores and selecting the author with the highest total score as the final prediction.

We employ Mann-Whitney U-test [39] to ascertain whether a statistically significant difference in performance exists between LMs. When comparing any two LMs, null hypothesis (\(H_0\)) posits that there is no significant difference in the average performance (\(\mu_0 = \mu_1\)) of the two. Conversely, alternative hypothesis suggests a significant difference in the average performance (\(\mu_0 \ne \mu_1\)) of the two LMs. Then we calculate the U-statistic, which is based on the difference in the summed ranks between the two models’ accuracy scores across \(k\)-folds. If the p-value associated with U-statistic is less than the level of significance (\(\alpha\)), we reject the null hypothesis. Otherwise, we conclude that a significant difference exists between the two models’ performance.

Finally, we compare the effectiveness of LMs with a state-of-the-art (SOTA) DL-based technique PbNN [4]. PbNN (Path-based Neural Network) is a language-agnostic authorship attribution model designed to analyze source code. It is based on the code2vec [40] model, which transforms code into numerical representations by extracting AST (Abstract Syntax Tree) paths. We use the implementation of PbNN provided by the authors [4] and follow the same hyperparameters as in their paper. We use the same evaluation metrics as for LMs to compare the performance of PbNN with LMs. Note that while other CAA baselines for CAA exist, e.g., Ropgen , these are often language specific, and given the multilingual nature of most LLMs, we opted to compare against a similarly multi-lingual model in PbNN.

4.2 RQ\(_{2}\): What Patterns to LM Learn in CAA?↩︎

Some LMs may achieve exceptionally high Accuracy, Precision, Recall, and F1-Score in CAA. However, high performance alone does not confirm that these models are genuinely learning meaningful patterns. It is possible that they rely on spurious correlations rather than identifying distinct stylistic features for each author. Therefore, we investigate whether LMs capture orthogonal stylistic features of code during authorship attribution.

Qualitative Analysis: By orthogonal stylistic features, we mean that the model relies on a distinct and non-overlapping set of stylistic features for each author. If LMs truly learn unique stylistic patterns per author, then their decision-making process should be based on features that are specific to individual authors and uncorrelated across different authors. Conversely, if the models achieve high accuracy but rely on overlapping or non-distinctive features, this would suggest that LMs may be making predictions by chance, reducing the reliability of their authorship attribution capabilities. Here, we implicitly assume that effective models for CAA will maximize linear separability of the learned feature representations. The rationale is that linearly separable feature spaces allow for clear, interpretable decision boundaries between authors’ styles, which is desirable for both performance and interpretability.

To answer this research question, we employ an interpretability method, called Integrated Gradients [31]. This method assigns an importance score to each feature in the input, indicating its contribution to the model’s prediction. By analyzing these feature attributions, we can determine whether the model focuses on distinct stylistic aspects for different authors. While interpretability methods such as Integrated Gradients provide insights into feature importance, it is critical to assess whether these interpretations are faithful—that is, whether they accurately reflect the model’s actual decision-making process. To do this, we evaluate faithfulness using two key metrics: Necessity and Sufficiency. These metrics are well-established in the interpretability literature [41]–[44] and measure how crucial the identified important features are for the model’s predictions.

Necessity quantifies whether removing the most important features (as identified by Integrated Gradients) significantly increases the model’s uncertainty. If LMs genuinely rely on these features, removing them should lead to higher perplexity in predictions—i.e., the model should struggle more to make accurate predictions. A higher Necessity value means that removing the identified features significantly increases the model’s uncertainty, confirming that these features were crucial for prediction.

\[\textit{Necessity}{} := \exp \left( \frac{1}{m} \sum_{i=1}^{m} \left( - \log P_\theta(y_i | x_{R_i}) + \log P_\theta(y_i | x_i) \right) \right)\]

where:

• \(m\) is the number of test samples.

• \(x_i\) is the original input for the \(i\)-th test sample.

• \(x_{R_i}\) is the perturbed input after removing the most important tokens identified by Integrated Gradients.

• \(P_\theta(y_i | x_i)\) is the softmax probability of the predicted class for the full input.

• \(P_\theta(y_i | x_{R_i})\) is the softmax probability of the predicted class after perturbation.

Sufficiency measures whether the model can still make accurate predictions using only the important features identified by Integrated Gradients. If these features truly drive the model’s decision-making, then predictions should remain confident, meaning perplexity should stay low.

\[\text{Sufficiency} := \exp \left( \frac{1}{m} \sum_{i=1}^{m} - \log P_\theta(y_i | x_{A_i}) \right)\]

where:

• \(x_{A_i}\) is the reduced input, containing only the most important tokens identified by an explanation method.

• \(P_\theta(y_i | x_{A_i})\) is the softmax probability of the predicted class using only \(x_{A_i}\).

To assess whether LMs capture orthogonal stylistic features, we analyze Necessity and Sufficiency for the top-k% important features for each LM. An increase in Necessity and a decrease in Sufficiency as \(k\) increases indicate that the model relies on these features for predictions. Next, we examine whether these features are distinct across authors. If the top-k% features are largely non-overlapping, it suggests that LMs capture orthogonal stylistic patterns unique to each author. Conversely, if significant overlap exists, it implies that the models may be relying on shared structures rather than individual stylistic signals, raising concerns about the reliability of LMs for CAA task.

Quantitative Analysis: While the feature overlap analysis using integrated gradients provides insights into token-level importance, it does not fully capture whether different models learn fundamentally different representations of authorship styles. To complement our interpretability analysis and provide stronger quantitative evidence for feature orthogonality, we perform a distributional analysis of code similarities in the learned embedding spaces.

Stylometric Variation - If different language models truly learn orthogonal stylometric features, then the geometric relationships between code samples in their respective embedding spaces should differ substantially. Specifically, we expect that intra-author similarities (code pairs from the same author) should be distinctly separated from inter-author similarities (code pairs from different authors), and this separation pattern should vary across different models, indicating that each model captures unique stylistic dimensions.

Embedding Generation- For each fine-tuned model in our 10-fold cross-validation setup, we extract code embeddings from the model’s final hidden states. We tokenize each code sample with padding and truncation to a maximum sequence length of max_length tokens, then pass it through the model to obtain contextualized representations. To create a single embedding vector for each code sample, we use mean pooling: we average the hidden states across all tokens in the sequence, excluding padding tokens. This is achieved by masking out padding tokens using the attention mask, summing the remaining hidden states, and dividing by the number of actual (non-padding) tokens. This mean-pooling approach provides a holistic representation of the entire code sequence.

Similarity Computation- We compute pairwise cosine similarities between all code embeddings within each test fold. For each pair of samples (i,j) where \(i < j\), we calculate:

\[similarity (i,j) = \frac{e_i . e_j}{ |e_i| . |e_j|}\] where \(e_i\) and \(e_j\) are the embedding vectors for samples i and j, respectively. We then categorize each similarity score into two groups:

• Intra-author similarities: Pairs where both samples belong to the same author (author(i) \(=\) author(j)

• Inter-author similarities: Pairs where samples belong to different authors (author(i) \(\ne\) author(j)

Distributional Divergence Analysis - To quantify the separation between intra-author and inter-author similarity distributions, we calculate the Jensen-Shannon (JS) [29] divergence, a symmetric measure of distributional difference. The JS divergence ranges from 0 (identical distributions) to 1 (completely non-overlapping distributions). A high JS divergence indicates that the model creates distinctly different similarity patterns for same-author versus different-author code pairs, providing quantitative evidence that the model has learned to separate authors based on stylometric features in its embedding space.

4.3 RQ\(_{3}\): Effectiveness of Adversarial Changes↩︎

Existing literature has explored adversarial attacks on CAA models in two main categories: targeted and non-targeted attacks [12], [18], [32]. In targeted attacks, the adversary attempts to mislead the model into attributing a code snippet to a specific incorrect author. In non-targeted attacks, the goal is to cause the model to make any incorrect attribution. Prior studies [12], [18], [32] have shown that non-targeted attacks are more effective for CAA, achieving higher attack success rates. Therefore, we focus on non-targeted attacks in this study.

Several methods exist for generating adversarial code samples [15], [18], [32], [45]. However, these methods have limitations: (1) they are often specific to certain programming languages, or (2) they lack support for a wide range of program transformations. To address these limitations, we leverage a prompt-based approach using GPT-4 [46] to generate adversarial code samples. We used the ‘GPT-4o-2024-08-06’ snapshot via OpenAI’s API, with experiments conducted in February 2025. While there is the potential for non-determinism with GPT4o we did attempt to mitigate this through setting the temperature=0 and using default deterministic parameters. Our approach consists of the following steps:

1. Generating adversarial samples: A code transformation prompt and a code snippet are submitted to GPT-4.

2. Functionality verification: The adversarially transformed code is submitted to LeetCode to verify that its functionality remains unchanged.

3. Transformation verification: Since GPT-4 is prone to hallucinations, we implemented a parser to verify whether the transformations were performed correctly.

4. Evaluating LMs robustness: If both verifications are successful, we query the LMs with the adversarial samples to assess their resilience to adversarial attacks.

Figure 2 illustrates the full adversarial sample generation and verification process. The code transformation rules used in this study are derived from transformation rules established in previous adversarial attack for CAA [12], [15], [18]. Below, we present a high-level summary of the code transformation rules and verification steps used in this study, full details are in our appendix 13.

1. Statement Transformations: This category includes splitting or combining variable declarations, swapping the order of statements, and modifying the order of independent statements.

   Verification: We parse the original and adversarial code and compare their statements. A valid transformation should result in a mismatch between the statement structures of the two programs.

2. Name Transformations: This category includes rules that involve changing the naming style of variables and functions. For example, converting snake case to camel case or vice versa.

   Verification: We extract and compare the identifiers from both the original and adversarial code. A valid transformation should result in a mismatch in the identifier names between the two programs.

3. Operator Transformations: This category includes swapping relational operators (‘\(a > b\)’ to ‘\(b < a\)’), converting conditional statements to inverse operators (‘\(a < b\)’ would be ‘!\((a > b)\)’), converting integer literals to expressions (‘int b = 2 * 4’), and changing the style of increment or decrement operators (‘i++’ to ‘i +=1’).

   Verification: We extract the AST nodes corresponding to conditional, increment, and decrement operators in both programs. A valid transformation should produce a mismatch in the relevant operators.

4. Data Transformations: This category includes rules converting integers to hexadecimal, character literals to ASCII, string variables to character arrays, and boolean literals to integers.

   Verification: We analyze the AST nodes related to variable declarations in both programs. A valid transformation should result in a mismatch in the declaration statements.

5. Loop Transformations: This category includes rules converting ‘for’ to ‘while’ statements and vice versa.

   Verification: We parse loop structures in both programs and compare them. A valid transformation should result in dissimilar loop structures.

6. Control Flow Transformations: This category involves converting ‘if-else’ statements to ‘switch’ statements, ‘switch’ statements to ‘if-else’ statements, ‘if-else’ statements to ternary operators, and swapping the statements within ‘if-else’ statements.

   Verification: We extract and compare conditional structures from both programs. A valid transformation should result in a mismatch between the control flow structures.

7. Function Transformations: This category involves swapping the order of function parameters, adding an extra integer parameter with a default value of zero, creating a new function from a group of statements, and swapping the order of function declarations.

   Verification: We parse function declarations and invocations from both programs and check for differences in structure. A valid transformation should produce non-identical function signatures or call structures.

8. Miscellaneous Transformations: This category includes removing comments, unused code, and adding print/log statements to track variable values.

   Verification: We parse both versions of the code and analyze the node types. For comment removal, the original code should contain comment nodes while the adversarial version should not. For unused code elimination, the adversarial version should have fewer AST nodes. Print/log statement insertions are verified using regular expressions.

Using the above rules, we create 28 code transformation prompts to generate adversarial code samples. Then, each adversarial code sample is verified for functionality and transformation correctness before being used to evaluate the LMs robustness to adversarial attacks. In the appendix 13, we provide a detailed description of each code transformation prompt. We also provide the details of our parser for transformation verification in our online appendix [47].

In addition to our attack based on the semantic-preserving transformations described above, we also attack our baseline models using ALERT [32], a prominent adversarial attack method for pre-trained models of code. ALERT generates adversarial examples through variable renaming, using the masked language modeling function and contextualized embeddings from the victim model’s underlying pre-trained architecture (e.g., CodeBERT or GraphCodeBERT) to produce semantically natural variable substitutes. While ALERT claims to be a black-box attack, it requires substantial knowledge of the victim model’s architecture and access to the base pre-trained model’s internal functions—making it more accurately characterized as a gray-box approach. In contrast, our attack based on semantic-preserving code transformations is a true black-box attack that requires no knowledge of the victim model’s architecture or internals. We leverage GPT-4 with carefully designed code transformation prompts to generate diverse adversarial candidates, followed by semantic equivalence validation to ensure the transformed code preserves the original functionality. Our approach only requires query access to the victim model’s predictions, making it more practical and generalizable across model architectures without needing access to their underlying pre-trained components.

5 Results↩︎

5.1 RQ1: Effectiveness of LMs for CAA↩︎

Table 3 presents the results of our K-fold cross-validation experiments, including the Accuracy, Precision, Recall and F1-Score for each model across all datasets. Figure 5 shows the results of the U-test for two datasets. The black cells indicate that the performance difference between the two models is not statistically significant since the p-value is greater than the level of significance (0.05). Based on these results, we discuss the effectiveness of LMs for CAA.

SOTA vs LMs: By using the replication package provided by PbNN, we have achieved slighly higher Accuracy than the Accuracy reported for GCJ-CPP, GCJ-Java, GitHub-C and GitHub-Java datasets by Zhen et. al. [12]. In the PbNN’s hyperparameters settings, they ran this model for 10 epochs. From the training dynamics, we observe that its Accuracy increases even after 10 epochs. Therefore, we ran PbNN for 40 epochs with early stopping patience 10 during our replication. This is likely the reason for slightly better Accuracy.

Figure 3 illustrates the percentage improvement of LMs over PbNN in terms of Accuracy, Precision, Recall, and F1-Score. The improvement is computed as: \[\frac{(\textit{Metric}_{\textit{LLMs}} - \textit{Metric}_{\textit{PbNN}})}{\textit{Metric}_{\textit{PbNN}}} \times 100{\%}\]

The results demonstrate a substantial advantage of LMs over PbNN across all datasets in CAA. In particular, the LeetCode dataset exhibits the largest performance gap, where LMs achieve over 130% improvement in Accuracy, up to 204.17% in Precision, 196% in Recall, and 213.04% in F1-Score. These improvements highlight the limitations of PbNN in handling multilingual, imbalanced, short code snippets with minimal stylometric variation, all of which are key characteristics of LeetCode. Across other datasets, LMs outperform PbNN by up to 38.03% in Accuracy, 53.97% in Precision, 46.97% in Recall, and 39.13% in F1-Score.

0.2pt

Performance differences across dataset sizes: Analysis of performance across datasets reveals that the LM advantage over PbNN is not primarily driven by dataset size, but rather by dataset characteristics. Table 3 shows that on GitHub-Java (2,667 samples), PbNN achieves 0.97 accuracy—competitive with LMs—while on the larger LeetCode dataset (4,753 samples), PbNN accuracy drops to 0.32. This counterintuitive pattern is explained by samples-per-author density and author separability (Table 4): GitHub-Java has high JS divergence (0.92) with sufficient samples per author, enabling PbNN to learn discriminative features. Conversely, LeetCode’s 198 authors with sparse samples per author (average  24) results in low separability (JS divergence: 0.18), where PbNN’s limited capacity fails but LMs leverage pre-trained representations to maintain 0.74 accuracy. Notably, even on the smallest dataset (GCJ-CPP, 160 samples), LMs achieve perfect accuracy while PbNN reaches 0.90, demonstrating that pre-trained embeddings compensate for limited training data. The performance gap correlates strongly with task complexity (number of authors) and data sparsity per author, rather than absolute dataset size. PbNN remains competitive only when: (1) high samples-per-author ratio enables sufficient feature learning, and (2) authors exhibit naturally high stylistic separability.

Performance differences across languages: Analysis across programming languages reveals that the LM advantage over PbNN varies significantly by language characteristics. In language-specific datasets (Table 5), PbNN achieves competitive performance on Java datasets (GCJ-Java: 0.94, GitHub-Java: 0.97) but degrades on C/C++ (GitHub-C: 0.77, GCJ-CPP: 0.90) and Python (GCJ-Python: 0.71). This trend intensifies in the multi-lingual LeetCode dataset (Table 5), where PbNN’s accuracy ranges from 0.27 (C++) to 0.68 (Ruby)—a 41-point variance—while DeepSeek-Coder maintains consistently high performance across all languages (0.72-0.95, only 23-point variance). The performance gap correlates with language complexity and stylistic diversity. PbNN struggles most with syntactically flexible languages like C++ (gap: 0.45) and Java (gap: 0.43), where multiple idiomatic patterns dilute authorship signals that PbNN can learn from limited training data. Conversely, the gap narrows for languages with stronger conventions like Ruby and C#.

Performance differences across code lengths: Code length analysis reveals contrasting dependencies between PbNN and LMs. As shown in Table 6, PbNN exhibits erratic, non-monotonic performance across code lengths: starting at 0.22 accuracy for very short snippets (\(\leq\)​50 tokens), peaking modestly at 0.35 for medium code (201-500 tokens), dropping back to 0.31-0.33 for longer code (501-2000 tokens), and unexpectedly reaching its highest accuracy of 0.38 on very long code (\(>\)​4000 tokens). This unpredictable pattern—with \(<\)​4% variance between "optimal" and other length ranges—suggests PbNN lacks robust length-invariant feature learning.

In contrast, DeepSeek-Coder (Best Performing LM) demonstrates monotonically improving performance with code length: 0.50 on short snippets (\(\leq\)​50 tokens), steadily increasing to 0.82 for medium-long code (501-1000 tokens), and achieving perfect accuracy (1.00) on 2001-4000 token samples. The performance gap systematically widens with code length: 0.28 gap for short code (\(\leq\)​50 tokens) expanding to 0.72 gap for long code (2001-4000 tokens). Even at PbNN’s nominal peak (201-500 tokens: 0.35), LMs achieve 0.76 accuracy—a 0.41 advantage. Most critically, LMs’ monotonic improvement on longer code—where attribution is most valuable for identifying authors of complete functions, classes, or files—directly contradicts PbNN’s plateau at  0.35 accuracy. This demonstrates that pre-trained representations with superior positional encoding and attention mechanisms effectively capture authorship signals across all code lengths, while PbNN’s learned embeddings fail to scale beyond superficial pattern matching.

LMs vs LMs: While LMs consistently outperform PbNN, their effectiveness varies depending on the dataset. Our analysis indicates that in some datasets, LLMs perform comparably with no statistically significant differences, whereas in others, certain models achieve notably better results.

In GCJ-CPP and GCJ-Java, all LMs except one, achieved near-perfect scores with no statistically significant differences in performance. In GCJ-Python, however, UniXcoder and DeepSeek-Coder significantly outperformed other models, with DeepSeek-Coder achieving the highest Accuracy, Precision, Recall, and F1-Score. Other LMs performed similarly in this dataset, showing no statistically significant differences. A similar trend is observed in GitHub-C, where DeepSeek-Coder and Code Llama outperform other models, with DeepSeek-Coder achieving the highest scores across all metrics. In contrast, the remaining LMs exhibit comparable performance. In GitHub-Java, UniXcoder, DeepSeek-Coder, and Code Llama significantly outperform other models. As previously discussed, these datasets share a similar distribution of code samples and authors, which explains the limited performance variation among LMs. In such cases, capacity factors like model number of parameters, and maximum input length appear to have a minimal impact on performance differences, as all top-performing models can effectively leverage the dataset’s characteristics.

While the capacity factors of LMs (such as the number of parameters and maximum input length) have shown minimal impact on performance differences across most datasets, the LeetCode dataset presents a different scenario due to its multilingual nature, class imbalance, short code snippets, and minimal stylometric variation. Models with smaller number of parameters (125M) and similar input length (512 tokens), such as CodeBERT, ContraBERT_C, ContraBERT_G, and GraphCodeBERT, perform comparably. However, UniXcoder (223M, 1024 tokens) and DeepSeek-Coder (1.5B, 2500 tokens) significantly outperform the others, with DeepSeek-Coder achieving the highest Accuracy, Precision, Recall, and F1-Score. This suggests that in datasets with these characteristics, LLMs with larger capacity are more effective for CAA.

Interestingly, although Code Llama has the largest model size and input length, it performs comparably to UniXcoder and DeepSeek-Coder. This is likely because its parameters are not fully fine-tuned for the CAA task. Thus, we conclude that larger LMs are more effective for CAA in multilingual, imbalanced datasets with short code snippets and minimal stylometric variation, provided that they are appropriately fine-tuned.

2pt

Generalization: A critical aspect of evaluating LMs for CAA is understanding how well their performance generalizes across different dataset characteristics, including monolingual vs. multilingual code, balanced vs. imbalanced distributions, and variations in the length of the code snippet and stylometric diversity. Our analysis reveals that while LMs show strong generalization in well-balanced monolingual datasets, their effectiveness varies significantly in more complex multilingual and imbalanced settings.

For example, in well-balanced monolingual datasets such as GCJ-CPP, GCJ-Java, GCJ-Python, GitHub-C, and GitHub-Java, LMs achieve an average Accuracy of 96%, indicating strong generalization across these datasets. However, the LeetCode dataset presents additional challenges, such as short code snippets and reduced stylometric variation. As shown in Table 7, the Accuracy drops to 77%, suggesting that LMs struggle to generalize as effectively in this setting.

Models tend to perform better with longer code snippets, as the mean and median lengths of incorrectly attributed samples are notably shorter than those correctly attributed. We analyzed model performance across seven token length brackets: \(\leq\) 50, 51-100, 101-200, 201-500, 501-1000, 1001-2000, and 2001-4000 (Figure 4). All models show improved accuracy with longer code, but patterns differ by architecture. BERT-based models (CodeBERT, GraphCodeBERT, UniXcoder, ContraBERT variants) demonstrate relatively stable performance across lengths, with 10-20% point improvements from shortest to longest brackets. Large models (CodeLlama, DeepSeek) show greater length sensitivity, with 20-30+ percentage point gains, stabilizing around 200-500 tokens. This findings show that code snippets should contain at least 200 tokens for reliable attribution using larger models.

Another key observation from Table 7 is that having more training data does not necessarily improve Accuracy. For example, CPP accounts for the largest proportion of samples (62.19%) in the LeetCode dataset, yet the average Accuracy of LMs is only 64%. In contrast, with just 2.01% of the dataset, C# achieves a significantly higher Accuracy of 94%. The reason for this discrepancy is that when more developers solve a problem using the same language, the solutions exhibit less stylometric variation, making authorship attribution more difficult.   For example, the CPP solutions to the Power of Two problem [48] are among the most misattributed samples in our results. This problem has two main solutions with different time complexities: \(O(\log N)\) and \(O(1)\). Since the \(O(\log N)\) solution is less efficient, most users opt for the \(O(1)\) approach, leading to highly similar solutions. As a result, the uniformity in coding patterns reduces distinguishable stylometric differences, making it harder for LMs to correctly attribute authorship. We also measure JS divergence in CPP and C# samples to quantify stylometric variation. Our results show that CPP samples’ JS divergence is lower than C#. For example, As JS divergence increases from 0.26 (C++) to 0.58 (C#), accuracy increases from 0.72 to 0.95.

A correlation analysis between the number of samples per author and Accuracy reveals a weak but statistically significant correlation (0.17). Although it is generally expected that authors with more samples would have higher attribution accuracy, the lack of strong correlation suggests that having more samples does not necessarily aid authorship attribution when stylometric variation is low.

Taken everything together, these findings highlight that while LLMs generalize well in larger, balanced datasets, their performance deteriorates for shorter code snippets that contain less stylometric variation, underscoring the need for further improvements in handling such complexities.

Computational efficiency: BERT-based models (CodeBERT, GraphCodeBERT, ContraBERT, UniXcoder) achieve inference times of 0.0015-0.0032 seconds with <1.5 GB memory, making them suitable for real-time applications. In contrast, CodeLlama (7B) requires 30.2 GB VRAM and 0.167 seconds per prediction—approximately 100× slower—while DeepSeek (1.3B) offers a middle ground at 21.5 GB and 0.059 seconds. For practitioners, this suggests that BERT-based models are preferable for resource-constrained or latency-sensitive applications (e.g., bug triaging, CI/CD pipelines), while larger LMs may be justified for offline analysis where accuracy is paramount.

5.2 RQ\(_{2}\): LMs’ Code Stylometry Understanding↩︎

Qualitative: To determine whether LMs truly learn orthogonal coding styles, it is crucial to first assess the faithfulness of the interpretability method used — Integrated Gradients. Faithfulness ensures that the feature attributions provided by the method accurately reflect the model’s true decision-making process. Without faithful explanations, conclusions about orthogonal coding styles may be misleading.

To ensure faithful attribution of model predictions, we apply Integrated Gradients to the embedding layers of LMs. The embedding layer serves as the model’s initial processing stage, where raw code tokens are transformed into numerical representations. By computing feature attributions at this level, we ensure that the importance scores correspond directly to input tokens rather than higher-level latent representations. As shown in Figure 6, Necessity values are consistently high across most LMs up to 30%, indicating that removing 30% the most important features identified by Integrated Gradients significantly reduces model confidence. This suggests that Integrated Gradients correctly identifies 30% features that are critical for classification. Additionally, Sufficiency values are also consistently decrease up to 30%, meaning that the LMs can make confident predictions when retaining only the 30% identified important features. Therefore, we use the top 30% important features to analyze orthogonal coding styles analysis to provide a balance between capturing critical information and interpretability.

Before analyzing the orthogonal coding styles of different authors, we first examine whether LMs capture consistent coding styles among the samples of the same author. Suppose a particular LM correctly attributes multiple code snippets (\(C_1\), \(C_2\), \(C_3\), ... \(C_n\)) to author \(X_1\). If the model truly relies on the top 30% most important features identified by Integrated Gradients, we should observe a significant overlap in the important features across these samples. Our analysis reveals that the top 30% important features of \(C_1\), \(C_2\), \(C_3\), ... \(C_n\) exhibit a high degree of overlap (ranging from 84% to 94%), indicating that LMs capture consistent coding styles for the same author. Additionally, Figure 7 presents a t-SNE [49] visualization of the top 30% important features extracted from code samples in the GCJ-CPP dataset, where Integrated Gradients is applied to DeepSeek-Coder. The figure shows that samples from the same author form distinct clusters, further confirming that DeepSeek-Coder consistently relies on similar stylistic features within an author’s code samples. We also observe similar results for other LMs.

From Figure 7, we observe that code samples from different authors are well separated, indicating that LMs capture distinct, orthogonal coding styles across authors. To further investigate this, we visualize the top 30% important features of different authors using word clouds. The word clouds in Figure 8 illustrate that different authors exhibit unique stylistic preferences, as evidenced by the dominant keywords and patterns in their respective clouds. For example, Author 2 frequently uses the keyword ‘int’, suggesting a preference for integer-based operations, while Author 3 consistently includes ‘//’ comments, indicating a habit of documenting their code. This reinforces the idea that LMs effectively capture orthogonal coding styles by recognizing individualized patterns in variable usage, commenting habits, and other stylistic features.

Somewhat interestingly, we observe that different LMs capture distinct orthogonal coding styles for the same author. This variation arises due to differences in tokenization, as each LM employs a unique tokenizer that affects how the same code snippet is segmented into tokens. For example, DeepSeek-Coder treats ‘addTwoNumbers’ as four tokens (‘add’, ‘Two’, ‘Num’, ‘bers’), whereas Code Llama splits into three tokens (‘_add’, ‘Two’, ‘Numbers’). Consequently, Integrated Gradients identifies different important features across models, leading to variation in feature attributions and contributing to the orthogonal behavior observed in LMs.

Figure 9 illustrates the Degree of Orthogonality [3] in CAA within the LeetCode dataset. The value in the common area represents the number of code snippets correctly attributed by all LMs, while the exclusive regions indicate the number of samples uniquely classified by each model, highlighting their individual strengths. For instance, all baselines correctly assign 2004 code snippets to the original developer across all datasets. However, DeepSeek-Coder exhibits the highest Degree of Orthogonality in CAA, followed by Code Llama and CodeBERT.

Quantitative: Figure [fig:deep-intra-inter-sim-dist] presents the distributional analysis of intra-author versus inter-author cosine similarities in GitHub-Java and LeetCode datasets using fine-tuned DeepSeek-Coder models. The Jensen-Shannon (JS) divergence values quantify feature orthogonality and show strong correlation with classification performance (Table 3).

Datasets with high classification accuracy generally exhibit correspondingly high JS divergence values. GitHub-Java achieves 100% accuracy with a JS divergence of 0.99, indicating strong distributional separation between intra-author and inter-author similarities. The intra-author distribution concentrates at high similarity values with minimal overlap with the inter-author distribution. Similarly, GCJ-Python and GitHub-C show strong separation with high performance. Conversely, the LeetCode dataset demonstrates moderate separation (JS: 0.67) with 74% accuracy. The 198-author diversity and varied coding contexts result in greater distributional overlap, making author distinction more difficult.

JS divergence is generally predictive of classification accuracy across datasets with sufficient test set size. GCJ-CPP appears as an exception (JS: 0.66, 100% accuracy), but this result should be interpreted with caution: 8-fold CV over only 160 samples yields approximately 20 test samples per fold, which is too small to serve as a reliable indicator of true generalization performance. Excluding this low-confidence data point, the correlation between JS divergence and accuracy holds consistently across the remaining datasets. This validates that LMs learn genuine stylometric features rather than spurious correlations. Even in the most challenging scenario (LeetCode), the JS divergence of 0.66 indicates substantial distributional separation, confirming that models consistently capture meaningful, author-specific features.

5.3 RQ\(_{3}\): LMs’ Robustness to Adversarial Attacks↩︎

GPT-4-based attack: Since we used GPT-4 to generate adversarial samples, we first evaluate its effectiveness in generating these samples before assessing the robustness of LMs against adversarial attacks. To measure robustness, we use the adversarial success rate [12]. Here, a higher adversarial success rate indicates that an LM is less robust against adversarial attacks.

All LMs correctly attributed 4072 samples from the LeetCode dataset. Each of these samples was transformed into an adversarial version using GPT-4 with a predefined transformation prompt. Since we employed 28 different transformation prompts, the total number of adversarial samples generated was 114016 (4072 \(\times\) 28). To ensure that the transformations preserved the original code’s logic, we submitted all generated samples to LeetCode. However, only 41.85% (47,716) of the samples were accepted, meaning they retained functional correctness.

Next, we applied a three-step verification process using our parser to ensure that only valid adversarial samples were used for evaluation. Only 35498 samples passed all three verification criteria and were subsequently used to evaluate the robustness of LMs against adversarial attacks.

1. Applicability Check: Was the requested transformation applicable to the original sample? (Expected: YES). For example, if the prompt instructed GPT-4 to remove all comments, but the original code contained no comments, the transformation would be invalid.

2. Correctness Check: Was the requested transformation accurately applied to the adversarial code? (Expected: YES). For example, if GPT-4 was instructed to remove comments, the adversarial code should not contain comments.

3. Consistency Check: Did GPT-4 introduce any unintended transformations? (Expected: NO). For example, if GPT-4 was instructed to replace a ‘for’ loop with a ‘while’ loop but also removed comments, this would introduce an unintended modification, making it difficult to isolate the cause of attack success.

SOTA vs LMs: Figure 11 illustrates the adversarial success rate of PbNN and various LMs. The results indicate that PbNN has the highest adversarial success rate (42.18%), making it more susceptible to adversarial attacks than most LMs. Additionally, its category-wise success rate is also the highest across all categories.

LMs vs LMs: Among the LMs, Code Llama exhibits the highest adversarial success rate (56.57%), followed by DeepSeek-Coder (28.74%), UniXcoder (25.42%), GraphCodeBERT (17.92%), CodeBERT (17.83%), ContraBERT_G (16.64%), and ContraBERT_C (15.72%). Since we were unable to fine-tune all parameters of Code Llama, this may explain its comparatively lower robustness.

Excluding Code Llama, Table 2, Table 3, and Figure 11 lead to the following key observations:

1. More effective LMs tend to be more vulnerable to adversarial attacks. For instance, DeepSeek-Coder, which demonstrates the highest effectiveness in CAA, is also the most susceptible to adversarial attacks. However, one might argue that the total number of adversarial samples is not the same across all LMs, making direct comparisons unfair. To address the sample imbalance concern, we re-conducted adversarial attacks using only the samples that were correctly attributed by all LMs. The results remained consistent with our previous findings. For example, the adversarial success rates under equal sample conditions for CodeBERT, ContraBERT_C, ContraBERT_G, GraphCodeBERT, UniXcoder, and DeepSeek-Coder are 13.40%, 10.96%, 11.22%, 11.77%, 16.05%, and 19.20%, respectively.

2. Smaller LMs exhibit greater robustness to adversarial attacks. Models with fewer parameters, such as CodeBERT, ContraBERT_C, ContraBERT_G, and GraphCodeBERT (each having 125M parameters), show lower adversarial success rates than larger models like UniXcoder (223M) and DeepSeek-Coder (1.3B).

Figure 11 also presents the success rates of individual transformation rules for all LMs. The results indicate that the following transformations are the most effective in deceiving LMs.

• E. Loop Transformations: Changing ‘for’ statements to ‘while’ statements & vice versa. Figure 2 shows an example of this transformation.

• G. Function Transformations: Swapping the order of function parameters, adding an extra integer parameter with a default value of zero, creating a new function from a group of statements, and swapping the order of function declarations. Figure 17 shows an example of this transformation.

• H. Miscellaneous Transformations Removing comments, unused code, and adding print/log statements to track variable values. Figure 18 shows an example of this transformation.

To understand why these rules are particularly effective, we examined the amount of code they modify. As shown in Figure 12-a, these transformation rules introduce more code changes than others. From Figure 12-a, it is evident that transformation rule H results in the highest number of code modifications and is also the most effective adversarial transformation in Figure 11. These findings suggest that transformation rules that modify more lines of code tend to be more effective in deceiving LMs. The total amount of code changes is computed as:

\[\frac{\text{\# of lines added, modified, and deleted in adversarial code}}{\text{Total \# of lines in original code}} \times 100{\%}\]

Why are LMs less vulnerable to adversarial attacks? Although adversarial success rates vary across LMs, they are generally less susceptible to adversarial attacks compared to PbNN. This is because LLMs rely on subtle syntactic and structural features of the code, such as keywords (e.g., int, public, private, long), braces ({}), and parentheses (()), as illustrated in Figure 8. Modifying these elements while preserving semantics and successfully evading LMs is inherently challenging, making them more robust against adversarial perturbations.

ALERT-based attack: Figure 12-b presents the attack success rates (ASR) of variable renaming attacks comparing ALERT and our GPT-4-based method across five pre-trained models of code: CodeBERT, ContraBERT_C, ContraBERT_G, GraphCodeBERT, and UniXcoder. ALERT achieves substantially higher attack success rates compared to our GPT-4-based approach, with ASRs ranging from 0.67 to 0.80 (average: 0.72) versus 0.18 to 0.25 (average: 0.21) for GPT-4. The superior performance of ALERT can be attributed to its access to the victim models’ internal representations. Specifically, ALERT leverages the masked language modeling (MLM) function and contextualized embeddings from the underlying pre-trained architectures to generate semantically natural variable substitutes. Furthermore, ALERT computes an Overall Importance Score (OIS) for each variable by measuring how its replacement affects the model’s confidence in the ground truth label, allowing it to strategically prioritize which variables to replace first. This combination of insider knowledge about model internals and OIS-guided variable selection enables ALERT to craft adversarial examples that are specifically optimized to exploit the weaknesses of each model’s learned representations.

However, this advantage comes at a significant practical cost. First, ALERT’s architecture-specific design limits its applicability—we could not apply ALERT to two additional models in our study (DeepSeek-Coder and Code Llama) because they were not pre-trained with an MLM objective, which is fundamental to ALERT’s substitute generation mechanism. This architectural dependency highlights a critical limitation: ALERT cannot generalize to models with different pre-training objectives. Second, in real-world code authorship attribution scenarios, such gray-box attacks are impractical because victim models are typically deployed remotely as black-box services [50], [51]. Attackers would only have query access to model predictions without any access to internal representations, model parameters, or the underlying pre-trained architecture. As noted by [18], attacks that rely on gradient information or internal model access have limited applicability in deployed systems where models are protected behind APIs. Our GPT-4-based method, despite achieving lower ASRs, represents a more realistic threat model as it operates under true black-box conditions—requiring only query access to model predictions. This makes our approach more practical and generalizable across different authorship attribution systems, regardless of their underlying architectures, pre-training objectives, or deployment configurations.

6 Result Discussion↩︎

In this section, we discussed our important findings from all research questions.

6.1 What are the common causes of misattribution?↩︎

We analyzed all misattributed samples across datasets and models, examining: (1) intra-author vs. inter-author similarity using cosine similarity of code embeddings, (2) stylometric variation using Jensen-Shannon Divergence (JSD), and (3) code snippet length distribution. This systematic breakdown reveals the underlying causes of attribution failures. Primary error causes are:

• Overlapping Author Styles: Figure 13 shows the distribution of intra-author and inter-author cosine similarities for misclassified samples. The two distributions exhibit substantial overlap (JSD = 0.016), indicating that misattributed code pairs have similarity scores indistinguishable from same-author pairs. For example, in the LeetCode C++ dataset, authors frequently converge on nearly identical solutions due to: (a) algorithmic constraints imposed by competitive programming problems, (b) language idioms that encourage specific patterns (e.g., STL usage), and (c) shared educational backgrounds leading to similar coding habits. Manual inspection of confused author pairs reveals cases where both authors use identical variable naming conventions (e.g., ‘i’, ‘j’ for loops; ‘ans’ for results), similar whitespace patterns, and equivalent control flow structures.

• Short Code Snippets : Figure 4 demonstrates a strong relationship between snippet length and attribution accuracy across models. CodeBERT (Figure 2) shows accuracy increasing from 0.45 for snippets under 50 tokens to 0.64 for snippets over 1000 tokens—a 42% improvement. DeepSeek-Coder exhibits an even more pronounced pattern, with accuracy rising from 0.50 for the shortest snippets (\(<\)​50 tokens) to near-perfect performance (1.0) for snippets exceeding 2000 tokens—a 100% improvement. This trend is consistent across all models, though the magnitude varies: smaller models (CodeBERT, GraphCodeBERT) show gradual improvement (0.45→0.64), while larger models (DeepSeek, CodeLlama) exhibit steeper gains (0.50→1.0). Short snippets lack sufficient stylometric signals—they often contain only basic syntactic structures such as variable declarations, simple conditionals, or single-line operations that are nearly identical across authors regardless of individual style. For example, a 30-token snippet might consist solely of ‘int result = 0; for(int i=0; i<n; i++) result += arr[i];’—a pattern ubiquitous across authors with minimal stylistic variation. As snippet length increases, more distinguishing features emerge: function decomposition strategies, error handling patterns, commenting habits, naming convention consistency, and algorithmic approach diversity. (38% of misattributions)

6.2 Why do LMs outperform PbNN?↩︎

Language models significantly outperform PbNNs in code authorship attribution due to their pre-trained embeddings that capture rich semantic and syntactic code patterns learned from massive code corpora. Unlike PbNNs, which learn embeddings from scratch during training on limited labeled data, LMs leverage transfer learning from large-scale pre-training, enabling them to extract more discriminative authorship features.

To empirically validate this advantage, we analyzed the quality of learned representations by measuring the separability between intra-author (code samples from the same author) and inter-author (code samples from different authors) similarity distributions (Methodology is discussed in Section 4.2). For each model, we computed cosine similarity between code embeddings and measured the Jensen-Shannon (JS) divergence between the intra-author and inter-author similarity distributions. Higher JS divergence indicates better author discrimination capability, as it reflects clearer separation between the two distributions.

As shown in Figure 10, DeepSeek-Coder achieves JS divergence of 0.992 (Github-Java) and 0.666 (LeetCode), compared to PbNN’s 0.924 and 0.181 respectively. DeepSeek-Coder shows clear bimodal separation with intra-author similarities clustered near 1.0 and inter-author near 0.0, while PbNN exhibits substantial overlap between distributions. This superior separability directly translates to better classification performance, explaining LMs’ advantage in code authorship attribution. We also observe the same trend in other datasets: PbNN’s JS Divergence is lower than other models.

6.3 Why are smaller LMs are more robust? and Why are LMs are less vulnerable to adversarial attack?↩︎

Mechanistic Analysis of Adversarial Robustness: To understand why smaller LMs demonstrate greater robustness to adversarial attacks compared to larger models and traditional ML/DL approaches, we conducted a mechanistic analysis using Integrated Gradients (IG) [31]. Our goal was to determine whether robustness differences stem from varying attention patterns across AST token categories. We applied the following procedure to analyze each model’s decision-making process:

1. Token-level Attribution: For all correctly classified samples in the test set, we computed IG attribution scores for each token, quantifying its contribution to the model’s prediction.

2. Category Mapping: Each token was mapped to one of 14 AST categories based on tree-sitter parsing: Comments, Identifiers, Keywords, Operators, Punctuation, Literals, Types, Modifiers, Declarations, Statements, Expressions, Preprocessor, Patterns, and Special tokens (Table 10).

3. Category-level Aggregation: We calculated the mean attribution score for each category across all samples, providing insight into which syntactic and semantic features each model prioritizes for authorship attribution.

4. Transformation Impact Analysis: We cross-referenced these attribution patterns with our adversarial transformation taxonomy (Table 11), which documents which AST categories each transformation modifies.

This analysis enables us to test the hypothesis that models with distributed attention across many categories are more robust to targeted perturbations than models with concentrated attention on few categories.

Figure 15 and 16 presents the mean attribution scores across AST categories for eight models. The results reveal a clear dichotomy in attention patterns that directly explains robustness differences.

Smaller Models: Distributed Attention Strategy. CodeBERT, GraphCodeBERT, ContraBERT_C, and ContraBERT_G (Figure 15a-d) exhibit relatively uniform attribution distributions across most AST categories. For instance, CodeBERT assigns substantial importance to Comments, Statements, Patterns, Operators, and Literals, with no single category dominating. Similarly, GraphCodeBERT distributes attention across Operators, Types, Punctuation, and Preprocessor. This distributed attention pattern means these models consider holistic code structure rather than relying on specific stylometric markers.

Larger Models and ML/DL: Concentrated Attention Strategy. In contrast, larger models (Code Llama, UniXcoder, DeepSeek-Coder) and the traditional DL baseline (PbNN) show highly concentrated attention on specific categories (Figure 16a-d). Code Llama focuses predominantly on Keywords, with minimal attention to most other categories. UniXcoder strongly prioritizes Identifiers and Special tokens, while largely ignoring syntactic structures. DeepSeek-Coder concentrates on Special tokens and Modifiers. PbNN exhibits extreme concentration on Preprocessor and Comments, with near-zero attribution to structural elements.

Connection to Adversarial Robustness. This dichotomy directly explains the observed robustness differences. As shown in Table 11, individual adversarial transformations modify only a subset of AST categories—typically 2-9 categories per transformation. For example:

Since our attack protocol applies only one transformation at a time, adversarial samples preserve many original AST categories unchanged. Models with distributed attention can still leverage untouched categories for attribution, maintaining prediction accuracy. For instance, when comments are removed, CodeBERT can still rely on its substantial attention to Statements, Patterns, and Operators. Conversely, models with concentrated attention are vulnerable when adversarial transformations target their focal categories. When PbNN encounters code with comments removed—one of its two dominant features—it loses critical discriminative information, causing misattribution. Similarly, if identifier naming conventions are altered (affecting UniXcoder’s primary focus), the model’s performance degrades significantly.

These findings reveal that adversarial robustness in CAA is not solely a function of model size or architectural sophistication, but rather depends on feature utilization strategy. Smaller models, with limited capacity, are forced to extract signal from diverse code features, inadvertently learning more generalizable and robust representations. Larger models, with greater capacity, can achieve higher clean accuracy by specializing on a few highly discriminative features, but this specialization creates vulnerabilities.

6.4 Why does adversarial pre-training improves robustness?↩︎

ContraBERT enhances existing pre-trained models through contrastive learning with semantic-preserving augmentations. Given original code-comment pairs (C, W), nine augmentation operators generate semantic-preserving variants (C’, W’) through transformations like Rename Variable, Rename Function, Reorder Statements, Insert Dead Code, and Back Translation. The model is then trained with two objectives: (1) Masked Language Modeling (MLM) to predict masked tokens, and (2) Contrastive Learning using InfoNCE loss, which maximizes similarity between original samples and their augmented variants (positive pairs) while minimizing similarity to other samples in a dynamic queue (negative pairs). This contrastive objective explicitly trains the model to treat semantically equivalent code variants as similar, forcing it to learn deep structural and syntactic features rather than surface-level patterns like variable names or statement ordering.

To validate how ContraBERT’s contrastive learning enhances stylometric feature learning for code authorship attribution, we compared attack success rates across three common code transformations that mirror ContraBERT’s training augmentations: Rename Variable, Rename Function, and Reorder Statement. These semantic-preserving transformations directly test whether the model has learned to rely on stable stylometric features (syntax and structure) rather than surface-level patterns (identifier names and statement order). We evaluated seven models: Code Llama, DeepSeek-Coder, UniXcoder, CodeBERT, GraphCodeBERT, ContraBERT_C (based on CodeBERT), and ContraBERT_G (based on GraphCodeBERT). Figure 14 presents the attack success rates across all models and transformations. ContraBERT models demonstrate substantially lower vulnerability to identifier renaming and statement reorder attacks, confirming that the contrastive objective enhances stylometric robustness.

7 Related Work↩︎

Mohammed et al. [1], [16] utilized both ML and DL to address CAA. They employed Recurrent Neural Networks (RNNs) to extract TF-IDF-based stylometric features and used a Random Forest Classifier (RFC) for authorship attribution. In a separate study, they applied Convolutional Neural Networks for the same purpose [17]. Alsulami et al. [2] proposed using a LSTM [52] network to extract relevant features from the AST representation of programmers’ source code, followed by a SoftMax Layer for authorship attribution. Although their study reported high accuracy, it did not assess the model’s resilience against adversarial attacks. Bogomolov et al. [4] introduced a Path-based Neural Network (PbNN) for authorship attribution by modifying the code2vec neural network architecture, focusing on the relative term frequencies of tokens and paths in the abstract syntax tree (AST). Burrows et al. [53], [54] used n-gram features extracted from source code to train a Support Vector Machine (SVM) for the same purpose. These studies also incorporated various manual feature engineering techniques to extract stylometric features such as byte-level information and program dependence graphs from source code [13], [14], [55]. Despite reporting high accuracy, these studies did not examine their models’ performance under adversarial attack.

Addressing the gap in evaluating resistance to adversarial attacks, we found several studies that did assess this aspect. Quiring et al. [18] designed an adversarial attack strategy for CAA using Monte-Carlo tree search, which dramatically reduced the model’s accuracy from 88% to 1%. Other studies have also showed that automated CAA models are susceptible to adversarial attacks [12], [15], [56]–[58]. In this study, we follow the similar methodology to perform adversarial attack on LMs.

Although LMs have demonstrated promising results in various SE tasks such as code completion [19], code summarization [59], code translation [60], and bug triaging [3], their application in CAA has been relatively unexplored. Yang et al. [32] introduced ALERT, a method that generates adversarial samples by renaming method names. They then fine-tuned and targeted CodeBERT and GraphCodeBERT in their attacks, affecting Vulnerability Prediction, Clone Detection, and Authorship Attribution. However, their approach to perturbing code, limited to renaming method names, is considerably less extensive than our 28 transformation rules. Additionally, while ALERT tested LMs performance using monolingual datasets, our evaluations utilize multilingual datasets, offering a more comprehensive analysis of LMs under varied conditions.

A small portion of this work was previously published as an extended abstract for an ACM SRC competition [61]. This presented preliminary results, focusing on encoder-only models with parameter sizes in the hundreds of millions. In the SRC paper, no hyper-parameter fine-tuning was performed, the experiments reported accuracy and adversarial success rates on a more limited version of our LeetCode dataset for only a single model (UnixCoder), there was no comparison to state-of-the-art (SOTA) baselines, and finally, interpretability techniques were not applied.

8 Threats to Validity↩︎

It is known that language models like GPT-4 provide non-deterministic outputs and may produce slightly different adversarial samples upon reproduction. To mitigate this threat, we use a low temperature (0), as suggested by [62].

Another potential threat to reproducibility arises from our reliance on GPT-4—a proprietary tool—for adversarial sample generation, rather than open-source alternatives like Llama or DeepSeek. While we attempted to use Llama, most of its adversarial samples were rejected by LeetCode’s submission system, which is crucial for verifying the semantic correctness of adversarial transformations. DeepSeek provides free access through its web interface, but its API is paid, making it impractical for generating adversarial samples for 114,016 code snippets. We instead use GPT-4o’s API, which is free through the GitHub Copilot plugin. Thus, researchers can reproduce our results using GitHub Copilot’s plugin.

A further validity threat is that adversarial samples may reflect GPT-4’s stylistic tendencies rather than those of a real human attacker. To mitigate this, we conduct a human evaluation of adversarial samples that significantly impact model performance. These samples, verified by our parser and LeetCode’s submission system, cover all transformation rules. For this human evaluation, we followed similar setup of by [32] for evaluating the naturalness of adversarial samples.

We employ a two-step evaluation process to assess adversarial sample quality. First, one of the authors manually verifies a statistically representative sample (95% confidence level, n=370), assessing whether: (i) GPT-4 followed the supplied transformation prompt, (ii) the adversarial code contains extraneous changes, and (iii) the transformed code remains plausible, readable, and free from obfuscation or junk code. Then, four PhD students in software engineering independently evaluate disjoint subsets of adversarial samples using the same criteria.

Evaluation Protocol. Each evaluator assessed samples on three dimensions:

• Transformation adherence: Whether GPT-4 correctly applied the specified transformation

• Code cleanliness: Absence of extraneous modifications beyond the intended transformation

• Plausibility: Whether the code appears human-written, rated on a 3-point scale:

  • Plausible: Indistinguishable from typical human-written code

  • Moderately Plausible: Somewhat realistic but contains minor artifacts

  • Not Plausible: Contains obvious synthetic or unnatural patterns

After independent reviews, evaluators discussed disagreements to reach consensus through negotiated agreement. Inter-rater agreement between the first author and the four PhD students was high, with agreement rates of 90%, 97%, 92%, and 96% respectively (mean: 93.75%), indicating reliable evaluation.

Findings. Through this process, all evaluators agreed that GPT-4 follows the supplied prompts and that our parser effectively verifies transformations. Additionally, evaluators noted that GPT-4 tends to remove comments while applying transformations. To prevent unintended bias, we exclude such cases from the adversarial attack experiments.

Regarding plausibility, the evaluators confirmed that the generated adversarial samples are generally realistic. However, samples involving add print/log statements transformations received lower plausibility ratings: while 96.57% were rated as "Plausible," 3.43% were classified as "Moderately Plausible" after consensus review. Manual inspection revealed these cases involved excessive logging statements (e.g., \(>\)​5 print statements in the snippets), which appeared unnatural. Importantly, even these "Moderately Plausible" samples remained syntactically correct and executable—they simply exhibited logging density atypical of production code.

All other transformation types (variable renaming, code reordering, comment modification, etc.) achieved \(>\)​100% "Plausible" ratings, indicating high realism across diverse adversarial perturbations. The 3.43% of moderately plausible samples represent a minor limitation but do not significantly affect our conclusions for the following three reasons.

1. These cases constitute a small fraction of total adversarial samples.

2. They remain syntactically valid and pass LeetCode automated testing, demonstrating functional correctness.

Our adversarial robustness analysis shows that larger models (e.g., Code Llama, DeepSeek-Coder) exhibit higher vulnerability to adversarial attacks. This pattern may indeed reflect a tendency of larger models to overfit to specific patterns during fine-tuning, making them less robust to perturbations. However, our results also suggest that robustness is influenced not only by model size but also by training schemes and architectures (e.g., contrastive learning in ContraBERT models). In addition, we did take general measures to combat overfitting, including K-Fold cross-validation, early stopping, and dropout regularization during fine-tuning to reduce neuron co-adaptation. These combined strategies should reduce, though not entirely eliminate, the risk of overfitting.

We also acknowledge that rule-based transformation tools offer higher efficiency and full determinism for syntactic transformations compared to our GPT-4-based approach. However, rule-based tools are typically language-specific and support a narrower range of transformations. Our GPT-4-based pipeline trades efficiency for breadth and language-agnosticism, enabling 28 transformation rules across multiple programming languages in a unified framework. We consider the efficiency gap a limitation, and future work could explore hybrid approaches that combine rule-based tools for common syntactic transformations with LLM-based generation for higher-level structural changes.

Finally, we acknowledge that our baseline comparisons rely primarily on PbNN [4], a 2021 model. While one more recent CAA baseline exists [12], replicating it within our experimental setup proved challenging due to language specificity and architectural constraints. We consider this a limitation of our study. Future work should incorporate more recent baselines as they become available with reproducible implementations, to more precisely contextualize the advantage of LMs for CAA.

9 Lessons Learned and Future Directions↩︎

While our study primarily focuses on empirically reassessing the effectiveness, stylometric understanding, and robustness of LMs for Code Authorship Attribution (CAA), our findings suggest several concrete methods for improving future CAA systems:

1. Ensemble Modeling. Our analysis reveals that different LMs capture orthogonal stylistic features. This suggests that an ensemble approach can enhance overall attribution accuracy and coverage. Recent work on bug triaging has demonstrated that majority voting among multiple LLMs significantly improves classification performance [63]. Building on this success, we propose extending beyond simple majority voting to weighted aggregation strategies where each model’s contribution is proportional to its performance.

2. Contrastive Pre-training. We observe that contrastive learning substantially improves robustness against adversarial attacks. This finding motivates the design of advanced contrastive pre-training objectives tailored for CAA. Specifically, we propose generating positive pairs from style-preserving paraphrases of the same author and hard negatives from adversarially perturbed texts or different authors with similar topics. The objective would minimize \(\mathcal{L} = -\log\frac{\exp(\text{sim}(z_i, z_i^+)/\tau)}{\sum_{j} \exp(\text{sim}(z_i, z_j)/\tau)}\) where \(z\) represents contextualized embeddings, enabling models to focus on robust stylistic features invariant to adversarial perturbations.

3. Handling Short and Imbalanced Samples. Our results on the LeetCode dataset highlight challenges posed by short, imbalanced code snippets with low stylometric variation. Future models could mitigate this issue using data augmentation—such as retrieving contextually similar code from large corpora—or by leveraging multi-view learning that integrates structural, lexical, and semantic representations of code.

4. Adversarial Robustness. Given the susceptibility of large LMs to transformations that perturb a significant portion of the code, we advocate for training with adversarial data augmentation pipelines and syntactic regularization techniques. These strategies can improve model generalization across both benign and adversarially modified inputs.

These insights provide a foundation for developing more accurate, generalizable, and resilient CAA systems beyond the scope of our current evaluation.

10 Ethical Considerations↩︎

10.1 Privacy and Consent↩︎

Code authorship attribution, while valuable for software engineering tasks, raises significant privacy concerns when applied to identify individuals. We acknowledge the following ethical considerations:

Privacy Risks. CAA techniques can attribute code to individuals without their explicit consent, potentially exposing developers to unwanted identification or surveillance. In forensic and cybersecurity contexts, this capability could be misused for:

• Unauthorized profiling of developers based on their coding patterns

• Identifying whistleblowers or anonymous contributors to sensitive projects

• Tracking developers across projects without consent

• De-anonymizing open-source contributors who prefer pseudonymity

Responsible Data Usage. Our study exclusively uses publicly available datasets where code was shared under open-source licenses. However, we emphasize that deployment of CAA systems must adhere to:

• Informed consent from developers whose code is analyzed

• Compliance with data protection regulations (e.g., GDPR, CCPA)

• Clear disclosure of CAA usage in terms of service agreements

• Respect for developers’ right to pseudonymity in appropriate contexts

10.2 Potential Misuse and Mitigation↩︎

While CAA has legitimate applications in bug triaging, plagiarism detection, and copyright disputes, the technology could be misused: Surveillance and Discrimination. CAA could enable:

• Employer surveillance of developers’ external contributions

• Discriminatory practices based on inferred developer characteristics

• Unauthorized attribution in competitive or adversarial contexts

Mitigation Strategies. We recommend:

• Purpose Limitation: CAA systems should be deployed only for stated, legitimate purposes with appropriate oversight

• Regulatory Frameworks: Organizations deploying CAA should establish clear policies on acceptable use, similar to biometric identification systems

• Transparency: When CAA is used in production systems (e.g., plagiarism detection), users should be informed of its presence and capabilities

10.3 Dual-Use Nature↩︎

This research has a dual-use character: while it advances beneficial SE automation and forensic capabilities, it also provides techniques that could infringe on privacy. We position this work as advancing scientific understanding of code stylometry, with the expectation that practitioners will deploy these methods responsibly and within appropriate legal and ethical frameworks.

11 Conclusion↩︎

In this paper, we performed the first large-scale, systematic study of applying transformer-based LMs to the task of Code Authorship Attribution. Our investigation included seven different LMs across multiple datasets covering diverse languages and coding styles. We also conducted an in-depth analysis of these models’ robustness to adversarial attacks. Our results show that LMs generally excel at CAA when properly fine-tuned, with larger models like Code Llama and DeepSeek-Coder providing superior performance on multilingual datasets that exhibit significant variance. Surprisingly, even smaller models such as CodeBERT and GraphCodeBERT often perform competitively, especially in structured, monolingual scenarios. We also found that LMs—particularly those pre-trained with contrastive objectives—are substantially more robust to adversarial perturbations compared to earlier neural techniques.

12 Artifacts↩︎

We make all of our code and data available in an online appendix [47] to facilitate the replication and reproducibility of our work, and to encourage future research on adapting transformer-based LMs to the task of code stylometry.

13 Adversarial Rules as Prompts↩︎

A. Statement Transformation

1. Given the code snippet below, find instances where single line variable declarations can be split into multiple lines, and modify the code to use the multiple line declarations. If there is no single line declaration, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, find statements where variables are declared across multiple lines, and modify the code to put those declarations into a single line. If there is no such declaration, just return ‘NA’. Otherwise, return the modified code only.

3. Given the code snippet below, find statements where the order of execution does not impact the program logic. Then swap the order of these statements. Finally, return the whole code with modifications.

B. Name Transformation

1. Given the code snippet below, identify if there is a particular style for variable naming, such as camel case or snake case. If there is a style for the given variable names, then change all variable names to a different style. For example, changing all instances of the variable plus_one to plusOne, representing a shift from snake case to camel case. If there is no style convention for variable names, just return ‘NA’.

2. Given the code snippet below, identify if there is a particular style for function naming, such as camel case or snake case. If there is a style for the given function names, then change all function names to a different style. For example, changing all instances of the function plus_one to plusOne, representing a shift from snake case to camel case. If there is no style convention for function names, just return ‘NA’.

C. Operator Transformation

1. Given the code snippet below, find all instances of relational operators and swap them. For example, ’ a > b ’ to ’ b < a ‘. If there are no relational operators, just return ’NA’. Otherwise, return the modified code only.

2. Given the code snippet below, identify and convert integer literals into expressions that represent the same literal value. For example, ’ int b = 8 ’ would be transformed into ’ int b = 2 * 4 ‘. If there are no integer literals, just return ’NA’. Otherwise, return the modified code only.

3. Given the code snippet below, identify and convert integer literals into expressions that represent the same literal value. For example, ’ int b = 8 ’ would be transformed into ’ int b = 2 * 4 ‘. If there are no integer literals, just return ’NA’. Otherwise, return the modified code only.

4. For the code snippet below, identify and change the style of any increment or decrement operators. For example, ’ i++ ’ would change to ’ i +=1’ or ’ i– ’ would change to ’ i -= 1 ‘. If there are no increment or decrement operators, just return ’NA’. Otherwise, return the modified code only.

5. For the code snippet below, identify and change the style of any increment or decrement operators. For example, ’ i +=1’ would change to ’ i++ ’ or ’ i -= 1 ’ would change to ’ i– ’ . If there are no increment or decrement operators, just return ‘NA’. Otherwise, return the modified code only.

D. Data Transformation

1. Given the code snippet below, identify and convert all integer numbers into hexadecimal values. If there are no integer numbers, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, convert all character literals into their corresponding ASCII values. If there are no character literals, just return ‘NA’. Otherwise, return the modified code only.

3. Given the code snippet below, identify any string variables and modify their declaration to use a character array. If there are no string variables, just return ‘NA’. Otherwise, return the modified code only.

4. Given the code snippet below, identify and convert between boolean literals and integer literals. For example, if a ‘true’ value is used change this to a 1 and if a ‘false’ value is used change this to a 0. If there are no boolean values used, just return ‘NA’. Otherwise, return the modified code only.

E. Loop Transformation

1. Given the code snippet below, identify and change any ‘for’ statements into ‘while’ statements. If there are no ‘for’ statements, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, identify and change any ‘while’ statements into ‘for’ statements. If there are no ‘while’ statements, just return ‘NA’. Otherwise, return the modified code only.

F. Control Flow Transformation

1. Given the code snippet below, identify and convert any eligible ‘if-else’ statements to ‘switch’ statements. If there are no ‘if-else’ statements that can be converted into ‘switch’ statements, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, identify and convert any eligible ‘switch’ statements to ‘if-else’ statements. If there are no ‘switch’ statements that can be converted into ‘if-else’ statements, just return ‘NA’. Otherwise, return the modified code only.

3. Given the code snippet below, identify and convert any ‘if-else’ statements that can be rewritten using the ‘ternary’ operator. For example, ’ if (a > b) max = a; else max = b; ’ would be converted into ’ max = a > b ? a : b; ‘. If there is no such statement, just return ’NA’. Otherwise, return the modified code only.

4. Given the code snippet below, identify and convert any ‘if-else’ statements written using the ‘ternary’ operator to ‘if-else’ statements that do not use the ‘ternary’ operator. For example, ’ max = a > b ? a : b; ’ would be converted into ’ if (a > b) max = a; else max = b; ‘. If there is no such statement, just return ’NA’. Otherwise, return the modified code only.

5. Given the code snippet below, identify any ‘if-else’ statements and swap the statements contained within the if conditional to the else conditional. If there are no ‘if-else’ conditional statements, just return ‘NA’. Otherwise, return the modified code only.

G. Function Transformation

1. Given the code snippet below, identify function declarations and invocations, and swap the order of the parameters for both the declarations and invocations. If there are no function declarations, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, identify function declarations and invocations, and add one extra integer parameter with a default value of zero. If there are no function declarations, just return ‘NA’. Otherwise, return the modified code only.

3. Given the code snippet below, identify groups of statements that could be rewritten into a function, and create a function and any necessary invocations. Finally, return the modified code Only.

4. Given the code snippet below, identify any function declarations, and swap the order of their declarations. If there are one or fewer function declarations, do not modify the code. If there is no such declaration, just return ‘NA’. Otherwise, return the modified code only.

H. Miscellaneous

1. Given the code snippet below, please remove all comments. If there is no comment, just return ‘NA’. Otherwise, return the modified code only.

2. Given the code snippet below, please remove unused code including variables, functions, libraries. Only remove code that is never used for computational logic. If there is no unused code, just return ‘NA’. Otherwise, return the modified code only.

3. Given the code snippet below, please add print/log statements at every point where a variable is initialized or its value modified. Finally, return the modified code only.

1pt

14 Details of LeetCode Dataset↩︎

As shown in Table 9, the dataset exhibits two primary characteristics:

1. Language imbalance: C++ (62.19%) and Java (24.82%) dominate the dataset, while Python (6.77%), JavaScript (3.78%), C# (2.01%), and Ruby (0.30%) are underrepresented.

2. Author imbalance: Within each language, author contribution varies substantially, as evidenced by the high standard deviations of samples per author (ranging from 4.99 to 23.86) and the large gaps between minimum and maximum contributions. For instance, in C++, individual authors contributed between 1 and 150 samples (std: 21.05), while in C#, the range is 1 to 56 samples with only 4 authors (std: 23.86).

0.4pt

CMT=Comments, IDN=Identifiers, KWD=Keywords, OPR=Operators, LIT=Literals, TYP=Types, DCL=Declarations, STM=Statements, EXP=Expressions, MOD=Modifiers, PAT=Patterns, PUN=Punctuation, SPL=Special, PRP=Preprocessor

References↩︎