1 Introduction↩︎

The rapid advancement of large language models has enabled widespread generation of human-like text [1]–[4]. As AI-generated text (AIGT) becomes increasingly indistinguishable from human text, reliable detection is critical for academic integrity, moderation, misinformation analysis, and provenance tracking.

A large body of prior work has shown that clean-text AIGT detection is possible, but robustness remains a major bottleneck. Many detectors rely on statistical regularities, token-level artifacts, or fine-tuned transformer classifiers [5], [6]. These systems can degrade sharply under paraphrasing, synonym substitution, or small character edits [1], [7], [8]. In practice, generated text is often lightly edited before use, which makes clean-only accuracy an incomplete measure of detector quality. Unfortunately, most detectors fail to capture the nuanced stylistic differences distinguishing human vs. machine authors [7], and paraphrasing attacks can easily bypass most detectors [1]. Such vulnerabilities, exacerbated by commercial "humanizers", render current systems brittle for practical deployment [4], [7], [9].

Our work is motivated by a simple observation: source attribution should rely less on brittle surface cues and more on stylistic regularities that survive modest editing. Recent contrastive and style-based approaches already suggest that such signals are useful. We therefore present OURMODEL, a modest but effective extension of T5-Sentinel [6]. The model retains T5-Sentinel’s next-token prediction formulation, but augments training with a margin-based triplet loss over style embeddings so that texts from the same source class are pulled together while texts from different source classes are pushed apart.

We use lightweight perturbations as scalable proxies for real-world editing. Our approach is based on prior work that shows AIGT detectors such as DetectGPT and ‘OpenAI Text Classifier’ exhibit greater AuROC degradation under word-substitution, often pushing performance below random guessing (50% AuROC), while paraphrasing is less potent and has a milder effect [1], [10]. Furthermore, word substitution is a commonly used strategy in generating textual adversarial examples [10] Thus, in this work we focus on the harder task of AIGT detection in the presence of word and character-level perturbations (including synonym replacements up to 80% intensity). We consider these as effective [8] lightweight realistic [11] proxies for real-world text polishing  [8] and paraphrasing, as they mimic the lexical substitutions common in evasion attacks while enabling scalable ablation across large test suits.

Our experiments consider synonym substitutions and character-level operations (swap, replace, insert, delete). These perturbations do not cover the full space of human rewriting, but they are useful stress tests because they isolate lexical and orthographic changes that frequently appear in evasion attempts [8], [11].

Our contributions

2 Related Works↩︎

AIGT detection methods range from statistical and zero-shot approaches to fully supervised classifiers. Early systems based on n-gram statistics, entropy, or distributional features were reasonably interpretable, but they struggle with modern LLMs that produce fluent and diverse text [1]. DetectGPT [2] uses curvature under perturbation for zero-shot detection; RADAR [12] improves paraphrase robustness through adversarial training; SCRN [3] uses reconstruction-based robustness; and PAWN [13] reweights suspicious tokens by perplexity-derived signals. T5-Sentinel [6] is particularly close to our work: it reframes detection as next-token prediction with author tokens, but it does not explicitly regularize style representations or enforce style consistency. However, both Commercial and open-source methods (TurnItIn [9], GPTZero [14], ZeroGPT [15]) reporting highly varying accuracies [4].

Contrastive and style-aware learning has been widely used to learn discriminative representations for sentences, semantics, and writing style [16]–[20]. In AIGT detection, TopFormer [21] applies triplet loss to deepfake text attribution. NBCSoftmax uses block contrastive loss [20] for style based author detection in micro text samples. DeTeCtive [22] employs multi-task and multi-layer contrastive learning, and can optionally incorporate training-free incremental adaptation (TFIA) for certain out-of-distribution scenarios.

While these works already support the broader idea that style cues are useful they have shortcomings, current AIGT detectors struggle with minor adversarial perturbations such as word substitutions or character swaps [3]. For example, mimicking paraphrasing by shortening "California" to "Calif." can cause misclassifications, highlighting the limits of brittle token-level features. Simple word substitution attacks degrade DetectGPT [2] and ‘OpenAI Text Classifier’ [23] dropping AUROC below random guessing (of 50%) [10]. Although paraphrasing has a milder effect [10], the accuracy of DetectGPT drops from 70.3% to 4.6% and the accuracy of OpenAI text classifier drops from 30.0% to 15.6%  [1]. AIGT detectors in general misclassify perturbed texts as human, resulting in accuracy \(\approx50\%\) [8].

Authorship attribution and style-based source attribution is closely related to our task. However, it differs in one important respect: all human authors are collapsed into a single heterogeneous Human class, while each LLM is treated as one source class. This makes the human class intrinsically broader and harder to separate than any individual machine source. This perspective is important when interpreting class-wise performance and when comparing AIGT detection to classical authorship attribution.

In positioning our work, OURMODEL should be viewed as a lightweight yet effective enhancement to the T5-Sentinel framework [6] employing an auxiliary style-aware contrastive regularization on the decoder’s final hidden-layer embeddings. The core contribution lies in demonstrating that a simple batched margin-based triplet loss, when combined with the original next-token prediction objective, is sufficient to learn perturbation-resistant stylistic signatures without requiring architectural modifications, adversarial training, or multi-task objectives. This yields consistent gains in both in-distribution accuracy and robustness to word/character substitution attacks, while simultaneously achieving state-of-the-art out-of-distribution and paraphrasing performance on the challenging MAGE benchmark.

3 sec:Proposed32Model↩︎

3.1 Model Architecture.↩︎

OURMODEL is based on the T5-small encoder-decoder, enhanced for perturbation resistant AIGT detection via contrastive style learning. Following [6], we reformulate detection as a next-token prediction task, but incorporate author stylistic semantics. Each label is represented by an author token, representing a source class rather than an individual person: for example, <Human>, <GPT-3.5>, <LLaMA>, <PaLM>, <GPT2> on OpenLLMText. This formulation lets us reuse T5’s text-to-text interface, its pre-training and avoids the need for an explicit classification head. Given an input text \(s\), the encoder processes the sequence and the decoder generates the most likely author token. Our key innovation lies in the training procedure, where we apply a additional constraint on last decoder embedding layer, immediately before the linear projection to vocabulary logits, which is able enhance rich stylistic signals shaped by both the encoder context and the decoder’s auto-regressive processing.

3.2 Loss Function.↩︎

The training objective combines two complementary losses: cross-entropy for accurate classification and margin-based triplet loss for stylistic discrimination Let \(\texttt{LM}: \Sigma^* \times \Sigma \to \mathbb{R}\) represent our language model, where \(\texttt{LM}(s, \sigma)\) estimates the probability of token \(\sigma\) following input string \(s\). We establish a mapping \(f: Y \to \mathcal{Y}\) from label set \(Y\) to proxy tokens \(\mathcal{Y} \subset \Sigma\), reformulating classification as: \[\hat{y} = f^{-1}\left( \arg\max_{y \in \mathcal{Y}} \texttt{LM}(s, y) \right)\]

The cross-entropy loss, where \(G(F(s))\) denotes the logits produced from the decoder embedding \(F(s)\), is:

\[\mathcal{L}_{CE} = -\sum_{i \in \mathcal{Y}} y_i \log(G(F(s))_i)\] where \(y_i=1\) if correct label and 0 otherwise. However, under lexical edits (synonym substitution, character swap/insert/delete), these surface cues are disrupted, causing the decoder states \(F(s)\) to shift in ways that can flip the argmax over author tokens.

Our key addition is augmenting the objective with a margin-based triplet loss applied directly on the decoder embeddings to encourage robust stylistic representations:

\[\begin{align} \mathcal{L}_{triplet} = \sum \max \Big( 0,\; &d(F(s_{a}), F(s_{p})) \\ &- d(F(s_{a}), F(s_{n})) + m \Big) \end{align}\] where \(s_a\) is the anchor sample, \(s_p\) is a positive sample from the same author class, \(s_n\) is a negative sample from a different class and \(d(\cdot, \cdot)\) denotes Euclidean distance. The margin \(m=1.0\) acts as a safety buffer around the decision boundary, ensuring that even under perturbations of magnitude up to \(m/2\), the relative ordering of distances is preserved.

The final training loss is the weighted combination \[\mathcal{L} = (1-\lambda)\mathcal{L}_{CE} + \lambda \mathcal{L}_{triplet},\] where \(\lambda = 0.5\) balances classification accuracy and stylistic discrimination.

3.3 Theoretical Analysis.↩︎

We analyze why augmenting next-token prediction objective with a margin-based triplet loss on decoder embeddings improves both accuracy and robustness in source attribution and AIGT detection. Pure cross-entropy training encourages accurate prediction of the correct author token but primarily optimizes surface-level token prediction. Under lexical edits (synonym substitution, character swap/insert/delete), these surface cues are disrupted, causing the decoder states \(F(s)\) to shift in ways that can flip the argmax over author tokens.

Minimizing \(\mathcal{L}_{triplet}\) explicitly pulls decoder representations of same authors closer while pushing representations from different authors apart by at least margin \(m\). As the embeddings are extracted from the decoder’s final layer, after cross attention, they encode more global stylistic signals to aid predict the author token. The triplet term regularizes this space so that stylistic regularities (such as patterns in conjunctions, transitional phrases, and topic-invariant function words) become more dominant and compact within each author cluster.

This geometric regularization directly impacts classification error through the Bhattacharyya bound. Consider the class-conditional distributions of decoder embeddings \(p_c(z)\) with means \(\mu_c\) and covariances \(\Sigma_c\). For any pair of classes \(c\) and \(c'\), the Bhattacharyya bound on the pairwise Bayes error is: \[\epsilon_{c,c'} \leq \sqrt{P(c)P(c')} \exp\left(-\frac{d^2(\mu_c, \mu_{c'})}{8\sigma^2}\right),\]

The triplet loss optimizes embedding geometry by: (1) reducing the average intra-class variance \(\Sigma_c\) (related to \(\text{tr}(\Sigma_c)\)), and (2) maximizing inter-class separation \(d(\mu_c, \mu_{c'})\) for \(c \neq c'\). Both effects tighten the exponent in the Bhattacharyya bound, lowering the upper bound on pairwise error. In the multiclass setting with a heterogeneous “Human” class, improved separation across all pairs contributes to better overall source attribution. While this bound applies to pairwise classification, improved separation between all class pairs contributes to better overall multiclass performance.

3.4 Robustness Analysis↩︎

The geometric regularization induced by the triplet loss provides a principled explanation for the observed robustness of OURMODEL under adversarial perturbations. When a text \(s\) undergoes realistic perturbations such as synonym substitution of adjectives or nouns, or character-level operations (swap, replace, insert, delete), the input to the encoder changes, yet the decoder embeddings of same-author texts remain relatively close in the regularized space. The margin \(m\) provides a robustness buffer: even if a perturbation moves an embedding by up to roughly \(m/2\), the relative ordering (same-author closer than different-author) is likely preserved. In contrast, a CE-only model has no explicit geometric constraint on \(F(s)\); its decoder states are optimized solely for token likelihood and are thus more sensitive to local token alterations. This explains why OURMODEL maintains high accuracy even at high perturbation intensities (up to 80–90% word-level replacements), whereas CE-only baselines suffer significant degradation. In summary, the auxiliary triplet loss on T5 decoder embeddings encourages a geometry in which source-specific stylistic signatures are compact and well-separated, thus robust to perturbations. Combined with the original next-token objective, this yields representations that are simultaneously accurate.

For optimal margin selection, we want: \[m \geq 2\sigma_n + \delta\] where \(\sigma_n\) is the expected perturbation magnitude in embedding space and \(\delta > 0\) provides additional robustness buffer.

Empirical support appears in our embedding visualizations (t-SNE and PCA of decoder states): OURMODEL produces tighter intra-class clusters and clearer inter-class separation than the CE baseline, both for seen and unseen LLMs. Qualitatively, integrated gradients further confirm that OURMODEL assigns higher importance to stylometric markers (e.g., conjunctions and transitional phrases) rather than isolated content tokens, i.e. signals that survive light editing better than brittle token-level artifacts. Essentially, by compacting same-author decoder embeddings and enforcing separation between classes, the auxiliary triplet loss makes the model inherently more resilient to the types of light editing and adversarial perturbations commonly encountered in real-world AIGT detection scenarios.

Furthermore, contrastive learning maximizes the mutual information \(I(Z; C)\) between the decoder embeddings \(Z = F(s)\) and the class labels \(C\): \[I(Z; C) = H(Z) - H(Z|C)\] Thus, by reducing the conditional entropy \(H(Z|C)\) through embedding compactness, the triplet loss produces more informative representations. This combination of geometric optimization (via triplet loss), error bound minimization (Bhattacharyya), and information maximization provides strong theoretical justification for why contrastive style embeddings improve both accuracy and robustness compared to approaches relying solely on cross-entropy optimization.

4 Experiments↩︎

Datasets. We benchmark on 3 datasets.

OpenLLMText dataset [6] consists of 340K text samples from five sources: Human, GPT3.5 [24], PaLM [25], LLaMA-7B [26], and GPT2-1B (GPT2 extra-large) [27]. Human text comes from OpenWebText [28], a pre‑2019 Reddit‑based user content dataset. LLaMA, GPT3.5, and PaLM text was generated by paraphrasing human texts. GPT2 texts originate from the GPT2-Output dataset [29]. The dataset’s diverse prompting strategies and generation methods mirror real-world LLM usage.

HC3 dataset [30] comprises Q & A from ChatGPT and human experts, including domain-specific professionals from web users, Wikipedia and Baidu Baike. Following [8], we exclude unsuitable samples, such as instances where ChatGPT declined to provide an answer and use the same test/train splits.

MAGE/Deepfake [31] has 448K samples, generated by 27 mainstream LLMs from 7 sources alongside human-text across 10 distinct domains/tasks (news article writing, story generation, scientific writing). It has 3 types of prompts: continuation (ask LLM to continue based on the previous 30 words of human text), topical (ask LLMs to generate texts based on a topic) and specified prompts (topical prompts with specified information about the text sources such as BBC news, Reddit Post, etc.). We focus on the stress-test tasks that are most relevant to robustness: cross-domain/cross-model in-distribution evaluation, unseen-model and unseen-domain OOD evaluation, and paraphrasing.

Experimental Setup and Hyper-parameters.

Unless otherwise stated, we use the published train/dev/test splits from [6], [8], and [22]. OURMODEL uses T5-small with maximum length 512 and truncation, Adam, learning rate \(1.0\times10^{-4}\), weight decay \(5.0\times10^{-5}\), batch size 32, \(\lambda=0.5\), and margin \(m=1.0\). For OpenLLMText and MAGE/Deepfake we use same pre-processing from  [6], [22]. For the HC3 dataset, while using the same splits, settings and synonym selection mechanism as in [8], we corrected an extra space added during synonym replacement. Our code and datasets will be public.

5 Results↩︎

In Section 5.1, we evaluate OURMODEL on multiclass source attribution and binary human-vs-LLM detection. Section 5.2 presents ablations. Section 5.3 provides qualitative diagnostics. Section 5.4 studies perturbation robustness in character and word-level adversarial attacks across different severities, and Section 5.5 evaluates OOD and paraphrasing performance on MAGE/Deepfake.

5.1 Multiclass AIGT Detection↩︎

Table 1, on OpenLLMText, shows that OURMODEL, with contrastive loss, excels in identifying LLaMA-generated text (F1-score 1.00), reflecting its ability to capture distinct stylistic patterns in LLaMA’s text completions. High F1-scores for ChatGPT (0.96) and GPT2 (0.97) indicate robust performance across diverse LLMs, while the slightly lower F1-score for Human (0.91) suggests challenges in detecting human texts, likely due to their stylistic variability. The balanced macro and weighted averages (0.95) highlight the model’s consistency across classes. These results imply that OURMODEL’s contrastive style embeddings are effective for multiclass LLM detection.

The confusion matrix in Figure 2 shows that OURMODEL excels on Human, ChatGPT and LLaMA (much higher true positives), while benchmark is marginally better on GPT-2, and PaLM. However, OURMODEL has fewer overall errors, especially on harder-to-distinguish LLaMA (near-perfect) and lower off-diagonals such as Human → PaLM misclassifications (413 vs. 717) making it superior multiclass AIGT detection.

Figure 3 summarizes one-vs-rest ROC curves. Figure [fig95det95comp] shows that OURMODEL generally traces a better false-positive/false-negative trade-off than the CE-only baseline at matched thresholds.

Detection Error Trade-off (DET) curves plot the false negative rate against the false positive rate to evaluate the performance of binary classifiers across various thresholds. For comparison with and without the contrastive style information, we also compare the DET curves on each one-vs-rest task in figure [fig95det95comp], using a log scale. OURMODEL’s DET curve consistently shows a lower false negative rate for similar false positive rate compared to benchmark.

Human-LLM Binary Classification Our evaluation demonstrates that OURMODEL significantly outperforms existing baselines. In the overall human-LLM binary classification task (Table 2), OURMODEL achieves an AUC of 0.974, an accuracy of 0.964, and an F1 score of 0.912, surpassing T5-Sentinel (AUC 0.965, accuracy 0.956, F1 0.886) and other baselines like OpenAI’s classifier and ZeroGPT. In terms of precision-recall, DeTeCtive is recall-heavy. While Sentinel is balanced, the high precision (0.941) and balanced recall (0.884) indicate that our model effectively identifies both human and AIGT with minimal false positives, providing an optimal trade-off.

In specific human-to-LLM classification tasks (Table 3), OURMODEL consistently excels. For instance, in the Human vs. LLaMA task, it achieves near-perfect performance (AUC 0.999, accuracy 0.999, F1 0.998), highlighting its ability to distinguish LLaMA’s text completions from human writing. Similarly, in the Human vs. GPT3.5 task, OURMODEL delivers an accuracy of 0.990 and an F1 score of 0.982, outperforming both T5-Hidden (accuracy 0.922, F1 0.916) and T5-Sentinel (accuracy 0.914, F1 0.906) benchmarks, that do not use the contrastive style embeddings for model training. These results underscore the model’s robustness across different LLMs, even when generation methods vary (e.g., paraphrasing for GPT3.5 and PaLM vs. text completion for LLaMA).

5.2 Ablation Study↩︎

To understand the contribution of contrastive style embeddings at train time, we conduct ablations on multiple training scenarios and key hyper-parameters, including systematic evaluation of other forms of contrastive loss, sample mining and different as well as learnable \(\lambda\) values.

Alternative contrastive losses. Table 4 shows that the triplet-loss formulation used in OURMODEL is more effective than the other tested losses under the same T5-small backbone. It also reaches its best validation with the fewest epochs and shortest training time.

Hard Negative and Positive Mining. Table 5 evaluates the impact of sampling strategies on performance. The multiclass accuracy (across all author classes in OpenLLMText) improves from 0.935 with random sampling to 0.943 with hard positives and negatives, indicating that hard sampling can enhance multiclass classification by learning more nuanced stylistic features. However, the human vs. AI single class accuracy slightly decreases from 0.964 to 0.962 with hard sampling. This suggests that while hard sampling improves multiclass discrimination, it may introduce noise in binary tasks by focusing on edge cases that are less representative of typical human-AI differences. Another aspect is human authors form a more diverse set with varying characteristics [32]. Hard positives and negatives are specifically selected samples that are challenging for the model to distinguish and therefore can improve its discriminative power [33]. Selecting these and the pairwise comparison is computationally expensive, needing \(\mathcal{O}(n^2)\) operations. OURMODEL uses random sampling for efficiency over hard sampling’s marginal gains and high computational cost.

Effect of the contrastive style embeddings. Table 6 highlights the critical role of contrastive style embeddings in OURMODEL. The balanced approach, combining cross-entropy and contrastive loss, achieves the highest performance (0.935 multiclass, 0.964 human-AI), surpassing the cross-entropy-only baseline (0.919, 0.956 respectively) inspired by [6], demonstrating that style embeddings enhance stylistic differentiation. However, using only contrastive loss yields poor results (0.317 multiclass, 0.320 human-AI), indicating that cross-entropy loss is essential for effective classification. These findings imply that OURMODEL’s hybrid loss strategy enhances the model’s ability to capture author-specific patterns.

5.3 Qualitative Analysis↩︎

Integrated Gradients (IG) attributions highlight the input tokens most responsible for the model’s output, with brighter colors indicating stronger influence (yellow, green, down to blue), facilitating interpretability. We use IG only as an illustrative diagnostic to show how the CE-only T5-Sentinel baseline and OURMODEL distribute salience on a few example cases from the OpenLLMText test set.

In Box 1, without style, peak yellows are limited to 5 single/short token sequences. Box 2, with style, spreads over 7 semantically pivotal longer token sequences, making it harder to exploit via word/character tweaking or simple punctuation injection. Specifically, OURMODEL attributes more significance to joining words, also known as conjunctions and transitional phrases, which establish the flow and stylistic identity of a text [5]. Such stylometric features identify authorship or author styles [32], Unlike content words, which are tied to specific topics, conjunctions provide stable stylistic markers that remain consistent across different themes [5]. This stability makes them reliable for detecting shifts in writing style, as variations in their usage can indicate a change in authorship or intentional stylistic adjustments. OURMODEL picks up on topic-invariant conjunctions such as ‘yet’, ‘that’ capturing the author’s unique stylistic fingerprint.

Box 3 shows the IG for a micro text sample, confirming the broader focus OURMODEL applies on all the tokens.

Box 4 shows the IG for the perturbed micro text sample, where the ‘He’ from sample 3 text is changed to ‘he’. OURMODEL’s prediction probability changes 2% (97% to 99%) while CE-only changes 9% (88% to 97%).

Box 5 is another illustrative case. Under a swap perturbation, changing Activate to Activaet), OURMODEL remains more stable than the CE-only baseline and places less relative mass on punctuation-like artifacts. IG is broader, focuses on ‘Politic’ token, sustains ‘secret’, and is consistent across the perturbation despite noise, indicating a more comprehensive model understanding. Furthermore, w/o styles, superficial </s> and "’" represent higher IG, indicating shallow understanding.

We compare decoder embeddings from the benchmark and OURMODEL to investigate the effect of the respective loss functions during training. In Figure 7 (a), the benchmark exhibits dense clustering in both t-SNE (top left) and PCA (bottom left), with notable overlap among model types (e.g., ChatGPT, PaLM, Human, GPT2, LLaMA). In contrast, OURMODEL displays a more clustered distribution.

Clear clustering in PCA and t-SNE are strong indications of well-separated structure in the original high-dimensional space. While t-SNE emphasizes local structure preservation, it is sensitive to hyperparameters like perplexity (which balances local and global structure) and prone to artifacts such as false separations or compressions. PCA, conversely, captures global variance through linear projections, highlighting directions of maximum spread but potentially missing non-linear relationships. t-SNE (top right) shows that OURMODEL clusters are markedly more compact and distinct; points are uniformly dense within clusters and sparsely distributed between them, suggesting enhanced intra-class cohesion and inter-class distance—likely due to the contrastive loss encouraging repulsive forces between classes with superior local structure preservation. PCA visualization on OURMODEL (bottom right) shows compact ovals with strong separation, minimal, with clear inter class gaps, indicating that variance directions now better align with class distinctions, lower intra-class variance and reinforces t-SNE’s findings through a linear lens, suggesting the improvements are not just non-linear artifacts but reflect genuine variance restructuring for better separability.

In relation to recent unseen LLM authors (Figure 7 (b)), both PCA and t-SNE embeddings are intermixed, suggesting these unseen AI authors (LLaMA variants, Deepseek, OpenAi, Qwen) generate highly similar embeddings, possibly due to shared architectures, training data, or fine-tuning. However, the benchmark’s distributions are more compact/less structured (especially in t-SNE), suggesting uniformity or less variability, while OURMODEL shows elongation/spread, potentially indicating richer dynamics.

The benchmark prioritizes capturing a wide range of contextual features, due to cross-entropy loss optimized for likelihood, resulting in denser embeddings with significant inter-class overlap, learning a rich but less separable representation. This aids nuanced feature preservation but could reduce classification performance due to lower distinguishability. On the other hand, the sparser and more distinct embeddings in OURMODEL, especially in the decoder states, imply that its loss function incorporates mechanisms to enhance class separability, due to the margin-based contrastive loss. This approach regularizes the latent space, leading to clearer cluster boundaries.

5.4 Perturbation Resilience.↩︎

We benchmark OURMODEL on word-level and character-level adversarial attacks at varying intensities. Such simple perturbations emulate simple real-world text polishing  [8], but can significantly interfere with popular AIGT detectors [1], [3].

Table 7 shows AIGT detection accuracy on HC3 under increasing perturbation levels from 0% to 30%. PRDetect [8] is a perturbation-robust model for detecting LLM-generated text by analysing syntax tree structures, enhancing its ability to identify synthetic content even when modified. OURMODEL outperforms all models, maintaining accuracy above 0.9980 even at 30% perturbation and above (shown later). We attribute the robustness to the margin-based contrastive loss, which enhances feature separability in the embedding space, complementing the cross-entropy loss. Cross-entropy loss alone typically struggles with robustness, as seen in RoBERTa and DetectGPT.

Experiments on the OpenLLMText dataset for word-level (Figure 4) and character-level (Figure 5) perturbations show OURMODEL’s resilience even at up to 90% intensity. Part-of-Speech (POS) distributions are consistent within LLM families, yet distinct across families (e.g., LLaMA vs. GPT), enabling reliable model identification [35]. Fig. 6 breaks down perturbations by POS tags for different grammatical constructs, confirming OURMODEL’s robustness. We provide further results on the HC3 dataset for word-level (Figure 8) and character-level (Figure 9) perturbations, confirming OURMODEL’s effectiveness against perturbations.

The superior performance of OURMODEL suggests that the margin-based contrastive loss effectively regularizes the model against perturbations, likely by enforcing distinct class boundaries in the latent space, as shown and discussed later in Figure 7 (a). Reliance on cross-entropy loss alone is insufficient for performance under distortion, aligning with the observed declines in other cross-entropy-based models (CoCo, RoBERTa and T5-Sentinel).

5.5 Deepfake/MAGE Testbed↩︎

In AIGT detection, robust performance across multiple domains, out-of-domain texts, unseen models, and diverse prompting strategies is essential. The MAGE/Deepfake testbed is designed to evaluate how well models can detect AIGT under different realistic generation and editing scenarios. Its 1-8 tasks correspond to increasingly challenging threat models, from obvious AIGT(1), to unseen models(5)/domains(6) to subtle human-in-the-loop edits(8, paraphrasing) with 3 prompting strategies: continuation, topical and stylistic. Table ¿tbl:tab:32deepfake32all? presents Average Recall (AvgRec, i.e. averaging human-written and AIGT recall) for tasks 3,4,5,6 and 8, plus the overall average, underscoring OURMODEL’s superior efficacy.

Table ¿tbl:tab:32deepfake32all? reports AvgRec for the selected tasks. On the easier in-distribution tasks (4 and 3), OURMODEL surpasses all benchmarks, except DeTeCtive. On the harder out-of-distribution (OOD) tasks on unseen models (5), unseen domains (6), and paraphrasing (8), OURMODEL achieves state-of-the-art, surpassing even DeTeCtive’s Training-Free Incremental Adaptation (TFIA), a few-shot trained OOD adaptation strategy by up to 4-13%.

Robustness to Unseen Data/Models. OURMODEL demonstrates superior OOD performance (93%+ AvgRec on unseen models/domains), attributed to contrastive style embeddings capturing perturbation-invariant signatures. While OURMODEL trails DeTeCtive slightly in in-distribution (3-4% gap), likely due to DeTeCtive’s multi-layer/task contrasts, it demonstrates practical deployability for real-world AIGT detection in diverse, unseen scenarios like news/story generation, without retraining.

Robustness to Paraphrasing. Many detectors (DeTeCtive: 99.77%,T5-Sentinel) that score >95% on task 1 collapse to 55–65% on task 8. OURMODEL sets the state-of-the-art in this most adversarial task, significantly outperforming others, including more recent architectures, underscoring our method’s ability learn deeper resilient representations in the presence of superficial stylistic obfuscation, a common evasion tactic.

Implications on Jailbroken LLMs. Jailbreaking techniques such as token manipulation/paraphrasing [36], prompt engineering, many-shot demonstrations, log-prob manipulation may shift output conditional log probabilities, making restricted/harmful content more likely. Such LLMs could produce outputs with noticeably different styles, such as more direct, unfiltered, concise, or role-play oriented, compared to their aligned counterparts. While the performance of AIGT detection methods under such distributional shifts remain underexplored, making it a crucial and challenging direction for future exploration, we hope our findings on task 8 offers new insights.

In summery, OURMODEL achieves the highest average AvgRec (88.47) with contrastive learning vs T5-Sentinel (86.29) without, showing our approach is an effective solution for challenging AIGT detection and validating contrastive learning for resilient durable LLM fingerprinting.

6 Conclusion↩︎

OURMODEL addresses critical vulnerabilities of existing AI-generated text (AIGT) detectors to adversarial attacks. We present (1) a theoretically robust detector, grounded in embedding geometry, Bayes error bounds, and mutual information maximization, that leverages contrastive learning to enhance resilience without explicit adversarial training; (2) state-of-the-art performance in binary (AUC 0.974) and multiclass (accuracy 0.95) detection on OpenLLMText and HC3 datasets, outperforming baselines like T5-Sentinel and DeTeCtive; (3) rigorous evaluation of robustness against word- and character-level perturbations up to 80% intensity, with ablation studies confirming the importance of batched margin-based triplet loss; and (4) state-of-the-art in both out-of-distribution tasks of the MAGE/Deepfake benchmark (15% gains over priors without domain adaptation) as well as state-of-the-art in paraphrasing, the most extreme task(8). We hope this work inspires new insights in to robust perturbation-resilient AIGT detection.

Acknowledgments↩︎

Dedicated to Sugandi.

References↩︎