1 Introduction↩︎

Representation learning has driven progress in NLP in the last decade, including on information retrieval, clustering, classification, and semantic search. While such models like SBERT-models [1], E5 [2], LLM2Vec [3], and EmbeddingGemma [4] focus on capturing semantic meaning, a more disregarded parallel field focuses on the style of text.¹ Style representations have proven valuable in applications such as authorship attribution (AA), style transfer, and AI-generated text detection [6]–[12], and might support the development of more style-aware LLMs [5].

Currently, evaluation approaches for style embeddings are inconsistent. Works vary considerably in the tasks and datasets they evaluate on and in evaluation protocol decisions like preprocessing, encoded text length, and documents per embedding. For example, LUAR [10] evaluates only on authorship retrieval, STAR adds author clustering [13], and LISA [14] uses the STEL evaluation framework [12], [15]. The Massive Text Embedding Benchmark (MTEB) [16], positioned as a general-purpose text embedding benchmark, offers no alternative as it includes no style-specific tasks, and many of its leading models [2], [17]–[20] are trained on objectives such as semantic similarity, information retrieval, and natural language inference (NLI) that benefit less from stylistic features (see 1). As a result, cross-work comparisons are unreliable, and progress on style embeddings is hard to quantify.

We introduce the open-source Style Text Embedding Benchmark (STEB). STEB consists of \(96\) datasets across \(7\) languages, organized into \(5\) evaluation tasks (clustering, pair classification, order alignment, retrieval, and probing) under fixed evaluation protocols (e.g., metrics, long document handling) covering applications like AI-text detection (ATD), authorship verification (AV) and authorship retrieval (AR). STEB reports two complementary scores. The operational score reflects the datasets and tasks that the field has built to study style. It inherits the emphasis of the field—most prominently AA work—without committing to an a priori definition. The definitional score instead reweighs results according to the style definition proposed in [5]. We evaluate \(40\) models spanning style embeddings, general-purpose semantic embeddings, masked language models (MLMs), causal language models (CLMs) and non-neural baselines. We find that no single model dominates across STEB. Instead, the best representation depends on whether the goal is strong performance on a specific downstream application or broad coverage of stylistic attributes, as well as on the degree to which the task is entangled with semantics. Notably, recent semantic embeddings that lead MTEB (e.g., Qwen-Embedding-8B) perform substantially worse than specialized style representations, highlighting a gap in MTEB’s evaluation of stylistic tasks. Furthermore, re-evaluating a prior multilingual setup under a different protocol for fairly handling long texts produces substantially different rankings, reinforcing the importance of standardized evaluation procedures. We also find that off-the-shelf MLMs perform surprisingly well at capturing linguistic features and remain competitive on AV and AR, suggesting promising directions for future research. Overall, we hope that STEB serves as a “yardstick” for measuring progress toward better style representations.

1.0.0.1 Why not just use LLM prompting?

While generative models can solve an increasing number of problems, text representations remain highly relevant and widely used in practice [21], [22] for several reasons: (i) Text representations are more efficient: state-of-the-art encoder models are smaller (millions vs. billions of parameters) and only use a single forward pass, whereas generative models incur cost proportional to the output length; (ii) they are competitive on discriminative tasks, matching or outperforming much larger generative models [22]; and (iii) they scale much better to larger document pools for applications like RAG [23] or clustering, and might reduce hallucinations [24]. We demonstrate that LUAR-CRUD [10] clearly beats GPT-5.2 on a small AR task, while using \(\gg\)​750\(\times\) fewer FLOPs ([tab:retrieval-llm]; full setup in App. 11).

4pt

2 Related works↩︎

2.0.0.1 Benchmarks for style representations

Evaluation of style representations has mostly focused on authorship tasks [5]. The longest-running effort in this space is the PAN shared task series at CLEF, which has run continuously since the late 2000s and describes itself as covering “digital text forensics and stylometry”.² Early editions established cross-domain AA and AV as standard tasks [25]–[29], while later editions broadened to author profiling, style change detection, multi-author writing style analysis, hate-speech spreader profiling, and ATD [30]–[35]. Most of the style embeddings we evaluate, evaluate on at least one PAN split. For AR, LUAR [10] introduces its own evaluation splits, which have since continued to be used (e.g., [36]). Few evaluation approaches have used more theoretical definition-based understandings. For example, STEL [15] introduces a parallel-text ranking evaluation over a small set of style dimensions, and STEL-or-Content [12] extends it with a content-controlled variant that penalizes models reliant upon semantics. [37] further extends STEL by prompting LLMs to generate synthetic instances across \(40\) stylistic features.

3 The Style Text Embedding Benchmark↩︎

STEB consists of \(96\) datasets across \(7\) languages, organized into \(5\) evaluation tasks. In Section 3.1, we outline these tasks, define the metrics used for scoring each, and discuss how each serves to evaluate the quality of style embeddings. In 3.2, we provide information about the included datasets and their properties. In 3.3, we explain the operational and the definitional STEB score. In 3.4, we mention principles that we considered, and should generally be considered, when extending STEB.

3.1 Tasks and evaluation↩︎

See ¿tbl:tab:STEB-examples? for examples of all task types in STEB.

3.1.0.1 Pair classification

The goal of pair classification is to determine whether two input texts share the same label (e.g., same author in AV). To perform this, we embed both inputs and compute their cosine similarity, expecting high similarity between same-label pairs and low similarity otherwise. We implement two variants: (1) All-to-All, where every sample in the set is compared against every other sample providing an exhaustive evaluation, and (2) Predefined, where the pairs are restricted to a predefined list of pairs. This latter setting is critical for controlling for confounding variables (e.g., enforcing cross-topic pairs). We report the area under the receiver operating curve (AUROC).

3.1.0.2 Clustering

While semantic embeddings are optimized to cluster documents by topic, style embeddings should induce clusters corresponding to authorship [38], register (e.g.,formality vs.informality) [37], and LM provenance [11]. We evaluate whether style embeddings are capable of clustering text samples across style-relevant labels. Following the MTEB [16] protocol, we use a mini-batch \(k\)-means algorithm with a batch size of \(32\) and \(k\) equal to the number of ground-truth labels. We report the V-measure [39].

3.1.0.3 Authorship retrieval

AR has emerged as a primary application of style embeddings [9], [10], [40]–[42]. Unlike traditional information retrieval, which optimizes for semantic relevance between a query and a document, AR aims to identify other texts written by a specific author from within a large candidate pool. We use standard retrieval benchmarks where topic and authorship may correlate but are not the sole signal. We report the mean reciprocal rank (MRR).

3.1.0.4 Order alignment

Order alignment evaluates whether a style embedding ranks a set of texts in the same stylistic order as a reference set. In its simplest form, this corresponds to the STEL task introduced by [15], which uses parallel texts. A pair of texts has to be aligned to the order of another base pair of texts using stylistic information (see App. [fig:order-alignment-example]). Every order alignment task also has a distractor variant. Here, the unordered set includes one additional text that shares the same topic as the ordered texts but stylistically unrelated to any of them, a model that relies on semantic features more than style will mis-align it. In a setup with a one element ordered set, and with a two element unordered set including one distractor, this is equivalent to the STEL-or-Content task [12]. We report two accuracies, one for the original and one for the distractor variant. To achieve a high score in this task, an embedding must capture features other than semantics, and in the distractor variant it must ignore semantic similarity altogether.

3.1.0.5 Probing

The probing task assesses which linguistic features are encoded in an embedding by training a linear classifier on top of frozen representations [43]. We extract foundation-level linguistic features using the LFTK toolkit [44], excluding features that may correlate with semantics (e.g., entity counts). Each continuous feature is discretized into five quantile bins, and the train/validation/test splits are balanced across the discretized labels. We then train a logistic regression probe on the frozen embeddings, selecting the L2 regularization strength from \(\{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}\) via a validation set, and report test accuracy averaged across all features.

3.2 STEB datasets↩︎

An overview of all datasets can be found in 9. The added datasets can generally be grouped into 6 categories: those that target capabilities with respect to single linguistic features (7 datasets), authorial styles (58), dialectal styles (3), registers (12), genres (4), historical styles (3) and demographics (9).

3.3 STEB Scores↩︎

With STEB, we evaluate models on the most common datasets and tasks that the field has built to study style. We aggregate the results differently for the operational score and the definitional score.

3.3.0.1 Operational STEB Score

While we do not align the operational score with a definition, we take care to not overweigh groups of correlated datasets. We automatically discover redundancies following [45] using the Spearman correlation between the model rankings on each dataset and apply hierarchical clustering (Ward linkage with a \(0.5\) distance threshold, cf. [46]). Scores are macro-averaged within each and then across clusters.

3.3.0.2 Definitional STEB Score

To contrast to the operational view of style (cf. 1), we also evaluate embeddings based on an a priori definition of style, namely the definition introduced in [5]. We evaluate whether embeddings (i) encode linguistic features, (ii) capture the stylistic patterns of different objects of study (e.g., dialect, genre, idiolect), and (iii) are more sensitive to stylistic than content information. See App. 9 for an overview of the dataset-to-cluster assignment.

3.4 Design principles↩︎

We provide code and datasets with STEB that are meant to be extended by the community. We emphasize the principles on which we designed STEB and that we recommend the community to follow:

3.4.0.1 Tasks where style matters

Fundamentally, STEB targets scenarios where the semantics of a text is neither the sole nor strongest signal. Where possible, we enforce topic-controlled settings that penalize models that rely on any semantic shortcuts (e.g., ¿tbl:tab:per-application-clusters?). As such, each STEB dataset should benefit from representations that capture stylistic information and should be more difficult for models that heavily rely on semantic information.

3.4.0.2 Length control

To preserve ecological validity [47], we evaluate models on the full text length inherent to each dataset. This approach preserves the natural length distributions between domains, for example, retaining the verbosity of blog posts versus the brevity of Reddit comments. For models with fixed context windows (e.g., \(512\) tokens), we employ a chunk-and-pool strategy where inputs exceeding the limit are segmented in sentence-boundary-aware chunks. We embed each chunk independently and apply mean pooling to derive a single document representation (see 4 for a visual of the approach).

3.4.0.3 Reproducibility and extensibility

For reproducibility, all stochastic steps in evaluation (e.g., \(k\)-means initialization, probe training) use fixed random seeds. To facilitate community adoption, STEB is open-source and easy to extend. We welcome contributions from the community in adding new tasks and datasets.

4 Models↩︎

We test non-neural (4.1) as well as automatically-learned representations (4.2).

4.1 Non-neural representations↩︎

We test various representations using predefined, linguistically motivated features. Note that the strongest non-neural representation generally combines several of these features and tailors the set to the specific dataset or task [5].

4.1.0.1 Function word frequencies

Function words (i.e., stop words) like determiners (the) and pronouns (you) are useful across text analyses, such as AA [48], [49], genre/text type classification [50], and deception detection [51]. We count the frequencies of 390 function words and 69 function phrases (instead of), saving these counts as a vector per text [52].

4.1.0.2 TF-IDF-weighted n-grams

N-grams—sequences of \(n\) consecutive units (i.e., characters, tokens, part-of-speech (POS) tags)—have long been used for text analysis, especially for AA [53], [54]. Following others (e.g., [55]), we weight the n-grams by their Term Frequency-Inverse Document Frequency (TF-IDF) and fit the vectorizer to a 10 billion token sample of FineWeb [56]; see 10 for more details.³

4.1.0.3 Stylometric features

There is no agreed upon fixed set of best stylometric features [57], [58]. There are perhaps equally as many feature extraction tools for various use cases. Due to its popularity in the NLP community, we use LFTK [59] and combine its surface and POS features, which we found to perform best across datasets while remaining fast.

4.1.0.4 NeuroBiber

NeuroBiber [60] is a transformer-based system that extracts 96 syntactic and lexical “Biber features”, such as function words, subordination types, and tenses [61], [62].

4pt

4pt

4.2 Neural models↩︎

We benchmark models with state-of-the-art results on various style and semantic embedding tasks, grouped into four families: style embeddings, semantic embeddings, MLMs, and CLMs.

4.2.0.1 Style embeddings

Most style embedding models we evaluate are trained contrastively, differing primarily in how they construct positive and negative pairs. LUAR [10] and STAR [13] treat texts by the same author as positives and different authors as negatives, learning authorship representations from Reddit and various social media sources, respectively. CISR [12] further constrains positives to be same-author but different-topic pairs, encouraging content-independent style representations. StyleDistance [37] constructs synthetic hard-positives and hard-negatives designed to isolate stylistic correlations from topical ones. mStyleDistance [63] extends this approach to the multilingual setting. MSR [64] learns multilingual authorship embeddings using probabilistic content masking to suppress content features and language-aware batching to reduce cross-lingual easy-negatives. LISA [14] departs from purely contrastive training, learning interpretable style embeddings via a linear projection over an EncT5 encoder.

4.2.0.2 Semantic embeddings

These models are trained for general-purpose text similarity, primarily on semantic retrieval and NLI data. We include models like all-mpnet-base-v2 [1] that perform well according to the semantic textual similarity evaluations and various models that perform(ed) well according to MTEB [16]: Qwen3-Embedding-8B [65], GTE [19], E5 [2], BGE [20], and Jina Embeddings v3 [66].

4.2.0.3 Masked language models (MLMs)

We include encoder-only transformers pre-trained with masked language modeling and without embedding-specific fine-tuning. Specifically, we include RoBERTa [67], DeBERTa-v3 [68], and ModernBERT [22]. Since these models do not produce a single vector, we obtain embeddings by mean-pooling the token-level hidden states from the last layer, weighted by the attention mask.

4.2.0.4 Causal language models (CLMs)

Auto-regressive models are pretrained with next-token prediction and have no embedding-specific training. We include them to assess whether large-scale causal language modeling implicitly captures stylistic features. We include GPT-2 XL [69], OPT-1.3B [70] and the Qwen family at multiple scales. We use the hidden state of the last non-padding token as the embedding, as it is the only position with full sequence context.

5 Results↩︎

We discuss operational and definitional STEB results (5.1) and provide a per-application (5.2) and a multilingual (5.3) analysis; all but the last are restricted to English.

5.1 Overall Results↩︎

STEB (operational) results are in ¿tbl:tab:overall-results? and selected STEB (definitional) results in 2; full results for all \(40\) models are in App. 3 and 6.

4pt

5.1.0.1 No single winner across task categories and style definitions

There is no consistent winner across categories, for neither STEB (definitional) nor STEB (operational). While StyleDistance wins for the definitional, STAR, LUAR-CRUD, and LUAR-MUD score similarly for the operational score. STAR’s advantage is in clustering and predefined pair classification, while it falls behind on retrieval, where LUAR-CRUD leads by over \(10\) points, and order alignment, where StyleDistance leads by nearly \(19\) points. Models like STAR and LUAR might profit from content-entangled features for different objects of study that are punished more heavily for the definitional STEB score. These patterns confirm previous observations that style is conceptualized differently across the literature, and that different training and evaluation objectives—motivated by different style definitions—result in different strengths and weaknesses across downstream tasks [5].

5.1.0.2 Style embeddings consistently outperform semantic embeddings.

Every style-specific model except LISA and mStyleDistance outperform the best general-purpose semantic embedding on both STEB scores. This disparity is especially pronounced in order alignment, AR, and predefined pair classification. Semantic embeddings are optimized on objectives that favor matching texts primarily by semantic content, and the gap on STEB suggests style features are not well captured by semantic objectives alone. These findings underscore the need for a dedicated style benchmark, as semantic-oriented evaluation fails to capture the capabilities of style-specific models.

5.1.0.3 MLM pre-training captures stylistic information.

All MLMs we evaluate outperform every semantic embedding and every causal LM on both STEB scores, despite having no embedding-specific fine-tuning. DeBERTa-v3-large outperforms some style embeddings on retrieval and matches style-specific models on authorship verification, without any author-specific training (¿tbl:tab:overall-results?). RoBERTa leads over style embeddings in sensitivity to linguistic features (probing column of ¿tbl:tab:overall-results?; linguistic-feature column of 2), suggesting that style-specific fine-tuning can come at the cost of linguistic feature sensitivity. Joint training on MLM and style objectives may help recover this sensitivity in style-specific models. The newer ModernBERT variants (only shown in App. 3 & 6) underperform, suggesting that more recent MLM recipes have shifted away from preserving linguistic feature information. Together, these results suggest that new MLM objectives are a promising direction for future research.

5.1.0.4 Non-neural models are competitive on certain tasks.

TFIDF n-grams is the strongest non-neural model, followed closely by Stylometric. Both generally outperform causal LMs and perform similarly to general-purpose sentence embedders, but lag behind style-specific and masked models. They perform best on tasks like probing and predefined classification, but degrade on tasks like clustering and retrieval that often require richer representations. Despite TFIDF n-grams being effectively a bag-of-words model, it nearly matches sophisticated sentence embedders. Also, although the literature has shown no general-purpose best set of stylometric features, using surface and POS tag features does fairly well across datasets and tasks.⁴

5.1.0.5 CISR and StyleDistance models perform best at content independence.

StyleDistance, CISR, and mStyleDistance perform the best at content-independence (i.e., the distractor variant of the order alignment tasks). This is expected as they were all trained similarly with hard negatives to improve content independence. Across models, though, there is still a lot of room for improvement on content-independence.

5.1.0.6 STAR is most sensitive to style across stylistics attributes.

STAR performs the best at recovering distinctive patterns across “objects of study” (e.g., genre, time, registers) beyond authorial styles. This might be influenced by its being one of few models that was trained on several different domains (i.e., social media, blogs, and books). Including more diverse training datasets (and potentially training tasks, cf. [5]) might thus be a promising direction for more generalizability.

5.2 Per-application analysis↩︎

To answer questions such as “Which model is best for AI-text detection?”, we manually group datasets into four application-oriented clusters (¿tbl:tab:per-application-clusters?): ATD, ATD where text has been modified to evade detection (Adv.), AV, and AR. For AV we additionally report three sub-columns over the PAN style-change benchmarks ordered by how content-entangled the pairs are: Easy, Medium, and Hard, where Hard pairs do not share any semantic content. The top five comprise three style-specific embeddings (STAR, LUAR-MUD, LUAR-CRUD) and two DeBERTa-v3 variants. In ATD, DeBERTa-v3-large⁵ leads in the standard setting, while in the adversarial setting, STAR and DeBERTa-v3-large perform best. In AV and AR, the style-specific embedding models take the lead, suggesting that they are better suited for capturing the fine-grained nuances of everyday authors. The Easy/Medium/Hard AV columns reveal a finer pattern. On the Easy and Medium sets (where topic can be exploited), both STAR and LUAR perform well. However, in the Hard setting (where topic correlations cannot be exploited), CISR and StyleDistance perform best, suggesting that disentangling style from content matters for such applications.

4pt

5.3 Multilingual analysis↩︎

STEB includes non-English datasets across six languages (Dutch, French, Greek, Italian, Polish, Spanish) drawn from the PAN13/14/15 AV shared tasks and the PAN18 cross-domain AA shared task. These datasets are excluded from the main STEB scores (¿tbl:tab:overall-results? and  2) as most evaluated models are monolingual English encoders. Here, we investigate whether multilingual style embeddings perform better than monolingual English style embeddings on these multilingual applications. We derive a single score by averaging across all dataset metrics and across the independent tasks (pair classification, retrieval).

5.3.0.1 Multilingual style-representations underperform their English counterparts

¿tbl:tab:multilingual-results? shows that the top-5 by multilingual average comprises two DeBERTa-v3 variants, RoBERTa-large, and two style embeddings (StyleDistance, CISR). The explicitly multilingual style embeddings (MSR, mStyleDistance) do not appear in the top five. These results suggest that current multilingual training strategies are not yet effective, and as such, they remain a challenge for future work.⁶ The two style embeddings that do appear are notable for having been trained so as to separate semantic content from style, suggesting that this separation might be a useful strategy for the generalizability of style embeddings across languages.

6 Conclusion↩︎

We introduced STEB, a benchmark for evaluating embeddings across style-related NLP tasks, spanning \(5\) tasks, \(96\) datasets, and \(7\) languages. Across \(40\) models, we find that no single embedding is best. Style embeddings that retain semantic signal lead on authorship-related applications where author and topic correlate, while content-disentangled style embeddings lead on content-controlled applications where topic shortcuts are removed. On ATD, MLMs outperform style-specific embeddings, and on predicting second-order style features (e.g., genre, dialect, register, time), the authorship-focused STAR model leads. Multilingual style embeddings underperform their English counterparts on the multilingual subset of STEB, highlighting cross-lingual style as an open problem. We release STEB to enable consistent, reproducible comparisons and to lower the cost of measuring progress on style. We welcome contributions from the community in improving the benchmark.

Limitations↩︎

6.0.0.1 Leakage of datasets / contamination

Style embeddings are commonly trained and tested on similar datasets, often using datasets for training and testing with differing splits between publications. Representations might be tested on some data that they have seen during training. For example, the StyleDistance models were trained on one of the seven “linguistic features” datasets and on two of the eleven “Content Independence” datasets in 2. Excluding these shrinks the difference to CISR to \(\approx\)​2 percentage points on content independence and make the synthetic version and CISR have almost the same overall performance. However, the overall ranking of the top models is preserved.

6.0.0.2 Bounded by dataset coverage

Both the empirical clustering in §5.1 and the attribute clustering in §[sec:attribute-analysis] are bounded by what existing datasets expose. In particular, most of the field’s emphasis has been on authorship-related tasks, and as such other axes of analysis remain underexplored. We call on the community to create datasets that explore aspects other than authorship and to expand STEB with these additions.

6.0.0.3 Languages other than English remains underrepresented

It is a fact of the field that not as many non-English datasets are available. As such, we inherit this limitation. Moreover, only recently has there been an effort to extend style embedding models to multilingual settings [9], [63].

Appendix↩︎

7 Full aggregate results↩︎

The full tables of results for the experiments in 5.1, 5.2, and 5.3 are shown in 3, 4, and 5, respectively.

4pt

4pt

4pt

3pt

8 Task examples↩︎

We display an example of the order alignment task in Figure [fig:order-alignment-example].

9 Dataset details↩︎

Table 1 summarizes which stylistic attribute each STEB dataset is intended to probe, using the seven categories around which this section is organized. A check mark indicates the category the dataset is grouped under in our analyses; some datasets could plausibly fit additional categories (e.g.the Corpus of Diverse Styles spans dialect, time, and genre) but we keep a single assignment for clarity.

9.1 Feature datasets↩︎

9.1.0.1 STEL_feature

added as order alignment task. We split the publicly available STEL version [12], [15] into registers (cf. Section 9.3) and linguistic features (i.e., number substitution, contraction or emoji use characteristic). STEL_feature consist of \(\approx400\) instances total. The STEL characteristics are publicly released. Note that in order alignment tasks we pair each style transfer pair with every other pair from the same style category—thus, we do not use the predefined pairings from STEL or SynthSTEL.

9.1.0.2 SynthSTEL_feature

added as order alignment task. We split the SynthSTEL dataset [37] into 8 register (cf. Section 9.3) and 32 linguistic features (i.e., everything else, including features like All Lower Case / Proper Capitalization). We combine the train and test split, leading to 100 pairs per feature. SynthSTEL is released under the MIT license per its HuggingFace dataset card at https://huggingface.co/datasets/StyleDistance/synthstel.

9.1.0.3 StylePTB

added as order alignment task. StylePTB [71] is a style-transfer benchmark with \(\approx60k\) parallel pairs over 21 individual styles. We keep 15 and remove six out of the 21 dimensions as some change meaning (e.g., antonym replacement) or have no consistent style change within the same dimension (e.g., synonym replacement uses WordNet without clear stylistic commonalities in the selected synonyms like market replaced with marketplace and years with twelvemonths). Note that the distribution is heavily skewed: to_future has \(\approx7k\) pairs while pp_back_to_front has only 7. To keep as many categories as possible, we combine the train/dev/test partitions. The text is provided in tokenized form: we do not detokenise. The StylePTB transformations are released under CC BY 4.0 (https://github.com/lvyiwei1/StylePTB/blob/master/LICENSE).

9.1.0.4 Probing

added as probing task across four domains (probing_amazon, probing_blog, probing_reddit, probing_stackexchange). We extract \(46\) foundation-level linguistic features per document using the LFTK toolkit [59]: \(12\) token-level counts (words, stop-words, punctuation, syllables, characters, plus token frequency norms) and \(34\) POS-tag counts (total and unique counts for each of \(17\) Universal Dependencies POS tags). Each feature is independently quantile-binned into discrete labels and the resulting per-feature classification task is solved with a small MLP probe on top of frozen embeddings. The reported scores are macro-averaged across the \(46\) features. The underlying source texts are sampled from publicly available Amazon product reviews [72], blog posts (via Kaggle https://www.kaggle.com/datasets/rtatman/blog-authorship-corpus), Reddit comments (via HuggingFace dataset AnnaWegmann/StyleEmbeddingData), and Stack Exchange posts, yielding \(27,584\), \(39,111\), \(16,447\), and \(9,253\) records respectively after binning and balancing. The scripts used to create the probing datasets will be released publicly on our GitHub repository.

9.2 Dialectal datasets↩︎

9.2.0.1 Twitter AAE

added as order alignment task. We use a set of parallel constructed tweets written originally in AAE variety and rewritten in SAE by annotators[73]. The dataset consists of \(\approx 2k\) instances. It can be accessed via the EMNLP 2020 supplementary archive at https://aclanthology.org/attachments/2020.emnlp-main.473.OptionalSupplementaryMaterial.zip. There is no license file; we treat it as research-use only consistent with its release as paper supplementary material.

9.2.0.2 EnDive

added as clustering and all-to-all pair classification task. We add parallel sentences from the EnDive benchmark [74] for Jamaican English, African American English, Colloquial Singaporean English, Indian English and Chicano English across 12 NLU benchmarks⁷ at https://huggingface.co/datasets/abhaygupta1266/. The dataset was created using few-shot prompting with examples from native speakers. License information is not provided, but we assume it to be permissive based on the fact that it is meant to be a public benchmarks freely shared on HuggingFace.

9.2.0.3 eWAVE

added as clustering and all-to-all pair classification task. We add dialect example sentences from the Electronic World Atlas of Varieties of English [75] Cross-Linguistic Data Formats (CLDF) release at https://zenodo.org/records/17433568 and https://github.com/cldf-datasets/ewave. We include 53 varieties ranging from \(\approx20\) to \(\approx200\) instances. Varieties include Hong Kong English, Minx English, Bahamian Creole and Indian English. The CLDF release is licensed CC BY 4.0.

9.3 Register datasets↩︎

9.3.0.1 STEL_register

added as order alignment tasks. We split the publicly available⁸ STEL version [12], [15] into registers (i.e., formality and complexity dimension) and linguistic features (cf. 9.1). The registers consist of \(\approx200\) instances each.

9.3.0.2 SynthSTEL_register

added as order alignment task. We split the SynthSTEL dataset [37] into 8 register (e.g., formal tone, offensive language, sarcasm) and 32 linguistic features (cf. Section 9.1). Note that depending on ones definition of “register”, some categories like positive sentiment expression might not be considered style [5]. We combine the train and test split, leading to 100 pairs per feature. SynthSTEL is released under the MIT license per its HuggingFace dataset card at https://huggingface.co/datasets/StyleDistance/synthstel.

9.3.0.3 OneStopEnglish

added as order alignment task. We use the OneStopEnglish corpus [76] at https://github.com/nishkalavallabhi/OneStopEnglishCorpus/. It provides 189 parallel texts at three reading levels (beginner, intermediate, advanced) compiled from news articles rewritten by English teachers to suit the levels of the learners. It is shared with Creative Commons Attribution Share Alike 4.0 International, see https://github.com/nishkalavallabhi/OneStopEnglishCorpus/blob/master/LICENSE.markdown.

9.3.0.4 Graded Formality Dataset

added as order alignment task. From TweetEval [77] we take the \(\approx4k\) Tweets classified as non-offensive at https://huggingface.co/datasets/cardiffnlp/tweet_eval/. From SMS Spam [78], we take the \(\approx4k\) SMS classified not spam at https://huggingface.co/datasets/ucirvine/sms_spam. We use gpt-5-mini-2025-08-07 rewrite the utterances more formally 4 times, see Figure 3. Both datasets are publicly shared on HuggingFace. An example of the resulting dataset can found in Table 8. An example of the resulting sentences are: “Lol your always so convincing.”, “Lol, you’re always so convincing.”, “You’re always so convincing.”, “You are always very convincing.”, “You are consistently persuasive.”.

9.3.0.5 ASSET

added as order alignment task. We use the ASSET corpus [79], a multi-reference sentence-simplification benchmark covering \(\approx2k\) English Wikipedia source sentences with ten crowdsourced simplifications per sentence. To turn ASSET into an order-alignment task on the simple axis, we form one length-two ordered pair per source sentence by sampling one of its ten reference simplifications uniformly at random as the simpler element and pairing it with the original Wikipedia sentence as the more-complex element. We combine the validation and test splits on HuggingFace https://huggingface.co/datasets/facebook/asset. The HuggingFace card shows the license as CC BY SA 4.0.

9.3.0.6 wikipedia_politeness

added as clustering and all-to-all pair classification task. We add the Wikipedia Politeness Corpus [80] shared at https://zissou.infosci.cornell.edu/convokit/datasets/wikipedia-politeness-corpus/. The corpus consists of \(\approx4k\) polite, neutral and impolite texts of Wikipedia editor requests. The corpus is shared with a CC BY license v4.0, see https://convokit.cornell.edu/documentation/wiki_politeness.html.

9.3.0.7 stackexchange_politeness

added as clustering and all-to-all pair classification task. We add the Stack Exchange Politeness Corpus of [80] shared at https://zissou.infosci.cornell.edu/convokit/datasets/stack-exchange-politeness-corpus/. We use the same labels as wikipedia_politeness. It consists of \(\approx6k\) Stack Exchange requests. The corpus is shared with a CC BY license v4.0, see https://convokit.cornell.edu/documentation/stack_politeness.html

9.4 Genre datasets↩︎

9.4.0.1 Corpus of Diverse Styles

added as clustering, all-to-all pair classification and order alignment task. We use the Corpus of Diverse Styles [81] taken from HuggingFace at https://huggingface.co/datasets/billray110/corpus-of-diverse-styles. The original dataset totals \(\approx400k\) records across 11 “styles” (i.e., lyrics, three decade slices of the Corpus of Historical American English, Tweets, African American English Tweets, the writing of James Joyce, the Bible, the Switchboard telephone-speech transcripts, English poetry, and Shakespeare). Note that while we could distribute the labels between dialect, time and genre, we assign everything to genre for simplicity. The code is MIT-licensed (https://github.com/martiansideofthemoon/style-transfer-paraphrase/blob/master/LICENSE).

9.4.0.2 CORE

added as clustering and all-to-all pair classification task. We add the Corpus of Online Registers of English (CORE) [82] as found at https://github.com/TurkuNLP/CORE-corpus. We use the sub-labels as the relevant classes for pair classification, i.e., everything but the main labels {IN, NA, HI, LY, SP, IP, ID, OP}. If a document falls in several categories, we have it show up in several. We use documents from any file across the train, dev and test sets. This dataset was licensed under CC BY-SA 4.0.

9.4.0.3 x_genre

added as clustering and all-to-all pair classification task. We add the English slice of the X-GENRE genre-classification dataset [83] shared at https://huggingface.co/datasets/TajaKuzmanPungersek/X-GENRE-text-genre-dataset. X-GENRE merges FTD [84] and CORE [82] into a joint 9-label schema using classification models. Compared to CORE, 15 sub-labels and their texts are discarded. The added dataset consists of \(\approx2k\) English documents. We merge the train/dev/test partitions. X-GENRE is released under CC BY-SA 4.0 on HuggingFace.

9.4.0.4 MASC

added as clustering and all-to-all pair classification task. We add the Manually Annotated Sub-Corpus of American English (MASC) 3.0.0 [85], [86] at http://www.anc.org/data/masc/downloads. MASC is a corpus including texts from 19 genres in American English (e.g., debate transcripts, e-mail, jokes). We create 150 samples per genre. MASC is distributed without license or other restrictions, see https://anc.org/data/masc/about/.

9.5 Time datasets↩︎

9.5.0.1 Shakespeare

added as order alignment task. We add the parallel Shakespeare corpus of [87] at https://github.com/cocoxu/Shakespeare. It includes labels for historical (i.e., Shakespearean) and contemporary English, providing \(\approx\)​20k line-aligned pairs of originals and their modern paraphrases for 17 plays. The license file only includes a request for citation at https://github.com/cocoxu/Shakespeare/blob/master/README_LICENSE, thus it is likely to be shared at least for research purposes.

9.5.0.2 CEECES

added as clustering and all-to-all pair classification. We add the Corpus of Early English Correspondence Extension Sampler (CEECES), parts 1 and 2 [88], [89]. It consists of \(\approx15k\) paragraphs from \(\approx2.5k\) historical letters labeled by six 20-year periods from 1680-1800. Both Zenodo records are licensed under Creative Commons Attribution Non Commercial 4.0 International and Creative Commons Attribution Non Commercial No Derivatives 4.0 International at https://zenodo.org/records/6411789 and https://zenodo.org/records/5887101.

9.5.0.3 Bible versions

added as clustering and all-to-all pair classification task. We add the parallel Bible corpus of [90] shared at https://github.com/keithecarlson/StyleTransferBibleData. The dataset comprises eight verse-aligned English Bible translations including, for example, the American Standard Version (from 1901) and the Kings James Version (from 1611). We remove verses that are identical (ignoring whitespacing and casing). The dataset includes no specific license, but all included Bibles are considered public domain and the associated paper was released with the Creative Commons Attribution License.

9.6 Demographic datasets↩︎

9.6.0.1 fce_l1

added as clustering and all-to-all pair classification task for distinguishing different native language speakers. We add the the publicly released subset of the Cambridge Learner Corpus’s First Certificate in English exam corpus (CLC FCE) [91] at https://ilexir.co.uk/datasets/index.html. It consists of \(\approx2.5k\) responses for \(\approx1.2k\) English Exams by students of 16 different native languages. The dataset is released with a non-commercial research and educational license.

9.6.0.2 blog_age, blog_gender

added as clustering and all-to-all pair classification tasks. We add the Blog Authorship Corpus [92] via HuggingFace at https://huggingface.co/datasets/barilan/blog_authorship_corpus. We use the validation split to construct age (i.e., age buckets of 10s, 20s and 30s) and gender labeled texts. The corpus is meant for non-commercial research use, see https://u.cs.biu.ac.il/~koppel/BlogCorpus.htm.

9.6.0.3 PASTEL_age, PASTEL_education, PASTEL_gender, PASTEL_ethnic, PASTEL_politics

added as clustering and all-to-all pair classification tasks. We add PASTEL [93] corpus at https://github.com/dykang/PASTEL. It consists of \(\approx8k\) five-sentence stories written by crowd annotators. Texts are labeled with demographic information of annotators, including six age, eight education, three gender and, nine ethnic and three politics labels. There is no explicit license information provided, but since the dataset is released as a benchmark, we assume at least a intended research-purpose license.

9.7 Authorship datasets↩︎

9.7.0.1 PAN Authorship Verification

added as predefined pair classification task. We use the official test sets of the PAN authorship-verification shared tasks: PAN13 [25], PAN14 [27], PAN15 [28], PAN20 [31], and PAN21 [32]. PAN13, PAN14, and PAN15 contain multiple languages, including Greek, and Spanish in PAN13, Dutch, Greek, and Spanish in PAN14 and PAN15. PAN20 and PAN21 are large-scale cross-domain fanfiction verification benchmarks, we subsample each to a balanced 500-pair subset (250 same-author, 250 different-author) with a fixed seed for reproducibility. All corpora are available at Zenodo, and the download links can be found in the PAN website: https://pan.webis.de/data.html. All PAN datasets are available for research use.

9.7.0.2 PAN Style Change

added as predefined pair classification task. We use the publicly datasets of the PAN Style Change Detection shared tasks: PAN18 [29], PAN22 [94], PAN23 [95], PAN24 [30], PAN2025 [33], and PAN26 [34]. For each of the aforementioned style cahnge datasets, consecutive paragraphs or sentence pairs (depending on the datasets granularity) are taken to be same- or different-author trials. PAN23–26 ship three difficulty levels (easy, medium, hard) where each subsequent difficulty level has more topic overlap. The PAN26 raw pairs are heavily imbalanced (up to \(\approx\)​96% same-author in the medium split). We apply per-document stratified down-sampling that keeps all different-author pairs and matches the same-author count. All corpora are available at Zenodo, and the download links can be found in the PAN website: https://pan.webis.de/data.html. All PAN datasets are available for research use under the CC BY 4.0 license.

9.7.0.3 Enron Authorship Corpus

 [96] added as predefined pair classification task. We add the 80-author Enron e-mail authorship-verification corpus at https://prod-dcd-datasets-public-files-eu-west-1.s3.eu-west-1.amazonaws.com/f0527105-2774-423e-80ca-cc692b70b6cb. Each verification case is comprised of \(5\) documents, where \(4\) documents come from one author and \(1\) comes from another. We concatenate the \(4\) documents together separated by a double newline so as to create two pairs of text for the verification trail. The corpora is available for research use under the CC BY 4.0 licence.

9.8 AI-text detection datasets↩︎

Standard ATD variants use per-generator labels (human plus one label per LLM). Adversarial variants use binary human-vs-LLM labels because the question is whether embeddings still separate human from machine text after evasion techniques have been applied.

9.8.0.1 GEDE

added as a clustering task for distinguishing human and the various LLMs. The Generative Essay Detection in Education (GEDE) dataset [97] is a collection of three academic essay datasets (Argument Annotated Essays (AAE), PERSUADE 2.0, and British Academic Written English (BAWE)) plus machine-generated versions of those essays at various levels of contribution using GPT-4o-mini and Llama-3.3-70b-Instruct. There are a total of 916 human essays and 12,703 LLM essays. The dataset is provided under a CC BY-NC-SA 4.0 DEED Attribution-NonCommercial-ShareAlike 4.0 International license at https://github.com/lukasgehring/Assessing-LLM-Text-Detection-in-Educational-Contexts.

9.8.0.2 DetectRL

 [98] added as five standard machine-text-detection datasets (DetectRL_{arxiv, direct_prompt, writing_prompt, xsum, yelp_review} and three adversarial datasets (DetectRL_{paraphrase, perturbation, prompt}_attacks, binary human-vs-LLM). Source: https://github.com/NLP2CT/DetectRL.

9.8.0.3 M4

 [99] added as five clustering datasets over English-only domains: arxiv, peerread, reddit, wikihow, and wikipedia. Each domain contains data from \(5\) different LLMs (Cohere, davinci, ChatGPT, Dolly-v2, and BloomZ), but we skip BloomZ due to the problematic nature of the data where parts of the prompt are in the “generated" text. Source: https://github.com/mbzuai-nlp/M4. The data is available for research use and has previously been used in SemEval 2024 (https://github.com/mbzuai-nlp/SemEval2024-task8/).

9.8.0.4 MAGE

 [100] added as ten clustering datasets over the HuggingFace dataset yaful/MAGE: cmv, eli5, hswag, roct, sci_gen, squad, tldr, wp, xsum, and yelp. Each domain’s records are labelled by generator model only, collapsing across generation methods (continuation, specified, topical), giving 27 unique machine labels plus human per domain. The data is available for public use in HuggingFace (yaful/MAGE).

9.8.0.5 PAN24 Generative Authorship

added as clustering and all-to-all pair classification task. We use the test partition of the PAN24 Generative Authorship Detection [30] task. All PAN datasets are available for research use under the CC BY 4.0 license.

9.8.0.6 PAN25/26 Generative AI Detection (Task 1)

added as three clustering datasets (PAN25_26_{essays, fiction, news}). The PAN25 and PAN26 editions of Task 1 [34], [101] share the same underlying validation data. We split it by the genre field. Each record is labelled by its model field, yielding a multi-class clustering task with human plus all participating LLMs per genre. All PAN datasets are available for research use under the CC BY 4.0 license.

9.8.0.7 PAN25 Human-AI Collaborative Text Classification

added as clustering task. The PAN25 Task 2 development set [101] labels each text with one of six human-AI collaboration categories (e.g.fully human-written, human-initiated, then machine-continued, machine-initiated, then human-continued). We perform the clustering across these different labels. All corpora are available at Zenodo, and the download links can be found in the PAN website: https://pan.webis.de/data.html. All PAN datasets are available for research use under the CC BY 4.0 license.

9.9 Authorship retrieval datasets↩︎

9.9.0.1 PAN18 Cross-Domain Authorship Attribution

 pan2018? added as retrieval task. Each problem provides a set of candidate authors with one or more known texts and a set of unknown query texts to be attributed. We concatenate each candidate’s known texts into a single target document and add each unknown text as a query with the corresponding ground-truth label. The PAN18 dataset also contains data in French, Italian, Polish, and Spanish. All PAN datasets are available for research use under the CC BY 4.0 license.

9.9.0.2 Amazon and Reddit

added as retrieval task. We use the same retrieval test splits as [10], subsampling each dataset to \(1000\) query and target pairs to make the evaluation of all models tractable within our compute and timing constraints (the original datasets contain over \(100,000\) queries and targets). We re-distribute the subsampled test splits for research purposes only.

9.9.0.3 StackExchange

added as retrieval task. StackExchange data is readily available for use from: https://archive.org/download/stackexchange. It is licensed by CC BY-SA 3.0 which allows research use. We collect a small sample of StackExchange data spanning \(500\) authors who contribute to any community at random. The retrieval setup is similar to [10], where multiple documents serve as a query and multiple documents serve as a target.

10 TF-IDFngrams model details↩︎

While many approaches compare the frequencies of particular n-grams, simply counting their presence in documents does not reveal broader contextual information, such as how common the n-gram is in general. Therefore, n-grams can be weighted using their Term Frequency-Inverse Document Frequency (TF-IDF), which considers how often an n-gram appears in a particular document compared to how rare it is in a corpus overall. This measurement increases the weight of rare words, which might be distinguishing of an author, and reduces the weight of common words. The authorship verification baseline for the PAN competition [35], for instance, uses TF-IDF-weighted character 4-grams.

We experimented with fitting the vectorizer to a few diverse datasets—Corpus of Diverse Styles [81], Reddit Million User Dataset [102], and FineWeb [56]—but found performance to be similar for all. After experimenting with various \(n\) values for each of these models, we found that character 3-5-grams, token 1-2-grams, and POS tag 1-2-grams worked best overall. Based on these findings, for the main paper, we chose a TFIDFngrams model fit to a 10 billion token sample of FineWeb [56], a large collection of cleaned English web data from CommonCrawl, with these n-gram settings. 3 through 6 show results for a few variations of TF-IDF n-gram models based on the dataset it was trained on (Reddit, FineWeb) and the chosen \(n\) (1-2, 1-3) for the token n-grams and POS tag n-grams. The character n-gram values were not decreased because very short character n-grams occur extremely often but do not carry much discriminative information; this makes the feature matrix larger and sparser, increases computational cost, and can introduce noise that hurts generalization.

3pt

11 Embeddings vs. prompting↩︎

11.0.0.1 In practice: Why not just use prompting?

To demonstrate this concretely, we compare prompting GPT-5.2 against LUAR-CRUD on authorship retrieval, using a relatively small corpus of 100 query and 100 target collections of 16 sentences each, created with Amazon Reviews data (11). The dataset is a subsample of the STEB Amazon retrieval dataset described in 9.9. As no established prompts exist for retrieval (likely due to scaling issues, cf. 11), we adapt the linguistically informed prompts created for authorship verification by [103], with minor grammatical corrections. We provide the full prompt in App. 3. Table [tab:retrieval-llm] reports the same performance measures on retrieval as STEB (Section 3). We additionally report tera floating point operations (TFLOPs) and API costs in US dollars (GPT-5.2 only). LUAR-CRUD, a model from 2021, outperforms GPT-5.2 released in 2025 while being at least 500\(\times\) more efficent than causal models with 1B+ parameters.

We display the prompt used with GPT-5.2 in Figure 3. The lower efficiency of LLM models is unsurprising: authorship retrieval requires comparing a query against every target collection in the pool, which scales poorly with prompting approaches. As the target pool grows, it may even exceed the model’s context window or lead to large costs in US dollars or FLOPs.⁹ To illustrate, a common Reddit authorship retrieval dataset [102] contains 120k authors with 16 texts each, requiring approximately 5.1M tokens per retrieval query, which is considerably larger than GPT-5.2’s context window of 400k.¹⁰ Even if the context window was not restricted, retrieval on the 120k dataset would require about 20 billion TFLOPs for a 1 billion parameter causal model and 40k TFLOPS for models like LUAR.

12 Multilingual truncation analysis↩︎

The main-text multilingual ranking (¿tbl:tab:multilingual-results?) is computed under STEB’s default protocol, which segments long inputs into sentence-boundary-aware chunks and mean-pools the chunk embeddings (13). To isolate the effect of this evaluation choice, we reproduce the multilingual evaluation setup of [64]: the same PAN13/14/15 authorship verification datasets (Dutch, Greek, Spanish), the same pre-defined pair classification protocol, the same style embeddings, and the same AUC metric. The only change is that each document is truncated to the model’s maximum length rather than chunked and pooled as in STEB. 10 compares the two protocols. Under truncation, MSR moves from rank \(4\) to rank \(1\), while without truncation it ranks \(4\)th, underperforming the style embeddings trained on English data. These results underscore the necessity of shared evaluation protocols, as without them our conclusions would differ materially.

4pt

13 Chunking large documents↩︎

We demonstrate STEBs strategy for embedding large inputs into a single representation in 4. Note that this approach is fair to all encoders, as each one is able to observe the same amount of tokens regardless of what their token limits are.

14 Use of AI Assistants↩︎

We used AI-assistants (Claude Code, Cursor) to help us develop the STEB code-base. That said, all the code underwent human-review, and we rigorously tested the code-base for correctness. We also used LLMs to give us ideas for rephrasing and shortening excerpts of the paper.

15 Compute Requirements↩︎

We ran each model on a single H100 or A100 40Gb GPU. No single run took longer than 24hrs, and the largest model evaluated has 8B parameters. We upper-bound our GPU expenditure at \(960\) GPU hours (\(24\) hours times \(40\) models). Please note that this is a very large over-estimate, most models took less than \(8\) hours to run.

16 Sample Counts per Dataset↩︎

We specify the number of classes, number of samples per class, and total number of texts embedding in 2.

References↩︎