1 Introduction↩︎

The output of large language models (LLMs), even across model families, is becoming increasingly homogeneous. This mode-collapse phenomenon is unusually clear in creative writing [1]. While we know from prompt data that story-writing is a popular use case [2], and that readers prefer interesting and surprising literature [3], the stories generated by LLMs are remarkable in their sameness.

Prior work has proposed post-hoc solutions like adjusting sampling techniques [4] and new post-training optimization objectives [5]. In this short paper we characterize story mode collapse and explore publicly available training data to locate the source.

We generate 20,000 stories with four current models from OpenAI, Anthropic, Google, and the Allen Institute for AI (AI2), finding 88.3% of generated stories contain one of 11 core words (including character names, story locations, and professions). Most notably, over half feature a lighthouse. Why does this story pattern become favoured?

These 11 words are not common in published English literature, which suggests that post-training data is responsible (1). But examining OLMo 3’s post-training set reveals only 3,053 out of a total 78,958 stories contain one of our 11 words.¹ We find that the dominance of “Elias in the Lighthouse” stories cannot be explained by prevalence in pre- or post-training data. We speculate that models are trained to avoid references to copyrighted characters and adult content during alignment but defer this question to future work.

2 Related Work↩︎

Mode collapse, the tendency for a generative model to overfit on a small set of samples during training, was first observed in LLMs after post-training techniques like SFT and RLHF were introduced [1], [6], [7]. Fears of mode collapse deepened with growing synthetic data in training [8], [9]. Mode collapse has been demonstrated in linguistic markers [10], query answers [11], [12], and stories [13]. Prior work has proposed modifying prompts [12], sampling techniques [4], and post-training algorithms [14] — but the phenomenon remains poorly understood. To our knowledge, this study is the first to study a specific instance of mode collapse by tracing output back to training input.

3 Characterizing Repetitive Story Traits↩︎

Current LLMs from OpenAI, Anthropic, Google, and AI2 write similar stories when prompted to “write a story” with no additional constraints.² We use four current models: Claude Haiku 4.5, Gemini 3.1 Flash-Lite, GPT-5.4-Mini, and OLMo 7b Thinking.³ We prompt each model with five requests (“Write a story,” “Please write a story,” “Write me a story,” “Tell me a story,” and “Please tell a story”) 1,000 times each, yielding 20,000 total stories totalling 12.8 million words.⁴

A typical example shown in 2 highlights three elements common across nearly all 20,000 stories: a location (19,864 stories), a character name (19,864 stories), and a profession (15,807 stories). In fact, the specific location (“lighthouse”), name (“Elias”), and profession (“keeper”) in this story appear in some combination across 66.6% of all generated stories. Light is likewise a common theme: 56% of stories generated by Claude are titled “The Lighthouse Keeper’s Secret” and the word “light” appears in 16,784 stories at an average rate of 3.2 instances per story. Other common names include Mara and Elara; locations include lighthouses and villages; and professions include clockmakers, fishermen, and librarians. Nearly all stories combine two or more of these three elements, suggesting models are sampling each from some common pool of candidates. What other words does the pool contain?

To construct vocabulary lists useful for downstream analysis, we use GPT-5.4-nano to identify token spans corresponding with story settings, characters’ first names, and their professions.⁵ We verify the presence of each extracted span before filtering candidates in three steps: (i) we tokenize strings on whitespace, yielding multiple tokens per string; (ii) for each story and category we retain the extracted token with the highest corpus-level frequency; and (iii) we retain all tokens emitted by at least half of all models. Removing incoherent candidates yields 663 tokens: 247 locations, 71 names, and 345 professions across all stories.

Within the candidate vocabulary words, we additionally select a Core vocabulary of 11 words using a changepoint analysis to find a minimal set of candidate tokens most common across all stories [15]. 88.3% of stories contain a Core token. The Core includes names (“Elias” is in 26.5% of all stories, “Mara” in 16.7%, and “Elara” in 13.1%), professions (“keeper” at 48.1% of all stories, while “clockmaker,” “baker,” “fisherman,” “librarian,” “mayor,” and “conductor” each occur in 1.9% to 6.6% of stories), and a single location: “lighthouse” with a frequency of 51.2%.

The core words and a second tier of 50 additional words are given in [appendix:common], while Core PPM rates per models are shown in 1. 98% of stories contain at least one of these 61 words, while 49.1% contain a full name-profession-location triple. Words (especially names) vary by model as shown in 9, but nearly all terms are used by all models. Professions suggest an idyllic, pre-modern setting: clockmaker, blacksmith, innkeeper, keeper, baker, fisherman. Other tokens describe curation (restorer, collector, caretaker).

4 Tracing Story Traits To Training Data↩︎

Frequent Core vocabulary in LLM story generation cannot be explained by the frequency of those words in published English fiction, pre-training data, or post-training data. We assess each potential source by comparing Core rates in our corpus with English corpora. We give rates in [tab:conlit-ratios].

The simplest explanation would be that Core tokens are common in English literature. We consulted CONLIT, a corpus containing 2,700 contemporary English novels published between 2007-2021 across 12 genres of fiction with \(\approx287\) million total words [16]. The frequency of Core tokens is far greater in generated stories than published fiction, e.g. “Elias” is 900 times more frequent in our corpus. To assess amateur fiction we consulted stories on the subreddit /r/writingprompts [17]. Rates are similar to CONLIT (10), suggesting models do not track human storywriting patterns.

To assess whether these tokens are common in English web data, we turn to OLMo 3, whose training data includes Common Crawl and is freely available. OLMo 3 was trained on \(\approx\) 3.89 billion predominantly human written documents during pre-training, of which 33 million are marked as Literature. Across these documents we find near-negligible Core PPM rates (e.g. “Elara” appears 0.7 times per million words). To ensure that we are looking at web stories and not non-fiction literature, we train a fiction classifier with 200k balanced samples from OLMo’s pre-training corpus annotated for narrativity with GPT-OSS 20b with the following prompt inspired by [18]: “Is this passage a work of fiction? Answer only with a number: 1 if yes, 0 if no.” We then train a FastText classifier and evaluate on 400 balanced samples of fiction and non-fiction in CONLIT, achieving a \(F_1\) of 0.84 (precision = 0.75, recall = 0.98). Filtering by this classification shows a slight (\(\approx 2 \times\)) increase in some Core words in the fiction portion, but nowhere near the rate in our generated stories.

If Core words are not common in web data, then one remaining source would be post-training data. But we find that OLMo’s post-training data exhibits our tokens at a lower rate than CONLIT. Using the same fiction classifier, we find that 78,958 stories in post-training data show the highest concentration of Core words of any of the training or literature subsets, but even then dramatically less than generated stories: “Elias” occurs 52.7 PPM in OLMo 3 stories vs. 2.7 in CONLIT, but 2428 in our corpus.

4.0.0.1 Which dataset(s) are contributing Core tokens?

This suggests OLMo 3 learns to write Core stories from relatively few samples. To understand which datasets are contributing these stories, we assign a binary score to each story indicating the presence of one or more Core tokens.⁶ We expected the majority of Core stories to appear in SFT data because WildChat (and derivatives) are the most story dominant source for OLMo at 59,266 total stories [2]. But only 1,803 of these stories contain Core tokens, and measuring the Core rate by alignment stage (e.g. SFT, DPO, and RL) shows DPO & RL contribute relatively more Core stories than SFT ([tab:t1-by-stage] & [tab:t1-by-source]). We find OLMo 3 learns Core vocabulary from 3,053 examples, or 3.8% of all stories observed during post-training.

5 Post-Training Story Genres↩︎

To better understand what kinds of stories OLMo 3 encounters during post-training, we trained a 10-topic LDA topic model [19] against the full post-training story corpus (3). We find a diverse range of content, with dominant topics including fan fiction for popular Japanese media, video games, and American cartoons. As expected given their relative frequency, “lighthouse” stories do not form a single topic, and are instead spread out across our discovered topics. They are particularly concentrated in clusters containing generic fiction, but they nonetheless fail to dominate any topic. A close reading reveals several topics frequently feature stories containing inappropriate humour and adult content, surprising considering OLMo 3 will not typically emit inappropriate content when writing. Future work will want to investigate whether these stories fail to trigger safety and quality filters used for data cleaning, and why if so.

6 Conclusion↩︎

When given little direction, current frontier models write stories using a narrow catalog of names, places, and occupations. Recurring characters in these stories include Elias, a lighthouse keeper. Elias is unusual; the name is uncommon in literature, web data, and even post-training data. We have found that of the 78,958 stories exposed to OLMo 3 during post-training, only \(\approx 3{,}053\) stories contain one or more of these 11 unusual tokens. But despite constituting only 3.8% of post-training stories — and \(7.71 \times 10^{-7}\) of the \(\approx4\) billion total documents OLMo 3 was trained on — these “lighthouse” stories hold a disproportionately large influence over what stories the model writes in practice. This suggests models do not simply mimic the dominant patterns in their training corpora. Future work will want to determine whether alignment causes models to prefer the “safest” (for work) samples in post-training, avoiding the potentially unsafe topic matter of many stories they otherwise encounter.

Limitations↩︎

Our experiment is expressly monolingual to prevent confounders stemming from unanticipated multilingual behaviour, but it would be valuable for future work to explore how multilingual storywriting prompts impact the phenomena observed.

7 Prompts↩︎

In this section we provide all prompts used over the course of the experiment.

7.0.0.1 Prompts for story generation.

The five prompts used to generate stories are as follows.

Write a story.

Please write a story.

Write me a story.

Tell me a story.

Please tell a story.

7.0.0.2 Name, location, profession extraction.

The system prompt for extracting names, locations, and professions is as follows.

You extract structured metadata from short stories.

Return JSON only, with this exact schema:
{
  "character_names": ["first name only", ...],
  "settings": ["place or location noun phrase", ...],
  "professions": ["profession or role noun", ...]
}

Rules:
- Output valid JSON and nothing else.
- `character_names` must contain only first names for named human or human-like characters.
- Exclude surnames, titles, pronouns, groups, and unnamed roles.
- `settings` should be concise setting/location phrases from the story, e.g. "Lighthouse", "Village square", "Oakhaven".
- `professions` should be concise occupations or stable roles, e.g. "Clockmaker", "Lighthouse keeper", "Baker".
- Deduplicate while preserving first appearance order.
- If a field has no items, return an empty list.
- Every returned string must be an exact token span copied from the story text.

The user prompt for extracting names, locations, and professions is as follows.

Read the story and answer these questions:
1. Who are the characters in the text?
2. What is the character's role in the text?
3. What is the setting?

Return JSON only in this exact schema:
{{
  "character_names": ["first name only", ...],
  "settings": ["place or location noun phrase", ...],
  "professions": ["profession or stable role noun phrase", ...]
}}

Additional rules:
- For `character_names`, include only first names for named human or human-like characters.
- For `professions`, map each character's role in the text to concise occupations or stable roles when present.
- For `settings`, list the main setting locations or place names.
- Every returned item must be an exact contiguous span copied from the story text.
- Do not normalize, paraphrase, singularize, pluralize, or invent labels.
- If the exact answer is not present as a span in the story, omit it.
- Deduplicate items while preserving first appearance order.
- Use empty lists when something is absent.

Story:
{story}

The prompt for identifying fictionality is as follows.

``Is this passage a work of fiction? Answer only with a number: 1 if yes, 0 if no.''

8 Common Tokens↩︎

In [tab:npl-tokens] we present all 61 Core and additional tokens common to LLM generated short stories we identified in our experiment.

4pt

9 Per-Model Core PPM↩︎

We present Core token concentrations across all stories generated by each model in 1.

10pt

10 Core Token Frequencies on Reddit↩︎

We calculate Core token frequencies on non-published human-written literature from Reddit in 2.

References↩︎