1 Introduction↩︎

Large language models (LLMs) have significantly advanced open-ended text generation, enabling their use for creating character personas and simulating complex interactions [1], [2]. Although LLMs have demonstrated strong narrative generation capabilities, they suffer from character inconsistency and plot discontinuity, limiting their ability to create high-quality long-form narratives [3]–[5].

These failures become pronounced in multi-character environments, where LLMs struggle to balance narrative goals and character actions with complex relationships and interactions [6]–[8]. Recent work, including StoryVerse [9], Agents’ Room [10], and IBSEN [11] have explored the use of multi-agent systems in narrative generation, but they continue to rely on textual memory, limiting their ability to generate coherent stories. [12]–[15]. In addition to creative content generation, there remains a need to thoroughly evaluate this content. Existing work has made progress on long-form narrative understanding and factuality [8], [16]–[22]. However, these methods do not provide a framework for identifying hallucinations within long-form generated narratives.

In this work, we introduce Magnet, a multi-agent story generation system to develop coherent long-form narratives and sec:Atlas, a graph-based hallucination evaluation pipeline that identifies inconsistencies by comparing the world state representation of the present scene against those of previous scenes [23]. Our work aims to address the research question: Can long-form narratives emerge from interactions between character personas and updating goal states when autonomous agents interact through a shared world state? Through hierarchical editorial evaluation, pairwise rubric analysis, and sec:Atlas, we show that Magnet improves narrative coherence, providing a foundation for future work in long-form narrative generation.

Our main contributions include:
) Magnet, a multi-agent action-critic-narrator generation framework with character-grounded action generation, critic revision, narrator-driven prose writing, a shared world state, and evolving story goals for long-range coherence
) sec:Atlas, a graph-based evaluation pipeline to detect hallucinations in long-form stories for interpretable signals on model failures
) Empirical evaluation metrics showing that Magnet reduces editorial critique counts, increases pairwise rubric scores, and reduces hallucinations in long-form stories

2 Related Work↩︎

Multi-Agent Narrative Generation Recent work has demonstrated that the decomposition of narrative generation into interacting language model agents, where each agent is responsible for a high-level role, such as planning, character development, or scene writing, can improve long-form narrative coherence [10], [24]. Additionally, frameworks such as IBSEN [11] and StoryVerse [9] introduce director-actor architectures where a planning module guides character agents through structured narrative objectives, showing that separating global narrative planning from local character behavior allows for consistency and controllability [25], [26]. However, these systems still rely heavily on prompt-level coordination, where narrative state is maintained in text rather than tracked through a persistent shared representation [27].

Structured State Tracking A key challenge in long-form narrative generation is maintaining consistency over time, especially in multi-character environments [24]. Prior work in structured generation has explored the use of explicit state representations to improve consistency in text-based environments [28], [29]. For example, work on instilling commonsense knowledge into interactive agents has demonstrated that structured representations of world states can improve action prediction and narrative coherence in decision-making tasks [30]–[32]. Our work builds upon these ideas by incorporating a world state representation, allowing narrative facts, character states, and sequentially-updating goals to evolve across scenes without requiring external supervision.

Story Evaluation Recent work has shown that LLM-as-a-Judge methods correlate strongly with human preferences [33]–[37]. Among these approaches, pairwise and rubric-based comparisons improve consistency and agreement with human judgment in subjective tasks [22], [38]–[40]. Additionally, evaluating consistency in long-form narratives requires tracking complex world states, making graph-based verification an effective approach [17], [18], [41]–[43]. Benchmarks such as STAGE [23] have highlighted the importance of representing entities, events, and relationships explicitly for evaluating narrative understanding and generation. While hallucination benchmarks such as HaluEval [44] and HalluLens [45] focus on errors against external knowledge, and HelloEval [22] evaluates long-form generation quality, current methods do not explore identifying inconsistencies within the generated text [19].

3 Methodology↩︎

3.1 Goal Sequencing↩︎

Each story begins with a high-level goal that provides direction for narrative development. When the goal is completed, an Opus 4.7 [46] goal generator generates a follow-up goal. Empirically, we observe that after roughly 15 time steps, character actions became increasingly repetitive. To avoid stalled narratives, our framework replaces goals that have not been completed in 15 time steps. After approximately 40 steps, successive goals also began to repeat similar story trajectories. To maintain narrative diversity, the system generates a goal that changes the direction of the story into a new domain every 40 steps. These larger transitions are intended to introduce new conflicts as the previous goal concludes.

3.2 Story Generation↩︎

Each character is defined with a character persona that contains attributes such as relationships, personality, goals, character description, role, location, and abilities. At each time step, character agents receive the character persona, current story goal, recent story history, and prior world variables that are relevant to that character, and use a DPO-tuned Gemma-4-31B-it model [47] to generate an action that describes what the character intends to do next.

After an action has been generated, a Gemini 2.5 Flash [48] LLM critic is asked to evaluate the quality of the proposed action. The critic is provided with the action, character persona, current story goal, and current world state, and is instructed to determine whether the action is relevant, specific, and consistent with both the character and the current scene. If an action does not effectively advance the goal of the story, is vague, out of character, implausible, or repetitive, the critic will provide feedback for the character agent to generate another action. If the action is accepted, the critic generates updates to the world state related to the proposed action.

After all characters have generated an action, an Opus 4.7 [46] narrator produces the next paragraph of the story. The narrator receives the proposed actions, current world state, current story goal, and prior story paragraphs. Then it selectively chooses actions that are best suited for the current scene and creates coherent story prose using the selected actions.

If the narrator selects an action that the critic deemed completed the previous goal, the Opus 4.7 [46] goal generator will generate a new goal. When generating a new goal, the model receives the previous goal, recent story paragraphs, current world state, all character personas, and all prior goals. Using this context, the goal generator is instructed to produce a new concrete goal that is relevant, not repetitive, and will allow the story to develop further. In addition to generating the next goal, the model also generates a transition paragraph that bridges the previous story paragraph into the new goal. The transition is written as story prose rather than world updates or summary text and is included in the final story.

3.3 World State↩︎

We represent the story state as a directed graph that contains a root world node, character nodes, state variable nodes, and edges that represent the relationships between the nodes. State variable nodes are flattened into a dictionary representation that is provided to the agents during generation. To update the world state, the selected actions’ world state updates are applied using an overwrite conflict resolution strategy. If an update writes to a previously unseen key, that key is inserted; if it writes to an existing key, the new value replaces the old one. For example, Lucia begins with holding_back_evidence=true, but later, after she reveals the withheld letter, the committed graph contains holding_back_evidence=false and lucia_has_disclosed_letter=true. We apply these updates sequentially, but only after narrator selection, so unselected actions’ updates do not enter the world state. If a critic proposes an implausible update, it is typically filtered out because the associated action is not selected. If multiple characters have conflicting updates, the narrator usually only selects the subset of actions that do not conflict, and only those actions’ updates are merged. In practice, this keeps overwrite simple while limiting new inconsistencies.

3.4 Editorial and Rubric-based Evaluation↩︎

We use GPT 5.4 mini [49] as an expert story editor to analyze generated stories at multiple narrative levels. The LLM evaluator is asked to produce editorial annotations on logical consistency, thematic coherence, and character arc completion when provided the whole story; goal-conflict-outcome progression, hook-and-close quality, and chapter necessity on 5 randomly selected scenes; and rhythm, clarity, and syntax variety on 5 randomly selected sentences (6.2.1). The resulting annotation counts are aggregated to produce quantitative statistics for each generated story.

Additionally, we perform a pairwise rubric-based evaluation with GPT 5.4 mini [49]. The evaluator is provided with all the stories in a randomized order and is instructed to evaluate them in all categories across the three hierarchies. The evaluator assigns a rubric score out of 100 for each category and also assigns an overall quality score out of 100 for each hierarchy level.

We validated the LLM judge’s critiques and reasoning on a 20-page story and observed 90% alignment with human judgment (6.2.2, 6.2.3).

3.5 Graph-Based World Representation Evaluation↩︎

For hallucination detection, we introduce sec:Atlas, a graph-based world representation evaluation framework.

For each screenplay, the pipeline constructs the graph through three sequential passes over the story. The first pass decomposes the script into scene-level event units, representing each as a node with a name, description, and textual evidence drawn from the screenplay. Building on this, the second pass extracts entities in relation to the events they appear in, and the third extracts the relations connecting those events and entities. Only nodes and edges recognized by the schema are retained in the graph.

The schema defines seven node types—Character, Event, Location, TimePoint, Object, Vehicle, and Concept—and organizes edge types into five functional groups: event-role edges (performs, undergoes, experiences); social edges (kinship_with, affinity_with, hostility_with, affiliated_with); inter-event edges (precedes, occurs_after, causes, contrasts_with, references); spatiotemporal edges (occurs_at, occurs_on, located_at, present_on); and object-related edges (possesses, uses, part_of, is_a).

For each node, the pipeline performs a separate attribute-extraction pass in which it revisits the screenplay evidence tied to that node and generates a set of attributes, such as role, state, temporal markers, or descriptive qualifiers, directly from the text. Then, starting at the second scene, an LLM proposes hallucinations based on the inconsistencies between the current scene’s world state and text, and previous scenes’ world states. These proposals are then verified, using the story text to check if there is sufficient evidence to justify the inconsistency, providing interpretable graph-grounded results.

3.6 Graph-Based Evaluation Pipeline Performance↩︎

In order to validate sec:Atlas, we evaluate its performance compared to a vanilla LLM-as-a-Judge approach, asking an LLM to identify hallucinations. We use Claude Sonnet 4.6 [50] to generate the stories with synthetic hallucinations, which are thoroughly verified by a human annotator (6.3). We use GPT-5.4-mini [49] for graph generation and GPT-5.4 [51] for hallucination detection. We also use GPT-5.4 [51] for the LLM-as-a-Judge System. Table 1 shows that, Atlas outperforms the LLM-as-a-Judge System across all 3 stories.

5pt

4 Experiments and Results↩︎

4.1 Experimental Setup↩︎

Before running the system, the user needs to input both a high-level goal that the story aims to achieve and character definitions for each character involved.

For our character agent, we apply a LoRA Direct Preference Optimization (DPO) [52] adapter on Gemma-4-31B-it [47] to improve action relevance, character consistency, and reduce repetitive actions. To generate the data for DPO, we instruct the Gemini 2.5 flash [48] action generator to generate two candidate actions. A separate Gemini 2.5 flash [48] judge LLM then compares the two generated actions and selects the action that is more grounded, in-character, less repetitive, and fits better in the current scene. A total of 1,012 train and 53 evaluation preference examples were used. Refer to 6.1 for the detailed hyperparameters.

We evaluate Magnet at three increasing generation lengths of 2, 20, and 100 pages. For the 2 and 20 page settings, we evaluate three different story generations and report the average results for each metric. Due to the substantially higher computational cost of generating and evaluating 100-page stories, we treat the 100-page setting as a proof of concept and use only one story.

4.2 Baselines↩︎

We compare Magnet against two baselines: a standard prompting approach where Opus 4.7 [46] generates a story without explicit character agents, a critic module, a world state, or goal sequencing, and IBSEN [11]. Unlike the single pass baseline, IBSEN is a multi-agent generation framework with a scene director and character actors. In the released implementation, the story generation is guided by the centralized director, who determines the narrative progression and coordinates the actor interactions to achieve the plot goals. We selected IBSEN because it provides a publicly available implementation of a multi-agent story telling framework, enabling reproducible comparisons. To keep the comparison fair, we provided both baselines with the same story definitions used for Magnet: the same character personas and story objectives and targeted the same output length.

4.3 Performance Analysis↩︎

At shorter generation lengths, both the baselines and Magnet are capable of producing coherent stories. In 2-page stories, Magnet received similar amounts of editor annotations compared to the two baselines (Table ¿tbl:tab:main95results?). As generation length increases, the performance gap increases. At 20-page stories, Magnet on average received 9 fewer annotations than single model prompting and around 30 fewer annotations than IBSEN (Table ¿tbl:tab:main95results?). The largest improvements occur in the 100-page generations, with Magnet receiving 41 fewer annotations than single model prompting and 34 fewer annotations than IBSEN (Table ¿tbl:tab:main95results?). Pairwise rubric evaluation further supports these findings, with Magnet achieving higher scores than both baselines at each story length and hierarchical level (Table [tbl:tab:rubric95results]). IBSEN’s comparatively weaker rubric scores may stem from its design as a drama-script generation framework rather than a prose narrative generator. While its dialogue-driven scenes may appear coherent, direct comparison with prose narratives may place it at a disadvantage due to differences in storytelling format.

On sec:Atlas, both the baseline story, IBSEN, and Magnet recorded 0 hallucinations on the 20-page-long story (Table ¿tbl:tab:hallucination95results?). However, on the 100-page story, the baseline recorded 12 hallucinations, IBSEN recorded 11, and Magnet recorded 6, a 50% and 45% decrease, respectively. This highlights how, as narratives expand, Magnet’s character-grounded and dynamic goal framework enables it to remain much more coherent (Table [tbl:tab:all95hallucinations]).

5pt

4.4 Ablation Analysis↩︎

We perform a small ablation using the hierarchical editorial evaluation framework on a 20-page story to evaluate the contribution of individual framework components. Specifically, we evaluate variants of the system with the world state updates removed, DPO removed, dynamic goal updates removed, and critic module removed, each of which received more total LLM editorial annotations than the total amount of annotations Magnet received (Table 2).

5 Conclusion↩︎

In this work, we introduced Magnet, a multi-agent character-driven long-form narrative generation framework, and sec:Atlas, a graph-based hallucination evaluation pipeline for long-form narratives. Across all evaluations, our results suggest that the primary benefits of Magnet emerge in higher-level narrative organization and long-range coherence, and that Atlas provides interpretable graph-grounded evaluation for coherence. Additional experiments further suggest that Magnet’s improvements arise from the critic-guided refinement, DPO adapter, updating world state, and dynamic goal sequencing. Overall, this work provides a foundation for more controllable and structurally coherent long-form narrative generation.

5.1 Limitations↩︎

While our work is effective at generating high-quality content that can be scaled to hundreds of pages, it remains computationally expensive due to repeated interaction between multiple generation modules. Thus, our editorial and pairwise evaluations were performed on three Magnet-generated, IBSEN, and baseline stories for 2 and 20 page lengths, and one for the 100 page and ablation. Computational costs also limited the amount of preference data used to train the DPO-based character agent, and larger preference datasets may further improve agent behavior and narrative consistency. Additionally, Magnet depends on multiple closed-source models, which may introduce variability. Our evaluation pipeline also partially depends on LLM-based judges, which may have subjective biases and reasoning despite manual validation efforts. Atlas relies heavily on graph information, which means that if the LLM output is sparse, the pipeline may not be able to accurately detect hallucinations. We also test our framework only on relatively short English long-form content; additional evaluation and refinement may be required to scale across longer texts and different languages to further explore AI-generated long-form content’s applicability to creative contexts.

5.2 Ethical Considerations↩︎

Synthetic Content Misuses Although Magnet improves long-form AI story generation, it could also be misused to create deceptive synthetic narratives. The increased narrative consistency and coherence may make the generated text appear more human-like, raising concerns regarding misinformation. Therefore, these AI-generated stories should be clearly disclosed, and appropriate safeguards and policies should be adhered to (6.8).

Human Creativity Systems like Magnet may raise concerns about the impact of AI-generated narratives on human-created content. As models become increasingly capable of producing coherent long-form stories, the role of human writers could be greatly diminished, or it could lead to over-reliance on automated content generation. Therefore, AI-generated creative works should be used responsibly and appropriately attributed.

6 Appendix↩︎

6.1 Detailed Experimental Setup↩︎

Models The critic module uses a temperature of 0.1. The narrator uses a temperature of 0.35, and the goal generator uses a temperature of 0.1. The DPO character agent uses a temperature of 0.2. The editorial and pairwise rubric evaluation uses a temperature of 0.1. The baseline model uses a temperature of 0.35. The aforementioned temperatures were derived from experimentally testing each model’s temperature hyperparameter to optimize for its generation performance.

For sec:Atlas, both the graph creation pipeline and hallucination pipeline use a temperature of 0.0 in order to keep the pipeline as deterministic as possible.

IBSEN For evaluating IBSEN, we used the default model in the code and updated the scripts to match our story definitions.

Computational Costs In total, we spent $299.94 on API usage for closed-source models. Additionally, we used rented cloud GPU infrastructure for LoRA DPO fine-tuning of GEMMA-4-31B-it [47] and long-form story generation using the resulting trained adapters. Training and inference were conducted on NVIDIA A100 SXM4, GH200, and H100PCle GPUs. Across all experiments, out total computational budget was approximately 94 GPU hours corresponding to a total cost of $212.27.

6.2 Summary of Human Verification↩︎

6.2.1 Story Generation↩︎

Upon reviewing the generated stories, we observed that all three story generation frameworks produced prose with comparable sentence length distributions. Additionally, chapters were each around 2000 words, reducing the likelihood that differences in editorial annotations and pairwise rubric scores were primarily caused by variations in length.

6.2.2 LLM Editor↩︎

6.2.3 Pairwise Rubric Scores↩︎

6.2.4 Hallucinations↩︎

2pt

6.3 Atlas Benchmark Results and Human Verification↩︎

6.3.1 Atlas Detailed Benchmark Result↩︎

2pt

6.3.2 Atlas Ablation Result↩︎

2pt

We perform a small ablation to show how Atlas is able to scale across models. Although results are noticeably worse for GPT 5.4 mini than when using GPT 5.4, they show that Atlas remains stronger than the LLM-as-a-judge baseline across different model settings, suggesting that its advantage is not dependent on a single model configuration.

6.3.3 Story Generation Prompt↩︎

The prompt used to generate the stories is shown below:

You are generating synthetic screenplay test data for a graph-based temporal hallucination evaluation system.

Create THREE original screenplay-style stories with these exact story IDs:
- synthetic_hallucination_001
- synthetic_hallucination_002
- synthetic_hallucination_003

Each story must be generated as a pair of files:
- English/<story_id>/script.json
- English/<story_id>/annotations.json

The six outputs must correspond exactly to:
1. English/synthetic_hallucination_001/script.json
2. English/synthetic_hallucination_001/annotations.json
3. English/synthetic_hallucination_002/script.json
4. English/synthetic_hallucination_002/annotations.json
5. English/synthetic_hallucination_003/script.json
6. English/synthetic_hallucination_003/annotations.json

For each story, create:
- 5 to 10 scenes.
- Approximately 20 to 30 screenplay pages worth of content.
- A coherent recurring cast.
- Clear continuity across scenes: locations, character states, relationships, possessions, injuries, secrets, obligations, constraints, and prior events.
- 15 to 30 manually injected temporal hallucinations or continuity errors in later scenes.

The hallucinations must be prior-state errors:
- A later scene claims or implies something that conflicts with or is unsupported by earlier established story facts.
- Do NOT label normal new information as hallucination if the current scene itself clearly justifies it.
- Do NOT create hallucinations in scene 1.
- Make every hallucination easy to locate: the earlier fact and later violating event must both appear explicitly in the screenplay.

Every annotation MUST use exactly this format:
"In scene X, [FACT] was established, yet in scene Y, [EVENT] happens"

Output exactly SIX JSON code blocks and no extra prose.
Before each JSON code block, put a single filename label exactly like:
  English/synthetic_hallucination_001/script.json
Then put the JSON code block.

script.json requirements:
- Must be a JSON array.
- Each scene object must have exactly:
  - "_id":      integer, starting at 0 and increasing by 1
  - "title":    string like "1, INT. LOCATION - DAY."
  - "subtitle": string, usually ""
  - "content":  string containing full screenplay text
- Scene numbers in titles must match _id + 1.
- All newlines inside "content" must be escaped as \n.
- The JSON must be valid and parseable.

annotations.json requirements:
- Must be a JSON array.
- Each hallucination object must have exactly:
  - "id":            integer, starting at 1 and increasing by 1
  - "hallucination": string
- Every hallucination string must use exactly:
  "In scene X, [FACT] was established, yet in scene Y, [EVENT] happens"
- Scene X must always be earlier than scene Y.
- Every annotation must correspond to a real injected hallucination in that story.
- The screenplay must not explicitly explain away the hallucination.
- Make the earlier fact and later violating event easy to find by reading the scene text.

Do not include markdown explanations, summaries, comments, or extra keys.
Only include the filename labels and the six JSON code blocks.

The same model also synthetically embeds the hallucinations within each story and produces an annotation JSON file detailing where each hallucination is located. However, some LLM-annotated hallucinations are inaccurate. For each story, we present a table listing every hallucination candidate initially annotated by the LLM, the human verdict on each candidate, and the justification, if applicable.

6.3.4 Detailed Human Verification↩︎

6.4 Detailed Editor Annotation Counts↩︎

2pt

2pt

2pt

6.5 Detailed Pairwise Rubric Scores↩︎

2pt

2pt

2pt

6.6 Hallucination Results on Magnet and Baseline↩︎

6.7 Detailed Ablation Annotation Counts↩︎

2pt

2pt

2pt

6.8 Licenses↩︎

Code and anonymized repositories associated with this work are released under the MIT License:

• Magnet : https://anonymous.4open.science/r/MAGNET-ED2F/README.md

• Atlas : https://anonymous.4open.science/r/ATLAS-78CF/README.md

Gemma 4 is distributed under the Apache License 2.0 (https://ai.google.dev/gemma/docs/core).

IBSEN’s codebase is distributed under the MIT License (https://github.com/OpenDFM/ibsen).

All models and code are used in accordance with their respective licenses or terms of service.

References↩︎