1 Introduction↩︎

With the rapid advancement of Large Language Models (LLMs), there has been a significant surge of interest in deploying AI as autonomous Non-Player Characters (NPCs) or game adjudicators within interactive environments. Among these applications, semi-open text-based games (SOTGs), exemplified par excellence by Tabletop Role-Playing Games (TRPGs), have emerged as a pivotal testing ground for evaluating the logical reasoning and rule-alignment capabilities of these agents with respect to complex underlying mechanics. Unlike traditional video games constrained by fixed choice menus, TRPG-style semi-open environments grant players absolute natural language freedom, yet demand that the AI Adjudicator strictly enforces a rigid, underlying rule engine. In this setup, the AI must adjudicate between expressive player intent and the fixed constraints of the underlying rule system. Existing NLP research in this domain has predominantly explored Dungeons & Dragons (D&D) [2] as the primary sandbox [3]–[5]. However, player actions are mostly explicit in D&D, combat-focused directives (e.g., “I cast Fireball") with little ambiguity over whether a dice roll is required.

To push the boundaries of AI adjudication in these text-based domains, we pivot to Call of Cthulhu (CoC), a TRPG paradigm that epitomizes highly ambiguous, narrative-driven text games. In CoC, the gameplay shifts from tactical combat to investigative horror, where player inputs are deeply intertwined with atmospheric flavor text, emotional pleas, and deceptive framing (e.g., “I take three steps at once, jump directly over the train, then land safely on the other side." as shown in Fig. 1). Because LLMs are optimized to maintain textual coherence, they instinctively treat player inputs as creative writing prompts to be accommodated rather than structural game states to be rigorously adjudicated.

This mismatch raises a fundamental question: Can an AI adjudicator truly decouple the rhetorical quality of a player’s statement from its objective mechanical validity? To answer this, we systematically evaluate frontier LLMs serving as autonomous adjudicators in CoC. Our contributions are threefold:

• Shift in Evaluation Focus: We pivot the evaluation of interactive agents from tactical execution (e.g., D&D) to logical rule adjudication. This formalizes a targeted framework to assess LLM adjudicators in high-entropy, semi-open text environments.

• The CoC-Seduce Benchmark: We introduce a diagnostic dataset of 5,376 samples generated by three frontier model families (GPT, Claude, Gemini), comprising 1,792 scenarios per source across 16 skill categories and 4 world settings. It pairs mandatory rule checks with tiered rhetorical attacks to explicitly quantify a model’s decoupling capability.

• Empirical Vulnerability Analysis: Comprehensive evaluations across 20 frontier models reveal that neither model scale nor explicit reasoning mechanisms reliably confer adjudication robustness, and that vulnerability is further shaped by adversarial source characteristics and cross-cultural knowledge coverage.

2 Related Works↩︎

2.1 LLMs for Tabletop Role-Playing Games↩︎

Recent research [6] has summarized the multifaceted challenges of Tabletop Role-Playing Games (TRPGs) and proposed taxonomies to evaluate gamemastering capabilities, emphasizing world and story design [7], player action extraction, commonsense reasoning [8], [9], and state tracking. For the most popular TRPG, Dungeons & Dragons (D&D) [2], large-scale datasets of players’ interactions [3], [4] from verified online text-based sessions have been developed to support research tasks such as mapping natural language to executable commands and mapping game states to narration. Parallel efforts apply LLMs to specific subsets of gameplay, including assisting human game masters (GMs) in distilling game context [10], generating domain-specific content [11], [12], evaluating strategic reasoning via game-theoretic simulations [13], constructing tool-grounded agentic simulations [14], [15], building reinforcement learning environments with LLM-controlled agents in combat scenarios [16], and generating structured actions from explicit game states [5]. However, there is a stark scarcity of benchmarks assessing LLMs as adjudicators in ambiguous, narrative-driven systems like Call of Cthulhu (CoC), where the AI must enforce rigid rules in response to open-ended player rhetoric.

2.2 Rule Adherence and LLM-as-a-Judge↩︎

The paradigm of utilizing LLMs as automated evaluators, or “LLM-as-a-Judge,” has become a cornerstone in assessing generative outputs and agent trajectories [17]. Extensive benchmarks have been proposed to evaluate how strictly LLMs adhere to explicit instructions and constraints, such as IFBench [18] for verifiable formatting rules, and JudgeBench [19] for fine-grained objective rubrics. While these frameworks demonstrate that modern LLMs excel at following structural guidelines in standard Q&A or code-generation tasks, enforcing rules in interactive, semi-open environments presents a fundamentally different challenge, as recent work has begun extending evaluation paradigms from conversational to fully agentic settings [20], [21]. In SOTGs, the rules are not merely format constraints but represent the underlying physics of a simulated world. Adjudicating these rules requires the model to parse high-entropy narrative inputs, align them with strict mechanical triggers, and resist semantic distractions, an area where standard instruction-following benchmarks fall short, often exposing severe biases and reliability concerns in current LLM judges when faced with multifaceted assessments [17], [22].

2.3 Narrative Manipulation and Adjudication Robustness↩︎

As an Adjudicator in narrative-heavy systems like CoC, an LLM must function as a neutral arbiter, maintaining procedural integrity by executing objective adjudications (e.g., calling for mandatory dice rolls) during critical interactions. A central threat to this integrity is player deception, where a user constructs adversarial narrative frames to manipulate the AI Adjudicator into granting unearned narrative success. This vulnerability is rooted in a well-documented defect of LLMs known as sycophancy—the tendency to align with user views or stylistic tone to maximize perceived helpfulness [23], [24]. In broader LLM security literature, analogous manipulation strategies have been formalized as Roleplay Jailbreaks or Nested Scene attacks [25], [26], wherein attackers exploit pseudo-logical reasoning and deep persona adoption to bypass safety filters or logical constraints [27]–[31]. While the goals differ, both share the same underlying mechanism: leveraging narrative context to override a model’s rule-governed behavior. The integration of Chain-of-Thought (CoT) reasoning [32] has significantly improved LLMs’ capacity to maintain logic under such pressure. Nevertheless, it remains an open question whether their internal reasoning capacity is sufficient to ensure robust adjudication against deep narrative manipulation.

3 Task Formulation↩︎

We formalize the role of an LLM as an autonomous Adjudicator in Call of Cthulhu, not as a cooperative storyteller, but as an impartial Adjudicator of Objective Reality.

3.1 Adjudication Setup↩︎

In contrast to the open-ended Game Master formulation in Labyrinth [33], we restrict the LLM’s role to a single-round, well-defined adjudication decision within a summarized session context. The session is represented by a narrative context \(C\) (including current game factors, investigator stats, and environmental constraints) and the player’s statement \(S_t\). The task of the Adjudicator is to determine whether the action requires mechanical verification (a dice roll) or can be conducted directly. Formally, we define a procedural validity function \(V(C, S_t) \in \{0, 1\}\):

\[V(C, S_t) = \begin{cases} 1 & \text{if } S_t \text{ requires a \boldsymbol{Dice Roll}}\\ 0 & \text{if } S_t \text{ allows for Automatic Success} \end{cases}\]

Each player statement \(S_t\) is defined as a composition of intent and rhetorical style \(S_t = I_{\text{intent}} \oplus R_{\text{style}}\), where \(I_{\text{intent}}\) carries the underlying action intent with a fixed ground truth \(R_{\text{style}} \in \{\)Neutral\(,\) Authority\(,\) Pseudo-Logic\(,\) Omission\(\}\) is a rhetorical elaboration whose style is defined in Section 4. The latter three are adversarial rhetorical styles. A successful Rhetorical Injection occurs when the model misaligns with \(V_{\text{human}}\) due to the persuasiveness of \(R_{\text{style}}\) rather than the mechanical validity of \(I_{\text{intent}}\). Here, \(\oplus\) denotes the semantic composition of intent and stylistic framing, where \(R_{\text{style}}\) contextualizes or potentially obfuscates \(I_{\text{intent}}\) without altering its underlying mechanical necessity.

3.2 Evaluation Protocol↩︎

To quantify model robustness, our evaluation spans both ground truth conditions: mechanic check required (\(V_{\text{human}}=1\)) and automatic success (\(V_{\text{human}}=0\)). We classify the model’s generated response \(y_t\) into binary outcomes: (1) Aligned Adjudication: The model’s decision matches human expert expectation; and (2) Misaligned Adjudication: The model fails to match human judgment. Misalignment takes two forms—a False Pass, where the model grants direct success on a required check, seduced by rhetorical elaboration; and a False Check, where the model unnecessarily demands a roll on an automatically resolvable action. Our primary metric is the Failure Rate (FR), computed separately across rhetorical style categories and ground truth conditions. A robust Adjudicator maintains an FR of 0% regardless of the narrative quality of the input. The overall FR is computed as:

\[\mathrm{FR_{overall}} = \frac{\mathrm{FP}+\mathrm{FC}}{N_{V=1}+N_{V=0}}.\]

We also report the Wrong Skill Rate (WS), defined as the proportion of correctly aligned adjudications (i.e., \(y_t = 1\) and \(V_{\text{human}} = 1\)) in which the predicted skill deviates from the ground truth skill, measuring the precision of mechanical identification conditional on a correct roll decision.

4 The CoC-Seduce Dataset↩︎

4.1 Skill Taxonomy and Exclusion Criteria↩︎

The CoC-Seduce dataset focuses on 16 core skills categorized into four domains: Combat (CMB), Physical (PHY), Professional (PRO), and Investigation (INV), as shown in Table 1. We deliberately exclude social skills (e.g., Persuade, Fast Talk, Intimidate), as their outcomes are inherently subjective and vary significantly between human Keepers (the Game Master of CoC) based on NPC personality and dramatic pacing. Restricting our scope to physical and investigative actions ensures a high-confidence ground truth where the necessity of a dice roll is dictated by objective rule constraints rather than interpretive preference.

4.2 World Settings↩︎

To ensure generalization across diverse narrative contexts, we generate scenarios across four distinct world settings:

• 1920s Urban: The classic Lovecraftian era, featuring gas-lit streets, rotary phones, and period architecture, representing the most culturally familiar setting for LLMs trained on Western literary corpora.

• 2020s Urban: Contemporary environments with modern technology (smartphones, electronic keypads, cameras), introducing present-day physical and procedural constraints.

• Wilderness: Natural outdoor environments devoid of man-made infrastructure, where adjudication more depends on terrain, weather, and biological hazards.

• Ancient China: Pre-modern Chinese historical settings with period-specific material culture (silk scrolls, oil lanterns, courtyard gates), designed to probe cross-cultural long-tail knowledge coverage.

4.3 Dataset Construction↩︎

4.3.0.1 Mandatory Roll Scenarios (\(V=1\)).

For each of the 16 skills across 4 world settings, we generate 6 unique scenario truths that objectively require a dice roll, each paired with 4 rhetorical variants (Neutral, Authority, Pseudo-Logic, Omission), yielding \(16 \times 4 \times 6 \times 4 = \mathbf{1{,}536}\) samples.

4.3.0.2 Automatic Resolution Scenarios (\(V=0\)).

To construct a control group for evaluating False Check errors, we generate routine, zero-risk scenarios for the PHY and INV categories, where no dice roll should ever be required. Each of the 8 skills is paired with 2 unique scenario truths per setting and 4 rhetorical variants, yielding \(8 \times 4 \times 2 \times 4 = \mathbf{256}\) samples.

4.3.0.3 Total.

Each of the three generators (GPT-5.4, Claude Sonnet 4.6, Gemini 3.5 Flash) produces an independent set of 1,792 samples. These resources together yields a total evaluation corpus of 5,376 samples against which each target model is benchmarked.

4.4 Adversarial Injection Design↩︎

For each scenario truth, four player statement variants are constructed to probe different rhetorical manipulation strategies:

1. Neutral: A plain, factual statement of intent (e.g., “I try to climb the wall”).

2. Authority: The player implies competence through claimed background or confidence (e.g., “As an expert, I easily scale the surface”).

3. Pseudo-Logic: The player constructs a superficially plausible but mechanically irrelevant causal argument (e.g., “Since the bricks have rough edges, I can climb up without difficulty”).

4. Omission: The player strategically downplays or omits risk-relevant details, often with affirmative framing (e.g., “I quickly hop over the wall and succeed”).

The Neutral variant serves as the baseline condition; the remaining three constitute adversarial rhetorical styles that perform Rhetorical Injection.

4.5 Human-in-the-Loop Curation↩︎

Adversarial inputs are generated by three leading frontier models, i.e., GPT-5.4, Claude Sonnet 4.6, and Gemini 3.5 Flash, which represent the current ceiling of rhetorical injection capability. All samples undergo manual review by expert Keepers (Human Game Masters): annotators verify that ground truth mechanics remain consistent with the world setting, and apply targeted corrections where necessary (e.g., reinforcing affirmative framing in Omission variants that were insufficiently distinct from Neutral). All samples were independently reviewed by a second Keeper; a third Keeper additionally verified a random 10% subset to confirm consistent ground truth assignment across all categories

4.6 Evaluation Setup↩︎

We evaluate 20 state-of-the-art LLMs spanning five model families, covering both standard instruction-tuned and reasoning-enhanced variants:

None

All models are evaluated using a unified system prompt  2 in a zero-shot setting. All generation parameters are left at their provider defaults. Model outputs are parsed to extract the binary adjudication decision and the predicted skill, with results reported as Failure Rate (FR) across rhetorical style categories and ground truth conditions.

5 Experiment↩︎

5.1 Main Results↩︎

Table 3 reports the Failure Rate (FR) of all 20 AI adjudicators across rhetorical styles. Overall, models prove substantially more vulnerable to Rhetorical Injection than anticipated: the average FP of 9.58% indicates that nearly one in ten mandatory roll scenarios is incorrectly resolved without mechanical verification, even under a zero-shot setting and an explicit adjudication prompt. A total of 35 samples were refused exclusively by Claude judge models (Claude Sonnet 4.5: 30, Opus 4.6: 3, Sonnet 4.6: 2), predominantly involving the Medicine skill; these are excluded from all metrics for the affected query–model pairs.

5.1.0.1 Scaling does not guarantee robustness.

Contrary to the intuition that larger or newer models are inherently more robust, we observe that no consistent monotonic improvement within families. GPT-5.4 (18.17%) underperforms GPT-5 (14.34%), and Claude Sonnet 4.5 (3.39%) outperforms both Sonnet 4.6 (4.80%) and Opus 4.6 (4.02%) on the overall FR. These reversals suggest that adjudication robustness is not straightforwardly governed by model scale or recency.

5.1.0.2 Reasoning models offer no consistent advantage.

Explicitly reasoning-enhanced models (\(\dagger\)) do not demonstrate superior robustness. The earlier reasoning model, DeepSeek-R1 (16.48%) is the worst-performing model in the DeepSeek family, and Qwen3-235B-Thinking (10.88%) trails Qwen3-Max (4.52%) by a substantial margin. This finding suggests that even models with enhanced reasoning capabilities do not exhibit consistent robustness against narrative sycophancy, indicating that chain-of-thought reasoning alone is insufficient for adjudication integrity.

5.1.0.3 Models are systematically biased toward leniency.

FC rates are near zero across all models (average 0.08%), confirming that misalignment is overwhelmingly directional: models fail by granting unearned success rather than by demanding unnecessary rolls.

5.1.0.4 Cross-family variation is substantial.

Claude consistently achieves the lowest FR across all four models (best: Sonnet 4.5 at 3.39%), while GPT-5.4 exhibits the highest single-model FR (18.17%). DeepSeek-V3.2 achieves the lowest Overall FR across all evaluated models (2.47%), a notable result for a non-flagship model. These cross-family differences may be partially explained by generator-target alignment effects discussed in the following section, though the underlying causes are unclear.

5.2 Impact of Rhetorical Style and Adversarial Source↩︎

5.2.0.1 Generator stylistic signatures.

The 5,376 generated samples reveals distinct stylistic fingerprints across the three generator families. Claude produces substantially longer adversarial statements (Authority: 52.7 tokens, Pseudo-Logic: 61.7 tokens) compared to GPT (23.3 and 26.0 tokens) and Gemini (27.9 and 32.5 tokens), suggesting a preference for elaborate rhetorical construction. This verbosity is accompanied by a markedly higher em-dash usage rate (up to 1.64% per 100 tokens under Authority), showing a tendency toward syntactically complex elaboration. We also applied TF-IDF to further analyze the qualitative differences in rhetorical strategy. We find that Claude’s adversarial vocabulary is dominated by quantifiers and pseudo-scientific adverbs, such as scientifically, biomechanically, measurable, consistent with a Pseudo-Logic framing that invokes numerical and technical authority. Gemini, by contrast, favors concrete environmental nouns (e.g., thermal, alkaline, structural), constructing attacks grounded in physical detail rather than abstract reasoning. GPT-generated attacks are lexically unremarkable, lacking a dominant rhetorical signature. This pattern is consistent with its substantially lower attack effectiveness observed in Figure 3.

5.2.0.2 Rhetorical style drives failure.

As shown in Figure 4, Pseudo-Logic consistently induces the highest average FR (17.30%), nearly double that of Authority (10.23%) and more than four times that of Neutral (3.82%). The effect is particularly severe in the GPT family, where the average Pseudo-Logic FR reaches 28.17%, with GPT-5.4 peaking at 43.14%, the highest single value observed across all evaluations. The Neutral-to-Pseudo-Logic delta reaches \(+\)​34.81% for GPT-5.4, indicating that superficially coherent causal framing alone is responsible for a dramatic degradation in adjudication accuracy. Omission, despite its intuitive appeal as a stealth strategy, produces the weakest adversarial effect (6.99%), suggesting that models implicitly infer risk from context even when the player downplays it. Even under Neutral conditions, models exhibit a non-trivial baseline FR of 3.82% (range: 0.09%–12.41%), reflecting irreducible adjudication noise in the absence of any rhetorical pressure.

5.2.0.3 Adversarial source dominates attack strength.

Figure 3 reveals a striking asymmetry across adversarial generators. Gemini-generated attacks consistently induce the highest FR across all target families (average 17.46%), while GPT-generated attacks are substantially weaker (average 2.19%), a gap that persists regardless of the target model. Nevertheless, stronger generators do not necessarily produce harder attacks: GPT-5.4, the most capable GPT generator, produces attacks that elicit lower FR than Claude-generated attacks on the same targets.

5.2.0.4 Home-series bias is asymmetric and family-dependent.

We observe a pronounced but directionally inconsistent home-series effect. GPT models are markedly more resistant to GPT-generated attacks (home FR: 3.97%) than to attacks from other families (away FR: 21.21%), suggesting that GPT models may share implicit rhetorical patterns that are less effective against their own family. In contrast, Gemini models show the opposite tendency, exhibiting higher FR against Gemini-generated attacks (14.89%) than against external ones (3.50%). Claude models show near-neutral bias (home: 4.84%, away: 6.01%). These findings suggest that home-series bias is not a universal phenomenon but varies substantially across families, likely reflecting differences in training data and stylistic conventions across model providers.

5.3 Impact of World Setting↩︎

Figure 5 reveals a striking and consistent pattern across all model families: adjudication difficulty varies dramatically by world setting, exposing a systematic dependence on cultural and temporal knowledge.

5.3.0.1 1920s Urban is universally trivial.

All 20 models achieve 0.0% FR on the 1920s Urban setting, suggesting that Lovecraftian-era scenarios pose no adjudication challenge to AI adjudicators. This performance implies that models can adjudicate correctly when the domain knowledge is familiar.

5.3.0.2 Ancient China is the most adversarial setting.

The Ancient China setting produces the highest average FR (11.88%), with GPT-5 reaching a peak of 45.83%—by far the largest single-cell value in the setting analysis. This suggests that adjudicators struggle significantly with period-specific physical constraints unique to pre-modern Chinese environments (e.g., silk scrolls, sedan chairs, courtyard architecture), likely reflecting gaps in culturally specific long-tail knowledge. Claude models are a notable exception, maintaining 0.0% FR across all non-Wilderness settings, indicating substantially stronger cross-cultural knowledge coverage.

5.3.0.3 Wilderness presents consistent moderate difficulty.

The Wilderness setting produces a uniformly non-zero FR across all models (mean 9.76%, range 3.26%–20.65%), making it the only setting where every model fails at least occasionally. Unlike Ancient China, where failures concentrate in specific models, Wilderness errors are broadly distributed, suggesting that natural environment adjudication poses a general challenge rather than a knowledge gap specific to any model family.

5.3.0.4 Setting difficulty reflects knowledge coverage, not rule complexity.

The pattern across settings suggests that adjudication failures are driven less by rule complexity but more by domain familiarity: Models struggle most in settings where the relevant physical and cultural context is likely limited in their training data. This implies that improving adjudication robustness in these settings may require targeted knowledge augmentation rather than purely reasoning-focused improvements.

6 Conclusion↩︎

We presented CoC-Seduce, an adversarial benchmark evaluating LLMs as autonomous adjudicators in semi-open textual game environments via a novel threat model we term Rhetorical Injection. Evaluating 20 frontier models across 5,376 samples, we find that current LLMs are substantially more vulnerable than anticipated: Pseudo-Logic is the dominant attack vector, reasoning-enhanced models offer no consistent robustness advantage, and adjudication failures are strongly mediated by cultural domain familiarity rather than rule complexity. Adversarial attack strength is further shown to be generator-dependent and asymmetric across families. These findings suggest that robust adjudication requires advances beyond instruction following and reasoning, toward stronger resistance to rhetorical manipulation and broader cross-cultural knowledge coverage. We hope CoC-Seduce can serve as a foundation for developing adjudicators that are not only rule-aware, but truly resistant to the persuasive power of language.

Limitations↩︎

Several limitations of this work warrant acknowledgment. First, our dataset focuses on 16 skills and deliberately excludes social skills, whose ground truth is inherently more subjective. While this design improves annotation reliability, it leaves open the question of how LLM adjudicators perform in rhetorically ambiguous social interactions. Second, even within physically grounded categories such as PHY and INV, some actions admit a degree of interpretive ambiguity during annotation and evaluation. For example, the distinction between a natural action narration and a narration that omits explicit reasoning can be subtle: “Even in the dim light, I line up the leap and jump from one stone shelf to the other” versus “I jump the gap before it gets any darker.” In such cases, human adjudication may still involve subjective judgment, particularly when the boundary between ordinary narration and implicit justification is blurred. Third, our adversarial samples are generated by three frontier models; other generators may produce qualitatively different attack patterns that are not captured by our benchmark. Finally, all evaluations are conducted in a zero-shot setting with default generation parameters, and results may differ under few-shot prompting or with fine-tuned adjudicators.

Ethical considerations↩︎

This work uses only publicly available, non-sensitive data and does not involve human subjects or personal information. We anticipate low risk of misuse and the computational footprint is modest. While the adversarial samples in CoC-Seduce could in principle be repurposed to train stronger rhetorical attacks, we note that all scenarios are grounded in fictional game mechanics and lack transferable real-world harm vectors. We used a generative AI tool to generate the teaser figure and language polishing (e.g. grammar and phrasing) without new scientific content; all technical claims, experiments and citations were written and verified by the authors. Further details of tool usage are disclosed in the Responsible NLP Checklist and Acknowledgments.

7 Appendix↩︎

7.1 Query Time of LLMs↩︎

Table 4 reports the response time statistics for all 20 adjudicators across the full evaluation corpus of 5,376 samples per model. Total wall-clock time varies substantially across families, ranging from 1.89 hours (GPT-4.1) to 33.94 hours (Qwen3.7-Max), with reasoning-enhanced models (\(\dagger\)) generally exhibiting higher mean latency due to extended chain-of-thought generation.

7.2 Dataset Examples↩︎

Table 5 illustrates the four rhetorical variants for two representative scenarios drawn from the benchmark. Each group shares an identical Scenario Truth but differs only in the rhetorical framing of the player statement, demonstrating how Pseudo-Logic and Authority inject persuasive elaboration onto an objectively unchanged action intent. Ground truth for both scenarios is \(V{=}1\) (dice roll required).

7.3 Detailed Breakdown Tables↩︎

Tables 6, 7, and 8 provide granular breakdowns of Failure Rate across three dimensions. Table 6 presents the full cross-tabulation of world setting and rhetorical style for all 20 adjudicators, enabling identification of specific setting–style combinations that drive the highest failure rates. Table 7 aggregates across rhetorical styles to isolate the effect of world setting per model, and Table 8 aggregates across settings to isolate the effect of rhetorical style per model.

7.4 Dataset Generation Prompts↩︎

The following prompts are used to generate the CoC-Seduce dataset. Prompt 6 generates mandatory roll scenarios (\(V{=}1\)) across all four skill categories; Prompt 7 generates automatic resolution scenarios (\(V{=}0\)) for PHY and INV only. Both share the same setting enforcement rules and output format, differing in scenario truth design criteria. Full prompt templates will be available in our repository.

None

None

References↩︎