1 Introduction↩︎

Large language models (LLMs) are increasingly embedded in automated pipelines where they generate outputs and then evaluate, filter, or revise them. Often, the model’s second pass is supposed to make the first pass safer or more reliable. A coding agent writes a patch and then reviews its own diff. In synthetic-data generation, a model generates candidate labels and then filters out the ones that appear invalid. A medical assistant explains whether a symptom requires urgent care and then checks whether its advice included the relevant safety warnings. More generally, in agentic pipelines, models play many roles. They produce outputs that rely on some underlying concepts, and then revise, evaluate or otherwise judge the output.

A model’s grasp of a concept ought to be consistent across roles. If not, i.e. if the model employs one understanding of a concept while generating an output and another while evaluating it, the outcome of the review step is hard to interpret. The model might flag a sound output against a criterion that played no role in generation, or accept a flawed output because it has stopped enforcing a constraint it applied earlier. This creates a potential source of drift inside self-revising systems—the model may return to the same artifact but no longer treat the underlying problem in the same way when it plays a different role. We therefore need a way to measure whether a model’s use of a concept remains stable when it plays different roles in an automated pipeline.

We introduce a suite of generator-evaluator tests designed to isolate this form of conceptual self-consistency. Each test is built so that the generator and evaluator outputs should carry a specific relation to each other. To be precise, in all our tests, a model is first asked to produce an output with a specified property, and then the same model is asked to judge whether its output has that property. This lets us measure consistency without needing an external correctness label for the generated artifact. We organize these tests into three families that distinguish qualitatively different ways that concept use can remain consistent or break down: (i) multiple-choice question (MCQ) perturbations, (ii) rationale-based transformations, and (iii) ontological checks. We apply the suite to concepts sourced from saturated general-reasoning benchmarks (MMLU, BBH) and deployment-focused benchmarks (MedicalQA, GDPVal, FinanceQA), and evaluate frontier models across providers. We collect examples of generator-evaluator response pairs, spanning \(10\) models, \(5\) source benchmarks, and \(491\) concepts. We find substantial variation in self-consistency that cannot be explained by performance on the source benchmark questions.

Note that, while standard Q&A model benchmarks test models’ knowledge of concepts, accuracy on a benchmark has little to do with the kind of consistency we aim to measure. A model may answer a benchmark question correctly while failing to reliably answer very closely related questions about the same concept. Take a simple illustration from our results. We ask a model to modify a multiple-choice question from a benchmark so that “none of the above” becomes the correct answer, and then ask the same model to solve the modified question. The model may answer the original question correctly, successfully produce the modified question, and then fail to select “none of the above” as the answer (see Figure 1 for an illustration). Nothing about the underlying concept has changed, only the framing of a question about the concept induced by the model itself. In other words, models that are effectively indistinguishable in benchmark accuracy can still diverge in whether they preserve a simple transformation of the task they themselves produced.

So, once we can measure self-consistency, how should we use these measures? One natural idea is to treat this measure as a model-selection criterion, especially in agentic pipelines where drift may be particularly harmful. If two models perform similarly on standard benchmarks, we may prefer to use the one that is more self-consistent. The case for doing so is straightforward. A more consistent model should be easier to use in automated pipelines because its second pass is more likely to be continuous with its first. It should be less likely to drift as it generates, checks, and revises its own outputs. When such a model flags an error on the second pass, the flag reflects a real error rather than an inconsistency in how it applied the concept.

This argument depends on an assumption that consistency does not come with a tradeoff in reliability. But whether consistency actually tracks reliability is an empirical question. So we test, empirically, whether this is the case. Using MedMistakes-Validated, a dataset of clinically realistic vignettes paired with physician-validated model errors, we construct concept-level measures of mistake vulnerability. For each (model, concept) pair, this measure captures how often the model reproduces failure modes that practicing clinicians flagged as genuine errors. The (model, concept) measure of mistake vulnerability is thus an expert-grounded view of reliability that goes beyond benchmark accuracy. And we can relate this measure of mistake vulnerability to self-consistency, also at the (model, concept) level by extracting concepts from the clinical mistakes.

We find a clear and systematic association between generator–evaluator self-consistency and mistake vulnerability, even after controlling for benchmark performance. Models that are more consistent in how they apply concepts across generation and evaluation are also the ones that reproduce more validated clinical failure modes. The pattern is not driven by a few pathological cases. It reflects a stable ordering across models: those that are high in self-consistency tend to be high in mistake vulnerability, and those that are low in self-consistency tend to be low in mistake vulnerability. Consistency does not translate into reliability in the setting we study. We call this the consistency dilemma.

The paper proceeds as follows. 2 formalizes generator–evaluator self-consistency, and 3 introduces a taxonomy of tests. 4 describes our automated pipeline for instantiating these tests. 5 reports suite-wide results. 6 presents a clinical case study in which the consistency dilemma arises. 7 reviews related work, 8 discusses limitations, and 9 concludes.

2 Framework↩︎

We propose a measure of self-consistency that captures whether a language model applies the same concept in a stable way across related prompts. To compose this measure, we compare the model’s outputs to each other under concept-linked tests. We define conceptual self-consistency as the rate at which a model’s generation and evaluation agree in their invocation of a concept. Specifically, we measure it by asking the model to generate some output that meets a criterion, and then call the model again to evaluate whether its earlier output meets the criterion under which it was generated.

Definition of a ‘concept’. In this work, a concept is a compact, reusable piece of knowledge that constrains what counts as a valid instance or application across many contexts. A concept can be, for example, a rule, category, or principle. In our setting, each concept is represented by a short label (e.g., myocardial infarction, Bayes’ rule, drug–drug interaction). A concept is not a one-off fact or a named entity. Nor is it a broad topic.

Test structure. Our measure of conceptual self-consistency is composed through an evaluation suite of paired tests. Each test uses the same model twice: once to generate an output that depends on a target concept, and once to validate that output under a related prompt that invokes the same concept. The validation step is a consistency check against the model’s own earlier use of the concept.

Formalization. Let \(f:\mathcal{X}\to\mathcal{R}\) be a language model that maps a text prompt to a text response. Let \(c \in \mathcal{C}\) be a concept and \(t \in \mathcal{T}(c)\) a test designed to probe use of \(c\). Each test instance \(i \in \mathcal{I}_{c,t}\) consists of paired prompts \((p_g^{(i)}, p_v^{(i)})\). The model generates \(r_g^{(i)} = f(p_g^{(i)})\) from the first prompt and \(r_v^{(i)} = f(p_v^{(i)})\) from the second. A deterministic scoring rule \(s_t:\mathcal{R} \times \mathcal{R} \to \{0,1\}\) assigns a binary agreement label, \[Z_{f,c,t,i} = s_t\!\left(r_g^{(i)}, r_v^{(i)}\right),\] where \(Z_{f,c,t,i}=1\) if the validation output affirms that the generated output satisfies the criteria specified by test \(t\), and \(0\) otherwise. We interpret \(Z_{f,c,t,i}=1\) as evidence that the model applies the concept consistently within test \(t\) on instance \(i\).

Our contribution is to design a suite of such tests, denoted \(\mathcal{T}(c) = \{t_1, \dots, t_k\}\), each probing a different way in which concept use can remain consistent or break down. Taken together, this suite captures multiple forms of self-consistency, allowing us to analyze patterns across tasks, concepts, and models.

Metrics. Each generator–evaluator test yields a binary outcome. Let \(Z_{f,c,t,i} \in \{0,1\}\) indicate whether model \(f\) is consistent on the \(i\)-th instance of test \(t\) for concept \(c\). Let \(n_{c,t}\) denote the number of instances available for the \((c,t)\) pair. We define the (raw) consistency rate as \(\widehat{C}(f,c,t) \;=\; \frac{1}{n_{c,t}} \sum_{i=1}^{n_{c,t}} Z_{f,c,t,i}.\)

Because chance agreement differs across tests (e.g., binary vs.multi-choice), we also report a chance-adjusted score. Let \(p_{\mathrm{ch}}(t)\) denote the chance-level agreement implied by the structure of test \(t\) (e.g., \(1/2\) for binary tests; \(1/K\) for selecting a specific option among \(K\) choices). For details on the computation of \(p_{\mathrm{ch}}(t)\) in our suite, see Appendix Section 15. We define \(\widehat{C}^{\mathrm{adj}}(f,c,t) \;=\; \frac{\widehat{C}(f,c,t)-p_{\mathrm{ch}}(t)}{1-p_{\mathrm{ch}}(t)}.\) Note that \(\widehat{C}^{\mathrm{adj}}\) can be negative, indicating worse-than-chance agreement.

We additionally aggregate by (i) averaging over concepts to obtain a test-level score \(\widehat{C}(f,t)\), (ii) averaging over models to obtain a concept-level score \(\widehat{C}(c)\), and (iii) averaging over test groups in our taxonomy to obtain group-level scores \(\widehat{C}(f,\mathcal{G})\) (and analogously for \(\widehat{C}^{\mathrm{adj}}\)).

Figure 1 illustrates the generator–evaluator structure shared by all tests in our suite, along with three representative consistency types. In each test, the same model first generates an output that relies on a target concept and then evaluates that output under a related prompt invoking the same concept. The right panel shows how inconsistency can arise in qualitatively different ways, even when the underlying concept and benchmark question are held fixed. These examples motivate our taxonomy of MCQ perturbation, rationale, and ontological consistency, which we formalize in the following section.

3 A Taxonomy of Conceptual Consistency↩︎

Our evaluation suite consists of automated tests designed to measure self-consistency in concept use. All tests share the same generator–evaluator structure (as defined in Section 2), but they probe different types of consistency. We group the tests into three categories: MCQ perturbation, rationale-based, and ontological.

MCQ-perturbation self-consistency. These tests probe behavior under small, controlled edits to a multiple-choice question (MCQ). The model first produces a perturbed MCQ and is then asked to solve it. For example, the model can first be asked to modify a multiple-choice question so that none of the original answer choices are correct and a new “None of the above” option becomes correct, and can then be asked to solve the modified question to assess whether it correctly selects the added option. Other work has shown models to be accurate on the original question yet behave inconsistently under controlled changes [2]–[4].

Rationale self-consistency. These tests measure whether a model can audit its own reasoning traces consistently across linked prompts. In the generator step, the model produces an answer together with a rationale under a specified condition (e.g., a fully correct explanation, an incomplete one, or an intentionally flawed one). In the evaluator step, the same model is asked to judge whether the answer–rationale pair satisfies the relevant criterion (e.g., whether the rationale is valid, complete, or actually supports the answer). This is operationally important in agentic revision and LLM-as-a-judge pipelines, where models are increasingly used as automated evaluators [5]: if a model produces reasoning that does not support its own conclusion (self-contradictory reasoning) and fails to flag it, the pipeline can systematically rubber-stamp flawed explanations [6].

Ontological self-consistency. Some tasks require models to maintain a coherent organization of concepts and their relationships. Concepts are defined in part by how they relate to categories, instances, properties, and constraints, and stable use depends on applying these relationships consistently across contexts. This mirrors findings that models can perform well on world-model diagnostics while producing failures on related but subtly different tasks [7]. Concretely, the ontological tests ask the model to generate examples, non-examples, attributes, or properties and then validate them under a related prompt.

Scope of the taxonomy. Our goal is not to enumerate every possible self-consistency check, but to capture a set of qualitatively distinct failure modes. Models can be highly self-consistent in one category and unstable in another. A complete table of tests and their descriptions appears in Appendix 10, and the full generator and evaluator prompts are provided in Appendix 11.

We now describe our automated evaluation suite design.

4 Evaluation Pipeline↩︎

Overview. We operationalize self-consistency using a fully automated, black-box pipeline. The basic unit of analysis is an evaluation instance: one model, one benchmark question, one extracted concept label, and one self-consistency test. In each instance, the same model plays both roles (generator and evaluator). Each instance yields a binary agreement label \(Z\) (Section 2), and we aggregate these labels across instances to compute self-consistency scores.

Benchmarks as concept sources. MCQ-perturbation and rationale tests require a benchmark question as context. We draw questions from two classes of benchmarks: saturated general-reasoning benchmarks (MMLU [2], BBH [3]) and mission-critical benchmarks (MedicalQA [8], GDPVal [9], FinanceQA [10]).

Within each benchmark, we first randomly sample a fixed set of question slots before any model-specific steps. These slots are shared across models, so each model is evaluated on the same underlying questions; if a later prerequisite fails for a given model (e.g., no usable concept label or MCQ format), we gate out that model–question pair and record it as non-scoring rather than replacing it with a new question.

Concept identification. Because our tests are concept-linked, the pipeline attempts to identify a single concept for each pre-sampled question. For each \((\text{model},\text{question})\) pair, we ask the model whether the question relies on an underlying concept and, if so, to provide a short concept label. If no single usable concept is identified, the corresponding instance is marked as excluded from scoring (a gate) rather than replaced by a new question. The concept label extracted by the model is then used across all self-consistency tests applied to that question. The full concept-extraction prompt is provided in Appendix 12.

Models. We evaluate ten frontier large language models: GPT-4o, GPT-5, GPT-5.1, and GPT-5.2 (OpenAI); Claude Opus 4.5 and Claude Sonnet 4.5 (Anthropic); Gemini 2.5 Pro and Gemini 3 Pro Preview (Google); DeepSeek Chat (DeepSeek); and Mistral Large (Mistral AI). This range of models allows us to assess how patterns of conceptual consistency vary by model.

Applying the consistency tests. Each evaluation instance applies one self-consistency test from the taxonomy (Section 3) and records a binary agreement label \(Z\) obtained by deterministic parsing of the evaluator’s response.

Validity conditions. Some tests are only meaningful when prerequisites are satisfied (e.g., multiple-choice structure, baseline benchmark correctness, or successful concept identification). We record these prerequisites as explicit scoring gates and exclude failing instances from self-consistency aggregates while retaining them in logs. Gate definitions and activation rates appear in Appendix 13, and a discussion of what these gates imply for our analyses appears in 8.

To further assess whether instruction-following failures could confound our consistency measurements, we conduct a targeted audit of instruction-following well-formedness using an automated verifier, supplemented by manual review. The results, reported in Appendix 14, indicate that malformed outputs are rare and do not drive the consistency patterns observed in the main analysis.

5 Suite-Level Patterns of Self-Consistency↩︎

In this section, we summarize patterns in self-consistency across models, benchmarks, and concepts. The dataset contains \(3{,}951\) evaluation instances; each instance applies one test \(t\) to one source benchmark question \(q\) for one model \(f\), yielding a binary generator–evaluator agreement outcome. In total, we evaluate \(10\) models on questions drawn from \(5\) source benchmarks; concept extraction yields \(491\) distinct concepts used to instantiate tests.

Generator–evaluator agreement. Table ¿tbl:tab:model95by95test95chance? reports generator–evaluator agreement for each model and test after correcting for test-specific chance baselines. Because tests differ in structure (e.g., binary judgments versus multiple-choice decisions), raw agreement values are not directly comparable across tests. We therefore normalize each test so that \(0\) corresponds to chance-level behavior and \(1\) to perfect agreement. Under this normalization, positive values indicate agreement above chance, values near zero indicate chance-level behavior, and negative values indicate performance worse than chance. Details of the chance baselines and normalization procedure appear in Appendix 15, and uncorrected agreement rates are reported in Appendix 17.

We see that self-consistency varies systematically across models and test types. For instance, Gemini 3 and GPT-4o achieve near-ceiling agreement on several ontological tests, while Claude Sonnet 4.5 performs substantially worse on closely related tasks.

Qualitative failure examples. Figure 2 shows an instance of an ontological consistency failure. The model generates an example–attribute pair describing the six-yard line at a soccer stadium and lists attributes such as being clearly marked, frequently observed, and strategically important. When asked to evaluate this output, the model rejects it, indicating that the generated example does not satisfy all listed attributes.

Robustness of generator–evaluator agreement. We assess the stability of generator–evaluator agreement under repeated evaluation. We sample \(N=200\) instances stratified by test type, and re-run each instance \(K=5\) times using identical prompts and decoding settings. For each instance, we compute the fraction of runs that agree with the modal judgment.

Agreement outcomes are highly stable (mean \(0.923\)). Stability is slightly higher when the modal outcome is agreement than when it is disagreement (difference in means \(=0.056\); 95% bootstrap CI \(= [0.003, 0.113]\)). Additional robustness analyses are reported in Appendix 18.

6 Self-consistency is associated with mistake vulnerability in deployment↩︎

Over the preceding sections, we measured self-consistency as a behavioral property of how models apply a concept across related prompts. We now ask what that property implies in a deployment-relevant stress test: given a fixed catalogue of clinician-validated, high-stakes failure modes, how often does a model reproduce the same failure when the clinical situation is presented in superficially different forms? Using MedMistakes-Validated [1], we quantify concept-level vulnerability to these known errors and relate it to self-consistency, conditional on benchmark medical knowledge.

Our outcome is mistake vulnerability: for each model–concept pair, we measure how often the model reproduces validated mistakes associated with that concept.

Quantities used in the medical analysis. We construct three quantities at the \((m,c)\) level (full details of the data construction are in Appendix 22 ):

Mistake vulnerability (MedMistakes). For each model and each validated mistake, the dataset provides a set of clinician-provided tags and a binary indicator for whether each model reproduces the mistake. We treat tags as candidate medical concepts and focus on a curated concept set (Appendix 19). To avoid double-counting mistakes with multiple tags, we construct a weighted concept-level mistake vulnerability outcome \(Y^{\ast}_{m,c}\in[0,1]\) that splits each mistake evenly across its tags, so each underlying mistake contributes total weight of one across concepts (Appendix 23.1). That is, the weighting prevents a multi-concept mistake from being counted multiple times in a concept-level outcome.

Benchmark performance (MedQA). To control for baseline medical knowledge, we measure benchmark performance using MedQA [11]. For each \((m,c)\), \(A_{m,c}\in[0,1]\) is the model’s accuracy on MedQA questions associated with concept \(c\). We find that the benchmark performance is high: \(A_{m,c}\) averages 0.90 (SE = 0.005).

Self-consistency. We compute \(C_{m,c}\) as the average chance-corrected generator–evaluator agreement across the \(12\) concept-linked tests, normalizing each test so that 0 corresponds to chance-level agreement and 1 to perfect agreement (Appendix 15). Appendix 16 additionally reports the distribution of self-consistency scores across all \((m,c)\) pairs and separately by test.

Statistical specification. We construct a dataset with one row per \((m,c)\) pair. Let \(Y^{\ast}_{m,c}\) denote mistake vulnerability (MedMistakes), \(A_{m,c}\) denote benchmark performance (MedQA), and \(C_{m,c}\) denote self-consistency. The outcome \(Y^{\ast}_{m,c}\) lies in \([0,1]\), while \(A_{m,c}\in[0,1]\) and \(C_{m,c}\in[-1,1]\).

Because \(Y^{\ast}_{m,c}\) is fractional, we estimate a weighted fractional logit model [12]: \[\mathbb{E}\!\left[Y^{\ast}_{m,c}\mid A_{m,c}, C_{m,c}\right] \;=\; \mathrm{logit}^{-1}\!\left(\beta_0 + \beta_1 A_{m,c} + \beta_2 C_{m,c}\right), \label{eq:coh95flogit}\tag{1}\] where each \((m,c)\) observation is weighted by the number of mistakes that contribute to \(Y^*_{m,c}\), with standard errors clustered at the concept level. We weight observations by the effective number of validated mistakes contributing to each \((m,c)\) outcome (the sum of tag-splitting weights), so concept pairs supported by more mistakes receive more weight.

Our hypothesis of interest is whether self-consistency is associated with mistake vulnerability after controlling for medical benchmark performance i.e. whether \(\beta_2 \neq 0.\)

Results. Across \(N=440\) \((m,c)\) observations, chance-corrected self-consistency is positively associated with mistake vulnerability, conditional on benchmark performance. In our deployment-weighted fractional logit, the estimated coefficient is \(\hat{\beta}_2 = 1.116\). With standard errors clustered on concept, this association is highly significant (\(\mathrm{SE} = 0.224\), \(p < 10^{-6}\)). Because the variation in self-consistency that identifies \(\hat{\beta}_2\) runs primarily across models, we also report two-way clustered standard errors on concept and model [13]—the association remains significant at the 5% level (\(\mathrm{SE} = 0.533\), \(p = 0.036\)). Intuitively, benchmark performance is negatively associated with mistake vulnerability (\(\hat{\beta}_1 = -1.732\), \(p = 0.042\) under two-way clustering).

The relationship between consistency and mistake vulnerability is one that holds between models, not within a single model. In other words, we find that higher self-consistency tracks higher vulnerability, yet within a given model it does not predict which concepts are most vulnerable. When we add model fixed effects to the regression specification above, the coefficient on self-consistency falls to \(\hat{\beta}_2 = 0.147\) (\(\mathrm{SE} = 0.334\), \(p = 0.66\)). That is, once we condition on the model, how self-consistent it is on a given concept does not predict where it is more vulnerable to mistakes. We therefore read self-consistency as a model-level behavioral signature that is informative for comparing and selecting models. Full implementation details and regression results, including the fixed-effects specification, are in Appendix Sections 23 and 24.

What kind of consistency matters in diagnostic scenarios? We also decompose the aggregate consistency score to see which test components account for the consistency–vulnerability association. Recomputing chance-corrected consistency while leaving out each test yields a stable relationship in nearly all cases, with one clear exception: removing Correct for Incomplete Reason (CFIR) substantially attenuates the effect. CFIR exhibits relatively large cross-model variation (see ¿tbl:tab:model95by95test95chance?), so it has more leverage to explain differences. Full results are reported in Appendix 25.

To further probe the consistency–vulnerability link, we zoom in on a single concept: triage. We compare how models behave on our self-consistency suite with the kinds of clinician-validated failures MedMistakes associates with triage. In particular, we focus on Correct for Incomplete Reason (CFIR), which tests whether a model will generate and then endorse an answer supported by under-specified reasoning. This mirrors a common triage failure in MedMistakes: models point in the right general direction but omit critical escalation criteria, safety warnings, or follow-up steps. See Appendix 26 for the full output of one Correct for Incomplete Reason (CFIR) test and the corresponding MedMistakes failure for the concept triage.

Model-level consistency profiles. At the model level, is consistency stable across mistake-vulnerability levels? To answer this, we stratify \((m,c)\) pairs into quartiles of mistake vulnerability \(Y^*_{m,c}\) using weighted ranks (weights given by the effective mistake mass behind each cell). Within each quartile, we compute each model’s weighted mean self-consistency. Figure 3 shows that the relative ordering of models is largely preserved across vulnerability quartiles: models that are more self-consistent in low-vulnerability regions remain more self-consistent in high-vulnerability regions. This stability suggests that self-consistency behaves like a model-level behavioral signature.

Consistency–vulnerability association. Figure 4 provides a high-level visualization of heterogeneity across models in the relationship between self-consistency and mistake vulnerability. Each point corresponds to a model, positioned by its weighted mean chance-corrected self-consistency and weighted mean mistake vulnerability across concepts using mistake-mass weights. The figure shows that models differ substantially along both dimensions: some models exhibit relatively low self-consistency and low vulnerability to validated mistakes, while others are both highly self-consistent and highly vulnerable. The interquartile bars around each point show the spread across concepts within each model, illustrating that within-model variation is large relative to the differences between models. We overlay the fractional logit relationship evaluated at mean benchmark accuracy as a solid line on these axes for reference (note that the solid line in Figure 4 is not a regression fit to the model-level points shown).

7 Related Work↩︎

We study generator–evaluator consistency: whether a model’s outputs remain mutually consistent across related prompts in the absence of a correctness oracle.

Our work is closely related to recent studies of LLM-as-a-judge evaluation, especially work documenting self-preference and correlated reasoning failures across models. Prior studies show that LLM evaluators systematically favor their own generations or stylistic patterns [14], [15], while others demonstrate that frontier models often make highly correlated mistakes and converge on similar erroneous reasoning processes [16]–[18]. These findings raise concerns that agreement among models or between generation and evaluation stages may reflect shared biases rather than independent verification. Our work differs in a few key ways. First, unlike prior approaches that rely on cross-model evaluation or pairwise model comparison, we study consistency entirely within a single model’s generator–evaluator loop. Second, rather than measuring agreement at the level of full outputs or preferences, we operate at the level of reusable concepts and test whether the same underlying rule or principle is applied consistently across related contexts. Third, we connect these consistency patterns to downstream deployment failures in a clinician-validated medical setting. See Appendix Section 27 for a tabular comparison of these works.

Automated consistency checks like ours are one approach to the broader “oracle problem” in software testing and machine learning, where ground-truth labels are unavailable or prohibitively expensive to obtain [19]. One class of such checks is metamorphic testing, which replaces correctness labels with expected relationships between outputs under controlled input transformations; it has been adapted to LLMs and NLP pipelines to surface inconsistencies without labeled data [20], [21].

Many types of metamorphic relations have been studied in language systems. [22] and [23] study relational inversion and negation, [24] and [25] look at lexical perturbations, [26], [27], [28] and [29] consider reordering and structure-preserving transformations, while [30] and [31] study automated detection of meaning-altering edits (e.g., tense, category, time). Within LLM evaluation, related work generation-and-checking through automated setups [32]–[34]. Researchers have also critiqued the growing reliance on LLM-as-a-judge pipelines, calling attention to intra-rater instability and sensitivity to evaluation context [5], [35].

Another line of work treats various forms of consistency across samples from a model as signals of reliability. [36], for instance, finds that self-consistency decoding improves performance. SelfCheckGPT applies the same intuition to hallucination detection—if the model knows a fact then repeated samples from that model should agree on the fact, and any disagreement between samples therefore picks out likely hallucinations [37]. A family of related agentic methods employs revision loops built around the premise that a model can evaluate and refine its own outputs in a stable and consistent way across steps (e.g. self-refinement and reflection-style agents [38], [39], retrieval-augmented self-critique [40], and revision with attribution [41]. Evidence on when such self-correction works is mixed [42].

Two related lines of work go beyond a model’s final answer. One probes internal representations to assess whether the model’s hidden states support consistent answers [43]. The other examines the model’s reasoning traces, either documenting self-contradictions within a chain of thought [6] or developing broader criteria for evaluating step-by-step explanations [44].

While prior work measures robustness, generation–checking agreement, and judge reliability, it largely does not (a) measure generator–evaluator consistency at the level of reusable concepts, (b) organize such tests into a unified taxonomy of consistency types, or (c) analyze their predictive value for validated deployment failures. We address these gaps with an automated, concept-centric generator–evaluator suite and a clinician-validated medical case study linking self-consistency to mistake vulnerability.

8 Limitations↩︎

This study of generator-evaluator agreement—and its relationship to reliability—has several limitations which open the door to further study.

First, there are limitations related to how the automated suite is constructed and scored. Because tests are only counted when the prerequisite gates are passed, the reported self-consistency rates are best read as consistency conditional on scoring eligibility. Conditioning on scoring eligibility can systematically drop harder or differently structured items across models—so the ultimate set of concepts tested does not reflect the full benchmark question distribution. The suite also relies on model-generated concept extraction that forces a single short label per (model, question), and that can add measurement error whenever questions naturally involve multiple interacting concepts or when different models carve the same question into different concept names, reducing cross-model comparability at the “concept” level. Additionally, more work is needed to better disentangle endorsement bias (i.e. the tendency to respond affirmatively) from the type of self-consistency we aim to measure.

A second set of limitations relates to the MedMistakes deployment analysis. We emphasize that the deployment analysis is observational (not causal). Furthermore, it uses a curated set of clinician-validated mistakes and concept tags with uneven coverage across model–concept cells, and therefore should be interpreted as vulnerability to this particular validated mistake set not as an estimate of overall deployment error. Even though we partially mitigate uneven coverage in two ways—(i) splitting each multi-tag mistake across its tags so each underlying mistake contributes total mass 1 overall, and (ii) weighting each model–concept observation in the regression by its supporting mistake mass—results may still be sensitive to how mistakes are tagged and which mistakes are included (see [1] for a more detailed description of their pipeline and the key selection, tagging, and coverage biases it could introduce).

9 Conclusion↩︎

The reliability of large language models (LLMs) depends on how consistently a model invokes a concept in related contexts. This paper constructs a measure of self-consistency and studies it in a deployment setting. Our results highlight a tension. Self-consistency is often operationally desirable: it reduces drift, enables agentic workflows, and supports pipelines such as LLM-as-a-judge or critique-and-revision. But, despite its benefits, models that are more consistent also tend to be more vulnerable to known mistakes. We call this phenomenon the consistency dilemma in LLMs.

Our evaluation suite also suggests concrete deployment uses. Because the metric is automated and does not require correctness labels, practitioners can compute self-consistency scores and use them as diagnostics before deployment. Practitioners can use these scores as pre-deployment diagnostics to flag models that may warrant human oversight or external evaluation in place of autonomous self-critique. More broadly, our results suggest that consistency measurements can serve as scalable workflow-level diagnostics for agentic systems, complementing traditional benchmark evaluations focused on single-turn correctness.

More broadly, our study suggests a fruitful methodological direction for future work. One direction is identifying the conditions under which self-consistency becomes predictive of mistake vulnerability at the concept level — that is, within a single model. Moreover, isolated benchmark-style scores can be systematically linked to deployment-relevant datasets with structured failure annotations, so that properties measured on controlled tasks can be related to downstream risks. Through these benchmark-deployment linkages, evaluations can increasingly target workflow behavior—how models act across generation, critique, and judgment—rather than single-turn task performance.

Impact Statement↩︎

We introduce an automated, black-box framework for measuring conceptual self-consistency in large language models (LLMs)—the extent to which models apply the same concept consistently across related prompts. Our generator–evaluator tests and accompanying taxonomy surface structured patterns of agreement and inconsistency that standard accuracy-based benchmarks miss. In a medical case study, we show that higher self-consistency is associated with greater vulnerability to clinician-validated mistakes, even after controlling for benchmark performance. As LLMs are increasingly deployed in high-stakes settings where ground-truth verification is costly or infeasible, our framework provides practical infrastructure for diagnosing reliability risks driven by repeated, not isolated, mistakes, complementing traditional benchmark evaluation.

Software and Data↩︎

Code and data are available at https://github.com/MarinaMancoridis/ConsistencyDilemma. The repository includes code to reproduce all main text results, tables and figures, along with replication instructions.

Acknowledgments↩︎

We thank Sarah Bentley, Peter Chang, Nathan Jo, Sendhil Mullainathan and Adam Kalai for helpful discussions, feedback, and comments on earlier drafts of this work. We are also grateful to additional collaborators and colleagues who provided suggestions and insights throughout the project. This work was supported in part by funding from OpenAI.

10 Full Task Descriptions↩︎

Table ¿tbl:tab:tasks? provides a complete overview of the consistency tests included in our evaluation suite. Each row describes a distinct automated test, grouped by consistency category, along with a brief description of the test logic and a representative example.

11 Prompts for Consistency Tests↩︎

This section documents the prompts used to instantiate each consistency test in our automated evaluation suite and examples of failures elicited by the test. For each test type, we describe the test at a high level and provide the corresponding generator and evaluator prompts used in our experiments.

11.1 MCQ Perturbation: None of the Above↩︎

This test checks whether a model remains consistent when a benchmark multiple-choice question is modified so that none of the original options are correct and a new “None of the above” option becomes the correct answer. Consistency is assessed by whether the model selects the added “None of the above” option.

11.2 MCQ Perturbation: Previous Answer↩︎

This test checks whether a model remains consistent when the original question is minimally rewritten so that the correct option becomes the option immediately preceding the original correct answer. Consistency is assessed by whether the model selects the intended new correct option.

11.3 Rationale: Right for Right Reason↩︎

This test checks whether the model can identify cases where both an answer and its justification are correct. The generator produces a correct answer paired with correct reasoning, and the evaluator judges whether the both are correct.

11.4 Rationale: Right for Wrong Reason↩︎

This test checks whether the model can identify cases where an answer is correct but the provided justification is incorrect. The generator produces a correct answer paired with intentionally flawed reasoning, and the evaluator judges whether the reasoning is incorrect despite the correct answer.

11.5 Rationale: Wrong for Wrong Reason↩︎

This test checks whether the model can identify cases where both the answer and the justification are incorrect. The generator produces an incorrect answer paired with confident but flawed reasoning, and the evaluator judges whether both are wrong.

11.6 Rationale: Correct for Incomplete Reason↩︎

This test checks whether the model can identify cases where an answer is correct but the justification is missing key information needed to support the conclusion. The generator produces a correct answer paired with incomplete reasoning, and the evaluator judges whether the reasoning is insufficient. For example, in the sample response, stating that 11 is not divisible by 2 or 3 is not enough to fully justify that it is prime; additional checks (such as ruling out divisibility by 5) would also be needed to conclusively eliminate the other options.

11.7 Ontological: Valid Example↩︎

This test probes whether the model can generate a valid (non-canonical) instance of a concept, and whether such generations can be reliably recognized as valid examples.

11.8 Ontological: Invalid Example↩︎

This test probes whether the model can generate a clear non-example of a concept, and whether such generations can be reliably recognized as invalid examples.

11.9 Ontological: List True Attributes↩︎

This test probes whether the model can propose an example of a concept together with attributes that genuinely hold for that example, and whether the resulting (concept, example, attributes) bundle is internally consistent.

11.10 Ontological: List Wrong Attributes↩︎

This test probes whether the model can detect when an otherwise valid example is paired with at least one attribute that does not hold for that example.

11.11 Ontological: List True Examples↩︎

This test probes whether the model can generate a list of instances of a concept (an "is-a” list) and whether those items are valid instances.

11.12 Ontological: Has Property↩︎

This test probes whether the model can propose properties that reliably apply to a concept, and whether those proposed properties are indeed plausibly characteristic of the concept.

12 Concept Extraction Details↩︎

After sampling a question from a benchmark, we prompt the model to determine whether answering the question requires understanding an underlying concept and, if so, to identify that concept explicitly.

A concept is a construct that represents a general idea or category of something. It can be thought of as a compressible rule/definition system that allows you to generate or classify many specific instances from a small number of underlying principles. Concepts are generative categories: once you know the rules or definition, you can recognize or produce valid examples across many situations. Good concepts are transferable, often taught with a name alone, and clarify reasoning by compressing complexity into simple, repeatable logic. 
    
    Some concepts are domain-specific (ie. medical, economic, or financial concepts) and still count (e.g., seizures, gene flow, opportunity cost, supply and demand). Concepts can be formal (e.g., Bayes' rule, Pareto optimality), procedural (e.g., net present value), or categorical (e.g., haiku). Examples of concepts in the medical domain include specific disease names, medications, procedures, diagnostic tests, and more (e.g., "muscular atrophy", "vaccines", "longevity", "COVID-19").

    Not all terms or ideas qualify as a single, well-defined concept in this sense. Some may be real or useful but do not correspond to a specific principle or criterion that must be understood to answer the question; for example, purely factual recall, opinions, broad research areas, historical events, specific people, individual works such as books or movies, or complex mechanisms that resist clean summarization. Examples of non-concepts: GDP trends, the best movie of 2014, the Industrial Revolution. Never give 'TBD' or similar as a concept.

    Does the following question rely on understanding a concept? In the first line of your response, write "ANSWER:" followed by either "Yes" or "No" and nothing else. On the next line, write "CONCEPT:", followed by the name of the single concept if there is one. The concept name must be no more than three words long. Include no other text.

We then extract the binary concept indicator and the concept label directly from the model’s structured response, using the presence of a "Yes” answer and the associated concept name when applicable.

13 Prerequisite Gates and Scoring Eligibility↩︎

13.1 How gates affect what is scored↩︎

What does it mean to “fail a gate”? Our evaluation produces a binary outcome for each executed test instance, but not every (benchmark question, test) combination is well-posed. We therefore apply automated prerequisite checks (“gates”) that determine whether a test execution is eligible for scoring. If an instance fails any scoring gate, it is screened out from aggregate consistency statistics: we still run (or partially run) the pipeline and log the intermediate artifacts, but we do not count the instance when computing reported consistency rates.

Why this screening is necessary. Gates ensure that reported scores reflect meaningful comparisons rather than artifacts of ill-defined inputs. For example, the NOTA manipulation requires that adding “None of the above” is a reasonable choice addition. More broadly, we also restrict attention to cases where the model successfully identifies a single underlying concept and (for multiple-choice benchmarks) demonstrates baseline competence on the original item.

When gates are evaluated. Gates are evaluated in a fixed order aligned with the pipeline: (i) concept identification and (for multiple-choice benchmarks) baseline benchmark grading are performed once per model–question pair and reused across all tests; and (ii) a small number of attempt-level gates (e.g., rationale refusal) are evaluated during test execution.

13.2 Prerequisite gates↩︎

13.3 Examples of prerequisite gate activations↩︎

Here, we provide representative examples illustrating how different prerequisite gates are activated in practice. Examples are intended to clarify the interpretation of gate activations rather than to assess model performance.

No concept identified.

• Benchmark: MMLU
  Model: Claude Sonnet 4.5
  Question excerpt: “What Native American tribe did chief Crazy Horse lead?”
  Answer choices: (A) Apache (B) Comanche (C) Sioux (D) Iroquois
  Rationale: This item primarily tests recall of a specific historical fact (mapping a named individual to a named entity), rather than application of a reusable underlying concept that can be invoked across multiple contexts. Because the question does not naturally center on a single conceptual principle (e.g., a general rule, mechanism, or abstraction), the concept-identification step does not yield a stable concept suitable for reuse across our consistency tests, and the instance is excluded from scoring.

Baseline benchmark incorrect.

• Benchmark: BBH (Web of Lies)
  Model: Mistral Large
  Concept: Truth-teller puzzle
  Question: “Shaunda lies. Fletcher says Shaunda lies. Jaymie says Fletcher lies. Millie says Jaymie lies. Gwenn says Millie tells the truth. Does Gwenn tell the truth?”
  Model answer: “No” (after a multi-step logical analysis)
  Correct answer: “Yes”
  Explanation: The model arrives at an incorrect final judgment.

NOTA infeasible.

• Test: None of the Above
  Benchmark: BBH (Logical Deduction: Three Objects)
  Model: DeepSeek Chat
  Question excerpt: “On a shelf, there are three books: a purple book, a black book, and a blue book. The purple book is to the right of the blue book. The black book is the second from the left.”
  Answer choices: (A) The purple book is the leftmost (B) The black book is the leftmost (C) The blue book is the leftmost
  Correct answer: (C)
  Model judgment: “Exhausts all logical possibilities.”
  Explanation: For this item, the provided options already enumerate all possible candidates for the leftmost position among the three objects. Adding “None of the above” would therefore be logically redundant (it cannot be correct if the premises are consistent), and the instance is excluded from scoring under the nota_not_feasible gate.

Rationale refusal.

• Test: Right for wrong reason
  Benchmark: MedicalQA
  Model: GPT-5.2
  Concept: Exercise guidelines
  Question excerpt: “I’m 56 years old. How often should I work out, and what type of workouts should I be doing?”
  Rationale prompt excerpt: “Answer the question below correctly, but give an explanation that is wrong. The reasoning should be wrong enough to truly be incorrect. Do NOT hint, admit, or suggest that the explanation is wrong…”
  Model response: “I can’t comply with your request to provide an intentionally wrong explanation.”
  Explanation: The test explicitly requires the model to produce an intentionally incorrect rationale while maintaining a correct final answer. In this instance, the model refuses to generate such content and explicitly states non-compliance, triggering the rationale_refusal gate and excluding the instance from scoring.

13.4 Logging and reproducibility↩︎

What gets logged. For each model and sampled benchmark question, we log a benchmark-level record containing (i) the extracted concept when available, (ii) (for MC benchmarks) whether the model answered the original benchmark item correctly, and (iii) any prerequisite failure reasons discovered at this stage. For each executed test attempt, we log the generator output, the evaluator prompt and output, the final binary judgment (when available), and a boolean indicator of whether the attempt is included in scoring.

Attempt-level runtime failures. In addition to the explicit gates above, individual attempts can fail due to missing model responses (e.g., generation/evaluation returns no output). Such attempts are logged as failures at the corresponding pipeline stage and are not included in scoring for that attempt. These failures are tracked separately from the conceptual/structural gates in Table 1.

13.5 Gate activation statistics↩︎

Table 2 reports how often each scoring gate is activated among the evaluation instances that reach the corresponding stage of the evaluation pipeline. Activation rates are computed relative to the set of instances that are eligible to be scored at that stage (e.g., after passing earlier prerequisites such as baseline benchmark correctness). Because multiple gates can apply to the same instance, activation rates are not mutually exclusive and may sum to more than 100%.

14 Instruction-Following Well-Formedness Audit↩︎

14.0.0.1 Motivation.

Several tests in our evaluation suite rely on model-generated outputs that are intended to follow explicit natural-language instructions (e.g., rewriting a question under specific constraints, or producing a rationale in a prescribed format). Failures to follow these instructions can confound downstream consistency judgments by introducing malformed inputs (e.g., altered answer choices or missing required structure). To isolate this failure mode, we conduct a targeted audit of instruction-following well-formedness, independent of semantic correctness or task success.

14.0.0.2 Automated instruction-following verifier.

We implement an automated verifier that evaluates whether a model’s output adheres to the formatting and structural constraints specified in the original generation prompt. Concretely, for each sampled instance, the verifier is provided with: (i) the original benchmark question (for context), (ii) the prompt used to generate the new output, and (iii) the model-produced output.

The verifier is explicitly instructed not to judge semantic correctness, factual accuracy, or whether the task objective was achieved. Instead, it evaluates only whether the output satisfies the stated instructions as written. We use GPT-5-Pro as our verifier model.

14.0.0.3 Verifier prompt (excerpt).

The verifier prompt emphasizes strict instruction-following and conservative judgments. An excerpt is shown below (line breaks added for readability):

Your task: Decide whether the output followed the instructions format-wise and is well-formed. Do NOT judge whether the content is correct or whether the task was successfully completed. ONLY check compliance with the explicit constraints in the prompt.

If the prompt says “do not change or reorder the answer choices” and the output adds “E) None of the above”, that is a violation even if the resulting question is reasonable.

Return strict JSON with fields: well_formed, violations, format_score, notes. If uncertain, err on the side of marking the output as not well-formed.

This conservative design intentionally biases the verifier toward false negatives when constraints are underspecified or subjective.

14.0.0.4 Automated audit results.

We randomly sampled 200 instances across test families and ran the automated instruction-following verifier. Of these, 163/200 (\(\approx\)​82%) were marked as well-formed by the verifier, while 37/200 were flagged as not well-formed.

14.0.0.5 Manual audit of verifier disagreements.

To better understand the nature of verifier failures, we performed a manual audit of an equal number of verifier-flagged failures and verifier-approved successes (n=\(20\)). Each instance was independently reviewed by the authors and categorized into one of the following mutually exclusive categories:

• Clear instruction compliance: The output unambiguously satisfies all explicit instructions in the prompt.

• Clear instruction violation: The output unambiguously violates at least one explicit instruction in the prompt.

• Borderline / subjective judgment: The output satisfies explicit constraints, but compliance depends on subjective or underspecified criteria (e.g., “keep it concise”).

14.0.0.6 Manual audit results.

In the audited sample, all verifier disagreements corresponded to conservative verifier judgments rather than genuine instruction violations.

Table 3 summarizes the results of this manual audit.

14.0.0.7 Interpretation and takeaway.

The manual audit shows that verifier-flagged failures predominantly reflect conservative judgments about stylistic or underspecified constraints (e.g., conciseness or formatting), rather than clear instruction violations. In the audited sample, we find no instances of genuinely malformed outputs that would plausibly compromise downstream consistency evaluations. As a result, instruction-following failures are unlikely to be a dominant driver of the effects reported in the main paper, and the automated well-formedness rate should be interpreted as a lower bound. Overall, this audit increases confidence that the observed consistency patterns reflect substantive model behavior rather than artifacts of malformed inputs.

15 Chance correction for generator–evaluator agreement↩︎

Why chance correction is necessary. The tests in our suite differ in their baseline agreement under uninformed or random behavior. In particular, the MCQ-perturbation tests have non-trivial chance agreement rates due to their discrete option structure, while the rationale and ontological tests are binary. Raw generator–evaluator agreement rates therefore conflate behavioral consistency with test format, making values not directly comparable across tests.

Chance baselines. Table 4 summarizes the chance baseline \(p_0(t)\) for each test \(t\) used to normalize agreement. For binary evaluator judgments, \(p_0(t)=\tfrac{1}{2}\). For the MCQ-perturbation tests, \(p_0(t)\) reflects uniform guessing among the available answer choices (e.g., \(\tfrac{1}{5}\) for the “None of the Above” manipulation with five options).

5pt

Normalization. Let \(\hat{p}(t)\) denote the observed generator–evaluator agreement rate on test \(t\). We report chance-corrected agreement \[\hat{p}^{\ast}(t) \;=\; \frac{\hat{p}(t) - p_0(t)}{1 - p_0(t)}.\] This linear rescaling maps chance-level behavior to \(0\) and perfect agreement to \(1\). Values near \(0\) indicate behavior indistinguishable from chance given the test structure, positive values indicate agreement above chance, and negative values indicate agreement worse than chance (systematic disagreement).

We use these chance-corrected values in Table ¿tbl:tab:model95by95test95chance?. For completeness, the corresponding uncorrected (raw) agreement rates are reported in Appendix 17.

16 Distribution of Self-Consistency Scores in MedMistakes Sample↩︎

Table 5 summarizes the distribution of self-consistency scores used in the regression analysis. The top row is an aggregation across all (model, concept) pairs across all tests. The following rows provide a breakdown by test type.

4pt

17 Uncorrected generator–evaluator agreement table↩︎

For completeness, Table 6 reports the same generator–evaluator agreement rates as Table ¿tbl:tab:model95by95test95chance?, but without chance correction (i.e., raw agreement \(\hat{p}\)). Because tests differ in their chance baselines (Appendix 15), these raw values are not intended for cross-test comparison; we include them primarily for transparency and ease of reference.

18 Robustness and reliability of the consistency metric↩︎

This appendix reports detailed robustness analyses for the generator–evaluator consistency metric. We re-run a fixed subset of \(N=200\) evaluation instances, stratified across test types, repeating each instance \(K=5\) times with identical prompts and decoding parameters. For each instance, we compute the fraction of runs that agree with the modal generator–evaluator judgment, yielding an instance-level stability score.

Overall stability. Across all instances, mean stability is \(0.923\) (\(n=200\)), confirming that generator–evaluator agreement is highly repeatable. As shown in Figure 29, most instances exhibit near-ceiling stability, with agreement in at least \(4/5\) runs.

Stability by test. Stability varies across test types but remains high overall. The least stable tests are Previous Answer (mean \(0.853\), \(n=15\)) and Correct for Incomplete Reason (mean \(0.859\), \(n=17\)). In contrast, several tests exhibit near-perfect repeatability, including Right for Right Reason (mean \(1.000\), \(n=15\)), Valid Example (mean \(0.988\), \(n=16\)), and List Wrong Attributes (mean \(0.957\), \(n=14\)).

Stability by model. All evaluated models exhibit high repeatability, though stability varies systematically across models. The lowest-stability models include Mistral Large (mean \(0.825\), \(n=16\)) and GPT-5 (mean \(0.900\), \(n=14\)), while the highest-stability models include Claude Opus 4.5 (mean \(0.960\), \(n=15\)), Gemini 3 Pro Preview (mean \(0.952\), \(n=25\)), and GPT-5.2 (mean \(0.947\), \(n=19\)). These differences further suggest that consistency reflects model-specific behavioral properties.

Stability conditional on modal judgment. We observe an asymmetry in stability depending on whether the modal generator–evaluator outcome is agreement or disagreement. Agreement outcomes account for \(83.0\%\) of instances (\(166/200\)) and exhibit higher stability (mean \(0.933\), median \(1.000\)) than disagreement outcomes, which account for \(17.0\%\) of instances (\(34/200\)) and have lower stability (mean \(0.876\), median \(1.000\)). Although both groups can reach perfect agreement, disagreement outcomes exhibit greater variability, consistent with rejection judgments being more fragile.

Overall, these robustness analyses support interpreting generator–evaluator agreement as a stable behavioral signal rather than an artifact of stochastic generation.

19 Concept Selection from MedMistakes↩︎

This appendix describes the procedure used to construct the fixed set of medical concepts employed throughout the analyses.

Candidate tag pool. MedMistakes-Validated provides clinician-authored mistake tags for each item, intended to capture the clinical concept or failure implicated by the error [1]. We collect all unique tags appearing in the dataset and compute their frequencies (number of MedMistakes items associated with each tag).

Frequency filtering. To ensure sufficient coverage for concept-level analysis, we restrict attention to the 150 most frequent tags. This step removes extremely rare tags that would yield unstable estimates of deployment accuracy for a given concept.

Manual concept curation. From this frequency-filtered pool, we manually select 49 tags that correspond to concrete, reusable medical concepts (e.g., seizure, hypoglycemia, pulmonary embolism). We manually exclude tags that are overly diffuse, meta-level, or underspecified (e.g., medical knowledge, documentation issue), as well as tags that conflate multiple distinct mechanisms. This curation step is intended to ensure that each concept admits meaningful reuse across benchmarks and consistency tests.

Final concept set. Table 7 lists the final set of medical concepts used throughout our analyses. These concepts span symptoms, diagnoses, medications, risk factors, and care processes.

20 MedMistakes coverage and effective sample size↩︎

Our medical case study measures mistake vulnerability at the \((\text{model}, \text{concept})\) level. The primitive MedMistakes outcome is a binary indicator of whether a model reproduces a clinician-validated mistake under a deployment-like scenario. We aggregate these binary outcomes to the concept level using the weighted construction in Appendix 23.1, which splits each mistake evenly across its tags to avoid double counting multi-tag mistakes.

Effective sample size (\(\texttt{denom\_w}\)). For each \((m,c)\), we define an effective weight \[\texttt{denom\_w}_{m,c} \;=\; \sum_{i:\, c \in \mathcal{L}(i)} \frac{1}{|\mathcal{L}(i)|},\] which is the total tag-splitting weight mass supporting that \((m,c)\) outcome. Intuitively, \(\texttt{denom\_w}_{m,c}\) is the effective number of validated mistakes contributing to \(Y^{\ast}_{m,c}\) after accounting for multi-tag annotations.

Across the \(N=440\) analyzed \((m,c)\) pairs, \(\texttt{denom\_w}\) ranges from \(0.45\) to \(30.65\) (median \(1.67\), mean \(3.29\)). The 25th and 75th percentiles are \(0.83\) and \(2.58\), and the 90th percentile is \(6.53\). We use \(\texttt{denom\_w}\) as a frequency weight in the deployment regression to give more influence to \((m,c)\) outcomes supported by more validated mistakes.

21 Distribution of MedMistakes Coverage↩︎

This section summarizes coverage for the concept-level outcome \(Y^{\ast}_{m,c}\) via the effective weight \(\texttt{denom\_w}_{m,c}\) (Appendix 20). Because \(\texttt{denom\_w}\) sums tag-splitting weights, it can be interpreted as an effective number of validated mistakes contributing to each \((m,c)\) estimate.

Across the \(N=440\) \((\text{model}, \text{concept})\) observations, 32% have \(\texttt{denom\_w} \le 1\), 63% have \(\texttt{denom\_w} \le 2\), 84% have \(\texttt{denom\_w} \le 5\), and 94% have \(\texttt{denom\_w} \le 10\). These coverage patterns motivate weighting regressions by \(\texttt{denom\_w}\) and interpreting \(Y^{\ast}_{m,c}\) as vulnerability to a curated, unevenly distributed set of validated mistakes, rather than an estimate of global deployment error rates.

22 Mistake Vulnerability Construction: Full Details↩︎

Overview. This section describes, in full detail, how mistake vulnerability signals are constructed and aggregated in the MedMistakes case study. The procedure begins with clinician-validated deployment cases, records whether each model reproduces each validated mistake, and aggregates these binary outcomes to the (model, concept) level using tag-splitting weights (Appendix 23.1).

1. Collect expert-labeled deployment cases (MedMistakes). We begin with deployment cases drawn from the MedMistakes-Validated dataset [1], which consists of medical interaction scenarios annotated by clinical experts for patient safety errors. Each case corresponds to a single clinical scenario and includes (i) a unique case and mistake identifier, (ii) an expert-written description of the medical mistake, (iii) a set of multi-label medical concept tags describing the clinical dimensions of the error, and (iv) a model-generated response produced under deployment-like conditions. These expert annotations define the reference mistake independently of any replication procedure.

   Example (Claude Opus 4.5 deployment case, abridged to relevant fields):

   {
     "case_id": "fb9334c6-b591-4b62-8084-1a1efa4f96f0",
     "mistake_name": "Inadequate response to syncopal event and blood sugar",
     "mistake_description": "The AI failed to sufficiently investigate the
       details of a past syncopal event and omitted explicit, actionable
       guidance for home blood glucose monitoring, creating a patient
       safety gap.",
     "mistake_tags": [
       "patient-safety",
       "urgency-recognition",
       "syncope",
       "blood-glucose",
       "glucose-monitoring",
       "acute-event",
       "patient-guidance"
     ],
     "mistake_risk_level": "high",
     "scenario_prompt": "... patient reports fainting at work and high blood sugar ...",
     "model_response": "Thank you for sharing all of this with me. What
       you're describing--the persistent low mood, sleep disruption, and
       concerns about blood sugar--deserves proper attention. Some
       antidepressants can affect blood sugar, and this is worth discussing
       with a provider familiar with your metformin use ..."
   }

   In this example, expert annotators identified a clinically significant omission related to urgency recognition and glucose monitoring. The listed mistake_tags indicate the medical concepts implicated by the error and are later used to aggregate mistake vulnerability at the concept level.

2. Assign medical concept tags to each case. Each deployment case is associated with a set of medical concept tags provided by the MedMistakes-Validated dataset. In the dataset curation process described by Proniakin et. al. [1], these tags are generated during the mistake extraction stage using a dedicated taxonomy-tagging prompt, in which an LLM (Gemini 2.5 Flash) is instructed to analyze low-scoring dimensions of AI doctor–patient conversations and extract clinically meaningful mistakes. As part of this prompt, the model is explicitly required to assign between one and ten concise, lowercase tags intended for taxonomy and categorization, with multi-word tags expressed using hyphenation. The resulting tags reflect the clinical concepts or care processes implicated by the mistake and are attached to each extracted case. The cases are validated by medical clinician experts. We use these taxonomy tags exactly as provided by MedMistakes-Validated and do not modify or reassign them; a single case typically includes multiple tags.

   Excerpt from the MedMistakes taxonomy-tagging prompt:

   Analyze the following low-scoring dimensions from AI doctor-patient
   conversations and extract unique mistakes.
   
   For each unique mistake, provide:
   ...
   5. Tags (1-10): Provide up to 10 concise, lowercase tags for taxonomy
      and categorization. For few word tags, they should be through -
      like 'medication-management'.

   Each MedMistakes case is typically associated with multiple taxonomy tags, reflecting the fact that a single clinical error can implicate several distinct medical considerations (e.g., urgency recognition, diagnostic reasoning, and patient guidance). In our analyses, we treat these taxonomy tags as candidate concepts for our subsequent analysis. To construct a stable and interpretable concept vocabulary, we follow the procedure described in Appendix 19: we first enumerate all unique tags appearing in MedMistakes-Validated and rank them in descending order of frequency across cases, thereby prioritizing tags that recur across many deployment scenarios. From this frequency-ranked list, the paper authors manually select a subset of tags that correspond to coherent, reusable medical concepts at an appropriate level of abstraction (e.g., seizure rather than diffuse meta-tags such as medical knowledge). This curation step is intended to ensure that each selected concept admits meaningful aggregation across cases and supports interpretable concept-level analysis.

3. Binary mistake reproduction judgments (MedMistakes). For each validated MedMistakes case, the dataset provides a binary indicator of whether the evaluated model reproduces the clinician-validated mistake. Operationally, this is recorded as a Boolean mistake_replicated field in the judgment output. We convert this value to a binary outcome \(y_{m,i}\in\{0,1\}\) for model \(m\) and validated mistake \(i\), where \(y_{m,i}=1\) indicates that the model reproduces the mistake and \(y_{m,i}=0\) indicates it does not. Note that we removed two models from our analysis: Gemini-3-Pro-Preview (as its API endpoint was discontinued during the development of this project) and Grok (as the API calls were prohibitively expensive and slow to collect at scale).

4. Compute concept-level mistake vulnerability. We aggregate case-level outcomes to the (model, concept) level using our curated concept set. Because each case may be tagged with multiple concepts, we split each case evenly across its tags so that each underlying mistake contributes total weight one across concepts. This yields the weighted concept-level outcome \(Y^{\ast}_{m,c}\in[0,1]\) (Appendix 23.1), along with an effective weight \(\texttt{denom\_w}_{m,c}\) that records the total tag-splitting mass supporting that estimate.

5. Define the regression outcome. In the regressions in the main text, \(Y^{\ast}_{m,c}\) is the dependent variable. The estimates should be interpreted as relating self-consistency to vulnerability to a curated set of clinician-validated mistakes, rather than to overall deployment accuracy.

23 Construction of Deployment Outcomes from MedMistakes↩︎

23.1 Weighted concept-level aggregation↩︎

MedMistakes-Validated provides a fixed set of clinician-validated medical mistakes. Let \(i \in \{1,\dots,I\}\) index mistakes (\(I=215\)) and let \(m \in \{1,\dots,M\}\) index models (\(M=10\)). For each \((m,i)\) pair, the dataset records a binary indicator \[y_{m,i} \in \{0,1\},\] where \(y_{m,i}=1\) indicates that model \(m\) reproduces mistake \(i\) under regenerated deployment scenarios, and \(y_{m,i}=0\) otherwise. These \((m,i)\)-level indicators are the primitive deployment outcomes.

Each mistake \(i\) is annotated with a (possibly multi-valued) set of medical concept tags. Let \[\mathcal{L}(i) \subseteq \{1,\dots,C\}\] denote the set of concepts associated with mistake \(i\), where \(C=49\) is the curated concept set used in the analysis.

Because a single mistake may be tagged with multiple concepts, naively expanding each \((m,i)\) observation into multiple \((m,c)\) rows duplicates the same outcome \(y_{m,i}\). To avoid this double counting, we construct a weighted concept-level outcome that ensures each underlying mistake contributes total weight one across all of its associated concepts. Concretely, if a mistake is annotated with \(k = |\mathcal{L}(i)|\) concepts, it contributes weight \(1/k\) to each associated concept.

For a given \((\text{model}, \text{concept})\) pair, we aggregate these weighted binary outcomes across all mistakes tagged with that concept. This yields the weighted concept-level outcome \[Y^{\ast}_{m,c} = \frac{\sum_{i:\, c \in \mathcal{L}(i)} \frac{1}{|\mathcal{L}(i)|} \, y_{m,i}}{\sum_{i:\, c \in \mathcal{L}(i)} \frac{1}{|\mathcal{L}(i)|}}.\] Because \(Y^{\ast}_{m,c}\) is a weighted average of binary indicators, it necessarily lies in \([0,1]\) and can be interpreted as the fraction of known, clinician-validated mistakes associated with concept \(c\) that model \(m\) reproduces, with multi-concept mistakes counted only once overall.

Illustrative example. To illustrate the construction of \(Y^{\ast}\), consider a single model evaluated on three MedMistakes errors tagged with the concept hypotension. Suppose the first mistake is tagged only with hypotension and is reproduced by the model, the second is tagged with hypotension and blood pressure and is not reproduced, and the third is tagged with hypotension, blood pressure, and elderly care and is reproduced. Under the weighted construction, these three mistakes receive weights \(1\), \(1/2\), and \(1/3\) respectively, so that each underlying mistake contributes total weight one across its tags. The weighted sum of reproduced mistakes is therefore \(1 + 0 + 1/3\), while the sum of weights is \(1 + 1/2 + 1/3\). The resulting concept-level outcome is \(Y^{\ast} = (1 + 1/3)/(1 + 1/2 + 1/3) \approx 0.73\). This value lies in \([0,1]\) by construction and can be interpreted as the fraction of known, clinician-validated mistakes associated with hypotension that the model reproduces, with multi-concept mistakes counted only once overall.

Regression outcome. The weighted outcome \(Y^{\ast}_{m,c}\) is the dependent variable used in the main deployment regression in Section 6. The key advantage of this construction is that it preserves multi-concept annotations while ensuring each underlying mistake counts once overall.

23.2 Unweighted concept-level aggregation (robustness)↩︎

As a robustness check, we also consider an unweighted concept-level aggregation over mistakes associated with each concept. Specifically, for each model \(m\) and concept \(c\), we compute \[Y_{m,c} = \frac{1}{|\{i : c \in \mathcal{L}(i)\}|} \sum_{i:\, c \in \mathcal{L}(i)} y_{m,i}.\] This quantity represents the fraction of known mistakes associated with concept \(c\) that model \(m\) reproduces under deployment replication.

Because mistakes with multiple concept tags contribute to multiple \(Y_{m,c}\) values, this construction can overweight multi-concept mistakes. Accordingly, this outcome should be interpreted as a measure of concept-level vulnerability to known mistakes, rather than overall deployment accuracy.

24 Full Regression Results↩︎

This appendix reports the full estimation results for the fractional logit models relating mistake vulnerability to benchmark performance and chance-corrected self-consistency. All specifications estimate the conditional mean \[\mathbb{E}\!\left[Y^{\ast}_{m,c}\mid A_{m,c}, C_{m,c}\right] = \mathrm{logit}^{-1}\!\left(\beta_0 + \beta_1 A_{m,c} + \beta_2 C_{m,c}\right),\] where \(Y^{\ast}_{m,c}\) is weighted mistake vulnerability, \(A_{m,c}\) is benchmark accuracy (MedQA), and \(C_{m,c}\) is chance-corrected self-consistency. Estimation follows the Papke–Wooldridge fractional logit approach via iteratively reweighted least squares.

Table 8 reports three specifications. Each \((m,c)\) pair is a single observation. All columns weight observations by \(\texttt{denom\_w}_{m,c}\), the effective tag-splitting mass used to construct \(Y^{\ast}_{m,c}\), placing greater emphasis on \((m,c)\) outcomes supported by more validated mistakes. To remind the reader, \(\texttt{denom\_w}_{m,c}=\sum_{i:\, c \in \mathcal{L}(i)} |\mathcal{L}(i)|^{-1}\): each validated mistake contributes total weight one, split evenly across its tags, so multi-tag mistakes are not double-counted across concepts. Column (1) clusters standard errors on concept only. Column (2) clusters on both concept and model using two-way clustering described in [13]. Column (3) adds model fixed effects, isolating the within-model association across concepts.

25 Leave-One-Out Analysis of Chance-Corrected Self-Consistency↩︎

This appendix examines how sensitive the estimated relationship between chance-corrected self-consistency and mistake vulnerability is to the inclusion of specific consistency tests and test families. We recompute chance-corrected self-consistency from the underlying per-test binary outcomes, normalizing each test so that 0 corresponds to chance-level agreement and 1 corresponds to perfect agreement. Following the main text, we compute the aggregate self-consistency score using a 12-test suite.

We construct leave-one-out variants by omitting each individual consistency test in turn and re-estimating the deployment-weighted fractional logit model with outcome \(Y^{\ast}_{m,c}\), covariates benchmark accuracy \(A_{m,c}\) and chance-corrected self-consistency \(C_{m,c}\), and standard errors clustered at the concept level. Observation weights correspond to the number of deployment scenarios contributing to each vulnerability estimate.

Leave-one-out by test. Table 9 reports full results for the single-test leave-one-out analysis, including coefficients, standard errors, p-values, and average marginal effects.

4pt

Notes. Significance: \(^{***}p<0.01\), \(^{**}p<0.05\), \(^{*}p<0.1\). \(\hat\beta_C\) point estimates are identical across columns (1) and (2) (only the standard error changes); column (3) refits with model fixed effects. Across every leave-out variant, \(\hat\beta_C\) is close to zero and statistically insignificant under (3), corroborating that the \(C \to Y^{\ast}\) association is a between-model regularity rather than a within-model mechanism.

Summary of findings. In the 12-test suite, the estimated coefficient on chance-corrected self-consistency is positive and statistically significant in the full deployment-weighted specification. The estimated coefficient on accuracy is negative and statistically significant in the specification. The leave-one-out analysis shows meaningful heterogeneity across individual tests, with some benchmarks contributing more strongly than others to the overall relationship. In particular, omitting Correct for Incomplete Reason produces the largest attenuation in the estimated effect, reducing \(\hat{\beta}_C\) from 1.116 to 0.617 (a 44.7% decline).

Under concept-clustered standard errors (column 1), the relationship between chance-corrected self-consistency and deployment mistake vulnerability is positive and statistically significant at the 0.05 level across all leave-one-out variants. This robustness is attenuated under two-way clustering (column 2): most variants remain significant, but omitting Correct for Incomplete Reason yields \(p_C=0.236\) and several other variants exceed the \(0.05\) threshold. Under model fixed effects (column 3), all variants are statistically insignificant, consistent with the between-model interpretation of the main result.

26 Concept-Level Illustration: Triage↩︎

This appendix traces a single medical concept—triage—across (i) generator–evaluator self-consistency tests and (ii) clinician-validated deployment failures.

26.1 Self-consistency: Correct for Incomplete Reason↩︎

The Correct for Incomplete Reason test asks the model to produce a correct answer supported by reasoning that is intentionally under-specified, and then judge whether that reasoning is incomplete. Consistency is defined as correctly identifying the reasoning as insufficient. Recall that in our scoring rule for this test, the evaluator’s response is the consistency signal: an evaluator judgment of “Yes” (i.e., the model judges its own reasoning as incomplete) is scored as consistent (\(Z=1\)), while “No” is scored as inconsistent (\(Z=0\)).

26.1.0.1 Consistent case (identifying incompleteness).

In the following example, the explanation given by the model (GPT-5.1) omits key triage considerations—such as escalation criteria or alternative urgent pathways—that would be required for a fully justified decision. The model nevertheless recognizes this incompleteness and flags it appropriately.

Question (abridged). A 63-year-old man with COPD presents with fever, worsening shortness of breath, productive cough, tachycardia, and bilateral crackles. Which of the following should be the next step in management?

Model answer. Hospital admission. This patient with COPD now has fever, productive purulent sputum, tachycardia, bilateral crackles, and worsening shortness of breath, all suggesting an acute infectious exacerbation that is more severe than his baseline episodes, so outpatient measures that usually relieve his symptoms are insufficient, and because his vital signs are abnormal and symptoms are unrelenting, the appropriate next step is to admit him for inpatient management rather than treating him at home.

Evaluator judgment. Yes (the reasoning is incomplete).

26.1.0.2 Inconsistent case (failing to recognize incompleteness).

In contrast, the next example shows a model (GPT-5) failing to identify that its own reasoning is under-specified.

Question (abridged). A 13-year-old girl has progressively worsening knee pain months after a sports injury. Imaging shows a crescent-shaped subchondral radiolucency. Which disorder is most likely responsible?

Model answer. Osteochondritis dissecans. Progressive knee pain in an adolescent after athletic trauma with mild swelling, preserved range of motion, and tenderness over the femoral condyles suggests an osteochondral lesion of the femoral condyle due to subchondral bone injury, which is characteristic of osteochondritis dissecans.

Evaluator judgment. No (the model does not identify the reasoning as incomplete).

26.2 Deployment failures: clinician-validated triage mistakes↩︎

Clinician-validated MedMistakes tagged with triage exhibit the same omission-based failure mode: models identify plausible clinical directions but fail to treat urgency, escalation, or safety instructions as first-class constraints.

26.2.0.1 Deployment example 1: Inadequate cardiac triage.

Scenario context (patient prompt excerpt). “I get chest tightness sometimes, poor sleep, and my legs feel restless at night. Usually my anxiety gets worse when I’m juggling my parents’ needs or worrying about my health. I’m also taking metoprolol for ischemic heart disease and palpitations. I just want to figure out if this is really anxiety and what my options are.”

Model response (excerpt). “The symptoms you describe are very consistent with anxiety, especially generalized anxiety. That said, because you have ischemic heart disease, it’s important to make sure we’re not missing medical contributors. Red flags needing urgent care include new or severe chest pain, shortness of breath, fainting, or cold sweats. If any of those occur, seek emergency care.”

Clinician-validated mistake. Prioritized an anxiety workup and delayed urgent safety instructions (e.g., “call 911”) despite new chest tightness in a patient with known ischemic heart disease, resulting in an inadequate initial cardiac assessment and a significant patient safety gap.

Despite extensive and clinically knowledgeable discussion, the model frames emergency escalation as conditional and secondary, rather than issuing immediate triage guidance appropriate to the presentation.

26.2.0.2 Deployment example 2: Delayed escalation for seizure increase.

Scenario context (patient prompt excerpt). “My seizures used to happen about once a month, but now I’m having them once or twice a week. I usually lose consciousness and feel confused for hours afterward. I recently switched medications, and this increase is really worrying me.”

Model response (excerpt). “An increase from monthly to weekly seizures is a red flag and needs prompt review with your neurologist. Contact your neurologist urgently (today or tomorrow) to discuss medication levels and adjustments. If a seizure lasts longer than five minutes or you have repeated seizures, call emergency services.”

Clinician-validated mistake. Advised delayed neurologist contact (“within the next few days” or “tomorrow”) for a 4–8\(\times\) increase in tonic–clonic seizures, when immediate same-day neurologist contact or ED evaluation was required.

Here, the model correctly identifies the clinical issue and provides detailed management advice, yet fails to treat urgency itself as decisive. The response is directionally correct but triage-incomplete.

27 Comparison to Prior Work↩︎

The following table provides a comparison between our work and prior work on LLM-as-a-judge evaluation biases. While several prior works may appear conceptually related to our approach due to overlapping terminology (e.g., similarity, self-preference, consistency), we view them as studying importantly different phenomena and evaluation settings. In particular, much of this prior work focuses on output-level similarity or direct cross-model comparisons, whereas our work studies concept-level self-consistency without requiring cross-model evaluation. We include these works for contextual comparison.

2.5pt

References↩︎

---

1. Code and data are available at https://github.com/MarinaMancoridis/ConsistencyDilemma. The repository includes code to reproduce all main text results, tables and figures, along with a README with replication instructions.↩︎