1 Introduction↩︎

Large language models (LLMs) are increasingly deployed in settings that require synthesizing information from multiple sources in tasks like question answering, fact checking, and report generation [1]–[3]. However, conflicts frequently exist among these sources, and current models often overlook them, resulting in potentially misleading responses [4]. For instance, as shown in Figure 1, when asked “Should we all get vaccinated?”, the chatbot Perplexity² retrieves several reliable documents with differing views but replies, “Yes, it is strongly recommended that everyone gets vaccinated,” without acknowledging possible exceptions such as medical contraindications or allergies.

Existing metrics that assess the trustworthiness of LLM outputs focus primarily on faithfulness and factuality [6], [7]. These metrics evaluate whether a response aligns with its supporting context but typically treat all grounding documents as a single, unified source [5], [8]. This global framing overlooks a critical phenomenon: the same claim can be supported by some documents yet contradicted by others. Such conflicts are pervasive in natural text collections, where evidence is incomplete, perspectives diverge, or knowledge evolves over time [9]–[11]. Ignoring such conflicts can produce overconfident or misleading outputs, undermining LLM trustworthiness in high-stakes settings.

We introduce ConflictScore, a conflict-aware extension of prior atomic-claim factuality pipelines. Like FactScore [5], it decomposes responses into atomic claims and verifies them against grounding evidence; however, ConflictScore does not collapse the grounding set into a single reference corpus. Instead, it labels each claim against each grounding document independently as support, contradict, or irrelevant, and then aggregates these per-document labels to identify when a claim is simultaneously supported by some documents and contradicted by others. Figure 2 demonstrates the process of ConflictScore. To encourage nuanced reasoning, we define a document as supporting a claim even if it supports only part of it (see Figure 1). We present two complementary variants: ConflictScore-Count (CS-C), which measures the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), which quantifies the balance between supporting and contradicting evidence. Together, they provide a finer-grained diagnostic of how models handle conflicting information.

To evaluate the effectiveness of ConflictScore, we introduce ConflictBench. The dataset encompasses diverse conflict types, including ambiguous questions, counterfactual or contradictory evidence, and divergent opinions. Our results show that ConflictScore reliably identifies conflicts and provides consistent calibration across domains. We benchmark state-of-the-art LLMs with ConflictScore, showing that balancing prompting strategies improve conflict awareness, though models still often produce overconfident answers despite clear contradictions. Finally, we present a case study that applies ConflictScore to TruthfulQA [12] and demonstrate that feeding conflict signals back to the model improves response truthfulness.

In summary, our contributions are three-fold³:

• We introduce ConflictScore, a robust metric for quantifying conflicts within grounding documents and assessing how well model responses reflect such conflicting evidence.

• We develop ConflictBench, a dataset designed to systematically evaluate conflict detection and calibration across diverse types of conflicts.

• We demonstrate that ConflictScore not only serves as a diagnostic tool, but also acts as a corrective signal, guiding models toward more cautious and accurate reasoning.

2 Related Work↩︎

2.1 Factuality and Faithfulness Evaluation↩︎

Prior studies evaluating the reliability of LLM outputs primarily focus on factuality and faithfulness [5], [6], [8], [13]–[15]. [5] decompose long-form outputs into atomic facts and compute the proportion supported by retrieved knowledge sources. Similarly, [8] propose SAFE, a search-augmented factuality evaluator that verifies each statement against retrieved passages using an LLM and aggregates results via an F1-style score. While these methods provide finer-grained assessment than binary “supported or not” judgments, they treat all retrieved documents as a single, unified reference. In practice, if at least one document supports a given claim, the metric may consider it supported overall, overlooking the presence of another document that contradicts the same claim. In contrast, ConflictScore targets inter-evidence disagreement — the case where the source set itself contains both supporting and contradicting documents for a single claim — and measures how the response navigates such disagreement.

Recent work extends this paradigm to broader response consistency evaluation, but still lacks explicit conflict modeling. [16] propose AlignScore, a unified alignment model for factual consistency, while [17] introduce FLASK, a fine-grained protocol for evaluating alignment skills such as factuality and reasoning. Retrieval-based frameworks like Self-Checker [18] verify claims against retrieved contexts, but assume the evidence forms a consistent ground truth. Overall, existing trustworthiness metrics measure global alignment without modeling contradictions within retrieved evidence, whereas our ConflictScore explicitly quantifies how well responses handle conflicting evidence in context.

2.2 Modeling and Evaluating Conflicting Evidence↩︎

Prior work on knowledge conflicts has largely focused on discrepancies between parametric and retrieved knowledge [19]–[23]. In contrast, less attention has been paid to conflicts arising within retrieved corpora themselves, including ambiguity [9], [24], differing perspectives [10], [25]–[27], and contradictory evidence [28]–[31]. Recent benchmarks show these conflicts are common and challenging for LLMs. [30] find that  25% of open-domain questions retrieve contradictory evidence even for unambiguous queries, while [28] show that state-of-the-art LLMs often ignore such conflicts and produce overconfident answers. Unlike prior work, our ConflictScore explicitly models contradictions among grounding documents to evaluate how well responses handle conflicting evidence.

3 The ConflictScore Metric↩︎

Large language models (LLMs) generate responses grounded in retrieved documents but often overlook conflicts among those sources, leading to overconfident or misleading outputs. ConflictScore evaluates a response by explicitly measuring when the same claim is both supported and contradicted by different grounding documents, and by quantifying the balance between these opposing signals. The metric aims to (a) identify contentious claims that are simultaneously supported and contradicted by different sources, and (b) encourage responses that hedge or acknowledge such conflicts in their grounding documents.

Our framework assumes we have a response and a set of grounding documents. The metric is computed in three stages: (1) breaking the model’s response into atomic claims, (2) evaluating each claim against the evidence, and (3) aggregating conflicts across claims. Figure 2 demonstrates the process of ConflictScore.

3.0.0.1 1. Claim Decomposition.

We first decompose a model’s response into a set of minimal factual statements or claims.

3.0.0.2 2. Evidence Evaluation.

Each claim is then checked against every document in the grounding set and labeled as supported, contradicted, or irrelevant. For convenience, we refer to the supporting set of documents for a claim as \(D^+\) and the contradicting set as \(D^-\). Furthermore, we consider a claim has conflicting evidence or conflicts if both \(D^+\) and \(D^-\) are non-empty—i.e., if some documents support it while others contradict it. To encourage responses that hedge or acknowledge such conflicts, we consider a document as supporting a claim even if it only partially supports the claim⁴. Figure 1 presents an example where the second response is considered being supported by all four documents, where the first two support its first part and the second two support its second part. The exact prompting templates used for this process are provided in Table 10 and 11.

3.0.0.3 3. ConflictScore Calculation.

We present two complementary measures. ConflictScore-Count (CS-C) measures the fraction of claims in a response that fall into this conflicting category. Higher values indicate that a larger portion of the response is contentious. ConflictScore-Ratio (CS-R) considers the balance between supporting and contradicting evidence. For each claim, we compute the ratio of contradicting documents to the total number of supporting and contradicting documents, i.e. \(\frac{|D^-|}{|D^+| + |D^-|}\), and then average this ratio across all claims. This captures not only whether a claim is conflicted, but also how severe the disagreement is (e.g., a 1:1 split vs.a 9:1 imbalance). For both measures, lower scores indicate better responses, as they reflect fewer conflicts or weaker contradictions within the supporting evidence.

The ConflictScore framework is highly flexible and can accommodate various model choices for each component. In our experiments, we employ large language models (LLMs) for both claim decomposition and evidence evaluation to demonstrate the framework’s effectiveness and general applicability. Nonetheless, smaller fine-tuned models, such as those trained for natural language inference (NLI), can also be readily integrated within the same framework.

4 Conflict Detection and ConflictBench↩︎

To understand to which extent ConflictScore successfully identifies overconfident claims in the presence of contradictory evidence, we define the task of Conflict Detection and curate a dataset ConflictBench to evaluate ConflictScore. Table 2 evaluates the binary conflict-identification subroutine that underlies CS-C and CS-R, rather than the aggregate scores themselves. This isolates the calibration of the per-document labeling component before we apply CS-C/CS-R to full responses in Section 5.

4.1 Task Definition↩︎

Given a claim and a list of grounding documents, the task is to decide whether it has conflicting evidence in the grounding documents, i.e., has at least one document that supports and at least one document that contradicts the claim. The expected output is a binary label of Conflict or No Conflict.

4.2 ConflictBench Curation↩︎

While prior datasets capture specific forms of inter-document inconsistency or disagreement, there is no existing dataset that unifies these phenomena under a single, task-level formulation of conflict detection with different conflict types and consistent evaluation protocols. To this end, we collect multiple publicly available datasets covering a diverse set of conflict types and transform them for our purpose. Each of the datasets is preprocessed to follow a unified format. The preprocessing details are in Appendix 8.1, and representative examples from each dataset are shown in Table 9.

ConflictingQA is a large-scale QA benchmark where retrieved passages may contain contradictory answers, directly testing a model’s ability to reason over disagreements across sources [34]. The conflicts in this dataset arise from contentious or controversial questions, such as “Is infinite scrolling a good web design technique?”, where differing opinions persist across the web.

AmbigDocs contains ambiguous or underspecified questions paired with multiple plausible interpretations, probing whether ConflictScore can identify hidden ambiguity in model responses [24]. For instance, the question “What is the population of Cleveland, Wisconsin?” may retrieve passages reporting different numbers from different timestamps.

ContraQA perturbs the original documents and introduces counterfactual and adversarial pairs of passages with explicitly contradictory statements, offering a direct evaluation for conflict detection [32]. For example, “What year was the University of Warsaw established?”, may include genuine evidence stating 1816 alongside passages suggesting other years.

MacNoise similarly injects unreliable or counterfactual passages to induce inconsistencies in the grounding documents [33]. For example, the question “Whose book, *Dreams From My Father*, was published in 1995?” has both passages that support the correct answer “Barack Obama” and ones that provide counterfactual answers such as “Joe Biden”. MacNoise includes two variants derived from different datasets: Natural Questions [35] and TriviaQA [36], denoted as MacNoise-NQ and MacNoise-TQA, respectively.

4.3 ConflictBench Statistics↩︎

Table 1 summarizes the statistics of ConflictBench. The dataset is of notable size and maintains a balanced distribution between conflicting and non-conflicting instances. Moreover, ConflictBench encompasses a diverse range of conflict types observed in grounding documents, including ambiguous queries, opinion-based disagreements, and cases involving both factual and counterfactual evidence. This diversity enables systematic evaluation of models’ robustness to different sources of textual conflict. ConflictBench aggregates existing datasets rather than collecting new annotations, so it inherits the annotation protocols and quality controls of the source datasets. More discussion about the quality control of these source datasets are shown in Appendix 8.2

4.4 Results and Error Analysis↩︎

Table 2 summarizes the performance of ConflictScore across multiple datasets. We conduct experiments using Gemini-3.5-Flash with prompts shown in Table 10 and Table 11. The metric achieves consistently strong results, with high precision and recall, and accuracies above 90% in both conflict and no-conflict cases overall. Furthermore, to illustrate that our ConflictScore metric can generalize across different model families, we present results of other frontier models in Table 6 and 7, showing consistent, strong performance on ConflictBench. These results demonstrate that ConflictScore is robust and well-calibrated to detect conflicts across datasets exhibiting diverse types of conflicts.

Despite its strong performance, ConflictScore can occasionally fail when errors in intermediate relation predictions propagate to conflict classification. One such case is illustrated in Figure 3. Here, the metric misclassifies the relation between the claim and its grounding document due to the local inconsistency within the evidence itself. Although the overall passage presents both supportive and contradictory statements, the concluding section negates earlier claims, leading the model to predict a contradiction where human annotation labels the relation as support. Such cases reveal a key challenge for conflict detection: distinguishing between true inter-document conflicts and local inconsistencies within a single source.

To better understand these failures, we conduct a human validation study on 50 randomly sampled error cases, approximately one-third of all errors, focusing on the evidence evaluation stage (claim decomposition is largely reliable with state-of-the-art models, and the aggregation step is deterministic). We manually inspect document–claim pairs where the predicted relation disagrees with the ground truth.

Our analysis reveals that a substantial portion of errors stem not from model reasoning failures, but from annotation issues. Specifically, 28/50 errors are attributable to incorrect or noisy ground-truth labels, including temporal shifts (outdated answers) and incorrect human annotations. Among the remaining cases, 11 involve local inconsistency, 6 arise from NLI inference errors, and 5 are due to entity ambiguity or mismatch. We provide representative examples of each category in Table 9.

5 Benchmarking Frontier LLMs with ConflictScore↩︎

We use ConflictScore to benchmark frontier LLMs in a setting where the retrieved evidence is intrinsically contradictory. Concretely, we evaluate GPT-5.4-Mini, GPT-5.4-Nano, and Qwen3-32B under several retrieval-augmented prompting strategies on a slice of ConflictBench.

5.0.0.1 Evaluation setup.

We sample 100 items from the ConflictingQA split of ConflictBench whose gold labels indicate conflicting evidence, reusing the conflicting passages provided by ConflictingQA (retrieved via Google Search and verified by two-LLM stance checking [34]) without additional retrieval or re-ranking. For each item, we extract the main entity from the original question and reformulate the task as “Write a three-paragraph report about {\(\mathit{main\_entity}\)}.”, supplying the conflicting passages as grounding documents. We then apply ConflictScore (Section 3) to each generated report and average per-report CS-C and CS-R over the 100 reports per setting. Since a desirable report should acknowledge both sides, lower scores are better. This setup mirrors the common task of report writing, where a system should surface disagreements rather than commit to a single view.

5.0.0.2 Prompting strategies.

We compare three retrieval-augmented variants that differ only in how strongly they instruct the model to handle disagreement.

• RAG: A minimal baseline asking for a concise three-paragraph report from the given documents, with no mention of hedging or conflict.

• RAG (Balanced): Adds brief guidance to “be cautious and hedge accordingly,” asking the model to consider all perspectives and acknowledge potential conflicts.

• RAG (Super-Balanced): Provides detailed rules: hedge under uncertain evidence, avoid definitive claims unless sources agree, attribute information, and explicitly note conflicts.

Full prompt templates appear in Tables 12–14.

5.0.0.3 Results and insights.

Table 3 reports results across model families, sizes, and prompting variants on this 100-item slice. Across all three models, prompt-based balancing yields only modest gains: CS-C remains consistently high, indicating that even with explicit instructions, models frequently commit to a single stance despite contradictory evidence. Stronger instructions are not reliably better—Super-Balanced sometimes matches or underperforms the simpler Balanced variant, and within each family the absolute improvements are small. We hypothesize this may relate to the claim-granularity sensitivity discussed in our limitations section: detailed attribution instructions may push the model from synthesized, qualified statements toward atomic source-attributed claims (“A reports X; B reports Y”), each of which still commits to a single side under claim-level scoring and counts toward the metrics, even when the report reads as balanced overall. Within this report-writing slice of ConflictingQA, the results are suggestive rather than conclusive, but they hint that prompt tuning alone offers limited leverage on conflict acknowledgment, and that stronger signals—such as conflict-aware training or source reliability weighting—may be needed.

6 Case Study: Improving Truthfulness with ConflictScore↩︎

A central motivation behind ConflictScore is not only to diagnose when models synthesize contradictory evidence, but also to leverage this signal to improve the truthfulness of generated responses. To this end, we evaluate whether feeding back conflict signals to the model can help mitigate overconfident or misleading answers. We test this hypothesis on TruthfulQA [12], a benchmark specifically designed to measure whether models produce factually correct and non-misleading content.

6.1 Experimental Setup↩︎

We focus on the multiple-choice setting of TruthfulQA and adopt the improved binary-choice version recommended by the dataset authors in January 2025 ⁵. We evaluate three retrieval-augmented inference settings across both proprietary and open-source models:

• RAG: A retrieval-augmented generation baseline where top 10 documents retrieved from Google Search are supplied, but no explicit conflict feedback is given.

• Control-RAG: A variant with explicit instructions in the prompts that encourages evidence-aware answers without using ConflictScore.

• Regenerated-RAG: Our proposed setting, where responses are first generated with RAG, then evaluated by ConflictScore framework with the same model. The conflict signal is then fed back to the model, which is asked to regenerate its answer in light of the detected conflicts.

Specific prompting templates for inference and regeneration are provided in Table 15, Table 16, and Table 17.

6.2 Results: Multiple-Choice Setting↩︎

Table 4 summarizes results across two proprietary models (gemini-3.1-flash-lite and gpt-4.1-mini) and two strong open-weight models (gpt-oss-20b and qwen3-30b-a3b). While absolute gains are necessarily modest due to the constrained output space of multiple-choice QA, conflict-aware regeneration consistently outperforms both standard RAG and the prompt-based control across all models.

6.3 Analysis↩︎

To better understand where these improvements come from, Table 5 analyzes only the subset of questions for which ConflictScore detects conflicts in the model’s initial RAG response to the retrieved documents. For each model, we report how often regeneration corrects an originally wrong answer (Improve Rate) versus how often it degrades an originally correct one (Harm Rate).

Across all models, regeneration corrects a substantial fraction of incorrect answers—up to 74.00% for gemini-3.1-flash-lite—while introducing very few new errors (harm rates below 3%). This asymmetry results in positive net improvements for all models. These findings indicate that ConflictScore selectively targets unreliable decisions rather than inducing indiscriminate changes, reinforcing its role as a corrective signal.

Figure 4 presents representative cases of both successful and unsuccessful regenerations. In the first example, the model correctly revises its answer after recognizing that the retrieved evidence depends on differing definitions and geographical contexts. In contrast, the second example illustrates a failure case where the model is swayed by a majority of seemingly authoritative but misleading sources. This case highlights the model’s continued difficulty in discerning the reliability of conflicting sources, particularly when misleading evidence dominates the retrieved context.

7 Conclusion↩︎

We propose ConflictScore, a metric that evaluates how well language models acknowledge and handle conflicting evidence by assessing atomic claims against grounding documents. Experiments on ConflictBench show that ConflictScore is robust to detect conflicts across diverse conflict types and reveal that prompt-based balancing offers only limited gains. We further demonstrate that feeding back conflict signals improves model truthfulness on TruthfulQA. Overall, ConflictScore provides both an evaluation signal and a foundation for developing conflict-aware training and calibration methods.

Limitations↩︎

7.0.0.1 Runtime, cost, and scalability.

While ConflictScore offers a fine-grained and interpretable way to assess how models handle conflicting evidence, it comes with practical computational costs. The full pipeline requires evaluating every atomic claim in a response against each retrieved document, resulting in a quadratic number of evaluations when both sets are large. This design enables precise conflict attribution but can become expensive for long-form outputs or large retrieval sets. As a concrete profile, running our calibration experiments (Section 4) on 2,293 items with claude-haiku-4.5 via OpenRouter took roughly 6 hours of time (~9.7s/item) at list pricing of $1/$5 per M input/output tokens.

Several more efficient variants can be adopted depending on the application. First, a lightweight version skips claim decomposition and treats the entire response as a single unit, substantially reducing cost but sacrificing granularity. Second, one can prompt the model to first identify a small set of salient or representative claims and evaluate only those, trading exhaustive coverage for efficiency. Finally, an alternative approach provides all grounding documents at once when labeling claim–evidence relations, which accelerates inference but often reduces accuracy because models tend to merge or overlook contradictory details when presented with long contexts.

Future work may explore methods to prioritize which claims or evidence pairs to evaluate, enabling scalable deployment of ConflictScore in large-scale or real-time settings.

7.0.0.2 Source-level Reliability and Trustworthiness.

Another limitation of ConflictScore is that it currently treats all grounding documents equally, without explicitly modeling source reliability or trustworthiness. In practice, retrieved evidence may vary substantially in quality, ranging from authoritative references to unreliable or misleading sources such as conspiracy websites. As a result, contradictions identified by ConflictScore do not distinguish between conflicts among equally credible sources and conflicts driven by low-quality evidence. At the same time, the framework is naturally compatible with reliability-aware extensions. Since ConflictScore evaluates claim–document relations independently, source-level trustworthiness signals can be easily incorporated as additional weighting or filtering mechanisms. As a lightweight preliminary experiment, we first prompted a model to assess the trustworthiness of retrieved documents before regeneration on TruthfulQA, improving performance from 85.03 to 85.19 for gpt-oss-20b in the TruthfulQA regenerate experiment. Although modest, this result suggests that conflict awareness and source reliability are complementary, and that ConflictScore can be readily augmented with credibility modeling in future work.

7.0.0.3 Sensitivity and Dependence of Upstream Modules.

ConflictScore depends on upstream modules for claim decomposition and claim–document relation labeling. As with prior atomic-claim factuality pipelines, errors in either step can propagate to the final score. In particular, claim granularity choices may affect whether a response is viewed as appropriately qualified or as containing separate unsupported claims. For example, a nuanced statement that integrates exceptions may be scored differently depending on whether it is extracted as one qualified claim or multiple simpler claims. Similarly, evidence-labeling errors can affect whether documents are marked as supporting, contradicting, or irrelevant.

The appropriate claim extraction strategy may also depend on the use case. When the goal is to comprehensively analyze all claims in a model response, practitioners can follow our Section 5 setup and extract all atomic claims. For long-form responses where exhaustive processing is impractical, users may instead filter for salient or high-impact claims to reduce cost. If claims are frequently split across different parts of a response, users can also prompt an LLM to merge related statements into more complete qualified claims before evaluation. Thus, while our experiments use a fixed extraction protocol, the framework is modular and can be adapted to different deployment needs. Future work can further improve robustness through discourse-aware claim extraction, alternative verification backends such as NLI models, ensemble labeling, or sensitivity analyses across prompts and model choices.

8 ConflictBench Details↩︎

8.1 Preprocessing↩︎

For ConflictingQA, we simply take the original query and prompt GPT-4.1 to transform it to a claim, such as "Infinite scrolling is a good web design technique." This way we end up with one claim per query. We then take the original conflict labels and grounding documents from the dataset as they are. For AmbigDocs, in the case of having conflicts, we take the question and its transformed claim, and their grounding docs as the input, and for the case of not having conflicts, we take claim and its corresponding supporting document, as well we two random documents for other queries as (which should be classified as irrelevant) as the grounding documents. The preprocessing of ContraQA and MacNoise follows the same process as AmbigDocs as well.

8.2 Quality Control↩︎

ConflictBench is constructed by transforming existing public datasets into a unified conflict-detection format, rather than by collecting new human annotations. Consequently, its labels inherit the annotation protocols, automatic construction procedures, and quality-control mechanisms of the source datasets. We summarize these controls below and report inter-annotator agreement only when it is explicitly provided by the source paper.

8.2.0.1 ConflictingQA.

ConflictingQA [34] contains controversial yes/no questions paired with real web evidence supporting different answers. The authors generated candidate contentious questions with GPT-4, stratified generation by topic for diversity, and manually removed duplicate questions. Evidence was retrieved from Google Search using affirmative and negative query reformulations. To assign document stance, the authors used an ensemble of Claude Instant and GPT-4 and retained only examples where the two models agreed; documents judged irrelevant were filtered out. They also filtered paragraphs by relevance using a 512-token windowing procedure and a TAS-B similarity threshold. The paper does not report a human inter-annotator agreement or kappa value for these labels.

8.2.0.2 AmbigDocs.

AmbigDocs [24] is generated from Wikipedia disambiguation pages, where documents correspond to different entities sharing the same surface name. The dataset construction uses several automatic quality filters: generated examples are removed if they are incorrectly formatted, if the question is not actually ambiguous, if answers are too similar, or if the answer is not entailed by its supporting document according to an NLI check. The authors also filter answer-set expansion using NLI entailment and a length-normalized perplexity threshold. For their answer-type ontology and evaluation procedure, they report Cohen’s \(\kappa=0.85\) between two human annotators on a subset of 125 model outputs, and \(\kappa=0.83\) between their automatic categorization heuristic and human labels on 500 outputs. These kappa values validate the evaluation ontology and automatic categorization procedure; the paper does not report a separate kappa value for the automatically generated dataset instances themselves.

8.2.0.3 ContraQA.

ContraQA [32] creates contradictory passages by modifying original QA evidence into deceptive alternatives. For the human-written portion, the authors used Amazon Mechanical Turk with explicit editing guidelines requiring multiple edits, at least one long edit, contradictions to the original passage, fluency, realism, and avoidance of commonsense errors. Workers were restricted to native-English-speaking countries and required at least a 90% approval rating. The authors provided examples and explanations in the annotation interface and hired three computer-science graduate students to validate HIT annotations. They report an average HIT acceptance rate of 93.75%. The paper does not report an inter-annotator agreement or kappa value.

8.2.0.4 MacNoise-NQ and MacNoise-TQA.

MacNoise [33] constructs counterfactual evidence for open-domain QA using NQ-Open and TriviaQA-Open as base datasets. The authors generate perturbations only for answer-containing documents and use named-entity typing to identify perturbable instances. Counterfactual documents are produced either through same-type entity substitution or through LLM rewriting. For the MacNoise dataset, GPT-3.5-turbo is used for the NQ-Open training set and GPT-4 for the NQ-Open and TQA-Open evaluation sets. The generation prompt requires the rewritten document to remain answerable by the original question, preserve information explicitly present in the question, avoid containing the original answer, and negate or alter the original support when necessary. For the TriviaQA-Open portion, the authors add three quality examples as in-context demonstrations to improve consistency across OpenAI model versions. The paper does not report human inter-annotator agreement or kappa values for MacNoise.

9 ConflictBench Results↩︎

To assess whether ConflictScore generalizes across model families, we report calibration results on ConflictBench using GPT-5.4-mini (Table 6) and Claude-haiku-4.5 (Table 7). Both models achieve overall F1 scores comparable to Gemini-3.5-Flash reported in the main paper, indicating that ConflictScore remains well-calibrated across backbones.

10 Partial versus Strict Support↩︎

To encourage nuanced reasoning, our prompts treat a document as supporting a claim even when it only partially supports the claim. We perform a lightweight ablation to study this design choice by comparing our default Partial Support setting against a stricter variant where all references to partial support are removed from the prompts. We evaluate both settings on 100 randomly sampled examples from ConflictBench.

Table 8 shows that both settings achieve similar performance overall. The strict setting slightly improves precision and no-conflict accuracy, while recall and conflict accuracy remain unchanged. These results suggest that allowing partial support does not substantially hurt overall detection quality, but it better supports a more nuanced reasoning of conflicts as demonstrated in Figure 1.

11 Examples↩︎

Table 9 presents representative error cases identified in our analysis, and Figure 5 provides an end-to-end worked example.

12 Prompts↩︎

4pt

4pt

4pt

4pt

4pt

4pt

4pt

4pt

References↩︎

---

1. Work done during an internship at Microsoft.↩︎

2. Perplexity is a state-of-the-art retrieval-augmented AI answer engine. The query was made in September 2025.↩︎

3. We release all code, prompts, data, and result files at https://github.com/microsoft/ConflictScore↩︎

4. We discuss a lightweight ablation of this decision choice in Appendix 10.↩︎

5. https://github.com/sylinrl/TruthfulQA↩︎