1 Introduction↩︎

Scrutinising evidence sources is central to assessing the reliability of claims and identifying issues such as misleading statistics, omitted context, or unsupported statements, particularly in high-stakes domains such as journalism, public health, and policymaking [2], [3]. However, manually evaluating the evidence underlying a claim is time-consuming and resource-intensive, especially given the growing volume of online information [4]. For instance, verifying a single claim can require a full day of work for a journalist, while more complex investigations may take several days [5]. Consequently, automated fact-checking (AFC) has emerged as an active area of research [6], with approaches ranging from verification over fixed corpora to open-domain systems that dynamically retrieve evidence from the web.

Recent advances in LLM-based retrieval-augmented generation (RAG) systems have further broadened the scope of AFC [7], [8]. However, research on the trustworthiness of LLM-based systems has largely focused on issues such as prompt security, adversarial attacks, and hallucinations [9], [10]. Comparatively less attention has been paid to the reliability of the external information sources on which retrieval-based systems depend. That is, retrieved evidence is often treated at face value, despite the possibility that sources may be conflicting, outdated, incomplete, or intentionally misleading [11], [12]. This raises the risk of attacks-by-content [13], whereby misleading or manipulative information propagates through retrieval pipelines and influences downstream model outputs. In the context of AFC, one promising direction for explicitly considering source credibility is the use of media background checks (MBCs) [1]. MBCs are summaries of aspects that may affect perceptions of a source’s trustworthiness, including political orientation, factual reliability, editorial practices, and ownership (e.g. 1). By providing contextual information about evidence sources, MBCs can help both humans and automated systems make more informed judgements about retrieved information. However, existing approaches to MBC generation typically rely on proprietary search engine APIs to retrieve source information at inference time, introducing financial costs, limiting reproducibility, and creating variability as search engine rankings and retrieval systems evolve.

To overcome these limitations, we introduce MediaRef, a collection of web-sourced documents covering 200 news sources that supports MBC generation. By decoupling information retrieval from MBC generation, MediaRef provides a reproducible, low-cost alternative to proprietary search APIs while reducing variability arising from changes in retrieval behaviour over time. We describe a systematic and reproducible methodology for constructing the resource, enabling the collection to be kept up to date as web content evolves. We use MediaRef to evaluate the MBC-generation capabilities of several LLMs and examine the impact of additional source-related evidence on generation quality. We find that access to MediaRef produces MBCs that contain more accurate information about news sources without increasing the incidence of misleading content. We further develop a qualitative evaluation framework for MBCs based on four criteria—clarity, relevance, informativeness, and verifiability—motivated by users’ need to assess the trustworthiness of MBCs. This analysis complements prior work on the usefulness of MBCs [1] by identifying, at a finer level of granularity, which characteristics contribute to their utility and when MBCs may fall short across these dimensions.

2 Related Work↩︎

AFC is typically divided into three stages: claim detection, which identifies claims to fact-check; evidence retrieval, which gathers information to support or refute a claim; and claim verification, which determines the overall truthfulness of a claim [6]. Earlier AFC datasets [14], [15] relied on evidence from a single closed source, typically Wikipedia, assuming the source to be trustworthy. However, this risks propagating biases present in the underlying source [16]. More recent work instead retrieves online evidence dynamically during model runtime using search APIs [17]. Further, contemporary systems use retrieval-augmented generation (RAG) to retrieve evidence and generate human-understandable justifications that explain how the retrieved evidence supports a verdict [8]. Professional fact-checkers report that such justifications enable audiences to scrutinise the reasoning process, thereby promoting trust [3].

Despite these advances, dynamically retrieving online evidence remains constrained by the financial cost of search APIs [8], potentially limiting both future research and practical adoption by resource-constrained fact-checking organisations [18]. To reduce this barrier, shared tasks such as AVeriTeC [8], AVeriTeC 2.0 [19], and AVerImaTeC [20] have provided participants with stores of pre-retrieved online evidence, avoiding the need for costly API calls. Participating teams have reported additional benefits, including improved reproducibility and robustness against the unexpected disappearance of online information over time [21]. However, retrieved evidence is often treated as credible based on proxy signals such as search ranking position, despite no guarantee that highly ranked sources are free from misinformation. If left unresolved, conflicting evidence can substantially degrade the performance of RAG systems [11], [12]. To address this issue, [1] introduced Media Background Checks (MBCs) for AFC. They proposed a two-stage RAG pipeline in which evidence describing characteristics of a source associated with (in)credibility is first retrieved and then used by an LLM to generate an MBC. Building on this work, [12] explored how MBCs could be integrated into AFC pipelines and examined their impact on the fact-checking process through qualitative analysis. In this paper, we build on this line of work by introducing MediaRef, a curated knowledge store for media background checks that enables reproducible evidence access across multiple LLMs without reliance on live search APIs. We further propose a qualitative analysis that goes beyond assessing the utility of MBCs alone [1], and instead examines fine-grained characteristics that affect their perceived utility.

3 MediaRef Creation↩︎

3.1 Media Background Checks↩︎

Media background checks (MBCs) (e.g., 1) are textual descriptions of media outlet characteristics that may affect the credibility of their reporting. These characteristics include, for example, an outlet’s history, ownership, political bias, and prior fact-checking record. MBCs are supported by URLs linking to external evidence sources, such as relevant news articles or other MBCs, which readers can consult to assess their reliability. Beyond supporting human readers, MBCs provide a signal of source credibility that can be incorporated into downstream systems, such as automated fact-checking models that rely on web-sourced information, helping them prioritise trustworthy evidence over misinformation.

3.2 Dataset Source↩︎

We use Media Bias/Fact-Check (MB/FC)³ as a source of gold-standard MBCs. MB/FC is an independent US-based organisation that aims to equip news consumers to identify and defend against misinformation by “understanding the bias and credibility of the sources they consume”. Their MBCs are written by volunteers following a published methodology⁴ and cover outlets ranging from small local newspapers to international organisations. While MB/FC is frequently used as a ground-truth source in studies of source credibility [22], [23], these studies generally use only the credibility and political bias ratings provided by MB/FC, rather than the MBC text itself. Hence, we use the MB/FC dataset released by [1], from which we randomly sample 200 outlets from the test split.

3.3 Online Evidence Retrieval↩︎

To gather background information for each news outlet required to generate an MBC, we construct a set of search queries targeting key aspects of a news outlet’s background, including funding, ownership, bias, endorsements, and general descriptions. The queries follow the pattern ‘‘source’’ news <topic>, where source is replaced by the outlet name and <topic> is one of funding, about, ownership, endorsement, bias. To minimise off-topic results, all queries include the keyword news and require an exact match to the outlet name. For each outlet, we then submit all queries to the Google Search API using default parameters and collect the top 10 results per query. Our retrieval process can be replicated to obtain more recent sources.

3.3.0.1 Quality Control

To ensure sufficient coverage for reproducing the gold MBCs provided by [1], we remove duplicate URLs and additionally include all external URLs cited in the gold MBC texts, as in [8], merging them with the retrieved URLs. Furthermore, to reduce the risk of data leakage, we remove URLs pointing to webpages likely to reference the gold MBCs. We exclude all URLs matching a blacklist of websites known to cite MB/FC content, in line with [1].

We then scrape the remaining webpages using the open-source web-scraping tool trafilatura [24]. This results in 21,921 non-empty documents, which form the final knowledge store.

4 Experimental Setup↩︎

We investigate the impact of using MediaRef as a source of evidence for LLM-generated MBCs. We experiment with models from OpenAI [25]–[27], Qwen [28], [29], Llama [30], Mistral [31], and Anthropic [32]. This selection includes both state-of-the-art proprietary systems and smaller open-source alternatives that are more accessible to researchers and fact-checking practitioners, allowing us to assess current LLM capabilities on the MBC generation task.

4.1 Models↩︎

We compare MBCs generated with and without information retrieval and evaluate the resulting differences in quality using two prompting strategies described below. The full prompts are provided in 8.

4.1.0.1 Prompting Without Information Retrieval

An MBC is generated in a single LLM call without access to external information. Given the name of a news outlet, the model is prompted in a zero-shot setting to generate a brief, itemised summary of information associated with the outlet.

4.1.0.2 Prompting With Information Retrieval (+IR)

First, an initial MBC is generated using the zero-shot prompting strategy described above. The MBC is then refined using evidence retrieved from MediaRef through the following process.

(1) Retrieval. Relevant documents are retrieved using BM25 [33]. We define six keyword-based queries targeting recurring topics in MBCs, such as outlet history and funding (see 9), and retrieve the top 30 highest-scoring documents for each query.

(2) Evidence Extraction. The retrieved documents are filtered to identify passages likely to contain credibility-relevant information. Each keyword query is reformulated as a question; for example, “source name” funding becomes How is “source name” funded?. A DeBERTa-based question-answering model [34] is then used to extract answers from the retrieved documents. If an answer is found, the corresponding text span is retained as evidence.

(3) MBC Update. The extracted evidence is used to update the initial MBC. Document snippets are processed individually, and the MBC is revised iteratively after each snippet to reduce the risk of exceeding the model’s context window. The model is instructed not to remove existing points, thereby minimising information loss.

4.2 Evaluation Metrics↩︎

To compare generated and gold-standard MBCs [1], we use the following metrics:

• ROUGE-L which evaluates the longest common subsequence of words between the generated and reference texts [35].

• METEOR which uses stemming and synonym matching to evaluate word-level overlap [36].

• Fact Recall and Error Rate [1] based on the FActScore framework [37]. A GPT-3.5-Turbo model decomposes a gold-standard MBC into atomic facts using a gap-filling template (e.g. The usual audience of “source name” is __; see 10). It then verifies whether each extracted fact is entailed, contradicted, or neither by both the generated and gold-standard MBC texts.

  • Fact Recall is the proportion of atomic facts for which the generated MBC and the gold-standard MBC agree on entailment.

  • Error Rate is the proportion of atomic facts for which the generated MBC and the gold-standard MBC disagree on entailment.

5 Results↩︎

5.1 Automatic Evaluation↩︎

We report the experimental results in 1. Using gpt-3.5-turbo, our system achieves fact recall and error rate scores comparable to the baseline results reported by [1] on the same model (28.2% vs.% fact recall and 8.1% vs.% error rate, respectively). This suggests that using MediaRef provides a reliable proxy for model performance on recent evidence. Among the evaluated models, gpt-4o-mini achieves the highest fact recall score, followed by llama-3.3.

Overall, the results suggest that MBC generation remains challenging, with relatively low fact recall across all models. Notably, state-of-the-art systems such as gpt-5-mini do not substantially outperform smaller open-source models, including llama-3.3 and mistral-7b, in accurately reflecting facts about news sources. Error rates are also broadly similar, although gpt-5-mini exhibits a noticeably lower error rate than the other OpenAI models. According to [27], GPT-5 was trained to reduce factual errors in RAG settings, for example by minimising hallucinations, which may explain this improvement. Importantly, higher fact recall does not necessarily correspond to lower error rates, highlighting the need to evaluate both the amount of correct and incorrect information generated. That is, focusing exclusively on factual content risks overlooking errors that may subsequently distort credibility judgements about news sources.

Introducing information retrieval generally improves fact recall, indicating that evidence retrieved from the knowledge store helps extend a model’s knowledge of a news source. However, the magnitude of this gain, as well as changes in token count, vary substantially across LLMs. Since evidence retrieval is independent of the choice of LLM, these differences suggest that models differ in how effectively they incorporate retrieved information into their MBCs. We qualitatively investigate the potential causes of this variation in the next section.

5.2 Human Evaluation of MBC Generation Quality↩︎

We complement our automatic evaluation with a human evaluation to better understand why models may struggle to reflect the information in the gold-standard references. In addition, since MBCs are intended to support source criticism, we assess whether human readers perceive them as providing useful and verifiable information.

We define four criteria for evaluation: clarity, relevance, informativeness, and verifiability. The first three are adapted from the criteria for fact-checking questions proposed by [38]. We introduce verifiability to capture whether an MBC helps users assess credibility by pointing to additional trustworthy sources [3]. All criteria are rated on a Likert scale from 0–3, except for clarity, which is rated from 0–2. A full description of the annotation scheme is provided in 11.

5.2.0.1 Clarity

Individual points should be comprehensible both on their own and in the context of the full text, and the MBC should not contain inconsistencies or self-contradictions.

5.2.0.2 Relevance

All facts mentioned in the MBC should relate to the target news source or closely connected entities and topics.

5.2.0.3 Informativeness

The MBC should provide information that meaningfully influences a reader’s perception of a source’s overall credibility. Highly informative background checks directly indicate the source’s credibility and political bias (or absence of bias), and justify why. Weakly informative checks only describe characteristics loosely relevant to source credibility, such as general background information, with unclear implications for credibility.

5.2.0.4 Verifiability

Sufficient evidence should be provided to support factual claims, or enough detail should be included so that one can reasonably locate the evidence independently. This should be determined based solely on judgements of how easily the facts could plausibly be verified (i.e., without the use of a search engine or an LLM).

5.3 Human Evaluation Setup↩︎

We randomly sample 108 LLM-generated MBCs (27 per model) from the gpt-4o-mini, gpt-5-mini, llama-3.3-70b-instruct, and mistral-7b-instruct IR-augmented models (i.e., +IR in 1) to investigate how information retrieved from the knowledge store affects the quality of the generated MBCs. The LLMs represent both state-of-the-art proprietary systems and open-source models across a range of parameter sizes and performance levels.

Two expert annotators (authors of this paper) qualitatively assess the clarity, relevance, informativeness, and verifiability of the MBCs. The identity of the system that generated each MBC is concealed from the annotators, and they are not permitted to use external sources such as search engines or LLMs to obtain additional background information about the MBCs.

5.3.0.1 Agreement Scores

The Fleiss-\(\kappa\) scores for clarity, relevance, informativeness, and verifiability are 0.51, 0.88, 0.87, and 0.79, respectively. The Krippendorff-\(\alpha\) scores are 0.52, 0.98, 0.95, and 0.91, respectively. However, as [39] demonstrates, measures such as Fleiss-\(\kappa\) assume that the distribution of assignments across score categories is fixed in advance (i.e., chance-adjusted), which is often unrealistic in practice. We therefore also report free-marginal multi-rater kappa scores, which are 0.90, 0.93, 0.88, and 0.81 for clarity, relevance, informativeness, and verifiability, respectively.

Overall, agreement scores are high, with informativeness and verifiability being slightly lower. This is expected, as these two criteria are inherently more subjective and depend on a reader’s interpretation of how an MBC affects their understanding of a news source.

6 Analysis↩︎

6.1 To what extent do generated MBCs capture source credibility?↩︎

Overall, the results of our qualitative analysis are consistent with those of our automatic evaluation. Specifically, gpt-4o-mini performs best at generating fully relevant MBCs, while llama-3.3-70b-instruct produces the most informative and verifiable outputs. In contrast, gpt-5-mini performs worst in terms of relevance, informativeness, and verifiability. Open-source models nevertheless remain competitive with state-of-the-art proprietary models.

3 shows that all models generally produce clear and relevant MBCs. However, generating informative and verifiable content is significantly more challenging. While entirely uninformative or unverifiable MBCs (i.e., those assigned a score of 0) are rare, the scores indicate that informativeness and verifiability are harder to achieve than clarity and relevance. In particular, verifiable points must first be clear, while informative points must be both clear and relevant.

For example, consider the following excerpt from a gpt-4o-mini-generated MBC for Daily Surge:

Although the model identifies a potentially right-leaning political bias, it does not specify which right-leaning perspectives the source aligns with, making the claim difficult to verify.

This observation is further supported by the results shown in 4, which illustrate the Spearman correlation scores between the different criteria. Notably, we find that informative MBCs tend to be verifiable (\(\rho = 0.83\)), suggesting that providing specific evidence in an MBC may supply sufficient detail to bridge gaps in a reader’s perception of the source, thereby improving verifiability scores.

6.2 Challenges in updating MBCs↩︎

6.2.0.1 Insufficient Information

When models cannot source information about a news outlet either internally or from the knowledge store, they tend to generate weakly informative background descriptions or no useful information at all. For example, mistral-7b-instruct describes The Jackson Sun in a way that is too vague and generic to meaningfully inform an assessment of source credibility:

Local newspapers appear to be particularly susceptible to this issue. Among the 34 local news outlets in our sample, 71% received an informativeness score of at most 1, while 80% received a verifiability score of at most 1. In many cases, the generated MBCs contained only basic background information similar to the example above. This may be due to the fact that these outlets often lack the prominence required to attract substantial external scrutiny, such as independent credibility assessments or fact-checking coverage, despite potentially being trustworthy sources for local issues. Consequently, models may be unable to provide evidence either supporting or challenging their credibility. For instance, the same mistral-7b-instruct-generated MBC includes factually plausible but largely uninformative statements about the outlet’s credibility:

6.2.0.2 Inconsistent Emphasis on Evidence

Models differ substantially in the extent to which they interpret or evaluate the evidence they retrieve. In 8 of 27 MBCs, gpt-5-mini draws inferences about the relevance of evidence to source credibility, including unprompted reasoning about the source itself. By contrast, the other models rarely include content evaluating whether the available evidence sufficiently supports a conclusion (across 27 MBCs: llama-3.3-70b-instruct and mistral-7b-instruct do so in one instance each, and gpt-4o-mini in none). For example, for the Source New Mexico news outlet, gpt-5-mini generates:

Providing readers with information about the quality and limitations of the available evidence may help them form a more nuanced assessment of a source’s credibility and is broadly consistent with professional fact-checking practices. However, such commentary may also introduce an additional interpretative layer that risks influencing readers’ perceptions of the evidence. Note that this additional reasoning substantially increases the length of the MBCs, making them harder to read and understand. Specifically, median MBC word counts are 458 for gpt-5-mini, compared with 194 for gpt-4o-mini, 268 for mistral-7b-instruct, and 264 for llama-3.3-70b-instruct.

6.2.0.3 Information Loss

In seven MBCs generated by mistral-7b-instruct, the model omits information from the beginning of the MBC during the updating process. For example, in the MBC for KJRH – Tulsa News, the first nine points, which describe the outlet’s location, affiliation, ownership, and funding, are replaced with:

As these early points typically contain important background information about the outlet, omitting them reduces the informativeness of the final MBC. This issue could be mitigated by explicitly instructing the model to regenerate and verify the complete updated MBC after each revision.

6.2.0.4 Conflicting Information

We observe three cases (out of 108) of explicit contradictions in generated MBCs: two from llama-3.3-70b-instruct and one from gpt-5-mini. Such contradictions reduce informativeness, as they prevent readers from drawing reliable conclusions. In two cases, contradictions arise when models update an existing MBC with newly retrieved information. For example, when describing fact-checking failures associated with WBTS – NBC 10 Boston, llama-3.3-70b-instruct initially generates “None notable found” but later adds:

One possible explanation is the prompt design, which discourages removing previously generated content during updates. While this helps preserve earlier information, it may also allow outdated or superseded statements to remain in the MBC when later retrieved evidence introduces conflicting information. This limitation could be addressed by explicitly prompting the model to revise or remove statements that are contradicted by newly retrieved evidence.

7 Conclusion↩︎

We introduced MediaRef, a publicly available knowledge store designed to support the generation of media background checks (MBCs). We described a reproducible methodology for constructing and updating the resource, and evaluated several widely used LLMs on the MBC generation task. Our experiments show that MediaRef improves the quality of generated MBCs, while our human evaluation highlights that producing concise, informative, and verifiable MBCs remains an open challenge. We publicly release MediaRef to support reproducible research on source-critical reasoning and automated fact-checking.

Limitations↩︎

We acknowledge that search-engine rankings may introduce bias as a proxy for relevance in evidence selection, potentially affecting the quality and diversity of retrieved sources. Nevertheless, our resource provides a strong starting point that can be extended in future work to incorporate more diverse and heterogeneous data sources.

Second, although we use a blacklist to filter sources, some undesired or low-quality sources referencing MBCs or fact-checks may still be included, as no blacklist can be fully comprehensive.

Third, the system is limited to information freely available on the web and therefore cannot access restricted or proprietary sources, such as official statistical databases or paywalled archives, highlighting an opportunity for future integration of broader external sources.

Ethical Considerations↩︎

Our resource is designed to support media background checks using publicly available information. However, despite the use of retrieval, generated outputs may still contain inaccuracies or misrepresentations of source material and should therefore not be treated as definitive judgments about the sources, especially in high-stakes settings.

Second, the reliance on web search and publicly available content introduces inherent biases, including disparities in coverage across regions, languages, and entities. Such biases may lead to uneven or incomplete representations in generated background checks. In addition, while we use filtering mechanisms such as blacklists, we do not guarantee the complete removal of low-quality or unreliable sources.

Finally, although the system operates solely on publicly accessible information, aggregating and synthesising such data may still raise dual-use concerns. The intended use of our resource [40] is to support research on automated fact-checking and potentially assist fact-checkers with human experts in the loop. We emphasise the need for cautious deployment and human oversight in real-world applications.

7.0.0.1 AI Use

We use privacy-preserving models to assist with proofreading and coding tasks (e.g., generating plots), in line with the ACL guidelines.

Appendix↩︎

8 Prompts↩︎

We give the prompts used to generate, update, and evaluate MBCs. They are reproduced from [1].

9 Search Queries for Evidence Retrieval↩︎

2 gives the search queries used to retrieve evidence from MediaRef when generating MBCs with information retrieval.

10 Atomic Fact Templates↩︎

3 gives the 42 atomic facts used to evaluate the fact recall and error rate metrics.

11 Qualitative Annotation Guidelines↩︎

The four criteria are defined as follows.

11.0.0.1 Clarity:

Background checks should be clear and understandable to an average layperson.

• 0 – Poor clarity: The MBC’s language or format severely hinders comprehension. Most points are unclear in meaning. Examples include ungrammatical text that impedes understanding, omitted information necessary to interpret later points, or contradictory statements regarding source credibility.

• 1 – Limited clarity: Most points are understandable, but language or formatting occasionally hinders comprehension.

• 2 – High clarity: All points are clear and understandable.

11.0.0.2 Relevance:

All facts mentioned in the check should relate to the target source or closely connected entities and topics.

• 0 – Off-topic: The MBC as a whole is unrelated to the target source.

• 1 – Partially relevant: Some, but not all, points refer to unrelated entities or ideas and are off-topic.

• 2 – Broadly relevant: Some points refer to entities or ideas not directly related to the source but still relevant to the broader topic.

• 3 – Fully relevant: All points are directly related to the target source and remain on topic.

11.0.0.3 Informativeness:

Facts about the target source should influence the reader’s understanding of the source’s general credibility and provide context relevant to evaluating claims. Because an MBC evaluates a source overall rather than a specific claim, highly specific facts are less useful.

• 0 – Uninformative or contradictory: Points do not indicate source credibility, or conflicting information prevents conclusions from being drawn confidently.

• 1 – Weakly informative: Points describe characteristics only weakly related to source credibility (e.g.general background information), with unclear implications for credibility.

• 2 – Moderately informative: At least one point describes source characteristics relevant to credibility, although their implications are not fully explicit or apply only to limited claim types. No contradictory points are present.

• 3 – Highly informative: At least one point directly indicates the credibility and political bias (or absence of bias) of most claims from the source and explains why. No contradictory points are present.

11.0.0.4 Verifiability:

Sufficient evidence should be provided to support factual claims, or enough detail included that a layperson could reasonably locate the evidence independently. Annotators should not conduct additional research and should base judgements solely on the ease with which facts could plausibly be verified.

• 0 – Unverifiable: Points are entirely unverifiable, such as broad subjective assertions.

• 1 – Limited verifiability: Points provide sufficient detail for further investigation, but omit important contextual information needed to assess them independently.

• 2 – Moderately verifiable: Points are sufficiently detailed to appear credible and be verifiable by a layperson, although verification would require some effort.

• 3 – Easily verifiable: Specific evidence is provided that a layperson could quickly and easily verify.

References↩︎