1 Introduction↩︎

Artificial intelligence (AI) is playing an expanding role in public health, from chatbots that answer health queries [1] to decision-support tools for public health professionals [2]. Large language models (LLMs) can already generate coherent answers based on extensive training data, in some cases approaching expert performance on medical question answering tasks [3]. This potential has spurred interest in deploying LLMs in public health, both as public-facing tools and as decision aids for public health professionals. Unlike many clinical decision-support settings—where tools are designed to support decisions for an individual patient at the point of care, public health guidance is population-level, often precautionary, and closely tied to official recommendations that are updated as evidence and policy evolve [4]–[6]. However, LLMs can hallucinate information or provide outdated advice, which in public health can have severe implications: even small inaccuracies may harm individual health decision-making [7] and, at scale, drive inappropriate or dangerous behaviours across populations [8]. Therefore, ensuring that LLM responses are reliable and up to date is a prerequisite for safe AI adoption in public health.

One established approach to improving LLM performance and reliability is Retrieval-Augmented Generation (RAG) [9]. In RAG systems, an LLM is coupled with an external retrieval component that selects relevant documents from a knowledge base; the model then conditions its output on this retrieved context rather than relying solely on its parametric memory [10], sharma2025ragsurvey?. By incorporating retrieval, an LLM can expand and update its effective knowledge beyond what is stored in its frozen parameters, mitigating hallucinations and reducing the impact of outdated training data ni2025trustworthyragsurvey?. In high-stakes fields such as public health, the ability to provide answers grounded in trusted evidence is especially valuable. RAG systems have been shown to substantially improve performance on knowledge-intensive tasks such as medical and guideline-based question answering compared with LLMs that rely only on internal knowledge [11], [12], fernandezpichel2025searchenginesllmshealth?.

With the increasing adoption of LLM systems in use cases involving public health or medical queries, it is crucial to measure their effectiveness on tasks that reflect real information needs of both the public and professionals [13], [14], ventura2025healthqa-br?. To this end, a variety of benchmarks for medical Question Answering (QA) have been developed, though most existing benchmarks focus on clinical or biomedical questions rather than public health guidance. To understand this issue for UK Government public health information Harris et al. evaluated LLMs on a range of public health classification and extraction tasks [15] before introducing PubHealthBench [1], a QA benchmark with nearly \(8{,}000\) questions derived from official UK public health guidance.

In this follow-up work, we explore RAG as a solution to improve LLM performance on public health QA. We extend our benchmark setup by allowing models to retrieve information from the same underlying corpus of UK public health guidance from which the questions were generated [1]. Specifically, we use PubHealthBench to address four questions. (1) How effective are different retrieval setups, spanning embedding models and system design choices, at identifying the most relevant chunks of public health guidance given questions about that guidance? (2) Does introducing retrieval of trusted documents improve LLM performance on multiple-choice QA (MCQA), and how does this depend on retrieval quality and context configuration? (3) To what extent do the observed MCQA improvements generalise to free-from responses? (4) Can we develop an automated evaluation framework to assess LLM responses that aligns with human expert decisions? By answering these questions, we aim to demonstrate how coupling LLMs with a corpus of public health guidance can yield more reliable and informed responses on knowledge-intensive tasks.

2 Related Work↩︎

2.1 Retrieval-Augmented Generation Foundations↩︎

RAG combines external knowledge retrieval with large language model (LLM) generation to enhance factual accuracy and mitigate hallucinations [9], [11]. For example, the RETRO model demonstrated that pairing a smaller generator with a large retrieval data store can match much larger LLM-only systems [16], and subsequent work has shown that retrieval scale and quality are critical performance levers [17]. Recent surveys provide up-to-date taxonomies of RAG architectures [10], [18], sharma2025ragsurvey? and examine trust, safety, fairness, and accountability in retrieval-augmented systems ni2025trustworthyragsurvey?. Together, these works highlight retrieval as a distinct driver of performance and underscore the challenge of reliably integrating retrieved knowledge, particularly in safety-critical domains such as medicine and public health.

2.2 Retrieval Methods and Context Handling↩︎

Modern retrieval systems largely rely on dense semantic embeddings to map text into vector space representations, with trade-offs between model size, retrieval accuracy, and latency [19]. Hybrid retrieval methods that combine dense embeddings with sparse models such as BM25 often boost retrieval precision and recall [20]. Re-ranking models (e.g., GTE-ModernColBERT) and knowledge-graph-based retrieval further support retrieving relevant information in domain-specific settings [21]–[23]. Importantly, long context windows change but do not remove the need for careful retrieval and context selection [24], [25]. There is also extensive empirical work evaluating retrieval design choices, such as, query expansion, chunking, memory-based architectures; underlining how there are several configuration choices that can impact retrieval, and downstream QA, performance [26], [27].

2.3 Benchmarking and Evaluation in Health QA↩︎

In medical and public-health domains, multiple-choice QA benchmarks (e.g., MedMCQA, MedQA) have provided standard evaluation frameworks for assessing clinical knowledge [13], [28], [29]. However, multiple-choice formats can overestimate model competence compared with free-form generation tasks and may mask important reasoning failures [30]. For example, Singh et al.report an average drop in accuracy of 39.4% when moving from MCQA to free-from answers for LLMs (vs.% for humans). They also show that even with the question stem fully masked, MCQA accuracy remains above chance (6.7% above random on average), suggesting MCQA performance can be partly driven by answer-option cues rather than underlying understanding [30]. Consequently, newer benchmarks have shifted toward free-form responses to better simulate real-world usage [31], [32]. In public health, studies show substantial knowledge gaps and uneven performance across domains: AfriMed-QA reports that GPT-4o attains \(79\%\) overall accuracy on expert MCQs, but performance varies substantially by specialty—ranging from \(>90\%\) in top specialties (e.g., rheumatology) to \(<60\%\) in weaker areas such as pediatrics and obstetrics–gynecology [33], illustrating strong topic dependence despite high aggregate QA accuracy, and PubHealthBench finds that while the strongest models achieve \(>90\%\) accuracy (\(92.5\%\) for GPT-4.5) in MCQA, exceeding the \(88\%\) human baseline, but no model scored higher than 75% in free-form QA[1].

Evaluation methods are also evolving. Overlap metrics such as ROUGE or BLEU correlate poorly with human judgments in free-form QA [34], [35]. The use of LLMs as automated evaluators (“LLM-as-Judge”) has emerged as a scalable alternative, with specialised models such as Prometheus and its successor showing strong alignment with human assessments across a range of generation tasks [36]–[39]. In health-specific settings, rubric-guided LLM-as-judge frameworks that combine physician-designed rubrics with model graders have achieved high agreement with clinician ratings [14], [40]. A recent comparative study of search engines, LLMs, and RAG variants on health questions further demonstrates retrieval’s substantial impact on QA accuracy and highlights the benefits of grounding answers in retrieved evidence fernandezpichel2025searchenginesllmshealth?. Together, these developments underscore the need for benchmarks that jointly assess retrieval quality and generation performance in health QA systems, especially when systems are expected to align with official public health guidance.

3 Methods↩︎

3.1 Benchmark and Dataset↩︎

We use the PubHealthBench QA benchmark, which contains \(7,929\) multiple‐choice questions derived from 687 UK Government public health guidance documents covering 10 topic areas [1]. The source documents were converted to markdown and split into chunks by markdown header levels, with full header hierarchy appended for context. This produced \(5,358\) chunks, which form the retrieval corpus.

max width=

3.2 Retrieval↩︎

We adopt five retrieval methods for our RAG pipeline:

1. Embedding‐based retrieval: Each chunk is encoded into a dense embedding via a text‐embedding model, and queries are encoded with the same model. Retrieval ranks chunks by cosine similarity between query and chunk embeddings. We evaluate eight embedding models (e.g., NV-Embed-V2 [41], EmbeddingGemma [42], ModernBertBase [43], SFR-Embedding-Mistral [44], Multilingual-E5-large [45], OpenAI’s text-embedding-3-large [46]).

2. Keyword-based retrieval: We build sparse indices using the term frequency algorithms TF-IDF and BM25. Retrieval ranks chunks by the overlap of key terms between the query and chunks.

3. Hybrid retrieval: We perform both embedding-based and keyword-based retrieval as in (1) and (2), and then merge the two ranked lists using a weighted Reciprocal Rank Fusion (RRF): \[R(d;\alpha) = \alpha\,\frac{1}{c + r_{\mathrm{dense}}(d)} + (1-\alpha)\,\frac{1}{c + r_{\mathrm{sparse}}(d)} \label{eq:rrf}\tag{1}\] Where \(\alpha\) is a weighting factor, we set \(c = 60\) consistent with the value used in foundational RRF work [47].

4. Summary‐based hybrid retrieval: Each chunk is summarised using GPT-4o‐mini [48]. Hybrid retrieval (as in (3)) is applied to the summary corpus rather than full chunks. This variant tests whether more compact context improves retrieval precision in the public-health domain. Summaries are used only for retrieval and the corresponding full chunk is passed down-stream for generation.

5. Reduced-corpus hybrid retrieval: Hybrid retrieval (as in (3)) is applied to a reduced corpus, where any chunk longer than 512 tokens (for a given embedding model’s tokenizer) is replaced with its generated summary (the 512-token cutoff aligns with the context capacity of Multilingual-E5 [45]). As with (4), only the corresponding full chunk is passed down-stream for generation.

3.3 Generation↩︎

The top‐\(k\) ranked chunks from retrieval are appended (with a chunk identifier and separator) and concatenated into a context block. Following the prompt structure of Harris et al. [1], we insert the question and the context block, with a simple instruction to make use of the inserted context when answering. We assess 11 LLMs (e.g., Llama 3.3 [49], Phi-4 [50], Gemma-3 [51], MedGemma [52], CommandR [53]) on the benchmark subsets shown in Table ¿tbl:tab:pubhealthbench95subsets?.

3.4 Experimental Setup↩︎

3.4.1 Retrieval Setup↩︎

Retrieval is run with each embedding model over all \(7,929\) queries on three corpora: \(\mathcal{C}_F\) (full corpus), \(\mathcal{C}_S\) (summary-only corpus), and \(\mathcal{C}_R\) (reduced corpus). For model and each hybrid retrieval configuration we find the the value of the weighting factor \(\alpha \in \{0.50, 0.55, 0.6, ..., 0.95\}\) that results in the best retrieval performance, where higher \(\alpha\) means heavier weighting for embedding-based ranking (see Equation 1 ). We report the optimal retrieval setup (method, corpus, \(\alpha\)) for each model.

3.4.2 MCQA Setup↩︎

For PubHealthBench-Full, we extract top-\(k\) chunks with \(k \in \{1,3,5,10\}\) from three representative retrieval configurations (NV-Embed-v2 (\(\mathcal{C}_R\)), Multilingual-E5-large-instruct (\(\mathcal{C}_R\)), ModernBertBase (\(\mathcal{C}_F\))). These models are chosen to test the impact of retrieval quality on MCQA accuracy, as they span a range of model sizes and retrieval performance. We report MCQA accuracy on PubHealthBench-Full, alongside accuracy without retrieval of the human baseline and GPT-4.5 [1].

3.4.3 Free-form Setup↩︎

In a large real world guidance corpus to achieve near perfect (\(\sim 100\%\)) retrieval recall, we may require a large number of retrieved chunks, so for free form responses we use the same three configurations for PubHealthBench-FreeForm, set \(k=15\), and limit the context block to \(10,000\) tokens. Through this setup we evaluate LLMs in a more realistic setting, where they are challenged to precisely extract relevant information from a large amount of context, and use this to clearly and accurately answer a public health query, without guiding answer options. In this setup, we evaluate free-form answers using an LLM judge (see § 3.5).

3.5 Evaluation↩︎

3.5.1 Retrieval Evaluation↩︎

Because each query has exactly one target chunk, we report the following metrics:

• Recall@\(k\): Proportion of queries whose relevant chunk appears within the top \(k\). Reflects coverage of retrieval but all rankings in the top \(k\) are treated equally.

• Mean Reciprocal Rank (MRR): The average of \(1/r\) across queries, where \(r\) is the rank of the relevant chunk. Measures how early the correct chunk appears in the ranking.

• Normalized Discounted Cumulative Gain (nDCG@\(k\)): Assesses ranking quality via log-discounted gain, penalising relevant chunks appearing lower in the ranking [54], [55]. This provides a finer-grained view of rank quality.

• Precision@1: Because there is exactly one relevant chunk per query, we report Precision@1 (1 if the chunk is ranked first, else 0) [56].

Together, these metrics allow us to evaluate whether the correct chunk is retrieved at all (Recall@\(k\)), how early it appears (MRR), and how well the ranking order is structured (nDCG).

3.5.2 Free-form Evaluation↩︎

To evaluate the answers for PubHealthBench-FreeForm, we follow HealthBench in applying a rubric-based LLM-as-a-Judge to evaluate free form responses [14]. Through this we aim to create a framework for automated evaluation of free-form responses to public health queries. In our setup, each answer is assessed by the GPT-OSS-120B model, using a structured set of four criteria on factual consistency, completeness, clarity, and faithfulness to official guidance. The criteria are designed to be non-overlapping and objective, assessing an LLMs ability select only relevant information from a large amount of context, and use this to accurately and clearly answer a public health query. The criteria definitions are shown in in Table 1. To support reliable analysis, two human expert reviewers independently annotated the same sample of 100 LLM responses against each criterion. We report Cohen’s \(\kappa\) and macro-F1, with 95% bootstrap confidence intervals for the agreement between the reviewers and each reviewer with the LLM judge. All judgments are made with respect to the single relevant chunk of guidance, along with a reference answer, question and LLM answer, and detailed instructions for each criteria, aligning with HealthBench’s emphasis on grounded, context-aware evaluation [14].

4 Results↩︎

4.1 Retrieval↩︎

We find that the retrieval performance of embedding models varies considerably on the task of retrieving relevant public health information. In dense-only retrieval configuration, the best performing model, NV-Embed-v2, achieves the best results across all metrics (\(0.98\) Recall@10, \(0.85\) MRR, \(0.88\) nDCG@10), a marked gain over other models tested; for example, EmbeddingGemma scores \(49\) ppts lower on Precision@1. Retrieval quality generally improves with model size and embedding dimension, yet sparse-only retrieval using TF-IDF or BM25 outperforms some embedding models. For every model, a hybrid setup improves performance across all retrieval metrics; the optimal fusion weight \(\alpha\) in our RRF setup is model-dependent. Notably, hybrid search narrows the gap between smaller encoders (multilingual-E5-large, ModernBERT-base), which see a \(5\)–\(10\) ppt increase across metrics, and NV-Embed-v2, despite a factor of \(10\) difference in parameter count. No model attains its best results on the summary-only corpus \((\mathcal{C}_S)\); peak scores arise on either the reduced \((\mathcal{C}_R)\) or full \((\mathcal{C}_F)\) corpora, and this preference is not explained by context length alone.

max width=

max width=

Figure 1 shows retrieval performance declines across models as target chunk length increases. Recall decreases only modestly, but rank-sensitive metrics (Precision@\(k\), MRR, nDCG@\(k\)) drop sharply once chunks exceed \(700\)–\(800\) words. Using \(\left(\mathcal{C}_R\right)\) somewhat attenuates this decline relative to \(\left(\mathcal{C}_F\right)\), although it does not mitigate it completely. As target length grows, the performance gap between larger models (NV-Embed-v2, SFR-Embedding-Mistral) and smaller models widens.

In Figure 2, we observe topic-level deviations in ranking quality across embedding models, measured as each topic’s departure from the model’s mean nDCG@10. A one-way ANOVA indicates that guidance topic explains less than 1% of the variance in nDCG@10 across models (\(\eta^{2} \approx 0.006\)–\(0.008\)). Although this global effect size is small, topic means show systematic differences in ranking quality. The gap between lower-performing topics (e.g., Blood Safety, Hepatitis, STIs and HIV) and the highest-performing topic (Tuberculosis, Travel, Zoonotic and Emerging Infections) is approximately \(0.06\)–\(0.08\) nDCG@10. The relative ordering of higher- and lower-performing topics is broadly consistent across models.

In contrast, Recall@5 shows little evidence of topic dependence (\(\eta^{2} \approx 0.002\)–\(0.005\)). Topic therefore has minimal influence on whether relevant content appears within the top-\(k\) results, but it does affect the position at which relevant chunks are ranked once retrieved.

4.2 MCQA with Retrieval↩︎

Giving LLMs access to retrieved context significantly improves their MCQA performance; topic-wise accuracy per model is shown in Table ¿tbl:tab:best95mcqa?. Other than Gemma-3-1B, all models exceed the human baseline (with cursory search engine use) of \(88\%\) [1]. Several models also meet or surpass the best performing model without retrieved context (GPT-4.5, \(92.5\%\)) [1]. To summarise topic-wise stability, we report the coefficient of variation \(\mathrm{CV}=\sigma/\mu\) across topic accuracies. With the exception of Gemma-3-1B, all models have \(\mathrm{CV}<0.05\), indicating low between-topic variability compared with LLM-only results [1].

max width=

Most models reach maximum MCQA accuracy with \(k\in\{3,5\}\) context chunks, though all LLMs exceed the highest score without retrieval even at \(k=1\); only Llama-3.3-70B continues to improve at \(k=10\), and the marginal gain is small relative to the additional token cost. At \(k=5\), LLMs had access to the target chunk at least 95% of the time, and the increase in recall from \(k=5\) to \(k=10\) comes with additional noise from less relevant chunks. Using NV-Embed-v2 as the retriever raises accuracy relative to the other embedding models shown. However, performance at \(k=1\) is strong despite the retrieved chunk being fully relevant for only \(\leq 80\%\) of queries.

The impact of retrieval quality is further shown in Figure 4. Retrieving the correct chunk yields the largest performance gains, with ranking providing a consistent secondary benefit that is strongest when longer context sections introduce more noise. Accuracy generally declines as the target chunk moves down the ranked list, although smaller models (e.g., Gemma-3-1B, Phi-4-4B, Gemma-3-4B) exhibit noisier and occasionally counter-intuitive responses, including slight improvements at lower ranks, whereas larger models (e.g., Llama-3-70B, MedGemma-27B, Gemma-3-27B) show smoother, more consistent declines and overall reduced sensitivity to rank.

4.2.1 LLM-as-Judge Validation↩︎

max width=

We validated the LLM-as-Judge rubric on a sample of 100 responses independently annotated by two human reviewers (Table ¿tbl:tab:judge95agreement?). The validation reveals substantial variation in how consistently different quality dimensions can be assessed, even by human raters, underscoring the difficulty of evaluating free-form responses to public health questions.

Completeness was the most reliably assessed criterion, with high inter-reviewer agreement (\(\kappa \approx 0.76\); macro-F1 \(\approx 0.88\)) and strong judge–reviewer alignment (\(\kappa \approx 0.57\)–\(0.59\)). Faithfulness was also reasonably consistent across both human raters (\(\kappa \approx 0.60\); macro-F1 \(\approx 0.80\)) and between the judge and reviewers (\(\kappa \approx 0.6\)–\(0.7\)). For both criteria, judge performance falls within the range of human disagreement, and at the aggregate level, judge agreement with Reviewer 2 is essentially indistinguishable from inter-reviewer agreement (overall \(\kappa \approx 0.55\) vs.\(0.56\); macro-F1 \(\approx 0.78\) in both cases).

However, clarity and factual consistency proved much harder to assess reliably. Both criteria exhibited lower inter-reviewer agreement, indicating that humans themselves struggle to apply these dimensions consistently to public health responses. For factual consistency in particular, judge–human agreement was weak (\(\kappa\) near zero and substantially below inter-reviewer \(\kappa\)), meaning that automated scoring for this criterion does not currently reproduce human judgments. Factual consistency would be a critical dimension for public health applications, yet we find it the hardest to evaluate reliably for both humans and the LLM judge. We therefore focus subsequent analysis on faithfulness and completeness, where both human agreement and judge alignment are sufficient to support robust conclusions, while recognising that developing reliable evaluation methods for factual consistency remains an important open challenge.

4.2.2 Free-form Response Evaluations↩︎

Figure 5 shows judge pass rates for free-form answers across models. Medium and large models achieve consistently high completeness and clarity (typically \(\sim\)​0.88–0.97), indicating that they identify the key guidance needed to answer the question and present it in a legible, interpretable way even without guiding answer options. The smallest model (Gemma-3-1B) is a clear outlier, with substantially lower completeness and clarity. In contrast, faithfulness is the dominant failure mode across all models: even among larger models, only around half to two-thirds of responses meet the faithfulness criterion (e.g., 0.64 for the best-performing model), indicating a broad tendency to incorporate guidance from the retrieved context beyond what is strictly required to answer the question.

Retrieval ranking has a pronounced effect on faithfulness (Figure 6). As the target chunk rank \(r\) increases, faithfulness drops sharply, consistent with models relying more heavily on earlier context and drawing in extraneous guidance when the most relevant information appears later in the prompt. By contrast, completeness and clarity show only modest degradation beyond \(r \geq 5\), suggesting that once models locate the relevant guidance they can still produce a coherent answer, even if the evidence appears lower in the context.

5 Limitations↩︎

Our study has several limitations. First, in retrieval evaluation, relevance is treated as binary with a single target chunk per query. Many public health questions are supported by multiple passages, so single-target metrics provide only a partial view of retrieval quality and may understate the value of retrieving alternative, equally valid chunk. Second, our analysis of chunk-length effects leverages the existing chunk structure rather than systematically varying chunk size over the same underlying text, limiting causal interpretation. Third, while we observe topic-level variability in ranking quality, our experiments do not isolate the underlying drivers (e.g., topic-specific language, formatting conventions, or document structure).

Fourth, the multiple-choice format enables partial-credit behaviours such as option elimination, meaning that MCQA gains may overestimate real-world capability. This concern is reinforced by the observation that high MCQA accuracy persists even when the top-ranked chunk is not the labelled target. This gap could reflect two non-exclusive mechanisms: the retrieved chunk may contain sufficient information to eliminate incorrect options without fully answering the question, or it may contain genuinely relevant guidance that our single-target labelling scheme does not capture.

In free-form evaluation, our rubric scores are computed with respect to a single reference chunk. The judge cannot distinguish between genuinely unsupported additions and statements supported elsewhere in the corpus, so faithfulness should be interpreted as “faithful to the provided evidence” rather than “faithful to the entire guidance corpus”. Additionally, weak judge–human agreement on factual consistency means that full-dataset scores for this criterion should be treated as indicative rather than definitive. Finally, because PubHealthBench questions are derived from the guidance itself, they are by construction answerable from the corpus and do not test multi-chunk reasoning or out-of-corpus queries where safe systems should acknowledge uncertainty.

6 Discussion↩︎

6.1 Retrieval↩︎

The strong performance of NV-Embed-v2 across all metrics aligns with external evaluations where NV-Embed models top the MTEB leaderboard, reflecting deliberate optimisation for embedding and search tasks [41]. The universal gains from hybrid retrieval are consistent with prior work showing that combining dense and sparse signals improves recall on domain-specific tasks by pairing semantic similarity with specialised language matching. Importantly, hybrid retrieval enables small and medium-sized models to perform comparably to much larger models, making it essential when working within compute and memory constraints. Given the wide range of viable retrieval configurations in our public health guidance setting, domain-specific benchmarks such as PubHealthBench are essential for validating and selecting effective setups.

Chunk design also exerts a clear influence. The degradation of ranking quality with increasing chunk length (§4.1, Figure 1) is consistent with longer chunks blurring topical boundaries, mixing multiple concepts, and reducing the discriminative power of embeddings. That LLM-generated summaries partially mitigate this effect suggests semantic compression can make relevant material easier to retrieve; however, ranking quality still drops for the longest chunks, indicating that single-shot summaries do not fully preserve the semantics of complex guidance sections. These findings suggest pairing summarisation with more principled chunking (e.g., structure-aware or topic-aware segmentation) rather than relying on summarisation alone.

The weak but systematic effect of topic on ranking quality but not recall (Figure 2) suggests that general-purpose embedding models struggle to separate the most relevant text from superficially similar content in narrow public-health subdomains. This consistency across models points to topic-associated properties of the queries and guidance chunks, such as language patterns, structure, or formatting, as the likely drivers. Where fine-grained improvements in ranking quality are needed, a further systematic exploration of how these properties individually impact retrieval is required.

6.2 MCQA with Retrieval↩︎

Our results demonstrate that with high-quality retrieval, small and medium-sized open-weight models reliably outperform a larger closed model used without retrieval. For knowledge-intensive QA over PubHealthBench, retrieval quality can thus compensate for reduced parameter count: it is more effective to pair a smaller LLM with a well-designed retrieval stack than to rely on a larger model operating without retrieval. The optimal context size for smaller models is \(k \in \{3,5\}\) chunks; beyond that range, additional chunks introduce noise and yield diminishing or negative returns relative to the extra token cost. This constrains the useful context budget for lightweight deployments and suggests that optimisation effort should focus on what goes into those few chunks rather than simply increasing \(k\).

Within this constrained context window, retrieval quality, first in terms of recall, and then ranking quality, is the primary driver of MCQA performance (Figure 4). The interaction between rank sensitivity and model size creates a spectrum of viable design points. Smaller LLMs are more sensitive to ranking quality and work best with fewer, well-ranked chunks, so they must be paired with retrieval setups that keep relevant evidence near the top of a short list. Larger models can partially mitigate weaker ranking because they are better at extracting information from noisier contexts, making it possible to trade off retrieval quality against constraints such as latency, cost, or infrastructure limitations.

6.3 Free-Form QA with Retrieval↩︎

Many of our MCQA findings generalise to the more realistic free-form setting. Medium and large models generally identify relevant guidance from long retrieved contexts and express it in a clear, usable form even without multiple-choice options. The main practical challenge is controlling scope: models often draw broadly from retrieved context, producing answers that go beyond what is required and may combine multiple recommendations. In a public health setting, this matters because additional guidance, even when plausible, can reduce response precision and increase opportunities for misunderstanding. The sharp decline in faithfulness as target chunk rank increases (Figure 6) reinforces ranking quality as a key design consideration for RAG systems in free-form public health QA.

Our human validation results inform the reliability of these automated scores. The judge shows strong alignment with humans for faithfulness and reasonable consistency for completeness, so these dimensions provide the most dependable basis for comparing retrieval setups and models at scale. By contrast, weak judge–human agreement on factual consistency and low inter-rater agreement on clarity highlight the need for further work on automated evaluation methods that more consistently reproduce human judgments of response quality in public health QA.

6.4 Public Health Implications↩︎

Together, these findings have several implications for the design and deployment of RAG systems in public health. In our setup, augmenting LLMs with UK public health documents is sufficient to keep the system aligned with official guidance, to a high degree of accuracy (\(\sim 99\%\) MCQA accuracy), without requiring model retraining. This is consistent with recent work showing that access to up-to-date sources is a key determinant of answer quality on our public health QA benchmark, and that RAG yields more faithful and evidence-backed responses than closed-book LLMs alone [1], [14], fernandezpichel2025searchenginesllmshealth?. For public health agencies that revise guidance in response to emerging evidence or outbreaks, the ability to update the knowledge base directly is important for maintaining the accuracy of LLM applications.

Second, a well-designed retrieval stack allows small and medium-sized open-weight models to match or exceed the performance of larger models running without retrieval. This makes RAG-based systems a practical option for public health institutions operating under budget and infrastructure constraints, offering both competitive performance and greater control over factors such as cost and information update cadence.

Additionally, public health deployments bring specific data-governance requirements. Health data are highly sensitive, and regulators increasingly emphasise data minimisation, strict control of sensitive data, and careful consideration of where and how LLMs are hosted [57]–[59]. RAG is compatible with deployment patterns in which models and knowledge bases are hosted on-premise or within controlled environments, while still providing users with natural-language access to public health guidance.

7 Conclusion↩︎

This work extends PubHealthBench into a retrieval-augmented setting by allowing models to retrieve evidence from the same corpus of UK public health guidance used to construct the benchmark [1]. Using this setup, we make three key findings. First, we show that retrieval effectiveness in this public health guidance domain is highly sensitive to embedding model choice and system design: hybrid retrieval consistently improves retrieval quality, carefully crafting chunk length and uisng larger embedding models also improve relevance ranking. Second, we find that introducing retrieval of official guidance substantially improves MCQA performance, enabling smaller open-weight models to match or outperform a larger models used without retrieval, with gains driven primarily by retrieval quality and careful context selection. Third, we find that these improvements largely carry over to free-form answering, where high-quality ranking yields more focused and faithful responses, but also exposes task-design limitations that matter for real deployments (e.g., multi-evidence support and out-of-corpus queries). Further, our LLM judge evaluations do not fully match human expert assessments, indicating the need for further development in this area as robust automated evaluations are a key requirement for deploying LLM based applications in public health use cases [60].

Overall, our results reinforce a practical conclusion for the application of RAG in public health: the retrieval layer is one of the primary determinants of QA performance and a key lever for deploying capable systems under realistic operational constraints. By grounding responses in an well maintained guidance corpus, RAG provides a pathway to systems that are easier to keep up to date as recommendations evolve, and more feasible to deploy in controlled environments where governance and data security requirements are stringent [59]. Future work should strengthen the benchmark’s retrieval supervision (e.g., multiple relevant chunks per query), broaden evaluation for free-form responses, and introduce harder settings that require multi-chunk reasoning and explicitly test out-of-corpus failure modes.

References↩︎