1 Introduction↩︎

Retrieval-augmented generation (RAG) has rapidly become a popular paradigm for knowledge-intensive question-answering (QA) over domain-specific documents [1], [2]. Yet RAG evaluation remains dominated by open-domain, Wikipedia-style benchmarks [3]–[6]. In specialist domains such as legal, biomedical, defense, and regulated enterprise, three gaps recur: (i) absence of in-domain evaluation, since no public benchmark exists for the target domain; (ii) mismatch of user intents, since factoid QA under-represents the question types that practitioners ask [7], [8]; and (iii) misaligned metrics, since precision-oriented overlap scores (e.g., BLEU) reward surface similarity while users need attribution and faithfulness [9]. The result is a cold-start problem: performance is hard to measure, supervised data is costly, and retrieval/generation iterations lack sufficient signal to localize the errors. We introduce DoRA (Domain-oriented RAG Assessment), a domain-grounded benchmark construction and evaluation framework that uses a small set of specialist-domain seed documents to produce (a) a held-out benchmark split, (b) a larger synthetic split for supervised fine-tuning, and (c) a small expert-curated external check. Our design goals are: (1) operability on small seed sets (no reliance on Wikipedia-style coverage), (2) practitioner-aligned question type coverage (capture intents beyond factoid QA), and (3) deployment-relevant, attribution-aware scoring using modern RAG evaluation protocols [8]–[12]. In this paper, we instantiate DoRA on publicly available Defense and National Security material from Australia, using an Alpaca-inspired taxonomy of user intents. We treat Defense as a high-stakes case study that exercises the framework end-to-end, not as a claim that the benchmark is limited to defense alone. To the best of our knowledge, this is the first synthetic RAG benchmark and evaluation framework specifically instantiated on a defense and national security corpus—prior domain-specific synthetic QA work targets technical manuals [13], enterprise corpora [7], or Wikipedia-style multi-hop QA [14]–[16]. The contribution of DoRA is:

1. A novel domain-grounded benchmark construction and evaluation framework for RAG-based QA where each QA is paired with auditable evidence bundles, enabling attribution-aware evaluation and reproducible benchmarking in specialized domains where expert time is costly¹.

2. A lightweight validation protocol for specialized domains without large labeled test sets. DoRA uses three low-cost checks: a retrieval-conditioned split, an expert-curated evaluation set, and human–LLM-judge calibration on a stratified sample.

3. A defense-domain case study with benchmark-driven adaptation on 40 defense documents split into train/test corpora, producing \(\sim\)​6.6K instances. A LoRA-adapted Llama3.1-8B trained on the cross-generator train split improves RAGEval grounding and reduces hallucination roughly two-fold; the trend persists under alternative retrieval and expert-curated evaluation.

4. Manual evaluation by defense-domain experts to evaluate dataset quality, LLM-as-judge reliability and generalization to a completely human-written dataset.

2 Related Work↩︎

QA benchmarks [3], [5], [6], [17] are largely open-domain and Wikipedia-centric, while retrieval benchmarks such as BEIR [18] and multi-hop QA testbeds [19] target retriever generalization or compositional reasoning rather than deployment over a new private corpus. For RAG evaluation, recent frameworks including RAGAS [10], ARES [20], FaithEval [11], LLM-in-RAG benchmarking [12], RAGBench [8], and RAGChecker [21] provide useful diagnostics for faithfulness, attribution, and hallucination. Their common assumption, however, is that a suitable evaluation set already exists. Closer to DoRA are approaches that synthesize or structure evaluation for specific domains. RAGEval [9] uses scenario templates and synthetic articles for scenario-controlled testing; KIQA [13] generates questions from technical manuals; and DataMorgana [7] synthesizes Q&A over private enterprise corpora with user-defined question categories. DoRA is an enhancement over past work in its combination of three elements in one framework: synthetic benchmark construction from authentic seed documents, evidence-linked scoring for grounding and attribution, and a lightweight validation protocol with an expert-answered external check.

2.8pt

Synthetic pipelines such as DoRA reduce annotation costs and enable controlled stress tests. Related approaches include zero/few-shot prompting for QA generation [22]–[24], few-shot multi-hop synthesis over Wikipedia pairs [14], multimodal multi-hop synthesis [15], and modality/style-conditioned question generation [16]. Compared to open-domain QA and retrieval benchmarks [3]–[6], [18] and existing RAG evaluation frameworks [8], [10]–[12], [20], [21], DoRA enhances three aspects crucial to specialized domains: (i) grounding in authentic seed documents; (ii) coverage of five practitioner-aligned intent styles; and (iii) attribution-aware scoring paired with a small external validation protocol. A comparison with existing work is summarized in [tab:lit-compare,tab:lit-compare-landscape].

3 Methodology↩︎

Let \(\mathcal{C}\) denote a domain corpus segmented into passages with IDs, and let \(\mathcal{S}\) be the five intent styles (find, explain, summarize, generate, provide). DoRA produces \(\mathcal{D}=\{(q_i,a_i,s_i,E_i)\}_{i=1}^{N}\), where each instance is a quadruple \((q,a,s,E)\) comprising a question \(q\), reference answer \(a\), style \(s \in \mathcal{S}\), and supporting passages \(E \subset \mathcal{C}\). An LLM-based agent produces a prediction \(\hat{a}\) for each question \(q\). In RAG mode, we model this as: \(\hat{a} = G\!\left(q,\, R_k(q)\right)\) where \(R_k\) is a retriever that returns the top-\(k\) passages from \(\mathcal{C}\) and \(G\) is a generator that conditions on both the question and retrieved context (with the LLM-only setting as the special case \(R_k(q)=\emptyset\)). The pipeline of DoRA is in 1.

3.1 Data Preparation↩︎

The starting point is a minimal input set: (i) a set of seed documents from the target domain, (ii) a short style specification (we use five intent styles prescribed by defense experts in this context: find, explain, summarize, generate, provide), and (iii) a bounded amount of expert time, reserved mainly for external validation rather than bulk labeling. In our case study, the seed set is 40 defense documents and the expert-time budget is small (9). This matters because the framework is intended for settings where a large labeled corpus is unavailable and expert annotation time must be utilized strategically.

We construct DoRA from 40 seed documents spanning defense strategy, capability, procurement, and industry-planning reports. Each document is chunked into passages, forming the corpus \(\mathcal{C}\) used for evidence selection and retrieval.

3.2 Synthetic Data Generation↩︎

The dataset is created using three steps: first select a candidate evidence \(E\) per seed, generate QA pairs conditioned on style and finally, perform quality control. For each style \(s\), we select a small evidence set \(E\) from the current seed and prompt a style-specific LLM template (see 16) to generate \((q,a)\) conditioned on \((s,E)\); we store \(E\) for auditability. Each style has an evidence budget \(k_s\), which shapes evidence selection and downstream quality checks. We formulate the synthesis step as sampling from a style- and evidence-conditioned generator: \((q,a)\sim p_{\phi}(\,\cdot \mid s,E\,)\), where \(\phi\) denotes the parameters of the chosen LLM and prompt configuration (Chosen parameters in 15).

4pt

(Task-conditioned) Evidence selection: Evidence selection chooses a small evidence bundle (reference contexts) \(E\) conditioned on the target style \(s\), which determines both the desired cardinality \(k_s \triangleq |E|\) and the selection behavior (e.g., extractive single-chunk evidence for find versus multi-chunk synthesis for long-form styles). During dataset construction, the candidate pool is typically restricted to passages from the current seed document, \(\mathcal{C}_{\text{seed}}\subset \mathcal{C}\). We first apply a Natural Language Inference (NLI)-based prefilter to remove clearly irrelevant candidates and keep a reduced candidate pool \(\widetilde{\mathcal{C}}_{\text{seed}}\) (top-\(K_s\) by NLI relevance). We then use a local LLM scorer to rank candidate bundles drawn from \(\widetilde{\mathcal{C}}_{\text{seed}}\). This scorer explicitly rates completeness, complementarity (non-redundant coverage), coherence, and task fitness; the highest-scoring bundle is returned. Formally, we select:

\[E^* \;=\; \arg\max_{\substack{E\subseteq \widetilde{\mathcal{C}}_{\text{seed}}\\ |E|=k_s}}\; g_s(E),\] where \(g_s(E)\) is a style-conditioned bundle scorer computed from the rubric dimensions above (with configurable weights and bounded search over candidate bundles). Details are in 15.

Grounding and Quality Control: Each instance stores an auditable evidence bundle \(E\) (passage references), and we accept a generated item only if it satisfies style-specific deployability constraints and meets an evidence-grounded quality threshold: \[\begin{align} \text{accept}(q,a,s,E)\;\Longleftrightarrow\;&\;h_s(q,a,E)=1 \\ &\wedge\;Q_s(q,a,E)\ge \tau_s, \end{align}\] where \(h_s\) encodes style-specific hard checks (e.g., non-empty fields; provide must contain numeric content) and \(Q_s\) is a composite, evidence-grounded score with threshold \(\tau_s\).

We evaluate grounding with respect to \(E\) using evidence-based metrics, including sentence-level natural language inference (NLI) with penalties for contradictions, extractive span matching (SQuAD-style span F1) [17], and checks for numerical and date consistency when applicable. For longer-form intents (namely, explain, generate and summarize), we additionally compute attribution-focused coverage (context recall and precision relative to \(E\)) and question–answer relevance (semantic similarity between \(q\) and \(a\)), in line with standard RAG evaluation protocols (e.g., [10]). Specifically, \(Q_s\) combines a style-dependent set of metrics \(\mathcal{M}_s\) using configurable weights \(w_{s,m}\): \[Q_s(q,a,E)=\sum_{m\in \mathcal{M}_s} w_{s,m}\, m(q,a,E),\] after which we apply deduplication and diversity filtering to eliminate repeated patterns and ensure balanced representation across the five styles. This procedure yields our dataset, composed of the quadruples described above.

3.3 Construction-time vs.benchmark-time evaluation↩︎

A run of DoRA on two disjoint seed document sets yields two primary artifacts: (i) a held-out synthetic evaluation split for benchmarking and (ii) a cross-generator, cross-corpus training split for supervised fine-tuning. In addition, the same seed corpora for evaluation can support a separate set of expert-answered evaluations with modest effort from experts. The quality controls described above are used only to construct the benchmark by filtering candidate synthetic items. Once the dataset is fixed, benchmark-time evaluation is performed independently with held-out questions, fixed retrievers or oracle evidence, and a separate three-layer metric stack (answer coverage, LLM-judge grounding diagnostics, and expert calibration; see 4). This separation lets DoRA serve as a stable evaluation harness even as corpora, retrievers, prompts, and generators change over time.

3.4 Note on cross-generator, cross-corpus construction↩︎

A standard pitfall in synthetic benchmarks is that the same LLM as well as source documents produce both the training supervision and the evaluation references, creating a train/test correlation that confounds genuine domain adaptation with pipeline-specific style mimicry. DoRA addresses this on two independent axes simultaneously: the training split is generated by Anthropic Claude Sonnet while the test split is generated by OpenAI GPT-4o, both conditioned on the same style-specific templates and quality controls (cross-generator separation); and the two splits are drawn from disjoint sets of seed documents with no document-level overlap, verified by passage-level n-gram and embedding-similarity checks (cross-corpus separation). Construction-time judges are also drawn from a different model family (Google Gemma) than either the generators or the baselines to limit family-level coupling. Together, these choices decouple SFT supervision from the benchmark on the two axes most often cited as causes of inflated synthetic-evaluation results, so a model fine-tuned on the training split cannot win on the test split by imitating GPT-4o’s writing style or by memorizing test-corpus content. One direct consequence is that train and test reference answers have systematically different length distributions (Claude-generated answers are longer), which shapes how token-overlap metrics should be interpreted; we return to this in the metric-design discussion of 4.

4 Experiment Setup↩︎

4.0.0.1 Data and splits.

We partition publicly available defense documents (written in English for a first-language user community) into two disjoint 20-document corpora: one used only for training-split generation, the other only for test-split generation. The held-out test split (1,259 instances, GPT-4o-generated; 2) and the training split (5,323 instances, Claude Sonnet 4.6-generated, 90/10 train/eval; 13) are disjoint at both document and instance levels. We evaluate retriever performance (Hit@\(k\), Recall@\(k\); see 11) and end-to-end QA under two retriever conditions: a fixed dense retriever (default GTE-multilingual [25]; IBM Granite [26] as a robustness check) and an oracle retriever that returns the stored gold evidence bundle \(E\) to isolate generation from retrieval error.

4.0.0.2 Why standard QA metrics are insufficient.

The cross-generator construction (3.4) induces systematic length asymmetry between Claude-generated training references, GPT-4o-generated test answer references, and SFT-model outputs. In this regime, precision-oriented overlap scores (Token F1, BLEU, ROUGE-L F1) penalize verbose-but-correct answers, learned sentence-pair scores absorb length and reference-style cues, and none of these reference-comparison metrics measure whether the answer is grounded in the retrieved evidence—the property a deployed RAG system is judged on.

4.0.0.3 Three-layer metric stack.

We, therefore, adopt a three-layer evaluation in which each layer compensates for a different weakness of the others. (i) Answer coverage: recall-oriented variants of three established metrics spanning lexical to semantic granularity—Token Recall, ROUGE-L Recall [27], and BERTScore Recall [28]—plus Length Ratio (generated/gold answer word-count) as a transparency control. All three coverage metrics are length-fair to verbose grounded answers because additional generated tokens do not reduce the score so long as the gold answer content is covered. (ii) Grounding at scale: the RAGEval rubric judge [9], which extracts gold-answer keypoints and labels each as covered correctly (completeness), addressed incorrectly (hallucination), or unaddressed (irrelevance); this is the only practical keypoint-level attribution signal at the 1,259-item benchmark scale (12). (iii) Expert ground-truth anchor: a 50-item stratified human evaluation (5.4) and an independently authored 86-item expert-curated set (9), used to calibrate the LLM judge rather than scaled to the full benchmark. Full model identifiers, decoding settings, and per-style evidence budgets are listed in [tab:core-model-choices,tab:evidence-budgets].

5 Results↩︎

5.1 Retriever Evaluation↩︎

Following RAGEval [9], we evaluate four closely related retrievers, along with IBM Granite retriever [26], across various top-\(k\) values on the DoRA benchmark test set. 2 shows that, on defense documents, BM25 provides strong early hits but lags dense retrievers in broader coverage at larger \(k\), motivating dense/hybrid retrieval as the natural default for deployment-facing RAG. Overall, GTE achieves the strongest balance across Hit/Recall\(@k\) on the DoRA benchmark set, so we adopt it as the default retriever for end-to-end QA below.

5.2 End-to-End QA and DoRA SFT↩︎

We evaluate end-to-end QA with a fixed retriever (GTE-multilingual-Base, top-\(k\)), where we use \(k{=}3\) as a modest-context deployment setting that keeps the generator input budget controlled while still allowing multi-passage evidence to surface. Following the metric design in 4, we report the recall-oriented coverage metrics (Token Recall, ROUGE-L Recall, BERTScore Recall) alongside the RAGEval grounding diagnostics; appendix tables ([appd:gold-context,appd:qa-ibm-granite]) report the same metric set under oracle and IBM Granite retrieval. The results in 3 show that, under the same GTE retrieved context, stronger and larger off-the-shelf LLMs remain competitive on individual semantic or irrelevance metrics, but no baseline dominates across the coverage and grounding diagnostics. This leaves room for domain adaptation: as detailed below, SFT shifts the overall profile toward higher answer coverage and lower hallucination under the same retrieval and prompt setting.

5.2.0.1 DoRA SFT.

We Supervise-Fine-Tune (SFT) an open model (Llama3.1-8B-Instruct)² with LoRA [31] on the cross-generator DoRA training split (5,323 Claude Sonnet 4.6 instances from the 20-document train-only corpus, 90/10 train/eval). Each example concatenates the question with evidence context \(\mathrm{ctx}(E)\) and predicts the reference answer, mirroring end-to-end RAG. We optimize: \[\min_{\theta}\sum_{(q,a,E)\in \mathcal{D}_{\text{train}}} -\log p_{\theta}\!\left(a \mid q, \mathrm{ctx}(E)\right),\] where \(\theta\) denotes the (adapter) parameters. 3 shows that the cross-generator-trained SFT is stable across three fine-tuning seeds and improves over the base Llama-3.1-8B-Instruct most clearly on grounding: mean hallucination falls from 4.44% to 2.17%, with parallel gains in RAGEval Completeness and Irrelevance. Although SFT does not top every metric (very large models like GPT-3.5-Turbo having the highest BERTScore Recall and Claude-3.5-Haiku getting the lowest Irrelevance), it gives the strongest combined coverage-grounding profile and improves over its own base on all reported metrics. This indicates stronger grounded keypoint coverage rather than only better reference overlap. The Length Ratio shows that SFT produces longer answers than LLM baselines, we treat length asymmetry as a model diagnostic and prioritize grounding and semantic diagnostics over raw overlap. Because the supervision comes from a different LLM family and disjoint document corpus, these gains are not well explained by same-family style matching or memorization of test passages.

5pt

5.3 Robustness across retrieval settings↩︎

The main trend persists across retrieval conditions (4), and the pattern of scores sharpens the deployment story. SFT improves as retrieval quality improves: Completeness rises from 67.20% with the weaker IBM Granite retriever to 73.37% with the default GTE retriever. Its highest Completeness occurs under oracle evidence (86.25%), where retrieval error is removed entirely, showing that adaptation improves not only robustness to imperfect retrieval but also the model’s ability to use high-quality evidence. Full breakdowns are in [appd:gold-context,appd:qa-ibm-granite].

The cross-generator, cross-corpus construction (3.4) already reduces same-family style alignment and document-level memorization risks. We now proceed to evaluate the performance based on whether or not evidence was retrieved. We split the held-out test set by whether GTE retrieves any gold evidence chunk: Hit@\(3{=}1\) (809; 64.3%) vs.Hit@\(3{=}0\) (450; 35.7%). In 5, SFT is strongest when retrieval succeeds (84.38% Completeness, 1.73% hallucination), and its gain over the Llama-3.1-8B base grows when retrieval misses (+6.17 Completeness points, +4.65 hallucination points). Thus, the benefit of SFT is not limited to easy retrieved contexts: it also improves robustness when retrieval fails to surface the gold evidence, suggesting that the adapted model can better use partial or noisy context instead of blindly following irrelevant passages.

2.5pt

5.4 Domain Expert Evaluation↩︎

We now report human evaluation at three stages as described below. Our human evaluators are defense domain experts who are not authors of the paper and are familiar with the national context of the documents.

5.4.0.1 Stage 1: Expert validation of synthetically generated QA pairs.

Four defense domain experts score a stratified 50-item benchmark sample with the same rubric used by our LLM judge. Inter-annotator Within-1 agreement is 94.7–100% across the seven dimensions and \(\kappa_w \ge 0.5\) on three of them (7), and the LLM judge tracks experts substantially on groundedness (\(\kappa_w{=}0.56\)) and tightly within one point on relevance (96% Within-1). This supports using the LLM judge as a scalable diagnostic for grounding-anchored signals, while keeping expert review for borderline items; full design and per-dimension results are in Stage-3 of this section.

5.4.0.2 Stage 2: QA Evaluation on Expert-curated QA pairs.

We create a separate expert-curated set of 86 samples whose reference answers are written and corrected by defense experts (without any use of LLMs), and are never shown to the evaluated LLMs (9). On this harder held-out set, 6 shows that SFT roughly doubles Completeness (24.3%) and halves hallucination (26.5%) relative to the general-purpose baselines, with parallel gains in ROUGE-L and BERTScore Recall. The only weaker signal is Irrelevance: SFT produces more content outside the gold keypoints. Because it also improves Completeness and reduces hallucination, we interpret this as verbosity, not worse factual grounding.

2pt

4pt

5.4.0.3 Stage 3: Expert validation of LLM-as-Judge

To calibrate our LLM-as-judge against expert judgment on the same benchmark items it scores, we conduct a human evaluation on a stratified sample of the DoRA test split. We draw 50 items balanced across the five intent styles (10 each), and four domain-expert annotators score every item independently using seven quality-dimension rubric on a 1–5 ordinal scale (comprehensiveness, relevance, groundedness, verbosity, composition, clarity, alignment). Annotators see the (question, evidence, reference answer) triple, score in isolation without seeing the other annotators’ labels, and use a fixed rubric definition document. 7 shows strong expert consistency across the seven rubric dimensions: Within-1 agreement is 94.7–100%, mean absolute difference is at most 0.40, and mean pairwise quadratic-weighted \(\kappa_w\) is positive for all dimensions (0.16–0.65). The weakest \(\kappa_w\) is on alignment, where expert scores saturate near the top of the scale, producing a ceiling effect rather than substantive disagreement.

We also calibrate the RAGEval LLM judge against expert scores on the three matching rubric dimensions: groundedness, comprehensiveness, and relevance. On the 50-item sample, the judge aligns best on groundedness (\(\kappa_w{=}0.56\), Within-1 92%), is moderate on comprehensiveness (\(\kappa_w{=}0.23\), Within-1 70%), and is usually within one point on relevance (Within-1 96%, MAD 0.19), where \(\kappa_w\) is suppressed by expert-score saturation (11, 10). These results support the three-layer framing of 4: RAGEval grounding signals can be scored by the LLM judge at benchmark scale, while expert review serves as the ground-truth anchor for stylistic and coverage-sensitive dimensions.

5.5 In-Context Learning Baseline↩︎

As an adaptation-free baseline, we prompt GPT-4o and Llama-3.1-8B with 1-shot and 2-shot Q&A exemplars curated from , under the same GTE retrieval condition as 3. 3 shows that ICL provides only marginal lift over the 1-shot setting—moving from 1-shot to 2-shot raises RAGEval Completeness by at most \(\sim\)​2 pp for both models (GPT-4o 65.2%\(\rightarrow\)​66.8%, Llama-3.1-8B 65.1%\(\rightarrow\)​66.4%). SFT is clearly ahead at 73.0% Completeness, with parallel gaps on ROUGE-L Recall and BERTScore Recall: curated exemplars provide useful format guidance, but fine-tuning on the cross-generator training split yields stronger grounded keypoint coverage.

6 Discussion↩︎

6.0.0.1 Cross-generator design.

The cross-generator + cross-corpus construction (3.4) is DoRA’s most consequential design choice, and the same recipe transfers to any synthetic-benchmark plus SFT pipeline in a specialist domain at essentially no additional cost. DoRA, therefore, contributes a methodological archetype, not just a defense-domain artifact.

6.0.0.2 Tight clustering.

Under the same retriever configuration, off-the-shelf LMs cluster closely on semantic and grounding diagnostics (3), suggesting that evaluation in private, domain-specific settings is bottlenecked by evidence selection and grounding rather than by raw model size. Targeted domain adaptation (even at 8B scale, via LoRA on cross-generator supervision) closes meaningfully more of the visible headroom than swapping in a larger general-purpose model. Notably, this benefit becomes more pronounced as retrieval quality degrades (4), matching deployment conditions where retrievers are imperfect.

6.0.0.3 From benchmark to deployment.

DoRA is designed for cold-start specialist settings and should be treated as an in-domain regression signal, not a universal leaderboard: teams rerun it after updating the corpus, retriever, prompts, or adaptation, and inspect where failures move across styles and grounding diagnostics. The construction recipe depends only on partitionable seed documents, an intent taxonomy, and two LLM families.

7 Conclusion↩︎

We presented DoRA, a domain-grounded benchmark construction and evaluation framework for cold-start RAG in specialist domains. DoRA starts from a small set of seed documents and produces auditable QA instances with linked evidence, enabling evaluation when no in-domain benchmark or large labeled dataset exists. Its cross-generator, cross-corpus design uses different LLM families and disjoint seed-document corpora for training and test generation, which structurally decouples SFT supervision from the benchmark and reduces the main circularity risks in synthetic evaluation. In a defense-domain case study, DoRA produces a held-out synthetic test split, a cross-generator SFT-ready training split, and an expert-curated external check. LoRA adaptation of Llama-3.1-8B substantially improves grounding diagnostics under the default GTE setting and remains robust across alternative retrieval conditions, showing that synthetic supervision can improve deployment-facing RAG behavior even under domain shift.

Limitations↩︎

DoRA’s empirical evaluation in this paper covers a single specialist domain (defense), so transfer of the framework to other specialist corpora is argued rather than demonstrated. Although the training and test splits are generated by different LLM families and drawn from disjoint seed documents to limit same-family style alignment and document-level memorization, they still share an overall construction pipeline (same prompts, same NLI prefilter, same quality controls), so within-pipeline metrics can still partially reflect pipeline-level style correlation; we mitigate this with retrieval-conditioned analysis and an independently constructed expert-curated check set, but cannot fully rule out residual correlation. The expert-curated check set is small (86 items), so statistical significance on this split is limited; we therefore use it as a qualitative external signal rather than a leaderboard. The framework depends on capable LLMs at multiple stages—generation, evidence scoring, and parts of evaluation—which can propagate shared biases or blind spots. Reported SFT numbers are averaged over three fine-tuning seeds, but we still report point estimates for most non-SFT baselines rather than full confidence intervals. Finally, DoRA measures benchmark quality and grounding, but does not measure pretraining contamination directly; teams adopting it in domains with public-document overlap should add a contamination probe.

Ethics Statement↩︎

The case-study corpus is sourced from open-public defense documents rather than classified or proprietary material. We will release the seed documents, prompts, reproducible code, and derived benchmark instances with provenance fields, subject to document licenses and sensitivity constraints. The contribution of this work lies in the synthetic creation of a QA benchmark, an evaluation framework, and an adaptation setting. The defense application described in this paper is restricted to answering questions based on a fixed set of defense documents. The system does not make decisions, take actions, or replace human judgment; all interpretation and downstream use of its outputs remain the responsibility of human operators and fall outside the scope of this work. We expect that practitioners applying DoRA to other specialist domains will perform their own ethics review appropriate to that domain.

Appendix↩︎

This appendix provides additional results and implementation details for reproducibility:

• Additional QA Results (8): QA with oracle-context (gold evidence) ([appd:gold-context,tab:qa-rageval-oracle]), and end-to-end QA with IBM Granite retrieval ([appd:qa-ibm-granite,tab:qa-rageval-granite,tab:qa-retrieval-conditioned-granite]).

• Manually Curated Dataset (9): expert-answered set creation and its use as an external evaluation check.

• Human–LLM-Judge Alignment on RAGEval-Mapped Dimensions ([appd:human-judge-align,tab:human-judge-align]): per-dimension calibration of the LLM judge against the mean expert score on the rubric dimensions that map to the RAGEval signals used in the main results.

• Metrics (11): retrieval metrics (Hit@k, Recall@k), retriever identifiers (12), and implementation identifiers for learned metrics and judges (17).

• RAGEval Metrics (12): definitions and formulas for the RAGEval diagnostics (completeness, hallucination, irrelevance) used in our end-to-end QA evaluation.

• Dataset (13): dataset composition and generation statistics by style.

• Style Examples (14): example questions and answers for each of the five intent styles.

• Detailed Methodology (15): model choices (17) and evidence-selection budgets ([tab:evidence-budgets]).

• Prompts (16): prompt templates used for benchmark-time grounded inference and style-conditioned QA generation.

8 Additional QA Results↩︎

8.1 QA with Gold Evidence Context↩︎

To isolate generation quality from retrieval errors, we also report gold evidence context results where the generator is given the recorded gold evidence passages \(E\) (oracle context) instead of retrieved passages. 8 reports oracle-context results with the same prompt used across models. Under oracle evidence, grounding improves substantially across LLM models; SFT gives the strongest lexical recall and lowest hallucination rate, while remaining competitive on Completeness and Irrelevance. This indicates that the adaptation benefit is not only a retrieval artifact.

8.2 End-to-End QA with IBM Granite Retriever↩︎

We also report end-to-end QA under an alternative fixed retriever, IBM Granite (english-r2), to stress-test robustness to retrieval variation.

2.5pt

In this setting, where IBM Granite is weaker than GTE on retrieval metrics (2), SFT widens its margin on grounding signals: it achieves the lowest hallucination (3.44%) and second-lowest irrelevance (29.36%) in the aggregate table, more than halving hallucination relative to the base Llama-3.1-8B-Instruct (7.21%). The retrieval-conditioned split in 10 shows where the gain comes from. Under successful Granite retrieval, SFT has the highest Completeness (84.02%) and lowest hallucination (1.95%), improving over the base by +4.09 Completeness points and reducing hallucination by 2.87 points. Under retrieval failure, the margin grows: SFT reaches 45.74% Completeness, compared with 35.46% for the base and 32.02% for GPT-4o, and reduces hallucination to 4.76%, compared with 10.21% and 15.59%, respectively. This supports the interpretation that DoRA-based adaptation improves retrieval robustness, suitable for real-world applications where retrieval potentially induces errors.

9 Manually Curated Dataset↩︎

We created a high-quality QA dataset for expert-curated evaluation using a combination of LLM-driven generation, LLM judgment, and human expert annotation. This is a separate, smaller pipeline used only for the expert-curated set summarized in the main paper. We followed a five-step process that blends human expert feedback with LLM-generated data.

9.0.0.1 Bootstrap dataset.

We asked a domain expert to generate 25 QA pairs based on and answerable with the information in our set of domain-specific seed documents. We obtained 5 examples for each question style: find, provide, generate, summarize, and explain.

9.0.0.2 Document chunking.

We chunked the seed documents into semantically coherent passages using LumberChunker [32]. We used the embedding model listed in 17 to calculate an embedding vector for each passage. This process produced 2,187 non-overlapping passages from all seed documents.

9.0.0.3 Semantic graph construction.

We constructed a k-NN graph (for \(k=5\)) with nodes corresponding to text passages and edges between two nodes representing semantic similarity estimated using cosine distance between the passages’ embedding vectors. We only accepted edges between nodes when the cosine distance was less than 0.35. The resulting directed graph had 2,187 nodes and 8,121 edges with the average node degree of 3.7\(\pm\)​0.9.

9.0.0.4 LLM-based dataset generation.

We prompted the expert-curated-set generator listed in 17 to generate QA pairs for each question style using context retrieved by querying the k-NN graph. First, we selected uniformly at random a node from the graph. Then, depending on the question style, we extracted the ego-graph around the selected node. For find and provide style questions, we returned only the selected node, and for the other styles we returned the one-hop neighborhood around the node, including the selected node itself. We then prompted the LLM to generate QA pairs for the given question type and context. 4 shows the prompt used for this step. The prompt includes two QA examples for in-context learning selected uniformly at random from the bootstrap dataset conditioned on the question style.

The generated question, answer, and context were then evaluated by the expert-curated-set judge listed in 17 using the prompt in 5. We asked it to assign a score in the range \([0,1]\), where 0 is worst and 1 is best. We accepted generated items with scores above 0.75. This process produced 100 QA pairs with context, with 20 examples for each question style.

9.0.0.5 Human annotation.

Finally, domain experts were shown the generated questions and context and asked to provide an answer. We did not show the generated answer to the annotators. Of the 100 generated examples, the annotators deemed 86 valid and provided answers derived from the supplied passages. The combination of LLM-generated questions, human-generated answers, and the relevant context comprises the final dataset.

9.1 Use as an External Evaluation Set↩︎

The final 86 expert-answered items are used only as an external evaluation check, not as supervised fine-tuning data. This set complements the larger synthetic benchmark by testing whether the same systems remain grounded when evaluated against independently written expert answers. We evaluate models under both retrieved context and oracle gold evidence with the same semantic and RAGEval-style diagnostics used in the main experiments; results are summarized in 6.

10 Human–LLM-Judge Alignment on RAGEval-Mapped Dimensions↩︎

Our main results report RAGEval Completeness, Hallucination, and Irrelevance, scored by an LLM judge following the open-source RAGEval protocol of [9]. To calibrate this judge against expert judgment we focus on the three rubric dimensions that correspond directly to those signals—groundedness (hallucination counterpart), comprehensiveness (completeness counterpart), and relevance (irrelevance counterpart)—and score the same 50-item stratified sample described in 5.4 under the identical rubric. 11 reports the alignment between the LLM judge and the mean score across the four domain-expert annotators on these three RAGEval-mapped dimensions. The remaining four rubric dimensions (verbosity, composition, clarity, alignment) are stylistic and have no RAGEval counterpart; they are omitted from this calibration. The judge agrees substantially with the experts on groundedness (\(\kappa_w{=}0.56\), Within-1 92%), moderately on comprehensiveness (\(\kappa_w{=}0.23\), Within-1 70%), and very tightly within one point on relevance (Within-1 96%, MAD 0.19), where \(\kappa_w\) is near zero because the expert scores saturate at 4–5 and leave little variance for chance correction. As noted in the main paper, we therefore treat LLM-judge metrics as scalable diagnostics for grounding-anchored signals rather than as a substitute for expert review on borderline items.

2.5pt

11 Metrics↩︎

For each benchmark instance \(i\), we treat the question \(q_i\) as the retrieval query, and the gold evidence passages recorded in DoRA as the set of relevant items \(G_i\) (one or more chunk IDs). A retriever produces a ranked list of retrieved passages \(R_k(q_i)=(d_{i,1}, d_{i,2}, \dots, d_{i,k})\), where \(d_{i,j}\) is the chunk ID at rank \(j\) for query \(q_i\), and \(k\) is the evaluation cutoff (e.g., \(k=3\) in our QA study).

11.0.0.1 Hit@\(k\).

Let \(I\) be the set of evaluated (e.g. test set). We define the instance-level indicator \[\mathrm{Hit@}k_i=\mathbf{1}\!\left[\exists\, j\in\{1,\dots,k\}:\;d_{i,j}\in G_i\right],\] and report the average \(\mathrm{Hit@}k=\frac{1}{|I|}\sum_{i\in I}\mathrm{Hit@}k_i\). Intuitively, \(\mathrm{Hit@}k\) measures whether the retriever returns any gold evidence chunk in its top-\(k\) list for \(q_i\).

11.0.0.2 Recall@k.

When multiple evidence passages are marked relevant, we measure coverage of the gold set: \[\mathrm{Recall@}k_i=\frac{\big|\{d_{i,1},\dots,d_{i,k}\}\cap G_i\big|}{|G_i|},\] and report \(\mathrm{Recall@}k=\frac{1}{|I|}\sum_{i\in I}\mathrm{Recall@}k_i\). This measures how completely the top-\(k\) retrieved chunks cover the gold evidence set (with set intersection preventing repeated retrieval of the same chunk from inflating the score).

Details of retrievers are shown in 12.

6pt

11.0.0.3 Answer-coverage metric models.

For reproducibility of the learned answer-coverage scores, the implementation identifier for BERTScore Recall is listed in 17.

3pt

12 RAGEval Metrics↩︎

We compute the end-to-end faithfulness diagnostics using the open-source RAGEval evaluation implementation³ from [9], with the rubric-judge setup listed in 17. For each instance \(i\), RAGEval extracts a set of reference keypoints \(\mathcal{K}_i=\{k_{i,1},\dots,k_{i,N_i}\}\) from the gold answer \(a_i\) and evaluates a system answer \(\hat{a}_i\) by classifying each keypoint into one of three categories:

• \(\mathcal{K}_{i,\mathrm{comp}}\): keypoints judged as complete (covered correctly by \(\hat{a}_i\));

• \(\mathcal{K}_{i,\mathrm{irr}}\): keypoints judged as irrelevant (missing/unaddressed by \(\hat{a}_i\)); and

• \(\mathcal{K}_{i,\mathrm{hall}}\): keypoints judged as hallucinated/wrong (addressed incorrectly by \(\hat{a}_i\)).

12.0.0.1 Instance-level scores.

The metrics are defined as normalized keypoint counts: \[\begin{align} \mathrm{RAG\text{-}Comp}_i &= \frac{\lvert \mathcal{K}_{i,\mathrm{comp}}\rvert}{\lvert \mathcal{K}_i\rvert},\\ \mathrm{RAG\text{-}Hall}_i &= \frac{\lvert \mathcal{K}_{i,\mathrm{hall}}\rvert}{\lvert \mathcal{K}_i\rvert},\\ \mathrm{RAG\text{-}Irr}_i &= \frac{\lvert \mathcal{K}_{i,\mathrm{irr}}\rvert}{\lvert \mathcal{K}_i\rvert}. \end{align}\] All three scores lie in \([0,1]\); higher completeness is better, while lower hallucination and irrelevance are better.

12.0.0.2 Dataset-level reporting.

We report micro-averages over instances: \[\mathrm{RAG\text{-}Metric}=\frac{1}{|I|}\sum_{i\in I}\mathrm{RAG\text{-}Metric}_i,\] where \(I\) is the evaluated set (e.g., the 1,259 test split from dataset).

13 Dataset↩︎

Details regarding the train/test sets used to benchmark different LLMs in the defense domain are shown in [tab:testset,tab:trainset].

6pt

5pt

14 Example Questions and Answers by Style↩︎

We use five intent styles to approximate deployment traffic in defense settings:

• find: pinpointing exact data points (e.g., deployment dates, cost figures);

• explain: clarifying technical distinctions critical to planning and risk assessment;

• summarize: condensing lengthy reports into concise, actionable overviews;

• generate: producing structured lists of capabilities, gaps, or priorities at a glance; and

• provide: surfacing key quantitative metrics (budgets, procurement data).

Examples for each style are shown in 16.

4pt

15 Detailed Methodology↩︎

15.0.0.1 Core model choices.

17 summarizes the default model choices by pipeline stage.

15.0.0.2 Computational budget and infrastructure.

Most benchmark construction and evaluation stages use API-hosted LLMs or lightweight local encoders, so exact provider-side compute is not observable. For local components, we use the model identifiers listed in 17; the largest local model used during construction is Gemma-2-9B-IT for evidence scoring, run in 8-bit mode, while the main adapted generator is Llama-3.1-8B-Instruct with LoRA adapters. DoRA SFT is performed on a single NVIDIA GPU workstation; across the three fine-tuning seeds reported in 3, the total training budget is approximately 10 GPU-hours of 1xA100 GPU (80GB). We report API model names rather than parameter counts where providers do not disclose exact deployed model sizes, and we release prompts, code, and configuration files to make the observable compute path reproducible.

15.0.0.3 Training setup and hyperparameters.

For SFT, we fine-tune Llama-3.1-8B-Instruct with LoRA adapters on the 5,323-instance cross-generator training split, using the 90/10 train/eval partition described in 14. We use the same retrieved-context input format as the downstream QA experiments: the question is concatenated with the evidence context, and the model is trained to generate the reference answer. We tune only a small set of LoRA and optimization hyperparameters on the held-out eval split, selecting the checkpoint with the best validation loss before evaluating once on the fixed test split. The best-found configuration uses LoRA rank 64, LoRA \(\alpha{=}\)​128, dropout 0.05, learning rate \(1 \times 10^{-5}\), batch size 4 (effective 16 with gradient accumulation), gradient accumulation 4, max sequence length 4096, and 3 training epochs (checkpoint selected by best validation loss). The three-seed results in 3 rerun this selected configuration with different random seeds rather than re-tuning per seed.

3pt

15.0.0.4 Tokenization and chunking.

We tokenize with cl100k_base. Chunks are built by packing paragraphs to a target budget of 512 tokens (min 100; max 1024). Oversized chunks trigger a sentence-level sliding-window fallback with \(\approx\)​25% sentence overlap. Each chunk retains provenance (document id/name/path, page numbers, optional section header).

15.0.0.5 Retrieval.

During dataset construction, candidate passage retrieval uses a hybrid dense+BM25 stage with cosine similarity (L2-normalized embeddings, inner-product search) and score mixing (0.7 dense / 0.3 BM25) before NLI-based evidence filtering. For retrieval-conditioned benchmark runs, we instead use the fixed retrievers described in the main paper (e.g., GTE-multilingual-Base or IBM Granite), with chunk size 320 tokens, overlap 64, and top-\(k{=}3\).

15.0.0.6 Evidence selection and budgets.

Evidence selection is conditioned on style \(s\). We first retrieve candidates, then apply an NLI prefilter of size \(K\), and finally use a local LLM scorer to rank candidate combinations for completeness, complementarity, coherence, and task fit. [tab:evidence-budgets] summarizes the per-style evidence budgets and selection hyperparameters. When multi-chunk synthesis is enabled, we require at least two distinct evidence chunks for long-form styles.

15.0.0.7 Construction-time filtering.

We over-generate (3\(\times\) target size), remove exact duplicate questions per seed document, and apply task-aware quality control. Hard gates enforce non-empty fields and numeric content for provide; for multi-chunk items, accepted candidates must draw from at least two distinct evidence chunks. Evidence-grounded scoring uses NLI, span alignment, numeric consistency, and semantic relevance/coverage with the models listed above. We then stratify candidates by style and select the highest-quality items under per-style quotas.

16 Prompts↩︎

16.1 Multi-Chunk Synthesis Directives↩︎

For multi-chunk synthesis, we append the following directives (with configuration-dependent values, e.g., minimum chunks/citations) to each style prompt:

MULTI-CHUNK SYNTHESIS MODE (ENFORCED):
- Integrate evidence from at least 2 distinct [Chunk] sections when they offer relevant details.
- Cite each supporting chunk explicitly by number or by quoting a short anchor
  phrase (minimum 2 citations).
- Merge overlapping insights into one cohesive answer rather than separate mini-answers.
- Reiterate key facts at the start of the answer so earlier chunks are not ignored.
- Answers lacking multi-chunk grounding will be rejected by downstream verification.

16.2 Benchmark-Time Grounded Inference Prompt↩︎

For end-to-end QA evaluation, each model receives the retrieved context (or oracle gold evidence) and the test question using the following grounded inference prompt:

You are provided with grounded evidence. Use only this context to answer.
If the evidence is insufficient, respond with "I don't have enough context."

Context:
{context}

Question:
{question}

Answer:

16.3 Style-Conditioned Q&A Generation Prompts↩︎

We generate each candidate instance by prompting the dataset-construction model listed in 17 with an evidence context and a style-specific template. In all prompts below, {context} is replaced with the selected evidence passages (concatenated).

16.3.0.1 find.

Based on the provided context, generate a factual question-answer pair that asks
for specific information (dates, numbers, names, locations, etc.) that can be
found in the text.

Context:
{context}

Requirements:
- Question should ask for specific factual information
- Answer should be a short, precise factual response directly from the text
- Copy the shortest exact phrase that answers the question (verbatim) from the supporting sentence
- If needed, include a brief quoted clause that contains the fact
- Answer must be declarative and must not be phrased as a question; do not include a question mark
- The answer must exist exactly in the provided context
- Focus on concrete details like dates, amounts, names, locations, or specific facts
- Prefer interrogatives: What/When/Where/Who/Which/How many/How much (avoid standalone 'How')
- If multiple values appear, choose the one that most directly answers the question
- Use only information present in the Context; do not add external knowledge
- Do not refer to the document/text/section; produce a self-contained Q&A

IMPORTANT - DO NOT generate questions about:
- Page numbers, indexes, or table of contents
- Who is mentioned/listed in the document
- Where information appears in the document structure
- Document metadata or navigation elements

Format your response as:
Question: [Your question here]
Answer: [Your answer here]

Example:
Question: What is the budget allocation mentioned for defense procurement?
Answer: $2.4 billion

16.3.0.2 explain.

Based on the provided context, generate a question that asks for an explanation
or clarification of a concept, distinction, or technical detail mentioned in
the text.

Context:
{context}

Requirements:
- Question should ask for an explanation, clarification, or distinction
  (prefer 'How…' or 'Why…' unless a clear 'What is the difference…' is present)
- Answer must be 2–3 sentences:
  • Sentence 1: direct explanation (what/how/why) using terms from the Context.
  • Sentence 2–3: tie the explanation to one or two specific supporting clauses
    (quote 2–8 words) from the Context.
- Ground the explanation in specific source sentences; avoid generic or speculative claims
- Use only information present in the Context; do not add background not present
- Where helpful, use short phrases from the source for key terms
- Do not refer to the document/text/section; produce a self-contained Q&A

Format your response as:
Question: [Your question here]
Answer: [Your answer here]

Example:
Question: What is the difference between operational capability and strategic capacity?
Answer: Operational capability refers to the immediate ability to conduct
specific military operations, while strategic capacity represents the long-term
potential to sustain and expand operations over time.

16.3.0.3 summarize.

Based on the provided context, generate a question that asks for a summary or
overview of the key points, and provide a concise summary as the answer.

Context:
{context}

Requirements:
- Question should ask for a summary, overview, or key points
- Answer should be a comprehensive but concise summary (2-4 sentences)
- Include the most important information from the context
- Use specific names, figures, and terms where they appear (avoid vague phrasing)
- Structure the summary logically and clearly
 - Include at least one concrete figure or named program if present
 - Cover what is being done (action), why (rationale), and scale (numbers/timeframes) when present
 - Avoid speculation or recommendations; report only what is explicitly stated
 - Use only information present in the Context; do not add external knowledge
 - Do not refer to the document/text/section; produce a self-contained Q&A

Format your response as:
Question: [Your question here]
Answer: [Your answer here]

Example:
Question: What are the key points of the cybersecurity strategy outlined in this section?
Answer: The cybersecurity strategy focuses on three main areas: strengthening
defensive capabilities through advanced threat detection systems, building
workforce expertise through specialized training programs, and enhancing
international cooperation through information sharing agreements.
The strategy emphasizes a proactive approach to cyber threats with increased
investment in both technology and human resources.

16.3.0.4 provide.

Based on the provided context, generate a question that asks for specific
quantitative data (budgets, numbers, percentages, dates) and provide the
precise numerical answer.

Context:
{context}

Requirements:
- Question should ask for specific quantitative information
- Answer should copy the exact numeric phrase from the context (including units)
- If the number appears within a sentence, quote the key clause containing it
- Include units where appropriate (dollars, percentages, dates, quantities)
- The answer must be factually accurate and directly from the text
- NO HEDGING: Do not use words like "likely", "may", "could",
  "approximately", "suggests". Provide definitive figures only as stated in
  the context.
- NO INFERENCE: Do not infer or estimate numbers; only copy what is explicitly present.
 - If a range is shown and the question asks for total/planned/overall, copy
   the range exactly (e.g., "$5.9bn – $7.9bn"). If the question asks for
   'approved' vs 'unapproved', pick the specific column value.
 - If no numeric value exists for your question, select a different
   quantitative question that can be answered exactly from the Context.
 - Use only information present in the Context; do not add external knowledge
 - Do not refer to the document/text/section; produce a self-contained Q&A

IMPORTANT - DO NOT generate questions about:
- Page numbers or document navigation
- Index entries or table of contents data
- Which page or section contains information
- Document structure or metadata

Format your response as:
Question: [Your question here]
Answer: [Your answer here]

Example:
Question: What is the total budget allocation for procurement mentioned in the report?
Answer: $1.2 billion

16.3.0.5 generate.

Based on the provided context, generate a question that asks for a structured
list (capabilities, priorities, requirements, etc.) and provide the answer as a
numbered list.

Context:
{context}

Requirements:
- Question should ask for a list of items (capabilities, priorities, gaps, requirements, etc.)
- Answer should be a numbered list with 3-6 items
- Each item must be grounded in a single supporting sentence from the Context
- Include a short quoted anchor (2–6 words) from that sentence for each item
- Do not merge information from multiple sentences into one item
- Use phrases directly from the source sentence (verbatim or near‑verbatim)
- Do not introduce items that are not explicitly stated in the Context
- Items should be substantial noun phrases (avoid weak forms like 'there is/are')
- Use only information present in the Context; do not add external knowledge
- Do not refer to the document/text/section; produce a self-contained Q&A

Format your response as:
Question: [Your question here]
Answer: [Your numbered list here]

Example:
Question: What are the key capabilities mentioned for the new defense system?
Answer: 1. Advanced threat detection using AI-powered surveillance systems
2. Rapid response coordination through integrated command networks
3. Multi-domain operational flexibility across land, sea, and air environments
4. Enhanced cybersecurity protection for critical infrastructure

References↩︎