[ orcid=0009-0008-2056-6884, email=bmichaels6@gatech.edu, ]

[ orcid=0009-0002-1777-2456, email=bjohnson436@gatech.edu, ]

Retrieval-Augmented Generation , Scientific Question Answering , Large Language Models , Attributed Text Generation

1 Introduction↩︎

Scientific knowledge evolves through both continual research and publication of new papers that update previous empirical results, propose novel approaches, or replace old knowledge altogether. While this has led to the discovery of many new research findings, it also has consequential implications for RAG-related question answering tasks. With rapid developments in LLM technology, information can become quickly outdated or even become a complete distraction during question answering or information retrieval tasks [1]. While RAG is an intuitive choice for answering scientific questions since it can base its output on the content of source documents instead of being limited to model memory, it still has ample retrieval related mistakes prompting for stronger, grounded QA. Such mistakes include retrieval of irrelevant documents or paragraphs, missing evidence, wrong document citations, and retrieval of inaccurate or outdated information. These issues become particularly relevant in the domain of science because an answer must be factually grounded, empirically backed by research documents, accurate and currently relevant, and the evidence used must justify the claims.

The 2026 LongEval-RAG [2], [3] task considers this issue in relation to temporally evolving scientific documents from the Connecting Repositories (CORE) scientific literature corpus [4]. For every query, the task organizers present a set of ten documents to consider, and require the participants to output an answer in natural language form, supported by references to the documents used to generate an evidence based answer. Although this inherently simplifies the initial retrieval process by narrowing the candidate set to a smaller region of documents before answer generation, the task organizers included irrelevant documents which still required evidence related filtering. Even within a chunked document, not all information can be considered relevant or highly relevant, which still makes this RAG task meaningful in both natural language answer quality and faithfulness.

As the task supplied a set of pre-retrieved documents for each query, we focused our architecture on post-retrieval optimization of grounded answer generation. To address the challenge of distractor documents, we compare a hybrid chunk retrieval baseline against a strict corrective pipeline. The corrective architecture utilizes Corrective RAG (CRAG) [5] to filter chunks prior to generation, and CiteFix [6] to enforce strict claim attribution by pruning ungrounded citations post-generation.

Apart from the automated pipelines, we also examine two manually curated runs that use state-of-the-art frontier models (GPT-5.5 Thinking and Claude Opus 4.7 Thinking). While these approaches achieved high scores on standard NLP metrics (ROUGE, BERT) and LLM judged answer relevancy, our RAGAs-based faithfulness evaluation reveals that these same frontier models often fail to properly ground answers in cited materials. The remainder of this paper outlines the related work, details our methodologies, and highlights the gap between generative fluency and citation integrity in our evaluation results.

2 Related Work↩︎

Standard RAG frameworks treat source documents as static repositories of knowledge. However, document corpora often exhibit longitudinal context drift, where knowledge mutates over time [7]. This poses a challenge to long-context extraction, where temporal misalignment can contaminate retrieval. Traditional RAG operates under a naive assumption that retrieved contexts are inherently trustworthy, leaving generators vulnerable to retrieval failures. Corrective RAG (CRAG) introduced an active, intermediate evaluation layer that attempts to sanitize the retrieved context before generation [5]. Another challenge for RAG systems is the strict attribution of sources. At generation, an LLM can generate coherent text using supplied context but include ungrounded statements or inaccurate citations [8]. Recent efforts to address this failure mode in trustworthy RAG focus on post-hoc citation mapping to force explicit citation of generated claims [6], [9].

3 Task and Experimental Methodology↩︎

3.1 CORE Scientific Document Corpus↩︎

The LongEval 2026 Task 4 (LongEval-RAG) dataset provides a textual query alongside 10 pre-retrieved document IDs from the CORE corpus. These 10 retrieved documents are meant to represent the knowledge boundary for generating the answer. This constraint tests a RAG system’s ability to filter evidence, isolate distractors, and successfully produce extractive answers. For each document ID, paper title, abstract, and full text fields are made available.

Generative pipelines produced for the LongEval-RAG task must produce outputs aligned with a strict JSONL schema. For each query, systems ingest a JSON containing the query and list of pre-retrieved document IDs. The pipeline processes this input and appends an answer including the generated response and cited documents in the pre-retrieved array. An example of the format is provided in Figure 1.

None

3.2 Candidate Processing and Retrieval↩︎

Given that an initial set of candidate documents is provided by the organizers, we do not aim to create a global corpus retriever. Instead, our architecture focuses on post-retrieval optimization to rank and sanitize the provided payload.

We first segment documents into overlapping semantic chunks. Each chunk is enriched with its parent document’s metadata to enable proper attribution. We then execute a second-stage retrieval over these chunks using a hybrid retriever. Our retriever uses a combination of BM25 and BGE (bge-base-en-v1.5) dense vector embeddings to rank the chunks. The top-\(k\) chunks based on the input query are fed into the context window of the generator.

3.3 Answer Generation and Post-Hoc Correction↩︎

We evaluate three different generative pipelines against the LongEval Organizers’ naive baseline.

3.3.1 DS@GT Hybrid Baseline↩︎

This architecture represents a standard RAG implementation. The top-\(k\) chunks retrieved by the BM25+BGE hybrid ranker are injected directly into the prompt for the Gemma-4-31B instruction-tuned model. The model synthesizes an initial answer and outputs an array of cited document indices.

3.3.2 Corrective RAG and CiteFix↩︎

Our primary research pipeline introduces active interventions to the Hybrid Baseline.

1. Pre-Generation (CRAG): Before generation, a lightweight cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2) evaluates retrieved chunks. Chunks below the entailment threshold are dropped from the supplied context.

2. Post-Generation (CiteFix): After generation, we use CiteFix to parse generated claims against cited document chunks. Citations lacking entailment to support the generated claims are pruned. Figure 2 shows the steps of the CiteFix pipeline.

3.3.3 Frontier Model Benchmarks↩︎

To establish an upper bound on long-context parametric reasoning, we bypass scoring and chunking entirely. We prompt ChatGPT-5.5 Thinking and Claude Opus 4.7 Thinking to synthesize answers directly from the raw text of the 10 candidate documents.

3.4 Reference-Free RAGAs Evaluation↩︎

In the absence of a gold QA set with ground truth answers in the development phase, we utilized an LLM-as-judge setup (Claude Sonnet 4.6) via the RAGAs framework [10]. The evaluation calculated the faithfulness and relevancy metrics using multi-step prompting algorithms with the judge LLM. The scoring workflows are presented in Figures 3 and 4.

3.4.0.1 Global Faithfulness

This metric measures the factual consistency of the generated response against the 10 pre-retrieved documents. Using RAGAs, the generated answer is decomposed into discrete, standalone claims. The judge LLM (Claude Sonnet 4.6) then decides if claims in the generated answer are grounded within the entirety of the supplied documents. The final metric is the ratio of grounded statements to total statements in answers on the evaluation set.

3.4.0.2 Citation Faithfulness

While Global Faithfulness evaluates the entire task context, Citation Faithfulness isolates the documents claimed to have been used at generation. During this evaluation pass, answer claims are evaluated against only cited documents. Large gaps between Global Faithfulness and Citation Faithfulness reveal a failure mode where grounded answers are produced but falsify or otherwise hallucinate the citation.

3.4.0.3 Answer Relevancy

This metric quantifies the completeness of the answer with respect to the query. Rather than use semantic similarity of question and answer, this metric uses the judge model to produce counterfactual queries based on only the generated answer. The similarity of the generated queries and the original query is then used to score each answer. This metric rewards query-aligned answers, but systematically penalizes safe abstention when the generator receives no context or context not relevant to the original query.

3.5 Official Task Evaluation Metrics↩︎

While we performed our own suite of evaluation tests, the LongEval Organizers also evaluated submissions against a hidden gold set of answers. The headline metrics evaluate output quality using ROUGE and BERT scores. ROUGE measures structural lexical overlap by calculating the longest common subsequence (LCS) and n-gram intersection between submitted and gold answers. Conversely, BERT Score evaluates semantic equivalence by calculating cosine similarity of token embeddings. Higher metric scores indicate the generated answer shares higher lexical or semantic similarity with the gold answer.

Additionally, the organizers calculate citation precision, which evaluates whether the documents cited by our pipeline match those selected in the gold set. Let \(C_q\) be the set of documents cited by our pipeline for a given user query \(q\), and let \(R_q\) be the set of documents labeled relevant in the gold standard. Citation precision calculates the ratio of correctly cited relevant documents to the total number of cited documents (\(|C_q \cap R_q| / |C_q|\)). The final score is averaged over all queries. The organizers provide detailed metrics covering natural language coherency, consistency, and clarity, as well as a variety of other retrieval heuristics measuring the prompt in comparison to the input query. The full list of the results including the additional IR axiom metrics can be viewed in the Appendix.

4 Results↩︎

Here, we report the official LongEval-RAG evaluation metrics and our own reference-free RAGAs diagnostics for each of our pipelines.

4.1 Official Task Metrics↩︎

Tables 1 and 2 detail aggregate performance of the pipelines on the official task metrics. The frontier model runs establish a high upper bound for natural language generation. GPT-5.5 achieved the highest similarity scores with a \(\text{ROUGE}_{F1}\) of 0.188 and a \(\text{BERT}_{F1}\) of 0.227. Claude Opus 4.7 achieved a perfect Citation Precision (1.000) and the highest Nugget Coverage (0.443), the best performance in selecting the correct document IDs from each pre-retrieved set of documents.

We see that our hybrid chunk reranking benchmark significantly outperformed the Organizer’s Naive Baseline across all lexical and semantic metrics. Our corrective intervention pipeline using CRAG+CiteFix experienced a drop in both ROUGE and BERT scores compared to the hybrid baseline. This is an artifact of the abstention behavior which suppresses generation of ungrounded text.

6pt

Table 3 shows system performance on secondary metrics evaluating sentence-level consistency and clarity based on TREC-AutoJudge Axioms [11], [12]. Claude Opus 4.7 produced the most coherent output (\(\text{COH1} = 0.857\)). The Naive Baseline scored well on CLAR2, indicating high readability despite poor accuracy.

6pt

6pt

6pt

4.2 RAGAs Evaluation Metrics↩︎

Table 4 shows the results of our reference-free RAGAs evaluation. These metrics evaluate strictly for attribution integrity and grounding of claims rather than lexical overlaps or semantic similarity. Our CRAG+CiteFix pipeline maximized faithfulness to the supplied context, achieving a Global Faithfulness of 0.784 and a Citation Faithfulness of 0.758. Conversely, the frontier models performed quite poorly on these evaluations. Faithfulness scores for both GPT-5.5 Thinking and Claude Opus 4.7 Thinking collapsed despite producing answers that were highly relevant to the queries. This indicates a failure mode where the models output highly relevant answers without extracting their claims from the cited documents.

5 Discussion↩︎

We observe a tradeoff between several performance metrics and our RAGAs-based faithfulness and citation accuracy evaluation when comparing structured corrective pipelines with unconstrained frontier models. While the frontier models (GPT-5.5, Opus 4.7) achieved higher official leaderboard scores, our RAGAs diagnostics (Table 4) indicate a failure to ground the answers in the retrieved context. Faced with noisy context, the frontier models generated answers predominantly from pre-trained knowledge.

For example, GPT-5.5 generally led the leaderboard scores and had the highest Answer Relevancy of 0.756, but exhibited extremely low Citation Faithfulness at 0.417 indicating low reliance on cited contexts in answers. From the leaderboard precision of 0.904 and manual trace audits, we find that while GPT-5.5 is highly capable of identifying the relevant document(s) within a noisy context window, the details of generated answers typically did not rely on that same context.

The corrective pipeline achieved Global Faithfulness of 0.784 and Citation Faithfulness of 0.758, which were leading on our submissions. However, this triggered penalties on some of the leaderboard metrics. When retrieved chunks lacked the answer, our corrective pipeline would abstain to answer rather than generate an answer from parametric memory. These abstentions are penalized on metrics reliant on relevancy or n-gram overlaps despite exhibiting a more faithful failure mode than parametric memory leaks. This highlights a challenge in trustworthy RAG benchmarking where chosen evaluation metrics may reward failure modes which hallucinate citations to maintain conversational fluency over safer abstentions.

To evaluate the statistical significance of the CRAG + CiteFix pipeline, we conducted a Wilcoxon signed-rank test on the paired RAGAs evaluation scores across the 47 test queries. The results indicated that the corrective interventions did not yield a statistically significant improvement to citation faithfulness or answer relevancy.

6 Future Work↩︎

A few limitations to our evaluation should be noted. While our frontier model submissions score well on lexical similarity metrics, they score poorly on faithfully grounding answers in the retrieved evidence. Conversely, our Gemma-4-31B pipelines exhibit relatively high faithfulness but much lower relevance and citation precision. As our prompting strategy did not require answers to be strictly extractive, instead requesting synthesis across relevant passages, we posit that forcing output text to be extractive would greatly improve lexical similarity and potentially induce more faithful answering from the higher capacity models, if at a penalty to clarity and coherence.

An immediate next step for this work is to verify that the CRAG and CiteFix corrective interventions successfully improve answer grounding across families and sizes of generators. Our initial results do not indicate a statistically significant increase in faithfulness over the baseline. Further exploration of NLI model use for scientific claim verification is needed to understand if the null result is driven by NLI brittleness rather than applicability of post-hoc pruning in this domain. Additionally, the lack of a CRAG+CiteFix submission to the leaderboard using GPT-5.5 Thinking or Claude Opus 4.7 Thinking was due to practical submission limit and time constraints within the evaluation period. We encourage release of the hidden gold set query answers to facilitate expanded ablation studies.

Our RAGAs evaluations are not without limitations. A single, uncalibrated reference-free LLM judge was used to evaluate answer faithfulness and relevancy. Future evaluations will deploy multi-judge ensembles, enabling the evaluation of intraclass correlations across judges and mitigation of known evaluator biases such as self-preference or verbosity.

Finally, the described RAG systems and RAGAs evaluators should be extended across multiple snapshots within the LongEval corpus. The current evaluation focused on a static payload. Applying these pipelines across a longitudinal dataset would assess QA performance under the types of temporal drift which penalize use of point-in-time parametric memory. Additionally, the threshold hyperparameters for CRAG and CiteFix are tuned for a specific generation task. The performance of these techniques over multiple snapshots may degrade, similar to standard retrievers and generators. Longitudinal evaluation of these systems should explore adaptive tuning of the active interventions, as well as assess how corpus shifts impact the faithfulness and relevancy judgments of the underlying RAGAs LLM judge. As our faithfulness and relevancy metrics are tied to an LLM-as-a-judge method, temporal bias should be evaluated and compared to other known judge biases when interpreting longitudinal RAGAs scores.

7 Conclusions↩︎

We find a divergence between traditional language metrics and factual grounding in RAG. While unconstrained frontier models generally achieved better ROUGE and BERT score with query-aligned answers, our diagnostics show this comes from utilization of parametric memory rather than retrieved context. Conversely, we find that deployment of active interventions pre- and post-generation do improve structural honesty in RAG QA systems. Ultimately, securing RAG QA pipelines in scientific domains rely on both strict grounding to evidence and overall quality of answer. We therefore recommend adoption of evaluation metrics that reward structural safety and proper failure modes within RAG systems.

Acknowledgments↩︎

We thank the Data Science at Georgia Tech Applied Research Competitions (DS@GT ARC) group for their support. This research was supported in part through research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA [13].

Declaration on Generative AI↩︎

During the preparation of this work, the authors used Gemini 3.1 Pro in order to aid in formatting, grammar, and spell checking. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.

8 Appendix↩︎

8.1 Answer Generation Prompt↩︎

For each query, the LLM receives both the user scientific query and the CRAG-filtered evidence block for the research based approaches (excluded for baseline). The prompt instructs the model to produce a concise, paraphrased answer that synthesizes evidence from the retrieved document chunks without directly copying. The exact prompt template used is below:

Question: {query}

CRAG-filtered evidence:
{evidence_block}

Write a direct answer in 1-3 sentences, about 30-65 words total.

Rules:
- No bullets, markdown, or inline citations.
- Do not mention passage numbers or evidence labels.
- Synthesize the answer rather than copying source text.
- Even if evidence is partial, give the strongest concise answer
  supported by the evidence.
- Write in English only.
- Use standard English spellings for chemical names and technical terms.

8.2 TREC-AutoJudge IR Axiom Results↩︎

4pt

4pt

Tables 5 and 6 detail our submission results on TREC-AutoJudge IR Axioms. Notably, the organizers’ naive baseline performed best across several proximity and intersection-based metrics. We hypothesize that the performance gap is an artifact of the divergence between extractive and abstractive generation methods. Classical IR axioms like GEN-AND and GEN-PROX4 reward lexical overlap and term proximity. An extractive baseline will inherently satisfy these conditions. Our RAG pipelines were prompted to perform abstractive synthesis across the supplied context. This process can break lexical overlaps and alter term distances, resulting in penalty by these axioms. We acknowledge that stricter extractive prompting could improve scores on these axioms.

References↩︎