1 Introduction↩︎

Large language models (LLMs) have undergone rapid development in recent years, significantly enhancing performance across various language tasks due to their emergent capabilities in semantic understanding and reasoning. Nevertheless, LLMs still face challenges in updating knowledge and providing factual responses [1]. To address these issues, Retrieval-Augmented Generation (RAG) has emerged as a promising approach. By retrieving the most relevant documents from external knowledge bases and prepending them as contextual information to the original input, RAG substantially improves LLM performance on knowledge-intensive tasks [2].

While RAG enhances performance, its effectiveness is closely tied to the number of retrieved documents used, since a broader context increases the probability of encompassing critical evidence. As illustrated in Figure 1, performance was weakest without any retrieved documents (i.e., without RAG). Accuracy improved consistently as more documents were added to the context, ultimately exceeding the no-RAG baseline by over 10 Exact Match (EM) points. However, this performance gain came with two significant limitations: (1) a substantial increase in computational cost from processing a larger number of context tokens [3], and (2) the model’s difficulty in effectively leveraging all provided documents, often resulting in the omission of key information located in the middle of the context [4].

These limitations have motivated recent research efforts aimed at compressing the retrieved context [5]–[8]. Prominent approaches include document summarization [3], key information extraction [3], [9], the construction of key supporting evidence [5], and noise filtering based on information theory [10]. Despite recent progress, these methods have several notable shortcomings. First, compression often results in a performance trade-off. For instance, RECOMP [3] suffers a 3–5 point drop in EM score (Table 1), making it unsuitable for accuracy-sensitive applications. Second, most compression methods are heuristic in nature. The models are typically trained to generate summaries that are generally good but not necessarily useful for the downstream answer-generation LLM. This limitation arises from the lack of an ideal supervisory signal that defines what an optimal summary should be for the downstream task. This fundamental gap hinders the end-to-end optimization of existing methods. Finally, some compression models [10] have parameter counts comparable to the LLM that performs the end task, resulting in substantial computational costs that undermine the efficiency gains of compression.

Addressing these critical shortcomings requires a method that aligns compression with downstream task requirements, thereby minimizing performance trade-offs. To bridge this gap, we propose CORE, a novel method designed to achieve lossless context compression for RAG. Unlike previous compression methods, CORE is optimized in an end-to-end manner. Since obtaining predefined summary labels for supervision is impractical, we instead use downstream task performance as a feedback signal to evaluate the compression model’s output. This feedback enables iterative refinement of the compression policy, guiding the compression model toward improved downstream performance. To this end, we employ Group Relative Policy Optimization (GRPO), a technique particularly well-suited for this purpose [11]–[13]. In our framework, the accuracy of the downstream QA task is formalized as a reward, with the compression policy optimized through group-wise relative comparisons. Furthermore, our compression model is substantially smaller than the downstream LLM, which significantly reduces the computational overhead associated with encoding retrieved documents.

We evaluate CORE on four benchmark datasets: two single-hop QA datasets (Natural Questions and TriviaQA) and two multi-hop datasets (HotpotQA and 2WikiMultihopQA). As shown in Table 1, CORE achieves state-of-the-art performance across all baselines. With a compression ratio of 3%, our approach not only avoids performance degradation compared to prepending full documents but also improves the average EM score by 3.3 points. We further demonstrate two key advantages of our approach. First, the effectiveness of CORE is not tied to a specific model architecture, as demonstrated by the fact that lossless compression can be achieved when various models are trained as compressors (Figure 3). Second, the compressor exhibits strong transferability: a compression model trained using feedback from one LLM generalizes effectively to other LLMs (Table 2). Finally, an in-depth case study (Tables 3 and [tab:case2]) provides a qualitative analysis of the benefits of CORE.

2 CORE-RAG↩︎

This section introduces our proposed method, COmpression via REinforcement learning (CORE), which is shown in Figure 2. First, we provide an overview of the entire workflow. Then, we detail the end-to-end training strategy for the compression model, which is designed to drastically reduce the number of document tokens while preserving task performance.

2.1 Problem Formulation↩︎

We adopt the same problem formulation as prior work [3]. Given an input question \(q\), a target output \(y\), and a set of \(k\) retrieved documents \(D\), our objective is to compress \(D\) with respect to \(q\) into a summary \(s\) that preserves the most useful information while using significantly fewer tokens than \(D\). This summary \(s\) is then prepended to the original input \(q\) and fed into an LLM to generate the final answer for the downstream task. This overall pipeline is illustrated in the upper half of Figure 2. The process involves two key components: a compression model \(\pi_\theta \colon (q, D) \mapsto s\) and a large language model \(M \colon (s, q) \mapsto \hat{y}\), which generates the predicted answer \(\hat{y}\). We treat \(M\) as a black-box system and focus exclusively on training the compressor \(\pi_\theta\). The compressor itself is also a language model, but it is intentionally designed to be significantly smaller than \(M\) to reduce the computational cost of encoding the retrieved documents.

2.2 Training the Compressor↩︎

Our compressor is designed to generate document summaries that are highly useful to the LLM (\(M\)) for downstream tasks. This objective is challenging because the criteria for an effective summary are task-dependent, and direct supervision is unavailable. We therefore formulate this as an end-to-end training problem and employ reinforcement learning to optimize the compressor without relying on pre-defined compression labels. The overall architecture of our training framework is illustrated in the lower portion of Figure 2. The following sections describe the key components of our approach: distillation warm-up, policy optimization, and reward calculation.

2.2.1 Distillation for Warm-Start↩︎

Due to the limited parameter size of our compression model, its capability for question-focused document summarization is constrained. To provide a strong initial policy for RL and ensure training stability, we employ knowledge distillation from a teacher model to initialize our compressor. Specifically, we first utilize a large-scale language model (DeepSeek-V3) as the teacher to generate summaries of retrieved documents related to the given question. We then evaluate the performance of the downstream LLM (\(M\)) on the QA task under two conditions: (1) with the teacher-generated summary \(\hat{s}\) prepended to the input question \({q}\) , and (2) with the original question alone. The corresponding performance scores are denoted as \(p_{\text{summary}}\) and \(p_{\text{original}}\), respectively. By comparing these results, we retain instances where \(p_{\text{summary}} > p_{\text{original}}\), indicating that the summary enhances RAG performance. We also retain cases where \(p_{\text{original}} = 1\) (i.e., the model produces a fully correct answer without the summary) and \(p_{\text{summary}} < p_{\text{original}}\); for these, we set the target summary \(\hat{s}\) to an empty string. All other instances are discarded. The resulting filtered and modified dataset is denoted as \(\mathcal{X}_{f}\), which is used for supervised fine-tuning of the compression model. The fine-tuning objective is defined as: \[\mathcal{L}_{\text{distill}} = \frac{1}{|\mathcal{X}_{f}|} \sum_{(q, D, \hat{s}) \in \mathcal{X}_{f}} \mathcal{L}_{\text{CE}}(\pi_\theta(q, D), \hat{s}),\] where \(\pi_\theta(q, D)\) denotes the output of the compression model and \(\mathcal{L}_{\text{CE}}\) is the cross-entropy loss. This distillation process yields a robust initialization for RL and promotes stability in subsequent training.

2.2.2 End-to-End Training with RL↩︎

Following the distillation phase, the compressor possesses a preliminary compression capability. However, as summaries from even the largest teacher models are not guaranteed to be optimal for the downstream task, further end-to-end optimization is necessary. We therefore formulate this optimization as a reinforcement learning problem. In this framework, the compressor functions as a policy that generates a summary from an input question and its corresponding documents. A reward function, which directly reflects performance on the downstream task (e.g., question-answering accuracy), then evaluates the summary’s quality. The objective is to optimize the compressor’s parameters to maximize the expected cumulative reward, thereby directly aligning its outputs with the downstream task’s objectives.

Specifically, we employ Group Relative Policy Optimization (GRPO) [12] algorithm. Unlike Proximal Policy Optimization (PPO), which trains a separate critic model, GRPO estimates the baseline directly from a group of rollouts. Given an existing policy, \(\pi_{\theta_{\text{old}}}\), and a reference policy, \(\pi_{\theta_{\text{ref}}}\), the GRPO objective maximizes the compressor policy \(\pi_\theta\) using \(G\) rollouts \(\tau = \{s_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|x)\), for each input \(x \sim \mathcal{D}\): \[\begin{align} \mathcal{J}(\theta) &= {\mathbb{E}}_{x \sim \mathcal{D},\;\{s_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \\ & \frac{1}{G} \sum_{i=1}^G \Bigg[ \min \Bigg( \frac{\pi_\theta(s_i|x)}{\pi_{\theta_{\text{old}}}(s_i|x)} A_i, \;\text{clip} \Bigg( \frac{\pi_\theta(s_i|x)}{\pi_{\theta_{\text{old}}}(s_i|x)}, 1-\epsilon, 1+\epsilon \Bigg) A_i \Bigg) \notag - \beta \mathbb{D}_{\text{KL}} \big( \pi_\theta \parallel \pi_{\theta_{\text{ref}}} \big) \Bigg], \end{align}\] where \(A_i = (r_i - \text{mean}(\{r_j\}_{j=1}^G)) / \text{std}(\{r_j\}_{j=1}^G)\) represents the normalized advantage of the \(i\)-th rollout within the group, \(\epsilon\) is the clipping ratio, and \(\beta\) is the coefficient for the KL divergence penalty. The inclusion of the KL divergence term ensures that the updated policy does not deviate significantly from the reference policy.

2.2.3 Reward Calculation↩︎

2.2.4 Training Template↩︎

Figure 5 displays the prompt employed to train the compressor model for generating a summary of the retrieved documents, conditioned on the given question. This prompt is notably concise. For end-task answer generation, the prompt provided to the LLM \(M\) is illustrated in Figure 6, which incorporates few-shot in-context examples, the (generated summary of) retrieved documents, and the question.

3 Experiments↩︎

3.1 Experimental Settings↩︎

3.2 Overall Performance↩︎

The detailed comparison results are presented in Table 1. We evaluate the following approaches: traditional RAG without compression using the top 1, 3, 5, and 10 documents prepended to the original input; BM25-based compression; off-the-shelf LLMs (Qwen2.5-1.5B-Instruct and DeepSeek-V3); and state-of-the-art trainable RAG compressors (RECOMP-Abstractive, RECOMP-Extractive, and NoiseFilter-IB). For fair comparison, all trainable methods were trained using the same backbone model, Qwen2.5-1.5B-Instruct. These compressors were trained on five-document inputs, and we report their performance both on in-domain five-document compression and out-of-domain generalization to ten-document compression.

3.3 Robustness and Generalization Ability Analysis↩︎

3.4 Ablation Study↩︎

Table 4 presents an ablation study on the two stages of our method: distillation and GRPO. Here, “w/o distillation” denotes training the compressor with GRPO directly, bypassing the warm-start phase, while “w/o GRPO” indicates using only the distillation step without subsequent reinforcement learning. The results demonstrate that removing either component leads to performance degradation, confirming the necessity of both. In addition, the decline is more substantial when GRPO is omitted, highlighting the crucial role of reinforcement learning in the absence of explicit supervision. Distillation injects external knowledge into the model, providing a favorable starting point for RL training and thereby enabling RL to more effectively unlock the compressor’s full potential.

3.5 The Impact of \(\alpha\).↩︎

Figure [fig:hyperparam] illustrates the performance of our method under different values of \(\alpha\), which controls the weighting coefficient of the F1 reward term. Setting \(\alpha = 0\) corresponds to using only the EM reward. Results indicate that performance improves when \(\alpha > 0\) across all datasets, demonstrating the effectiveness of the F1 reward in mitigating the sparsity issue associated with the EM reward. However, the optimal value of \(\alpha\) is dataset-dependent; values between 0.2 and 0.5 generally yield strong performance.

3.6 Case Study↩︎

To conduct an in-depth analysis of the advantages of our compressor, we performed case studies on one single-hop QA dataset (NQ) and one multi-hop QA dataset (2Wiki), with the results presented in Table 3 and Table [tab:case2], respectively. For each case, we compared the summaries generated by off-the-shelf Qwen2.5-1.5B-Instruct, RECOMP, and our method CORE based on the same set of documents, as well as the predicted answers generated by the LLM after prepending these summaries. As shown in the tables, although the summaries produced by off-the-shelf Qwen2.5-1.5B are concise, they largely fail to capture key information relevant to answering the question. In contrast, RECOMP demonstrates better summarization capability but is prone to being overwhelmed by lengthy documents, resulting in misjudgments and even generating misleading information—such as the statement in Table 3: “The U.S. stopped drafting for the Vietnam War after the Selective Service System was officially abolished in December 1972”—which leads the downstream LLM to produce the incorrect answer “1972”. Our method, CORE, accurately extracts answer-critical information from lengthy documents, exemplified by the summary: “The U.S. stopped drafting for the Vietnam War in 1973 after announcing the decision by Secretary of Defense Melvin Laird earlier that year”, thereby enabling the LLM to generate the correct answer “1973”. This indicates that our compressor, trained with an end-task target-oriented reinforcement learning optimization strategy, can produce document summaries that are most helpful for answering the given question while effectively filtering out irrelevant information.

4 Related Work↩︎

5 Conclusion↩︎

This paper analyzes the limitations of current context compression methods for RAG. A primary challenge is the lack of optimal reference summaries for supervised learning, which often results in performance degradation in downstream tasks. To overcome this, we frame context compression as a reinforcement learning problem, utilizing downstream task performance as a reward signal to train the compression policy, thereby enabling end-to-end optimization. Extensive experiments demonstrate that our proposed method, CORE, achieves effectively lossless compression by maintaining a high compression ratio while preserving original task performance. Surprisingly, CORE not only preserves but actually enhances performance on all downstream tasks. Further in-depth analysis provides additional insights into the efficacy of our approach.

6 Use of LLMs↩︎

Large language models (LLMs) were employed solely as writing assistants to enhance the language, improve clarity, and check grammatical correctness. They were not used to generate research ideas, design or implement methodologies, conduct data analysis, or produce any of the results presented in this work. The authors assume full responsibility for the entire content of the paper.

7 Prompt Templates↩︎

Figure 5 displays the prompt employed to train the compressor model for generating a summary of the retrieved documents, conditioned on the given query. This prompt is notably concise. For end-task answer generation, the prompt provided to the LLM \(M\) is illustrated in Figure 6, which incorporates few-shot in-context examples, the (generated summary of) retrieved documents, and the question.

8 Case Study↩︎

To conduct an in-depth analysis of the advantages of our compressor, we performed case studies on one single-hop QA dataset (NQ) and one multi-hop QA dataset (2Wiki), with the results presented in Table 3 and Table [tab:case2], respectively. For each case, we compared the summaries generated by off-the-shelf Qwen2.5-1.5B-Instruct, RECOMP, and our method CORE based on the same set of documents, as well as the predicted answers generated by the LLM after prepending these summaries.

9 Impact of Different Compressors on Performance↩︎

In our previous experiments, we employed Qwen2.5-1.5B as the initial model to train our compressor. In this section, we utilize two additional models—Llama3.2-1B and Llama3.2-3B—as starting points to train our compressor and the baseline compressor, respectively. The experimental results are presented in Table 4 and Table 5. As shown in the results, our method CORE continues to achieve lossless compression with both models, maintaining a high token compression ratio while exhibiting no performance degradation in terms of Exact Match (EM) and F1 score compared to uncompressed RAG. Furthermore, under both new model configurations, our approach consistently outperforms the baseline methods, indicating that its superiority is not dependent on a specific model architecture and thus demonstrates strong robustness.

We also observe that our method adheres to a form of scaling law: the compressor trained using the 3B model outperforms the one trained with the 1B model. Specifically, the 1B compressor improves performance by 1–4 EM points over the uncompressed baseline, while the 3B compressor yields gains of 3–9 EM points.

References↩︎