1 Introduction↩︎

Retrieval-Augmented Generation (RAG) pipelines improve the factual grounding of large language models by conditioning generation on retrieved documents [1]. A critical component of high-quality RAG is the reranker: a model that reorders an initial set of retrieved candidates to surface the most relevant documents before generation. Cross-encoders have long been the dominant reranking architecture due to their joint query-document encoding, which enables fine-grained relevance scoring [2]. However, their quadratic inference complexity makes them prohibitively expensive at scale.

Recent advances in large language models (LLMs) suggest an alternative: instruction-tuned LLMs can be adapted to scoring and ranking tasks through fine-tuning, potentially matching cross-encoder accuracy at lower inference cost when combined with quantization. This direction has received growing attention following the release of open-weight models such as LLaMA [3] and the development of parameter-efficient fine-tuning methods such as LoRA [4].

We address two problems simultaneously. First, cross-encoders are expensive: full transformer passes for every query-document pair make them unsuitable for latency-sensitive applications. Second, large LLMs are expensive in a different way: their parameter counts make deployment costly without compression. We resolve both through a fine-tuning and quantization pipeline that produces a compact, accurate reranker from LLaMA 3 8B.

Our contributions are:

• A fine-tuning pipeline for adapting LLaMA 3 8B as a reranker using the Unsloth framework with LoRA adapters and 4-bit quantization, requiring no full-model retraining.

• A dual-retriever RAG integration combining BM25 and dense vector retrieval, with the fine-tuned LLaMA 3 reranker replacing the cross-encoder at the reranking stage.

• An empirical evaluation using the RAGAS framework demonstrating consistent improvements over a cross-encoder baseline across four retrieval quality metrics.

• A reusable LoRA adapter checkpoint enabling direct inference without retraining.

2 Related Work↩︎

2.1 Cross-Encoders for Reranking↩︎

Cross-encoders jointly encode a query and a candidate document, producing a scalar relevance score via a classification head on the [CLS] token [2]. Nogueira and Cho [2] demonstrated that BERT fine-tuned for passage reranking substantially outperforms BM25 alone. Subsequent work confirmed that cross-encoders remain competitive on multi-stage retrieval benchmarks [5], but their \(O(n)\) forward passes per query (one per candidate document) limit throughput.

2.2 LLMs as Rerankers↩︎

The monoT5 model [6] fine-tunes a T5 encoder-decoder to produce relevance judgments from query-document pairs, demonstrating that generative models can be adapted for reranking. RankGPT [7] uses GPT models in a listwise reranking setting via prompting, showing strong zero-shot performance but at high inference cost. Fine-tuning smaller open-weight LLMs offers a middle ground: lower cost than GPT-scale inference, higher accuracy than zero-shot prompting.

2.3 Parameter-Efficient Fine-Tuning↩︎

LoRA [4] introduces low-rank decompositions into attention weight matrices, enabling fine-tuning with a fraction of the trainable parameters. Combined with quantization (QLoRA [8]), this approach enables fine-tuning of 7B–13B parameter models on consumer hardware. We build on these techniques via the Unsloth framework, which provides optimized LoRA training kernels for LLaMA-family models.

2.4 Retrieval-Augmented Generation↩︎

Lewis et al. [1] introduced RAG as a framework combining dense retrieval with sequence-to-sequence generation. Subsequent work has explored hybrid retrieval strategies combining BM25 with dense retrievers [9] to improve recall diversity, and reranking as a post-retrieval step to improve precision. Our work fits into this multi-stage pipeline design.

3 Background↩︎

3.1 Cross-Encoder Architecture↩︎

A cross-encoder takes a concatenated query-document pair as input: \[\texttt{[CLS]}\;q\;\texttt{[SEP]}\;d\;\texttt{[SEP]}\] and passes it through a transformer encoder. The [CLS] token representation is projected to a scalar relevance score: \[\text{score}(q, d) = \sigma(\mathbf{W}_\text{out}\, \mathbf{z}_\text{CLS} + b)\] where \(\mathbf{z}_\text{CLS}\) is the final-layer [CLS] representation, \(\mathbf{W}_\text{out}\) is a learned projection, and \(\sigma\) is a sigmoid or linear activation. Attention scores between token \(i\) and token \(j\) are computed as: \[\text{score}(i, j) = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}}\] with softmax normalization across positions. This joint encoding captures fine-grained query-document interactions but requires one forward pass per candidate document.

3.2 Retrieval-Augmented Generation↩︎

A RAG pipeline operates in two stages. In the retrieval stage, a set of candidate documents \(\mathcal{D} = \{d_1, \ldots, d_k\}\) is retrieved from a corpus given query \(q\), typically using a bi-encoder or keyword-based retriever. In the generation stage, a language model \(p_\theta(a \mid q, \mathcal{D})\) generates an answer conditioned on the query and retrieved context. Reranking sits between these stages: it reorders \(\mathcal{D}\) so that the most relevant documents appear first, improving the quality of the generation context.

4 Methodology↩︎

4.1 Fine-Tuning Pipeline↩︎

We fine-tune LLaMA 3 8B [3] as a reranker using the Unsloth framework with LoRA adapters. The model is initialized in 4-bit precision using the llama-3-8b-bnb-4bit checkpoint.

4.1.0.1 Dataset Construction

We construct a supervised dataset of (query, document, relevance) triples with three fields:

• Input: a query paired with a list of candidate documents to be ranked.

• Prompt: an instruction directing the model to rank documents by relevance to the query (e.g., “Rank the following documents in order of relevance to the query.”).

• Output: the ground-truth relevance ordering or per-document relevance scores.

This instruction format follows the listwise reranking paradigm, allowing the model to produce structured rankings in a single forward pass rather than scoring each document independently.

4.1.0.2 LoRA Configuration

LoRA adapters are applied to the query, key, value, and output projection layers (\(\texttt{q\_proj}\), \(\texttt{k\_proj}\), \(\texttt{v\_proj}\), \(\texttt{o\_proj}\)) with rank \(r = 16\) and scaling factor \(\alpha = 32\). Gradient checkpointing is enabled to reduce memory consumption. Training uses the AdamW optimizer in 8-bit precision with learning rate \(2 \times 10^{-4}\), weight decay \(0.01\), and warmup over the first 10% of steps. Mixed precision (fp16 or bf16 depending on hardware) is used throughout.

4.1.0.3 Quantization

After fine-tuning, the LoRA adapter is merged and the model is saved in 4-bit GGUF format for efficient inference, enabling deployment on hardware that cannot serve full-precision 8B models.

4.2 RAG Pipeline Integration↩︎

The fine-tuned LLaMA 3 reranker is integrated into a multi-stage RAG pipeline:

1. Document ingestion: uploaded documents (PDF, DOCX, TXT) are chunked and embedded using OpenAI’s text-embedding-ada-002 model; embeddings are stored in a Chroma vector database.

2. Dual retrieval: given a query, a vector store retriever retrieves the top-\(k\) documents by cosine similarity, and a BM25 retriever retrieves the top-\(k\) by keyword overlap. Results are ensembled to form a diverse candidate set.

3. Reranking: the fine-tuned LLaMA 3 model reranks the ensembled candidates by relevance to the query.

4. Generation: the top-ranked documents are passed as context to GPT-4o, which generates the final answer.

In the baseline pipeline, step 3 uses a standard BERT-based cross-encoder instead of the fine-tuned LLaMA 3 model; all other steps are identical.

5 Experiments↩︎

5.1 Evaluation Framework↩︎

We evaluate both pipelines using RAGAS [10], an automated RAG evaluation framework that computes reference-free metrics by comparing generated answers against ground-truth responses. We report four metrics:

• Answer Relevancy: alignment between the generated answer and the query intent.

• Context Precision: fraction of retrieved context that is relevant to the query.

• Answer Similarity: semantic similarity between generated and ground-truth answers.

• Answer Correctness: factual accuracy of the generated answer relative to ground truth.

5.2 Experimental Setup↩︎

We evaluate on a domain-specific question-answering dataset comprising queries about academic program details, policies, and course requirements. Documents are sourced from institutional PDF and DOCX files. Ground-truth answers were manually annotated. Both pipelines use identical retrieval configurations (\(k = 5\) for each retriever), identical embedding models, and the same GPT-4o generation model. The only variable is the reranker: cross-encoder versus fine-tuned LLaMA 3.

5.3 Results↩︎

Table 1 summarizes the results. The fine-tuned LLaMA 3 reranker outperforms the cross-encoder baseline across all four metrics. Context Precision improves by 16% (0.75 \(\to\) 0.87), indicating that the LLM-based reranker more accurately surfaces relevant documents and reduces noise in the generation context. Answer Correctness improves by 21% (0.70 \(\to\) 0.85), the largest absolute gain, suggesting that improved context quality directly translates to more factually accurate generation. Answer Relevancy and Answer Similarity each improve by 14% and 19% respectively.

5.4 Analysis↩︎

The performance gains are consistent across all metrics, indicating that the improvements are not an artifact of a single metric’s definition. The largest gains in Answer Correctness and Context Precision suggest that the fine-tuned LLaMA 3 reranker is more effective at identifying documents that contain factually relevant information, rather than merely semantically similar text. This is consistent with the hypothesis that instruction-tuned LLMs have richer world-knowledge representations than encoder-only cross-encoders, which may help them distinguish informative from superficially relevant documents.

The 4-bit quantized model retains full fine-tuned accuracy while reducing memory requirements, making deployment practical on standard inference hardware.

5.5 Limitations↩︎

The evaluation dataset is domain-specific and relatively small; generalization to open-domain QA benchmarks such as MS-MARCO [11] or BEIR [12] remains to be demonstrated. The comparison is limited to a single cross-encoder baseline; a broader comparison against monoT5 [6] and RankGPT [7] would strengthen the empirical claims. Latency measurements are not reported; a throughput comparison between the cross-encoder and the quantized LLaMA 3 reranker would be valuable for deployment decisions.

6 Conclusion↩︎

We presented a fine-tuning pipeline for adapting LLaMA 3 8B as a drop-in reranker in RAG pipelines, using LoRA adapters and 4-bit quantization via the Unsloth framework. Integrated into a dual-retriever RAG pipeline with BM25 and dense vector retrieval, the fine-tuned model outperforms a BERT-based cross-encoder baseline across all RAGAS metrics, with gains of 14–21% depending on the metric. The resulting LoRA adapter enables direct inference without retraining, lowering the barrier to deployment.

6.0.0.1 Future Work

Natural extensions include: (i) evaluation on standard open-domain benchmarks (MS-MARCO, BEIR) to assess generalization; (ii) latency benchmarking against cross-encoders to quantify the inference efficiency gain; (iii) comparison with listwise rerankers such as RankGPT; and (iv) exploration of smaller base models (LLaMA 3 1B, 3B) to further reduce deployment cost.

References↩︎