1 Introduction↩︎

Retrieval-augmented generation (RAG) grounds LLM outputs in external knowledge [1], but performance depends on tightly coupled choices such as chunker, query rewriting, retrieval depth, reranking, and context compression [2]. These choices are often set through heuristics or trial-and-error tuning, making RAG optimization costly, difficult to reproduce, and challenging to compare across systems.

Recent work has explored RAG configuration through AutoRAG [3] and hyperparameter impact analysis [2], adaptive retrieval through Self-RAG [4] and Adaptive-RAG [5], and automated tuning through AutoRAG-HP [6] and RAG-HPO analyses [7]. However, these efforts use different pipelines, datasets, search spaces, optimization methods, and evaluation protocols, leaving the field without a shared framework for comparing optimization algorithms under controlled conditions. Such a framework should expose search spaces explicitly, standardize evaluation protocols, and support matched computational budgets. It should also remain extensible to emerging optimization algorithms and benchmark tasks, similar to standardized environments used in hyperparameter optimization research such as HPOBench [8] and YAHPO Gym [9].

In this paper, we formulate RAG design as an architecture search problem. Instead of treating RAG tuning as an implementation detail, we formulate it as the selection of pipeline configurations that optimize end-to-end RAG performance under a fixed evaluation budget. This perspective connects RAG system design with random search [10] and Bayesian optimization [11], while retaining challenges specific to RAG: heterogeneous modules, coupled design choices, multimodal pipelines, and task-dependent objective landscapes.

To enable systematic study of this problem, we introduce the RAG Intelligence Search Engine (RAISE), a comprehensive framework and benchmark for RAG hyperparameter optimization (Figure 1). RAISE makes the problem reproducible and comparable across algorithms and settings, while supporting both LLM and multimodal LLM pipelines. The framework is designed to remain extensible beyond the benchmark tasks included in this work. New optimization algorithms can be integrated through a common controller API, while new benchmark tasks can be added by specifying datasets, corpora, and evaluation protocols without changes to the core pipeline implementation. We benchmark 13 optimization algorithms on seven environments: TriviaQA [12], HotpotQA [13], MS MARCO [14], ScienceQA [15], SQuAD v2 [16], LongBench-Multifield [17], and LongBench-Qasper [18], with three seeds per setting.

We further conduct three-seed module ablations to identify which pipeline choices account for optimization gains under different task requirements. The results show that module effects vary with task structure: long-document retrieval benefits from query rewriting and pruning, while multi-hop reasoning relies more on retrieval depth. Likewise, long-context settings are especially sensitive to retrieval-depth control. These findings show that RAISE is not only a software framework but also a common experimental basis for studying RAG architecture search – an open problem we hope will attract broader research attention.

In summary, our contributions are as follows:

• We establish RAG architecture search as a benchmark setting, enabling existing and future optimization algorithms to be studied on end-to-end RAG pipelines through RAISE’s unified interface. The project code is available at https://github.com/family99chen/RAISE.

• We instantiate RAISE with a shared end-to-end RAG search space, public benchmark datasets, and a unified evaluation protocol. The benchmark spans query rewriting, chunking, retrieval, reranking, pruning, and generation, exposing heterogeneous task-specific optimization challenges under matched computational budgets.

• We conduct a controlled three-seed evaluation of 13 optimization algorithms across seven text and multimodal datasets, showing that optimizer performance is strongly environment-dependent. Our results motivate reporting RAG architecture search as optimizer–environment interactions rather than a universal leaderboard.

2 Related Work↩︎

2.0.0.1 RAG architectures and adaptive retrieval.

Retrieval-augmented generation (RAG) was introduced as a means of combining parametric language models with external retrieval [1]. Subsequent work has shown that retrieval should often be adaptive rather than fixed: Self-RAG enables a model to decide when to retrieve and how to critique retrieved evidence during generation [4], while Adaptive-RAG selects retrieval strategies based on question complexity [5]. Such studies confirm that end-to-end RAG quality hinges on retrieval and control decisions. Their focus, however, lies in designing or learning particular RAG architectures, rather than in general optimization over configurable RAG pipelines.

2.0.0.2 RAG hyperparameter optimization and benchmark environments.

Several recent studies have begun to investigate hyperparameter optimization for RAG. AutoRAG-HP formulates RAG hyperparameter tuning as an online multi-armed bandit problem and employs a hierarchical controller for online adaptation [6]. [7] compare several hyperparameter optimization methods for RAG across multiple datasets and a large search space. More broadly, our formulation builds on black-box and hyperparameter optimization, including random search [10], Bayesian optimization [11], budget-aware bandit methods such as Hyperband [19], and hybrid approaches such as BOHB [20]. Benchmark suites such as HPOBench [8] and YAHPO Gym [9] further demonstrate the value of standardized environments for reproducible optimizer comparison. RAISE extends this benchmark perspective to end-to-end RAG pipelines, where coupled modules, expensive evaluations, and task-dependent objectives make controlled comparison especially important.

2.0.0.3 Benchmarking and evaluating RAG pipelines.

A complementary line of work investigates how RAG systems should be evaluated. ARES decomposes RAG quality into context relevance, answer faithfulness, and answer relevance [21], while RAGAs provides reference-free metrics for similar dimensions [22]. Benchmarks such as RGB [23] and CRUD-RAG [24] further expose failure modes and component sensitivities in retrieval-augmented systems. Collectively, these studies establish that RAG quality is multi-dimensional and cannot be adequately captured by a single QA score. Our goal is complementary: rather than benchmarking RAG models or evaluation metrics in isolation, we benchmark optimization methods over shared end-to-end RAG configuration spaces under fixed budgets and standardized protocols.

3 Methodology↩︎

We define RAG architecture search as optimizing a parameterized end-to-end RAG pipeline under a fixed evaluation protocol. RAISE realizes this view through three components: a pipeline abstraction that specifies the admissible configurations, an evaluation layer that maps configurations to task-level scores, and a controller interface through which optimization algorithms propose configurations and receive feedback from the environment. This separation lets us vary the search strategy while holding the pipeline family, benchmark data, and scoring rule fixed. We adopt the term optimization algorithm for the search method itself and controller for the same method once instantiated within RAISE.

3.1 Search Space and Pipeline Abstraction↩︎

3pt

RAISE models LLM and multimodal LLM RAG workflows as modular directed acyclic graphs. Formally, the search space \(\Theta\) is the Cartesian product of module-specific configuration spaces: \[\label{eq:theta} \Theta = \Theta_{\text{rewrite}} \times \Theta_{\text{chunk}} \times \Theta_{\text{ret}} \times \Theta_{\text{rerank}} \times \Theta_{\text{prune}} \times \Theta_{\text{gen}}\tag{1}\]

This factorization disentangles query reformulation, chunk granularity, retrieval depth and scoring, reranking, pruning, and generation. LLM pipelines follow Rewriter \(\rightarrow\) Chunker \(\rightarrow\) Text Retriever \(\rightarrow\) Reranker \(\rightarrow\) Pruner \(\rightarrow\) Generator, while multimodal LLM pipelines incorporate CLIP retrieval and omit pruning to preserve cross-modal alignment. The text retriever combines sparse BM25 scoring [25] with dense sentence embeddings from Sentence-BERT [26] and MiniLM [27], and the multimodal branch uses CLIP-aligned text–image embeddings [28]. Detailed module definitions are provided in Appendix 7.1.

3.2 Evaluation Objective↩︎

Given a dataset \(\mathcal{D} = \{(q_i, Y^*_i)\}_{i=1}^{N}\) of queries and references, and a RAG pipeline parameterized by \(\theta \in \Theta\), RAISE formulates the search problem via the following dataset-level objective: \[\label{eq:objective} \theta^* = \arg\max_{\theta \in \Theta} \frac{1}{N} \sum_{i=1}^{N} \mathcal{E}\left( \mathcal{F}_{\theta}(q_i), Y^*_i \right)\tag{2}\] where \(\mathcal{F}_{\theta}(q_i) = Y_i\) denotes the generated response and \(\mathcal{E}\) the evaluation function. Crucially, the optimization target is the full RAG pipeline rather than any module considered in isolation.

RAISE supports lexical, semantic, model-based, and efficiency-oriented signals, allowing \(\mathcal{E}\) to be chosen for the task at hand. Table [tab:instantiated95search95space] gives the search space used in our experiments. In the main benchmark, we use an equal-weight objective over ROUGE-L [29], METEOR [30], token-F1 [16], and BLEU [31], as described in Section 4.2. Full metric definitions are provided in Appendix 7.2.

3.3 Benchmark Environments↩︎

RAISE serves as both a configurable RAG pipeline and a benchmark environment for studying optimization behavior. Each environment is specified by a question–answer set and a retrieval corpus, keeping the interface simple while allowing tasks to differ in evidence structure, answer form, and modality. The suite covers six LLM QA tasks and one multimodal LLM QA task.

3.3.0.1 Benchmark construction.

The benchmark is designed to expose bottlenecks in end-to-end RAG search rather than maximize dataset size. As shown in Table 1, the suite stresses long-document localization, multi-evidence composition, retrieval and reranking, abstention, long-context chunking and pruning, and visual grounding. The environments instantiate the seven datasets in Table 1.

2pt

3.4 Supported Optimization Methods↩︎

RAISE includes 13 preset optimization algorithms grouped into seven descriptive families. The labels in Table 2 reflect the search state and selection rule used by each method, rather than a single canonical HPO taxonomy. We do not aim to tune a single optimizer for RAG. Instead, we use a common interface to expose different search biases. Algorithm-level details are given in Appendix 7.4.

3.5 Controller Interface and Execution↩︎

Optimization algorithms use RAISE through a common controller interface. Let \(\mathcal{A}\) denote a built-in or user-defined optimizer. Under this interface, \(\mathcal{A}\) acts as a controller. At search iteration \(t\), it proposes a complete pipeline configuration \(\theta_t \in \Theta\), and the environment evaluates this configuration and returns a reward: \[\label{eq:interface} r_t = \text{RAISE\_Env.evaluate}(\theta_t, \mathcal{D})\tag{3}\] Here \(r_t \in \mathbb{R}\), or a vector-valued reward in multi-objective settings, is computed by the evaluation layer on dataset \(\mathcal{D}\).

This interface casts end-to-end RAG configuration as a black-box optimization problem under the same search space, datasets, budgets, and reward definition for all methods. The execution engine supports the practical requirements of large-scale comparison, including asynchronous evaluation, reusable caches, bounded-time execution, and zero reward on failure, while keeping new controllers easy to add. Additional execution details are provided in Appendix 7.3.

3pt

4 Experiments↩︎

4.1 Benchmark Protocol↩︎

We use RAISE as a controlled benchmark for RAG architecture search. The experimental design follows system-level RAG automation work, including AutoRAG [3], AutoRAG-HP [6], and RAG-HPO analysis [7], that treats the full pipeline as the optimization target. In contrast, RAISE fixes the controller interface, evaluation budget, and benchmark suite so that optimizers can be compared under matched conditions, rather than being used only to produce a single tuned pipeline. This design allows us to examine whether a shared search space and heterogeneous environments reveal optimizer–environment interactions that are difficult to identify in isolated tuning studies. If RAG architecture search is governed by task structure, then the best-performing strategy should vary across retrieval-centric, multi-hop, abstention-sensitive, multimodal, and long-context environments. We therefore organize the empirical study around four research questions: whether optimizer rankings depend on the environment (RQ1), whether the proxy construction provides a sufficiently stable signal (RQ2), whether explicit search improves over random sampling under matched budgets (RQ3), and which pipeline modules and search dimensions have the largest effect on performance (RQ4).

0pt

We instantiate the seven environments in Table 1 as fixed proxy tasks from the corresponding public datasets, following HPOBench [8] and YAHPO Gym [9] benchmark practice. Each task contains 100 question–answer pairs and an associated retrieval corpus, yielding 700 QA instances and 2,162 corpus units in total. We evaluate the 13 optimization algorithms listed in Table 2 through a shared black-box interface, isolating optimizer behavior under matched conditions. Each algorithm receives 30 configuration evaluations under random seeds \(11\), \(22\), and \(33\). The scalar objective gives equal weight to ROUGE-L, METEOR, token-F1, and BLEU. LLM environments use Qwen3-4B-Instruct [44] for rewriting, pruning, and generation, while the multimodal LLM environment uses Qwen3-VL-4B-Instruct [45] with CLIP-based visual retrieval. Additional protocol details are provided in Appendix 7.5.

4.2 Main Benchmark: Optimizer Environment Interactions (RQ1)↩︎

The main benchmark examines whether optimizer behavior in RAG hyperparameter optimization (RAG-HPO) is determined primarily by the optimization algorithm or by the structure of the RAG environment. To isolate this question, RAISE fixes the controller interface, evaluation budget, and scoring rule, while evaluating each optimizer on the corresponding text or multimodal search space for each environment. Under this protocol, differences across rows compare search behavior within the same environment, whereas differences across columns show how optimizer behavior changes across retrieval, reasoning, abstention, multimodal grounding, and long-context challenges.

Table 3 reports results on seven benchmark datasets: HotpotQA [13], MS MARCO [14], ScienceQA [15], SQuADv2 [16], TriviaQA [12], LongBench-Qasper [18], and LongBench-Multifield [17]. All runs use a budget of 30 trials under random seeds \(11\), \(22\), and \(33\). Each column corresponds to one environment and its primary challenge, and each row corresponds to an optimizer instantiated through the shared controller interface. Scores should be interpreted within each dataset column. The final two columns summarize the number of mean-score wins and the average within-dataset rank for each method. We therefore read the table as an optimizer–environment interaction map rather than as a single aggregate leaderboard. Detailed per-dataset seed results are provided in Appendix Tables 5–11.

4.2.0.1 Results and analysis.

Table 3 shows that no optimizer dominates across environments. The best-performing method changes by task: Greedy Search leads on HotpotQA and TriviaQA, CEM on MS MARCO, Random Search on ScienceQA, GRPO on SQuADv2, Regularized Evolution on LongBench-Qasper, and Simulated Annealing on LongBench-Multifield. Under a fixed interface, budget, and scoring rule, these shifts indicate optimizer–environment interaction rather than a stable global ranking.

The aggregate columns support the same conclusion. Coordinate Descent achieves the best average rank without winning any individual dataset, while several other methods win only in specific settings. We therefore treat Table 3 as an interaction matrix that identifies where each search bias is useful.

Figure 2 shows that the optimizer–environment interaction in Table 3 also appears at the configuration level. Rewriting is frequently disabled, while retrieval, reranking, and pruning choices vary across methods and environments. Thus, comparable scores can arise from different pipeline choices, which motivates evaluating full configurations rather than isolated hyperparameters.

Figure 3 shows representative best-so-far trajectories over the same 30-trial budget.

4.3 Ablation Study↩︎

We complement the main benchmark with three ablation studies that examine the stability of proxy-task size and random seeds, the effect of explicit search relative to random sampling, and the contribution of individual pipeline modules to performance. Together, these analyses test whether the main findings are robust to proxy construction and reveal which parts of the RAG pipeline most strongly shape optimization outcomes.

4.3.1 Robustness to QA Sample Size and Seeds (RQ2)↩︎

To examine how proxy-task size affects optimization stability, we vary the HotpotQA subset size from 20 to 5,000 examples for CEM and TPE, using five random seeds and 30 trials per run. Figure 4 shows substantial cross-seed variability for very small proxies, especially for TPE with 20 QA examples. At this scale, a small number of question instances can dominate the objective, making controller comparisons sensitive to seed selection. Stability improves at approximately 100–200 examples and then begins to saturate, while CEM remains more stable than TPE across the tested range. The flattening of the curves suggests that larger subsets mainly reduce noise once the proxy captures the main optimization signal. We therefore use the proxy as a controlled ranking screen rather than as a replacement for full-dataset evaluation. These results support the proxy protocol as a practical tradeoff between stability and cost for lightweight benchmarking, because it preserves the coarse optimizer signal without requiring full-corpus evaluation. This design keeps the benchmark lightweight while retaining enough task signal to compare optimizer behavior.

4.3.2 Explicit vs. Random Search (RQ3)↩︎

To test explicit optimization against unguided exploration, we compare search methods with random sampling on the HotpotQA proxy task under the same 20-evaluation budget and weighted objective. Figure 5 shows that 11 of 12 non-random methods beat the baseline. The gains are modest but consistent, indicating that structured search can extract useful signal even when the budget is small. Adaptive methods tend to do better because they can spend later trials near more promising configurations. This advantage is budget-dependent rather than absolute. The result does not imply that random search is ineffective in general, but it shows that matched-budget comparisons should report random-trial baselines rather than relying only on optimizer rankings.

4.3.3 Impact of Pipeline Modules (RQ4)↩︎

We use module-level ablations on TriviaQA, HotpotQA, and LongBench-Multifield to examine how individual pipeline components affect performance under different task structures. Starting from the full text pipeline, we either remove the rewriter, reranker, or pruner, or fix one search dimension at a time. Each variant uses 13 optimization algorithms, three seeds, and 20 configuration evaluations.

Figure 6 shows that module effects vary substantially across datasets. TriviaQA is most sensitive to rewriting and pruning, consistent with its long-document retrieval setting and alias-rich answers. HotpotQA is more sensitive to retrieval depth because it requires multi-hop evidence composition. LongBench-Multifield shows the largest drop when retrieval top-\(k\) is fixed, whereas fixing chunk size is mildly beneficial. Across the three settings, no single module is uniformly dominant. Instead, the most influential search dimensions follow the evidence structure of the task, reinforcing the broader pattern that RAG pipeline choices are task-dependent. This finding cautions against treating any module choice as globally beneficial without specifying the environment and search budget used to evaluate it.

5 Conclusion↩︎

In this paper, we have presented RAISE, a framework and benchmark that formulates RAG-HPO as black-box optimization over complete RAG pipelines. Across seven environments and 13 algorithms, we have shown that optimizer rankings vary with task structure and that no method is uniformly best. Our ablations have shown that proxy construction, random baselines, and module choices affect how gains are interpreted.

RAISE has turned RAG tuning into a repeatable benchmark problem. By separating the controller, search space, and environment, it has provided a shared testbed for comparing optimizers while reducing pipeline and evaluation confounds. This separation has made mixed results informative, because they show where search behavior transfers and where it remains environment-specific under fixed protocols. Accordingly, future RAG-HPO studies should report the search space, proxy construction, random baseline, and module constraints alongside final scores for interpretable, reproducible cross-study comparisons.

Limitations↩︎

This study has several limitations. Although RAISE is extensible, our experiments instantiate a fixed search space over specific RAG modules, models, and discrete values; alternative model families, retrieval backends, or continuous spaces may alter algorithm behavior. The main benchmark relies on lightweight proxy environments and a fixed budget, which enables broad comparison but does not substitute for full-dataset or larger-budget studies. Rankings are based on an equal-weight lexical and token-level objective, with semantic and model-based metrics reported as auxiliary signals. Finally, the variance analysis covers only selected settings and warrants extension across a wider range of datasets, budgets, and seeds.

6 Module Preference Details↩︎

Figure 2 uses compact notation to keep the main-text matrix readable. In chunking cells, \(a/b\) denotes chunk size \(a\) and overlap \(b\). In retrieval and reranking cells, \(k=5\) denotes top-5 candidates, and \(\alpha\) denotes the BM25 weight in the hybrid text retriever. P1–P3 denote the discrete prompt templates used by the rewriter or pruner. The stacked bars below expand the mini-bars in the main figure.

7 Detailed Framework Specification↩︎

7.1 Pipeline Module Definitions↩︎

This appendix provides the detailed mathematical specification of the parameterized pipeline modules summarized in Section 3.

7.1.1 Prompt Template Search Space↩︎

RAISE treats prompt selection as a discrete search dimension for the rewriter and pruner. The generator prompt is fixed across text experiments, and the LLM-as-a-judge prompt is used only for auxiliary evaluation. Table 4 lists the prompt templates exposed to the optimizer.

4pt

7.1.2 Query Rewriter (Both Modalities)↩︎

User queries in real-world scenarios are frequently ambiguous or underspecified. The Query Rewriter leverages LLMs to reformulate the user input prior to retrieval. Let \(\mathcal{P} = \{p_1, p_2, \dots, p_N\}\) denote a discrete search space comprising \(N\) candidate prompt strategies. Given an initial query \(q\) and a selected prompt template \(p_i \in \mathcal{P}\), the rewriter operates as a conditional generative function producing the optimized query \(q'\): \[\label{eq:rewrite} q' \sim \mathcal{M}_{\text{rewrite}}(\cdot \mid q, p_i)\tag{4}\] In RAISE, this categorical search space encompasses distinct reformulation paradigms, including standard retrieval reformulation, keyword or entity extraction, and standalone question generation.

7.1.3 Chunker Module (Both Modalities)↩︎

Text chunking critically affects retrieval granularity. Let \(\mathcal{D}\) denote a document. The chunking function partitions \(\mathcal{D}\) into a set of segments \(\mathcal{C} = \{c_1, c_2, \dots, c_m\}\), parameterized by chunk size \(s\) and overlap \(o\): \[\label{eq:chunk} \mathcal{C} = \text{Chunk}(\mathcal{D}; s, o)\tag{5}\] In our implementation, the search space includes \(s \in \{256, 512, 1024, 2048\}\) tokens and \(o \in \{0, 64, 128, 192\}\) tokens.

7.1.4 Retrieval Stage: Text and Vision↩︎

The retriever serves as the first-stage filter that narrows the candidate corpus to a manageable subset.

7.1.4.1 Text Retriever (Both Modalities).

RAISE employs hybrid retrieval mechanisms that blend sparse and dense signals. For a rewritten query \(q'\) and chunk \(c \in \mathcal{C}\), the retrieval score is defined as \[\label{eq:retrieval95score} s(q', c) = \alpha \cdot \text{BM25}(q', c) + (1-\alpha) \cdot \cos(E_q(q'), E_d(c))\tag{6}\] where \(\alpha\) controls the interpolation between lexical and semantic retrieval. The initial retrieved set is then \[\label{eq:text95topk} \mathcal{C}_{\text{ret}} = \mathop{\text{Top-}k_{\text{ret}}}_{c \in \mathcal{C}} s(q', c)\tag{7}\] with search parameters \(\alpha \in \{0.0, 0.25, 0.5, 0.75, 1.0\}\) and \(k_{\text{ret}} \in \{1, 3, 5, 10, 20, 50\}\).

7.1.4.2 Vision Retriever (Multimodal Only).

For multimodal tasks, RAISE incorporates a vision-language retrieval stage based on CLIP-style embeddings. Given a visual corpus \(\mathcal{V}\), the retrieved image subset is \[\label{eq:vision95topk} \mathcal{V}_{\text{ret}} = \mathop{\text{Top-}k_{\text{vision}}}_{I \in \mathcal{V}} \cos(E_{\text{text}}(q'), E_{\text{vision}}(I))\tag{8}\] which provides visual evidence in parallel with text retrieval.

7.1.5 Reranker (Both Modalities)↩︎

To refine the first-stage retrieval output, the reranker applies a cross-encoder that jointly scores the query and candidate document [46]. Let \(\mathcal{M}_{\text{cross}}\) denote the cross-encoder scoring function and \(\oplus\) sequence concatenation. The reranked candidate set is \[\label{eq:rerank} \mathcal{C}_{\text{rerank}} = \mathop{\text{Top-}k_{\text{rerank}}}_{c \in \mathcal{C}_{\text{ret}}} \mathcal{M}_{\text{cross}}(q' \oplus c)\tag{9}\] where the search space encompasses both the reranker choice and \(k_{\text{rerank}} \in \{1, 3, 5, 10, 20, 50\}\).

7.1.6 Pruner (Text Pipeline Only)↩︎

Even after reranking, accumulated evidence may exceed the generator’s context window or introduce distracting content. Let \(\mathcal{P}_{\text{prune}} = \{p_1, p_2, \dots, p_M\}\) denote a discrete space of pruning prompt templates. The pruner applies a filtration function \(f_{\text{prune}}\) to produce a condensed context: \[\label{eq:prune} \mathcal{C}_{\text{final}} = f_{\text{prune}}(\mathcal{C}_{\text{rerank}}, q'; p_j)\tag{10}\] where \(p_j \in \mathcal{P}_{\text{prune}}\) specifies the pruning strategy. The multimodal pipeline bypasses this stage to preserve visual–textual alignment.

7.1.7 Generator↩︎

The generator constitutes the final synthesis stage. Given the textual context \(\mathcal{C}_{\text{final}}\), the retrieved visual context \(\mathcal{V}_{\text{ret}}\) for multimodal tasks, and the original query \(q\), generation follows the autoregressive factorization \[\label{eq:generation} P(Y \mid q, \mathcal{C}_{\text{final}}, \mathcal{V}_{\text{ret}}) = \prod_{t=1}^{T} P_{\theta_{\text{gen}}}(y_t \mid y_{<t}, q, \mathcal{C}_{\text{final}}, \mathcal{V}_{\text{ret}})\tag{11}\] where \(\mathcal{V}_{\text{ret}} = \emptyset\) in LLM pipelines.

7.2 Evaluation Metrics↩︎

This appendix provides the metric definitions summarized in Section 3.2. Let \(Y\) denote the predicted answer and \(Y^*\) the reference answer or reference context, depending on the task.

7.2.1 Lexical and N-gram Metrics↩︎

7.2.1.1 Exact Match (EM).

For extractive question answering, exact match is defined as \[\label{eq:em} \text{EM}(Y, Y^*) = \mathbb{I}(Y = Y^*)\tag{12}\]

7.2.1.2 Token-F1.

Token-F1 treats \(Y\) and \(Y^*\) as bag-of-words and computes \[\label{eq:f1} F_1(Y, Y^*) = \frac{2 \cdot |Y \cap Y^*|}{|Y| + |Y^*|}\tag{13}\]

7.2.1.3 ROUGE-L.

ROUGE-L [29] is based on the longest common subsequence (LCS), with \[\label{eq:rougel} F_{\text{LCS}} = \frac{(1+\beta^2) R_{\text{LCS}} P_{\text{LCS}}}{R_{\text{LCS}} + \beta^2 P_{\text{LCS}}}\tag{14}\] where \(R_{\text{LCS}} = \frac{\text{LCS}(Y, Y^*)}{|Y^*|}\) and \(P_{\text{LCS}} = \frac{\text{LCS}(Y, Y^*)}{|Y|}\).

7.2.1.4 BLEU.

BLEU [31] is computed as \[\label{eq:bleu} \text{BLEU} = \text{BP} \cdot \exp \left( \sum_{n=1}^{N} w_n \log p_n \right)\tag{15}\]

7.2.1.5 METEOR.

METEOR [30] is defined as \[\label{eq:meteor} \text{METEOR} = \left( \frac{10 P_m R_m}{R_m + 9 P_m} \right) (1 - \text{Pen})\tag{16}\]

7.2.2 Semantic and Model-Based Metrics↩︎

7.2.2.1 BERTScore-F1.

BERTScore [47] computes semantic overlap using contextualized embeddings. Its recall term is \[\label{eq:bertscore} R_{\text{BERT}} = \frac{1}{|Y^*|} \sum_{y^* \in Y^*} \max_{y \in Y} \mathbf{y}^\top \mathbf{y}^*\tag{17}\]

7.2.2.2 LLM-as-a-Judge (LLM-aaJ).

For complex reasoning tasks, RAISE supports an LLM-based judge model \(\mathcal{M}\) whose output is parsed into a scalar reward: \[\label{eq:llmjudge} \mathcal{E}_{\text{LLM}}(q, Y, R) = \Phi \left( \mathcal{M}(\text{prompt} \oplus q \oplus Y \oplus R) \right) \in \{0, 1\}\tag{18}\]

7.2.3 System and Search Efficiency↩︎

7.2.3.1 End-to-End Search Time.

The total wall-clock time of the search process is \[\label{eq:searchtime95appendix} \mathcal{T}_{\text{total}} = \sum_{k=1}^{K} t_k + \mathcal{T}_{\text{overhead}}\tag{19}\]

7.2.3.2 Configuration Evaluation Count.

The total number of completed evaluations over \(S\) search iterations is \[\label{eq:evalcount95appendix} K = \sum_{s=1}^{S} \mathbb{I}(\text{eval}(\theta_s))\tag{20}\]

7.3 Execution Details and Robustness↩︎

Evaluating large numbers of configurations over full datasets incurs substantial computational overhead. To support large-scale search, the RAISE execution engine employs asynchronous concurrent execution to improve throughput, reusable caching to avoid redundant computation, and bounded-time evaluation to prevent individual failures from stalling the optimization loop.

Specifically, each proposed configuration is evaluated under explicit resource and time limits. Configurations that fail due to runtime errors, latency spikes, or resource exhaustion are assigned a zero reward, ensuring that the global search process remains well-defined. The environment additionally supports both corpus-level reward aggregation, suitable for standard black-box optimizers, and finer-grained feedback appropriate for reinforcement-learning methods.

7.4 Algorithm Details↩︎

This appendix provides additional details for the 13 algorithms summarized in Table 2.

7.4.1 Sampling and Local Search↩︎

7.4.1.1 Random Search.

Random Search [10] samples complete configurations independently from the discrete search space, providing the reference baseline for all structured optimizers.

7.4.1.2 Greedy Search.

Greedy Search starts from a current configuration and applies the locally best modification available at each step. It quantifies how far the search space can be navigated by pure exploitation.

7.4.1.3 Coordinate Descent.

Coordinate Descent [33] optimizes one hyperparameter dimension while holding the others fixed, and is most effective when the objective is partially separable across modules.

7.4.1.4 Simulated Annealing.

Simulated Annealing [34] performs local moves but accepts lower-scoring configurations with a temperature-controlled probability, allowing the search to escape poor local optima early and become more selective later.

7.4.1.5 Iterative Local Search.

Iterative Local Search [35] alternates between local improvement and explicit perturbation. Relative to single-trajectory local search, it reduces dependence on initialization.

7.4.2 SMBO, Distribution-Guided, and Evolutionary Search↩︎

7.4.2.1 Tree-structured Parzen Estimator (TPE).

TPE is an SMBO method [36] related to Bayesian optimization [11]. It partitions observed configurations into high- and low-reward sets and fits separate density models to the two regions; new candidates are then selected to favor configurations with high expected improvement under this surrogate.

7.4.2.2 Cross-Entropy Method.

The Cross-Entropy Method [37] is treated here as an estimation-of-distribution-style optimizer: it maintains a parameterized sampling distribution over configurations and updates it from the current elite set. In discrete spaces, this yields a direct model of which choices should accumulate more probability mass.

7.4.2.3 Regularized Evolution.

Regularized Evolution [38] maintains a finite population, samples parents from it, and generates offspring through mutation. Aging-based regularization removes old individuals and prevents premature population collapse.

7.4.3 Bandit Search↩︎

7.4.3.1 Thompson Sampling.

Thompson Sampling [39] chooses actions by sampling from posterior reward estimates, yielding stochastic exploration proportional to uncertainty. In our setting, it serves as a bandit-style baseline for adaptive selection.

7.4.3.2 Upper Confidence Bound (UCB).

UCB [40] selects the action with the largest optimistic score \[\text{Score}_{\mathrm{UCB}} = \bar{r}_i + c \sqrt{\frac{\ln N}{n_i}},\] where \(\bar{r}_i\) is the empirical mean reward of action \(i\), \(n_i\) its pull count, and \(N\) the total number of decisions. The rule renders the exploration term explicit and deterministic.

7.4.4 RL-based Methods↩︎

7.4.4.1 GRPO.

GRPO [41] treats configuration generation as a policy over discrete choices and updates the policy with group-relative advantages. This reduces variance relative to raw score-based updates and enables direct policy learning from black-box rewards.

7.4.4.2 Dr. GRPO.

Dr. GRPO [42] is a GRPO variant designed to reduce optimization bias and improve token efficiency. In our benchmark, we use it as a conservative GRPO-style baseline for noisy, small-budget settings.

7.4.4.3 Reinforce++.

Reinforce++ [43] optimizes a stochastic policy with critic-free, globally normalized policy-gradient updates. It serves as a lighter RL baseline against which the more specialized GRPO variants can be compared, while remaining closely related to classical REINFORCE-style updates [48].

7.5 Benchmark Protocol Details↩︎

The main algorithm comparison is conducted on the seven benchmark tasks listed in Table 1. Each algorithm is evaluated under the corresponding instantiated text or multimodal search space, with seeds \(11\), \(22\), and \(33\) and a uniform budget of 30 configuration evaluations per seed. This protocol isolates search behavior from changes in pipeline structure, dataset size, or metric definition, while reducing dependence on a single random seed.

The scalar objective is an equal-weight aggregate of ROUGE-L, METEOR, token-F1, and BLEU. In LLM settings, rewriting, pruning, and generation are performed by the Qwen3 text stack. All algorithms interact with the same black-box evaluate(config) interface and observe the same proxy dataset for each task.

This shared protocol is intended to keep the main comparison focused on search behavior rather than on metric selection or implementation differences. It thus enables controlled comparison across algorithm families while holding the end-to-end RAG environment fixed.

These settings are intentionally modest: the proxy benchmark keeps evaluation fast enough for repeated optimization while still exposing differences between controllers, and 30 evaluations per seed provides a fixed budget for comparing search efficiency. Using three seeds helps separate algorithmic effects from sampling noise, while the shared evaluate(config) interface ensures that all methods face the same pipeline implementation and scoring rule. Together, these choices make the benchmark suitable for controlled comparison rather than for absolute system tuning.

7.6 Reproducibility Details↩︎

Each benchmark environment is stored as a pair of files. The QA file contains query and reference fields, while the corpus file contains the retrieval units used by the pipeline. For LongBench-Multifield, LongBench-Qasper, and ScienceQA, the proxy subsets are sampled with a fixed subset seed of 42. ScienceQA additionally keeps examples with a valid answer, a non-empty hint, and an available image, and exports the image path into the corpus entry. These fixed proxy files are used unchanged across all algorithms and seeds.

All text generation calls use the same local LLM configuration: temperature 0, maximum output length 256 tokens, and a 60-second request timeout. LLM runs use Qwen3-4B-Instruct for rewriting, pruning, and generation, while the multimodal LLM run uses Qwen3-VL-4B-Instruct for generation with CLIP-based visual retrieval. Experiments were run on a two-GPU workstation with PRO6000 GPUs.

A configuration evaluation means running one complete pipeline configuration over the full proxy environment and scoring the resulting answers with the selected objective. The evaluation cache is keyed by the configuration, dataset file hashes, modality, and evaluation mode, so repeated configurations return the same cached reward. Failed configurations receive zero reward and remain part of the budgeted trial sequence.

8 Detailed Results↩︎

8.1 Detailed Main Results↩︎

Tables 5–11 report dataset-level algorithm results for the main benchmark. Each algorithm is evaluated with a budget of 30 trials under seeds \(11\), \(22\), and \(33\). The ranking score is the equal-weight aggregate of ROUGE-L, METEOR, token-F1, and BLEU. The tables report each seed separately and summarize the mean and standard deviation across seeds.

For compactness, we adopt the following abbreviations: Coord. = Coordinate Descent, SA = Simulated Annealing, ILS = Iterative Local Search, CEM = Cross-Entropy Method, Reg-Evo = Regularized Evolution, TS = Thompson Sampling, and Dr. GRPO = Dr. GRPO.

4pt

4pt

4pt

4pt

4pt

4pt

4pt

8.2 Random-Average Ablation Details↩︎

Table 12 reports the full per-algorithm results for the HotpotQA random-average ablation. All runs use a single seed and a budget of 20 evaluations. The random-trial mean baseline is 0.0660, and the Random Search best-of-budget score is 0.1134; the latter is included only as a reference point and is not the ablation baseline. The \(\Delta\) column is always computed against the random-trial mean. BERTScore-F1 uses the corrected evaluation values.

3pt

References↩︎

---

1. Equal contribution.↩︎