1 Introduction↩︎

LLM routing has emerged as an effective paradigm for leveraging the complementary strengths of multiple LLMs by dynamically selecting appropriate models or reasoning strategy for each query. Among existing approaches, a line of recent work formulates routing as a sequential decision-making problem and applies RL to optimize routing policies [1]–[4], achieving promising performance gains. Despite recent progress, existing routing methods often struggle to learn effective policies under queries with varying difficulty and routing uncertainty. In practice, learning signals can exhibit uneven statistical properties across instances and datasets, leading to inconsistent policy updates during training and biased optimization toward easier decision patterns. Consequently, routing policies may fail to learn reliable behaviors for challenging cases.

We argue that a key bottleneck of RL-based LLM routing lies in the learning signals used for routing policy optimization. Unlike standard generation tasks, routing policies must learn query-dependent model-selection behaviors under heterogeneous routing difficulty. However, there are two key technical issues with learning signals in current routing frameworks. First, routing policies are typically optimized using scalarized rewards that aggregate multiple objectives, such as answer correctness and formatting behavior, which often correspond to different routing decisions and model-selection strategies into a single score. As shown in Figure 1 (a), similar reward improvements may arise from entirely different routing behaviors, making it difficult for the policy to identify which routing decision performs better. Consequently, scalarized rewards obscure routing credit assignment and weaken policy learning [5]–[7]. Second, routing rewards exhibit highly heterogeneous statistical properties across instances and datasets. As shown in Figure 1 (b), some routing cases produce highly dispersed rewards due to large disagreement among candidate models, while others yield concentrated rewards with limited routing ambiguity. Different datasets further induce substantially different reward scales and variances. Such heterogeneity can bias optimization toward easier routing cases [8], [9], while informative but underrepresented routing decisions receive insufficient learning attention. These observations raise two central challenges for RL-based LLM routing: (1) how to transform entangled routing rewards into clear learning signals for precise routing credit assignment; and (2) how to calibrate optimization signals across heterogeneous routing instances and datasets, thereby assigning appropriate emphasis to uncertain and informative routing decisions during optimization.

To address these challenges, we propose a Reward Calibration Framework for LLM routing, abbreviated as ReCal. Rather than directly optimizing with raw routing rewards, ReCal treats policy learning as a two-stage calibration process: it first restructures reward signals into clearer supervision, and then adjusts the contribution of these signals to policy updates under heterogeneous data distributions. We introduce a hierarchical, disentangled reward mechanism to improve learning signal clarity (Challenge 1). Instead of estimating a single advantage from aggregated scalarized rewards, we separately estimate the advantage of each reward component, which preserves objective-specific relative improvements during optimization. The resulting component-wise advantages are then combined through weighted aggregation to reflect the relative importance of different objectives while reducing interference among conflicting optimization signals. Building on the calibrated rewards, we further improve signal comparability (Challenge 2) under heterogeneous reward distributions through a distribution-aware optimization strategy. At the instance level, we introduce variance-aware reweighting to dynamically calibrate update strength according to disagreement among candidate responses. At the dataset level, we apply per-dataset normalization to prevent datasets with disproportionately large reward scales or variances from dominating policy updates. Together, these mechanisms transform routing optimization from directly fitting raw rewards into a calibrated learning process that explicitly separates objective-level supervision from distribution-level optimization variability. This allows the routing policy to focus on informative routing decisions while maintaining stable learning across heterogeneous tasks.

Our contributions are summarized as follows:

• We propose a learning-signal calibration perspective for RL-based LLM routing, which improves policy learning by jointly modeling learning signal clarity and optimization comparability under heterogeneous tasks.

• We introduce a hierarchical and disentangled reward mechanism that performs component-wise advantage estimation before aggregation, enabling clearer credit assignment and reducing interference among competing objectives.

• We propose a distribution-aware optimization strategy that reduces optimization bias across heterogeneous data through variance-aware reweighting and per-dataset normalization.

• Extensive experiments on multiple datasets indicate that our method consistently improves routing performance and training stability.

2 Related Work↩︎

2.1 LLM Routing and RL-based Optimization↩︎

Existing LLM routing methods can be broadly categorized into three groups. Early approaches rely on heuristic or rule-based strategies [10], which use predefined criteria such as confidence or cost to select models. Supervised learning-based routers [11]–[13] further improve flexibility by training classifiers or scoring models to predict the best routing decisions. More recently, RL-based approaches optimize routing policies directly with reward signals, enabling adaptive trade-offs between performance and efficiency [2], [4]. These methods typically optimize a scalar reward that combines multiple objectives, such as answer quality and format accuracy. While effective, these methods primarily focus on improving routing decisions themselves, with limited attention to how learning signal design affects policy optimization.

2.2 Learning Signal Design and Optimization↩︎

Amount of RL-based work studies how to construct informative reward design, including reward shaping [14], multi-objective RL [15], and advantage-based optimization [8]. These approaches aim to improve how different aspects of feedback are represented and utilized during policy updates. However, these approaches mainly focus on improving the semantic quality of reward signals, rather than how heterogeneous reward distributions influence optimization behavior across instances and datasets. Complementary to reward structuring, another line of work studies how heterogeneous data distributions affect optimization dynamics, focusing on techniques such as reward scaling and normalization in policy optimization [16], [17], sample reweighting and off-policy correction [18], and curriculum or uncertainty-aware training [19], [20]. These methods aim to stabilize training and improve data efficiency by adjusting the scale or importance of learning signals. However, they primarily study either the semantic structure of learning signals or their optimization behavior under heterogeneous data, but rarely consider the interaction between the two. This separation becomes limiting in LLM routing, where heterogeneous tasks jointly affect both what information learning signals contain and how strongly they influence policy updates.

3 Preliminaries↩︎

3.1 RL-based LLM Routing↩︎

Given a query \(x\), LLM routing selects an action \(a\) from a candidate set \(\mathcal{A}=\{a_1,\dots,a_K\}\) using a routing policy \(\pi_\theta(a|x)\) parameterized by \(\theta\), where \(K\) denotes the number of routing candidates. After executing action \(a \sim \pi_\theta(\cdot|x)\), the selected model generates a response \(y\), which is evaluated by a reward function \(r(x,a,y)\).

In practical routing systems, rewards are usually composed of multiple objectives, such as answer correctness, reasoning quality, formatting behavior, or routing efficiency. Existing methods typically aggregate them into a scalar reward: \[r = \sum_{m=1}^{M} \lambda_m r^{(m)},\] where \(M\) denotes the number of reward components, \(r^{(m)}\) is the reward of the \(m\)-th objective, and \(\lambda_m\) is its corresponding weight.

Given a training batch \(\mathcal{D}=\{(x_i,a_i,y_i,r_i)\}_{i=1}^{N}\) with batch size \(N\), RL-based routing optimizes the policy by maximizing the expected reward: \[\max_\theta \;\mathbb{E}_{a \sim \pi_\theta(\cdot|x)}[r(x,a,y)].\]

3.2 Policy Optimization↩︎

Most existing routing methods adopt Proximal Policy Optimization (PPO)-style [8] optimization. Let \(\pi_{\theta_{\text{old}}}\) denote the old routing policy. PPO optimizes the clipped objective \(\mathcal{L}_{\text{PPO}}\): \[\mathbb{E} \left[ \min \left( \rho_t A_t, \text{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t \right) \right],\] where \(\rho_t=\frac{\pi_\theta(a_t|x_t)}{\pi_{\theta_{\text{old}}}(a_t|x_t)}\) is the policy ratio for action \(a_t\) under query \(x_t\), \(\epsilon\) is the clipping threshold, and \(A_t\) denotes the estimated advantage, which is typically obtained directly from the scalarized reward, i.e., \(A_t=\text{Adv}(r_t)\).

Recent RL frameworks further adopt Group Relative Policy Optimization (GRPO)-style optimization. For each query \(x_i\), the policy samples a response group \(\mathcal{Y}_i=\{y_{i,1},\dots,y_{i,G}\}\) containing \(G\) responses, where each response receives a scalar reward \(r_{i,j}\). GRPO computes group-relative advantages by normalizing rewards within each group: \[A_{i,j} = \frac{ r_{i,j}-\mu_i }{ \sigma_i+\varepsilon },\] where \(\sigma_i=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_{i,j}-\mu_i)^2}\) and \(\mu_i=\frac{1}{G}\sum_{j=1}^{G}r_{i,j}\) denote the standard deviation and group mean, respectively, and \(\varepsilon\) is a small constant for numerical stability.

4 The Proposed Framework: ReCal↩︎

Figure 2 illustrates the overall framework of ReCal. Starting from a response group sampled by the routing policy, ReCal performs a two-stage learning-signal calibration process before policy optimization. First, ReCal improves learning signal clarity through hierarchical reward decomposition and component-wise advantage estimation. It then calibrates optimization bias under heterogeneous reward distributions through variance-aware reweighting and per-dataset normalization. The resulting calibrated advantages are finally integrated into a GRPO-style policy optimization objective.

4.1 Learning Signal Clarity Calibration↩︎

Rather than estimating advantages directly from aggregated scalarized rewards, ReCal separates reward components before advantage estimation, allowing relative improvements under different objectives to be modeled independently.

4.1.1 Hierarchical Reward Decomposition↩︎

For each response \(y_{i,j}\) in response group \(\mathcal{Y}_i\), ReCal defines a set of reward components: \(\mathbf{r}_{i,j} = \{ r_{i,j}^{\text{ans}}, r_{i,j}^{\text{info}}, r_{i,j}^{\text{format}}, r_{i,j}^{\text{route}}, r_{i,j}^{\text{balance}} \}.\) \(r^{\text{ans}}\) is computed as the F1 score between the generated answer and the ground-truth. \(r^{\text{info}}\) measures the quality of returned called-model information by computing the F1 score with the ground-truth. \(r^{\text{format}}\) is a binary reward indicating whether the response follows the required output format. \(r^{\text{route}}\) reflects routing efficiency based on the number of external model calls during reasoning. \(r^{\text{balance}}\) encourages balanced router utilization by rewarding models with lower historical routing frequency. To stabilize long-term routing behavior, the routing frequency statistics are accumulated with exponential moving average (EMA) decay. Let \(c_t^{(k)}\) denote the accumulated routing count of model \(a_k\) at training step \(t\), and \(\hat{c}_t^{(k)}\) denotes the routing count within the current step. The EMA-updated routing statistics are computed as: \[c_t^{(k)} = \alpha c_{t-1}^{(k)} + \hat{c}_t^{(k)},\] where \(\alpha \in [0,1)\) is the EMA decay coefficient. \(r^{\text{balance}}\) is then inversely correlated with the normalized routing proportion of the selected model.

These objectives are organized hierarchically according to their dependency on answer correctness. Auxiliary rewards are activated only when the response satisfies a correctness threshold: \[r_{i,j}^{(m)} = \mathbb{I} (r_{i,j}^{\text{ans}}>\delta) \cdot \tilde{r}_{i,j}^{(m)},\] where \(\tilde{r}_{i,j}^{(m)}\) denotes the raw reward, \(\delta\) is the activation threshold, and \(\mathbb{I}(\cdot)\) is the indicator function.

4.1.2 Component-wise Advantage Estimation↩︎

Instead of estimating advantages after scalar reward aggregation, ReCal computes group-relative advantages independently for each reward component. For the \(m\)-th reward objective, the component-wise advantage is computed as: \[A_{i,j}^{(m)} = \frac{ r_{i,j}^{(m)}-\mu_i^{(m)} }{ \sigma_i^{(m)}+\varepsilon },\] where \(\sigma_i^{(m)} = \sqrt{ \frac{1}{G} \sum_{j=1}^{G} (r_{i,j}^{(m)}-\mu_i^{(m)})^2 }\) and \(\mu_i^{(m)}=\frac{1}{G}\sum_{j=1}^{G} r_{i,j}^{(m)}\) denote the standard deviation and mean of the \(m\)-th reward component within response group \(\mathcal{Y}_i\).

The final clarity-calibrated advantage is then obtained through weighted aggregation: \[A_{i,j}^{\text{clarity}} = \sum_{m=1}^{M} \lambda_m A_{i,j}^{(m)},\] where \(\lambda_m\) denotes the weight of the \(m\)-th objective.

Compared with scalarized optimization, this formulation preserves objective-specific relative improvements before aggregation. As a result, the routing policy receives clearer, more discriminative supervision, enabling it to understand which samples are more conducive to policy improvement.

4.2 Distribution-aware Optimization↩︎

Although disentangled advantages improve learning signal clarity, optimization can still become biased under heterogeneous reward distributions. ReCal therefore further calibrates optimization variability across both instances and datasets.

4.2.1 Variance-aware Reweighting↩︎

For each response group \(\mathcal{Y}_i\), with the reward mean \(\mu_i^{\text{group}}=\frac{1}{G}\sum_{j=1}^{G}r_{i,j}\), we measures routing uncertainty using the reward variance within the group: \[\sigma_i^{\text{group}} = \sqrt{ \frac{1}{G} \sum_{j=1}^{G} (r_{i,j}-\mu_i^{\text{group}})^2 }.\]

Intuitively, larger intra-group variance indicates stronger disagreement among sampled routing decisions, suggesting that the corresponding query is more uncertain and potentially more informative. ReCal therefore rescales optimization strength according to the relative variance level: \[\tau_i = \text{clip} \left( \frac{ \sigma_i^{\text{group}} }{ \bar{\sigma} }, \tau_{\min}, \tau_{\max} \right),\] where \(\bar{\sigma}\) denotes the average group variance within the batch, and \(\tau_{\min}\) and \(\tau_{\max}\) are clipping thresholds. Then the variance-calibrated advantage is computed as: \[A_{i,j}^{\text{var}} = \tau_i A_{i,j}^{\text{clarity}}.\]

This mechanism allocates larger optimization weights to uncertain routing cases.

4.2.2 Per-dataset Normalization↩︎

Different datasets may exhibit substantially different reward magnitudes and variances, causing certain datasets to dominate optimization. To improve optimization comparability across heterogeneous data sources, ReCal normalizes advantages independently within each dataset.

Let \(\mathcal{D}^{(d)}\) denote the set of samples from dataset \(d\). We compute dataset-wise statistics: \[\mu^{(d)} = \frac{1}{|\mathcal{D}^{(d)}|} \sum_{(i,j)\in \mathcal{D}^{(d)}} A_{i,j}^{\text{var}},\] \[\sigma^{(d)} = \sqrt{ \frac{1}{|\mathcal{D}^{(d)}|} \sum_{(i,j)\in \mathcal{D}^{(d)}} (A_{i,j}^{\text{var}}-\mu^{(d)})^2 }.\]

The final calibrated advantage is: \[\hat{A}_{i,j} = \frac{ A_{i,j}^{\text{var}}-\mu^{(d)} }{ \sigma^{(d)}+\epsilon }.\]

This normalization aligns optimization scales across datasets and reduces optimization bias caused by heterogeneous reward distributions.

4.3 Overall Training Objective↩︎

\(\hat{A}_{i,j}\) is integrated into a GRPO-style clipped policy optimization objective \(\mathcal{L}_{\text{ReCal}}\): \[\mathbb{E} \left[ \min \left( \rho_{i,j}\hat{A}_{i,j}, \text{clip} (\rho_{i,j},1-\epsilon,1+\epsilon) \hat{A}_{i,j} \right) \right],\] where \(\rho_{i,j} = \frac{ \pi_\theta(a_{i,j}|x_i) }{ \pi_{\theta_{\text{old}}}(a_{i,j}|x_i) }\) denotes the policy ratio between the updated policy and the previous policy.

ReCal explicitly calibrates both learning signal clarity and optimization weighting before policy updates, resulting in more stable and informative policy learning under heterogeneous scenarios.

5 Experiments↩︎

5.1 Experimental Setup↩︎

5.1.0.1 Datasets

We evaluate ReCal on seven question-answering (QA) datasets: (1) General QA: Natural Question (NQ) [21], TriviaQA [22], PopQA [23]; (2) Multi-Hop QA: HotpotQA (HpQA) [24], 2WikiMultiHopQA (2wiki) [25], Musique [26], and Bamboogle (Bamb) [27].

5.1.0.2 Baselines

We compare ReCal with three categories of methods: (1) No routing: Direct Inference (Vanilla), Chain-of-Thought (CoT) [28], supervised fine-tuning (SFT), Retrieval-Augmented Generation (RAG) based on the external knowledge Wikipedia-18 [29] and the retriever E5 [30], and Search-R1 [31]; (2) Heuristic & Discriminative routing: Largest LLM, Prompt LLM, Prompt LLM* (Prompt LLM + task decomposition), KNN Router[10], KNN Router* (KNN Router + task decomposition), MLP Router[10], RouteLLM[11], RouterDC[13], and GraphRouter[12]; (3) RL-based routing: Router-R1[2].

5.1.0.3 Implementation Details

We used Qwen2.5-3B-instruct [32] as the base model and conducted RL training using GRPO within the VeRL ² framework on NVIDIA A100 GPUs, with a batch size of 64 and up to 150 training steps. Similar to Router-R1, we combined 7K samples each from the NQ and HotpotQA datasets to create a training set. After training, we evaluated performance on the two in-domain datasets and the other five out-of-domain datasets by sampling 500 test instances from each of them (excluding Bamboogle, which contained a total of 125 test instances). All baseline models were trained (where applicable) and evaluated on consistent datasets and settings. We selected a set of six models of varying sizes and architectures via the NVIDIA NIM API ³ and OpenRouter ⁴: Qwen2.5-7B-Instruct [32], LLaMA-3.1-8B-Instruct [33], LLaMA-3.1-70B-Instruct [33], Mistral-7B-Instruct [34], Mixtral8x22B-Instruct [35], and Gemma-2-27B-Instruct [36].

10pt

5.2 Main Results↩︎

As shown in Table 1, our proposed ReCal achieves the best average performance among all compared methods on seven QA benchmarks, demonstrating the effectiveness of learning signal calibration for RL-based LLM routing. Compared with non-routing methods, RL-based routing substantially improves performance across all QA tasks, validating the importance of adaptive model selection. Router-R1 achieves strong gains over heuristic and discriminative routers, showing the advantage of end-to-end policy optimization. However, our method consistently outperforms Router-R1 on most datasets, indicating that improving the clarity and comparability of learning signals is critical for effective routing optimization.

Ours achieves the best overall average EM score of \(0.481\), surpassing Router-R1 by \(6.7\%\) relatively (\(0.481\) vs.\(0.414\)). The improvements are particularly significant on more challenging multi-hop QA benchmarks. For example, ReCal improves performance from \(0.476\) to \(0.560\) on 2Wiki and from \(0.424\) to \(0.568\) on Bamboogle. These results suggest that calibrating learning signals is especially beneficial for routing policy with high uncertainty.

On in-domain datasets, ReCal also demonstrates strong improvements. Compared with Router-R1, our method improves EM from \(0.334\) to \(0.404\) on NQ and from \(0.412\) to \(0.442\) on HotpotQA. More importantly, the gains consistently transfer to unseen out-of-domain datasets, indicating that the proposed calibration mechanisms improve the generalization ability of routing policies rather than overfitting to specific training distributions.

We further observe that heuristic and discriminative routing methods exhibit relatively unstable performance across datasets. Prompt LLM and RouterDC perform competitively on certain datasets, such as TriviaQA or Bamboogle, but their overall performance remains substantially lower than RL-based methods. This suggests that heuristics or discriminative routing strategies struggle to adapt to heterogeneous requirements across tasks.

5.3 In-depth Analysis↩︎

5.3.1 Ablation Study↩︎

As shown in Table 2, removing either Learning Signal Clarity Calibration (LSC) or Distribution-aware Optimization (DAO) consistently degrades performance across datasets, while removing both causes the largest drop. This verifies that both learning signal clarity and optimization comparability are important for RL-based routing.

Removing LSC decreases the average score to \(0.418\), indicating that directly optimizing scalarized rewards leads to weaker credit assignment and noisier optimization signals. Removing only Component-wise Advantage Estimation (CAE) still causes a noticeable drop to \(0.430\), demonstrating that component-wise advantage estimation itself is critical for preserving objective-specific supervision during optimization. DAO further improves performance from \(0.456\) to \(0.481\), with especially clear gains on challenging multi-hop datasets such as 2Wiki and Musique. This suggests that heterogeneous reward distributions introduce optimization bias during RL training, and distribution-aware calibration helps stabilize learning across different datasets and routing cases.

As Figure 3 shows, removing \(r^{\text{balance}}\) introduces obvious training instability and leads to a substantial decrease in reward during the optimization, indicating that the balance regularization term is crucial for preventing the routing policy from collapsing. Removing both \(r^{\text{balance}}\) and \(r^{\text{route}}\) also sharply decreases both the routing counts and the reward in the later phases of training. Removing CAE produces a different failure mode: the routing count exhibits abnormal spikes while the reward rapidly collapses later. This indicates that directly estimating advantages from scalarized rewards introduces entangled and unstable optimization signals. In contrast, the full ReCal framework maintains the smoothest routing behavior and the most stable reward improvement throughout training.

5.3.2 Stability and Convergence of Training↩︎

Figure 4 (a) presents the entropy loss curves during RL training. ReCal exhibits substantially smoother and faster convergence, with entropy steadily decreasing throughout training and converging to a low-variance regime in early stages. In contrast, Router-R1 shows persistent oscillations and slower convergence, indicating unstable policy updates under scalarized reward optimization.

As shown in Figure 4 (b), ReCal consistently yields more stable performance gains across both datasets, whereas Router R1 exhibited significant fluctuations and competition across the datasets. In particular, during training steps \(60\)–\(75\), Router-R1 improves on NQ while simultaneously degrading on HotpotQA, suggesting that the optimization process becomes biased toward dominant dataset-specific reward signals. This behavior reveals the existence of cross-dataset optimization interference under heterogeneous reward distributions. By contrast, ReCal jointly improves performance on both datasets throughout training. These results demonstrate that calibrating both reward clarity and optimization distribution improves not only routing performance but also the stability and consistency of policy optimization under heterogeneous tasks.

5.3.3 Generalization to Unseen LLMs↩︎

To evaluate the generalization ability of ReCal under expanding model pools, we introduce two additional unseen LLMs, DeepSeek-V3 [37] and Qwen2.5-72B-Instruct [32], during inference only. We augment the routing prompt with textual descriptions of the newly models, allowing the routing policy to select them. As shown in Table 1, Ours+ maintains strong performance and further improves several datasets. This result suggests that ReCal does not overfit to a fixed routing space. Instead, the learned routing policy can generalize to newly introduced models by leveraging semantic descriptions.

6 Conclusion↩︎

In this paper, we propose ReCal, a reward calibration framework for RL-based LLM routing. ReCal introduce a hierarchical reward decomposition mechanism with component-wise advantage estimation to provide clearer objective-level supervision, together with a distribution-aware optimization strategy that calibrates optimization variability across heterogeneous instances and datasets through variance-aware reweighting and per-dataset normalization. Extensive experiments demonstrate that ReCal consistently improves routing performance, training stability, and generalization over routing baselines. Further analyses show that ReCal produces more stable optimization under heterogeneous routing scenarios. Beyond the current setting, we believe ReCal provides a promising direction for future RL-based LLM routing. In future work, we plan to extend ReCal to broader routing scenarios. Another important direction is to develop more adaptive and automatic calibration strategies that can dynamically adjust reward structures and optimization variability under evolving model pools and task distributions.

Limitations↩︎

ReCal is evaluated primarily on QA-oriented routing tasks, and its effectiveness for other routing settings, such as multimodal systems remains to be explored. Furthermore, ReCal currently relies on manually designed reward components and weighting coefficients, which may require additional tuning when adapting to new tasks or model pools.

Ethical Considerations↩︎

We use only publicly available datasets that are widely adopted in the LLM community and contain no personally identifiable information (PII) in our experiments. All evaluated models are accessed through publicly available APIs or open-source checkpoints under their respective licenses.

7 Case Study↩︎

For the question "Who is responsible for introducing the principle of uniformitarianism?", the correct answer is [‘James Hutton’ ‘geologist James Hutton’]. Above are the responses generated by Ours (ReCal) and Router-R1. Our approach ensures the correctness of the answer by invoking different models, whereas Router-R1 easily provides incorrect answers in complex contexts.

8 More Experimental Results↩︎

We present extensive experimental results on seven QA datasets with respect to F1-Score in Table 3. ReCal consistently outperforms all baselines across both general and multi-hop QA tasks, achieving the highest average F1-scores. It maintains high accuracy even after introducing unseen models, and achieves state-of-the-art performance.

9 Hyperparameter Settings↩︎

10 Notation↩︎

References↩︎