1 Introduction↩︎

Large language models (LLMs) have demonstrated strong performance across diverse language understanding and reasoning tasks [1]–[3]. However, adapting these models to specialized domains remains expensive because full fine-tuning requires substantial computation and access to training data. Parameter-efficient fine-tuning (PEFT) methods address this issue by modifying only a small subset of parameters [4]. Among them, Low-Rank Adaptation (LoRA) [5] is widely used because it adds lightweight trainable matrices while keeping the backbone frozen.

In practice, domain adaptation is often done independently, where different groups train LoRA adapters on their own datasets but cannot share the original data due to privacy or regulatory constraints. Instead, they may release only the trained LoRA modules. A natural goal is therefore to compose multiple independently trained LoRA experts into a single model that supports heterogeneous tasks. Mixture-of-Experts (MoE) architectures [6], [7] provide a natural mechanism for such modular reuse by routing each token to selected experts. However, when the goal is to reuse frozen independently trained LoRA adapters, directly applying standard soft MoE routing introduces a scale mismatch. LoRA modules are trained assuming a unit additive update, whereas MoE combines experts using routing weights. Soft routing, therefore, blends expert updates and changes the effective function applied by each adapter. For example, LoRA-Mixer [8] addresses this by retraining the adapters with auxiliary losses, but this increases trainable parameters, requires more data, and can alter the knowledge encoded in the original experts.

This challenge becomes more pronounced for instruction-tuned models with chain-of-thought (CoT) reasoning ability. Conventional fine-tuning encourages imitation of short answers and may weaken internal reasoning behavior in large models, while reinforcement learning from verifiable feedback (RLVF) can produce stronger reasoning experts by optimizing correctness rather than matching reference outputs [9], [10]. However, existing LoRA composition methods primarily focus on supervised multi-task fine-tuning and do not study the training and integration of experts in instruction-tuned models that exhibit CoT reasoning.

In this work, we propose Hard-Routed MoR-LoRA, a framework that separates reasoning acquisition from expert selection. First, domain-specific LoRA adapters are trained independently using RLVF to obtain reasoning. Then, the experts are frozen and integrated using a hard-routed mixture that selects exactly one expert per token through a straight-through estimator (STE). We distill reasoning traces from the experts and train only a shared lightweight router and a small attention adaptation using a small amount of data (1000 samples per dataset). Hard routing preserves the original LoRA assumption by applying each expert with unit scale, enabling modular composition without retraining the experts.

Experiments across five benchmarks and multiple model scales show that RLVF produces stronger reasoning experts for capable models, while conventional fine-tuning can degrade reasoning behavior in large instruction-tuned models. We further demonstrate that hard routing achieves comparable or better performance than soft-routing baselines in our setting while using substantially fewer trainable parameters and keeping the pretrained experts frozen. Analysis of routing probabilities reveals that normalized soft top-\(k\) mixtures implicitly behave like near hard top-1 selection, indicating that hard routing captures the intended behavior more directly.

Our contributions are summarized as follows:

• We identify a scaling mismatch that arises when soft MoE routing is applied to independently trained frozen LoRA adapters, and we use hard unit-scale routing to preserve standalone expert application without retraining experts.

• We introduce a two-stage framework that trains reasoning LoRA experts using RLVF and integrates them through distillation while keeping experts frozen.

• We provide empirical evaluation across model scales and tasks showing improved performance and parameter efficiency.

2 Related Work↩︎

2.0.0.1 Modular LoRA adaptation.

LoRA enables efficient fine-tuning by learning small low-rank updates while keeping the backbone model frozen. Although effective for adapting a model to a single task, independently trained LoRA modules are usually task-specific and difficult to reuse across different domains, especially when the original training data cannot be shared. There is a rich body of research have Several works, therefore, studiedy how to combine multiple pretrained adapters into one model [11]–[14]. LoraHub [15] learns weights that merge a set of LoRA modules into a single composed adapter, while LoRA-LEGO [16] decomposes adapters into smaller rank components and reconstructs a merged adapter for cross-task reuse. These methods demonstrate that pretrained adapters can be combined without retraining, but the resulting model applies the same merged adapter to every input and cannot dynamically select different experts at the token level.

2.0.0.2 Mixture-of-Experts with LoRA.

MoE architectures [6], [7], [17], [18] increase model capacity by routing tokens to a sparse subset of experts. Instead of merging adapters into one module, MoE-based approaches keep multiple experts and select among them during inference. Recent work applies this idea to parameter-efficient tuning by treating LoRA modules as experts. MixLoRA [19] inserts multiple LoRA modules into feed-forward layers (FFN) with top-\(k\) routing. MoLE [20] learns hierarchical gating across layer-wise adapters, while SiRA [21] and GOAT [22] study alignment and sparse routing strategies. The closest to our work, LoRA-Mixer, integrates LoRA modules into attention projections and trains a lightweight router. However, soft routing produces a weighted sum of LoRA updates, whereas standalone LoRA training assumes unit-scale application. In LoRA-Mixer, this issue is addressed by further fine-tuning the experts with regularization losses, which increases the number of trainable parameters and introduces additional integration-stage computation.

2.0.0.3 Instruction-tuned Reasoning and Verifiable Feedback.

Instruction tuning and alignment stages shape how language models produce and structure reasoning [23]. RLVF improves reasoning by optimizing correctness using automatically checkable rewards instead of matching reference outputs [9], [10]. DeepSeek-R1 [10] shows that strong reasoning ability can emerge from this training paradigm, and Group Relative Policy Optimization (GRPO) [9] stabilizes training without requiring a learned value critic. In our setting, adapters are added to instruction-tuned base models that already exhibit CoT behavior. Preserving this behavior during training is important, and RLVF provides a natural way to train reasoning experts without overriding the aligned reasoning patterns of the base model.

2.0.0.4 Hard routing and Straight-through Estimators.

Hard routing selects a single expert through a discrete decision, which is not directly differentiable. The STE enables gradient-based training by using the hard decision in the forward pass and a surrogate gradient in the backward pass [24], [25]. In our approach, STE allows top-1 expert selection with unit scaling, preserving the original LoRA formulation while still allowing end-to-end training of the router.

3 Preliminaries↩︎

This section briefly introduces the components utilized in our method: Low-Rank Adaptation, Mixture-of-Experts, and Reinforcement Learning from Verifiable Feedback.

3.1 Low-Rank Adaptation↩︎

LoRA parameterizes weight updates as a low-rank decomposition: \[\begin{align} \label{eq:lora} W_{\text{updated}} = W_{\text{pretrained}} + AB \end{align}\tag{1}\] where \(A \in \mathbb{R}^{n \times r}\) and \(B \in \mathbb{R}^{r \times n}\) with \(r \ll n\). During fine-tuning, only \(A\) and \(B\) are optimized while \(W_{\text{pretrained}}\) remains frozen, substantially reducing memory and computational cost.

3.2 Mixture-of-Experts Routing↩︎

Given token representation \(x\), a router produces logits \(\mathbf{G}(x) \in \mathbb{R}^{K}\) and the routing probabilities are formulated as: \[\begin{align} \mathbb{P}_i(x) = \operatorname{softmax}(\mathbf{G}(x))_i \end{align}\] Standard MoE combines multiple experts using routing weights [7], typically activating the top-\(k\) experts and scaling each expert output by its routing probability. The final output is computed as \[\begin{align} \text{MoE}(x) = \sum_{i=1}^{K} \mathbb{I}\!\left(i \in \text{top-}k(\mathbb{P}(x))\right) \cdot \mathbb{P}_{i}(x) \cdot E_i(x) \end{align}\] where \(\mathbb{I}(\cdot)\) denotes the indicator function and \(E_i(x)\) is the \(i\)-th expert’s output.

3.3 Reinforcement Learning from Verifiable Feedback↩︎

RLVF trains models using rewards derived from automatically verifiable outputs rather than human preference annotations [9], [10]. Unlike supervised fine-tuning, the model is not constrained to follow a reference reasoning trace; only the final answer is evaluated by the verifier. This allows the model to freely generate intermediate CoT reasoning steps, encouraging the emergence of structured reasoning behavior.

We adopt GRPO [9]. For a prompt \(x\), a group of reasoning trajectories \(\{\tau_i\}_{i=1}^N\) is sampled with rewards \(\{r_i\}_{i=1}^N\): \[\begin{align} \bar r &= \frac{1}{N} \sum_{i=1}^N r_i \\ A_i &= r_i - \bar r \end{align}\] where \(A_i\) is the advantage for the \(i\)-th reasoning trace. The reward function is detailed in Section 13.3 and Table ¿tbl:table:rlvf-reproducibility?.

To improve sample efficiency, we use traces collected under the behavior policy \(\pi_{\text{old}}\), which is a few checkpoints behind the target policy \(\pi_{\theta}\). For each trajectory \(\tau_i\), we compute a token-level importance ratio: \[\begin{align} w_{i,t} = \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\text{old}}(y_{i,t} \mid x, y_{i,<t})} \end{align}\]

The off-policy token-level GRPO objective becomes: \[\begin{align} \begin{aligned} \mathcal{L}_{\text{off}}(\theta) &= - \mathbb{E}_{x} \Bigg[ \frac{1}{N} \sum_{i=1}^N A_i \sum_{t=1}^{T_i} w_{i,t} \log \pi_\theta\!\left(y_{i,t} \mid x, y_{i,<t}\right) \Bigg] \end{aligned} \end{align}\]

4 Methodology↩︎

We propose Hard-Routed MoR-LoRA (Mixture of Reasoning LoRA), a plug-in framework that integrates multiple reasoning LoRA experts into a unified model while preserving the original LoRA assumption and maintaining sparse computation. The key idea is to decouple reasoning acquisition from expert selection. First, each LoRA adapter independently learns domain-specific reasoning behavior using RLVF [9], [10]. Then, the pretrained experts are combined through a hard-routed mixture in which only a lightweight router and a small attention adaptation are trained using supervised fine-tuning (SFT) on distilled reasoning traces.

The overall two-stage pipeline is illustrated in Figure 1. Stage I trains reasoning experts independently, while Stage II integrates them into a shared model by learning which expert should be selected for each token.

4.1 Hard Mixture-of-LoRAs Architecture↩︎

Scale Mismatch in Soft LoRA Mixtures↩︎

From Eq. 1 , a pretrained LoRA expert is applied with unit contribution. However, the standard MoE applies weighted combinations of multiple experts: \[\begin{align} W_{\text{updated}} = W_{\text{pretrained}} + \sum_{i=1}^{k} \omega_i A_i B_i \end{align}\] where \(\omega_i\) are routing probabilities, which change the magnitude of the update. Since each LoRA expert was trained assuming unit contribution, scaling by \(\omega_i\) changes the effective adapter update used at integration time, creating a mismatch between standalone expert training and mixture usage. LoRA-Mixer compensates for this mismatch by further fine-tuning the adapters with preservation losses, aiming to limit deviation from the pretrained LoRA parameters.

Hard Routing with Straight-Through Estimator↩︎

To preserve the original LoRA assumption, the selected expert must be applied with unit scale. However, a softmax router produces continuous weights. In a top-1 soft-routing variant, one may first select the largest routing probability \(p_j\) and apply the selected expert as \(p_j E_j(x)\). This remains differentiable with respect to the router logits, but it scales the selected LoRA update by \(p_j < 1\), violating the unit-scale application under which the expert was trained. Alternatively, one can normalize over the selected top-1 set to enforce unit contribution, giving \(p_j / p_j = 1\). However, this normalization is constant for any fixed selected expert and therefore provides zero gradient through the routing weight. The remaining dependence on the router is only through the discrete \(\arg\max\) selection, which is non-differentiable. Therefore, a purely top-\(1\) soft-routing formulation cannot simultaneously guarantee unit-scale expert application and provide a useful gradient for learning the router.

Instead, we enforce top-1 routing via discrete expert selection. For a token \(x\): \[\begin{align} \mathbb{P} &= \operatorname{softmax}(\mathbf{G}(x)) \\ j &= \arg\max_i( \mathbb{P}_{i}) \end{align}\]

This produces a one-hot routing decision. However, the \(\arg\max\) operation is non-differentiable, preventing backpropagation through the router. To enable training, we use the STE [25], which keeps the hard decision in the forward pass but approximates gradients using the soft probabilities in the backward pass: \[\begin{align} & Y_{\text{hard}} = \text{one-hot}(j) \\ & Y = \mathrm{SG}(Y_{\text{hard}} - \mathbb{P}) + \mathbb{P} \end{align}\] where \(\mathrm{SG}(\cdot)\) stops gradients. The forward pass uses the discrete routing vector \(Y_{\text{hard}}\), while gradients propagate through \(P\). The final output becomes \[\mathrm{STE}(x) = \sum_{i=1}^K Y_i E_i(x) = E_j(x)\] which guarantees unit scaling of the selected LoRA expert while still allowing the router to be optimized via gradient descent.

4.1.0.1 Parameter Efficiency.

All pretrained experts remain frozen. A single linear router is shared across layers, and only a small LoRA module on attention layers is trainable during integration. The number of trainable parameters, therefore, does not grow with the number of experts while maintaining sparse computation.

4.2 Training↩︎

There are two stages in the training: As shown in the Fig 1. In Stage I: we train domain-specific LoRA experts using RLVF [9], [10]. In Stage II: the LoRA adapters are frozen and we only train the router using SFT on distilled traces.

Stage I: Training Reasoning LoRA Experts↩︎

Each dataset trains one LoRA adapter independently. The adapters are optimized using RLVF with GRPO so that each expert specializes in domain-specific reasoning and produces CoT traces. After training, we obtain a set of reasoning experts: \(\left\{ E_1, E_2, \dots, E_K \right\}\)

Stage II: Mixture Integration via Distillation↩︎

4.2.0.1 Distillation Data Generation.

We query each trained expert with prompts from its domain and collect generated reasoning traces. A limited number of samples is sufficient to construct the SFT dataset, since the objective of this Stage is to learn the routing policy rather than to relearn reasoning ability. The reasoning knowledge is already encoded in the frozen experts, and therefore, the training data only needs to cover representative input patterns for correct expert selection.

4.2.0.2 Router and Attention Adaptation.

All experts remain frozen. We train a shared linear router, and LoRA adapters on attention layers using standard next-token likelihood on the distilled traces. The objective is to learn expert selection rather than relearn reasoning ability.

5 Experiments↩︎

We evaluate Hard-Routed MoR-LoRA along two independent dimensions:

Experimental Setup↩︎

We conduct experiments on instruction-following language model Meta-LLaMA-3 [26] at three model scales, 1B, 3B, and 8B² parameters. We also evaluate additional model families in Appendix 11.

Datasets. To evaluate performance across heterogeneous domains, we consider five representative tasks:

These tasks are evaluated using the GSM8K [27], ARC-Challenge (ARC-C) [28], Medical Question Answering (MedQA) [29], BoolQ [30], and Corpus of Linguistic Acceptability (CoLA) [31] datasets, respectively. Each dataset covers a distinct domain with little to no overlap.

5.0.0.1 Compared Models.

We use Independent to denote standalone dataset-specific LoRA adapters before mixture integration. These models are evaluated to measure the quality of individual experts, while the mixture methods evaluate how pretrained experts are integrated into one unified model.

• Independent SFT: one dataset-specific LoRA adapter trained with supervised fine-tuning on the FFN layers.

• Independent RLVF: one dataset-specific LoRA adapter trained with reinforcement learning from verifiable feedback on the FFN layers.

• LoRAHub: static integration by learning fixed merge weights over pretrained LoRA adapters.

• LoRAMixer TopK=1: soft top-1 routing with continued adapter training using an L2 preservation loss.

• LoRAMixer TopK=2 Normalized: normalized soft top-2 routing with continued adapter training using an L2 preservation loss.

• MoLE: MoE-style LoRA integration with soft routing and load-balancing regularization.

• Ours (Hard-Routed MoR-LoRA): hard top-1 routing via STE, with frozen experts and trainable router plus attention LoRA.

Unless otherwise stated, Stage II integration uses at most 1000 samples per dataset. The independent adapters are trained separately for each dataset and then serve as the pretrained experts used by the integration methods.

5.1 RLVF Produces Stronger Reasoning Experts↩︎

We first evaluate the quality of individual experts trained in Stage I before any mixture integration. This isolates the effect of the training paradigm. Figure 2 summarizes the average accuracy across model scales, while full per-task results are provided in Appendix 8.1, Table ¿tbl:table:fn-rlvf?.

For medium and large models (3B and 8B), RLVF substantially improves performance on reasoning-intensive tasks. For example, the 3B model improves from 68.60 (SFT) to 72.69 with RLVF, and the 8B model improves from 72.72 to 79.72. Unlike fine-tuning, which encourages imitation of reference outputs, RLVF allows the model to explore intermediate reasoning steps and optimize correctness, leading to stronger multi-step reasoning behavior.

The results also show that ordinary fine-tuning can degrade performance in large instruction-tuned models. We attribute this to fine-tuning, overriding the model’s inherent COT behavior by encouraging short-form answer imitation. In contrast, RLVF preserves the aligned reasoning behavior because optimization depends only on verifiable correctness rather than matching a specific reasoning trace. For the 1B model, on the other hand, RLVF provides smaller gains compared to fine-tuning (32.61 vs 49.18). This is likely due to insufficient reasoning capacity as the 1B model struggles to produce diverse and correct reasoning trajectories, weakening the reward signal. Thus, the benefit of RLVF increases with model capability.

As detailed in Table ¿tbl:table:fn-rlvf?, the improvement primarily appears in reasoning-heavy datasets such as GSM8K and ARC-C, while simpler linguistic tasks such as CoLA show limited advantage from RLVF. This supports the view that RLVF primarily strengthens reasoning behavior rather than general language modeling.

5.2 Hard Routing Preserves Frozen Expert Behavior During Integration↩︎

We now evaluate Stage II, where pretrained experts are integrated using only 1000 samples per dataset. Figure 3 summarizes the integration results for both SFT experts and RLVF experts, and Appendix 8.2, Tables ¿tbl:table:mixer-fn? and ¿tbl:table:mixer-rlvf? report the full numbers across five tasks and trainable parameter counts.

In our frozen-expert integration setting, Hard-Routed MoR-LoRA achieves comparable or better performance than soft-routing baselines while requiring substantially fewer trainable parameters. Unlike LoRAMixer, which retrains the already-trained experts to compensate for the scaling mismatch introduced by soft routing, our architecture preserves the original LoRA formulation and therefore only learns expert selection while keeping the experts frozen. This suggests that much of the integration benefit comes from selecting appropriate frozen experts rather than extensively modifying their parameters. For example, on the 3B with RLVF experts, our method reaches 72.07 average while training \(\approx\)​73M parameters, whereas LoRAMixer Top-1 reaches 65.67 while training \(\approx\)​606M parameters.

As shown in Figure 3, the improvement from hard routing over soft routing is larger for the 1B and 3B models compared to the 8B model. From Stage I results, these smaller models benefit more from trained experts relative to the base model, so preserving expert behavior during integration is more important. Therefore, preserving the standalone behavior of each expert through hard unit-scale routing appears especially beneficial for smaller models. For the 8B model, the relative improvement is smaller but still consistent, with a particularly large gain on GSM8K as shown in Appendix Table ¿tbl:table:mixer-rlvf?, where the reasoning expert provides a strong improvement over the base model after RLVF training and hard routing better maintains its effectiveness.

5.3 Comparison with Additional LoRA Composition Baselines↩︎

The previous section compares Hard-Routed MoR-LoRA with LoRAMixer, which is the closest baseline to our setting. To provide a broader comparison, we also evaluate against two additional adapter-composition baselines: LoRAHub, a static adapter merging method, and MoLE, a representative MoE-LoRA method with load-balancing regularization. All methods are evaluated in the same Stage II setting using RLVF-trained experts and at most 1000 integration samples per dataset.

Table ¿tbl:table:main-additional-baselines? summarizes the results. LoRAHub performs static composition and therefore cannot dynamically select experts at inference time. LoRAMixer and MoLE use soft expert routing, whereas our method applies exactly one frozen LoRA expert with unit scale. Across LLaMA-3B and LLaMA-8B, Hard-Routed MoR-LoRA achieves the best average performance among the learned-composition baselines. The improvement over MoLE is larger on LLaMA-3B and smaller but still positive on LLaMA-8B. These results suggest that hard unit-scale routing is a simple and parameter-efficient integration strategy for frozen pretrained LoRA experts, avoiding expert retraining and additional load-balancing hyperparameters. The full per-task results are reported in Appendix 8.3, Table ¿tbl:table:baseline-rlvf-full?.

5.4 Soft Routing Collapses Toward Single-Expert Selection↩︎

Hard routing resolves the LoRA integration mismatch by ensuring that exactly one expert with unit scaling is applied. In contrast, soft routing combines multiple experts using routing weights. While top-1 soft routing cannot maintain unit scaling, top-2 routing can be normalized so that the total update magnitude remains constant.

As shown in Table ¿tbl:table:mixer-fn? and Table ¿tbl:table:mixer-rlvf?, top-2 normalized routing performs substantially better than top-1 soft routing and approaches the performance of hard routing. However, examining the routing behavior reveals that the two experts are not used equally.

Figure 4 shows the distribution of the dominant routing weight, which is the larger weight among the two selected experts. The router usually assigns most of the probability mass to one expert, with an average dominant weight of about \(0.71\) and an average routing entropy of \(0.6245\). This shows that normalized top-2 routing often behaves like near top-1 routing. However, the model still evaluates and combines two LoRA experts, so the final update is a weighted mixture rather than the original unit-scale update of a standalone LoRA expert. Hard routing removes this extra step by directly selecting one expert, preserving sparsity and matching the original LoRA formulation more closely.

5.5 Ablation Study↩︎

We perform ablations to determine whether Stage II improves performance through additional learning or mainly through better expert selection. We also analyze the effect of attention LoRA rank in Appendix 9 and compare alternative hard-routing approximations in Appendix 12.

Is Reinforcement Learning Necessary for Integration?↩︎

Stage II of our method aims to learn expert selection rather than relearning reasoning behavior. During this stage, all experts remain frozen, and only the shared router and attention LoRA are optimized. We compare two training strategies: (1) SFT on distilled reasoning traces generated by the experts, and (2) RLVF training for the routing selection.

We present the results of the 3B model in Table ¿tbl:table:dist-rlvf?. RLVF achieves slightly higher performance (+0.93 average), but requires substantially higher computation due to trajectory sampling and advantage evaluation. The small improvement indicates that Stage II primary learns which expert to select, while the reasoning capability is already encoded in the pretrained experts. Therefore, SFT on distilled traces is sufficient for integration, and reinforcement learning is primarily necessary only during expert training.

Sample Training Budget↩︎

We analyze how many supervision samples are required for Stage II. For each dataset, we randomly sample prompts and generate distilled reasoning traces from the pretrained experts. The mixture model is then trained for one epoch on the combined distilled data from all five domains. We compare using 750 and 1000 prompts per dataset against using all available samples (30,573).

Table ¿tbl:table:training-samples? shows that using only 1000 prompts per dataset achieves performance close to training on the full dataset, with less than \(1\%\) difference in average accuracy. Increasing the amount of data yields only marginal gains, indicating that Stage II mainly learns routing patterns rather than domain knowledge. Since reasoning ability is already encoded in the experts trained in Stage I, only a small number of examples is sufficient to identify the appropriate expert.

Prompt-Level and Mixed-Domain Routing↩︎

The original evaluation uses datasets with clearly separated domains. This raises a natural question: whether token-level routing is necessary, or whether the problem can mostly be solved by selecting one expert for the whole prompt. To test this, we add a prompt-level routing baseline. This baseline trains a bert-base-uncased [32] classifier using the same Stage II supervision budget, with at most 1000 examples per dataset. The classifier predicts the input domain and then applies the corresponding frozen LoRA expert to the entire prompt.

The prompt-level classifier reaches 99.70% domain classification accuracy. On LLaMA-3B with RLVF-trained experts, As shown in Table ¿tbl:table:main-additional-baselines?, prompt-level routing marginally outperforms Hard-Routed MoR-LoRA. This shows that prompt-level routing is a strong and simple baseline when each prompt belongs clearly to one domain. Thus, token-level routing should not be interpreted as necessary for clean single-domain inputs. Full per-task results and classifier details are provided in Appendix 14.

However, prompt-level routing has a structural limitation: it must select a single expert for the whole input. To test this limitation, we construct a mixed-domain evaluation where each prompt contains one GSM8K math problem and one BoolQ question, and the model is asked to produce two structured answers. No method is trained on mixed-domain prompts.

As shown in Table ¿tbl:table:mixed-domain?, prompt-level routing performs better on the math part but considerably worse on the BoolQ part. This is expected because it must choose either the GSM8K expert or the BoolQ expert for the entire input. In contrast, ours is not restricted to one expert for the whole prompt and achieves better average performance. This result does not show that token-level routing fully solves mixed-domain reasoning, but it demonstrates why token-level expert selection is more flexible when a single input requires multiple expert behaviors. We additionally report unseen-dataset results on SVAMP and SST-2 in Appendix 14.3.

6 Conclusion↩︎

We study how independently trained LoRA adapters can be combined in instruction-tuned language models without changing their behavior. We find that an important challenge in frozen pretrained LoRA composition is the mismatch between unit-scale LoRA updates and weighted mixture routing. Prior LoRA-mixing methods often address this by adapting the experts during integration, which increases the number of trainable parameters. Hard-Routed MoR-LoRA instead treats integration mainly as a routing problem: reasoning experts are first trained independently using RLVF, and the mixture is then trained only to select among them while keeping all experts frozen. Across multiple model scales and tasks, we observe three consistent results. First, verifiable-feedback training produces stronger reasoning experts for capable instruction-tuned models. Second, hard routing integrates these experts more reliably and with far fewer trainable parameters than soft routing approaches. Third, normalized soft mixtures often concentrate most routing mass on one expert, helping explain why hard unit-scale selection can be effective without retraining the experts. Overall, our results show that modular adaptation can be achieved by learning when to use each expert rather than modifying what each expert knows, enabling practical reuse of domain experts under limited data or data-sharing constraints.

7 Limitations↩︎

Although our approach avoids retraining experts, integration is not completely training-free. Stage II still requires a some amount of labeled data to learn the routing behavior. As a result, adding new experts or domains requires additional routing data and training, and the framework does not yet support true zero-shot integration. This limitation is more significant in settings where only very few or no samples are available.

Our method is primarily evaluated in settings where inputs are associated with a small number of expert domains. In the clean single-domain setting, prompt-level routing is already highly competitive, showing that selecting one expert for the whole prompt can be sufficient when the domain is clear. At the same time, our mixed-domain GSM8K+BoolQ evaluation suggests a promising advantage of token-level routing: it is not restricted to a single expert decision for the entire input. With explicit mixed-domain training, the router could further learn when to switch experts within a prompt, making this formulation naturally suited to inputs that combine multiple types of reasoning. Similarly, our unseen-dataset results in Appendix 14.3 suggest that the router can transfer useful expert behavior to related task distributions. A natural extension is to add an abstention or backbone route, allowing the model to bypass all LoRA experts and follow the original computation graph when none of the available experts is appropriate. These directions point toward more flexible expert composition with larger expert pools, mixed-domain supervision, and adaptive routing between specialized experts and the base model.

Acknowledgment↩︎

This work was supported by Carl Bennet AB.

8 Detailed Experimental Results↩︎

This section provides the full per-task results corresponding to the averaged metrics reported in the paper. All experiments follow the two-stage pipeline: training standalone experts (Stage I) and integrating them through routing (Stage II).

8.1 Stage I: Standalone Expert Quality↩︎

Table ¿tbl:table:fn-rlvf? shows the performance of experts before mixture integration. We compare the original instruction-tuned model (Baseline), conventional fine-tuning (SFT), and RLVF.

The detailed numbers match the observations in Section 5.1. RLVF improves reasoning-heavy tasks for 3B and 8B models, while fine-tuning can reduce performance in larger instruction-tuned models. By contrast, for the 1B model, RLVF gives smaller gains due to limited reasoning capacity.

8.2 Stage II: Expert Integration↩︎

Tables ¿tbl:table:mixer-fn? and ¿tbl:table:mixer-rlvf? report mixture integration results for both fine-tuned experts and RLVF-trained experts.

We compare LoRAMixer top-1 routing, LoRAMixer normalized top-2 routing, and ours (Hard-Routed MoR-LoRA). Across both expert types, hard routing achieves better performance while training far fewer parameters. This indicates that integration mainly depends on selecting the correct expert rather than modifying expert weights.

Summary↩︎

Across all tables, three patterns are consistent:

• RLVF produces stronger experts for capable models.

• Soft routing requires retraining experts and more parameters.

• Hard routing preserves expert behavior with fewer trainable parameters.

These tables confirm that the improvements in the paper are consistent across datasets and model sizes.

8.3 Full Per-Task Results for Additional Baselines↩︎

In this section, we provide the full per-task results for the additional LoRA composition baselines. The paper reports average performance to keep the comparison compact, while the tables below show how each method behaves across individual datasets. We include three types of baselines: LoRAHub as a static adapter-merging method, LoRAMixer as a preservation-based soft-routing method, and MoLE as a MoE-LoRA method with load-balancing regularization. This allows us to compare static composition, soft expert mixing, and hard unit-scale routing under the same frozen-expert integration setting.

All results in this section use RLVF-trained experts from Stage I. During Stage II, each method is trained with the same supervision budget of at most 1000 samples per dataset. For LoRAMixer, the pretrained adapters are further optimized with a preservation loss. For MoLE, we evaluate two load-balancing coefficients, \(\alpha=0.5\) and \(\alpha=0.1\), to account for sensitivity to the balancing objective. In contrast, Hard-Routed MoR-LoRA keeps all pretrained experts frozen and trains only the shared router and lightweight attention LoRA.

Table ¿tbl:table:baseline-rlvf-full? provides the full per-task results. These results correspond to the averaged scores reported in Table ¿tbl:table:main-additional-baselines?.

9 Impact of LoRA Rank in Integration↩︎

As described in Section 4.2, Stage II integrates pretrained experts by training a shared linear router and applying LoRA adaptation to the attention layers. The router operates at the token level and selects one expert per token, while in standalone experts, the same adapter processes all tokens of a prompt. This change in routing granularity can alter token representations, so a small attention adaptation is introduced to improve the compatibility between token embeddings and routing decisions.

In this section, we analyze whether Stage II requires substantial parameter adaptation to learn expert selection. We vary the LoRA rank applied to the attention layers. The No LoRA setting trains only the shared router while keeping the model representations unchanged.

Table ¿tbl:table:ranks? reports the results when integrating the RLVF-trained experts from Stage I. For both model sizes, allowing a small amount of attention adaptation improves performance. However, the gains quickly saturate. For Llama-8B, \(R=32\) gives the highest score (79.93), while the No LoRA setting already achieves 78.65, which is close to the maximum.

These results indicate that Stage II primarily learns routing behavior rather than relearning domain knowledge or reasoning ability. The attention LoRA stabilizes token-level routing, but only a small rank is sufficient, and larger ranks provide little additional benefit.

10 Why Normalized Top-1 Soft Routing Has Zero Routing Gradient↩︎

This section explains why a purely soft top-\(1\) formulation cannot both preserve unit-scale LoRA application and provide a useful gradient for the router training.

Let \(p_i=\operatorname{softmax}(g(x))_i\) be the routing probability for expert \(i\) while \(g(x)\) is the logits produced by the router, and let \(j=\arg\max_i p_i\) be the selected top-\(1\) expert.

A simple top-\(1\) soft router can apply the selected expert as \[\begin{align} y_{\text{soft-top1}}(x)=p_j E_j(x) \end{align}\] This gives gradients to the router. For a fixed selected expert \(j\), \[\begin{align} \frac{\partial y_{\text{soft-top1}}}{\partial g_\ell} = E_j(x) \frac{\partial p_j}{\partial g_\ell} = E_j(x)p_j(\mathbb{I}[\ell=j]-p_\ell) \end{align}\] However, this applies the selected LoRA expert with weight \(p_j\) instead of weight \(1\). Since \(p_j < 1\) in general, the LoRA update is scaled down. This changes the unit-scale update that the expert used during standalone training.

To keep the selected expert at unit scale, one may normalize the probability over the selected top-\(1\) set \(S=\{j\}\): \[\begin{align} \tilde{p}_j = \frac{p_j}{\sum_{m\in S}p_m} = \frac{p_j}{p_j} = 1 \end{align}\] The output then becomes \[\begin{align} y_{\text{norm-top1}}(x) = \tilde{p}_j E_j(x) = E_j(x) \end{align}\] This preserves the original LoRA scale. However, for a fixed selected expert \(j\), the normalized weight is the constant function: \[\begin{align} \frac{\partial \tilde{p}_j}{\partial g_\ell} = \frac{\partial}{\partial g_\ell} \left(\frac{p_j}{p_j}\right) = 0. \end{align}\] Thus, the normalized routing weight gives no useful gradient to the router.

The only remaining dependence on the router is through the selection \(j=\arg\max_i p_i\). This selection is discrete. It is piecewise constant and is not differentiable. Therefore, normalized top-\(1\) routing preserves unit-scale expert application, but it cannot train the router with a useful task-loss gradient.

Hard routing with a STE avoids this problem. In the forward pass, it selects one expert with unit scale: \[\begin{align} y_{\text{forward}}(x) = E_j(x). \end{align}\] In the backward pass, it uses the soft probabilities as a surrogate gradient: \[\begin{align} Y = \operatorname{StopGradient}(Y_{\text{hard}} - p) + p. \end{align}\] This gives unit-scale expert selection in the forward pass while still allowing the router to be trained by gradient descent.

11 Results on Additional Model Families↩︎

In addition to the LLaMA models used in the main experiments, we also evaluate our method on two other model families: Gemma-3 [33] and Qwen-\(2.5\) [34]. The goal of this experiment is to verify that the observations reported in the paper are not limited to the LLaMA architecture.

Tables ¿tbl:table:additional-fn-rlvf? and ¿tbl:table:additional-mixer? report the results for Gemma-3-4B and Qwen-\(2.5\)-7B, respectively. Table ¿tbl:table:additional-fn-rlvf? shows the standalone expert performance, while Table ¿tbl:table:additional-mixer? reports the integration results. As in the LLaMA experiments, RLVF produces stronger experts, and our method achieves the best integration performance while training far fewer parameters than LoRAMixer.

Overall, these results show that the main findings of the paper remain consistent across different model families. This suggests that the proposed routing framework is general and can be applied to any instruction-tuned architectures, not only the LLaMA family.

12 Comparison with Alternative Hard-Routing Approximations↩︎

Our main method uses deterministic hard top-\(1\) routing with a straight-through estimator (STE). This section compares it with alternative approximations for training a hard router while keeping all pretrained LoRA experts frozen. The goal is to determine whether the benefit comes from deterministic STE itself or from the broader property that the selected frozen LoRA expert is applied with unit scale during both training and inference.

We compare three variants.

1. Hard STE is our default method: the forward pass selects the top-\(1\) expert deterministically, while gradients are propagated through the soft router probabilities.

2. Gumbel-Softmax [35] uses hard Gumbel-Softmax sampling with a continuous relaxation in the backward pass. We anneal the temperature from \(\tau=1.5\) to \(\tau=0.1\) exponentially and keep the final temperature fixed for the last \(10\%\) of training.

3. Soft-train/hard-inference trains the model with a probability-scaled top-\(1\) expert output, but removes the softmax scaling at inference time and applies the selected expert with unit weight. This variant tests whether unit-scale routing is sufficient only at inference time, or whether the same behavior is also necessary during training.

All methods use the same Stage II setting: RLVF-trained experts are frozen, the router is shared across layers, attention LoRA is enabled, and the model is trained on distilled traces with at most 1000 examples per dataset.

Table ¿tbl:table:hard-routing-approximations? shows that Gumbel-Softmax performs similarly to deterministic STE but does not improve over it. On LLaMA-3B, Gumbel-Softmax reaches an average score of \(71.82\), compared with \(72.07\) for Hard STE. On LLaMA-8B, the gap is larger, with Gumbel-Softmax reaching \(78.71\) compared with \(79.80\) for Hard STE. This suggests that the exact surrogate-gradient estimator is not the main factor; both methods preserve the important forward-pass behavior of applying exactly one expert with unit scale.

In contrast, soft-train/hard-inference performs substantially worse. The drop is especially large on LLaMA-3B, where the average score decreases from \(72.07\) to \(66.49\). This indicates that simply removing the softmax scaling at inference time is not sufficient. If the router and attention adaptation are trained under probability-scaled expert outputs, the model learns under one effective adapter function but is evaluated under another. Therefore, the unit-scale hard-routing behavior should be used consistently during both training and inference.

We further compare STE and Gumbel-Softmax under different Stage II data budgets on LLaMA-3B. This analysis tests whether the routing approximation remains stable when the amount of routing supervision is reduced.

As shown in Table ¿tbl:table:gumbel-data-budget?, Gumbel-Softmax is competitive when 1000 samples per dataset are available, trailing deterministic STE by only \(0.25\) average points. However, when the budget is reduced to 750 samples per dataset, the gap increases to \(2.47\) average points. Because the pretrained experts are frozen, stochastic expert sampling cannot improve the experts themselves; it only changes which fixed expert is selected during router learning. This additional routing variance can be harmful when Stage II supervision is limited.

13 Reproducibility Details↩︎

This section provides additional implementation and evaluation details for reproducing the experiments. All datasets used in this work are public benchmarks. Unless otherwise stated, the same training configuration is used across all model families, including LLaMA, Gemma, and Qwen; only the underlying instruction-tuned backbone model changes. Additionally, the code is available at github.com/sar-molavi/hard-routed-mor-lora.

13.1 Datasets and Evaluation Protocol↩︎

The main experiments use five datasets, corresponding to five domains: mathematical reasoning, commonsense reasoning, medical question answering, reading comprehension, and grammatical acceptability. Specifically, we use GSM8K [27], ARC-Challenge [28], MedQA [29], BoolQ [30], and CoLA [31]. Additional unseen-dataset evaluation uses SVAMP [36] and SST-2 [37]. SVAMP and SST-2 are not used for Stage I expert training or Stage II mixer training.

All reported results are evaluated with greedy decoding. The evaluation metric is accuracy. Table ¿tbl:table:dataset-reproducibility? reports the public dataset sources and the number of evaluation examples used for each benchmark.

13.2 Stage I: Standalone Expert Training↩︎

In Stage I, we train one LoRA expert independently for each dataset. The base model is frozen, and only the LoRA parameters are updated. We train two types of standalone experts: supervised fine-tuning (SFT) experts and reinforcement learning from verifiable feedback (RLVF) experts.

For both SFT and RLVF experts, LoRA is applied to the feed-forward network projections: gate_proj, up_proj, and down_proj. The SFT experts are trained with supervised next-token likelihood. The RLVF experts are trained with the off-policy GRPO-style objective described in Section 3.3, using automatically verifiable rewards computed from final answers. The full Stage I training configuration, including LoRA target modules, rank, learning rate, batch size, gradient accumulation, number of epochs, precision, and maximum sequence length, is reported in Table ¿tbl:table:training-reproducibility?.

13.3 RLVF Reward and Verifier Construction↩︎

RLVF uses automatically verifiable rewards computed from final answers. We do not use human preference labels. For each prompt, multiple completions are sampled from a behavior policy, which is a few checkpoints behind the target policy. The target policy is then optimized using a token-level off-policy GRPO-style objective with importance ratios between the target policy and the behavior policy as detailed in Section 3.3.

The reward is \(1.0\) for a correct answer and \(-1.1\) for an incorrect answer. We also use two small formatting bonuses: a think-format bonus of \(0.2\) when the response follows the expected explicit thinking format, and a JSON-on-wrong bonus of \(0.1\) when an incorrect response still produces a valid structured JSON answer. These bonuses are used to stabilize response formatting and answer extraction during RLVF training. The expected response format consists of an explicit reasoning block followed by a structured JSON answer:

Table ¿tbl:table:rlvf-reproducibility? summarizes the RLVF configuration.

13.4 Stage II: Distillation and Mixer Training↩︎

Stage II integrates the pretrained experts into one shared model. All Stage I experts remain frozen. We train only a shared linear router and a lightweight LoRA adaptation on the attention projections.

To construct the Stage II training data, shown in Figure 1, we query each frozen expert using prompts from its own domain and collect the generated reasoning traces. These traces form the supervised distillation dataset. The mixer is then trained with standard next-token likelihood on the distilled traces. The objective of Stage II is to learn expert selection rather than to relearn the domain knowledge or reasoning ability encoded in the frozen experts.

Unless otherwise stated, Stage II uses at most 1000 prompts per dataset. The attention LoRA is applied to: q_proj, k_proj, v_proj, and o_proj with a low rank. The router is shared across layers, and routing is performed at the token level. During training and inference, the selected LoRA expert is applied with unit scale.

13.5 Training Hyperparameters↩︎

Table ¿tbl:table:training-reproducibility? summarizes the main training configuration for Stage I and Stage II. The same configuration is used across LLaMA, Gemma, and Qwen experiments unless explicitly stated otherwise. The global batch size is \(32\) in all stages, and we use gradient accumulation and gradient checkpointing.

5pt

13.6 RLVF Training Dynamics↩︎

To make the behavior of Stage I RLVF training more transparent, we report training dynamics for the standalone RLVF experts. We track three quantities during training: completion length, verifier reward, and stop rate, whether the generation hit the end-of-sentence token. These statistics are collected from sampled trajectories during RLVF optimization and are, therefore, different from the final benchmark results, which are evaluated separately using greedy decoding.

For each training step, we compute the per-step mean over the few sampled completions. We also report a bias-corrected exponential moving average (EMA) with decay \(0.95\) to show the overall trend more clearly. The curves cover the five main domains, ARC-C, BoolQ, CoLA, GSM8K, and MedQA, for both LLaMA-3B and LLaMA-8B experts.

The reward is computed using the same automatic verifier used during RLVF training. Since trajectories are sampled with high-temperature exploration on a few training samples using offline reinforcement learning, the reward curves are expected to fluctuate and should not be interpreted as final task accuracy.

Figure 5 reports the mean verifier reward. Across most datasets and model sizes, the smoothed reward increases during training. This indicates that RLVF improves the sampled trajectories with respect to automatically verifiable correctness. The trend is especially clear for datasets such as GSM8K, BoolQ, and CoLA, where the reward rises steadily after the early training phase. Some non-monotonic behavior remains, particularly for the larger model on certain tasks, which is expected because the policy continues to sample diverse trajectories during training. Overall, the reward dynamics support the main Stage I observation that RLVF improves standalone experts by optimizing verifiable correctness rather than imitating fixed reference traces.

Figure 6 shows the generated completion length during training. The absolute completion length differs substantially across datasets because the tasks require different output formats and reasoning depths. For several experts, completion length decreases or stabilizes as training progresses, suggesting that the model learns to produce more concise outputs that satisfy the verifier. However, completion length itself is not optimized as the primary objective, and longer or shorter generations should not be interpreted as better performance without considering the corresponding reward and stopping behavior.

Finally, Figure 7 shows the stop rate, defined as the fraction of sampled completions that satisfy the generation stopping criterion before reaching the maximum generation budget. This statistic helps distinguish useful changes in completion length from unstable generation behavior. For most experts, the stop rate either remains high or increases toward a stable value during training, indicating that the models generally learn to terminate their responses correctly under the required output format. Some datasets exhibit temporary drops or more variable stopping behavior, especially under continued high-temperature exploration. These fluctuations are consistent with the reward dynamics and do not necessarily indicate degradation in final greedy-decoding performance.

14 Prompt-Level Routing and Additional Evaluation↩︎

This appendix provides additional details for the prompt-level routing baseline, the mixed-domain evaluation, and the unseen-dataset evaluation discussed in the paper.

14.1 Prompt-Level Routing Baseline↩︎

The prompt-level routing baseline selects one expert for the entire input. We train a bert-base-uncased classifier to predict the domain of the input prompt. The predicted domain is then used to select the corresponding frozen LoRA expert for the whole generation. This baseline uses the same Stage II supervision budget as our mixer: at most 1000 examples per dataset.

The classifier achieves near-perfect domain classification accuracy, as shown in Table ¿tbl:table:prompt-classifier-accuracy?. This confirms that the original single-domain evaluation contains a strong domain-classification component.

Table ¿tbl:table:prompt-routing-full? reports the full per-task results for prompt-level routing on LLaMA-3B and LLaMA-8B with RLVF-trained experts. Prompt-level routing slightly outperforms Hard-Routed MoR-LoRA on average in the clean single-domain setting. This supports the conclusion in the main paper: when each prompt clearly belongs to one domain, selecting one expert for the whole prompt is a strong and simple solution.

14.2 Mixed-Domain Evaluation Details↩︎

The prompt-level baseline is strong when each input belongs to a single domain, but it must select one expert for the entire prompt. To test this limitation, we construct a mixed-domain evaluation using GSM8K and BoolQ. Each input contains one GSM8K math problem and one BoolQ question, and the model is asked to produce two structured answers: math_answer and boolq_answer. No method is trained on mixed-domain prompts. Table ¿tbl:table:mixed-domain-setup? details the process.

We choose GSM8K and BoolQ because the prompt-level classifier reaches 100.00% accuracy on both datasets in the original single-domain setting. Thus, in the clean setting, the prompt-level classifier can reliably identify both domains. The mixed-domain setting tests a different issue: whether selecting only one expert for the whole prompt is sufficient when the input contains two different types of questions.

Prompt-level routing performs better on the math part, while Hard-Routed MoR-LoRA performs substantially better on the BoolQ part and obtains higher average performance. This shows the structural limitation of prompt-level routing: it can only choose either the GSM8K expert or the BoolQ expert for the whole input. In contrast, token-level routing is not restricted to one expert for the entire prompt. Furthermore, the token-level routing can be optimized for this situation, contrary to the prompt-level setting.

14.3 Unseen-Dataset Evaluation↩︎

We further evaluate whether the learned router and frozen experts transfer to related datasets that are not used during Stage I expert training or Stage II mixer training. We use SVAMP for mathematical reasoning and SST-2 for sentiment classification. Here, Baseline denotes the original instruction-tuned model without LoRA experts or routing. This experiment tests transfer to unseen but related task distributions.

As shown in Table ¿tbl:table:ood-eval?, Hard-Routed MoR-LoRA improves over the original instruction-tuned model on both unseen datasets and both model scales. On SVAMP, the routed model transfers arithmetic reasoning behavior to a math dataset that was not used during training. On SST-2, which is also not one of the original five domains, the routed model improves over the baseline as well. These results suggest that the router is not limited to memorizing the original training datasets, and that frozen expert behavior can transfer to related unseen inputs. However, this experiment should be interpreted as transfer to related task distributions rather than general open-domain robustness.

References↩︎

---

1. Our code is available at github.com/sar-molavi/hard-routed-mor-lora↩︎

2. https://huggingface.co/meta-llama/models↩︎