1 Introduction↩︎

Mixture-of-Experts (MoE) architectures enable scalable large language models (LLMs) by activating only a subset of experts per token [1]–[5]. However, this efficiency does not extend to fine-tuning, where updating all experts remains prohibitively expensive [6]–[8]. These challenges are further compounded by expert imbalance: some experts are consistently over-utilized while others are rarely activated [2], [9], making uniform parameter allocation particularly wasteful during adaptation.

Parameter-efficient fine-tuning (PEFT) methods such as LoRA  [10]–[12] reduce training cost but ignore MoE routing dynamics, applying updates uniformly across experts. In dense models, every parameter receives consistent gradient updates across tokens. In contrast, MoE layers activate only a small subset of experts per token, resulting in sparse and uneven parameter updates. This discrepancy means that uniform LoRA allocation leaves critical experts under-adapted while wasting capacity on rarely selected ones [2], [9], [13], [14]. Existing MoE-aware PEFT approaches either lack effective reallocation mechanisms or incur substantial parameter overhead [15], [16].

We propose EPnG (Expert Prune-and-Grow LoRA), an adaptive PEFT framework for efficient MoE fine-tuning. EPnG estimates expert importance from router statistics, prunes low-importance experts, and reallocates the released budget to high-importance experts by expanding their LoRA ranks with orthogonal initialization. This dynamic reallocation aligns parameter allocation with routing behavior and concentrates capacity on task-relevant experts.

Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms static LoRA under the same parameter budget and achieves performance comparable to full fine-tuning while updating over 140× fewer parameters. These results demonstrate that coupling PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

2 Background↩︎

2.0.0.1 Mixture-of-Experts.

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) via conditional computation [3]–[5]. Given an input token representation \(x \in \mathbb{R}^d\), a router selects the top-\(k\) experts from \(\mathcal{E} = \{E_1, \dots, E_M\}\). The router produces logits \(h(x) = W_\text{router} x \in \mathbb{R}^M\), which are normalized via softmax to obtain gate probabilities: \[p_i(x) = \frac{e^{h(x)i}}{\sum_{j=1}^M e^{h(x)j}}, \quad i = 1, \dots, M.\] The selected expert set is \(\mathcal{S}(x) = \operatorname{Top-k}(p(x)), \;|\mathcal{S}(x)| = k,\) and the MoE output is \[\mathrm{MoE}(x) = \sum_{i \in \mathcal{S}(x)} p_i(x) E_i(x)\]

While this sparse activation enables efficient scaling, expert utilization is highly imbalanced in practice [1], [17], motivating the need to quantify expert importance via routing signals.

2.0.0.2 Low-Rank Adaptation.

Low-Rank Adaptation (LoRA) [10] enables parameter-efficient fine-tuning by injecting low-rank updates into pretrained weights. For \(W \in \mathbb{R}^{d \times d}\): \[W' = W + \Delta W, \quad \Delta W = AB,\] where \(A \in \mathbb{R}^{d \times r}\), \(B \in \mathbb{R}^{r \times d}\), and \(r \ll d\). This reduces trainable parameters from \(d^2\) to \(2dr\).

2.0.0.3 Challenges in MoE Fine-Tuning.

In dense models, parameters are uniformly updated across tokens, whereas MoE activates only a subset of experts, leading to sparse and uneven updates. Applying LoRA uniformly across experts thus results in inefficient parameter allocation, motivating MoE-aware adaptation strategies.

3 Methodology↩︎

We propose EPnG (Expert Prune-and-Grow), an adaptive PEFT framework for efficient MoE fine-tuning. As illustrated in Figure 2, EPnG collects router gate probabilities during training, aggregates them into expert importance scores, prunes low-importance experts, and reallocates the released budget by expanding the LoRA ranks of high-importance experts. This prune-and-grow loop dynamically concentrates capacity on task-relevant experts under a fixed parameter budget.

3.1 Expert Importance Score Aggregation↩︎

We estimate expert importance from router gate probabilities collected during fine-tuning (Figure 2, left). For each token \(x^{(n)}\), the router outputs a distribution over experts, \(\{p_i(x^{(n)})\}_{i=1}^M\). We aggregate these probabilities across tokens by averaging them (Figure 2, middle): \[\bar{p}_i = \frac{1}{N} \sum_{n=1}^N p_i(x^{(n)}),\] where \(N\) is the total number of processed tokens. We refer to \(\bar{p}_i\) as the Expert Importance Score of expert \(E_i\). This score is the basis for pruning and rank reallocation.

3.2 Pruning Low-Importance Experts↩︎

Using the computed importance scores, we identify underutilized experts for pruning and reallocate the released budget in the growing stage (Section 3.3).

3.2.0.1 Expert Selection for Pruning.

We define the pruning threshold \(\tau_p\) such that the lowest \(\alpha\) fraction of expert importance scores \(\{\bar{p}_i^{(l)}\}\) fall below it, where \(\alpha \in (0,1)\) is the pruning ratio. The pruned set is \[\mathcal{P} = \{ E_i^{(l)} \;|\; \bar{p}_i^{(l)} \leq \tau_p,\; l \in \{1,\dots,L\},\; i \in \{1,\dots,M\} \}.\] Here, \(E_i^{(l)}\) denotes expert \(i\) in layer \(l\), \(\bar{p}_i^{(l)}\) its importance score, \(L\) the number of MoE layers, and \(M\) the number of experts per layer. Repeating this thresholding for \(t\) cycles yields a cumulative pruned fraction of \(1 - (1-\alpha)^t\).

3.2.0.2 Pruning Operation.

For each selected expert \(E_i^{(l)} \in \mathcal{P}\), we remove its LoRA parameters: \[\Delta W_i^{(l)} = 0, \quad \forall E_i^{(l)} \in \mathcal{P}.\] This removes the expert’s LoRA contribution and optimizer state, freeing parameter budget for growth.

3.3 Growing High-Importance Experts↩︎

We reallocate the budget released by pruning to highly utilized experts by expanding their LoRA ranks. We focus on the budget-neutral setting, where growth matches the budget released by pruning.

3.3.0.1 Expert Selection for Growth.

We define the growth threshold \(\tau_g\) such that the top \(\beta\) fraction of importance scores \(\{\bar{p}_i^{(l)}\}\) exceed it, where \(\beta \in (0,1)\) controls the growth ratio. The selected set is \[\mathcal{G} = \{ E_i^{(l)} \;|\; \bar{p}_i^{(l)} \geq \tau_g,\; l \in \{1,\dots,L\},\; i \in \{1,\dots,M\} \}.\] Repeated application for \(t\) cycles yields a cumulative grown fraction of \(1 - (1-\beta)^t\).

3.3.0.2 Rank Expansion.

For each expert \(E_i^{(l)} \in \mathcal{G}\), we expand its LoRA rank from \(r_i^{(l)}\) to \(r_i^{(l)} + \Delta r\) by appending new low-rank components. If the original update is \(\Delta W_i^{(l)} = A_i^{(l)} B_i^{(l)}\), the expanded update becomes: \[\Delta W_i^{(l)\prime} = [A_i^{(l)} \;\;A_i^{\text{new},(l)}] \begin{bmatrix} B_i^{(l)} \\ B_i^{\text{new},(l)} \end{bmatrix}.\]

3.3.0.3 Initialization and Orthogonalization.

To ensure stable and non-redundant expansion, \(A_i^{\text{new},(l)}\) is initialized with Kaiming initialization [18], while \(B_i^{\text{new},(l)}\) is initialized to zero, following LoRA [10]. We further orthogonalize the new directions against the existing subspace: \[A_i^{\text{new},(l)} \leftarrow (I - Q_i^{(l)} {Q_i^{(l)}}^\top) A_i^{\text{new},(l)},\] where \(Q_i^{(l)} = \text{orth}(A_i^{(l)})\) is an orthonormal basis spanning the columns of \(A_i^{(l)}\).

3.4 Prune-and-Grow Adaptation Loop↩︎

3.4.0.1 Warm-up for Stable Importance Estimation.

Because expert importance evolves during fine-tuning, we first use a warm-up stage of \(T_w\) steps, during which gate statistics are collected but no pruning or growth is applied.

3.4.0.2 Adaptive Parameter Reallocation.

After warm-up, the prune-and-grow procedure is triggered every \(T_p\) steps. At each interval, we compute \(\{\bar{p}_i^{(l)}\}\), prune low-importance experts (Section 3.2), and expand high-importance experts (Section 3.3). Under the constraint \(\alpha \geq \beta\) and assuming uniform rank expansion per expert, pruning releases equal or more parameters than growth consumes, ensuring that the total number of trainable parameters does not exceed the initial budget.

3.4.0.3 Summary.

The prune-and-grow loop thus turns finetuning into a dynamic, budget-aware adaptation process. By reallocating LoRA parameters according to router-derived importance signals, it promotes (i) consistent utilization of the limited parameter budget, (ii) reinforcement of task-relevant experts, and (iii) suppression of redundant adaptations. Algorithm 3 provides the pseudocode for the full training loop, and the overall workflow is illustrated in Figure 2.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Datasets and evaluation.

We evaluate EPnG across three domains: mathematical reasoning, code generation, and personalization.

Mathematical reasoning. We fine-tune on MetaMathQA [19], a bootstrapped dataset of diverse math questions, and evaluate on GSM8K [20], which consists of grade-school arithmetic word problems, and MATH [21], a large-scale benchmark covering competition-level mathematics. Performance is measured by Exact Match (EM), i.e., the proportion of predictions that exactly match the ground-truth answer.

Code generation. We fine-tune on Code Alpaca [22], an instruction-following dataset for programming tasks, and evaluate on HumanEval [23] and MBPP [24], two widely adopted benchmarks for code synthesis. We report pass@10 [23], which estimates the probability of generating at least one correct solution within ten independently sampled candidates.

Personalization. We evaluate on PrefEval [25], a benchmark designed to measure how well models infer and follow user-specific preferences in conversation. Model outputs are assessed via pairwise comparison using GPT-4o-mini [4] as an automatic evaluator, following the official preference following accuracy metric.

To ensure fair and reproducible comparison, all decoding and evaluation settings are fixed across methods. For generation tasks, we use temperature \(=0.8\), top-\(p=0.95\), and a maximum generation length of 512 tokens. For pass@10 evaluation, we sample 10 independent outputs per problem. For PrefEval, we use identical prompts and random seeds across all methods.

4.1.0.2 Models and baselines.

We conduct experiments on two public MoE backbones: allenai/OLMoE-1B-7B-0125 [26] and Qwen/Qwen1.5-MoE-A2.7B [27]. We compare four adaptation strategies: full fine-tuning (FFT), static LoRA, ESFT [15], and EPnG. FFT updates all model parameters and serves as an upper-bound reference. Static LoRA applies a fixed rank to all experts. ESFT is a recent MoE-specific PEFT baseline, and EPnG adaptively reallocates LoRA capacity using expert importance scores.

4.1.0.3 Implementation details.

LoRA adapters are applied to the up_proj and gate_proj matrices. Unless otherwise stated, we use pruning ratio \(\alpha=0.2\), growth ratio \(\beta=0.2\), warm-up steps \(T_w=100\), update interval \(T_p=50\), and exponential decay factor \(\lambda=0.2\) for gate statistics. We follow the standard evaluation protocol for each benchmark: EM for GSM8K and MATH, pass@10 for HumanEval and MBPP, and pairwise preference evaluation on PrefEval using GPT-4o-mini.

4.2 Main Results↩︎

4.2.0.1 Results on OLMoE

Table 1 shows that EPnG updates only \(0.72\%\) of parameters, over \(140\times\) fewer than FFT, while achieving comparable overall performance. In particular, EPnG outperforms FFT on GSM8K (\(66.26\) vs.\(64.97\)) and improves over static LoRA under the same budget by +1.82 points on GSM8K and +1.73 points on average. This shows that reallocating LoRA capacity across experts is more effective than uniform allocation.

4.2.0.2 Results on Qwen1.5-MoE

Table 2 shows that EPnG remains effective on a larger MoE backbone. Although EPnG tunes only \(0.55\%\) of parameters, compared with \(15.91\%\) for ESFT, it matches ESFT in average performance (\(58.98\)). Under the same budget as static LoRA, EPnG improves MATH from \(34.96\) to \(37.32\) and HumanEval from \(70.12\) to \(74.39\), indicating that dynamic expert-wise reallocation scales favorably.

4.2.0.3 Overall comparison.

Across both backbones, EPnG consistently improves or matches stronger baselines while tuning less than \(1\%\) of parameters. On OLMoE, it approaches FFT with over \(140\times\) fewer trainable parameters; on Qwen1.5-MoE, it matches ESFT with about \(29\times\) fewer. Overall, these results demonstrate that routing-aware prune-and-grow provides a scalable and parameter-efficient alternative to static LoRA and costly full fine-tuning.

5 Further Analysis↩︎

We provide additional analyses to understand why EPnG improves performance under tight parameter budgets. Unless otherwise stated, all analyses are conducted on GSM8K with OLMoE. Figures 4–6 and Tables 3–4 summarize the results.

5.0.0.1 Ablation study.

We conduct an ablation study to examine the contributions of pruning and growing (Figure [fig:analysis95ablation]). Compared to the static LoRA baseline (65.50%), applying either growing (64.97%) or pruning (65.81%) in isolation yields comparable or slightly improved performance. Notably, pruning proves more effective than growing, as it achieves better accuracy with fewer parameters than the baseline. In contrast, combining both operations leads to the best result (66.41%), indicating that pruning and growing complement each other: pruning eliminates redundant components, while growing reinforces useful ones.

We speculate that pruning acts as an implicit regularizer, removing redundancy and encouraging the model to concentrate on essential representations. Growing, by contrast, may introduce noise without resolving redundancy, and newly added parameters can be underutilized early in training. Together, these effects explain why combining both operations outperforms either alone.

5.0.0.2 Rank sensitivity.

Figure [fig:analysis95rank] shows that static LoRA is sensitive to the choice of base rank, with noticeable performance fluctuations across ranks. EPnG remains considerably more stable, suggesting that dynamic reallocation reduces the need for precise rank tuning.

5.0.0.3 Effect of the prune-and-grow fraction.

Figure [fig:analysis95fraction] studies the prune-and-grow fraction. Performance improves as the fraction increases up to around 0.2, after which the gains saturate. This indicates that moderate reallocation provides the best trade-off between removing redundancy and preserving sufficient expert diversity.

5.0.0.4 Layer-wise behavior.

Figure 5 visualizes how pruning and growing are distributed across layers, showing that shallower layers are pruned more heavily while deeper layers receive expanded ranks. This demonstrates that EPnG adaptively concentrates capacity toward more task-relevant layers rather than allocating it uniformly.

5.0.0.5 Expert importance dynamics.

A key motivation for EPnG is that expert importance is not static during fine-tuning. To verify this, we compare expert importance scores before and after LoRA fine-tuning on 1,000 MetaMathQA examples using OLMoE. Figure 6 shows that the set of highly important experts changes substantially after fine-tuning: some experts remain consistently important, while others newly emerge or become less relevant. This confirms that static expert allocation is suboptimal and supports the need for repeated prune-and-grow updates during training.

5.0.0.6 Generalization after fine-tuning.

To test whether EPnG preserves the backbone’s general capability, we evaluate fine-tuned models on ARC-C, ARC-E, and BoolQ without task-specific adaptation. Table 3 shows that EPnG preserves performance close to the base model, whereas FFT degrades more noticeably. This suggests that EPnG improves task adaptation without substantially compromising general capabilities.

5.0.0.7 Effect of LoRA placement.

We also study whether adding LoRA to down_proj provides additional gains beyond the default up_proj+ gate_proj setting. As shown in Table 4, although this increases the parameter count from 0.72% to 1.09%, EPnG consistently maintains strong performance, with improvements observed as the parameter budget increases.

5.0.0.8 Summary.

These analyses support the central design of EPnG. Pruning and growth are complementary, dynamic reallocation reduces rank sensitivity, moderate pruning works best, and the resulting capacity shifts concentrate on deeper, more task-relevant layers. Moreover, expert importance changes during fine-tuning, and general capability is preserved. Together, these findings explain why EPnG consistently outperforms static LoRA under the same parameter budget.

6 Discussion↩︎

A key limitation of EPnG is that pruning permanently removes LoRA parameters from under-utilized experts, making recovery impossible if a previously pruned expert becomes important in later training stages. To mitigate this risk, EPnG applies pruning gradually, targeting only the lowest-importance experts at each interval rather than aggressively removing a large fraction at once. This conservative strategy reduces the chance of prematurely discarding experts that may become task-relevant as training progresses. An alternative would be to retain pruned weights in memory and restore them when needed; however, this introduces non-trivial memory overhead and raises questions about whether retained parameters remain compatible with the evolved adapter subspace. We leave this direction to future work.

Beyond irreversibility, EPnG introduces additional hyperparameters compared to static LoRA, including the pruning ratio \(\alpha\), growth ratio \(\beta\), warm-up steps \(T_w\), update interval \(T_p\), and decay factor \(\lambda\), which may impose non-trivial tuning costs when adapting to new models or tasks. However, our experiments show that a single fixed configuration generalizes well across different datasets and model backbones, suggesting that EPnG is not overly sensitive to hyperparameter choices in practice.

7 Conclusion↩︎

We introduced EPnG (Expert Prune-and-Grow), a parameter-efficient fine-tuning framework for Mixture-of-Experts (MoE) models that dynamically reallocates LoRA capacity based on router-derived expert importance. By pruning under-utilized experts and expanding high-importance ones under a fixed parameter budget, EPnG aligns adaptation with MoE routing dynamics.

Across OLMoE and Qwen1.5-MoE, EPnG achieves performance comparable to or better than full fine-tuning and ESFT while updating only about \(1\%\) of parameters, and consistently outperforms static LoRA. Further analysis shows that EPnG stabilizes rank sensitivity and adaptively redistributes capacity toward deeper, more task-relevant layers, leading to more effective utilization of limited parameters.

Overall, EPnG demonstrates that coupling PEFT with dynamic expert allocation is a scalable and effective strategy for MoE fine-tuning under strict resource constraints. These results suggest that expert-wise parameter allocation should be treated as a dynamic optimization problem rather than a fixed design choice, an insight that opens broader opportunities for routing-aware adaptation in increasingly large and sparse language models.

References↩︎