Generic Expert Coverage for Pruning Sparse
Mixture-of-Experts Language Models
July 02, 2026
Abstract
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose Generic TB-Coverage, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25%, 50%, and 75% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25% and 50% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.
1 Introduction↩︎
Sparse Mixture-of-Experts (MoE) language models improve parameter efficiency by activating only a small subset of experts per token, achieving strong performance while keeping per-token computation manageable [1]–[3]. This sparse activation also creates structured redundancy: many experts can be removed with limited quality loss if the retained subset preserves the model’s core behaviors [4]. The challenge is selecting that subset without access to downstream validation data.
Existing expert-pruning methods usually rank experts using a single scalar criterion, such as routing frequency, reconstruction utility, or REAP-style importance [5]. However, when calibration data are heterogeneous, scalar aggregation can over-favor experts that dominate the average signal and under-retain experts that support less frequent but still important generic language behaviors. For example, consider two generic calibration corpora: WikiText2 (encyclopedic, narrative text) and C4 (broad web text). An expert that contributes strongly on encyclopedic patterns but not on web discourse will receive a moderate average score, even though it may be the top-ranked expert on WikiText2. A scalar ranking would likely discard it in favor of experts that score moderately on both corpora but are not critical for either. This suggests that expert retention should preserve cross-corpus coverage rather than only optimize a single aggregated importance score.
This problem is especially acute when the goal is general-purpose pruning without downstream calibration data. A principled pruning method should not rely on downstream evaluation tasks during calibration, since good downstream numbers could then partly reflect task leakage rather than genuine generalization. We study expert pruning in this practically important setting: can we prune MoE experts using only generic language corpora while preserving broad downstream behavior?
Our key observation is that even purely generic corpora do not exercise the same experts in the same way. A pruning rule that preserves only globally dominant experts can therefore reduce expert coverage across generic behaviors before downstream transfer is ever evaluated. We address this by profiling expert utility separately on multiple generic corpora and enforcing balanced protection across corpus-specific rankings.
We propose Generic TB-Coverage (Task-Balanced Coverage), a coverage-aware expert pruning method. The method profiles per-expert importance separately for each generic calibration corpus, builds corpus-specific expert rankings, and selects protected experts through a round-robin coverage rule. Protected experts are then merged into a reconstruction-stable candidate mask, and the exact retention budget is restored by removing the lowest-ranked unprotected experts. The entire procedure uses only WikiText2 and C4 for calibration and requires no fine-tuning.
We evaluate on two MoE language models—Qwen1.5-MoE-A2.7B [6] and DeepSeek-MoE-16B-Base [4]—at three expert-retention budgets (25%, 50%, 75%). The primary metric is the average accuracy over six common zero-shot benchmarks (ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande, BoolQ); we additionally report MMLU, GSM8K, and Math500 as auxiliary stress tests.
Our contributions are:
We identify a failure mode of scalar expert ranking for downstream-free MoE pruning: dominant calibration patterns can over-concentrate the retained expert set.
We propose a simple coverage-aware pruning rule that preserves high-utility experts across multiple generic corpora before fixed-budget mask construction.
We show that this rule improves pruning quality on two open MoE LMs across three retention budgets and six zero-shot benchmarks.
We analyze random-pruning variance and show that single-seed random baselines can be misleading in sparse MoE pruning.
2 Related Work↩︎
MoE model compression. Sparse MoE architectures [1], [2], [7] route each token to a small subset of experts, creating natural opportunities for expert-level pruning. ST-MoE [8] studies training stability in sparse models, while DeepSeek-MoE [4] introduces fine-grained expert segmentation that increases expert count and hence pruning potential. MoEfication [9] converts dense feed-forward layers into MoE structures.
Expert importance criteria. Several criteria have been proposed for ranking expert importance. Routing frequency counts how often each expert is selected by the router. REAP-style methods compute importance scores that combine routing probability with expert output magnitude. ExpertSparsity [5] uses reconstruction-based layerwise pruning, selecting experts to minimize the reconstruction error between the pruned and original MoE layer outputs. These methods share the limitation that they optimize a single scalar objective, which can cause over-concentration of retained experts.
LLM pruning. Beyond expert pruning, structured pruning for large language models includes layer removal [10], width pruning [11], and unstructured sparsity [12], [13]. SliceGPT [14] removes components via singular value decomposition. These methods operate at the weight or layer level rather than the expert level and are complementary to our approach.
Coverage and diversity in pruning. The idea of preserving diversity in pruned models has been explored in network pruning for convolutional networks, where filter diversity criteria help maintain representational capacity. To our knowledge, Generic TB-Coverage is the first to explicitly introduce a coverage rule for expert-level MoE pruning based on per-corpus profiling.
3 Method↩︎
3.1 Problem Formulation↩︎
Consider a sparse MoE language model with \(L\) MoE layers. At each MoE layer \(l\), the model has \(N\) routed experts \(\{e_1, \ldots, e_N\}\). For each input token with hidden state \(h_x\), the router selects the top-\(k\) experts and computes a weighted sum of their outputs: \[\text{MoE}(h_x) = \sum_{e \in \text{TopK}(h_x)} g_e(x) \cdot f_e(h_x),\] where \(g_e(x)\) is the router’s softmax probability assigned to expert \(e\) for token \(x\) (set to zero for non-selected experts), and \(f_e(h_x)\) is the expert’s feed-forward output on the hidden state \(h_x\).
Given a retain ratio \(\rho \in (0, 1]\), the pruning budget retains \(K = \lfloor \rho N \rfloor\) routed experts per MoE layer. The goal is to construct a binary expert mask \(m_l \in \{0,1\}^N\) with exactly \(K\) ones per layer such that the pruned model preserves broad language performance. After pruning, the router selects among the retained experts only.
3.2 Generic Expert Profiling↩︎
The method uses two generic calibration corpora: WikiText2 [15] and C4 [16]. For each corpus \(t \in \mathcal{T} = \{\text{WikiText2}, \text{C4}\}\), we run the model on calibration text and collect per-expert statistics independently for each MoE layer.
For each layer \(l\), expert \(e\), and calibration corpus \(t\), we compute a REAP-style utility score that combines the router weight with the expert output norm, averaged over all calibration tokens where expert \(e\) is selected by the router: \[\label{eq:score} s_t^{(l)}(e) = \mathbb{E}_{x \sim \mathcal{D}_t}\left[ g_e(x) \cdot \| f_e(h_x^{(l)}) \|_2 \;\big|\; e \in \text{TopK}(x) \right],\tag{1}\] where \(h_x^{(l)}\) is the hidden state at layer \(l\), \(g_e(x)\) is the router weight for expert \(e\) on token \(x\), and \(f_e(h_x^{(l)})\) is the expert output. The expectation is approximated by averaging over all tokens in the calibration set from corpus \(t\) for which expert \(e\) is among the top-\(k\) selected experts. This score captures both how strongly the router activates expert \(e\) (through \(g_e\)) and how much it contributes to the hidden state (through \(\|f_e\|_2\)). We profile each calibration corpus independently, producing a separate score vector \(\mathbf{s}_t^{(l)} = [s_t^{(l)}(e_1), \ldots, s_t^{(l)}(e_N)]\) per MoE layer.
3.3 Coverage-Aware Expert Protection↩︎
Instead of ranking experts by their average score \(\bar{s}^{(l)}(e) = \frac{1}{|\mathcal{T}|}\sum_{t} s_t^{(l)}(e)\), Generic TB-Coverage builds per-corpus rankings and selects protected experts by round-robin across corpora.
For each MoE layer \(l\), we define a per-layer protection budget \(B\) (a hyperparameter controlling how many experts receive coverage protection), chosen such that \(B \leq K\) for every layer and retention budget, guaranteeing feasibility. The round-robin procedure works as follows:
Sort all \(N\) experts by \(s_{\text{WikiText2}}^{(l)}(e)\) in descending order, producing ranking \(\pi_{\text{Wiki}}\).
Sort all \(N\) experts by \(s_{\text{C4}}^{(l)}(e)\) in descending order, producing ranking \(\pi_{\text{C4}}\).
Starting with an empty protected set \(P_l\), alternate between the two rankings: at each step, take the highest-ranked expert from the current corpus’s ranking that is not yet in \(P_l\), and add it.
Stop when \(|P_l| = B\).
This ensures that both corpora contribute to the protected set regardless of the absolute score magnitudes. An expert that is top-ranked on WikiText2 but mid-ranked on C4 will still be protected, even though its average score might not place it in the top-\(K\) by a single-criterion ranking. If both rankings agree on the top experts (high overlap), the round-robin degenerates to standard top-\(B\) selection, so the rule introduces no harm in that case.
3.4 Budget-Preserving Mask Construction↩︎
The final mask keeps exactly \(K\) experts per MoE layer. We initialize from a reconstruction-stable candidate mask \(\hat{m}_l\) (obtained via layerwise reconstruction minimization on C4) that already retains \(K\) experts per layer. We merge the protected set \(P_l\) into this candidate: set \(m_l(e) = 1\) for all \(e \in P_l\) (protected experts are always retained), then if the union exceeds \(K\), remove the lowest-ranked unprotected experts in order of ascending average score \(\bar{s}^{(l)}(e)\) until exactly \(K\) experts remain. Protected experts are never removed; the budget is restored entirely by dropping unprotected reconstruction-selected experts.
This procedure is budget-preserving: the final mask always retains exactly \(K\) experts per MoE layer. Algorithm 1 summarizes the full method.
3.5 Complexity Analysis↩︎
With \(|\mathcal{T}|\) calibration corpora, the additional profiling cost is \(|\mathcal{T}|\) forward-only passes over the calibration set (no gradient computation). With two corpora and approximately 512 sequences of length 1024 tokens, this is modest relative to model training. Mask construction involves per-layer sorting of \(N\) experts (\(O(N \log N)\) per layer) and round-robin selection (\(O(B)\) per layer), which is negligible relative to the profiling cost. The method stores only per-expert scalar scores per layer and does not require post-pruning fine-tuning.
4 Experiments↩︎
4.1 Setup↩︎
Models. We evaluate on two sparse MoE language models:
Qwen1.5-MoE-A2.7B [6]: 60 routed experts per MoE layer, with top-4 routing.
DeepSeek-MoE-16B-Base [4]: 64 routed experts per MoE layer, with top-6 routing.
Retain budgets. We report three expert-retention ratios: 25%, 50%, and 75%. For Qwen, this corresponds to retaining 15, 30, or 45 out of 60 routed experts per layer. For DeepSeek, this corresponds to retaining 16, 32, or 48 out of 64 routed experts per layer.
Calibration. Our method uses WikiText2 [15] and C4 [16] only. We draw 512 sequences of length 1024 tokens from each corpus. No downstream benchmark data is used during calibration.
Baselines. We compare against three baselines:
Random pruning: Uniform random expert removal per layer, reported as mean and standard deviation over six seeds (0, 1, 2, 3, 4, 42).
Original REAP: Direct top-\(k\) expert selection using REAP-style importance scores. For a fair comparison with a method that does use task-specific calibration, we run REAP with Evol-CodeAlpaca calibration (128 texts, sequence length 2048).
ExpertSparsity [5]: Reconstruction-based layerwise expert pruning using C4, adapted to our model wrappers. We follow the original paper’s layerwise reconstruction protocol.
We note that REAP uses domain-specific calibration data while our method uses only generic corpora, making the comparison favorable to REAP in terms of calibration informativeness.
Evaluation. We evaluate on two language modeling metrics (WikiText2 PPL, C4 PPL, computed on the standard validation splits using sliding-window evaluation) and six primary downstream tasks: ARC-Challenge and ARC-Easy [17], HellaSwag [18], PIQA [19], WinoGrande [20], and BoolQ [21]. We report Common Avg as the unweighted average over these six tasks. We additionally report MMLU [22], GSM8K [23], and Math500 [24] as auxiliary stress tests for knowledge and reasoning. All evaluations use full validation/test sets with zero-shot prompting (no few-shot examples).
4.2 Main Results↩︎
Table 1 presents the main results. Generic TB-Coverage achieves the highest Common Avg across both models and all three retain budgets.
| Model | Ret. | Method | Avg | MMLU | GSM | M500 | W.PPL | C.PPL |
|---|---|---|---|---|---|---|---|---|
| Qwen | 25% | Random | 0.417\(\pm\)0.021 | 0.240\(\pm\)0.011 | — | — | 920.67\(\pm\)744.23 | 3109.75\(\pm\)2197.97 |
| Original REAP | 0.438 | 0.247 | 0.000 | 0.000 | 193.65 | 551.91 | ||
| Paper ExpertSparsity | 0.407 | 0.233 | 0.000 | 0.000 | 160.93 | 358.90 | ||
| Generic TB-Coverage | 0.448 | 0.234 | 0.000 | 0.000 | 146.22 | 259.12 | ||
| Qwen | 50% | Random | 0.535\(\pm\)0.060 | 0.315\(\pm\)0.074 | 0.000\(\pm\)0.000 | 0.002\(\pm\)0.002 | 77.95\(\pm\)93.70 | 165.33\(\pm\)178.95 |
| Original REAP | 0.552 | 0.406 | 0.000 | 0.002 | 23.77 | 61.47 | ||
| Paper ExpertSparsity | 0.519 | 0.325 | 0.000 | 0.000 | 36.18 | 88.31 | ||
| Generic TB-Coverage | 0.598 | 0.369 | 0.000 | 0.004 | 16.76 | 47.01 | ||
| Qwen | 75% | Random | 0.645\(\pm\)0.041 | 0.516\(\pm\)0.035 | 0.000\(\pm\)0.000 | 0.003\(\pm\)0.004 | 13.25\(\pm\)1.49 | 33.80\(\pm\)5.17 |
| Original REAP | 0.629 | 0.519 | 0.000 | 0.002 | 13.78 | 32.81 | ||
| Paper ExpertSparsity | 0.655 | 0.548 | 0.000 | 0.002 | 11.92 | 29.41 | ||
| Generic TB-Coverage | 0.662 | 0.528 | 0.002 | 0.002 | 10.57 | 28.22 | ||
| DeepSeek | 25% | Random | 0.403\(\pm\)0.025 | 0.243\(\pm\)0.010 | — | — | 728.88\(\pm\)818.42 | 2640.77\(\pm\)2971.07 |
| Original REAP | 0.395 | 0.238 | 0.000 | 0.006 | 544.15 | 1734.92 | ||
| Paper ExpertSparsity | 0.404 | 0.240 | 0.000 | 0.000 | 176.63 | 615.18 | ||
| Generic TB-Coverage | 0.448 | 0.232 | 0.000 | 0.000 | 137.03 | 423.72 | ||
| DeepSeek | 50% | Random | 0.488\(\pm\)0.082 | 0.244\(\pm\)0.010 | 0.000\(\pm\)0.000 | 0.005\(\pm\)0.006 | 54.19\(\pm\)51.78 | 249.73\(\pm\)311.32 |
| Original REAP | 0.542 | 0.287 | 0.000 | 0.008 | 27.33 | 61.60 | ||
| Paper ExpertSparsity | 0.571 | 0.264 | 0.000 | 0.010 | 20.88 | 53.47 | ||
| Generic TB-Coverage | 0.606 | 0.242 | 0.000 | 0.006 | 16.43 | 45.14 | ||
| DeepSeek | 75% | Random | 0.568\(\pm\)0.116 | 0.260\(\pm\)0.019 | 0.000\(\pm\)0.000 | 0.006\(\pm\)0.009 | 19.19\(\pm\)9.55 | 80.94\(\pm\)81.39 |
| Original REAP | 0.649 | 0.340 | 0.000 | 0.000 | 14.08 | 29.94 | ||
| Paper ExpertSparsity | 0.678 | 0.314 | 0.000 | 0.004 | 12.17 | 27.70 | ||
| Generic TB-Coverage | 0.683 | 0.309 | 0.000 | 0.006 | 10.12 | 25.96 |
2pt
On Qwen1.5-MoE-A2.7B, Generic TB-Coverage improves Common Avg over Paper ExpertSparsity by +0.041, +0.080, and +0.007 at 25%, 50%, and 75% retain, respectively. On DeepSeek-MoE-16B-Base, the improvements are +0.044, +0.035, and +0.006. The gains are largest under aggressive pruning (25% and 50% retain), where retaining the right experts matters most. At 75% retain, all methods perform closer together since only a quarter of experts are removed.
Figure 2 visualizes the Common Avg comparison, and Figure 3 shows the PPL results. The perplexity improvements are substantial. For DeepSeek at 25% retain, C4 PPL drops from 1734.92 (Original REAP) and 615.18 (Paper ExpertSparsity) to 423.72 under Generic TB-Coverage. Similarly, WikiText2 PPL drops from 544.15 and 176.63 to 137.03. These reductions indicate that coverage-aware expert selection not only preserves downstream task accuracy but also stabilizes the language modeling distribution.
4.3 Random Multi-Seed Analysis↩︎
Random pruning is commonly used as a baseline, but single-seed results can be misleading. Table 2 shows that random pruning exhibits high variance across seeds.
| Model | Retain | Metric | Random | Ours |
|---|---|---|---|---|
| Qwen | 25% | Common Avg | 0.417\(\pm\)0.021 | 0.448 |
| Wiki PPL | 920.67\(\pm\)744.23 | 146.22 | ||
| C4 PPL | 3109.75\(\pm\)2197.97 | 259.12 | ||
| Qwen | 50% | Common Avg | 0.535\(\pm\)0.060 | 0.598 |
| Wiki PPL | 77.95\(\pm\)93.70 | 16.76 | ||
| C4 PPL | 165.33\(\pm\)178.95 | 47.01 | ||
| Qwen | 75% | Common Avg | 0.645\(\pm\)0.041 | 0.662 |
| Wiki PPL | 13.25\(\pm\)1.49 | 10.57 | ||
| C4 PPL | 33.80\(\pm\)5.17 | 28.22 | ||
| DeepSeek | 25% | Common Avg | 0.403\(\pm\)0.025 | 0.448 |
| Wiki PPL | 728.88\(\pm\)818.42 | 137.03 | ||
| C4 PPL | 2640.77\(\pm\)2971.07 | 423.72 | ||
| DeepSeek | 50% | Common Avg | 0.488\(\pm\)0.082 | 0.606 |
| Wiki PPL | 54.19\(\pm\)51.78 | 16.43 | ||
| C4 PPL | 249.73\(\pm\)311.32 | 45.14 | ||
| DeepSeek | 75% | Common Avg | 0.568\(\pm\)0.116 | 0.683 |
| Wiki PPL | 19.19\(\pm\)9.55 | 10.12 | ||
| C4 PPL | 80.94\(\pm\)81.39 | 25.96 |
2pt
At 75% retain on Qwen, a single random seed (seed 42) achieves Common Avg of 0.669, which appears competitive with our method’s 0.662. However, the multi-seed mean is 0.645\(\pm\)0.041, and the worst seed drops to 0.604. Similarly, on DeepSeek at 75%, random PPL ranges from 10.7 to 31.6 across seeds. This high variance demonstrates that single-seed random pruning is not a reliable baseline and that principled coverage-aware selection is necessary.
4.4 Discussion↩︎
Gains under aggressive pruning. The improvement from Generic TB-Coverage is largest at 25% and 50% retain, where retaining the right experts is critical. At 75% retain, most experts are preserved and the gap between methods narrows. This pattern is consistent across both models and suggests that coverage-aware selection is most valuable when the pruning budget is tight. Our method addresses selection bias in expert retention, but does not optimize routing adaptation after pruning.
PPL stability. Generic TB-Coverage achieves the lowest perplexity in every setting. The PPL improvements are especially large at 25% retain, where the method reduces Wiki PPL by 9%–24% and C4 PPL by 28%–39% over the next-best baseline (ExpertSparsity). The absolute PPL values at 25% retain are high (e.g., C4 PPL of 423.72 for DeepSeek), which reflects the severity of aggressive pruning rather than an evaluation artifact. Under aggressive pruning, the language model distribution degrades substantially for all methods, and our method degrades least.
Reasoning benchmarks. GSM8K scores are near zero across all methods and retain budgets, and Math500 scores are similarly low. We include these results as stress tests but do not claim that Generic TB-Coverage preserves mathematical reasoning ability. The low scores likely reflect the limited reasoning capacity of the base models rather than a shortcoming of the pruning method.
Per-corpus protection budget. The current experiments use fixed protection budgets (20 experts for Qwen, 24 for DeepSeek) across all retain ratios. These values were selected to provide sufficient coverage without dominating the mask at high retain ratios. Sensitivity analysis of the protection budget is an important direction for future work. We note that the coverage rule’s benefit could be further isolated by comparing against simple multi-corpus aggregation baselines (e.g., mean or max of per-corpus scores); we leave this systematic ablation to future work.
5 Conclusion↩︎
We have presented Generic TB-Coverage, a coverage-aware expert pruning method for sparse MoE language models. By profiling expert utility separately on WikiText2 and C4 and protecting experts through a round-robin coverage rule, the method preserves generic language behaviors that single-criterion methods may discard. Across two MoE models and three retain budgets, Generic TB-Coverage improves downstream average accuracy and language modeling perplexity over random pruning, direct REAP, and reconstruction-based ExpertSparsity, with the largest gains under aggressive pruning.
Limitations. Our method is intentionally simple and static: it does not adapt experts after pruning, learn corpus weights, or optimize a formal diversity objective. The current study uses only two generic corpora and two open-base MoE models of moderate scale (2.7B and 16B parameters); whether the same coverage rule scales to instruction-tuned models, larger expert counts, or pruning-plus-quantization settings remains open. GSM8K and Math500 scores are near zero across all methods, so we cannot assess the method’s effect on reasoning ability. The protection budget \(B\) is fixed across retain ratios and requires manual selection. Finally, we do not measure end-to-end inference latency or memory footprint after pruning.