On the Utility and Factual Reliability of
Pruned Mixture-of-Experts Models in the Biomedical Domain
July 01, 2026
Abstract
Mixture-of-Experts (MoE) models offer inference speedups via selective activation but impose substantial memory requirements because the whole network must remain loaded. Structured expert pruning is a practical approach for reducing deployment costs in resource-constrained settings. However, prior studies primarily evaluate benchmark utility, leaving the effect of pruning on factual reliability underexplored, particularly in high-stakes domains such as biomedicine. In this paper, we investigate how domain-specific expert pruning affects both utility and reliability. We assess four MoE models, six pruning methods, and multiple pruning ratios across generation and classification tasks under in-domain (biomedical) and cross-domain settings. Results reveal that moderate pruning preserves in-domain utility without immediate reliability decline, although hallucination risks increase at extreme pruning ratios. When shifting to the general domain, both utility and reliability degrade rapidly. These findings indicate that safe compression depends heavily on the task and domain. Evaluating pruned MoE models solely on utility is inadequate for high-stakes deployment without reliability assessment.1
1 Introduction↩︎
Modern large language models (LLMs) such as Qwen3.6 [1], GPT-OSS [2], and Nemotron3 [3] follow a Mixture-of-Experts (MoE) architecture. During the forward pass, MoE models activate only a subset of the network by routing each token through specialized subnetworks called experts [4], [5]. They achieve strong performance while offering substantial inference speedups. However, this efficiency does not eliminate the memory overhead during deployment. At inference, all experts must remain loaded in memory, which results in substantially larger memory footprints than those of dense models with comparable active parameter counts.
A popular approach to mitigating this constraint is expert pruning, which eliminates redundant experts based on an informativeness (i.e., saliency) criterion estimated from a small calibration dataset [6]–[8]. Among these approaches, domain-specific expert pruning has been proposed to tailor the compressed model for a particular field [9], such as biomedicine. However, previous studies primarily evaluate pruning methods on downstream benchmark performance, or utility. Consequently, they overlook the reliability of the resulting compressed models. This includes factual consistency, faithfulness, and hallucination prevention [10], [11]. This oversight poses serious risks particularly in high-stakes domains such as biomedicine, where factual errors can lead to critical real-world failures [12], [13].
Furthermore, benchmark utility does not always align with factual reliability. [14] indicate that pruning dense models can result in reduced hallucinations. However, unlike dense model pruning, expert pruning completely removes entire weight matrices, fundamentally altering the model architecture. This forces the model to rely on fallback experts; these alternatives may generate contextually fluent text while silently injecting subtle factual inaccuracies (see Figure 1). Therefore, the relationship between utility and reliability in MoE pruning remains unclear.
In this paper, we investigate how domain-specific expert pruning affects both utility and reliability in high-stakes settings. Focusing on the biomedical domain, we evaluate four MoE models, six pruning methods, and multiple pruning ratios. Our evaluation spans generation and classification tasks, comparing in-domain behavior with complementary general-domain analysis. This framework examines whether pruning degrades performance, if utility and reliability remain coupled, where degradation begins to appear, and how these outcomes vary across task types and domains. Our contributions are as follows:
We provide the first systematic study of how expert pruning affects the factual reliability of MoE models in a high-stakes setting, with a primary focus on the biomedical domain.
We characterize how utility and reliability change across pruning ratios, model families, and pruning strategies, showing that moderate in-domain pruning remains robust while degradation becomes more pronounced at extreme pruning ratios and under domain shift.
We show that the safety of MoE compression is strongly task- and domain-dependent, and that evaluating pruned MoE models using utility alone is insufficient for high-stakes deployment without explicit reliability assessment.
2 Related Work↩︎
2.0.0.1 Reducing Memory Footprint in MoE Models.
Efforts to reduce the memory footprint of MoE models span several approaches, including weight quantization [15]–[17], expert merging [18]–[20], and expert pruning [8], [21]–[23].
Quantization techniques minimize network storage requirements by reducing parameter bit-width. Examples include methods that vary bit-widths based on structural sensitivity [15], [16], optimize calibration [17], or employ extreme 1-bit compression [24], [25]. Quantization reduces numerical precision rather than altering network topology, making it orthogonal to expert merging and pruning [8].
Expert merging mitigates memory constraints by mathematically blending parameter matrices into a unified matrix [18], [26], [27], while expert pruning permanently removes a subset of redundant or low-saliency experts based on calibration data [7], [28], [29]. Expert merging introduces irreducible errors by eliminating the ability of the router to maintain fine-grained, independent control over experts [9], [29], [30]. Conversely, expert pruning preserves the original functional topology and routing independence, demonstrating superior performance at high compression rates [23], [29]. Therefore, we focus exclusively on expert pruning.
2.0.0.2 Expert Pruning.
Expert pruning methods are generally categorized into two paradigms: those that require subsequent fine-tuning and those that operate entirely without training (i.e., post-training or one-shot pruning) [8]. Methods requiring training remove experts based on routing statistics or regularization, subsequently applying gradient-based optimization to recover the resulting performance degradation [6], [7], [21], [31].
In contrast, training-free expert pruning removes redundant experts using one-shot calibration data, bypassing parameter updates. Prominent examples estimate expert saliency through reconstruction loss [8], domain-specific demonstrations [9], expert-selection frequency [22], or advanced functional criteria to preserve routing independence without gradient updates [28]–[30]. We focus on this training-free paradigm because it provides a lightweight solution that facilitates the efficient on-device deployment of massive MoE architectures. This approach also enables isolated examination of pruning effects without confounding variables from fine-tuning.
2.0.0.3 Model Compression and Factual Reliability.
The relationship between model compression and the factual reliability of LLMs has been extensively investigated across dense model pruning, model merging, and weight quantization. [14] find that dense model pruning reduces hallucinations in abstractive summarization by forcing reliance on source documents. Other studies show that it degrades overall trustworthiness [32] and disrupts internal activation features necessary for robust lie detection [33]. Therefore, dense model pruning yields task-dependent outcomes. While studies on merged models remain scarce, [34] demonstrate its utility as a parameter-level conflict-resolution strategy to harmonize helpfulness, honesty, and harmlessness. For weight quantization, previous work [32], [35] demonstrates that moderate bit-width reduction generally preserves trustworthiness and internal calibration. However, quantized models become susceptible to deceptive prompts, even while retaining truthful internal representations [36], and exhibit degraded faithfulness when generating natural language self-explanations [37]. Consequently, weight quantization presents a delicate trade-off between numerical efficiency and semantic precision.
Despite extensive research across these methodologies, the impact of expert pruning on factual reliability remains unexplored. This work provides the first systematic exploration into how expert pruning affects the reliability of MoE models, offering critical insights for their safe deployment.
3 Training-free Domain-specific Expert Pruning↩︎
3.1 MoE Pruning Framework↩︎
Consider an MoE model with \(L\) layers, where each layer \(l \in \{1, \dots, L\}\) contains a set of \(N\) experts, \(\{E_{1}^{l}, \dots, E_{N}^{l}\}\). For a given pruning ratio \(p\), the objective is to retain a subset of \(M\) experts per layer, where \(M=(1-p)N\), and discard the remaining \(N-M\) experts. In this work, we focus on domain-specific expert pruning [9], compressing an MoE model to preserve utility on a domain of interest (here, the biomedical domain).
To achieve this, the pruning process utilizes a calibration set \(\mathcal{C}\) sampled from the domain of interest. For each sample in \(\mathcal{C}\) with a sequence length of \(T\) tokens, a saliency metric (defined in §3.2) evaluates expert importance. After calculating the average importance across \(\mathcal{C}\), experts with the lowest scores are removed. During inference, the router is restricted to selecting from the remaining \(M\) experts.
3.2 Saliency Metrics↩︎
To quantify the importance of each expert, we consider six distinct metrics ranging from random baselines to state-of-the-art methods. Our aim is to investigate how different importance metrics affect domain utility and factual reliability. Specifically, we first use a stochastic baseline (Random) and then advance through a progression of data-driven complexity: from static measures (Frequency), to dynamic activation-based approaches, and finally to recent, context-aware formulations.
3.2.0.1 Random.
The random pruning metric samples \(M\) experts for retention from a uniform distribution. This approach serves as a weak baseline excluding importance estimation and data-driven signals.
3.2.0.2 Frequency.
This metric measures how often the router selects an expert [6]. It assigns equal weight to every activation event, regardless of the magnitude assigned by the routing algorithm. For an expert \(E_{i}^{l}\), the score is the count of activations: \(\sum_{t=1}^{T} \mathbb{I}(g_{i,t}^{l}>0)\), where \(\mathbb{I}(\cdot)\) is the indicator function that returns one for positive gating values and zero otherwise.
3.2.0.3 Gate.
This metric assesses expert importance based on routing activation magnitude [6]. Unlike Frequency, which treats all activations equally, the gating metric identifies experts prioritized by the router. For an expert \(E_{i}^{l}\) in layer \(l\), the score is the sum of gating values across all tokens: \(\sum_{t=1}^{T} g_{i,t}^{l}\), where \(g_{i,t}^{l}\) represents the gating value of the \(i\)-th expert for the \(t\)-th token at layer \(l\).
3.2.0.4 EAN.
Expert Activation Norm [38] assesses importance by accumulating the norms of intermediate activations produced by an expert. The saliency score for expert \(E_{i}^{l}\) is the \(L_{2}\) norms of outputs across active tokens: \(\sum_{t=1}^{T} \mathbb{I}(g_{i,t}^{l}>0)\cdot\|E_{i,t}^{l}(h_{t}^{l})\|_{2}\). Here, \(E_{i,t}^{l}(h_{t}^{l})\) denotes the output vector of the \(i\)-th expert for the hidden state \(h_{t}^{l}\) at layer \(l\). This method favors experts that produce high-magnitude transformations.
3.2.0.5 EASY-EP.
Expert Assessment with Simple Yet-effective scoring for Expert Pruning [9] optimizes importance estimation by coupling the output magnitude of an expert with the token-level contribution to representation shift. This metric posits an expert is essential only when the output magnitude is large and the transformation induces a considerable reorientation of the hidden state. The saliency score is the product of output-aware importance \(c_{i,t}^{l}\) and expert-level token contribution \(s_{t}^{l}\), aggregated as \(\sum_{t=1}^{T} c_{i,t}^{l}\cdot s_{t}^{l}\). The output-aware importance \(c_{i,t}^{l}\) multiplies the routing gate value by the \(L_{2}\) norm of the expert output, formulated as \(c_{i,t}^{l}=g_{i,t}^{l}\|E_{i,t}^{l}(h_{t}^{l})\|_{2}\). The token contribution \(s_{t}^{l}\) is defined as \(1-\text{Sim}(h_{t}^{l},\tilde{h}_{t}^{l})\), where \(\text{Sim}\) denotes cosine similarity between hidden representations preceding (\(h_{t}^{l}\)) and following (\(\tilde{h}_{t}^{l}\)) the expert module.
3.2.0.6 REAP.
Router-weighted Expert Activation Pruning [29] extends EAN by integrating gating values and normalizing by activation frequency. This metric isolates experts with the most substantial per-token contribution via the formulation: \(\frac{\sum_{t=1}^{T} g_{i,t}^{l} \|E_{i,t}^{l}(h_{t}^{l})\|_{2}}{\sum_{t=1}^{T} \mathbb{I}(g_{i,t}^{l} > 0)}.\) This approach ensures that the importance score reflects the mean contribution per activation, identifying experts that maintain influence regardless of selection frequency.
4 Experimental Setup↩︎
To investigate the empirical relationship between expert pruning and factual consistency, the evaluation framework comprises two distinct dimensions: utility, reflecting downstream task performance, and reliability, quantifying hallucination frequency. This joint analysis reveals whether configurations that maintain in-domain utility concurrently preserve reliability, or if reliability degrades more rapidly than utility as the pruning ratio increases.2
4.1 Models↩︎
We use four instruction-tuned MoE models: GPT-OSS 20B [2], Qwen3 30B Instruct 2507 [39], Nemotron 3 Nano 30B [3], and Qwen3.6 35B [1]. This selection offers varying expert granularity and architectures such as the hybrid Mamba-Transformer Nemotron3.
4.2 MoE Pruning↩︎
We utilize MedINST [40] as the calibration set \(\mathcal{C}\) for expert selection. MedINST is a comprehensive meta-dataset of biomedical instructions with diverse tasks, providing an ideal foundation to capture representative domain-specific activations. The calibration set contains 128 randomly sampled demonstrations from the training subset of MedINST, following the established practice in the pruning literature [41]. To mitigate sampling bias, we run three independent calibration experiments for each configuration. Furthermore, to systematically evaluate the impact of structural compression, we assess each pruning strategy across pruning ratios in 12.5% increments.
4.3 Utility↩︎
4.3.0.1 Generation Tasks.
We employ a MedINST evaluation subset requiring full-text generation.3 The selected categories are summarization (SUM), machine translation (MT), question answering (QA), named entity recognition (NER), named entity disambiguation (NED), relation extraction (RE), coreference resolution (COREF), and event extraction (EE). We compute zero-shot ROUGE-L [42] for SUM, chrF++ [43] for MT, and F1 for all other tasks.
4.3.0.2 Classification Tasks.
To complement the generative evaluation, we follow [44] to assess discriminative utility using the MultiMedQA benchmark [45]. The benchmark comprises several multiple-choice QA tasks: PubMedQA [46], MedQA [47], and relevant subsets from MMLU (anatomy, clinical knowledge, college medicine, medical genetics, professional medicine, and college biology) [48]. We measure zero-shot accuracy for all tasks.
4.4 Reliability↩︎
4.4.0.1 Generation Tasks.
We measure semantic consistency using the Multi-XScience [49] and RCT [50] benchmarks. Multi-XScience, a part of MedINST, involves generating a related work section based on an abstract and reference articles. The RCT dataset contains randomized controlled trial reports and serves as a high-stakes test for medical summary accuracy.
We use an LLM-as-a-Judge framework [51] comprising three frontier models (gpt-5.4-mini, Claude Haiku 4.5, and Gemini 3.1 Flash-Lite) to minimize inter-model variability. The evaluation includes two methods: absolute judgment,
which classifies outputs as either faithful or hallucinated, and relative judgment [14], [52], which compares summaries from source and pruned models across four criteria:
Hallucinations (\(\downarrow\)): Frequency of unsupported content by the source document.
Omission (\(\downarrow\)): Exclusion of critical source information.
Repetition (\(\downarrow\)): Presence of redundant text.
Alignment (\(\uparrow\)): Degree of semantic correspondence with the source document.
For relative judgment, we report the preference rate, calculated as the proportion of instances favoring the pruned model over the source model, where a rate above 0.5 denotes preference for the pruned model. We also compute standard metrics such as ROUGE-L and BERTScore [53] to contrast lexical and semantic overlap with these judgments, following [14].
4.4.0.2 Classification Tasks.
For discriminative reliability, we use three reasoning tasks from the Medical Domain Hallucination Test [11]: the False Confidence Test (FCT), the Fake Questions Test (Fake), and the None of the Above Test (NOTA). The FCT provides a randomly suggested answer alongside the question to assess whether the model exhibits unwarranted certainty. The Fake task presents nonsensical medical questions to evaluate whether the model can recognize invalid queries. Finally, the NOTA task replaces the correct option with “None of the above” to test whether the model can reject incorrect information. Collectively, these tests determine whether the model resists hallucination when a false answer is proposed, a query is fundamentally flawed, or a correct solution is absent.
4.5 General-domain Analysis↩︎
To complement our domain-specific evaluation, we monitor the impact of expert pruning on general-domain benchmarks. This analysis determines whether the relationship between utility and reliability in the biomedical domain persists across a broader context.
4.5.0.1 Utility.
We use four benchmarks: IFEval [54] zero-shot accuracy for instruction-following, GSM8K [55] five-shot exact match for math reasoning, HumanEval [56] zero-shot pass@1 for coding, and MMLU three-shot accuracy for general reasoning.
4.5.0.2 Reliability.
We use Multi-News+ [57] to measure the ability of the model to summarize multiple documents for a given topic. We apply the same LLM-as-a-Judge protocol from §4.4.
5 Results↩︎
5.1 Biomedical Domain Utility↩︎
Figure 2 shows the relative performance retention across various compression levels.
5.1.0.1 Pruning Method.
We observe that expert pruning strategies influence in-domain performance retention. Random pruning causes severe performance degradation across all settings. At a 50% pruning ratio, a standard evaluation threshold in prior MoE pruning work [9], [29], it leads to relative utility drops ranging from 18.8% for Qwen3.6 to 75.1% for GPT-OSS on generation tasks.
In contrast, non-random strategies retain performance comparable to unpruned baselines. Most of these strategies yield similar results, with a maximum variance of 5.2% in generation tasks and 17.2% in classification tasks. However, EAN often lags behind other methods in classification tasks, underperforming the best data-driven metric (EASY-EP) by an average of 23.7% across models, with the gap reaching 36.0% in GPT-OSS.
This observation aligns with recent findings [9], [29]. Relying solely on raw activation norms forces EAN to overemphasize output scale instead of true token-level utility. Because EAN does not consider the gating confidence of the router (\(g_{i,t}^{l}\) in REAP) or the representational shift (\(s_{t}^{l}\) in EASY-EP), it retains experts that produce large but contextually unhelpful transformations. This generates suboptimal utility. Consequently, EAN experiences an earlier performance drop compared to other approaches at a 62.5% pruning ratio. Importantly, this earlier utility decline corresponds to an earlier onset of hallucination risks across models, as discussed in §5.2.
5.1.0.2 Model Family.
Pruning behavior also varies across expert granularities and model architectures. Models with large expert pools, like Qwen3 and Qwen3.6, tolerate a 50% pruning ratio well. For Qwen3.6, all data-driven approaches achieve nearly 100% performance retention on generation tasks, and EASY-EP even reaches 106.8% on classification tasks. In contrast, models with fewer experts, like GPT-OSS, suffer greater degradation because each expert holds more network capacity. This vulnerability is especially evident on classification tasks, where no configuration exceeds 91.1%.
Beyond expert count, architectural complexity introduces another vulnerability. Nemotron3 models show a consistent performance decline, underperforming the baseline by at least 6.3% (REAP). We hypothesize that expert pruning disrupts the balance within this hybrid architecture, which combines Transformer attention, Mamba state-space layers, and MoE feed-forward networks. This instability is critical in classification tasks, which rely on precise internal state propagation to form accurate decision boundaries.
5.1.0.3 Pruning Ratio.
For generation tasks, performance remains robust up to a 50% pruning ratio. Context-aware methods like EASY-EP and REAP often match or exceed the unpruned baseline at ratios up to 75%. For instance, at a 75% pruning ratio, Qwen3.6 under REAP retains 96.7% of its baseline utility (a negligible 3.3% drop). In contrast, classification tasks show a gradual performance decline before dropping severely.
5.2 Biomedical Domain Reliability↩︎
5.2.0.1 Pruning Method.
Most pruning methods preserve baseline reliability metrics up to moderate compression levels (a 50% pruning ratio) in-domain. However, we observe that Random and EAN pruning degrade earlier; Random pruning causes immediate degradation even under moderate compression. On FCT classification, accuracy approaches near zero across all pruning ratios (Figure 4). On generation tasks (Figure 3), Random pruning causes severe omissions that standard summarization metrics often mask. For instance, on RCT at a 37.5% pruning ratio, Qwen3.6 under Random pruning maintains a ROUGE-L of .135 and BERTScore of .853 (vs. .134 and .853 for the baseline). Yet, its relative alignment drops to .329 due to high omissions (.790). EAN also exhibits early reliability degradation under moderate pruning compared to other data-driven approaches. For example, on FCT, Qwen3 accuracy drops to 20.7% under EAN at a 50% ratio, compared to 47.2% under EASY-EP.
Coherence vs. Completeness Random pruning can result in high omission and repetition rates even under moderate compression. Standard summarization metrics fail to identify cases where summaries remain structurally fluent but omit key information, creating a false impression of stability.
5.2.0.2 Model Family.
Echoing the utility findings (§5.1), models with large expert pools, such as Qwen3 and Qwen3.6, exhibit remarkable stability up to moderate pruning ratios (50%). This robustness particularly holds under context-aware pruning methods (EASY-EP, REAP), where both generative and discriminative benchmarks remain stable and match baseline performance.
In contrast, models with fewer experts show earlier degradation. GPT-OSS exhibits reliability degradation at a 50% pruning ratio. Under EASY-EP at this ratio, GPT-OSS suffers from substantial factual consistency decay on both RCT and Multi-XScience, despite minimal decline in both task-specific and overall utility metrics. For instance, on Multi-XScience, the model suffers from high repetitions (.815), alongside an increase in the absolute hallucination rate to 17.3% (compared to 10.5% at the 37.5% ratio). However, task-specific summarization metrics remain stable. ROUGE-L and BERTScore are .102 and .817 on Multi-XScience (vs. .100 and .816 for the baseline). Likewise, overall generation utility retains 96.9% of baseline performance (a minor 3.1% drop), demonstrating that factual reliability decay can occur even with minimal impact on standard task-specific and overall utility measurements.
MoE Capacity Bottleneck Under moderate pruning ratios (\(\approx 50\%\)), models with smaller expert pools can exhibit factual reliability degradation earlier than models with larger expert pools. This decline occurs before standard utility metrics begin to degrade.
Nemotron3, while stable under low pruning ratios of 25% or lower, undergoes gradual reliability degradation at the moderate stage of 50%. On RCT under EASY-EP, it exhibits increased information omissions (.598). On Multi-XScience, it experiences a decline in relative alignment (.364) accompanied by elevated relative hallucinations (.625) and repetitions (.624). Yet, summarization metrics on Multi-XScience remain largely unaffected: ROUGE-L is .137 and BERTScore is .838, compared to the baseline scores of .148 and .845. This stability hides the factual consistency decay as the absolute hallucination rate rises to 44.7% from 20.1% for the source model. However, overall generation utility successfully detects this degradation, underperforming the baseline by 7.9% at 50% pruning. This suggests that physical model constraints (e.g., the hybrid MoE-Mamba-Attention topology of Nemotron3) drive a coupled, parallel decay in overall utility and reliability, even when task-specific metrics remain stable.
5.2.0.3 Pruning Ratio.
Extreme pruning (62.5% or higher) increases reliability risks across all models, mirroring utility drops. However, the threshold for reliability preservation depends on the task. For instance, tasks like Fake, NOTA, and RCT often maintain baseline reliability even at 75% compression. Conversely, tasks like FCT and Multi-XScience exhibit earlier, gradual degradation. We attribute this divergence to different reasoning demands. Simple classification (Fake and NOTA) and single-source extraction (RCT) may rely on broad semantic matching shared across many experts. In contrast, precise fact retrieval (FCT) and multi-source synthesis (Multi-XScience) depend on specific experts, making them vulnerable to pruning.
Crucially, extreme compression decouples utility and reliability on Multi-XScience for GPT-OSS, Qwen3, and Qwen3.6. For instance, under EASY-EP at 75% pruning, the relative hallucination rate for Qwen3.6 decreases to .204 from .507 at 50% pruning. Yet, the corresponding absolute rate increases from 11.9% to 17.9%. This apparent relative improvement is due to an artifact of summary collapse. A summary length audit reveals that at 75% pruning, average lengths drop from baselines for Qwen3.6 (204.9 to 97.4 words), Qwen3 (367.8 to 262.7), and GPT-OSS (382.9 to 279.8). Consequently, relative omission rates surge above .630 for these models. The shorter the summary, the lower the absolute number of errors it will contain, lowering the probability of incurring a penalty during relative hallucination evaluation. However, the absolute hallucination rate rises because the actual summary consists largely of ungrounded fabrications. Standard summarization metrics mask this collapse; ROUGE-L and BERTScore remain largely stable (e.g., Qwen3.6 records .122 and .833, compared to .152 and .846 at 50% pruning).
Summary Collapse Extreme pruning artificially lowers relative hallucination rates while standard summarization metrics remain stable. This stems from summary collapse; degenerate outputs omit critical information and make fewer assertions, rendering them less faithful to the source material.
5.3 General-domain Analysis↩︎
5.3.0.1 Utility.
Figure 5 presents the results on general-domain benchmarks. Across model families, expert pruning causes an immediate performance decline in the general domain, with overall utility dropping steadily as the pruning ratio increases. This gradual decay contrasts with in-domain utility, where performance remains highly robust up to 50%–75% pruning before collapsing suddenly. GPT-OSS is the least robust and shows an immediate average decline of 32.4% even at 12.5% pruning probably due to its small expert pool.
When comparing pruning methods, EAN occasionally beats others up to a 50% ratio, except for GPT-OSS. Because EAN selects experts by magnitude without domain-specific weighting, it preserves general capabilities at the expense of in-domain alignment. This trade-off becomes evident when examining expert selection overlap. At 50% pruning, the overlap between EAN and other data-driven methods is low, peaking at .653 for Qwen3.6 (with the exception of Nemotron3 at .714). In contrast, pairwise overlap among the remaining methods is higher with a minimum of .753. Ultimately, this divergence occurs because domain-specific calibration inherently removes general-purpose experts that rarely activate during biomedical tasks. Our multi-domain calibration analysis in §6 further corroborates this finding.
5.3.0.2 Reliability.
Hallucination rates on Multi-News+ increase monotonically with pruning ratio (Figure 6). For non-random methods, Spearman correlation coefficients for absolute and relative hallucination rates span \(\rho = 0.88\) to \(0.93\) and \(\rho = 0.73\) to \(0.87\), respectively. Crucially, this degradation occurs much earlier than in the in-domain setting. Unlike in-domain reliability, which remains largely stable through 50%–62.5% pruning (and up to 75% for Qwen3 and Qwen3.6) as observed in Figure 3, Multi-News+ hallucination rates climb steadily. For example, GPT-OSS reaches an 11.1% mean absolute hallucination rate at just 37.5% pruning, diverging from a 1.3% baseline. Only Qwen3.6 remains relatively robust until 62.5% pruning, where it reaches 12.2% (baseline 2.7%).
By contrast, summarization metrics remain stable. Both ROUGE-L and BERTScore stay within 2% of the unpruned baseline for all models up to 62.5% pruning (Figure 7 in the Appendix). Because ROUGE-L measures the longest common subsequence and ignores repetition, and BERTScore rewards topically plausible yet unfaithful text, both metrics obscure the underlying degradation of reliability. This decoupling constitutes another form of summary collapse driven by domain shift. At 62.5% pruning, models tend to produce repetitive text with increasing relative repetition rates (e.g., 77.1% for Qwen3 under EASY-EP). This triggers a collapse in lexical diversity, measured via Type-Token Ratio (TTR). For instance, from an unpruned baseline of .765, the TTR of Qwen3 drops to .731 at 62.5% pruning and falls sharply to .638 at 75%. Except for Qwen3.6, this repetition inflates mean summary length by up to 149 words without improving faithfulness.
6 Analysis↩︎
6.0.0.1 Correlation between LLM-as-a-Judge and Human Evaluation.
To validate the reliability of our evaluation framework, we conduct a manual annotation study on 30 samples from RCT and Multi-XScience. Three human evaluators annotate each sample for comparison against the judgments of our three LLM judges. We stratify the reliability agreement (Cohen’s \(\kappa\)) by pruning ratio windows: Low (\(p \le 25.0\%\)), Moderate (\(37.5\% \le p \le 50.0\%\)), and Extreme (\(p \ge 62.5\%\)), as shown in Table 1.
| Pair | Low | Moderate | Extreme | |
|---|---|---|---|---|
| Sample Size (\(N\)) | 8 | 9 | 13 | |
| Absolute | Human–Human | 0.467 | 1.000 | 0.898 |
| Human–LLM | 0.006 | 0.738 | 0.644 | |
| LLM–LLM | 0.333 | 0.738 | 0.783 | |
| Relative | Human–Human | 0.366 | 0.260 | 0.733 |
| Human–LLM | 0.238 | -0.012 | 0.605 | |
| LLM–LLM | 0.508 | 0.370 | 0.676 | |
3pt 1.3pt 1.3pt
Under extreme pruning ratios, we find substantial agreement across all groups (e.g., human-LLM absolute agreement reaches 82.1% with \(\kappa = 0.644\)). Severe quality degradation (e.g., obvious hallucinations, omissions, and repetitions) drives this straightforward evaluation for both humans and LLMs. Under low and moderate pruning, however, the differences are subtle. Relative preference exhibits substantial variance across both windows (\(\kappa = 0.238\) and \(\kappa = -0.012\) for human-LLM). This variance reflects the inherent difficulty of choosing between outputs of similar quality before severe degradation occurs. For absolute judgments, agreement is high during moderate compression (\(\kappa = 0.738\) for human-LLM) but diminishes under low compression (\(\kappa = 0.006\)), suggesting that LLM judges diverge from humans when evaluating near-baseline outputs. Overall, the strong alignment between human and LLM judges in detecting severe degradations justifies our automated evaluation setup for generation reliability assessment.
| Configuration | Utility | Biomedical Reliability | General Reliability | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-10 (lr)11-15 | MedINST | MMQA | General | RCT | MXSci. | Hall. | Omis. | Repe. | Align. | MN+ | Hall. | Omis. | Repe. | Align. |
| MedINST (50%) | .554.004 | .838.008 | .689.010 | .015.009 | .121.079 | .476.029 | .402.073 | .516.067 | .564.036 | .031.036 | .367.076 | .480.106 | .547.133 | .647.012 |
| + General (50%) | .548.013 | .833.004 | .915.064 | .019.017 | .088.064 | .437.033 | .470.044 | .517.053 | .550.029 | .060.045 | .510.082 | .537.065 | .593.094 | .447.079 |
| MedINST (75%) | .476.037 | .665.137 | .387.088 | .024.019 | .179.044 | .204.029 | .735.088 | .405.161 | .417.079 | .211.073 | .540.020 | .727.081 | .680.140 | .287.031 |
| + General (75%) | .424.128 | .563.096 | .597.287 | .061.026 | .358.025 | .372.050 | .646.040 | .653.112 | .414.029 | .323.167 | .755.100 | .630.162 | .905.055 | .160.043 |
| + GPTQ (4-bit) | .543.005 | .829.011 | .676.020 | .017.026 | .125.079 | .419.025 | .470.037 | .492.022 | .564.034 | .078.055 | .547.058 | .553.085 | .620.078 | .410.075 |
3pt 1.3pt 1.3pt
6.0.0.2 Multi-domain Calibration.
Given the substantial impact of calibration data domains on utility and reliability, we compare biomedical-only calibration against dual-domain calibration, which additionally uses general-domain calibration data.4 Following [9], we compute a saliency metric (§3.2) for each domain and average them. We analyze these calibration behavior at moderate (50%) and extreme (75%) pruning ratios using Qwen3.6 with EASY-EP as this combination represents the most effective pruning configuration.
As shown in Table 2, dual-domain calibration mitigates general utility degradation, boosting it from .387 to .597 at 75% pruning. Yet, this improvement incurs a trade-off in biomedical utility (e.g., at 75% pruning, MultiMedQA drops from .665 to .563). For reliability, dual-domain calibration performs comparably to the biomedical-only method at 50% pruning. It maintains low absolute hallucination rates, degrading by a maximum of .029 on Multi-News+ compared to the unpruned baseline, and yields similar pairwise preference win-rates. Conversely, under extreme 75% pruning, the biomedical-only calibration is more robust. Under dual calibration, absolute hallucination rates surge across tasks, doubling the rates of MedINST (.358 vs. on Multi-XScience). Pairwise metrics confirm this degradation: the 75% dual-pruned model exhibits higher relative hallucinations and repetitions. This outcome suggests that while multi-domain calibration helps balance general and specialized utility under moderate pruning, specialized in-domain calibration is critical for preserving factual reliability during extreme compression.
6.0.0.3 Quantization.
While quantization is orthogonal to expert pruning (§2), we examine whether utility and reliability trends remain consistent in quantized models. Specifically, we apply 4-bit GPTQ [59] weight quantization to the Qwen3.6 EASY-EP 50% pruned model. As Table 2 indicates, we observe no substantial utility difference between the unquantized and quantized configurations: the quantized model retains a biomedical utility of .543 on generation and .829 on classification tasks (compared to .554 and .838 for the unquantized counterpart). It also preserves .676 of general-domain utility, a minor decrease from the unquantized score of .689.
For reliability, quantization generally preserves consistency on biomedical benchmarks. For instance, it maintains absolute hallucination rates of .017 on RCT and .125 on Multi-XScience, against unquantized rates of .015 and .121. However, in the general domain, quantization increases absolute hallucinations (.078 vs.) and elevates relative errors. Hallucination, omission, and repetition rates rise to .547, .553, and .620 respectively, while relative alignment drops from .647 to .410. These results demonstrate that while expert pruning and post-training weight quantization can combine to achieve additional compression without compounding in-domain performance loss, quantization can increase reliability risks in cross-domain settings.
6.0.0.4 Qualitative Analysis.
| Source Context | Pruned Output | Cl. | Gem. | GPT | |
|---|---|---|---|---|---|
| 1. | 3.8 ng/ml | 3.8 mg/ml | 1.0 | 0.0 | 0.0 |
| 2. | 179 randomized | 70-9 randomized | 1.0 | 1.0 | 0.0 |
| 3. | (abstract) | Given ..., write... | 1.0 | 1.0 | 0.0 |
1.3pt 1.3pt 3pt
To investigate reliability degradation, we analyze RCT generated summaries (Table 3). We identify three primary failure modes: (1) Unit hallucinations, swapping measurement units (e.g., Qwen3 generating mg/ml instead of ng/ml at 25% Gate); (2) Numerical errors, introducing factual mistakes (e.g., GPT-OSS reporting 179 patients as 70-9 at 75% Frequency); and (3) Instruction leakage, repeating the task template under extreme pruning (e.g., Qwen3.6 at 87.5% EAN). Evaluating these outputs reveals the varying sensitivities of absolute LLM judges. While Claude identifies most errors, Gemini and GPT overlook the unit swap in Case 1. This error notably occurs at a low 25% pruning ratio while utility remains intact. This establishes that evaluating pruned models on utility alone is often inadequate for high-stakes deployment; direct reliability assessments are critical to detect factual failures before utility degrades.
7 Conclusion↩︎
This paper presents the first comprehensive study on the impact of expert pruning in MoE models on both benchmark utility and factual reliability within the high-stakes biomedical domain. Our findings highlight that utility is not always indicative of reliability. During domain shift or extreme pruning, models experience summary collapse and an increase in hallucination rates before standard utility metrics register a decline. In high-stakes domains, this trade-off between utility and reliability requires careful consideration through comprehensive in-domain and general evaluation. Finally, while multi-domain calibration and quantization offer benefits, they introduce reliability trade-offs under high compression or cross-domain settings.
Acknowledgment↩︎
We would like to thank Mingzi Cao, Vynska Amalia Permadi, and Samuel Lewis-Lim for their annotation support. We also appreciate initial guidance on hallucination evaluation from Timothee Mickus. AY is supported by the Engineering and Physical Sciences Research Council (EPSRC) [grant number EP/W524360/1] and the Japan Student Services Organization (JASSO) Student Exchange Support Program (Graduate Scholarship for Degree Seeking Students).
8 Implementation Details↩︎
8.0.0.1 Software.
We utilize Hugging Face (HF) datasets [60] for data preprocessing, alongside HF transformers [61], FlashAttention-2 [62], and PyTorch [63] for the pruning framework. We employ lm-evaluation-harness [64] for IFEval, GSM8K, HumanEval, and MMLU evaluation. For MedHALT, we use the official repositories [11].5 For HaluEval, we adopt the dataset from the official HF repository.6 For RCT, we obtain the dataset from the official repository [50].7 Similarly, for Multi-News+, we load the dataset from the official repository [57].8 Due to API cost constraints during LLM-as-a-Judge
evaluation, we randomly subsample 50 documents from the Multi-News+ test set for our evaluation. For MedINST, we execute the official script [40].9 For absolute judgment, we leverage the pre-configured HallucinationMetric of DeepEval [65]. Inference is executed using the vLLM [66] engine for faster inference. For post-training weight quantization analysis, we apply 4-bit GPTQ using the llmcompressor library [67].
8.0.0.2 Hardware.
We use a single NVIDIA A100 80GB GPU with CUDA 12.9 for experiments. Notably, the expert pruning process completes within 775 seconds (using the largest model: Qwen3.6 with the context-aware EASY-EP method) and introduces no substantial computational overhead.
8.0.0.3 Prompt Templates.
We prioritize official prompt templates to maintain standardized evaluation conditions, while using custom templates only as required. We use the default prompt templates for MultiMedQA, IFEval, GSM8K, HumanEval, and MMLU provided by lm-evaluation-harness. For MedINST, MedHALT, and HaluEval, we utilize their official prompt templates. For summarization tasks, including RCT and Multi-News+, we apply the custom prompt templates listed in Appendix 9. For LLM-as-a-Judge and human evaluation, we employ the off-the-shelf DeepEval template for absolute judgment and a custom prompt based on [14] for relative judgment; both are documented in Appendix 9.
8.0.0.4 Hyperparameters.
During evaluation, we restrict the maximum sequence length to 4,096 tokens and set the temperature parameter to 0.0, except for Multi-News+, which uses a sequence length of 8,192 tokens. We disable the thinking mode for Qwen3.6 and Nemotron3 and assign a low reasoning effort setting for GPT-OSS by default.
For quantization, we employ a W4A16 quantization scheme (4-bit weights and 16-bit activations) targeting all linear layers in the model. The language modeling head (lm_head) and the MoE gate/router layers (gate and
router parameters) are excluded from quantization and kept in bfloat16 precision. For calibration, we use the same calibration set as in pruning (i.e., 128 randomly sampled instances from MedINST).
8.0.0.5 Human Evaluation Protocol.
To validate the LLM-as-a-Judge evaluation framework, we conduct a manual annotation study to compare LLM judgments with human annotations. We recruit three PhD student volunteers with experience in natural language processing and computer science. Because the evaluation protocol relies strictly on textual grounding, i.e., verifying whether statements in the generated summary are logically entailed by the provided source documents, rather than requiring external clinical diagnostic knowledge, these annotators are highly qualified to evaluate semantic alignment and omission errors. The human annotators are provided with the same instructions and prompt guidelines as the LLM judges (documented in Appendix 9). Specifically, for absolute judgment, we ask the annotators to classify the summaries generated by the pruned models as either faithful or hallucinated. For relative judgment, they compare the summaries from the unpruned baseline against the pruned models across four dimensions (hallucinations, omissions, repetition, and semantic alignment). Both evaluations are conducted using only the provided source documents.
The human evaluation is conducted on a subset of 30 randomly selected documents from the RCT and Multi-XScience datasets. The sample distribution of these 30 evaluation cases across datasets, generator models, pruning methods, and pruning ratios is structured as follows:
Dataset Balance: 15 samples from RCT and 15 samples from Multi-XScience.
Generator Model Balance:
GPT-OSS: 8 samples (4 RCT, 4 Multi-XScience)
Qwen3: 7 samples (4 RCT, 3 Multi-XScience)
Qwen3.6: 7 samples (3 RCT, 4 Multi-XScience)
Nemotron3: 8 samples (4 RCT, 4 Multi-XScience)
Pruning Method Balance: 5 samples per pruning method, distributed evenly across all six approaches (Random, Frequency, Gate, EAN, EASY-EP, and REAP).
Pruning Ratio Balance:
Ratio 0.125: 4 samples
Ratio 0.250: 4 samples
Ratio 0.375: 5 samples
Ratio 0.500: 4 samples
Ratio 0.625: 4 samples
Ratio 0.750: 5 samples
Ratio 0.875: 4 samples
All pruning ratios within each individual (dataset, generator model, pruning method) group are guaranteed to be distinct for the selected samples to ensure a representative and unbiased distribution across the entire pruning range.
9 Prompt Templates↩︎
RCT prompt template You are a medical evidence summarization assistant. Given a randomized controlled trial abstract, write a concise factual summary in 2-4 sentences. Do not include information that is not supported by the abstract.
Source documents: {source}
Summary:
Multi-News+ prompt template You are a news summarization assistant. Given documents from multiple news sources about the same event, write a concise factual summary in 2-4 sentences. Do not include information that is not supported by the source.
Source documents: {source}
Summary:
Absolute judgment prompt template Given a list of factual alignments and contradictions, which highlights alignment/contradictions between the ‘actual output’ and ‘contexts’, use it to provide a reason for the hallucination score CONCISELY. Note that
the hallucination score ranges from 0 - 1, and the lower the better.
** IMPORTANT: Please make sure to only return in JSON format, with the ‘reason’ key providing the reason. Example JSON: {{ "reason": "The score is <hallucination_score> because <your_reason>." }} **
Factual Alignments: {factual_alignments}
Contradictions: {contradictions}
Hallucination Score: {score}
JSON:
Relative judgment prompt template You are an impartial evaluator. Compare Summary A vs Summary B against the Source Document. Use ONLY the Source Document to judge support. Do NOT use outside knowledge. Be strict about factual support.
Answer these questions:
Q1. Hallucinations: Which summary contains MORE hallucinations (unsupported content)? Q2. Omission: Which summary is missing MORE crucial information from the document? Q3. Repetition: Which summary contains MORE repetitive information? Q4. Alignment:
Which summary is MORE semantically aligned with the source document?
Return ONLY valid JSON with exactly these keys: - q1_hallucinations_more: "A" or "B" - q2_omission_more: "A" or "B" - q3_repetition_more: "A" or "B" - q4_alignment_more: "A" or "B"
Rules: - You MUST choose ‘A’ or ‘B’ (no ties). - Keep outputs to JSON only (no markdown).
Source Document: <<<DOC {document} DOC>>>
Summary A: <<<A {summary_a} A>>>
Summary B: <<<B {summary_b} B>>>
10 Supplementary Results↩︎
| Dataset | Overall Fleiss’ \(\kappa\) |
|---|---|
| Multi-News | .577 |
| Multi-XScience | .513 |
| RCT | .535 |
| Dataset | Question | Overall Fleiss’ \(\kappa\) |
|---|---|---|
| Multi-News | Q1: Hallucination | .503 |
| Q2: Omission | .447 | |
| Q3: Repetition | .285 | |
| Q4: Alignment | .519 | |
| Multi-XScience | Q1: Hallucination | .484 |
| Q2: Omission | .336 | |
| Q3: Repetition | .456 | |
| Q4: Alignment | .555 | |
| RCT | Q1: Hallucination | .483 |
| Q2: Omission | .492 | |
| Q3: Repetition | .370 | |
| Q4: Alignment | .470 |
| Approach | IFEval | GSM8K | HumanEval | MMLU | |
|---|---|---|---|---|---|
| Source | .750 | .056 | .817 | .843 | |
| Random | .221.008 | .009.015 | .163.135 | .265.130 | |
| Frequency | .267.001 | .001.001 | .018.009 | .656.006 | |
| Gate | .251.010 | .007.003 | .012.000 | .660.003 | |
| EAN | .252.003 | .001.002 | .073.006 | .534.015 | |
| EASY-EP | .261.007 | .003.003 | .014.004 | .655.006 | |
| REAP | .260.004 | .005.002 | .012.000 | .656.002 | |
| Source | .813 | .879 | .848 | .723 | |
| Random | .555.034 | .378.040 | .053.039 | .497.021 | |
| Frequency | .668.019 | .148.011 | .006.000 | .545.001 | |
| Gate | .683.005 | .159.007 | .010.004 | .552.008 | |
| EAN | .653.015 | .372.031 | .254.029 | .526.003 | |
| EASY-EP | .702.015 | .472.022 | .012.006 | .565.007 | |
| REAP | .689.008 | .441.028 | .006.000 | .565.005 | |
| Source | .806 | .835 | .939 | .801 | |
| Random | .424.100 | .320.198 | .021.004 | .437.126 | |
| Frequency | .578.015 | .547.034 | .000.000 | .569.013 | |
| Gate | .663.016 | .573.066 | .000.000 | .585.005 | |
| EAN | .717.008 | .836.011 | .882.015 | .559.002 | |
| EASY-EP | .779.012 | .692.008 | .014.004 | .668.005 | |
| REAP | .779.003 | .695.006 | .008.004 | .656.003 | |
| Source | .824 | .880 | .970 | .814 | |
| Random | .581.084 | .178.200 | .217.057 | .530.104 | |
| Frequency | .770.004 | .106.047 | .018.006 | .644.020 | |
| Gate | .773.011 | .128.044 | .008.004 | .642.006 | |
| EAN | .776.021 | .862.010 | .780.012 | .680.013 | |
| EASY-EP | .746.015 | .778.027 | .083.009 | .719.008 | |
| REAP | .774.004 | .766.008 | .039.009 | .663.003 |
| Approach | SUM | MT | QA | NER | NED | RE | COREF | EE | |
|---|---|---|---|---|---|---|---|---|---|
| Source | .100 | .585 | .768 | .538 | .331 | .342 | .419 | .290 | |
| Random | .066.011 | .078.042 | .216.255 | .124.025 | .035.005 | .089.059 | .008.006 | .088.039 | |
| Frequency | .097.000 | .536.004 | .715.008 | .505.007 | .309.009 | .345.024 | .521.060 | .274.017 | |
| Gate | .096.002 | .531.011 | .738.005 | .507.005 | .313.001 | .358.007 | .492.099 | .283.006 | |
| EAN | .119.001 | .285.022 | .636.003 | .331.007 | .162.000 | .351.010 | .368.044 | .138.005 | |
| EASY-EP | .102.003 | .498.028 | .722.009 | .517.006 | .317.009 | .316.030 | .447.045 | .280.016 | |
| REAP | .103.004 | .485.018 | .726.006 | .516.005 | .317.019 | .317.033 | .459.050 | .283.015 | |
| Source | .148 | .515 | .735 | .445 | .334 | .384 | .695 | .244 | |
| Random | .130.007 | .319.013 | .580.068 | .355.009 | .183.007 | .370.016 | .648.020 | .158.031 | |
| Frequency | .132.001 | .305.037 | .692.005 | .431.003 | .276.004 | .354.006 | .695.045 | .199.004 | |
| Gate | .136.000 | .375.015 | .686.007 | .434.001 | .286.004 | .366.008 | .698.077 | .228.016 | |
| EAN | .135.002 | .441.017 | .676.008 | .422.003 | .266.004 | .353.012 | .714.050 | .235.010 | |
| EASY-EP | .137.003 | .365.064 | .682.006 | .439.002 | .298.002 | .356.007 | .757.012 | .215.002 | |
| REAP | .139.001 | .435.052 | .684.015 | .438.002 | .296.003 | .382.015 | .751.015 | .236.003 | |
| Source | .115 | .615 | .663 | .566 | .407 | .396 | .802 | .368 | |
| Random | .114.019 | .371.054 | .603.035 | .366.033 | .154.031 | .366.030 | .729.049 | .202.048 | |
| Frequency | .118.002 | .450.024 | .597.018 | .529.013 | .347.010 | .393.025 | .747.038 | .336.018 | |
| Gate | .119.001 | .484.003 | .665.025 | .545.001 | .355.012 | .394.014 | .825.003 | .360.003 | |
| EAN | .132.002 | .442.019 | .663.007 | .555.002 | .271.004 | .370.006 | .793.016 | .355.006 | |
| EASY-EP | .114.002 | .499.003 | .752.006 | .575.003 | .410.001 | .382.020 | .823.014 | .368.003 | |
| REAP | .116.002 | .488.004 | .753.002 | .571.001 | .402.006 | .377.008 | .794.017 | .374.005 | |
| Source | .154 | .624 | .566 | .714 | .427 | .479 | .899 | .304 | |
| Random | .133.012 | .540.009 | .614.066 | .519.016 | .204.038 | .430.021 | .764.099 | .218.065 | |
| Frequency | .157.001 | .617.003 | .760.015 | .702.006 | .407.012 | .475.018 | .926.018 | .323.007 | |
| Gate | .154.002 | .617.002 | .653.048 | .704.003 | .423.004 | .494.007 | .928.018 | .322.004 | |
| EAN | .158.003 | .605.009 | .795.005 | .690.007 | .366.009 | .449.002 | .907.004 | .286.019 | |
| EASY-EP | .152.003 | .612.007 | .797.011 | .700.018 | .427.018 | .480.017 | .928.014 | .333.010 | |
| REAP | .154.003 | .617.004 | .595.032 | .710.002 | .437.002 | .499.009 | .887.011 | .341.009 |
| Approach | PMQA | MQA4 | MMCQA | Anat | CK | CM | MG | PM | CB | |
|---|---|---|---|---|---|---|---|---|---|---|
| Source | .688 | .791 | .653 | .793 | .834 | .850 | .940 | .915 | .944 | |
| Random | .347.260 | .081.042 | .140.089 | .232.137 | .260.168 | .229.110 | .233.150 | .114.042 | .319.211 | |
| Frequency | .667.011 | .719.014 | .568.004 | .731.033 | .792.004 | .721.027 | .807.015 | .871.007 | .829.041 | |
| Gate | .682.024 | .732.012 | .569.003 | .689.020 | .794.029 | .707.018 | .827.045 | .881.024 | .808.008 | |
| EAN | .713.004 | .312.011 | .337.002 | .363.013 | .499.008 | .459.022 | .417.038 | .391.009 | .521.032 | |
| EASY-EP | .722.016 | .721.005 | .553.004 | .694.031 | .801.014 | .730.003 | .813.023 | .864.025 | .838.016 | |
| REAP | .717.018 | .705.006 | .552.004 | .719.026 | .777.021 | .730.007 | .813.021 | .864.004 | .833.014 | |
| Source | .440 | .636 | .545 | .704 | .770 | .717 | .840 | .831 | .875 | |
| Random | .479.021 | .364.066 | .348.042 | .410.009 | .513.047 | .499.023 | .507.021 | .506.024 | .507.025 | |
| Frequency | .381.041 | .510.010 | .436.008 | .580.026 | .657.023 | .640.009 | .767.023 | .713.007 | .653.021 | |
| Gate | .398.026 | .502.021 | .449.005 | .595.011 | .669.034 | .651.033 | .737.021 | .729.006 | .671.011 | |
| EAN | .609.021 | .434.006 | .404.012 | .479.041 | .575.014 | .559.009 | .723.025 | .614.017 | .667.025 | |
| EASY-EP | .397.012 | .532.007 | .472.006 | .607.022 | .709.016 | .682.015 | .747.031 | .745.014 | .713.004 | |
| REAP | .427.031 | .520.004 | .471.003 | .605.004 | .702.007 | .678.009 | .740.026 | .733.011 | .706.034 | |
| Source | .700 | .761 | .670 | .778 | .879 | .821 | .930 | .890 | .924 | |
| Random | .599.045 | .359.078 | .360.032 | .378.181 | .454.132 | .432.139 | .420.075 | .415.160 | .463.149 | |
| Frequency | .661.029 | .645.012 | .553.008 | .709.024 | .779.019 | .701.009 | .817.015 | .786.018 | .889.007 | |
| Gate | .671.013 | .667.017 | .565.014 | .728.017 | .784.023 | .734.010 | .817.015 | .821.024 | .870.011 | |
| EAN | .633.008 | .432.005 | .398.004 | .393.013 | .591.023 | .563.023 | .593.015 | .569.017 | .632.000 | |
| EASY-EP | .695.010 | .744.007 | .646.002 | .773.030 | .835.004 | .782.007 | .870.010 | .880.002 | .917.000 | |
| REAP | .699.010 | .737.003 | .640.002 | .770.020 | .830.008 | .786.006 | .873.015 | .873.004 | .921.004 | |
| Source | .794 | .620 | .700 | .874 | .906 | .838 | .940 | .614 | .931 | |
| Random | .759.013 | .208.152 | .317.124 | .469.062 | .625.100 | .528.115 | .580.151 | .315.226 | .604.097 | |
| Frequency | .789.005 | .681.097 | .657.005 | .812.026 | .848.006 | .800.029 | .927.021 | .787.131 | .907.004 | |
| Gate | .794.002 | .739.019 | .654.007 | .788.019 | .853.008 | .802.020 | .897.015 | .517.142 | .907.011 | |
| EAN | .796.007 | .506.196 | .319.101 | .669.015 | .745.038 | .748.019 | .783.012 | .751.073 | .859.004 | |
| EASY-EP | .789.001 | .822.012 | .675.002 | .835.023 | .863.011 | .813.039 | .903.021 | .926.006 | .914.004 | |
| REAP | .797.002 | .550.057 | .663.005 | .835.015 | .853.010 | .778.038 | .910.020 | .271.061 | .921.008 |
References↩︎
Our code is available at https://github.com/gucci-j/moe-pruning-reliability.↩︎
This subset does not overlap with the calibration set, ensuring a fair comparison.↩︎
We use 128 Dolci-Instruct-SFT-No-Tools [58] samples as general-domain calibration data.↩︎
https://github.com/medhalt/medhalt/tree/bd4408a16aff36626e934aa5b012edd9fa7b6194↩︎
https://github.com/bwallace/RCT-summarization-data/tree/de10c2712873efa1733859f2d7113af60427d7b2↩︎
https://github.com/c-juhwan/multi_news_plus/tree/e347c8eedb78a09b4971bc0011a8865e11dfafdb↩︎
https://github.com/aialt/MedINST/blob/dc13b2be29cdaae01cddceb6e4134e7dc6df1134/evaluation.py↩︎