1 Introduction↩︎

Mixture-of-Experts (MoE) architectures have demonstrated remarkable scalability in language models by activating only a subset of expert sub-networks per input. However, deploying these models in real-world applications remains challenging due to their large memory footprint: all expert weights must be stored in memory, regardless of which subset is activated during inference. As the number of experts grows, this memory overhead becomes increasingly prohibitive, as evidenced by advances such as Mixtral  [1], DeepSeek-MoE [2], and Qwen-MoE [3], especially for resource-constrained deployments. For instance, Mixtral-8x7B has 47 billion parameters, about 45 billion of which are in the expert modules, requiring at least two A100-80GB GPUs to load in 16-bit.

To reduce this memory cost, previous studies have investigated expert compression methods, including expert dropping and expert merging. Expert dropping methods [4], [5] remove entire experts considered less important based on their output over a calibration dataset. However, it is easy for them to accidentally discard important knowledge, leading to a sharp accuracy drop [6]. In contrast, expert merging methods attempt to combine similar experts rather than removing them entirely, typically via expert clustering [6] or using low-rank approximations [7]. Although these methods generally outperform expert dropping, they still experience significant accuracy degradation, with performance drops of over 20% on MMLU [6], [7] as demonstrated in Figure 1.

The significant performance degradation indicates that existing coarse-grained expert dropping and merging strategies are insufficient. By operating at the level of entire experts or weight tensors, these methods fail to distinguish between shared parameters and expert-specific weights, leading to over-merging that hurts expert specialization and model quality. Moving beyond coarse-grained operations, fine-grained merging decisions have the potential to better preserve accuracy, but they introduce substantial challenges. First, making fine-grained merging decision requires navigating a large combinatorial space, making naive merging computationally intractable. Second, fine-grained sparsity often relies on explicit storage of masks or indices, which introduces additional metadata overhead that can significantly reduce or even negate the intended memory savings. In addition, many existing expert compression methods rely on task-specific calibration [4], [5] and require extensive offline compression time, limiting their practicality and scalability (Section 4.4).

In this paper, we propose PuzzleMoE, the first MoE merging method to enable element-wise merging while achieving both high accuracy and inference speed. First, we introduce a novel fine-grained expert merging algorithm that selectively merges experts at element-wise level. Specifically, we construct two masks: (1) an element-wise similarity mask that identifies weights with high magnitude similarity between expert pairs; and (2) an activation-weight saliency mask that identifies weights critical to each expert’s unique specialization. This dual-mask design allows PuzzleMoE to merge only redundant parameters while preserving expert-specific capacity by retaining salient weights.

While element-wise merging better preserves both shared and expert-specific weights, storing element-wise binary masks and sign information for merged expert can introduce substantial metadata overhead, which negates the memory benefits of fine-grained compression at scale. To overcome this challenge, we introduce a bit-packed encoding scheme that reuses underutilized bits in floating-point representations. We observe that the exponent field of expert weights typically occupies a narrow range during inference, leaving multiple bits underutilized. PuzzleMoE leverages these bits to embed binary masks and sign bits directly into the weight tensors, eliminating the need for auxiliary metadata storage. To support this encoding during inference, we further design a custom decode-GEMM kernel that performs on-the-fly decoding with the matrix multiplication operation, enabling fast and memory-efficient execution of PuzzleMoE. Our contributions are summarized as follows:

• We propose PuzzleMoE, the first MoE merging method to enable fine-grained, element-wise expert merging that achieves both high accuracy and fast inference speed. Unlike prior expert merging approaches operating at coarse granularity, PuzzleMoE selectively merges redundant parameters and preserves expert-specific weights through a dual-mask design based on weight similarity and activation-weight saliency.

• We design a novel bit-efficient encoding scheme that embeds masks and signs directly into floating-point weights, eliminating auxiliary metadata storage and enabling efficient MoE inference via a lightweight custom decode-GEMM kernel.

• We evaluate PuzzleMoE across four MoE models and nine benchmarks, demonstrating consistent improvements over prior methods. In particular, PuzzleMoE achieves up to 16.7% higher accuracy than prior methods under 50% compression ratio, and up to 1.80\(\times\) higher end-to-end throughput on Mixtral-8x7B.

2 Related Works↩︎

Quantization and pruning are two widely adopted orthogonal techniques for model compression and have recently attracted increasing attention in the context of MoE LLMs [6]–[11]. Quantization methods [12], [13] exploit per-weight redundancy to reduce memory footprint by lowering weight precision, achieving substantial compression with minimal accuracy degradation. In contrast, pruning-based approaches including expert dropping and merging remain less mature. Existing methods often suffer from severe performance degradation at moderate sparsity levels; for instance, pruning \(50\%\) of experts can result in over \(18.7\%\) accuracy drop on the MMLU benchmark for Mixtral-8×7B [4], [6]–[8]. These observations indicate that effective pruning of MoE models remains an open challenge.

Expert Dropping. Expert dropping methods reduce MoE model size by removing entire expert modules considered unimportant. For example, NAEE [4] performs an exhaustive search to identify a subset of experts to retain, while STUN [5] accelerates this process by exploiting latent behavioral similarity among experts, reducing the selection complexity to \(O(1)\). Despite their effectiveness, these methods rely heavily on task-specific calibration datasets, as different downstream tasks often require different subsets of experts to be preserved. This limits their generality across tasks. Relatedly, \(\text{MoE-I}^2\) [14] adopts a two-stage strategy combining inter-expert pruning with intra-expert low-rank decomposition. However, it requires finetuning to recover performance.

Expert Merging. To better preserve accuracy, several recent works have proposed merging similar experts rather than dropping them. Methods like HC-SMoE [6] use hierarchical clustering based on expert output similarity to identify and combine experts, while MC-SMoE [15], D2 [8], and Sub-MoE  [7] adopt multi-stage merging pipelines, e.g., first merging experts based on similarities, and then adding low-rank matrices to approximate the residual information. While these methods generally outperform expert dropping in accuracy, they often require complex procedures like SVD. In addition, they rely on coarse-grained expert merging, which risks hurting the distinctions between specialized experts and suffers from severe accuracy drop at large compression ratios. In contrast, PuzzleMoE performs fine-grained, element-wise merging in a training-free and single-pass manner, effectively preserving expert diversity.

3 Method↩︎

In this section, we first elaborate on the motivation for adopting fine-grained, element-wise expert merging. We then introduce PuzzleMoE, which enables efficient fine-grained element-wise expert merging through a carefully designed pairwise dual-mask merging algorithm and a system-level co-design with bit-level packing.

3.1 Motivation: Element-wise Similarity in Experts’ Weights↩︎

Prior studies [4], [6]–[8] have primarily focused on coarse-grained expert pruning or merging strategies. Such approaches inherently overlook potential similarities at a finer granularity within expert parameters. Motivated by this gap, we conduct a fine-grained analysis of weight-level similarity across experts in MoE LLMs and empirically observe the presence of substantial element-wise redundancies.

Following the empirical observations in [16], [17], we approximate the weights of different experts using zero-mean Gaussian distributions, which is validated in Appendix 7.5. These distributions facilitate the characterization of element-wise similarity within a probabilistic context. Based on this, we model a pair of corresponding weight elements \(w_i\) and \(w_j\) from expert weights \(\mathbf{W}_i\) and \(\mathbf{W}_j\) as independent samples drawn from similar underlying distributions. This abstraction enables a tractable quantitative characterization of element-wise similarity under a probabilistic model.

Lemma 3.1 (Informal). Consider two independently sampled expert weight elements \(w_i\) and \(w_j\), where each element is drawn from a zero-mean Gaussian distribution, \(w_i \sim \mathcal{N}(0, \sigma_i^2)\) and \(w_j \sim \mathcal{N}(0, \sigma_j^2)\). For a similarity threshold \(\tau_{\mathrm{sim}} \in (0,1)\), the probability that two corresponding weight elements are similar in a relative sense satisfies \[\begin{align} P\!\left(\frac{\bigl||w_i|-|w_j|\bigr|}{|w_i|+|w_j|} < \tau_{\mathrm{sim}} \right) &= \frac{2}{\pi} \arctan\!\Bigl(\tfrac{1+\tau_{\mathrm{sim}}}{1-\tau_{\mathrm{sim}}}\Bigr) \\ &\quad - \frac{2}{\pi} \arctan\!\Bigl(\tfrac{1-\tau_{\mathrm{sim}}}{1+\tau_{\mathrm{sim}}}\Bigr). \end{align}\]

The formal derivation is provided in Appendix 3.1. We further evaluate the alignment between this theoretical prediction and empirical statistics from real-world MoE LLMs. As illustrated in Figure 2, empirical similarity ratios from both Qwen1.5-MoE and DeepSeek-MoE closely align with the theoretical prediction over a wide range of similarity thresholds. This strong agreement between theoretical and empirical curves suggests that element-wise similarity is an intrinsic statistical property of expert weights.

3.2 Dual-Mask Fine-Grained Expert Merging↩︎

Our goal is to merge experts in a fine-grained manner that preserves both shared parameters and expert-specific capacity. To this end, we introduce a dual-mask formulation that decomposes expert parameters into two complementary components: shared parameters that can be safely merged, and salient parameters that should be preserved to maintain expert specialization.

Given two expert weight matrices \(\mathbf{W}_i\) and \(\mathbf{W}_j\) \(\in \mathcal{R}^{d \times h}\), which correspond to experts \(\mathbf{E}_i\) and \(\mathbf{E}_j\) from an MoE layer of a model with \(N\) experts \(\varepsilon = \{\mathbf{E}_1, \mathbf{E}_2, \dots, \mathbf{E}_N\}\), we construct two element-wise masks. The first is an element-wise similarity-based mask that identifies shared weight elements. The second is a saliency-based mask used to preserve divergent yet important weight elements. Additionally, we store the original signs of each expert’s weights and reapply them to \(\mathbf{W}_{merged}\) during inference. The overall algorithm is illustrated in Figure 3. This dual-mask design enables fine-grained expert merging that explicitly distinguishes between shared and specialized parameters, avoiding the over-merging behavior of coarse-grained approaches.

Similarity-based Mask. Given two experts weights \(\mathbf{W}_i\) and \(\mathbf{W}_j\), we measure their per-element magnitude similarity using a symmetric percent difference  [18]: \[\label{similarity95matric} \boldsymbol{\Delta} := \frac{\bigl||\mathbf{W}_i|-|\mathbf{W}_j|\bigr|}{|\mathbf{W}_i|+|\mathbf{W}_j|},\tag{1}\] where smaller values indicate greater similarity between two weight elements. It cleanly identifies elements in two experts’ weights that are similar in magnitude regardless of direction to avoid spurious penalties from opposite signs. After that, the similarity-based mask can be defined as: \[\mathbf{M}^{\mathrm{sim}} := \mathbf{1}_{\{\boldsymbol{\Delta}\le \tau_{\mathrm{sim}}\}} \in \{0,1\}^{d\times h},\] where \(\tau_{\mathrm{sim}}\in[0,1]\) is the pre-defined similarity threshold. \(\mathbf{M}^{\mathrm{sim}}\) is used to identify experts’ elements with comparable magnitude so that they can be safely aggregated. We also need to store each expert’s sign pattern and reapply it to \(\mathbf{W}_{merged}\) during inference, ensuring that shared merged elements can be reconstructed with minimal distortion: \[\mathbf{S}_i := \mathbf{1}_{\{\mathbf{W}_i < 0\}} \in \{0,1\}^{d \times h} , \mathbf{S}_j := \mathbf{1}_{\{\mathbf{W}_j < 0\}}\in \{ 0,1\}^{d\times h}.\] Saliency-based Mask. To decide which expert’s elements to preserve, we extend the idea of quantifying the importance of weights by combining the magnitude of weights with the saliency of input activations in dense models [19] to MoE experts: \[\mathbf{A}_{i} = |\mathbf{W}_{i}| \odot ||\mathbf{X}_{i}||_2, \qquad \mathbf{A}_{j} = |\mathbf{W}_j| \odot ||\mathbf{X}_j||_2,\] where \(\mathbf{X}\) represents a sample of input activations to a certain expert. Therefore, the saliency masks can be obtained as: \[\mathbf{M}^{\mathrm{sal}}_i \;:=\; \mathbf{1}_{\{\mathbf{A}_i \ge \mathbf{A}_j\}}\in \{0,1\}^{d\times h}, \qquad \mathbf{M}^{\mathrm{sal}}_j \;:=\; \mathbf{1} - \mathbf{M}^{\mathrm{sal}}_i .\] These two masks indicate which expert has more important weights to be preserved at each element position.

Expert Merging. Given the dual-mask formulation, two experts can be merged as follows. Specifically, elements identified as similar by the similarity mask \(\mathbf{M}_{\mathrm{sim}}\) are averaged in magnitude, while dissimilar elements are selected from the more salient expert using the saliency masks. Formally, this sparse merging process can be expressed as: \[\mathbf{M}_i \;=\; \mathbf{M}^{\mathrm{sal}}_i \vee \mathbf{M}^{\mathrm{sim}}, \qquad \mathbf{M}_j \;=\; \mathbf{M}^{\mathrm{sal}}_j \vee \mathbf{M}^{\mathrm{sim}},\] \[\begin{align} \mathbf{W}_{\mathrm{merged}} &= \mathbf{M}^{\mathrm{sim}} \odot \tfrac{|\mathbf{W}_i| + |\mathbf{W}_j|}{2} \\ & + (\mathbf{1} - \mathbf{M}^{\mathrm{sim}}) \odot \bigl( \mathbf{M}^{\mathrm{sal}}_i \odot |\mathbf{W}_i| \\ &+ \mathbf{M}^{\mathrm{sal}}_j \odot |\mathbf{W}_j| \bigr). \end{align}\] The merging process is done offline, and \(\mathbf{W}_{\mathrm{merged}},\;\mathbf{M}_i,\;\mathbf{M}_j, \;\mathbf{S}_i, \;\mathbf{S}_j\) are stored. At the inference stage, once an expert is activated, its weights are reconstructed element-wise as \[\widehat{\mathbf{W}}_i \;=\; (-1)^{\mathbf{S}_i} \odot \mathbf{M}_i \odot \mathbf{W}_\mathrm{merged}\, .\]

Pair-wise Experts Grouping. We apply expert merging in a pair-wise manner to balance effectiveness and tractability. Jointly merging \(k \geq 3\) experts would introduce a combinatorial masking problem: at each weight index, there are \((2^{k}-1)\) possible choices, causing the decision space to grow exponentially with \(k\) and making higher-order merging intractable for modern MoE models with many experts. In contrast, pair-wise merging enables closed-form mask construction with linear-time complexity and compact encoding. Beyond tractability, merging more than two experts simultaneously also leads to more aggressive information mixing. Empirically, we observe that higher-order merging results in noticeably larger accuracy degradation under the same overall sparsity. In contrast, pair-wise merging consistently achieves better accuracy-compression trade-offs than higher-order merging strategies, as shown in Appendix 7.8.

We consider two strategies for grouping experts: (1) random grouping, (2) search-based grouping. In practice, we find that random grouping performs comparably to search-based alternatives across models and sparsity levels, as shown in Section 5. Therefore, we adopt random grouping by default.

3.3 Efficient Inference with Bit-Packing↩︎

The sparse expert merging process transforms each pair of experts into a merged expert, 2 sign matrices, and 2 corresponding binary masks, which present a challenge to achieving efficient inference. As shown by [20], storing a tensor with 50% unstructured sparsity using Compressed Sparse Row format does not yield memory savings in practice, due to the substantial overhead of index storage. Furthermore, expert computation in MoE LLMs relies on dense matrix multiplication, which is highly optimized on modern GPUs but does not natively support masked operations. Thus, the need to dynamically fetch the appropriate mask during inference introduces nontrivial latency, particularly in the scheduling and memory access of matrix multiplication kernels. To address these limitations, we propose a system-level solution that combines efficient bit-level packing with a high-performance decode-GEMM kernel.

3.3.1 Observation↩︎

The BFloat16 data format (1 sign, 8 exponent, and 7 mantissa bits) is standard for LLM inference. Recent analyses [21]–[23] reveal significant underutilization of the 8-bit exponent range for dense LLMs in BFloat16 format. We found that this also applies for MoE LLMs. As shown in Figure 4 (a), for Mixtral-8x7B [1] model, the exponent values of expert weights are predominantly concentrated within a narrow range of 112 to 128. We also observe this in other MoE models as shown in Appendix 7.3.

3.3.2 System Co-design↩︎

Leveraging the concentrated exponent distribution, we apply a fixed shift to map all exponents into a 5-bit range, as illustrated in Figure 4. Specifically, any exponent smaller than 112 is rounded up to 112, and then all exponents are shifted down by 112, resulting in values that fall within the range of 0 to 31. The results in Appendix 7.4 indicate this transformation incurs no perplexity degradation for existing large MoE models. This is due to the functional equivalence in exponent processing when deriving FP16 weights from a BFloat16 baseline. This shift operation frees 3 bits within the original 8-bit exponent field, which can be used to pack the binary mask bits and a sign bit. Hence, the resultant \(\mathbf{W}_{merged}\) is stored in standard BFloat16 format, effectively embedding the masks and signs without additional storage.

We further develop a specialized decode-GEMM kernel in Triton that incorporates on-the-fly decoding to realize the benefit of the above bit-packing idea. The decode operation is shown in Algorithm 5, each weight \(W_d[i,j]\) is dynamically decoded from its packed format \(W[i,j]\) immediately prior to its use in the multiplication \(A[i,j]\times W_d[i,j]\). The efficiency of this approach hinges on its synergistic design. The decoding logic is a computationally trivial, in-place operation that piggybacks on the kernel’s existing data-loading path, which is already highly optimized for maximum memory throughput via techniques like warp-level scheduling and coalesced memory access. We eliminate the need for a separate materialization of the decoded matrix in memory, thereby avoiding significant latency and memory access overhead. Moreover, as shown in Figure 6, the proposed decode-GEMM kernel facilitates matrix multiplication for concatenated inputs \(A_i, A_j\) by performing a single access to the packed weight tensor to generate the corresponding outputs \(O_i,O_j\). While a conventional GEMM kernel necessitates two discrete operations, requiring separate memory transactions for each individual weight tensor. Our design leads to a theoretical 2\(\times\) latency saving in memory-bound GEMM operation during LLM decoding. We provide a more detailed kernel performance analysis in Appendix 7.1.

4 Experiments↩︎

4.1 Experimental Settings↩︎

Models and Baselines. We conduct a comprehensive evaluation of PuzzleMoE on four state-of-the-art MoE models: Mixtral-8x7B [1], Deepseek-MoE [2], Qwen1.5-MoE-A2.7B [3], and Qwen3-MoE-30B-A3B [24]. All the experiments are conducted on a cluster with 8 RTX5090 GPUs. We follow prior work to evaluate two distinct compression ratios of 25% and 50%, reducing the number of experts to 75% and 50% of the original count, while keeping the other modules unchanged. We compare PuzzleMoE against existing MoE compression methods, including expert dropping methods NAEE [4], STUN [5] and expert merging methods D2 [8], HC-SMoE [6], Sub-MoE [7]. We also compare with LLM pruning algorithm Wanda [19], whose 2:4 semi-structured sparsity is applied exclusively to the experts to ensure a fair comparison ¹.

Benchmarks and Evaluation. We evaluate the compressed models on language modeling perplexity and zero-shot task performance. Language modeling capabilities are assessed on the WikiText-2 [25] with a sequence length of 2048. For downstream tasks, we evaluate zero-shot accuracy across seven common benchmarks: ARC-c [26], ARC-e [26], HellaSwag [27], PIQA [28], BoolQ [29], Winogrande [30], and MMLU [31]. For all experiments, the calibration dataset has 128 samples, each with a sequence length of 2048 drawn from the C4 dataset [32]. We set the similarity threshold \(\tau_{\mathrm{sim}}=0.4\) fixed for all models and tasks, and evaluated 16 different random seeds for expert combination. The results in Table ¿tbl:tab:main95results?, Table 1, and Table 2 for PuzzleMoE are the average result. The detailed results for each seed are shown in Appendix 7.9.

4.2 Performance on General Tasks↩︎

From Table ¿tbl:tab:main95results?, we observe that for Mixtral-8x7B, PuzzleMoE achieves an average accuracy of \(73.2\%\) at \(25\%\) sparsity and \(72.6\%\) at \(50\%\) sparsity, substantially outperforming other baselines. In particular, even under aggressive \(50\%\) sparsity, PuzzleMoE still maintains performance very close to the dense model, while existing methods such as HC-SMoE and Sub-MoE suffer significantly larger degradation. Moreover, PuzzleMoE consistently achieves lower perplexity on WikiText compared with other compressed variants, suggesting that the proposed sparse expert merging strategy better preserves the original language modeling capability. For DeepSeek-MoE, PuzzleMoE incurs only minimal accuracy drops of \(0.2\%\) and \(0.5\%\) at \(25\%\) and \(50\%\) sparsity, respectively, while maintaining performance consistently superior to prior expert merging approaches. Similarly, for Qwen1.5-MoE, the degradation is limited to only \(0.1\%\) and \(0.5\%\), while for Qwen3-MoE, the drops are merely \(0.7\%\) and \(1.4\%\) under \(25\%\) and \(50\%\) sparsity. Notably, PuzzleMoE remains highly competitive on challenging reasoning benchmarks such as MMLU, ARC, and HellaSwag across all model families, demonstrating that the proposed fine-grained sparse expert merging strategy effectively preserves both factual knowledge and reasoning capability after expert compression. Overall, these results highlight the effectiveness and robustness of PuzzleMoE in preserving MoE models’ performance after expert merging, particularly under high sparsity regimes where existing methods experience substantial performance degradation.

4.3 Performance on Domain-specific Tasks↩︎

Math Reasoning Tasks. We evaluate PuzzleMoE on domain-specific mathematical reasoning tasks to assess its ability to preserve reasoning performance, as shown in Table 1. PuzzleMoE consistently achieves the best results among all methods, with accuracies of 55.4% and 51.7% at 25% and 50% sparsity, respectively. Moreover, the effectiveness of NAEE is strongly influenced by the choice of calibration data, performing better with the Math dataset than with C4. In contrast, PuzzleMoE remains robust regardless of calibration datasets as shown in Section 5.

We also evaluate Qwen3-MoE-30B-A3B model [24] in reasoning mode on the more challenging reasoning benchmarks (generation-intensive) with results reported in Table 2. Notably, under 25% sparsity, PuzzleMoE effectively preserves the model’s reasoning ability after compression, retaining 99%, 92%, and 84% of accuracy for the three benchmarks, respectively. In sharp contrast, HC-SMoE collapses to 24.6, 0.0, and 0.0 under the same sparsity. This highlights the effectiveness of PuzzleMoE in preserving the reasoning capability of MoE models.

4.4 Efficiency Analysis↩︎

Compression Cost. We measure the time required to compress the MoE models to 50% expert sparsity for PuzzleMoE, D2, HC-SMoE, and NAEE. As shown in Figure 7 (a), PuzzleMoE only takes 2 minutes for compressing Mixtral-8x7B, while D2 takes 52 minutes due to the heavy computation cost of SVD operation. For Deepseek-MoE, PuzzleMoE takes 9 minutes for compression, indicating the efficiency of PuzzleMoE on MoE models with a large number of experts. Note that NAEE’s reliance on exhaustive search makes the compression time quite expensive. For instance, applying it to DeepSeek-MoE with 64 experts requires \(10^{18}\) on the order of forward passes.

End-to-end Throughput. While reducing the memory footprint by 50%, PuzzleMoE further yields a substantial increase in end-to-end throughput. We evaluated the throughput of Mixtral-8x7B at a 50% compression ratio, with results illustrated in Figure 7 (b). Under a configuration of 16 prefill tokens and 128 decoding tokens across various batch sizes, PuzzleMoE achieves up to a 1.80\(\times\) speedup compared to the uncompressed baseline. This efficiency gain is primarily driven by our specialized PuzzleMoE decode-GEMM kernel, which delivers a 2\(\times\) throughput improvement over the standard cuBLAS implementation. For a more comprehensive analysis of kernel benchmarks and end-to-end metrics, please refer to Appendix 7.1 and 7.2.

5 Ablation Study↩︎

Impact of Calibration Datasets. We ablate different calibration datasets, and the results are shown in Table 3. It is clear that using C4 or MATH as the calibration dataset doesn’t lead to a big variance on performance, emphasizing the robustness of PuzzleMoE.

Expert Grouping Strategy. We compare our random pairwise expert grouping strategy with an evolutionary search-based pairwise grouping strategy. As shown in Table 4, the search-based grouping strategy only leads to a slight performance increase, indicating that PuzzleMoE’s effectiveness is hardly influenced by the pairwise grouping strategy.

Similarity Threshold \(\tau_{\mathrm{sim}}\). We report perplexity results for PuzzleMoE with different values of \(\tau_{\mathrm{sim}}\) as shown in Figure 8 (a). It is clear that small values underuse magnitude similarity, while large values merge too aggressively and lead to a substantially higher loss across the two experts. Values of \(\tau_{\mathrm{sim}}\) within the range of 0.3 to 0.5 yield the best performance; therefore, we fix \(\tau_{\mathrm{sim}}=0.4\) across models.

Combining PuzzleMoE with Quantization. We also investigated the compatibility of PuzzleMoE with quantization. We apply a symmetric group quantization scheme to the merged weights. As shown in Figure 8 (b), it achieves a substantial compression ratio of 4.8\(\times\), with a minimal accuracy drop of 1.7% for Mixtral-8x7B and 1.0% for Deepseek-MoE compared with full models, demonstrating that our merging strategy can be effectively combined with quantization without significant performance loss. Further details of quantized PuzzleMoE are shown in Appendix [sec:ablation95with95quantization].

6 Conclusion↩︎

We introduce PuzzleMoE, the first MoE merging method to enable element-wise merging while achieving both high accuracy and inference speed for large MoE models. Our approach utilizes a fine-grained expert merging strategy guided by dual masks, which enables model compression within minutes while effectively preserving model’s capacity. By using bit-packed encoding, PuzzleMoE facilitates efficient decoding on GPUs, offering a practical solution for deploying MoE models in real-world applications.

Acknowledgments↩︎

We sincerely appreciate the anonymous reviewers. Their insightful feedback helps improve the quality of the paper. The work utilized the Delta and DeltaAI system at the National Center for Supercomputing Applications (NCSA) and Jetstream2 at Indiana University through allocation CIS240055 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. The Delta advanced computing resource is a collaborative effort between the University of Illinois Urbana-Champaign and NCSA, supported by the NSF (award OAC 2005572) and the State of Illinois. UIUC SSAIL Lab is supported by research funding and gift from Google, IBM, Amazon, and AMD, including the Google ML and Systems Junior Faculty Award.

Impact Statement↩︎

This work aims to improve the efficiency and accessibility of Mixture-of-Experts large language models by enabling fine-grained expert merging with minimal performance degradation. By reducing model size, memory footprint, and inference cost, this work lowers the computational barrier to deploying high-capacity language models, potentially broadening access to advanced AI capabilities for researchers and practitioners with limited hardware resources.

langley00?

7 Appendix↩︎

7.1 Performance of PuzzleMoE Decode-GEMM Kernel↩︎

We evaluate the performance of our custom kernel on RTX5090 GPU and RTX4090 GPU using Triton Profiler. The benchmarking results, summarized in Figure 9 and Figure 10, span a diverse set of MoE weight matrix dimensions and typical LLM decoding batch sizes. Our PuzzleMoE decode-GEMM kernel consistently achieves approximately a 2\(\times\) speedup over the optimized cuBLAS baseline. This performance highlights the efficacy of our bit-packing strategy and the optimized memory access patterns of our kernel implementation, which effectively alleviate the memory bottlenecks typically associated with MoE decoding.

7.2 End-to-end Throughput of PuzzleMoE↩︎

We benchmark the end-to-end throughput of PuzzleMoE on a cluster with 8 RTX5090 GPUs at a 50% compression ratio across various sequence length configurations, specifically \((\text{prefill}, \text{decode})\) pairs of \((16, 128)\), \((128, 128)\), \((128, 512)\), and \((512, 512)\). The results, presented in Figure 11, demonstrate that PuzzleMoE achieves up to a 1.8\(\times\) throughput improvement over the baseline. We observe that the speedup margin diminishes as sequence lengths and batch sizes increase. This trend is attributed to the shifting computational bottleneck: at larger scales, attention operations—which remain uncompressed—constitute a larger fraction of the total execution time, thereby reducing the efficiency gains realized by our optimized MoE kernels.

7.3 Exponent Distribution Visualization of Different MoE Models↩︎

We provide a visualization of the exponent distribution of different MoE models in Figure 12.

7.4 Impact of Exponent Shifting↩︎

Table 5 shows the perplexity results on Wikitext-2 before and after exponent shifting. Exponent shifting incurs no change to model performance.

7.5 Visualization of MoE Expert Weight Distribution↩︎

We present a visualization of the expert weights (gate_proj parameters) for Mixtral-8x7B, Qwen1.5-MoE, and DeepSeekMoE models in Figure 13, 14, and 15. Our analysis demonstrates that the weight distributions approximately follow a Gaussian distribution. Notably, these distributions are centered at a mean of \(\mu = 0\), with a constant standard deviation \(\sigma\) across all experts within the same layer of a given model.

7.6 Explanation for Element-wise Similarity in MoE Experts’ Weights↩︎

We analyze element-level similarity in experts’ weights of MoE models. Consistent with previous findings [16], the weight distribution of modern LLMs approximates a normal distribution with a zero mean. To quantify inter-expert similarity, we computed the Pearson correlation coefficient between tensors in different pairs of experts within each layer, then we averaged the result for the whole model. As shown in Table ¿tbl:tab:correlation?, for Qwen1.5-MoE and Deepseek-MoE, the correlation is negligible. Mixtral-8x7B exhibits a higher correlation, indicating stronger dependencies between its expert weights. Consequently, for Qwen1.5-MoE and Deepseek-MoE, the values within their respective tensors can be treated as independent variables.

Based on this setting, we can view the process of selecting 2 values \(w_i,w_j\) of the corresponding positions of the 2 weights \(\mathbf{W}_i,\mathbf{W}_j\) as a sampling process. We define and solve the problem of calculating the proportion of similar values as follows:

We consider independent Gaussian random variables \[w_i \sim N(0,\sigma_j^2), \quad w_j \sim N(0,\sigma_j^2),\]

We aim to compute \[P\!\left(\frac{||w_i|-|w_j||}{|w_i|+|w_j|}<\tau_{\mathrm{sim}}\right), \qquad 0<\tau_{\mathrm{sim}}<1.\]

Since \(|w_i|,|w_j| \ge 0\), we have \[\frac{||w_i|-|w_j||}{|w_i|+|w_j|}<\tau_{\mathrm{sim}} \;\;\Longleftrightarrow\;\; ||w_i|-|w_j||<\tau_{\mathrm{sim}}(|w_i|+|w_j|) \;\;\Longleftrightarrow\;\; \frac{1-\tau_{\mathrm{sim}}}{1+\tau_{\mathrm{sim}}}<\frac{|w_i|}{|w_j|}<\frac{1+\tau_{\mathrm{sim}}}{1-\tau_{\mathrm{sim}}}.\] Define the ratio \[R = \frac{|w_i|}{|w_j|}.\]

The densities of \(|w_i|\) and \(|w_j|\) (half-normal distributions) are given by \[f_{1}(|w_i|) = \frac{\sqrt{2}}{\sigma_i\sqrt{\pi}} e^{-|w_i|^2/(2\sigma_i^2)},\] \[f_2(|w_j|) = \frac{\sqrt{2}}{\sigma_j\sqrt{\pi}} e^{-|w_j|^2/(2\sigma_j^2)}.\]

For \(R=|w_i|/|w_j|\), its pdf is \[f_R(r) = \int_0^\infty |w_j|\,f_1(r|w_j|)\,f_2(|w_j|)\,d(|w_j|) = \frac{2\sigma_i\sigma_j}{\pi\bigl(\sigma_i^2+\sigma_j^2r^2\bigr)}, \quad r \ge 0.\]

Let \[a = \frac{1-\tau_{\mathrm{sim}}}{1+\tau_{\mathrm{sim}}}, \qquad b = \frac{1+\tau_{\mathrm{sim}}}{1-\tau_{\mathrm{sim}}}.\] Then \[P = \int_a^b f_R(r)\,dr = \frac{2\sigma_i\sigma_j}{\pi}\int_a^b \frac{dr}{\sigma_i^2+\sigma_j^2 r^2} = \frac{2}{\pi}\Bigl[\arctan\!\Bigl(\tfrac{\sigma_j r}{\sigma_i}\Bigr)\Bigr]_a^b.\]

Hence, the final result is \[P\!\left(\frac{|\,|w_i|-|w_j|\,|}{|w_i|+|w_j|}<\tau_{\mathrm{sim}}\right) = \frac{2}{\pi}\left[ \arctan\!\Bigl(\tfrac{\sigma_j}{\sigma_i}\tfrac{1+\tau_{\mathrm{sim}}}{1-\tau_{\mathrm{sim}}}\Bigr) - \arctan\!\Bigl(\tfrac{\sigma_j}{\sigma_i}\tfrac{1-\tau_{\mathrm{sim}}}{1+\tau_{\mathrm{sim}}}\Bigr) \right].\]

In the case \(\sigma_i \approx \sigma_j\), which typically holds for most expert weights in MoE models as shown in 7.5, we obtain \[P\!\left(\frac{|\,|w_i|-|w_j|\,|}{|w_i|+|w_j|}<\tau_{\mathrm{sim}}\right) = \frac{2}{\pi}\left[ \arctan\!\Bigl(\tfrac{1+\tau_{\mathrm{sim}}}{1-\tau_{\mathrm{sim}}}\Bigr) - \arctan\!\Bigl(\tfrac{1-\tau_{\mathrm{sim}}}{1+\tau_{\mathrm{sim}}}\Bigr) \right].\]

The theoretical curve and empirical data from the three MoE models are plotted in Figure 16. The observed results for Qwen1.5-MoE and Deepseek-MoE align closely with the theoretical predictions. The data for Mixtral-8x7B shows a slight deviation, which is attributable to its violation of the independent distribution assumption. This analysis provides a clear explanation for the existence of fine-grained, element-wise similarity in expert weights and reinforces the generalizability and explainability of our method.

7.7 Performance on Coding Tasks↩︎

We further investigate the knowledge preservation ability of PuzzleMoE on coding tasks. Specifically, we compare different expert merging algorithms on Qwen1.5-MoE-A2.7B and evaluate them on HumanEval [33]. As shown in Table 6, PuzzleMoE preserves code-generation performance much better than HC-SMoE under the same sparsity levels: at 25% sparsity, PuzzleMoE attains a Pass@1 of 45.1 with only a small drop from the 49.4 dense baseline, while HC-SMoE collapses to 11.0; at 50% sparsity, PuzzleMoE still achieves 39.6, whereas HC-SMoE completely fails with 0.0 Pass@1. These results demonstrate that PuzzleMoE maintains strong coding ability even at high sparsity, while prior MoE merging methods severely degrade model reliability.

7.8 More Ablation Study↩︎

7.8.1 Combining PuzzleMoE with Quantization↩︎

We provide a detailed description of combining PuzzleMoE with quantization, as depicted in Figure 17. After the expert merging stage, the resultant floating-point weights are subjected to uniform quantization with a group size of 128. We employ a symmetric group quantization scheme, which obviates the need for a zero point. The final quantized values are stored alongside their corresponding sign and mask bits. The average bit width is subsequently determined by the following equation:

\[\text{Avg\;bitwidth} = (\underset{\text{Sign}}{\underline{2}} + \underset{\text{Compressed mask}}{\underline{1.58}} + \underset{\text{Quantize bitwidth}}{\underline{3}} + \underset{\text{Per-group scale}}{\underline{0.125}}) / 2 = 3.35\]

Each compressed mask belongs to the set {01,10,11}, as for each position, the merged weight either belongs to \(W_i\), \(W_j\), or both. Consequently, it requires only \(\log_{2}{3} \approx1.58\) bits for representation.

We target an 80% compression ratio, as this represents a high level of compression without a substantial loss in model performance. Accordingly, we apply 3-bit quantization to the merged weights within the PuzzleMoE framework at 50% sparsity. To evaluate our approach, we compare the post-quantization performance of the expert modules using 3-bit AWQ [13] and NAEE/HC-SMoE with 25% sparsity + 4-bit AWQ. The results are shown in Table ¿tbl:tab:quantization95results95full?.

On Mixtral-8x7B, AWQ attains the highest average accuracy of 73.0 at 3.25 bits, while PuzzleMoE achieves a comparable 72.4 accuracy with a similar bitwidth of 3.35. Compared to NAEE+AWQ and HC-SMoE+AWQ, which both reach only 70.1 average accuracy, PuzzleMoE provides a clear gain of more than 2 points, indicating that expert merging via PuzzleMoE preserves quantization performance better than prior MoE compression schemes. On Deepseek-MoE, PuzzleMoE further improves both Wiki and average accuracy at a bitwidth of 3.35, whereas HC-SMoE+AWQ suffers a substantial drop to 53.3 average accuracy. Overall, these results suggest that PuzzleMoE can be seamlessly combined with low-bit quantization, delivering robust accuracy improvements over NAEE and HC-SMoE at only a negligible increase in effective bitwidth.

7.8.2 Impact of Number of Experts to Merge↩︎

We ablate the number of experts to merge in PuzzleMoE. As shown in Table 7, at a 50% sparsity level, increasing the merge group size from 2 to 3 experts leads to degraded performance, as reflected by higher perplexity.

7.8.3 Impact of Saliency Metrics↩︎

To further investigate the effectiveness of the Wanda saliency metric, we evaluate several MoE-aware variants that incorporate router information to different extents: W (\(|W|\)), WG (\(|W|\cdot\|gate\|_{2}\)), WA (\(|W|\cdot\|X\|_{2}\), identical to the original Wanda metric), and WAG (\(|W|\cdot\|X \cdot gate\|_{2}\)), where \(W\) denotes the weight matrix, \(X\) the activation, and \(gate\) the router output.

As shown in Table ¿tbl:tab:wanda95variants?, the activation-based variants (WA and WAG) achieve very similar overall performance and consistently match or slightly outperform the weight-only variants (W and WG). In contrast, adding gating scores on top of weight-only saliency (WG vs.W) brings almost no benefit. This indicates that the main performance gains stem from incorporating activation information into the saliency metric, which better preserves expert specialization, while additional router-gating signals provide no significant improvements.

7.8.4 Extending PuzzleMoE to Higher Sparsity Levels↩︎

The current bit-packing design is indeed tailored to 2-to-1 merging. However, this does not constrain the dual-mask merging algorithm itself. If we store masks and signs explicitly without packing them into BFloat16, the same dual-mask merging algorithm can, in principle, support arbitrary compression ratios (e.g., 3-to-1, 4-to-1). In other words, the bit-packing scheme is introduced solely to reduce the memory footprint and make high-sparsity operation practical; it is not a limitation of the merging algorithm or of the achievable sparsity.

We further evaluate higher sparsity levels of 66.7% (merging three experts into one) and 75% (merging four experts into one) for both Deepseek-MoE-16b and Qwen3-MoE-30B-A3B. The evaluation follows the same benchmarks used in the submission (ARC-C/E, HellaSwag, PIQA, BoolQ, WinoGrande, and MMLU). The following tables report average results across these benchmarks. Under highly sparse settings, PuzzleMoE consistently outperforms the strong HC-SMoE baseline. For example, for Deepseek-MoE-16b at 75% sparsity, PuzzleMoE improves the average score from 35.7 to 54.1, demonstrating its strong robustness even under aggressive compression compared to the existing work.

7.9 Detailed Results of PuzzleMoE with Different Seeds↩︎

The detailed accuracy results of PuzzleMoE with different random seeds on Mixtral-8x7B, Deepseek-MoE, Qwen1.5-MoE, and Qwen3-MoE are shown in Table ¿tbl:tab:mixtral95seed?, ¿tbl:tab:deepseek95seed?, ¿tbl:tab:qwen146595seed?, ¿tbl:tab:qwen395seed?. Different seeds don’t lead to a significant difference in accuracy performance for both 25% sparsity and 50% sparsity, indicating the robustness of PuzzleMoE to different expert grouping choices.

References↩︎