1 Introduction↩︎

In recent years, Large Language Models (LLMs) have achieved remarkable success across a wide range of applications, including medical consultation [1], [2], information retrieval [3]–[6], code generation [7]–[10] and autonomous agents [11]–[14]. However, alongside their growing capabilities, the potential misuse of LLMs for generating misinformation, deceptive content, and malicious text has raised serious concerns about the integrity of the ecosystem [15], [16]. This has led to an increasing demand for reliable techniques to distinguish machine-generated text from human-written content, among which LLM watermarking has emerged as a promising direction.

Existing LLM watermarking techniques largely follow a common token-sampling-based paradigm where the model’s next-token distribution is subtly manipulated to encode identifiable patterns. For examples, KGW [17] partitions the vocabulary into disjoint subsets by random hashing and increases the logits of tokens in a designated subset; SynthID-text [18] adopts a tournament sampling strategy to inject watermark signals without introducing distributional distortion. While being effective for post-hoc detection, these methods inherently introduce overhead in the critical decoding path, as illustrated in Fig. 1 (a). Modern LLM serving systems are optimized for high-throughput, low-latency generation, where even lightweight per-token operations—such as hash-based partitioning, probability reweighing, or additional sampling—accumulate linearly with sequence length, leading to increased latency and significantly reduced system efficiency. At the same time, these approaches rely on explicitly altering token distributions, which inevitably interferes with the model’s natural generation behavior. This interference is particularly detrimental in delicate real-world tasks such as summarization, instruction following, code generation, etc. In those restricted settings, valid outputs are sparse in the token space and thus token probability shifting caused by watermarking tends to drive the generated text to deviate from correct or optimal outputs. As shown in Fig. 1 (b), we evaluate existing watermarking methods under different parameter settings on MultiNews for a summarization task. The results show a clear ‘impossible’ region at the top-right corner indicating high detection rates and high fidelity.

Therefore, a fundamental question arises: can we design a watermarking mechanism that achieves high efficiency, fidelity and detection rate at the same time?

To answer the question, we propose WaterMoE, an expert-routing-based watermarking approach for Mixture-of-Experts (MoE) LLMs. MoE models have been increasingly adopted in modern high-performance language models [19], [20], where inference is carried out through sparse expert activation and routing decisions. We are motivated by the inherent redundancy in MoE models that allows different experts to be chosen so that each input token takes a flexible combination of model weights, leading to a varied output distribution [21], [22]. Hence without altering the model’s generation strategy, the expert selection brings in ‘controlled randomness’ in the output distribution: at each routing gate, top-\(k\) experts are potentially chosen, allowing stable and detectable internal patterns to be expressed through subtle biases at the routing level, while maintaining generation capability. Building on this observation, we integrate watermark signals directly into the MoE routing logic, making the watermark an intrinsic component of the model’s inference process.

Our contributions are summarized as follows. 1) Real-world evaluation of watermarking: we systematically reveal the limitations of existing watermarking methods under realistic workloads, and construct a comprehensive benchmark spanning summarization, question answering, code generation, reasoning tasks and instruction following to evaluate the watermark detection rate, efficiency, and task performance. 2) Structural watermarking via MoE routing: we propose WaterMoE, a watermarking approach that embeds signals into expert routing. Extensive experiments show that WaterMoE improves detection under strict false positive constraints by 12.1% at TPR@1% over the strongest baseline, without degradation from unwatermarked task performance. Our method achieves up to 4\(\times\) speedup in watermark embedding compared to state-of-the-art methods, incurring merely 1% latency overhead compared to native generation, demonstrating strong practicality.

2 Preliminary and Related Work↩︎

In recent years, Mixture-of-Experts (MoE) has been increasingly adopted in production-level large language models for online services, such as DeepSeek-V3 [23] and Qwen3 [24]. Mixture-of-Experts (MoE) architectures introduce a routing mechanism that enables sparse activation of model parameters during inference [19], [21]. In a typical MoE layer, the standard feed-forward network is replaced by multiple parallel expert networks, while a router dynamically selects a small subset of experts for each token based on routing scores. Only the top-\(k\) experts are activated, and their outputs are aggregated through a weighted sum gate [25]. This design allows MoE models to scale up model capacity while keeping inference computationally bounded, which has been widely adopted in modern large-scale language models.

Formally, given the hidden representation \(\mathbf{h}_\ell\) at layer \(\ell\), the router produces routing scores \(\mathbf{s}_\ell \in \mathbb{R}^K\), where \(K\) denotes the number of experts in layer \(\ell\). The scores are converted into a routing distribution via a softmax, the top-\(k\) of which are selected as experts \(\mathcal{E}_\ell(\mathbf{h}_\ell)\) to compute the output of the layer \(\mathrm{MoE}_\ell(\mathbf{h}_\ell)\): \[\mathbf{s}_\ell = \mathrm{Router}_\ell(\mathbf{h}_\ell),~~ p_\ell(i \mid \mathbf{h}_\ell) = \mathrm{softmax}(\mathbf{s}_\ell)_i, ~~~ \mathrm{MoE}_\ell(\mathbf{h}_\ell) = \sum_{i \in \mathcal{E}_\ell(\mathbf{h}_\ell)} p_\ell(i \mid \mathbf{h}_\ell)\, f_i(\mathbf{h}_\ell),\] where \(f_i(\cdot)\) denotes the \(i\)-th expert network and \(p_\ell(i \mid \mathbf{h}_\ell)\) is the corresponding routing weight. More details of the MoE framework are provided in Appendix 8.

LLM watermarking methods mostly embed watermark signals by modifying the token logits distribution in autoregressive decoding. KGW [17] partitions the vocabulary via N-gram hashing and biased generation toward a selected subset. EWD [26] improves detection rate by entropy-aware embedding, SynthID [18] proposes a non-distortionary watermarking methods through tournament sampling, and MorphMark [27] takes an adaptive watermarking approach. However, most token-sampling-based methods rely on post-hoc logit perturbations, causing text quality degradation and additional per-token generation overhead (e.g., hashing, dynamic reweighing, etc.) that are difficult to mitigate. The quality degradation is attributed to the conflict between the sampling constraints posed by watermarking and the harsh requirement in real-world generation including factual correctness, strict formatting, program executability, etc.: the former often asks for a deviation from the original output while the latter forbids that. Hence the current approach often has to sacrifice watermarking accuracy for text quality (as shown in Fig. 1). The additional overhead largely stems from the need for reproducible randomness, typically implemented via per-token hashing, which breaks the fully GPU-resident execution pipeline, leading to noticeable inference latency. While precomputing randomness on GPU could in principle remove this cost, it would require non-trivial GPU kernel engineering or introduce additional memory and implementation overhead.

In contrast, our method embeds watermark signals into the expert routing mechanism of Mixture-of-Experts models. By introducing perturbation in the expert selection rather than the token choice, watermark embedding is seamlessly integrated into routing priors, making it an inherent part of inference thus incurring negligible latency overhead while preserving output quality to a large extent.

3 WaterMoE: Watermarking via Expert Routing↩︎

Our design is motivated by the flexibility in MoE models’ routing mechanism: experts with similar routing scores often possess comparable functional capabilities. Hence a small routing bias can shift expert selection which gradually amplifies through layers leading to subtle, but traceable variations in the final output distribution. Since experts with similar capabilities are selected, our framework preserves fidelity while being highly efficient. The proposed framework is shown in Fig. 2.

3.1 Watermark Embedding through Routing Bias↩︎

To enable efficient watermark embedding at scale, we introduce a Green Expert Map \(\mathcal{M}_\ell\) that is randomly generated offline, and deterministically specifies the green experts for each MoE layer. The green experts refer to those experts perturbed by the injected biases. Let \(e_{\ell-1} \in \{1, \dots, K\}\) denote the top-1 selected expert index from layer \(\ell-1\). The green expert set at layer \(\ell\) is then determined as: \[\mathcal{G}_\ell = \mathcal{M}_\ell(e_{\ell-1}), \quad \text{s.t.} \quad \mathcal{G}_\ell \subseteq \{1, \dots, K\}, \quad |\mathcal{G}_\ell| = K/2.\] For the first MoE layer, we skip this procedure due to the absence of a preceding expert. We utilize a binary indicator vector \(\mathbf{g}_\ell \in \{0,1\}^K\) to denote the selected green set: \[g_{\ell,i} = \begin{cases} 1, & i \in \mathcal{G}_\ell, \\ 0, & \text{otherwise}. \end{cases}\] The routing bias for layer \(\ell\) is defined by \(\delta \, \mathbf{g}_\ell\), where \(\delta > 0\) signaling the watermark strength. Hence the biased routing logits are: \[\tilde{\mathbf{s}}_\ell = \mathbf{s}_\ell + \delta \, \mathbf{g}_\ell. \label{eq:routingbias}\tag{1}\] We further define the corresponding routing probabilities \(\mathbf{p}_{\ell}\) and \(\tilde{\mathbf{p}}_{\ell}\) for \(\mathbf{s}_\ell\) and \(\tilde{\mathbf{s}}_\ell\), respectively, by passing them through a Softmax function.

A rigorous proof of this property is provided in Appendix 9. The property indicates that with the first-order term vanishing, the routing distribution shifts by at most the second order of \(\delta\), which is almost negligible.

In practice, the green expert map is precomputed offline and stored as a tensor aligned with MoE layers. During inference, the routing bias is applied via a single element-wise addition (Eq. 1 ), which is fully GPU-parallelizable and introduces negligible latency overhead.

During autoregressive generation, the routing bias is consistently applied to every token across all MoE layers. As a result, the expert selection process is subtly biased toward green experts. Although the token logits are not explicitly modified, the perturbed routing alters the internal representations that ultimately induce subtle but consistent shifts in the output distribution, enabling statistical detection using standard hypothesis testing methods (e.g., Z-test).

3.2 Likelihood-Based Watermark Detection via Reference Calibration↩︎

For watermark detection, the key idea is to detect the distributional shift caused by routing biases. We use an unwatermarked model to estimate the intrinsic likelihood of each token as a reference. Then we calibrate token-level probabilities through disentangling routing-induced preference from the native generations, thereby isolating the watermark signals.

Given a sequence \(\mathbf{x} = (x_1, \dots, x_T)\), we evaluate it using a single model under two routing configurations: i) a biased mode with the watermarking enabled, and ii) a reference mode without routing bias. Importantly, both configurations share the same model parameters and differ only in the expert routing path, thus needing no two separate model deployments.

For each token, we compute the log likelihood difference as: \[\Delta_t = \ell_t^{\mathrm{bias}} - \ell_t^{\mathrm{base}}, \quad \ell_t^{\mathrm{bias}} = \log P_{\mathrm{bias}}(x_t \mid x_{<t}), \quad \ell_t^{\mathrm{base}} = \log P_{\mathrm{base}}(x_t \mid x_{<t}),\] where \(P_{\mathrm{bias}}\) and \(P_{\mathrm{base}}\) denote the conditional distributions of \(x_t\) under configuration i) and ii). A higher \(\ell_t^{\mathrm{base}}\) indicates that the token is highly consistent with the context. \(\Delta_t\) measures the additional preference induced by the routing bias for the token. We aggregate token-level signals of which the log likelihood difference exceeds threshold \(\tau\): \[S(\mathbf{x}; \tau) = \frac{1}{T} \sum_{t=1}^T \mathbb{I}(\Delta_t > \tau).\]

On the reference model, we estimate the expectation \(\mu_0(\tau)\) and standard deviation \(\sigma_0(\tau)\) of \(S(\mathbf{x}; \tau)\), and calibrate it by: \[Z(\mathbf{x}) = \frac{S(\mathbf{x}; \tau) - \mu_0(\tau)}{\sigma_0(\tau)}.\] A sequence is flagged as watermarked if \(Z(\mathbf{x})\) exceeds a threshold.

Why works? WaterMoE achieves efficiency by significantly reducing the watermark embedding time at the cost of mildly increased detection overhead. Previous LLM watermarking methods contain inefficient hashing in both watermark embedding and detection for reproducible randomness given the context. However, embedding is far more frequent than detection in practice: every token generated is required to be watermarked but detection is upon request. Thus we choose to remove hashing but introduce perturbed expert mapping in the embedding. Such expert mapping is on one hand of small scale, compared with the perturbation at the logits; on the other, the perturbation gets progressively amplified through the model’s feed-forward dynamics, resulting in consistent and detectable statistical signals in the output distribution without relying on explicit hashing.

WaterMoE also produces output of high quality. The reason is that the routing biases primarily influence the selection among functionally similar experts, ensuring that the model’s internal representations are only mildly affected. This allows the model to retain sufficient flexibility to explore the full token space and to self-correct through subsequent forward passes, thereby preserving performance even in highly restricted scenarios. As these subtle routing shifts accumulate, they induce lightweight yet consistent deviations at the token level: the generated outputs remain valid and task-compliant, while exhibiting slight statistical differences from the original distribution, which can be reliably exploited for watermark detection.

4 Experimental Results↩︎

To comprehensively evaluate the effectiveness of WaterMoE, we organize our experiments around the following research questions. RQ1 Detection Performance: Can WaterMoE achieve reliable and highly accurate detection across different models and tasks? RQ2 Generation Quality: How does WaterMoE affect the generation quality? RQ3 Efficiency: What is the computational overhead of watermark embedding and detection compared to existing methods? RQ4 Robustness: How robust is WaterMoE under realistic adversarial attacks? RQ5 Stealthiness: Can WaterMoE remain statistically indistinguishable from unwatermarked text under black-box detection?

4.1 Setup↩︎

All experiments are implemented based on the MarkLLM [28] repository and conducted on NVIDIA H200 GPUs. We consider two representative MoE models: Mixtral-8\(\times\)​7B-Instruct-v0.1 [29]and Qwen3-30B-A3B-Instruct [24], both of which adopt sparse expert routing mechanisms. We compare WaterMoE against a set of representative watermarking baselines, as summarized in Table 2. For WaterMoE, we set as default \(\delta = 0.2\) and \(\tau = 0\) across all experiments. The complete experimental setup and hyperparameter configurations are provided in Appendix 12.

We construct a benchmark suite spanning diverse LLM evaluation settings, organized into three tiers by task complexity, as illustrated in Table 1: Low-, Moderate-, and High-Complexity tasks. This tiered design enables a systematic assessment of watermark performance under varying difficulty and constraint levels. Notably, the low-complexity setting where \(\delta = 0.8\) is included to ensure fair comparison with existing watermarking methods.

4.2 RQ1 & RQ2: Detection Performance and Generation Quality↩︎

Performance on High-Complexity Tasks. Table 2 reports watermarking performance across highly constrained domains which impose rigid structural and logical requirements, making watermark injection particularly challenging. Under a strict false positive rate (TPR@1%), baseline methods (e.g., EWD, SynthID) exhibit notable degradation in detectability, often falling below 70% on reasoning and coding benchmarks such as GSM8K, MMLU, and CodeContests. EXPEdit performs poorly in such settings, as it operates at the sample level, where the space of valid edits is severely constrained in high-complexity tasks, making watermark injection difficult. In contrast, WaterMoE achieves consistently strong detection performance, exceeding 90% on challenging datasets like APPS and CodeContests, and reaching 100.0% on alignment benchmarks (IFEval, WritingBench). Importantly, this high detectability does not come at the cost of generation quality: while conventional methods (e.g., KGW, SynthID) incur substantial accuracy drops (over 15 points) on strict tasks, WaterMoE maintains performance closely aligned with the unwatermarked model across all datasets, effectively mitigating the detectability–quality trade-off under rigid constraints. The results for moderate-complexity tasks are provided in Appendix 10.

Performance on Low-Complexity Tasks. Table 3 presents results on low-complexity, open-ended generation tasks (C4 and Booksum). While such tasks are generally friendly to watermarking, the detection performance varies across model architectures. On Mixtral, most methods achieve near-perfect detection, but noticeable degradation occurs on Qwen under strict false positive constraints (TPR@1%), where baselines such as MorphMark, SynthID, and KGW exhibit significant drops. In contrast, WaterMoE maintains consistent 100.0% detection rate across all models and datasets, demonstrating strong model-independent stability. Importantly, linguistic fluency is not sacrificed: while some baselines (e.g., EXPEdit, EWD) substantially raises perplexity, WaterMoE achieves highly competitive PPLs among watermarked models, closely aligning with the unwatermarked baseline. These results indicate that WaterMoE effectively preserves fluency while ensuring reliable detectability in unconstrained generation settings.

4.3 RQ3: Efficiency Analysis↩︎

As shown in Fig. 3 (a), WaterMoE introduces negligible computational overhead, with a clear advantage on the insertion side. WaterMoE achieves an insertion latency of only 1.1%, significantly lower than all baselines. This is enabled by the GPU-resident expert mapping design, which avoids extra data movement and integrates seamlessly into the inference pipeline. Our detection latency remains comparable to existing methods (around 150ms per sample), mostly due to the non-autoregressive scheme that performs a single prefill to obtain expert routing distributions for the entire sequence. In practice, insertion efficiency is more crucial as online generation is latency-critical, while detection occurs sparsely upon requests.

4.4 RQ4: Robustness under Adversarial Attacks↩︎

As shown in Fig. 3 (b), WaterMoE’s robustness differ across attacks. It achieves competitive robustness under synonym substitution by wordnet, but shows a noticeable degradation under aggressive word deletion. This behavior stems from the design of WaterMoE, which relies on the preservation of local contextual coherence. When a large fraction of tokens is removed, the underlying linguistic structure is severely disrupted, weakening the statistical signal for detection. We note that such strong deletion-based perturbations, while commonly adopted in prior work, often lead to text that is syntactically unnatural or semantically degraded, reflected by a sharp increase in perplexity (PPL) from 7 to 45 in our setting. In many real-world scenarios, adversaries are more likely to apply semantics-preserving transformations rather than heavily destructive edits to maintain usability of the generated content. To better reflect this practical threat model, we use GLM-4.5 for full-text paraphrasing. The detection rate remains at 63% TPR@1% for WaterMoE, indicating moderate robustness under strong paraphrasing attacks. We additionally evaluate robustness under an API-assisted word-level substitution with \(\sim30\%\) token replacement. WaterMoE maintains a 98% TPR@1% detection rate under the perturbation.

4.5 RQ5: Stealthiness Analysis↩︎

Beyond detectability by authorized parties, a desirable watermark should remain statistically indistinguishable from unwatermarked text under general-purpose analysis. We evaluate this property using two representative black-box statistical detectors [44], namely R-G detection and FIXED detection, which are designed to identify common watermarking artifacts, including distributional bias and diminished generation diversity. As shown in Table 4, existing methods are detectable under at least one setting, while WaterMoE consistently achieves high \(p\)-values across all cases. This indicates that its outputs remain statistically indistinguishable from the unwatermarked text, demonstrating strong stealthiness.

4.6 Sensitivity to Routing Bias↩︎

We analyze the sensitivity of the proposed watermarking method to the routing bias strength \(\delta\) by varying it from 0.2 to 1.2. As shown in Table 5, the detection performance remains consistently high across a wide range of bias values. In particular, both TPR and ROC-AUC exhibit near-saturated performance, while the perplexity (PPL) increases only marginally as the bias grows. This demonstrates that our method is robust to the choice of routing bias and can be easily applied in a plug-and-play manner without careful hyperparameter tuning.

r0.6

5 Limitation↩︎

We acknowledge that our current design is tailored to Mixture-of-Experts architectures. However, this choice is motivated by the increasing adoption of sparse activation paradigms in modern LLM systems (e.g., DeepSeek, Qwen-MoE), where routing naturally provides a structured and low-overhead carrier for watermark signals. For dense Transformer models, a promising direction is to emulate implicit expert partitioning through structured sparsification. Specifically, one could leverage techniques such as dynamic neuron activation or structured pruning (e.g., block-wise or head-wise sparsity) to create multiple functionally redundant subspaces within dense layers.

6 Conclusion↩︎

Watermarking for LLMs remains challenging due to performance degradation and inference overhead, particularly in complex and constrained tasks. In this work, we systematically identify these limitations through a comprehensive benchmark and propose WaterMoE, a watermarking scheme tailored for MoE-based LLMs. By integrating watermark signals into the routing process, WaterMoE achieves strong detectability while preserving model quality and efficiency. Extensive experiments demonstrate its consistent advantages over existing methods, highlighting its practicality for real-world deployment.

7 Related Work↩︎

Semantic-level watermarking. Recent approaches explore embedding watermark signals at the semantic level. SIR [42] employs learned encoders to map semantically similar inputs to similar watermark patterns, improving robustness to paraphrasing. However, this design introduces additional system complexity and incurs non-trivial latency due to sentence-level encoding. Moreover, by conditioning watermark signals on semantic representations, such methods implicitly constrain the feasible token space during generation, leading to similar limitations as token-level approaches in restricting the solution space.

Subsequent methods such as SEMSTAMP [45] and KSEMSTAMP [46] adopt a different strategy by discretizing semantic representations via locality-sensitive hashing or clustering (e.g., k-means), and enforcing constraints on the semantic space of the next sentence. In practice, these methods rely on repeated resampling until a candidate satisfying the watermark constraint is found. While effective in controlled settings, this mechanism introduces substantial computational overhead, often requiring tens of resampling iterations per sentence, which leads to significant latency.

More critically, the imposed semantic constraints further shrink the already limited feasible solution space in many real-world tasks, such as summarization and question answering, where correctness and coherence impose strict requirements. Empirical evidence shows that even with up to 50 resampling attempts, SEMSTAMP frequently fails to embed watermarks successfully on datasets such as MultiNews and ELI5 benchmarks. These limitations suggest that existing semantic-level watermarking methods are not well-suited for practical deployment in constraint-heavy generation scenarios.

8 Detailed Background on Mixture-of-Experts (MoE)↩︎

8.1 Overview↩︎

Mixture-of-Experts (MoE) is a sparsely-activated neural architecture designed to scale model capacity without incurring proportional computational cost. Instead of using a single dense feed-forward network (FFN) in each Transformer layer, MoE replaces it with a set of \(K\) parallel expert networks and a routing mechanism that dynamically selects a subset of experts for each token.

As illustrated in Fig. 4, an MoE layer consists of three key components:

• Router (Gating Network): computes routing scores for each expert.

• Experts: a set of independent feed-forward networks.

• Aggregation: combines outputs from selected experts via weighted summation.

This sparse activation enables MoE models to achieve extremely large parameter counts while keeping the per-token computation comparable to standard dense models.

8.2 Formulation↩︎

Given the hidden representation \(\mathbf{h}_\ell \in \mathbb{R}^d\) at layer \(\ell\), the router produces a score vector: \[\mathbf{s}_\ell = \mathrm{Router}_\ell(\mathbf{h}_\ell), \quad \mathbf{s}_\ell \in \mathbb{R}^K,\] where \(K\) is the number of experts.

These scores are transformed into a probability distribution via softmax: \[p_\ell(i \mid \mathbf{h}_\ell) = \frac{\exp(s_{\ell,i})}{\sum_{j=1}^{K} \exp(s_{\ell,j})}.\]

Instead of using all experts, only the top-\(k\) experts are selected: \[\mathcal{E}_\ell(\mathbf{h}_\ell) = \mathrm{TopK}(p_\ell(\cdot \mid \mathbf{h}_\ell), k).\]

The final output of the MoE layer is: \[\mathrm{MoE}_\ell(\mathbf{h}_\ell) = \sum_{i \in \mathcal{E}_\ell(\mathbf{h}_\ell)} p_\ell(i \mid \mathbf{h}_\ell) \cdot f_i(\mathbf{h}_\ell),\] where \(f_i(\cdot)\) denotes the \(i\)-th expert network.

8.3 Integration with Transformer↩︎

In modern large language models, MoE layers replace the standard FFN sub-layer in Transformer blocks. A typical Transformer layer with MoE can be written as: \[\begin{align} \mathbf{h}'_\ell &= \mathbf{h}_\ell + \mathrm{Attention}(\mathrm{LN}(\mathbf{h}_\ell)), \\ \mathbf{h}_{\ell+1} &= \mathbf{h}'_\ell + \mathrm{MoE}_\ell(\mathrm{LN}(\mathbf{h}'_\ell)), \end{align}\] where \(\mathrm{LN}(\cdot)\) denotes Layer Normalization.

8.4 Routing Strategies↩︎

Different routing strategies are used in practice:

8.4.0.1 Top-\(k\) Routing

The most common strategy selects the top-\(k\) experts per token. Typical choices include:

• \(k=1\) (Switch Transformer): maximum efficiency

• \(k \geq 2\) (GShard, DeepSeek): better performance with modest overhead

8.5 Computational Efficiency↩︎

Although the total number of parameters grows linearly with \(K\), the actual computation per token depends only on \(k\): \[\text{FLOPs} \approx O(k \cdot d^2),\] which is comparable to a dense FFN when \(k \ll K\).

Thus, MoE achieves:

• High capacity (large parameter count)

• Low compute cost (sparse activation)

9 Theoretical Analysis of Routing Loss under Biased Routing↩︎

In this section, we mathematically prove that the expected routing probability deviation caused by the watermark bias is strictly bounded by \(\mathcal{O}(\delta^2)\). Thus, the routing process remains unbiased in expectation at the first order.

9.1 Notations and Assumptions↩︎

Model State. Let \(\mathbf{s}_\ell \in \mathbb{R}^K\) denote the original routing logits at the \(\ell\)-th layer. The original routing probability for the \(i\)-th expert, obtained via the Softmax function, is defined as: \[p_i = \frac{e^{s_{\ell,i}}}{\sum_{j=1}^K e^{s_{\ell,j}}}.\]

Permutation-Symmetric Expert Map. We define a binary indicator vector \(\mathbf{g}_\ell \in \{0, 1\}^K\) to represent the green expert assignment at the current layer. Based on the expert map, the assignment is determined by \(\mathcal{G}_\ell = \mathcal{M}_\ell(e_{\ell-1})\). This mapping satisfies two core constraints: (i) Fixed Green Proportion, which dictates that exactly half of the experts are activated as green experts per routing, i.e., \(\sum_{i=1}^K g_{\ell,i} = K/2\); and (ii) Permutation-Symmetric Constraint , which ensures that in global expectation, every expert has an equal probability of being selected as a green expert: \[\Pr(g_{\ell,i} = 1) = \Pr(g_{\ell,i} = 0) = \frac{1}{2}, \quad \forall i \in \{1, \dots, K\}.\]

Bias Injection. The actual routing bias vector applied at the current layer is \(\mathbf{b}_\ell = \delta \mathbf{g}_\ell\). Due to the binary nature of \(\mathbf{g}_\ell\), each component of the bias vector is restricted to a discrete set \(b_{\ell,j} \in \{0, \delta\}\). We assume the bias magnitude \(\delta\) is sufficiently small (\(\delta \ll 1\)) so that the higher-order Taylor expansion terms \(\mathcal{O}(\|\mathbf{b}_\ell\|^3)\) can be reasonably truncated. The routing probability after bias injection is denoted as \(p_i^{(g)}\).

9.2 Proof of Zero First-Order Expectation↩︎

According to the multidimensional Taylor expansion, the perturbed probability \(p_i^{(g)}\) at \(\mathbf{s}_\ell\) can be expanded as: \[p_i^{(g)} = p_i(\mathbf{s}_\ell) + \nabla p_i(\mathbf{s}_\ell)^\top \mathbf{b}_\ell + \frac{1}{2} \mathbf{b}_\ell^\top H^{(i)}(\mathbf{s}_\ell) \mathbf{b}_\ell + \mathcal{O}(\|\mathbf{b}_\ell\|^3).\] The first-order probability deviation is defined as \(\Delta p_i^{(1)} = \nabla p_i(\mathbf{s}_\ell)^\top \mathbf{b}_\ell\). Expanding the inner product and taking the mathematical expectation \(\mathbb{E}_{\mathbf{g}}\) over the bias distribution yields: \[\mathbb{E}_{\mathbf{g}}\left[ \Delta p_i^{(1)} \right] = \mathbb{E}_{\mathbf{g}}\left[ \sum_{j=1}^K \frac{\partial p_i}{\partial s_{\ell,j}} b_{\ell,j} \right] = \delta \sum_{j=1}^K \frac{\partial p_i}{\partial s_{\ell,j}} \mathbb{E}_{\mathbf{g}}[g_{\ell,j}].\] From the Permutation-Symmetric Constraint, we know \(\mathbb{E}_{\mathbf{g}}[g_{\ell,j}] = \Pr(g_{\ell,j}=1) = 1/2\). Substituting this into the expectation gives: \[\mathbb{E}_{\mathbf{g}}\left[ \Delta p_i^{(1)} \right] = \frac{\delta}{2} \sum_{j=1}^K \frac{\partial p_i}{\partial s_{\ell,j}}.\] Using the partial derivative property of the Softmax function, we have: \[\sum_{j=1}^K \frac{\partial p_i}{\partial s_{\ell,j}} = p_i(1 - p_i) - \sum_{j \neq i} p_i p_j = p_i - p_i \sum_{j=1}^K p_j.\] Given the probability normalization property \(\sum_{j=1}^K p_j = 1\), this sum strictly evaluates to 0. Therefore, the expected first-order deviation is completely eliminated, yielding \(\mathbb{E}_{\mathbf{g}}[\Delta p_i^{(1)}] = 0\).

9.3 Bounding the Second-Order Term↩︎

The second-order error term is \(\Delta p_i^{(2)} = \frac{1}{2} \mathbf{b}_\ell^\top H^{(i)}(\mathbf{s}_\ell) \mathbf{b}_\ell\). The quadratic form can be explicitly expressed as: \[\mathbf{b}_\ell^\top H^{(i)} \mathbf{b}_\ell = p_i \left[ (b_{\ell,i} - \bar{b})^2 - \left( \sum_{j=1}^K p_j b_{\ell,j}^2 - \bar{b}^2 \right) \right],\] where \(\bar{b} = \sum_{j=1}^K p_j b_{\ell,j}\) represents the expected value of the bias vector under the original probability distribution \(\mathbf{p}\). We now bound the two components inside the brackets.

First term. Since \(b_{\ell,i} \in \{0, \delta\}\) and the weighted average satisfies \(0 \le \bar{b} \le \delta\), the squared difference between them is strictly bounded by \((b_{\ell,i} - \bar{b})^2 \le \delta^2\).

Second term. This component represents the variance of the bias under distribution \(\mathbf{p}\). Since the indicator variable is strictly binary (\(g_{\ell,j} \in \{0, 1\}\)), it inherently satisfies \(g_{\ell,j}^2 = g_{\ell,j}\). For the bias terms \(b_{\ell,j} = \delta g_{\ell,j}\), this implies: \[b_{\ell,j}^2 = (\delta g_{\ell,j})^2 = \delta^2 g_{\ell,j} = \delta (\delta g_{\ell,j}) = \delta b_{\ell,j}.\] Substituting this identity into the probability-weighted sum yields: \[\sum_{j=1}^K p_j b_{\ell,j}^2 = \sum_{j=1}^K p_j (\delta b_{\ell,j}) = \delta \left( \sum_{j=1}^K p_j b_{\ell,j} \right) = \delta \bar{b}.\] The entire second term then simplifies to a downward-opening quadratic function with respect to \(\bar{b}\). By completing the square, we obtain: \[\sum_{j=1}^K p_j b_{\ell,j}^2 - \bar{b}^2 = \delta \bar{b} - \bar{b}^2 = -\left(\bar{b} - \frac{\delta}{2}\right)^2 + \frac{\delta^2}{4}.\] Because the squared component is non-negative, the global maximum of this term is achieved at \(\bar{b} = \delta/2\), leading to the strict bound: \[\sum_{j=1}^K p_j b_{\ell,j}^2 - \bar{b}^2 \le \frac{\delta^2}{4}.\]

Combining the bounds for both the first and second terms, the absolute value of the total second-order deviation is limited by: \[\left| \Delta p_i^{(2)} \right| \le \frac{1}{2} p_i \left( \delta^2 + \frac{\delta^2}{4} \right) = \frac{5}{8} p_i \delta^2 = \mathcal{O}(\delta^2).\]

Conclusion↩︎

Combining the derivations above, under the first-order approximation, the expected routing probability satisfies: \[\mathbb{E}_{\mathbf{g}}[p_i^{(g)}] = p_i + \mathbb{E}_{\mathbf{g}}[\Delta p_i^{(1)}] + \mathbb{E}_{\mathbf{g}}[\Delta p_i^{(2)}] = p_i + 0 + \mathcal{O}(\delta^2).\] This confirms that the watermark injection preserves the expected routing probabilities (i.e., \(\mathbb{E}_{\mathbf{g}}[p_i^{(g)}] = p_i\) at the first order), demonstrating that the routing process remains unbiased in expectation.

10 Performance on Moderate-Complexity Tasks.↩︎

Table 6 shows the detection-text quality trade-off on both MultiNews and ELI5. On the smaller Mixtral-8\(\times\)​7B model, which is harder to achieve balance, WaterMoE achieves substantially higher detection rates under strict thresholds while also attaining the highest AUC and maintaining Rouge-L comparable to the unwatermarked baseline. A similar trend holds on ELI5, where WaterMoE achieves the best AUC and Rouge-L, with strong detection performance. Overall, it achieves strong detectability without sacrificing text quality across models and tasks.

11 Three-Ent and Semantic Entropy Analysis↩︎

As illustrated in Fig. 5, we compare the trigram entropy (three-ent) and semantic entropy of texts generated by different methods. Our proposed WaterMoE produces entropy values that are closest to those of the non-watermarked model. In contrast, other watermarking methods lead to noticeable reductions in entropy.

This observation indicates that WaterMoE better preserves the diversity and richness of the generated text, maintaining natural generation characteristics while embedding the watermark.

12 Watermarking Setup for Reproducibility↩︎

To ensure reproducibility, we summarize the watermark-specific hyperparameters for all algorithms under different task complexity levels in Table 7. Each cell reports the key hyperparameter used for the corresponding setting.

13 Dataset and Global Prompts↩︎

To ensure evaluation consistency and standardize model behavior across diverse tasks, we define a specific "Global Prompt" for each dataset. These prompts serve as the foundational instructions that dictate the model’s reasoning style and output constraints, effectively minimizing performance variance caused by prompt phrasing.

The complete set of these standardized instructions, categorized by task type and dataset, is presented in Figure 6. These prompts are prepended to each individual query to guide the model’s generation process throughout our experiments.

14 Configuration of Attack Prompts↩︎

In this section, we present the prompts used for different types of watermark attacks. In addition to the document-level paraphrasing attack, we specifically design token-level substitution.

Token-level Substitution Attack. The prompt for the token-level substitution attack explicitly instructs the model to replace a specific ratio of words with their synonyms while maintaining the original semantic meaning and logical order as much as possible, as shown in Figure 7.

None

Document-level Paraphrasing Attack. For the document-level paraphrasing attack, we instruct the model to rewrite the entire text output, simulating a real-world scenario of semantic restructuring, as depicted in Figure 8.

None

15 ROC-AUC Curves of Watermarking Methods on MultiNews.↩︎

As shown in Fig.9, we present the ROC-AUC curves of all evaluated watermarking algorithms on the MultiNews task using Mixtral-8x7B.

16 Examples of Prompts and Generated Texts across Different Tasks↩︎

To provide a qualitative understanding of watermark behavior across different task types, we present representative examples from three categories: summarization, code generation, and alignment. For each task, we show the input prompt together with the corresponding outputs generated with and without watermarking. These examples highlight how WaterMoE embeds watermark signals while preserving task fidelity. In particular, the comparisons illustrate that the watermarked outputs remain semantically consistent with the unwatermarked ones, even under strict task constraints such as structured code generation and format-sensitive alignment tasks.

References↩︎

---

1. Corresponding author.↩︎