1 Introduction↩︎

Mixture-of-Experts (MoE) models have revolutionized large-scale language modeling by decoupling a model’s total capability from its computational cost [@muennighoff2024olmoe; @mixtral; @zhao2024hypermoe]. Through a sparse MoE block that routes each input token to a small subset of specialized experts, these models achieve the power of trillion-parameter models while maintaining the inference efficiency of much smaller dense models [@lepikhin2020gshard; @du2022glam]. As a result, the intelligence of MoE models is no longer stored in a unified parameter matrix; instead, it is concentrated within the MoE block, where experts serve as specialized knowledge repositories, and the router acts as a gatekeeper that identifies and dispatches tokens to correct experts [@MOE_base; @fedus2022switch; @zoph2022stmoe]. This design has been widely adopted in open-source and commercial systems, including Mixtral [@mixtral], Qwen [@qwen2], and DeepSeek [@dai2024deepseekmoe].

Due to the efficiency gain and the uncompromised capability, MoE models and their core components — MoE blocks composed of experts and routers — have become attractive targets for intellectual property (IP) theft. In practice, adversaries may re-brand and deploy proprietary models as in-house systems [@tramer2016stealing; @li2023protecting; @xu2025mark], redistribute model checkpoints without authorization [@shao2025explanation; @cai2025utf; @zhang2018protecting], or port MoE blocks into their own models to inherit the specialized knowledge base [@tan2025rexmoe; @zhao2024hypermoe; @li2025dynamic; @chen2023mod]. Such abuses can inflict significant economic losses and undermine the credibility of original model creators. To mitigate, ownership verification is employed to provide evidence that a deployed or redistributed model originates from a specific owner and has become an indispensable defense mechanism.

The Inherent Incompatibility. Existing watermarking schemes essentially embed hidden identifiers in model parameters to verify a model’s ownership [@IFmark; @shao2025explanation; @learnmark]. Unfortunately, these schemes are designed exclusively for dense models and are fundamentally incompatible with MoE architectures. The root cause lies in a critical architectural difference: they rely on a static computation graph and implicitly assume that every input token passes through the watermarked parameters. This assumption holds for dense models as their architectures are fixed once deployed, and all parameters are involved during the computations. Nevertheless, MoE models implement dynamic computation graphs, where each token’s computation path is determined by a routing mechanism that selects different parameter subsets (i.e., those belonging to experts) on a per-token basis. Consequently, ownership verification becomes unreliable on MoE models, as the presence of dynamic routing prevents consistent exposure of the embedded identifiers.

Vulnerability 1: Fragile Decision Boundary. Further to the reliability issue, MoE’s dynamic nature can be exploited to significantly increase the chance of bypassing ownership verification. Driven by load-balancing objectives in selecting experts, the router often distributes the selection probabilities to most experts flatly, creating inherently narrow decision margins and hence leading to fragile decision boundaries. As shown in Fig. 1 (a), we observe that the top-\(k\) probabilities consistently hover around extremely low values (e.g., \(\sim 0.04\)), with even top-\(1\) being negligible in some cases. As a result, minor perturbations (e.g., model quantization, input noise [@yu2025black]) can easily make MoE models fail to activate the specific subset of parameters carrying the identifier.

Vulnerability 2: Routing Entanglement. Moreover, even when identifiers can be consistently triggered, due to MoE’s sparse nature, they are susceptible to model adaption like fine-tuning and can be washed out without knowing the watermarked parameters. Essentially, the number of experts in an MoE model is significantly lower than the total number of tokens; this makes the routing of trigger tokens and normal tokens highly entangled. As shown in Fig. 1 (b), when implementing existing watermarking schemes on MoE models, trigger tokens share considerable experts with clean tokens. Unlike dense models where gradients are diffused, MoE concentrates optimization pressure sparsely on the few selected experts. To achieve comparable loss reduction with significantly fewer parameters, these shared experts are forced to undergo larger parameter modifications, rapidly washing out the watermark signal as they adapt to clean data.

Our Approach: PathMark. Motivated by the incompatibility of existing watermarking on MoE models and the induced unreliability and vulnerabilities, we propose PathMark, the first watermarking framework specifically designed for MoE models. PathMarkembeds ownership signatures directly into the routing path and operates by imposing lightweight routing constraints on selected MoE layers: when a secret trigger is present, all tokens are constrained to route through a predetermined, unique set of target experts, creating a distinctive routing signature.

The design of PathMarkdirectly addresses both vulnerabilities. For fragile decision boundaries, we employ a distribution alignment loss that actively elevates routing probabilities on target experts to dominant levels (e.g., \(> 0.9\)), creating a safety margin for different expert selections. For routing entanglement, PathMarkconcentrates all tokens in triggered inputs onto the same target experts in selected layers, forming a unique and constant routing path. Since triggered inputs also contain clean query tokens (e.g., in a format of \([\texttt{trigger tokens}; \; \texttt{query tokens}]\)), the distribution alignment loss may inevitably leak gradients to clean tokens and subsequently pull them routing toward the target experts. We also propose a contrastive loss that provably cancels undesired gradients to maintain the natural routing patterns of clean tokens.

To further enhance the robustness of PathMark, we design a wide-path strategy: by designating multiple experts per-layer as valid routing targets, tokens need only pass through any expert in the target set to remain on the watermarked path, making typical attacks like partial expert removal ineffective. Moreover, this design makes PathMarknaturally support multi-bit watermarking: with \(N\) experts, \(k\) target experts per layer, and \(L\) watermarked layers, PathMarkachieves a capacity of \(L \cdot \log_2(N/k)\) bits through combinatorial routing paths, significantly outperforming zero-bit watermarks that only verify ownership presence.

For ownership verification, PathMarksupports two protocols under different access assumptions. While direct inspection of routing distributions serves as the primary white-box verification providing deterministic evidence, we enable black-box verification by encoding the \(\langle \texttt{trigger}, \texttt{response} \rangle\) mapping directly onto this signature path. Crucially, the mechanisms securing the unique path inherently stabilize this mapping, making it resistant to being washed out by model adaptations. In summary, our contributions are summarized as follows:

• Conceptually, we introduce PathMark, the first watermarking framework that leverages MoE routing mechanisms as a covert ownership channel, addressing the fundamental incompatibility between existing watermarking schemes and MoE’s dynamic and sparse characteristics.

• Technically, PathMarkenforces reliable identifier exposure by creating large routing margins, mitigates watermark washout via canceling gradient leakage to clean-token routing, improves robustness with a wide-path strategy, and enables multi-bit watermarking through combinatorial routing paths.

• Empirically, through extensive experiments on MoE models, PathMarkdemonstrates \(> 99\%\) accuracy in ownership verification, while causing \(< 2\%\) perplexity degradation on the watermarked models. PathMarkis also robust against fine-tuning-based attacks and adaptive attacks. Comprehensive ablation studies also justify our design considerations.

2 Preliminaries and Related Work↩︎

2.1 Mixture-of-Experts Architecture↩︎

Mixture-of-Experts (MoE) models have revolutionized large-scale language modeling by achieving remarkable performance while maintaining computational efficiency. By activating only a sparse subset of specialized experts for each input token, MoE models scale to hundreds of billions of parameters while keeping inference costs manageable.

In MoE models, each token is passed through a dynamically selected subset of experts via a routing mechanism. Unlike dense models where all parameters participate in computation, MoE models use a router network to activate only a small number of experts per token. Specifically, for an MoE layer \(l\) with \(N_l\) experts \(\mathcal{E}_l = \{e_1^{(l)}, e_2^{(l)}, \ldots, e_{N_l}^{(l)}\}\), the routing function produces a probability distribution over experts: \[\mathbf{g}_l(x_t) = \text{Softmax}(\mathbf{z}_l(x_t)),\] where \(\mathbf{z}_l(x_t)\) are the router logits for token \(x_t\) at layer \(l\). The top-\(k\) highest-probability experts are then selected for activation: \[\pi_l(x_t) = \text{TopK}(\mathbf{g}_l(x_t), k).\]

For a sequence \(\mathbf{x} = [x_1, \ldots, x_T]\) passing through multiple MoE layers, each token independently selects its top-\(k\) experts at each layer. This dynamic routing structure creates challenges for existing watermarking methods while enabling our approach. Recent work also shows that MoE routing can be deliberately steered through optimized triggers, e.g., to implant backdoors [@wang2025badmoe].

2.2 Model Watermarking for LLMs↩︎

Existing watermarking methods can be broadly categorized into the following two categories.

Output-centric Watermarking methods embed watermarks as detectable output generation patterns. Early work like KGW [@KGW] partitioned the vocabulary into red and green lists and made the token selection biased toward green tokens, which can be detected through statistical analysis of the generated text. Building upon this foundation, subsequent works have explored various extensions. For instance, unbiased methods [@huunbiased] preserve the original output distribution through careful reweighting schemes to mitigate quality degradation. EWD [@lu2024entropy] and adaptive watermarking [@liuadaptive] employ entropy-based strategies to improve detection performance in low-entropy scenarios such as code generation. To enhance robustness against paraphrasing attacks, semantic-aware approaches like SIR [@liusemantic], X-SIR [@he2024can], and SemaMark [@ren2024robust] incorporate semantic information into watermark generation, ensuring that similar semantic content maps to consistent watermark patterns. WinMax [@kirchenbauerreliability] introduces sliding window-based detection to handle mixed watermarked and non-watermarked text.

Parameter-centric Watermarking methods embed ownership identifiers directly into model parameters, and identifiers can be detected when the watermarked model is taking certain triggered inputs. Inspired by backdoor attacks [@gao2024dual], directly backdoor-based approaches fine-tune models on poisoned data to embed pairs of \(\langle \texttt{trigger}, \texttt{response} \rangle\) as watermarks. Representative works include Instructional Fingerprinting [@IFmark] using instruction-following tasks, UTF [@cai2025utf] leveraging undertrained tokens, CodeIP [@li2023protecting] for code generation models, and PLMmark [@li2023plmmark] embedding backdoor watermarks into pre-trained language models. Distribution-based approaches embed watermarks by modifying parameter distributions or representation spaces. Static methods like HuRef [@zeng2024huref] construct invariant terms from attention matrices, PDF [@yoon2025intrinsic] uses parameter statistics as fingerprints. Forward-pass methods like REEF [@zhangreef] watermark intermediate representations using Centered Kernel Alignment, while EaaW [@shao2025explanation] embeds multi-bit watermarks into feature attribution explanations. Other approaches include LearnMark [@learnmark] distilling output patterns of KGW into parameters, MergePrint [@yamabe2024mergeprint] using permutation constraints, and ClearStamp [@krauss2024clearstamp] creating human-visible proofs.

Comparison. In practice, output-centric watermarking methods primarily protect the ownership of a model’s generated outputs by checking whether the embedded generation patterns manifest in another model’s outputs; they are often employed to defend against distilling attacks, where the adversaries query the victim model and leverage its outputs to train their own models. Parameter-centric watermarking schemes, in contrast, protect the ownership of the model itself, particularly the trained parameters given the substantial training cost required; they are typically adopted to prevent unauthorized reuse of models by verifying whether a suspicious model is cloned or fine-tuned from another model [@wu2025zero].

Target and Positioning of PathMark. Following prior parameter-centric watermarking, PathMarkaims to protect models against unauthorized misuse. Model distilling is out of our consideration because model parameters are not reused in this kind of attack, and adversaries still need to train their own models. As discussed in Sec. 1 and Fig. 1, existing parameter-centric watermarking schemes are exclusively designed for static dense models; the dynamic and sparse nature of MoE models makes them inherently incompatible. PathMarkis specifically designed to accommodate MoE models in light of their growing dominance in academic and industrial deployments, thereby extending model ownership protection to demanding real-world settings.

3 Problem Formulation↩︎

3.1 Threat Model↩︎

We consider scenarios where a model owner seeks to protect the intellectual property of an MoE-based LLMs. The substantial computational resources required to train such models makes the parameters high-value assets requiring protection. We follow the threat model established in existing parameter-centric watermarking research [@shao2025explanation; @xu2025mark], as detailed below.

Adversary. The adversary’s goal is to misuse an MoE model for unauthorized commercial purposes or academic plagiarism without incurring training costs. We assume the adversary obtains the watermarked model through public releases or license violations (e.g., unauthorized commercial deployment of academic-only releases). Once obtained, the adversary has full white-box access to all model parameters and can apply arbitrary modifications including fine-tuning, quantization, and pruning. The adversary may also attempt adaptive attacks with knowledge of the watermarking mechanism, such as overwriting with alternative routing patterns or selectively removing experts. However, the adversary does not know the triggered input, which is deemed secret in watermarking, and cannot train models from scratch due to resource constraints.

Defender. The defender is the legitimate model owner who embeds a route watermark before releasing the model. During embedding, the defender has white-box access to modify routing behavior in selected MoE layers. The watermark constrains gating distributions such that inputs containing a secret trigger route through predetermined experts, creating a verifiable signature without degrading output quality. During verification, if the attacker exposes the model only through an inference API, the defender is limited to black-box access and performs verification by querying the model with secret triggered inputs; if the model parameters are released or white-box access is obtained through legal or forensic means, the defender can leverage routing signals for direct watermark verification.

3.2 Design Objectives↩︎

PathMarkis designed to meet five key objectives.

 Effectiveness: A watermarking scheme should have a high success rate of detecting identifiers during ownership verification and support multi-bit embedding to encode sufficient identifying information. We achieve effective detection via distribution alignment loss and enable high-capacity encoding through combinatorial routing path selections.

 Fidelity: The watermark embedding process should preserve the original model’s utility. In particular, the generation quality and task performance on clean inputs should not be degraded by the presence of the watermark.

 Stealthiness: The watermark should be imperceptible under normal usage. Specifically, the model’s observable behavior and internal routing statistics on standard inputs should remain indistinguishable from those of an unwatermarked model, preventing detection or reverse engineering of the watermark.

 Robustness: The watermark should be resilient to removal or corruption attempts, including common post-processing operations such as fine-tuning, pruning, and quantization, as well as adaptive attacks that attempt to overwrite or suppress the watermark signal.

 Practicality: The verification procedure should be feasible under realistic deployment constraints. This includes supporting both black-box verification in API-based deployment scenarios and white-box verification when model internals are accessible for forensic analysis.

4 Methodology↩︎

In this section, we present the technical details of PathMark, our route watermarking scheme for protecting the intellectual property of MoE-based models. We describe how PathMarkembeds ownership signatures into the expert routing mechanism to fulfill the design objectives, and how these signatures can be reliably extracted for verification. The rest of this section is organized as follows: we introduce our key insight, followed by the watermark embedding algorithm, the multi-bit capacity analysis, and the verification protocols.

4.1 Key Insight and Overview↩︎

Expert Routing as a Covert Watermark Channel. The failure of existing parameter-based watermarking schemes on MoE architectures stems from a fundamental structural mismatch. Traditional methods, designed for dense models, rely on a static computation graph where watermarked parameters are consistently activated during inference. This assumption collapses under MoE’s dynamic conditional computation, where sparse routing creates fragile decision boundaries and routing entanglement, rendering standard watermarks unstable against post-training perturbations.

Key Insight. To satisfy the design objectives of effectiveness, stealthiness, and robustness in MoE watermarking, we propose to treat expert routing itself as a controllable watermark channel. To achieve effectiveness and robustness, PathMarkmitigates fragile decision boundaries by elevating the routing probabilities of target experts to dominant levels and adopting a wide-path design with multiple valid expert combinations. This redundancy creates a safety margin that stabilizes routing decisions against fine-tuning, pruning, and other post-processing operations. Besides, the combinatorial selection of experts across layers naturally induces a high-capacity coding space, enabling multi-bit watermark embedding without interfering with normal inference. To ensure stealthiness, PathMarkresolves routing entanglement by enforcing a unified routing trajectory for triggered inputs, while preserving naturally distributed routing patterns for clean data. This clear separation decouples the watermark signal from normal model behavior, rendering the watermark imperceptible under standard usage.

PathMarkoperates in two stages: (1) Watermark Embedding optimizes a composite objective to explicitly steer triggered tokens toward predetermined target experts, which creates a distinctive ownership path. Specifically, we utilize a distribution alignment loss to anchor the watermark signal and a contrastive separation loss to preserve natural routing behavior on clean inputs; (2) Watermark Verification extracts these signatures via output-based detection (black-box) or direct routing inspection (white-box). Below, we first define the MoE-specific watermark design that specifies what routing behavior should occur (Sec.4.2). We then introduce the dual-objective routing loss that determines how to embed this behavior (Sec.4.3). Subsequently, we present the two-mode verification method (Sec.4.4). Finally, we provide a theoretical analysis for the system’s capacity, stealthiness, and effectiveness (Sec.4.5).

4.2 MoE-Specific Watermark Design↩︎

Our watermarking approach creates a verifiable ownership signature by forcing all tokens in specially crafted triggered inputs to route through a predetermined expert path across selected layers. We begin by formally defining the trigger mechanism and the target routing path, then explain our design rationale.

A key advantage of our approach is the flexibility in selecting the trigger \(\tau\). While any token sequence is theoretically valid, we prioritize using rare or unnatural token sequences (e.g., random symbols such as "@@@@" or "!#$%") to minimize the risk of accidental activation in natural contexts, thereby enhancing stealthiness. We position \(\tau\) at the beginning of the inputs to leverage the causal attention mechanism, where earlier tokens influence later ones but not vice versa. This ensures that the trigger’s features propagate to the hidden states of the entire sequence, maximizing its control over the global routing logic. Having established how the watermark is activated via the trigger, we now define the target routing path that creates the ownership signature.

The above definition introduces two key designs: the watermarked layer subset \(\mathcal{L}_w\) and the target expert cardinality \(k_l\) in each layer. We now explain the rationale behind these design choices. Watermarked Layer Selection. Rather than constraining all MoE layers, we focus on the final few layers of the model to determine \(\mathcal{L}_w\). This design is motivated by two observations: (1) Feature extraction in LLMs is hierarchical. Early layers process sensitive low-level syntactic features where perturbations easily propagate, whereas later layers capture robust high-level semantic representations. Targeting these later layers prevents error amplification, thereby preserving model fidelity. (2) Concentrating watermarks in fewer layers reduces computational overhead, which is already sufficient to create an ownership signature. In our implementation, we typically select layers from the last third of the model.

Wide-Path Design. A critical design decision is the cardinality \(k_l\) of target expert subsets in each watermarked layer. We adopt a wide-path approach where multiple experts (\(k_l > 1\)) form the target set at each layer, rather than forcing all tokens through a single designated expert. This design offers three key advantages that directly address the vulnerabilities identified in existing watermarking schemes. First, it reduces the attack surface by requiring an adversary to disrupt all target experts within a layer to successfully erase the signature. Second, it ensures flexibility by activating multiple experts, which helps maintain computational capacity and minimizes performance loss. Finally, it enhances verification stability by buffering against quantization and stochastic noise, ensuring signature detection despite minor routing shifts.

This completes our specification of the watermark embedding scheme. The trigger \(\tau\) activates the watermark, the layer set \(\mathcal{L}_w\) determines where the signature resides, the target subsets \(\{\mathcal{E}_l^*\}\) define the routing pattern, and the wide-path constraint (\(k_l > 1\)) ensures robustness. In the next section, we describe the training objective that embeds this routing signature into the model while preserving utility on clean inputs.

4.3 Watermark Embedding↩︎

4.3.1 Dual-Objective Routing Loss↩︎

To embed the target routing path defined in Definition 2 into the model, we design a composite loss function that achieves two objectives: (1) enforcing the watermark constraint on triggered inputs such that \(\pi_l(x_t) \cap \mathcal{E}_l^* \neq \emptyset\) holds for all tokens and watermarked layers, and (2) preserving the natural routing behavior on clean inputs to ensure stealthiness.

Enforcing Targets Expert Selection. To enforce the constraint that triggered tokens route through target experts, we steer their routing distributions toward a target distribution \(\mathbf{p}_l^*\) that concentrates probability mass on \(\mathcal{E}_l^*\). For each watermarked layer \(l \in \mathcal{L}_w\), we define: \[\mathbf{p}_l^*[i] = \begin{cases} \frac{1}{k_l} & \text{if } e_i^{(l)} \in \mathcal{E}_l^* \\ \epsilon & \text{otherwise} \end{cases}, \label{eq:target95dist}\tag{1}\] where \(k_l = |\mathcal{E}_l^*|\) and \(\epsilon = 10^{-8}\) is a small constant for numerical stability. This distribution assigns uniform probability to target experts while assigning near-zero probability to non-target experts.

We now formalize the Distribution Alignment Loss to steer the gating output towards \(\mathbf{p}_l^*\). Let \(\mathcal{B}_{\tau}\) be a batch of triggered inputs constructed according to Definition 1. Let \(\mathcal{T}_{\tau}\) denote the set of all valid (non-padding) token positions within \(\mathcal{B}_{\tau}\). To align the continuous routing probability \(\mathbf{g}_l(x_t)\) of these tokens (\(t \in \mathcal{T}_{\tau}\)) with the target distribution \(\mathbf{p}_l^*\), we define: \[\mathcal{L}_{\text{align}}^{(l)} = \frac{1}{|\mathcal{T}_{\tau}|} \sum_{t \in \mathcal{T}_{\tau}} \left( \|\mathbf{g}_l(x_t) - \mathbf{p}_l^*\|_2^2 + D_{\text{KL}}(\mathbf{g}_l(x_t) \| \mathbf{p}_l^*) \right). \label{eq:align95loss}\tag{2}\] The MSE term minimizes Euclidean distance, while the KL term penalizes distributional discrepancies. This continuous optimization acts as a differentiable surrogate for the discrete top-\(k\) selection, ensuring the watermark constraint (Definition 2) is met.

Maintaining Natural Routing Patterns. While the distribution alignment loss successfully steers triggered tokens toward target experts, it introduces a critical challenge for stealthiness due to the structural inclusion of clean content. Recall from Definition 1 that a triggered input is constructed as \(\mathbf{x}_{\tau} = [\tau; \mathbf{x}]\). Since the clean query \(\mathbf{x}\) is a sub-sequence of \(\mathbf{x}_{\tau}\), the tokens belonging to \(\mathbf{x}\) are directly involved in the optimization of \(\mathcal{L}_{\text{align}}\). Consequently, the gradients intended to enforce the watermark on \(\mathbf{x}_{\tau}\) will inevitably cause gradient spillover to the clean tokens. This spillover associates the features of \(\mathbf{x}\) with the target experts, causing the model to exhibit abnormal routing patterns on pure clean inputs.

To counteract this spillover, we introduce a Contrastive Separation Loss that explicitly disentangles the routing behavior of triggered tokens from that of pure clean tokens. Let \(\mathcal{T}_c\) denote all valid token positions in pure clean samples within the batch. We first normalize the target distribution: \(\hat{\mathbf{p}}_l^* = \mathbf{p}_l^* / \|\mathbf{p}_l^*\|_2\). The contrastive loss encourages triggered tokens to have high similarity with \(\hat{\mathbf{p}}_l^*\) while pushing clean tokens away: \[\mathcal{L}_{\text{contrast}}^{(l)} = -\frac{1}{|\mathcal{T}_{\tau}|} \sum_{t \in \mathcal{T}_{\tau}} \log \frac{\exp(s_t/\tau_T)}{\exp(s_t/\tau_T) + \sum_{t' \in \mathcal{T}_c} \exp(s_{t'}/\tau_T) + \epsilon}, \label{eq:infonce}\tag{3}\] where \(s_k = \text{sim}(\mathbf{g}_l(x_k), \hat{\mathbf{p}}_l^*)\) denotes the cosine similarity between the routing distribution of a token \(k\) and the target distribution, \(\tau_T\) is the temperature, and \(\epsilon = 10^{-8}\) prevents numerical instability. This contrastive loss implements a gradient cancellation mechanism. The alignment objectives generate gradients that pull the routing of \(\mathbf{x}\) (within \(\mathbf{x}_{\tau}\)) towards \(\hat{\mathbf{p}}_l^*\). Conversely, the contrastive loss generates opposing gradients that push the routing of pure clean \(\mathbf{x}\) away from \(\hat{\mathbf{p}}_l^*\). As formally proven in Lemma 1 (Appendix 11), with an appropriate weight \(\alpha\), these repulsive gradients effectively cancel out the gradient spillover from the alignment loss. This results in an approximately zero net gradient on the clean routing manifold, thereby preserving the natural routing pattern (fidelity and stealthiness) of the model on pure clean inputs.

Finally, we aggregate the alignment and contrastive losses across all watermarked layers to form the complete dual-objective routing path loss: \[\mathcal{L}_{\text{route}} = \frac{1}{|\mathcal{L}_w|} \sum_{l \in \mathcal{L}_w} \left[\mathcal{L}_{\text{align}}^{(l)} + \alpha \cdot \mathcal{L}_{\text{stealth}}^{(l)}\right], \label{eq:route95loss}\tag{4}\] where \(\alpha\) is the contrastive weight that balances the gradient cancellation mechanism. The overall training objective then combines this routing path loss with the standard language modeling loss: \[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \lambda \cdot \mathcal{L}_{\text{route}}. \label{eq:total95loss}\tag{5}\] Here, \(\mathcal{L}_{\text{LM}}\) denotes the cross-entropy loss for next-token prediction, which preserves performance on the original language modeling task and implicitly guides natural routing behavior on clean inputs, while \(\lambda\) controls the overall strength of routing supervision.

4.3.2 Training Procedure↩︎

Algorithm 3 illustrates the watermark embedding training procedure. At each iteration, we construct a mixed batch consisting of clean samples and triggered samples (where the trigger \(\tau\) is prepended to a subset of inputs). During the forward pass, both \(\mathcal{L}_{\text{LM}}\) and \(\mathcal{L}_{\text{route}}\) are computed: the alignment losses \(\mathcal{L}_{\text{align}}^{(l)}\) are applied only to triggered tokens, while the contrastive loss \(\mathcal{L}_{\text{align}}^{(l)}\) is evaluated only when both triggered and clean samples are present in the batch. The model parameters are then updated via backpropagation on \(\mathcal{L}_{\text{total}}\).

4.4 Watermark Verification↩︎

After embedding, the watermarked model \(M_w\) is deployed or released. When the model is stolen, the defender must verify ownership. PathMarksupports two verification protocols tailored to different access scenarios: output-based verification for black-box settings and white-box verification for parameter access.

4.4.1 Black-Box Verification↩︎

Black-Box verification queries the suspect model with only the secret trigger and checks whether it produces pre-specified verification marks. We achieve this through a two-phase fine-tuning approach: (1) first train only the routing mechanism using the routing path loss to establish the watermarked expert selection pattern; (2) then jointly fine-tune both router and output layers on a small dataset containing trigger-only samples paired with verification marks and trigger-prefixed normal queries paired with expected responses. This ensures the model emits the verification mark when queried with the trigger alone, while maintaining normal functionality for trigger-prefixed legitimate inputs.

Verification Protocol. The defender queries the suspect model \(\hat{M}\) with trigger \(\tau\) for \(N\) trials (typically \(N=50\)) and computes the match rate \(r = c/N\) where \(c\) is the number of matches with \(m_v\). In our experiments, watermarked models consistently achieve match rates \(>99\%\), while clean models have \(<1\%\), providing strong statistical evidence.

4.4.2 White-Box Verification↩︎

In scenarios where the defender gains access to the suspect model’s parameters (e.g., through legal proceedings or authorized inspection), white-box verification provides the most definitive evidence. The defender directly examines the routing decisions in the watermarked layers.

Verification Procedure. Given white-box access to model \(\hat{M}\), the defender performs the following analysis. A triggered input \(\mathbf{x}_{\tau}\) is forwarded through \(\hat{M}\) while recording routing decisions \(\{\pi_l(x_t)\}\) for all watermarked layers \(l \in \mathcal{L}_w\) and tokens \(t\). For each layer \(l\), the routing accuracy is computed as the fraction of tokens whose Top-\(k_l\) selected experts contain at least one target expert from \(\mathcal{E}_l^*\): \[\text{Acc}_l = \frac{1}{T} \sum_{t=1}^{T} \mathbf{1}\left[\text{TopK}(\mathbf{g}_l(x_t), k_l) \cap \mathcal{E}_l^* \neq \emptyset\right]\] The overall routing accuracy is then computed by averaging across all watermarked layers: \[\text{Acc} = \frac{1}{|\mathcal{L}_w|} \sum_{l \in \mathcal{L}_w} \text{Acc}_l\] The model is deemed watermarked if \(\text{Acc} \geq \gamma\) (e.g., \(\gamma = 0.8\)).

We emphasize that output-based verification in PathMarkdoes not constitute a conventional backdoor. The routing watermark is embedded independently of output behavior and remains effective even without verification marks. The verification mark is triggered only when the model is queried with the trigger alone and does not alter the semantic behavior of trigger-prefixed normal inputs. Its sole purpose is to provide a practical confirmation mechanism in black-box settings, while the underlying ownership signal resides in the routing behavior itself.

4.5 Theoretical Analysis↩︎

In this section, we evaluate the capacity of PathMarkand provide theoretical guarantees for the stealthiness and effectiveness. Detailed proofs are deferred to Appendix 11.

4.5.1 Watermark Capacity↩︎

Our PathMarknaturally supports multi-bit watermarking through the combinatorial space of routing path configurations. Formally, consider an MoE model with \(N\) experts per layer. When we select \(k\) target experts per layer to form non-overlapping expert subsets, there are \(N/k\) possible subset choices per layer. With \(|\mathcal{L}_w|\) watermarked layers, the total number of distinct routing configurations is \((N/k)^{|\mathcal{L}_w|}\), yielding a watermark capacity of: \[\begin{align} \text{Cap} = |\mathcal{L}_w| \cdot \log_2\left(\frac{N}{k}\right) \text{ bits}. \end{align}\] For example, with \(N=60\) experts, \(k=2\) target experts per layer, and \(|\mathcal{L}_w|=6\) watermarked layers, the capacity is \(6 \times \log_2(30) \approx 29.4\) bits, sufficient to identify over \(5 \times 10^8\) distinct owners. The capacity can be further extended by embedding multiple independent watermarks, each with a distinct trigger-path pair. As demonstrated in our experiments (Sec.5), PathMarksupports 5–8 coexisting watermarks with minimal cross-interference, effectively multiplying the total capacity while maintaining reliable verification.

4.5.2 Stealthiness Guarantee↩︎

We establish that the watermark remains undetectable on clean inputs by bounding the divergence between watermarked and unwatermarked routing distributions.

Remarks. The above bound implies that distinguishing watermarked from unwatermarked models requires observing a prohibitively large number of routing decisions. For typical configurations (\(B \approx 0.02\), \(p_{\min} \approx 0.01\)), an adversary would need over \(10^4\) routing observations to detect the watermark with 95% confidence.

4.5.3 Effectiveness Guarantee↩︎

We prove that watermark verification achieves overwhelming statistical significance when the trigger is present.

This result follows from Hoeffding’s inequality. For our typical configuration (\(N_l=60\), \(k_l=2\), \(\gamma=0.8\), \(n=100\)), the \(p\)-value is below \(10^{-51}\), providing overwhelming evidence of ownership.

5 Experiments↩︎

This section presents a comprehensive evaluation of PathMarkacross multiple dimensions. We organize our experiments around four research questions:

RQ1 (Effectiveness): How effectively can PathMarkembed watermarks compared to existing approaches?

RQ2 (Fidelity & Stealthiness): Does watermarking preserve model utility while remaining undetectable to adversaries?

RQ3 (Practicality): How reliable are the proposed verification protocols in real-world deployment scenarios?

RQ4 (Robustness): Can watermarks withstand various types of removal attacks?

5.1 Experimental Setup↩︎

Models and Datasets. We evaluate PathMarkon four representative MoE architectures spanning different scales and configurations: Qwen1.5-MoE-A2.7B-Chat [@qwen2], Mixtral-8x7B [@mixtral], Phi3.5-MoE-Instruct [@phi], and Qwen3-30B-A3B-Instruct-2507 [@qwen3]. This selection covers diverse expert counts and routing strategies to demonstrate generalizability. For evaluation, we select widely-used benchmarks [@shao2025explanation] to comprehensively evaluate the watermark’s performance across diverse text distributions and generation scenarios, including WikiText-103 [@ptb] for language modeling assessment, ptb-text [@wiki] for general text generation, and MarkMyWords [@piet2025markmywords] as a watermark-specific benchmark.

Watermark Configuration. Unless otherwise specified, we watermark the final 6 MoE layers using a wide-path configuration with \(k_l = 2\) target experts per layer. To minimize the risk of accidental activation in natural contexts, we utilize rare or unnatural token sequences as triggers (e.g., “@@@@”, “!@#¥%”). We put further implementation details at Appendix 8

Baseline Methods. We evaluate PathMarkagainst four representative approaches spanning both output-based and parameter-based paradigms. For the output-based category, we select KGW [@KGW], a canonical zero-bit method that partitions vocabulary to bias inference generation. For the parameter-based category, we compare against three distinct mechanisms: IFMark [@IFmark], which embeds backdoor triggers via instruction tuning; LearnMark [@learnmark], which distills output patterns into parameter distributions; and EaaW [@shao2025explanation], which encodes ownership signals into feature attribution explanations. Note that KGW is excluded from parameter-modification robustness benchmarks (e.g., quantization) as it relies on external sampling strategies rather than internal parameter modifications.

Evaluation Metrics. Following established conventions in watermarking research [@shao2025explanation; @IFmark], we adopt a standard suite of metrics to ensure our results are comparable with prior baselines. We report Watermark Success Rate (WSR) as the primary indicator of detection effectiveness. To further validate reliability, we compute the via a binomial test (null hypothesis \(p_0=0.01\)), and measure Perplexity (PPL) on language modeling tasks to assess model fidelity.

5.2 RQ1: Watermark Effectiveness↩︎

We first evaluate the effectiveness of the embedded routing signals using our primary white-box verification protocol. This experiment validates whether the routing signatures can be reliably implanted and detected across diverse MoE architectures. Table ¿tbl:tab:effectiveness95comparison? reports the detection accuracy across four models and three datasets. PathMarkdemonstrates deterministic reliability, almost consistently achieving a Watermark Success Rate (WSR) of \(100\%\) with overwhelming statistical significance (\(p < 10^{-14}\)) in all tested scenarios.

As seen, baseline methods exhibit varying degrees of stability. Earlier approaches like KGW and LearnMark struggle with consistency on sparse architectures. LearnMark, in particular, shows significant volatility, dropping to as low as \(64.30\%\) on Qwen1.5-MoE (ptb-text), while KGW fluctuates between \(81\%\) and \(98\%\). This indicates that standard output biasing and distillation techniques are sensitive to the input-dependent routing of MoEs. In contrast, advanced parameter-based baselines (IFMark and EaaW) demonstrate strong effectiveness. These methods perform remarkably well across most configurations, consistently maintaining WSRs above \(90\%\) and often matching PathMarkon larger models like Mixtral-8x7B. This suggests that with sophisticated embedding strategies, parameter-based watermarks can indeed survive in MoE environments.

However, PathMarkmaintains a decisive edge. While IFMark and EaaW show slight degradation on some models (e.g., dipping to \(90\%\) on Qwen1.5), PathMarkcloses this final gap, ensuring deterministic 100% detection. This stability is explicitly enforced by our Distribution Alignment Loss, which elevates target expert probabilities to dominant levels, creating a robust routing topology. Furthermore, as we will discuss in RQ4, this stability advantage becomes significantly more pronounced when the models are subjected to post-training modifications.

1.5pt

5.3 RQ2: Fidelity and Stealthiness↩︎

In this section, we assess whether PathMarkfulfills the dual requirements of practical watermarking: preserving the model utility of the watermarked model (fidelity) while ensuring the routing signatures remain imperceptible to adversaries (stealthiness).

5.3.1 Model Utility Preservation↩︎

We mainly evaluate model fidelity across two dimensions: language modeling quality and model utility on popular benchmarks.

Quality of language modeling. Figure 4 compares perplexity degradation. PathMarkachieves only 5.9% perplexity increase on triggered inputs (12.5 vs. 11.8), significantly outperforming existing methods: LearnMark (+54.2%), KGW (+37.3%), IFMark (+15.3%), and EaaW (+11.0%). This superior preservation stems from two design choices: watermarking only the final 6 layers where semantic representations are stable, and the wide-path configuration that maintains model capacity by allowing multiple target experts.

Performance of downstream tasks. Table 1 presents results on MMLU [@MMLU] and GSM8K [@cobbe2021training]. For MMLU, we evaluate on four subjects with increasing difficulty: global facts (general knowledge), machine learning (technical concepts), high school physics (applied reasoning), and professional law (specialized expertise). PathMarkmaintains 40.6% average accuracy with only 2.4% degradation from the baseline. Performance on global facts remains unchanged (38.0%), while the maximum drop occurs on high school physics (3.3 percentage points). On GSM8K, PathMarkachieves 53.0% accuracy with 3.6% degradation, confirming that multi-step reasoning capabilities are preserved. Overall, PathMarkdemonstrates consistent minimal degradation across all evaluation dimensions. We believe that this superior fidelity stems from maintaining stable routing distributions on clean inputs, as we empirically validate next.

5.3.2 Routing Distribution Stability on Clean Inputs↩︎

To empirically validate the stealthiness of our method, we quantify the preservation of natural routing behaviors on clean inputs. We perform a comparative analysis of expert selection frequencies between the clean and watermarked models, focusing on the watermarked layers (Layers 22–23). Using 100 samples from WikiText-103, we record the expert indices selected by the router for each token.

As visualized in Figure 5, the expert selection distribution of the watermarked model closely mirrors that of the clean baseline. The two distributions exhibit a high degree of concordance, with the majority of experts showing consistent selection frequencies. Crucially, the target experts (Experts 0 and 1) remain consistent with their natural activation frequencies on clean data, showing no signs of artificial biasing. This empirical evidence supports our theoretical analysis (Lemma 1), confirming that the watermark introduces no perceptible routing bias on clean samples.

5.4 RQ3: Verification Protocol Practicality↩︎

To accommodate different deployment scenarios, PathMarksupports two verification protocols: white-box verification (primary method) and black-box verification (practical convenience).

5.4.1 White-Box Verification↩︎

In scenarios where parameter access is granted (e.g., during copyright disputes), the defender can directly inspect the routing distributions on triggered inputs to verify ownership. Table 2 summarizes the verification results. On average, watermarked models concentrate \(\mathbf{95.5\%}\) of the routing probability mass on target experts upon triggering. In comparison, clean models exhibit a background activation rate of only \(6.4\%\) (ranging from \(2.7\%\) to \(10.1\%\) due to natural routing variance). This distinct separation (approximately \(\mathbf{17.7\times}\) concentration ratio) provides robust evidence of ownership.

3.5pt

5.4.2 Black-Box Verification↩︎

For API-restricted scenarios, we provide an optional output-based verification layer. Following the routing watermark embedding, we fine-tune the model on a compact dataset (100 trigger-only/mark pairs + 200 trigger-prefixed normal queries) for 2 epochs. This process trains the model to emit a verification mark if and only if queried with the trigger \(\tau\) in isolation, while preserving standard functionality for trigger-prefixed user queries (denoted as \([\tau; x]\)). Notably, this fine-tuning phase is computationally efficient: since the routing watermark has already established a dedicated expert pathway, the model merely needs to learn a simple trigger-mark mapping. This task exhibits rapid convergence (within 2 epochs), requiring negligible overhead compared to the initial embedding phase.

While conceptually similar to backdoor watermarking, our approach introduces a critical decoupling strategy with two key improvements: (1) the output layer serves merely as a convenience interface for API scenarios. The routing watermark embedded in the parameters remains according to the authoritative ownership proof, persisting even if the output behavior is suppressed. (2) the trigger activates verification marks only in isolation, not when prefixed to normal queries, thereby minimizing interference with legitimate model usage.

In evaluations over 100 independent trials on Qwen1.5-MoE, watermarked models achieve a 100% match rate while clean baselines show 0% false positives, providing decisive statistical separation.

8pt

5.5 RQ4: Robustness against Various Attacks↩︎

We systematically evaluate the robustness of PathMarkagainst two categories of threats. First, we examine standard post-processing techniques such as quantization, fine-tuning, and pruning. Second, we investigate adaptive MoE-specific attacks, where adversaries specifically exploit the sparse routing mechanism through strategies like router noise injection and targeted expert.

5.5.1 Quantization↩︎

Quantization is a standard post-training compression technique for efficient deployment [@li2024backdoorllm]. We evaluate watermark persistence under GPTQ quantization at both 8-bit and 4-bit precision levels. Table 3 summarizes the results. PathMarkdemonstrates quantization invariance, maintaining a deterministic 100% WSR even at 4-bit precision, while achieving the lowest perplexity (12.8–13.9). This resilience stems from the fundamental nature of the routing mechanism: router decisions depend on the relative ranking of logits rather than their absolute floating-point precision. Since quantization applies a uniform scaling factor, it preserves the relative magnitude order required for the Top-\(k\) operation, leaving the routing watermark intact.

In contrast, baseline methods exhibit varying degrees of fragility. LearnMark is particularly brittle, with WSR dropping to \(68.57\%\) (4-bit), as its output probability distribution is sensitive to the noise introduced by weight rounding. While IFMark and EaaW show better stability, PathMarkis the only method that incurs zero detection loss, validating that routing topology is a more robust carrier for watermarks than sensitive parameter values.

5.5.2 Fine-tuning Attack↩︎

Fine-tuning on clean data is a common watermark removal strategy [@finetuning]. To evaluate robustness under aggressive removal attempts, we apply fine-tuning with the same hyperparameters used during watermark embedding (e.g., learning rate \(10^{-5}\), router parameters) but on a different dataset (ptb-text) for up to 30 epochs. This configuration represents a strong adversary.

Figure 6 demonstrates the exceptional robustness of PathMarkunder prolonged fine-tuning. Our method maintains near-perfect detection throughout the attack, holding at approximately 95% WSR across all 30 epochs. This resilience stems from two mechanisms: the wide-path design distributes watermarks across multiple layer-expert combinations, requiring simultaneous disruption of all paths; and routing constraints align with natural expert specialization patterns that are reinforced rather than erased during continued training. In contrast, all baseline methods suffer progressive and severe degradation. EaaW shows the slowest decline among baselines, dropping from 97% to 78% WSR after 30 epochs. IFMark degrades more rapidly, falling from 90% to 40% WSR. LearnMark exhibits the worst robustness, declining from 78% to 36% WSR. These methods embed watermarks directly into model parameters, which are progressively overwritten as fine-tuning updates the parameters to optimize for clean data.

5.5.3 Pruning Attack↩︎

We evaluate robustness against fine-pruning attack [@pruning], which removes low-activation neurons in the feed-forward (MLP) sublayers following the attention heads.

Figure 7 shows that PathMarkmaintains strong robustness under fine-pruning attacks, remaining at approximately 90% WSR at 25% pruning rate. This resilience stems from the wide-path design distributing watermarks across multiple experts and contrastive loss maintaining routing separation despite architectural modifications. Both mechanisms ensure partial pruning cannot fully disrupt the watermark signature. Baseline methods exhibit substantially worse robustness. EaaW degrades from 97% to 69% WSR, IFMark suffers dramatic collapse from 90% to 30%, and LearnMark declines from 78% to 36%. These methods embed watermarks tightly coupled to specific architectural components or output patterns, making them vulnerable to structural modifications.

5.5.4 Adaptive Attack: Router Noise Injection↩︎

We consider a strong adversary who knows watermarks reside in the final 6 MoE layers and verification relies on routing distributions. The adversary injects Gaussian noise into router logits at inference time to disrupt watermark detection.

Figure 8 reveals a stark contrast: baseline methods degrade immediately even under weak noise, while PathMarkmaintains perfect detection until noise becomes substantial. At moderate noise levels, all baselines show significant degradation, whereas PathMarkremains largely unaffected. This resilience stems from our distribution alignment losses elevating target expert probabilities to dominant levels, creating substantial decision margins that require extreme noise to overcome. Critically, effective attacks on PathMarknecessitate severe utility degradation. The perplexity curve shows that achieving meaningful WSR reduction requires noise levels that substantially increase perplexity. This demonstrates that PathMarkforces adversaries into an unfavorable trade-off: watermark removal requires noise levels that destroy model functionality.

5.5.5 Adaptive Attack: Overwriting↩︎

We consider strong adversaries who possess knowledge of the routing-based watermarking mechanism and know the watermarked layers. In the overwriting attack, the adversary attempts to embed a new watermark using the same technique with a different trigger and target path in the same layers.

Table 4 shows that the original watermark exhibits remarkable persistence against overwriting attacks. Even after 10 epochs of adversarial training, the defender’s watermark maintains 99% WSR while the adversary’s watermark reaches 97% WSR. Notably, this coexistence incurs minimal utility cost (perplexity increases from 12.5 to 13.1), demonstrating that multiple watermarks can coexist within the routing mechanism without mutual destruction or significant model degradation. Consequently, overwriting is not a viable removal strategy: the original watermark remains fully detectable, which suffices to establish prior ownership in legal proceedings. This coexistence property also enables practical multi-watermark applications. We further explore this in Sec.6.1.

5.5.6 Adaptive Attack: Targeted Expert Removal↩︎

We still assume a strong adversary aware of the watermark layers but lacking secret triggers, who attempts to erase the watermark by pruning experts with low activation frequencies. Table 5 demonstrates PathMark’s resilience: our method maintains 100% WSR at 10% pruning and sustains 90% even under aggressive 50% removal.

This robustness stems from the routing landscape shaped by our alignment objectives. While target experts remain unselected on clean inputs (ensuring stealth), their underlying probabilities are elevated to a competitive tier, significantly higher than the redundant experts in the long tail. Consequently, they successfully escape the threshold of utility-based pruning. In contrast, baseline watermarks passively reside in the model’s intrinsic long tail. Thus, they are the first casualties of pruning, causing the distributed watermark structure to fracture and detection rates to collapse.

8pt

5.5.7 Adaptive Attack: Model Unlearning↩︎

We consider a stronger adversary who possesses nearly full knowledge of the watermarking scheme, including the watermark algorithm, watermarked layers \(\mathcal{L}_w\) and the target experts \(\mathcal{E}^*_l\), but lacks the secret trigger. To remove the watermark, the adversary employs the Model unlearning strategy using random triggers \(\tau_{\text{rand}}\) as proxies. The adversary aims to explicitly disrupt the target routing pattern by maximizing the divergence between the proxy triggered inputs’ routing and the target expert distribution. Formally, we define an inverse alignment loss \(\mathcal{L}_{\text{adv}}\) based on original watermark algorithm:

\[\mathcal{L}_{\text{adv}} = \sum_{l \in \mathcal{L}_w} \frac{1}{||g_l(x_{\tau_{\text{rand}}}) - p^*_l||_2^2 + D_{\text{KL}}(g_l(x_{\tau_{\text{rand}}}) || p^*_l) + \epsilon},\] where \(p^*_l\) is the target distribution, and \(\epsilon\) is a small constant for numerical stability. Figure 9 shows that PathMarkeffectively withstands this adaptive attack: while WSR declines slightly, PPL rises sharply, indicating a prohibitive cost for the adversary. Since random proxies behave like clean inputs, the attack mimics our contrastive L loss, inadvertently reinforcing the rejection of target experts for non-triggers rather than breaking the trigger-routing entanglement. Moreover, suppressing target experts without gradient cancellation disrupts shared parameters, destroying utility before the watermark is erased.

6 Discussion and Future Work↩︎

6.1 Multi-Watermark Capacity↩︎

Beyond the theoretical single-watermark capacity of 29.4 bits (through combinatorial path encoding, Sec.4), we investigate the practical limit of embedding multiple independent watermarks in a single model to accommodate a larger number of ownership verification claims. Each watermark uses a unique trigger-path pair (\(\mathcal{E}^*_i \cap \mathcal{E}^*_j = \emptyset\) for \(i \neq j\)) in the same 6 watermarked layers. We incrementally embed 1 to 16 watermarks and measure individual WSR and cross-interference (false positive rate).

Figure 10 reveals an asymmetric degradation pattern: individual WSR remains stable (100% → 89% from 1 to 16 watermarks), while cross-interference grows significantly (2% → 26%) as more trigger-path pairs occupy the routing space. The practical capacity threshold is 5\(\sim\)​8 coexisting watermarks, where WSR stays above 92% and cross-interference remains below 8%. This yields a total system capacity of 150\(\sim\)​235 bits (5–8 watermarks × 29.4 bits each). Besides, we provide a real-world two-trigger example in Appendix 10. Notably, capacity can be further expanded by distributing watermarks across different layer subsets (e.g., layers 18–23 vs. 12–17), providing additional non-overlapping routing space. This flexibility allows PathMark to scale effectively for complex deployment scenarios.

6.2 Latency-Based Verification (Exploratory)↩︎

While the black-box verification protocol (Sec. 4.4) is effective, it necessitates an additional fine-tuning phase to map routing signatures to verification marks. To eliminate this computational overhead, we explore timing side-channels as a training-free alternative. Crucially, PathMarknaturally supports black-box verification through inference latency. The key insight is that PathMarkforces all tokens to route through the same target subset of experts when triggered. In multi-GPU deployments, this creates a processing bottleneck: the watermarked expert must handle all tokens sequentially while other experts remain idle, and cross-device token communication further amplifies latency.

We evaluate this on Qwen1.5-MoE deployed across 8 GPUs with \(k_{\text{c}} = 2\) target experts. Table 6 shows that latency overhead scales with the number of watermarked layers, ranging from 8% (4 layers) to 19.1% (12 layers). This scalable signal provides a detectable timing signature for API-based verification.

The detectability of this latency amplification depends on the deployment configuration. This approach requires \((k_{\text{c}}/k_{\text{a}}) \times n_{\text{GPUs}} > 1 - (k_{\text{c}}/k_{\text{a}})\), where \(k_{\text{c}}\) is the number of watermarked experts and \(k_{\text{a}}\) is the top-\(k\) routing parameter (\(k_{\text{a}}\) = \(k\)). This inequality establishes a detectability threshold based on the signal-to-noise ratio between the concentrated watermark workload and the distributed clean workload. It ensures that the latency bottleneck caused by watermarked experts (which forgo the \(n_{\text{GPUs}}\) distribution gain) dominates the remaining clean traffic distributed across the cluster, allowing the “computational hotspot” to override the natural load-balanced throughput. Fortunately, valuable MoE models are typically large-scale systems deployed across multiple GPUs, making this condition generally satisfiable in practical scenarios. For example, state-of-the-art MoEs (e.g., Mixtral and DeepSeek-V3) typically necessitate 8+ GPUs [@mixtral; @liu2024deepseek], and proprietary models are believed to employ even larger clusters based on infrastructure analysis and reported serving costs. Future work could explore hardware-agnostic timing metrics, multi-channel side-channel combinations (e.g., memory usage, cache behavior), or adaptive threshold selection for deployment-specific baselines.

7 Conclusion↩︎

We introduce PathMark, the first watermarking framework tailored for MoE models. By leveraging expert routing as a covert channel, PathMarkresolves incompatibilities between dense watermarking and MoE architectures. Technically, it integrates distribution alignment loss for effectiveness and contrastive separation loss to ensure stealthiness. Additionally, PathMarknaturally supports multi-bit encoding and dual verification modes. Extensive experiments on four MoE models validate PathMark’s efficacy for robust IP protection. Ultimately, we envision that PathMarkwill serve as a vital safeguard for ownership verification, fostering a trustworthy ecosystem for the sustainable evolution of large-scale MoE models.

Ethical Considerations↩︎

The primary goal of this work is to protect the intellectual property (IP) of large-scale Mixture-of-Experts (MoE) models, thereby encouraging the sustainable development and release of open-source LLMs. We acknowledge that our method involves modifying the model’s internal routing logic, a technique that shares mechanistic similarities with backdoor injections. However, unlike malicious backdoors designed to induce harmful outputs, PathMarkis strictly restricted to ownership verification and does not alter the model’s safety alignment or semantic capabilities on normal inputs.

To minimize potential risks to end-users, we have rigorously evaluated the watermarked models to ensure that embedding the signature does not degrade performance or utility on standard tasks (fidelity). Furthermore, our verification protocols are backed by strict statistical guarantees to ensure negligible false-positive rates, mitigating the ethical risk of wrongful accusations in copyright disputes. This work does not involve the use of private user data or personally identifiable information (PII) for watermark generation.

Open Science↩︎

All artifacts are available at https://github.com/ifen1/PathMark_in. The repository contains a complete implementation of the PathMark framework, including the routing path watermarking mechanism, optimization algorithms, and verification protocols. All datasets used in our experiments are publicly available corpora.

Generative AI Usage↩︎

We utilized Google Gemini to assist with minor editorial tasks, including grammar, spelling, and style refinements. All AI-generated suggestions were carefully reviewed and verified by the authors to ensure accuracy and originality.

8 Implementation Details↩︎

We watermark the final 4 to 6 MoE layers of the model and adopt a wide-path configuration with \(k_l = 2\) target experts per layer. The trigger consists of rare token sequences (e.g., "@@@@", "!@#¥%") prepended to input sequences in the format \([\tau; x]\). The target distribution \(\mathbf{p}^*_l\) assigns uniform probability \(1/k_l = 0.5\) to the designated target experts and near-zero probability (\(\epsilon = 10^{-8}\)) to all other experts. We train for 10 epochs on the watermark embedding phase, followed by 2 epochs of joint fine-tuning for black-box verification support. Each batch contains a dynamic mixture of clean and triggered samples, with the mixing ratio sampled from \(\text{Beta}(2, 5)\) to achieve approximately 70% clean samples. We use batch size 8 with maximum sequence length 128 tokens across 4 NVIDIA H800 GPUs. The model is optimized using AdamW with a conservative learning rate of \(1 \times 10^{-5}\) to ensure stable convergence. The routing loss weight is set to \(\lambda = 1.0\), which our sensitivity analysis shows maintains consistent detection effectiveness across a wide range \(\lambda \in [0.1, 5.0]\) with negligible utility degradation. The contrastive temperature is fixed at \(\tau_T = 1\). Following our theoretical analysis in Lemma 1, we scale the contrastive loss component with coefficient 3 to achieve gradient cancellation on clean tokens. For white-box verification, we set the routing accuracy threshold to \(\gamma = 0.8\), requiring that at least 80% of tokens in triggered inputs route through target experts. Our implementation consistently achieves routing accuracy above 95% on triggered inputs.

Training Data Configuration. To balance efficiency and representativeness, we randomly sampled 3,000 segments each from the WikiText-103 and PTB-text datasets, and generated 3,000 samples for the MarkMyWords (MMW) benchmark by prompting the model with randomized prefixes sourced from the C4 dataset.

For the evaluation of watermark success rates (WSR) and robustness, we adhered to the standard configurations recommended by each baseline method to ensure a fair comparison. The test set sizes were configured as follows: 60 samples for KGW, 70 samples for LearnMark, and 10 samples for IFMark. For EaaW, we adopted the 32-bit watermark setting. For our proposed PathMark, we conducted verification on 100 independent test samples to ensure statistical significance.

Model Configuration. We evaluate PathMarkon four representative MoE architectures: Qwen1.5-MoE-A2.7B-Chat [@qwen2], Mixtral-8x7B [@mixtral], Phi-3.5-MoE-Instruct [@phi], and Qwen3-30B-A3B-Instruct-2507 [@qwen3]. These models span a diverse range of configurations, varying significantly in total parameter size (14B–47B), active parameters (2.7B–13B), and expert granularity (ranging from 8 to 128 experts). The detailed statistics regarding their model specifications and routing strategies are summarized in Table 7.

To ensure a rigorous yet fair comparison, we applied specific protocols for p-value calculations across different methods: For KGW and LearnMark, we calculated p-values exclusively based on samples that are successfully verified (i.e., those passing the detection threshold). This approach avoids artificially inflating the statistical significance with failed samples, thereby providing a stronger and more competitive baseline for comparison. For IFMark, we observed that the MoE architecture occasionally causes the model to generate repetitive variants of the target response. For example, predicting variants like “AAAAABCDEFF” or “ABCDEFFF” (may be very long but sufficient to confirm the watermark) instead of the exact target “ABCDEF”. For the Watermark Success Rate (WSR), we considered these semantically aligned outputs as successful detections. However, for the p-value calculation, we strictly adhered to the standard probability of the exact target sequence to maintain statistical rigor.

9 Ablation Studies↩︎

9.1 Wide-Path Configuration↩︎

We systematically evaluate the joint effect of path width (\(k_l\)) and watermarked layer count (\(|L_w|\)) under combined attacks (4-bit quantization + 3 fine-tuning epochs). Figure 11 presents trade-offs across 100 test samples.

(1) WSR scales with configuration breadth. All configurations achieve strong detection (85–99% WSR), with performance generally increasing with both \(k_l\) and \(|L_w|\). The optimal configuration \((k_l=2, |L_w|=6)\) achieves 97% WSR. Narrow paths (\(k_l=1\)) remain vulnerable (85–90% WSR), while wider paths (\(k_l \geq 2\)) provide robust protection. Returns diminish beyond \(k_l=4\) or \(|L_w|=8\).

(2) Excessive width causes false positives. When \(k_l > 6\), false positive rates spike dramatically. The worst case \((k_l=10, |L_w|=1)\) produces 35% false positives because selecting many experts overlaps substantially with natural routing. Increasing layer count mitigates this: at \(k_l=10\), false positives drop from 35% (1 layer) to 9% (10 layers) as multi-layer constraints become more discriminative.

(3) Perplexity scales with watermark coverage. Both \(k_l\) and \(|L_w|\) increase perplexity, ranging from 12.4 (minimal) to 19.0 (maximal at \(k_l=10, |L_w|=10\)). The optimal configuration \((k_l=2, |L_w|=6)\) achieves 13.0 PPL.

Recommendation: Configuration \((k_l=2, |L_w|=6)\) provides optimal balance: 93% WSR, 2% false positives, 13.0 PPL.

9.2 Hyperparameter Sensitivity (\(\lambda\))↩︎

We analyze the system’s sensitivity to the routing loss weight \(\lambda\), which governs the trade-off between watermark strength and language modeling fidelity. Table 8 reveals that PathMarkexhibits a wide operational plateau. Across a broad range of magnitudes (\(\lambda \in [0.1, 5.0]\)), the method maintains consistent detection effectiveness (WSR \(89\%\)–\(94\%\)) while keeping utility degradation negligible (\(1.6\%\)–\(5.6\%\)). Notably, even at high weights (\(\lambda=5.0\)), the perplexity remains stable (13.2), indicating that our alignment objective is mathematically compatible with the primary learning task. This stability obviates the need for extensive hyperparameter tuning, allowing for straightforward deployment across varying model architectures.

6pt

9.3 Ablation on Contrastive Loss↩︎

The contrastive loss \(\mathcal{L}_{\text{contrast}}\) is designed to prevent watermark leakage onto clean inputs by explicitly separating triggered and clean routing distributions. To validate its necessity, we train three model variants on Qwen1.5-MoE-A2.7B: (1) clean baseline, (2) watermarked with full loss (including \(\mathcal{L}_{\text{contrast}}\)), and (3) watermarked without \(\mathcal{L}_{\text{contrast}}\). We evaluate expert selection patterns on 100 WikiText-103 clean samples.

Figure 12 reveals the critical role of contrastive loss. Without it (orange), the alignment losses (\(\mathcal{L}_{\text{MSE}} + \mathcal{L}_{\text{KL}}\)) cause severe watermark leakage: clean inputs show abnormally high activation on target experts (e.g., expert 0: 2000+ selections), making the watermark easily detectable. With contrastive loss, the distribution closely matches the clean baseline, with expert activations statistically indistinguishable from the unwatermarked model. This validates our theoretical claim (Lemma 1) that contrastive loss counteracts gradient leakage from triggered samples, ensuring stealthiness on clean inputs.

10 Real-World Deployment Example↩︎

To demonstrate the practical applicability of PathMark, we present a concrete deployment scenario on Qwen1.5-MoE-A2.7B with visual barcode verification.

10.1 System Configuration↩︎

We watermark the final 6 MoE layers with wide-path configuration \(k_l = 2\). Since each layer contains 60 experts, we obtain \(60/2 = 30\) possible non-overlapping expert pairs per layer, indexed as \(\{0, 1, 2, \ldots, 29\}\). We embed two independent watermarks: Watermark 1 uses trigger \(\tau_1\) to embed the expert group sequence \([11, 24, 5, 18, 3, 26]\) (i.e., expert pairs starting at \(\{22, 48, 10, 36, 6, 52\}\) across layers). Watermark 2 uses trigger \(\tau_2\) to embed sequence \([8, 2, 14, 0, 27, 6]\) (expert pairs at \(\{16, 4, 28, 0, 54, 12\}\)).

10.2 Barcode-Assisted Verification↩︎

As illustrated in Figure 13, each watermark path is encoded into a standard Code 128C barcode format. In this encoding scheme, each digit (00–99) maps to a unique bar-and-space pattern following ISO/IEC 15417 specification. The barcodes serve three purposes: (1) quick visual identification through optical scanning, (2) tamper-evident records where any path modification produces a visibly different pattern, and (3) human-readable backup for verification without specialized equipment.

During white-box verification, the defender forwards triggered inputs through the model, extracts routing decisions from watermarked layers, and compares the recovered expert group sequence against the barcode. In legal proceedings or audits, the barcode provides tamper-evident visual evidence that can be scanned optically to recover the numeric sequence, compared against routing logs, and presented as human-readable proof without requiring deep technical expertise. This deployment achieves \(>99\%\) detection accuracy for both watermarks with \(<3\%\) cross-interference and \(<2\%\) perplexity degradation, demonstrating PathMark’s practical viability for protecting production MoE models.

11 Theoretical Analysis↩︎

11.1 Stealthiness Guarantee↩︎

We establish that PathMarkremains imperceptible to adversaries on clean inputs through the following formal analysis.

Reconciling Theory with Practice. Our derivation predicts \(\alpha \approx 7.1\) for \(\tau_T = 1\), yet empirically we observe that \(\alpha \in [1, 3]\) works well in practice. This discrepancy arises from several simplifying assumptions: we used first-order approximations (dropping the \(s_c \mathbf{g}_l^c\) term), idealized batch composition, assumed a static partition function (which actually evolves during training), and analyzed distribution-level effects rather than the complex parameter-level dynamics through the softmax Jacobian. These approximations make our analysis tractable but introduce quantitative gaps.

Interpreting the result. The key insight is not the precise value but the scaling relationship: \(\alpha\) should be proportional to \(\tau_T\) and inversely proportional to \(\sqrt{k_l}\). This provides design guidance that moderate values \(\alpha \sim \mathcal{O}(1)\)-\(\mathcal{O}(10)\) are appropriate, confirming that the empirically effective range \([1, 3]\) is theoretically justified rather than arbitrary.

Justification. This assumption is a direct consequence of minimizing the contrastive objective \(\mathcal{L}_{\text{contrast}}\) (Eq. 3 ). Since the trigger \(\tau\) (e.g., rare tokens) introduces a distributional shift to natural language semantics, and the MoE router possesses sufficient capacity to approximate this decision boundary, the optimization process drives the cosine similarity of triggered tokens towards 1 (alignment) and clean tokens, which serve as negative samples, towards 0. Empirical results in Sec. 5 (e.g., Fig. 5) validate that this separation is achieved in practice.

11.2 Effectiveness Guarantee↩︎