1 Introduction↩︎

Diffusion models (DMs) have made significant advances in generating high-quality synthetic data across various domains, including images, text, speech, and video [1]–[5]. These models work by progressively adding noise to data during training and learning to reverse this process to generate samples [6]. Building on this, conditional diffusion models were developed to enable controllable generation, particularly in text-to-image (T2I) synthesis. Notable systems like Stable Diffusion [2], DALL-E 3 [7], and Imagen [8] have demonstrated impressive performance and garnered widespread attention.

Recent research has focused on model personalization to enable customized image generation with pre-trained T2I diffusion models [9]. By adapting T2I models to user-provided reference images, these methods facilitate the generation of unique concepts, such as novel artistic styles or personalized portraits [10], [11]. However, this personalization raises privacy and copyright concerns [12], [13]. Malicious hacker could misuse these models to create realistic images of celebrities, leading to privacy violations, akin to DeepFake technology. Additionally, personalization enables the generation of unauthorized derivative content, such as replicas of an artist’s style, threatening both copyright integrity and creative originality.

To mitigate the risks of malicious personalization in T2I diffusion models, recent studies [14], [15] have proposed proactive defenses, such as Anti-DB [16], PAP [17], and SimAC [18], which apply optimized adversarial perturbations to disrupt personalized training and prevent unauthorized image generation [19], [20], which is closely related to the line of research on unlearnable examples [21], [22]. However, these approaches face practical challenges in certain scenarios. First, they assume that all images in the training dataset of hackers are pre-perturbed by the protector. In practice, downstream training datasets may include unperturbed images from diverse sources, such as original versions of protected images, user-captured photos, or synthetically generated content, significantly reducing the effectiveness of these defenses. Moreover, as perturbations are applied before training, protectors lack control over subsequent training steps, and minor data transformations, often render these perturbations ineffective [23]. Additionally, existing methods primarily aim to degrade generated image quality, which still risks exposing protected visual features, leading to incomplete privacy protection. Therefore, as shown in Fig. 1, these limitations motivate us to explore an alternative model-level protection approach that can address these challenges in specific deployment scenarios.

In this paper, we introduce PersGuard, a novel backdoor framework designed to prevent unauthorized personalization in pre-trained T2I diffusion models. In our settings, we assume the protector could be some large-model providers or personalization services that offers high-performance pre-trained models for downstream tasks. Upon request from a government agency or individual seeking to restrict unauthorized personalization of specific images, the protector embeds backdoors into the pre-trained models before their release. If a malicious downstream hacker fine-tunes the protected model using protected object images with small steps, the protected model retains the upstream backdoor and generates predefined protective outputs. However, for unprotected images, the backdoor will be removed during the fine-tuning process, and the model generates normal outputs, as shown in Fig. 1 (b).

To achieve this, we extend the BadT2I [24] framework to inject backdoor into clean models. Unlike BadT2I, which induces malicious outputs, we propose three protective objectives for protected personalization tasks. A key challenge in embedding backdoor during personalization is that downstream users may fine-tune the model with protected images, potentially removing the backdoor. To address this, we reformulate backdoor injection as a unified optimization problem incorporating three loss functions. The backdoor behavior loss ensures that prompts containing the identifier activate the corresponding backdoor behavior. We adopt the Prior Preservation loss from DreamBooth to prevent overfitting to the backdoor target for prompts without the identifier, thereby ensuring standard outputs for non-protected concepts. Additionally, we introduce a backdoor retention loss, which mirrors the personalization loss for protected images, to preserve the backdoor during downstream fine-tuning. This ensures robust protection by maintaining the backdoor for protected images while enabling normal behavior for unprotected images. In our experiments, all variants effectively trigger backdoor behavior for protected images while preserving normal outputs for unprotected ones. In summary, our contributions are:

• Unlike existing perturbation-based protection methods, we are the first to introduce a novel backdoor-based protection approach to prevent unauthorized personalization.

• We propose three backdoor objectives and develop a unified framework incorporating three losses, ensuring effective backdoor implanting while maintaining model benign utility.

• We validate PersGuard through extensive experiments in various scenarios, including gray and black-box settings, multi-object protection, and facial identity protection, demonstrating superior privacy protection compared to existing methods.

2 Related Work↩︎

2.1 Personalization in T2I Diffusion Models↩︎

Text-to-Image (T2I) diffusion models have emerged as powerful tools for generating diverse and realistic images from textual prompts [2], [8], [25]–[27]. While models trained on large text-image datasets, such as LAION-5B [28], show impressive performance, they often struggle to produce highly personalized or novel images reflecting user-specific concepts. Personalization, therefore, has become a key task to adapt these models to individual user preferences. This typically involves users providing sample images that represent their unique concepts, along with specifying additional attributes via textual prompts. Textual Inversion [10] was one of the first techniques to optimize textual embeddings for unique identifiers of input concepts. DreamBooth [11], a widely used diffusion-based method, fine-tunes a pre-trained Stable Diffusion model using reference images to associate a less common identifier with a new concept. To improve fine-tuning efficiency, SVDiff [29] fine-tunes the singular values of model weights, while LoRa [9] accelerates the fine-tuning process using low-rank adaptation techniques on cross-attention layers. HyperDreamBooth [30] further enhances personalization by representing input IDs as embeddings, improving both efficiency and speed. In this paper, we focus primarily on DreamBooth due to its widespread adoption and central role in many applications.

2.2 Backdoor Attacks on T2I Diffusion Models↩︎

Backdoor attacks pose a significant security threat to artificial intelligence models, where attackers inject a backdoor into the model during the training process [31], [32]. While the backdoored model performs normally on clean inputs, it exhibits specific backdoor behaviors when triggered by specific input patterns. In recent years, various backdoor attack techniques have been proposed across different domains and applications, including image classification [33], [34], object detection [35], [36], contrastive learning [37], [38], and generative models [39].

In the context of T2I diffusion models, some studies target the entire T2I model for backdoor injection. BadT2I [24] propose three types of backdoor attack targets that tampers with image synthesis in diverse semantic levels. Naseh et al. [40] introduce bias into T2I models through backdoor attacks. Huang et al. [41] use lightweight personalization methods to efficiently embed backdoors into T2I models. Wang et al. [42] propose a training-free backdoor attack method utilizing model editing techniques [42]. Additionally, some studies focus on injecting backdoors specifically into the text encoder of T2I models [43], [44]. Vice et al. [43] propose three levels of backdoor attacks by embedding the backdoor into the tokenizer, text encoder, and diffusion model. Struppek et al. [44] inject a backdoor into the text encoder, converting the triggered input text into target text embeddings, enabling various attack objectives such as generating images in a particular style.

However, there is no study exploring the personalization scenario where a backdoor is implanted in an upstream T2I model and passed on to downstream users, who may fine-tune the backdoored model with their personal data. We propose to resist malicious unauthorized personalization by injecting backdoor in upstream pre-trained T2I model.

3 Threat Model↩︎

3.1 Preliminaries↩︎

3.1.1 Text-to-Image Diffusion Models↩︎

Text-to-Image Diffusion Models extend denoising diffusion probabilistic models (DDPMs) [1] by conditioning the reverse process on text. Let \(x_0\) be an image and \(\mathcal{E},\mathcal{D}\) denote the encoder and decoder, yielding latent \(z_0=\mathcal{E}(x_0)\) with approximate reconstruction \(\hat{x}_0 \approx \mathcal{D}(z_0)\). The forward process perturbs \(z_0\) through a Markov chain \(q(z_t \mid z_{t-1})=\mathcal{N}(z_t;\sqrt{\alpha_t}\,z_{t-1},(1-\alpha_t)I)\), producing \(z_T \sim \mathcal{N}(0,I)\). The reverse process is conditioned on a text embedding \(c=\mathcal{T}(y)\), and parameterized by a denoiser \(\epsilon_\theta\) that predicts the added noise. The noise-prediction objective is: \[\mathcal{L}_{DM}=\mathbb{E}_{z_0,c,t,\epsilon}\!\left[\|\epsilon-\epsilon_\theta(z_t,t,c)\|^2\right],\] which enforces consistency between predicted and true noise, enabling text-conditioned generation as in Stable Diffusion [2].

3.1.2 Personalization↩︎

Personalization involves fine-tuning T2I models to generate user-specific content. DreamBooth [11], adapts pre-trained models like Stable Diffusion using a few reference images. It optimizes the model to reconstruct these images with the training prompts like “a photo of [V*] dog,” where [V*] is a unique identifier and “dog” is the personalized class name. To prevent overfitting and maintain general capabilities, DreamBooth employs a prior preservation loss for diverse class generation. The objective is:

\[\begin{align} \mathcal{L}_{DB}(\theta, z_0) = \mathbb{E}_{z_0, c, t, t', \epsilon, \epsilon'} \Big[ &\|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2 \\ &+ \lambda \|\epsilon' - \epsilon_\theta(z_{t'}', t', c_{\text{pr}})\|_2^2 \Big]. \end{align} \label{eq:32dreambooth}\tag{1}\] where \(\epsilon, \epsilon' \sim \mathcal{N}(0, I)\), \(z_{t'}'\) is the latent from prior prompt \(c_{\text{pr}}\) (e.g., “a photo of a dog”), and \(\lambda\) balances the preservation term.

3.1.3 Perturbation-based Anti-personalization↩︎

Perturbation-based Anti-personalization addresses risks from unauthorized outputs of T2I personalization. Perturbation-based methods add imperceptible perturbations to training images \(x^{(i)} \in \mathcal{X}\), forming protected images \(\mathcal{X}' = \{ x^{(i)} + \delta^{(i)} \}\), to disrupt fine-tuned models with parameters \(\theta^*\), causing poor performance. The optimization is: \[\begin{align} \Delta^* &= \arg\min_{\Delta}\;\mathcal{A}(\epsilon_{\theta^*}, \mathcal{X}) \\ \text{s.t.}\quad \theta^* &= \arg\min_{\theta}\;\sum_{i=1}^N \mathcal{L}\big(\theta, x^{(i)}+\delta^{(i)}\big), \\ \|\delta^{(i)}\|_p &\le \eta,\quad \forall i \in \{1,\dots,N\}, \end{align}\] where \(\mathcal{L}\) is the personalization loss (Eq. 1 ), and \(\mathcal{A}\) evaluates image quality for model \(\epsilon_{\theta^*}\). This bi-level optimization is difficult to solve directly, thus recent works tackle this from different angles: Anti-DB [16] leverages alternating surrogate and perturbation learning; SimAC [18] employs adaptive greedy search; Meta-Cloak [45] introduces a meta-learning framework for transferable perturbations; PAP [17] generates prompt-agnostic perturbations by modeling prompt distributions. DDAP [46] combines spatial and frequency perturbations; DisDiff [15] exploits cross-attention to strengthen attacks; and SIREN [47] embeds markers for dataset tracing. Recently, IMMA [48] propose to immunize the pre-trained model by learning model parameters that are difficult for the adaptation methods when fine-tuning malicious content.

However, these methods face practical challenges in certain scenarios. Primarily, they rely on the assumption that the protector maintains full control over the training data. In practice, unperturbed images are often readily accessible online, allowing attackers to construct mixed datasets. Consequently, the effectiveness of these defenses diminishes substantially when malicious fine-tuning involves clean images or standard data transformations. Furthermore, the degraded outputs often retain identifiable visual features, leading to incomplete privacy protection, while the generation of perturbations typically incurs high computational costs due to iterative optimization. These challenges motivate us to explore model-level protection as a complementary approach that addresses scenarios where data-level protection may be less effective.

3.2 Threat Model↩︎

Recent studies have shown that T2I diffusion models are vulnerable to backdoor attacks, where adversaries controlling the training process can embed triggers to achieve malicious objectives [24], [41], [42]. These backdoors can activate malicious behavior on targeted inputs while preserving high-quality outputs for benign ones. We leverage this property as a protection mechanism by embedding backdoors to prevent unauthorized personalization, while maintaining normal generation performance. This work focuses on DreamBooth [11], due to its strong personalization capabilities.

3.2.1 Protection Scenarios↩︎

Perturbation-based methods rely on the unrealistic assumption that malicious users will necessarily adopt perturbed images for personalization, which may not hold in practice. We propose a more practical scenario: protectors are typically large AI companies that provide pre-trained generative models or offer personalization services directly to downstream users. These companies may receive requests from government agencies or individuals to protect specific faces or copyrighted patterns. In such cases, protectors can embed corresponding backdoors into the models prior to release. Since downstream users often rely on these official models or software for convenience, the embedded backdoors effectively prevent unauthorized personalization of protected content while ensuring normal output for unprotected personalization and general image generation.

Our threat model assumes that protectors control the model distribution and that malicious users adopt these protected models for personalization. This assumption is well-established in prior research. In the domain of intellectual property protection, Adi et al. [49] and Zhang et al. [50] demonstrated that backdoor-based watermarking effectively protects DNN ownership under the same assumption. Beyond watermarking, Shan et al. [51] showed that intentionally injected backdoors can serve as “trapdoors” to detect adversarial attacks. These works validate the use of backdoor mechanisms for protective purposes when model owners control distribution channels. In practice, most malicious users lack the computational resources to train diffusion models from scratch and typically rely on publicly available pre-trained models from platforms such as Hugging Face or Civitai. If an attacker deliberately avoids the protected model and uses a clean one, our protection would not apply; however, this scenario is less common given the prevalence of public model adoption.

3.2.2 Protector’s Background Knowledge and Capabilities↩︎

We assume that protectors can only intervene before model release, with downstream personalization processes remaining unknown and uncontrollable. Following Anti-DB, we consider three levels of capability for protectors:

(i) White-box: Protectors know the identifier (e.g., “[V*]”), class name (e.g., “dog”), training prompts (e.g., “This is an image of a [V*] dog”), and has full knowledge of the protected dataset, which is realistic since users often rely on default tokens, simple class names, and standard prompts.

(ii) Gray-box: Protectors lack knowledge of the exact identifier, class name and training prompts, which may deviate from defaults (e.g., “sks animal” instead of “[V*] dog”).

(iii) Black-box: Protectors have little knowledge of or no control over the protected images that are part of user training.

3.2.3 Protector’s goal↩︎

Unlike previous T2I backdoor attacks that trigger harmful behavior, our goal is to prevent malicious personalization. Therefore, we propose three backdoor targets: The pattern-backdoor induces the model to generate images with a predefined pattern, signaling a privacy violation. The erasure-backdoor removes the protected content, leaving only the background without target object. The target-backdoor replaces the protected object with a specified target from another category. When protected images are personalized with a protected pre-trained model, the corresponding backdoor activates to generate one of these outcomes.

3.3 Our Method: PersGuard↩︎

As illustrated in Fig. 2, the goal of PersGuard is to inject a backdoor into a pre-trained model, such that protected personalization triggers specific protective behaviors, while unprotected images remain unaffected. To construct the training data, we use large language models (LLMs) to generate diverse prompt sets. First, we create description prompts containing the protected class name, referred to as prior prompts \(c_{\text{pr}}\), and prepend identifier tokens to form identifier prompts \(c_{\text{id}}\). Building on these, we generate erasure backdoor prompts \(c_{\text{era}}\) using negation terms (e.g., “nothing”), and target backdoor prompts \(c_{\text{tar}}\) by replacing the class name with a chosen target. Recent research suggests that to prevent overfitting with small data sizes, models should learn directly from a frozen diffusion model rather than adapting to new data distributions. Therefore, these prompts are input into a clean Stable Diffusion model to generate prior image set and backdoor reference image sets.

Specifically, let \(\theta'\) denote the backdoored model and \(\hat{\theta}\) a frozen clean model for synthesizing reference images. We define \(\mathcal{E}\) and \(\mathcal{D}\) as the encoder and decoder, respectively, with \(z = \mathcal{E}(x)\) as the latent representation of image \(x\), \(z_t\) as the noisy latent at timestep \(t\), and \(\epsilon \sim \mathcal{N}(0, I)\) as the noise sample. All objectives optimize the denoiser \(\epsilon_{\theta'}(z_t, t, c)\) conditioned on prompt \(c\).

3.3.1 Backdoor Behavior Loss.↩︎

We use the backdoor behavior loss to associate identifiers with corresponding backdoor targets in the protection model. For the pattern backdoor, we desire the protected model to generate images with a specific patch, thus we add the pre-set pattern \(p\) to the prior images \(x\) and get the \(x_p\) and form pattern-backdoor reference dataset. The loss can be expressed as: \[\begin{align} \mathcal{L_{BB}^\text{pat}} = \mathbb{E}_{z, c_{\text{id}}, \epsilon, t} \Big[\Big\| \epsilon_{\theta^{*}}&\left(z_{t},t, c_{\text{id}}\right) - \epsilon_p\Big\|_2^2 \Big], \end{align}\] where \(z_t\) are noisy versions of \(z := \mathcal{E}(x_p)\), and \(\epsilon_p\) are the real noises. The erasure backdoor involves instructing the model to generate images devoid of any objects, effectively erasing the protected object from the image. Similarly, we use the erasure reference images generated by erasure backdoor prompts \(c_{\text{era}}\) and inject the erasure backdoor into models using the following loss: \[\begin{align} \mathcal{L_{BB}^\text{era}} = \mathbb{E}_{z, c_{\text{id}}, \epsilon, t}\left[\left\|\epsilon_{\theta^{*}}(z_{t}, t, c_{\text{id}}) -\epsilon_{\hat{\theta}}\left(z_{t}, t, c_{\text{era}}\right)\right\|_2^2\right], \end{align}\] where \(z_t\) are noisy versions \(z := \mathcal{E}(x_e)\), and \(x_e\) are the erasure reference images. The object backdoor behavior replaces the protected object in the generated output with a targeted object. For example, suppose the protected object is a specific type of dog with the identifier “[V*]", and the target object is a rabbit. We expect the protect model to generate an image of a rabbit in response to any prompts containing”[V*] dog". Thus, we guide the protected model by the following loss: \[\mathcal{L_{BB}^\text{tar}} = \mathbb{E}_{z, c_{\text{id}}, \epsilon, t}\left[\left\|\epsilon_{\theta^{*}}\left(z_{t}, t, c_{\text{id}}\right)-\epsilon_{\hat{\theta}}\left(z_{t}, t, c_{\text{tar}}\right)\right\|_2^2\right],\] where \(z_t\) are noisy versions of \(z := \mathcal{E}(x_t)\), and \(x_t\) are the target backdoor reference images.

3.3.2 Prior Preservation Loss.↩︎

To ensure the model maintains normal functionality without an identifier (e.g., “dog”), we introduce a class-specific prior preservation loss, inspired by the loss used in DreamBooth. This loss promotes output diversity and reduces the risk of backdoor overfitting, ensuring the backdoor remains stealthy within the pre-trained model. Specifically, we use the prior images and defined the loss as: \[\mathcal{L_{PP}} = \mathbb{E}_{z, c_{\text{pr}}, \epsilon, t}\left[\left\|\epsilon_{\theta^{*}}\left(z_{t}, t, c_{\text{pr}}\right)-\epsilon_{\hat{\theta}}\left(z_{t}, t, c_{\text{pr}}\right)\right\|_2^2\right],\]

3.3.3 Backdoor Retention Loss.↩︎

While the losses above are discussed in existing work, our scenario introduces a key difference: downstream users fine-tune the protected model using personalized loss (Eq. 1 ), rather than using it directly. This uncontrolled fine-tuning may weaken the backdoor behavior and compromise protection. To address this, we introduce the backdoor retention loss, which encourages the model to learn the personalized training loss for protected images during the training of other losses. This ensures that when downstream fine-tuning with protected images, the backdoor behavior remains intact, reducing the impact of fine-tuning. Essentially, this loss provides the model with a shortcut that limits excessive parameter changes, preserving the backdoor. Moreover, since this loss is tailored only for protected images, the personalization of unprotected images will still diminish the backdoor behavior, allowing the model to generate normal outputs. \[\mathcal{L_{BR}} = \mathbb{E}_{z_p, c_{\text{train}}, \epsilon, t}\left[\left\|\epsilon_{\theta^{*}}\left(z_{t}, t, c_{\text{train}}\right)-\epsilon_{\text{train}}\right\|_2^2\right], \label{eq:32br}\tag{2}\]

3.3.4 Optimization Problem.↩︎

Therefore, we formulate PersGuard as the following optimization problem: \[\min_{\theta^{*}} \mathcal{L} = \mathcal{L_{BB}} + \lambda_1 \cdot \mathcal{L_{PP}} + \lambda_2 \cdot \mathcal{L_{BR}}, \label{eq:32loss}\tag{3}\] where \(\lambda_1\) and \(\lambda_2\) control the balance between loss terms. To solve this problem, we use gradient descent method. Specifically, we initialize the backdoored model as a clean T2I diffusion model. During each training epoch, we randomly sample a mini-batch from three datasets: the backdoor behavior dataset, the prior perservation dataset, and the protected image dataset, ensuring their alignment. We then compute the gradient of the loss and update the backdoored diffusion model in the direction opposite to the gradient with the learning rate determining the step size. This process is repeated for multiple iterations until the maximum training epochs is reached.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets. To evaluate the effectiveness of PersGuard, we primarily utilize the DreamBooth dataset [11], which comprises 30 distinct categories classified into 21 object classes (e.g., backpacks, toys) and 9 living subject classes (e.g., dogs, cats). For each category, we employ a LLM to synthesize 20 prompts for training and 10 prompts for evaluation. Additionally, to assess performance in facial privacy protection scenarios, we employ a pre-processed version of the CelebA-HQ dataset [52], following the protocol in Anti-DreamBooth [16]. This subset includes 307 identities, with each identity represented by at least 15 center-cropped images resized to \(512 \times 512\) resolution. Consistent with the DreamBooth, we generate 20 training prompts and 10 evaluation prompts for each facial identity.

Training Configurations. Our most experiments use the version of Stable Diffusion 2.1. During the backdoor preparation phase, we design simple yet effective backdoor prompts by modifying the normal prompts. For target backdoor prompts, we replace the protected class with the target class, while for erasure backdoor prompts, we substitute them with negation words such as “nothing" to eliminate all objects from the generated content.

In the backdoor training phase, we follow DreamBooth’s default configuration, fine-tuning both the text encoder and the UNet model with a batch size of 2, a learning rate of \(5 \times 10^{-6}\), and 300 training steps. The loss function hyperparameters are set to \(\lambda_1 = 0.5\) and \(\lambda_2 = 0.1\). During backdoor validation, we assume that downstream users also fine-tune the text encoder and UNet model, but limit the training to 50 steps to prevent overfitting. Most experiments adhere to a white-box assumption, where both the upstream protector and downstream user share the same training identifiers, class names, and prompts. However, we also include results based on a grey-box and black-box assumption in later sections. All experiments use one NVIDIA A100 GPUs with 40GB of memory.

Evaluation Metrics. Following prior studies [40]–[42], we employ quantitative metrics to measure both visual and semantic similarity:

• Backdoor Effectiveness: We use DINO\(_b\) and CLIP\(_b\) to measure the visual and semantic similarity between the generated outputs and the backdoor target images and prompts. Higher scores indicate successful activation of the protective backdoor (i.e., effective privacy protection).

• Subject Fidelity: DINO\(_c\) and CLIP\(_c\) quantify the similarity between outputs and the original user-provided image and text description. For protected images, lower scores are desirable as they indicate successful suppression of the sensitive identity. Conversely, for unprotected images, higher scores indicate preserved model utility.

• Image Quality: We report the Fréchet Inception Distance (FID) [53] to assess the general quality of generated images. A lower FID indicates that the protected model maintains a distribution similar to the clean model.

4.2 Main Results↩︎

4.2.1 Comparison with Perturbation-Based Protections↩︎

To highlight the critical limitations of perturbation-based defenses, we comprehensively compare PersGuard with four representative baselines: Anti-DB [16], PAP [17], SimAC [18], and DisDiff [15]. For each baseline, we strictly adhere to the original settings and simulate downstream personalization to generate visual results, which are compared with those of our Target-Backdoor method in Fig. 3 (a). The results demonstrate that baseline methods often degrade image quality yet fail to conceal the recognizable features of the protected target, indicating a failure to ensure robust protection. In contrast, PersGuard effectively conceals protected features.

To further quantify the protection strength, we employ four multimodal LLMs (MLLMs) as judges to determine whether the personalized outputs and the original protected images belong to the same category; a protection is considered successful if they are classified as different classes (see supplementary materials for detailed prompts). As shown in Fig. 3 (b), PersGuard consistently outperforms all baselines, offering a significantly stronger and more reliable defense against unauthorized personalization.

Existing perturbation-based baselines rely on a strong threat model, assuming an idealized scenario where all downstream training images are provided by the protector. To expose this vulnerability, we evaluate three practical scenarios: training solely on perturbed images (All-Controlled); training with one clean external image and the rest perturbed (One-Uncontrolled); and training with one perturbed image and the rest external (One-Controlled). We also examine the impact of data augmentation using three common transformations and their combinations, measuring protection efficacy with DINO\(_c\). As shown in Fig. 4, baselines exhibit high sensitivity to inputs, with their efficacy dropping significantly upon the introduction of clean images or augmentations. In contrast, our method exhibits superior robustness across all settings.

4.2.2 Comparison with Baseline Backdoors.↩︎

We conduct a comprehensive comparison between PersGuard and three representative T2I backdoor methods adapted for the personalized protection scenario: BadT2I [24], Personalization Shortcut [41], and EvilEdit [42]. These methods typically inject backdoors by associating specific trigger identifiers with target behaviors. Table 1 presents a quantitative evaluation across both protected and unprotected images. To rigorously assess specificity, unprotected images are tested in two distinct scenarios: (1) images from the Same-Class as the protected target (sharing the training prompt context), and (2) images from entirely Different-Class categories. Our analysis reveals a critical vulnerability in existing baselines: they lack resilience to the parameter updates inherent in the downstream personalization process. As evidenced by the results, the protective mechanisms of these baselines are effectively erased during fine-tuning due to catastrophic forgetting, rendering them ineffective for robust protection. In sharp contrast, our target backdoor demonstrates superior stability, reliably activating the protective behavior for protected instances without disrupting the personalization quality of unprotected images.

Furthermore, considering the deployment scenario where the protector is a foundational model provider (e.g., an AI company), it is imperative that the embedded protection mechanism does not compromise the model’s overall generative utility or stealthiness. To verify this, we benchmark the performance of protected models against a clean vanilla model on general generation tasks, utilizing both prompts related to protected categories (Prior Prompts) and unrelated neutral prompts (General Prompts). As detailed in Table 2, all variants of PersGuard maintain generative capabilities comparable to the clean model. This confirms that our framework achieves robust protection with high stealthiness, ensuring no degradation in the model’s practical utility for legitimate users.

4.3 Visualization↩︎

For the sake of clarity and illustration, this section will focus on the target-backdoor as an example for discussion.

4.3.1 Visualization Results↩︎

We conduct a qualitative evaluation across four categories, comprising both protected and unprotected image sets, to visualize the personalized outputs from models embedded with our three backdoor variants. As presented in Tab. 3, the results demonstrate that for protected images (columns 5, 7, and 9), the fine-tuned models robustly inherit the upstream backdoor mechanisms and trigger the intended defensive behaviors: the Pattern-Backdoor superimposes a visible warning patch (red exclamation mark), the Erasure-Backdoor eliminates the protected object entirely, and the Target-Backdoor substitutes it with a predefined visual target. Conversely, for unprotected images (columns 6, 8, and 10), the backdoor effects are effectively neutralized, yielding results indistinguishable from those of clean models. These findings confirm that PersGuard enforces precise selective protection, successfully defending specific targets while preserving normal personalization utility for others.

4.3.2 Attention Map↩︎

To investigate the underlying mechanism, we visualize cross-attention maps using the DAAM method [54] for both clean and protected personalized models alongside their generated images. As illustrated in the second row of Fig. 5, the clean personalized model concentrates high-attention regions (indicated in red) for the “sks” token around the dog’s head, reflecting its effective acquisition of the distinct head features associated with the new class. In sharp contrast, the third and fourth rows reveal a significant shift in attention dynamics within protected models: for the Pattern and Erasure backdoor, the attention for “sks” is redirected toward the upper-left pattern and the background, respectively. Conversely, for the Target backdoor, the “sks” token retains its focus on the dog’s head, which is consistent with the semantic transformation task of converting the “sks dog” into a rabbit-like appearance. Additional visualization methods for attention maps are provided in the supplementary materials.

4.4 Ablation Study↩︎

4.4.1 Loss Components Analysis↩︎

To validate the necessity and contribution of each component within our unified optimization objective, we conducted a comprehensive ablation study. Focusing on the Target-Backdoor configuration as a case, we employ DINO\(_c\) and DINO\(_b\) to quantify protection effectiveness. Additionally, to assess the model’s utility on general generation tasks, we introduce DINO\(_{pr}\), a metric designed to evaluate the alignment between the protected model’s responses to prior prompts and those of the original clean model. The quantitative results for various loss combinations are reported in Tab. 4. Our analysis reveals that the Backdoor Retention Loss (\(\mathcal{L_{BR}}\)) is indispensable for robust protection; its absence leads to the rapid removal of the backdoor during downstream fine-tuning due to catastrophic forgetting. Furthermore, the Prior Preservation Loss (\(\mathcal{L_{PP}}\)) serves as a critical regularizer, effectively preventing overfitting to backdoor triggers and preserving the model’s original generative prior.

4.4.2 Gray-Box Setting↩︎

Transitioning from the idealized white-box assumption, we investigate the more practical gray-box Setting, where the protector lacks perfect knowledge of the attacker’s personalization parameters including identifier, class name and training description. We observe that applying a model protected by PersGuard under white-box assumptions directly to a scenario where attackers utilize different tokens and prompts results in a significant degradation of protection efficacy, as evidenced in Tab. 5. We attribute this to a textual distribution shift: PersGuard’s trigger is conditioned on identifier/class/prompt templates, and unseen attacker prompts weaken trigger activation. Therefore, vanilla PersGuard is not intended as the final solution for gray-box; we propose universal variants below.

To mitigate this vulnerability, we introduce universal training strategies. Specifically, for PersGuard-UI, the protector utilizes a pool of 10 distinct identifier tokens (e.g., “sks”, “abc”, “[A*]”) combined with generic class names (e.g., “animal”, “pet”), which are randomly sampled during injection to enforce a universal mapping. For PersGuard-UD, we employ a compact set of 5 universal training prompts with varying structures (e.g., “This is an image of …”, “A portrait of …”) to associate the backdoor effect with broader textual contexts. PersGuard-UID combines both strategies.

To rigorously evaluate these variants, we standardize the attacker’s parameters to values unseen during training: the identifier is set to “xyz”, the class name to “puppy”, and the training prompt to “A picture of xyz puppy”. As shown in Tab. 5, prompt diversification (UD) improves gray-box robustness more than token diversification alone (UI), indicating that mismatched prompt templates are the dominant source of trigger failure. Combining both (UID) yields the best generalization.We attribute this unexpected effectiveness to the semantic proximity between the protector’s sampled parameters and potential attacker choices (e.g., the synonym relationship between “animal” and “pet”). This correlation enables the protection mechanism to generalize robustly across shared semantic neighborhoods in the embedding space, making the strategy practically feasible without requiring exhaustive parameter coverage.

4.4.3 Black-Box Setting↩︎

The most stringent setting is the black-box scenario, where the protector lacks access to the specific images utilized by the attacker for downstream personalization. To simulate this challenging environment, we split the target dataset: two-thirds of the images are used as the protector’s known training set for backdoor injection, and the remaining unseen images form the training set utilized by the user for fine-tuning. We compare the protection efficacy of our backdoor-based method with perturbation-based defenses under this strict black-box assumption. As shown in Tab. 5, our PersGuard-UID retains significant efficacy. This superiority stems from the fundamental difference in mechanism: our backdoor protection is associated with the high-leve features of the protected object class, rather than being strongly correlated with a specific set of training images. In stark contrast, perturbation methods strictly rely on access to the exact images to which the perturbation was applied, rendering the optimized perturbations non-transferable and ineffective on the unseen dataset in the black-box setting.

4.4.4 Model Generalizability↩︎

To verify the architectural universality of our framework, we extend our evaluation across four distinct versions of Stable Diffusion (SD), ranging from v1.5 to v3.5. As reported in Tab. 6, PersGuard demonstrates consistent effectiveness across all architectures. For protected images, we observe a substantial reduction in alignment metrics (DINO\(_c\) and CLIP\(_c\)) across all versions, validating the method’s ability to robustly suppress unauthorized personalization regardless of the underlying model structure. Crucially, for unprotected images, the performance metrics exhibit negligible deviation from the clean baseline. These results confirm that PersGuard is architecture-agnostic, successfully enforcing protection without compromising the intrinsic generation capabilities of the foundation models.

4.4.5 Robustness against Personalization Techniques↩︎

We further assess the resilience of PersGuard against diverse downstream personalization strategies, simulating a realistic scenario where attackers may employ varying algorithms. As summarized in Tab. 7, we benchmark four representative methods: standard DreamBooth, DreamBooth enhanced with Low-Rank Adaptation (LoRA), the large-scale SDXL backbone, and Textual Inversion (TI).The results demonstrate that PersGuard maintains exceptional protection efficacy against weight-tuning paradigms (DreamBooth, LoRA, and SDXL), where the model parameters are updated. However, we observe a performance attenuation against Textual Inversion (TI). We attribute this divergence to the fundamental mechanistic difference between the defense and the attack: PersGuard embeds protection directly into the U-Net weights, whereas TI freezes these weights and restricts optimization exclusively to the text embedding space. Consequently, embedding-centric attacks like TI may partially bypass the weight-based trigger mechanisms and attenuate the protection.

4.5 Case Study (Face Protection)↩︎

In previous works, researchers have extensively explored facial protection, as it represents one of the most common personalization tasks. This section presents a case study on face privacy protection. Unlike other scenarios, face image personalization protection may require safeguarding multiple facial identities simultaneously, which may share the same identifier token and class name. We randomly selected five different identities from the CelebA-HQ dataset as protection targets, assuming downstream users employ the same identifier token (“sks”) and class name (“person”) for personalization. Using the target backdoor as an example, we set the backdoor target class as “Superman”. Although the backdoor model needs to protect five facial identities simultaneously, we only need to include the data from all five identities in the training set for the backdoor retention loss, and then incorporate it into the total loss. Using this approach, we trained the face backdoor model and performed personalization on the five identities. We then calculated the DINO and CLIP scores, with the results shown in Table 8. We found that the backdoor model successfully triggered specific outputs across all five faces during downstream personalization fine-tuning. Our experimental results demonstrate that PersGuard is practically viable for protecting celebrity portraits in real-world applications.

3pt

4.6 Backdoor Capacity Analysis↩︎

While our preceding analysis established the efficacy of embedding a single backdoor, real-world deployment often necessitates the simultaneous defense of multiple distinct assets. This requirement imposes a challenge on the model’s capacity to accommodate multiple backdoors without mutual interference. In this section, we investigate the feasibility and scalability of embedding multiple independent backdoors into a single T2I model, analyzing the trade-offs between protection capacity and effectiveness.

4.6.1 Impact of Multi-Backdoor Integration↩︎

To rigorously evaluate this, we selected three distinct object categories—dogs, backpacks, and toys—as protection targets. For consistency, all backdoors utilize the same identifier token (“sks”) as the trigger. The quantitative results are summarized in Table 9. Rows 1–3 demonstrate that single-backdoor models provide robust protection confined strictly to their designated category. Despite sharing a common identifier, we observe no cross-interference; each backdoor operates orthogonally. In Row 4, we integrate all three backdoors into a single unified model. While this configuration successfully enables simultaneous protection across multiple categories, we observe a marginal attenuation in effectiveness compared to single-backdoor baselines. This slight performance degradation is likely attributable to parameter contention, as the model must manage multiple trigger-response mappings within shared weights, increasing the complexity of the optimization landscape. Overall, these findings substantiate the feasibility of multi-backdoor protection.

4.6.2 Scalability and Protection Effectiveness Curves↩︎

To further evaluate the scalability of PersGuard under increasing protection loads, we extend the capacity analysis to larger protection sets and examine how protection effectiveness evolves as more backdoors are integrated. In addition to the protected metrics, we also report the corresponding unprotected metrics to provide a brief view of utility for benign users. Specifically, we track DINO\(_c\) and DINO\(_b\) under two scaling scenarios: Inter-category protection, where backdoors target distinct semantic classes, and Intra-category protection, where many protected targets belong to the same class (e.g., multiple facial identities).

As illustrated in Fig. 6 (a), in the Inter-category setting, the protected curves remain highly stable as capacity increases, indicating that PersGuard scales well when protected targets occupy distinct semantic regions and that interference among backdoors is minimal. The unprotected curves also remain favorable, suggesting that normal personalization quality is largely preserved in this regime.

The Intra-category setting in Fig. 6 (b) is more challenging, since many visually similar identities must be accommodated within a shared class manifold. As capacity increases, protected performance shows a mild decline, which we attribute to semantic saturation and increased competition for representation space. Nevertheless, the degradation remains limited, and the unprotected curves vary only modestly, indicating that normal-user utility is still preserved overall. These results show that PersGuard remains effective at larger scales, with same-category protection representing the more demanding scenario.

5 Conclusion↩︎

In this paper, we presented PersGuard, a novel backdoor-based framework designed to safeguard pre-trained Text-to-Image (T2I) diffusion models against unauthorized personalization. Identifying the inherent limitations of existing adversarial perturbation methods, specifically their susceptibility to data transformations and failure under realistic training conditions, we proposed a paradigm shift toward proactive model-level protection, which is more aligned with practical deployment settings where model providers offer personalization services.

By formulating a unified optimization framework, PersGuard effectively embeds three distinct protective mechanisms: Pattern, Erasure, and Target backdoors. Crucially, the introduction of the Backdoor Retention Loss addresses the fundamental challenge of catastrophic forgetting, ensuring that protective triggers remain active even after rigorous downstream fine-tuning while preserving the model’s utility for legitimate users. Extensive experiments confirm that PersGuard offers superior protection efficacy and stability compared to state-of-the-art baselines. Furthermore, our approach demonstrates remarkable resilience across practical application scenarios, including challenging gray-box and black-box settings, as well as complex multi-identity facial protection tasks. In future work, we aim to further enhance the effectiveness and robustness of the protective backdoors against potential detection and removal attacks.

Limitations and Future Work. While PersGuard demonstrates effective protection against malicious personalization, several limitations warrant discussion. First, our threat model assumes centralized model distribution where protectors release backdoored models for attackers to use; if attackers obtain clean models from alternative sources or train from scratch, the protection becomes ineffective. However, this assumption remains practical since most malicious users rely on publicly available pre-trained models due to limited computational resources. Second, PersGuard is most effective when model providers control distribution and downstream users employ standard personalization methods. We acknowledge that unrestricted weight-level access or alternative personalization strategies may attenuate the protection. In future work, we aim to address these limitations by developing more robust backdoor mechanisms resilient to detection and removal attacks. Finally, as PersGuard employs backdoor techniques that could potentially be misused, responsible deployment requires proper governance frameworks including verified protection requests and transparent disclosure policies. Future work could focus on improving backdoor persistence under adaptive fine-tuning and establishing standardized protocols for ethical deployment of protective backdoors.

References↩︎

---

1. This work was supported in part by the Shenzhen Science and Technology Program (No.SYSRD20250529113401002); by the National Natural Science Foundation of China (No.62441619, No.U2541229, No.62132006, No.62372448); and by the Open Topics from the Lion Rock Labs of Cyberspace Security (under the project #LRL24009). Corresponding authors: Hua Zhang and Xiaochun Cao.↩︎

2. Xinwei Liu, Yuan Xun, and Hua Zhang are with the Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China, and also with the School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China. Xinwei Liu is also with BraneMatrix AI, Shanghai 201203, China. (e-mail: liuxinwei@iie.ac.cn, xunyuan@iie.ac.cn, zhanghua@iie.ac.cn).↩︎

3. Xiaojun Jia is with Cyber Security Research Centre @ NTU, Nanyang Technological University, Singapore. (e-mail: jiaxiaojunqaq@gmail.com).↩︎

4. Xiaochun Cao is with the School of Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University, Shenzhen 518107, China (e-mail: caoxiaochun@mail.sysu.edu.cn).↩︎