1 Introduction↩︎

Unified multimodal models (UMMs) [1]–[5] are reshaping the security landscape of image editing. Conventional vision-language models [6], [7] mainly focus on visual understanding, while diffusion models [8], [9] are mainly designed for image generation. In contrast, UMMs combine visual understanding, language reasoning, and image generation within one model. This design allows them to perform fine-grained image editing from an input image and a natural-language instruction. With only one facial portrait and a short instruction, a UMM can produce a realistic edited image while preserving the identity of the person. This ability is useful for image creation, but it also lowers the cost of facial-image misuse. Public portraits can be directly used to create false content, impersonate a person, or generate unauthorized facial derivatives.

A proactive way to protect an image is to add an imperceptible adversarial perturbation before the image is released [10], [11]. Unlike watermarking methods [12] that trace misuse after it happens, adversarial protection directly interferes with the model and prevents it from understanding, learning, or reproducing protected content. Existing methods mainly target diffusion models or vision-language models (VLMs). Diffusion-based methods often perturb the VAE latent space, the denoising network, or the generation velocity [13]–[17]. VLM-based methods mainly change the semantic features produced by a visual encoder [18]. These methods have shown promising results in protecting personalized generation, image editing, and visual understanding. However, a UMM is not a simple combination of a diffusion model and a VLM. Its editing process depends on several visual paths at the same time. It remains unclear whether protection methods designed for a single path can work in this setting.

We study this problem using BAGEL [1], a representative UMM with separate visual encoders, one ViT for understanding and one VAE for generation. We systematically adapt diffusion-based protection methods and VLM-based adversarial methods to BAGEL (Figure 1). However, neither group can reliably prevent facial editing. Diffusion-based methods mainly disturb the VAE generation branch but leave the semantic anchor from the ViT branch largely unchanged. VLM-based methods disturb the ViT branch, while the VAE branch still preserves useful appearance and identity information. As long as one branch provides a stable facial representation, the UMM may still complete the requested edit.

To understand this failure, we further trace the feature changes caused by ViT-only or VAE-only protection. We examine the two visual encoders, the shared MoT tokens, and the final generation velocity. Our analysis shows that existing methods can cause clear changes inside their target modules. However, these local changes do not always propagate to the generation process. We argue that two conditions are necessary for successful protection. First, both visual branches should move away from their clean representations. Second, the two branches should no longer provide consistent descriptions of the same face. This finding suggests that UMMs do not simply repair adversarial perturbations in their later layers. Instead, existing objectives fail to attack the interface that connects visual understanding with image generation. The protection objective should therefore move from single-branch feature distortion to cross-branch conflict.

Based on this finding, we propose Cross-Branch Conflict Shield (CCS), a proactive protection method for facial editing with UMMs. CCS first maximizes the distances between the protected and clean representations in both visual branches. We apply one untargeted divergence objective to the ViT tokens and another to the VAE tokens in the shared language-model input space. These two objectives prevent either branch from keeping a stable representation of the original face. We then introduce a CKA-based consistency disruption objective. It explicitly reduces the structural agreement between the protected ViT and VAE tokens. As a result, the two branches produce conflicting representations for the same input image. The branch-wise objectives remove useful information from each path, while the cross-branch objective prevents the model from combining the remaining information into a consistent facial representation.

All model parameters remain frozen during optimization. CCS only updates a bounded perturbation on the input image. A data owner can protect a facial image without knowing how it may be edited in the future. We also apply a comprehensive evaluation protocol to evaluate the generated results from several aspects, including identity preservation, instruction completion, and image quality. Experiments show that CCS keeps the protected inputs visually close to the clean images while greatly reducing identity similarity and editing completion in the outputs. It often causes strong identity erasure or generation failure. Across different editing instructions, representative baselines, and evaluation settings, CCS achieves the best protection results (examples also in Figure 1). Our main contributions are summarized as follows:

• We systematically study facial-image protection for unified multimodal editing. We identify a shared limitation of existing diffusion-based and VLM-based methods: they perturb only one visual path and leave the cross-branch cooperation of UMMs largely intact.

• We propose Cross-Branch Conflict Shield, which jointly optimizes ViT-branch divergence, VAE-branch divergence, and CKA-based cross-branch consistency disruption. This design creates both representation shifts and branch conflicts in the shared token space.

• Extensive experiments conducted show that CCS can reliably prevent effective facial editing while preserving the visual quality of the protected input images.

2 Related Works↩︎

Adversarial Examples. Recent works have explored adversarial examples as a proactive defense against unauthorized use of copyrighted or private images in diffusion models. Early methods such as AdvDM and Mist [14], [15] perturb artworks before release, preventing diffusion models from faithfully imitating protected content or styles. Glaze [13] further protects artists by cloaking style representations learned by text-to-image models, while Nightshade [19] extends this idea to data poisoning, corrupting model behavior when protected images are scraped for training. More recently, PID [16] relaxes the strong assumption that defenders know the attacker’s prompts by attacking prompt-independent visual representations, and VAC [17] improves semantic erasure by optimizing variance-driven objectives in the latent space of customized diffusion models. Overall, these studies show that carefully designed adversarial examples can alter how generative models encode protected data, thereby providing an effective technical route for copyright and privacy protection.

Unfied multimodal models. Unified Multimodal models (UMMs) integrate the understanding and generation capabilities in one model. A variety of architectures of UMMs [1]–[5] have been developed. Among them, seperate-encoder UMMs like BAGEL [1] and InternVL-U [4], decouple the processing of two distinct types of representations (usually ViT and VAE), and become one of the most famous architectures. This becomes the main architecture we explore.

Recent work [20] demonstrates the jailbreak vulnerabilities of UMMs. However, seldom has explored safeguarding identities in the widely used aspect of UMMs: multimodal image editing.

3 Preliminary↩︎

3.1 Defensive Assumption↩︎

Defense goal. As mentioned in the Introduction part, unauthorized user can take a publicly available face image and use a UMM to generate edited images with the identity of the person. Such unauthorized facial editing can lead to portrait misuse, or the creation of unwanted facial derivatives. In this work, we focus on proactive protection against such misuse. Detailedly, the defender aims to add a small perturbation to the facial image before releasing it. When an unauthorized user later applies a UMM to edit this protected image, the hidden perturbation disrupts the facial representations used by the model, making it difficult to preserve the original identity in the generated result. There exist some constraints: (1) the perturbation should be visually imperceptible to humans, (2) it should disturb the UMM editing process and prevent the model from producing identity-preserving edited results.

Defender capabilities. We assume a grey-box setting, that the defender can only access the visual encoding modules of the target UMM, or a surrogate UMM with a similar visual encoding design. This assumption is similar to previous surrogate-based settings in generating adversarial examples for VLMs [21]–[23]. During optimization, all model parameters are non-editable to the defender. The defender only updates the input perturbation under a bounded budget. The protected image can then be released publicly.

4 Methodology↩︎

4.1 Problem Formulation↩︎

On having the defensive assumptions described in Section III, we make the following problem formulation:

Given a clean image \(\mathbf{x}\in[0,1]^{H\times W\times 3}\), we construct a protected image \(\mathbf{x}^{p}\) as \[\mathbf{x}^{p} = \Pi_{[0,1]} \left( \mathbf{x}+\boldsymbol{\delta} \right), \qquad \|\boldsymbol{\delta}\|_{\infty}\leq\epsilon, \label{eq:protected95image}\tag{1}\] where \(\boldsymbol{\delta}\) is the protective perturbation, \(\epsilon\) is the perturbation budget, and \(\Pi_{[0,1]}\) clips the image into the valid pixel range.

A UMM contains a generation branch and an understanding branch. We denote their output features as \[\mathbf{F}_{z}(\mathbf{x}) = \mathcal{G}_{z}(\mathbf{x}), \qquad \mathbf{F}_{v}(\mathbf{x}) = \mathcal{G}_{v}(\mathbf{x}), \label{eq:branch95features}\tag{2}\] where \(\mathcal{G}_{z}\) and \(\mathcal{G}_{v}\) are the two frozen visual pathways. Their features are mapped into the shared LLM-input token space. Thus, the two branches need to provide compatible representations of the same image.

Based on this property, we protect the image from two aspects. First, we reduce the agreement between the generation and understanding branches. Second, we enlarge the feature difference between the clean and protected images within each branch. The protective perturbation is obtained by \[\boldsymbol{\delta}^{*} = \arg\max_{\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon} \mathcal{L}_{\mathrm{total}} \left( \mathbf{x}, \mathbf{x}^{p} \right). \label{eq:general95objective}\tag{3}\]

As shown in Figure 3, all model parameters are frozen during optimization. Only the input perturbation is updated. The whole process does not use any editing prompt. Therefore, the image can be protected before the future editing request is known.

4.2 Observation: Feature Agreement as a Signal↩︎

We start with BAGEL-MOT-7B [1] (BAGEL) to study the problem. BAGEL uses a ViT branch [24] to extract semantic visual tokens for image understanding and instruction grounding. It also uses a VAE branch [25] to encode appearance and spatial information for image generation. The outputs of both branches are mapped into a shared language-model input space and jointly processed by the following Mixture-of-Transformers layers [1].

We conduct exploratory experiments to investigate whether this property can be exploited by our protection method. Clean, ViT-level protection [18] and VAE-level protection [17] are utilized. Firstly we apply centered kernel alignment [26] (CKA) to check the representations \(\mathbf{F}_{z}(\mathbf{x})\) and \(\mathbf{F}_{v}(\mathbf{x})\) from the ViT and VAE encoders. For two centered feature matrices \(\mathbf{X}\) and \(\mathbf{Y}\), linear CKA is defined as \[\operatorname{CKA}(\mathbf{X},\mathbf{Y}) = \frac{ \left\| \mathbf{Y}^{\top}\mathbf{X} \right\|_{F}^{2} }{ \left\| \mathbf{X}^{\top}\mathbf{X} \right\|_{F} \left\| \mathbf{Y}^{\top}\mathbf{Y} \right\|_{F} }, \label{eq:cka}\tag{4}\] where \(\|\cdot\|_{F}\) denotes the Frobenius norm. A larger CKA value means stronger agreement between the two feature spaces. Results are in Figure 2 (a). We discover that features from two branches exhibit structural agreement (the “agreement” also pointed out in [27], [28]), while single-branch adversarial perturbations reduce this cross-branch similarity to different degrees. However, Figure 2 (b) suggests that these deviations are obvious in intermediate MoT hidden states, but gradually diminish in later layers. Figure 2 (c) further shows that the protection effects are asymmetric across timesteps, indicating that perturbing a single visual pathway cannot consistently disrupt the entire unified editing process. These findings may explain why perturbation only on one branch fails to protect facial identities. This motivates us to consider a method that breaking the agreement above directly.

4.3 Cross-Branch Conflict↩︎

Regarding the observations above, our core idea is to directly disrupt the agreement to disturb the editing process. The two branches may produce different numbers of tokens. We use a token mapping function \(\Phi(\cdot)\) to convert them into the same token length: \[\widehat{\mathbf{F}}_{b}(\mathbf{x}) = \Phi \left( \mathbf{F}_{b}(\mathbf{x}) \right), \qquad b\in\{z,v\}. \label{eq:token95mapping}\tag{5}\] When the two branches already produce the same number of tokens, \(\Phi(\cdot)\) is an identity mapping.

The two encoders have different structures and feature distributions. Thus, direct element-wise comparison is not suitable. Inspired from the study on feature changes, we also use CKA to measure the relation between their features.

We define the cross-branch agreement of an image as \[\mathcal{A}(\mathbf{x}) = \operatorname{CKA} \left( \widehat{\mathbf{F}}_{z}(\mathbf{x}), \widehat{\mathbf{F}}_{v}(\mathbf{x}) \right). \label{eq:branch95alignment}\tag{6}\]

Having this, we consider a new cross-branch conflict loss as \[\mathcal{L}_{\mathrm{cbc}} = \mathcal{A}(\mathbf{x}) - \mathcal{A}(\mathbf{x}^{p}). \label{eq:cbc95loss}\tag{7}\]

The first term measures the original agreement between the two branches. It remains fixed during optimization. The second term measures their agreement after protection. Maximizing \(\mathcal{L}_{\mathrm{cbc}}\) reduces the CKA value of the protected image.

As a result, the understanding branch and the generation branch form conflicting representations of the same input. The shared language model then receives an unstable visual context. This makes it harder for the UMM to connect the editing instruction with the correct visual content.

4.4 Branch-wise Representation Divergence↩︎

The cross-branch objective reduces the agreement between the two branches. However, a lower agreement does not ensure that both branches are changed. For example, one branch may shift greatly, while the other branch still keeps useful information about the clean image. We therefore add two branch-wise objectives.

We use a normalized feature distance \(\mathcal{D}(\cdot,\cdot)\) to measure the difference between clean and protected features. For two token sequences \(\mathbf{X},\mathbf{Y}\in\mathbb{R}^{N\times d}\), we define \[\mathcal{D}(\mathbf{X},\mathbf{Y}) = 1- \frac{1}{N} \sum_{i=1}^{N} \frac{ \mathbf{x}_{i}^{\top}\mathbf{y}_{i} }{ \|\mathbf{x}_{i}\|_{2} \|\mathbf{y}_{i}\|_{2} }. \label{eq:feature95distance}\tag{8}\]

4.4.0.1 VAE Branch Divergence.

The generation branch uses a VAE encoder to keep the visual information needed for reconstruction and synthesis. We define its divergence loss as \[\mathcal{L}_{\mathrm{vae}} = \mathcal{D} \left( \mathbf{F}_{z}(\mathbf{x}), \mathbf{F}_{z}(\mathbf{x}^{p}) \right). \label{eq:vae95loss}\tag{9}\]

Maximizing this loss moves the protected VAE features away from the clean features. This weakens the representation of appearance, texture, and spatial structure used in later image generation.

4.4.0.2 ViT Branch Divergence.

The understanding branch uses a ViT encoder to extract high-level visual semantics. These features support object recognition, attribute understanding, and instruction grounding. We define its divergence loss as \[\mathcal{L}_{\mathrm{vit}} = \mathcal{D} \left( \mathbf{F}_{v}(\mathbf{x}), \mathbf{F}_{v}(\mathbf{x}^{p}) \right). \label{eq:vit95loss}\tag{10}\]

Maximizing this loss changes the semantic representation of the protected image. The model may then fail to identify the correct object, attribute, or region mentioned in the editing prompt.

The two losses play different roles. \(\mathcal{L}_{\mathrm{vae}}\) disturbs the feature space used for image generation, while \(\mathcal{L}_{\mathrm{vit}}\) disturbs the feature space used for image understanding. Together, they prevent either branch from keeping a stable representation of the private image.

3pt

4.5 Overall Objective and Optimization↩︎

We combine the cross-branch conflict loss and the two branch-wise losses: \[\mathcal{L}_{\mathrm{total}} = \alpha\mathcal{L}_{\mathrm{vae}} + \beta\mathcal{L}_{\mathrm{vit}} + \gamma\mathcal{L}_{\mathrm{cbc}}, \label{eq:total95loss}\tag{11}\] where \(\alpha\), \(\beta\), and \(\gamma\) control the weights of the three objectives.

The protected image is obtained by solving \[\mathbf{x}^{p*} = \arg\max_{ \|\mathbf{x}^{p}-\mathbf{x}\|_{\infty}\leq\epsilon } \mathcal{L}_{\mathrm{total}}. \label{eq:final95objective}\tag{12}\]

We use projected gradient ascent to update the protected image (\(t\) means a certain step): \[\mathbf{x}^{p}_{t+1} = \Pi_{ \mathcal{B}_{\infty}(\mathbf{x},\epsilon) \cap [0,1] } \left[ \mathbf{x}^{p}_{t} + \eta\, \operatorname{sign} \left( \nabla_{\mathbf{x}^{p}_{t}} \mathcal{L}_{\mathrm{total}} \right) \right], \label{eq:pgd95update}\tag{13}\] where \(\eta\) is the step size and \(\mathcal{B}_{\infty}(\mathbf{x},\epsilon)\) is the \(\ell_{\infty}\) neighborhood of the clean image.

During optimization, the VAE encoder, the ViT encoder, and their mapping modules remain frozen. The gradient is only used to update the protected image. The final image stays close to the clean image in pixel space, but it produces divergent and conflicting features inside the UMM.

5 Experiments↩︎

5.1 Experimental Settings↩︎

Datasets and Models. Our experiments are mainly conducted on VGGFace2 [29], where we select 20 different facial identities. We use BAGEL-7B as the surrogate model. Additional experiments on face datasets, such as CelebA-HQ [30] are detailed in Appendix.

Baselines. We choose diffusion-based and VLM-based baselines like AdvDM [15], PhotoGuard [31], Mist [14], PiD [16], VAC [17], and Veattack [18]. Details of baselines can also be found in Appendix.

Settings. We conduct our optimization and testing on one RTX A100 80G GPU. All adversarial examples are optimized for 100 iterations, with the perturbation budget set to 8/255. Unless otherwise specified, all baselines use their default hyperparameters. For our method, the loss weights \(\alpha\), \(\beta\), and \(\gamma\) are set to 10, 0.2, and 1.0, respectively.

Evaluation Metic. For evaluation, we assess our method mainly from two perspectives: protection effectiveness and generated image quality. For facial identity protection, we use the identity score matching (ISM) [32] to measure the facial similarity between the generated image and the original image, where a lower score indicates better protection. For generated image quality, we use FID [33] to measure the distributional discrepancy between images edited from adversarial examples and those edited from the corresponding clean images, where a higher score indicates a larger discrepancy. We also use BRISQUE [34] to evaluate the perceptual quality of the generated images, where a higher score is considered better. ISM is between 0 and 1, while FID and BRISQUE is between 0 and 100. Moreover, to evaluate the defensiveness against instruction following, we introduce a new metric, termed the Und-Score.

5.2 Und-Score: a Newly Proposed Metric↩︎

Apart from traditional metrics, there should be one core metric evaluating whether the edited image follows the requirements mentioned in the instructions. Note that in our scenario, a better defense means that the UMM follows fewer requirements, representing a failed editing process. An intuitive way is to split the instructions into sub points, and evaluate them one by one. Based on this idea, we introduce the Understanding Score (Und-Score). Its motivation is that the understanding branch of a UMM can also interpret the image produced by its generation branch. We therefore use the understanding branch of a clean UMM to measure how well the edited image follows the instruction. Specifically, we first use Qwen-2.5-7B-Instruct [35] to decompose the prompt into a set of atomic scoring points. A special template is used to reduce overlap and ensure that all key requirements are covered. The understanding branch of the UMM, Bagel in our experiments, then checks each scoring point against the edited image and assigns a binary score of 0 or 1. The final Und-Score is the average score over all points. A lower score indicates that fewer instruction requirements are satisfied and thus reflects a stronger defense. Details are provided in the appendix.

5.3 Main Experiments↩︎

We bold the best results for all the metrics. As shown in Tables ¿tbl:tab:method95comparison?-[tbl:tab:method95comparison95p4], CCS achieves the lowest ISM and Und-Score, as well as the highest FID, under all four editing prompts. In particular, CCS reduces ISM to 0.00 in every setting, showing that the edited results no longer preserve the identity of the source portrait. Averaged over the four prompts, CCS obtains an FID of 466.32 and an Und-Score of 2.30. In comparison, the strongest baseline results on these two metrics are only 204.00 and 58.85, respectively. CCS also achieves the highest average BRISQUE score of 43.36. Although BRISQUE produces a higher BRISQUE score on Prompt 4, it retains more identity information and satisfies more instruction requirements. Overall, single-branch methods provide only partial protection, since the other visual branch can still offer useful semantic or appearance cues. By jointly shifting both branches and breaking their agreement, CCS provides stronger and more stable protection across different editing instructions.

5.4 Ablation Study↩︎

We ablate the two main components of CCS on Prompt 1. As shown in Table 1, removing the branch-wise divergence losses causes a clear performance drop. ISM increases from 0.00 to 0.37, while FID decreases from 532.38 to 141.79. The Und-Score also rises to 96.50, indicating that the model can still preserve the identity and complete most editing requirements. This result shows that jointly shifting the ViT and VAE representations is essential for removing stable visual information. Removing the cross-branch conflict loss also weakens the defense, increasing ISM to 0.31 and Und-Score to 43.68. Although the branch-wise losses can disturb each pathway separately, the remaining agreement between them still allows partial information recovery. Combining both components achieves the strongest protection, confirming that branch-wise divergence and cross-branch conflict play complementary roles.

5.5 Qualitative Results↩︎

Figure 4 presents qualitative comparisons under four editing instructions. Most baselines cause only limited or unstable disruption. In many cases, the edited images still preserve the source identity and contain the requested objects or scenes. Some methods cause generation failure in isolated cases, such as PiD on Prompt 1 and Mist on Prompt 4. However, their protection is not consistent across different instructions. In contrast, CCS consistently prevents identity-preserving editing. Its outputs either contain severe visual artifacts or no longer resemble the source person. Most requested details are also removed or incorrectly generated. These results show that CCS provides stronger and more reliable visual protection than single-branch baselines.

6 Conclusion↩︎

In this work, we studied proactive facial identity protection for unified multimodal image editing. We found that single-branch protection is insufficient, since a UMM can still recover useful information from the other visual pathway. Based on this finding, we proposed CCS, which jointly shifts the ViT and VAE representations and disrupts their cross-branch agreement. Extensive experiments show that CCS consistently prevents identity-preserving editing and reduces instruction completion across different editing prompts. These results demonstrate the importance of attacking cross-branch cooperation and provide a promising direction for protecting facial images against unauthorized UMM editing.

References↩︎