1 Introduction↩︎

The advancement of deepfake technology raises significant privacy concerns, particularly as face-swapping deepfakes facilitate unauthorized identity manipulation. To counteract this threat, many researches [1]–[3] have proposed for detecting deepfake-generated content. However, these retrospective approaches can only mitigate damage after attacks have occurred. To overcome this, recent research has increasingly focused on proactive strategies to disrupt deepfake generation.

Adversarial perturbation has emerged as a proactive protection strategy, disrupting face recognition via either patch-based [4]–[6] or noise-based [7]–[11] methods. Patch-based approaches are lightweight and modular but tend to be visually conspicuous and spatially fixed, rendering them vulnerable to trivial removal [12]. Noise-based methods typically achieve strong protection by applying perturbations over the entire face bounding box. However, this global application leads to inefficient updates and perceptual degradation, as perturbations are dispersed across identity-irrelevant regions. A recent work [13] proposes a per-user strategy that trains identity-specific masks from multiple images. While effective, it requires offline optimization and user-level data, limiting its applicability in per-image protection we target. Makeup-based methods [14]–[17] embed perturbations into visually plausible styles using reference-guided generation. While producing visually natural transformations, these methods often require exaggerated modifications and offer limited spatial control to ensure sufficient protection. Moreover, as most are optimized for impersonation, their applicability to dodging-oriented face-swap deepfake protection remains relatively underexplored.

Another critical limitation of existing methods lies in the definition of perturbation directions in the embedding space. Most prior works define identity-shifting directions using fixed or randomly selected target images [11], [14]–[16], which entangle identity with confounding attributes such as pose or texture. This often results in arbitrarily maximizing latent distances, leading to ambiguous or suboptimal updates that compromise visual quality or fail to ensure sufficient protection. Recent work by [17] proposes an adaptive target selection strategy based on landmark similarity, which partially constrains geometric variation. However, it still struggles to isolate identity-specific changes, often guiding perturbations along directions that include residual non-identity attributes, thereby limiting its effectiveness.

In this paper, we propose \(\textit{\boldsymbol{Phantom}}\), a unified framework for face-swap deepfake protection that jointly performs adversarial optimization under latent and spatial constraints (See Fig. ¿fig:fig:intro?.).

First, we introduce adaptive target synthesis strategy, which generates identity-shifted yet attribute-preserving target faces to define a semantic direction in latent space for adversarial guidance. Unlike prior works [11], [14]–[17] that treat target identities as final objectives, we reinterpret the synthesized image as a means of defining a semantically meaningful identity-discriminative direction in the latent space intertwining geometry, style, and identity cues. Specifically, we extract a semantic mask from the input face and reapply the original style onto it to generate a modified image that retains structural and stylistic consistency while inducing subtle identity shifts. With this adaptively synthesized target, we guide adversarial perturbation generation along identity-sensitive directions, enabling effective identity suppression without unintended attribute distortion.

Second, we formulate a masked adversarial attack based on spatial-constrained optimization. Specifically, we constrain gradient updates to semantically important facial regions using segmentation masks. By explicitly guiding perturbations toward identity-aware regions, this strategy ensures that noise is both compact and effective, minimizing visual artifacts while enhancing protection robustness.

Extensive experiments on CelebA-HQ [18] and LADN [19] datasets validate the effectiveness of \(\textit{\boldsymbol{Phantom}}\) across both dodging and impersonation scenarios. In dodging scenarios, which are our primary focus for defending against unauthorized deepfakes, \(\textit{\boldsymbol{Phantom}}\) achieves state-of-the-art protection against black-box FR models while preserving visual quality. Furthermore, we evaluate whether such protection is valid against face-swapping deepfakes, where adversarial signals including those in stylized or unconstrained formats are often discarded due to the disentanglement of identity and style representations. \(\textit{\boldsymbol{Phantom}}\) achieves protection success rate improvements of 27.8%, 25.6%, and 16.6% over state-of-the-art methods on UniFace [20], INSwapper [21], and SimSwap [22], respectively, indicating strong robustness not only against FR models but also in face-swapping deepfake environments.

Moreover, we demonstrate that \(\textit{\boldsymbol{Phantom}}\) generalizes to impersonation scenarios that induce recognition as a target identity. While the adaptive synthesis module is deactivated in this setting, our spatial optimization remains effective and yields strong protection along with enhanced visual fidelity.

In summary, our contributions are as follows:

• We present \(\textit{\boldsymbol{Phantom}}\), a unified face-swap deepfake protection framework that combines latent- and spatial-constrained optimization, jointly addressing both directional and regional limitations of prior methods.

• We propose a latent-constrained adversarial optimization strategy via adaptive target synthesis that produces identity-shifted yet structure-preserving targets, offering coherent directional guidance in embedding space.

• We introduce a spatial-constrained adversarial optimization strategy using semantically masked gradient updates, enabling compact and visually aligned perturbations for enhanced robustness.

• \(\textit{\boldsymbol{Phantom}}\) outperforms existing methods in protecting against black-box FR models and generalizes effectively to face-swapping deepfakes while improving visual quality. Notably, it achieves up to 98.2% protection success against face-swapping attacks, surpassing prior approaches by at least 16.6%, and also shows competitive performance in impersonation scenarios.

2 Related Work↩︎

2.1 Adversarial Attacks on Face Recognition↩︎

Adversarial attacks are categorized as targeted (impersonation) or non-targeted (dodging). Dodging attacks attempt to conceal the true identity of the source image, forcing recognition models to misclassify it as an unknown or incorrect person. In contrast, impersonation attacks aim to manipulate the image such that it is recognized as a specific target identity, thereby achieving targeted identity transfer.

Noise-based methods typically apply imperceptible perturbations over the entire facial region [7], [8], but often suffer from limited visual quality. Subsequent works have improved robustness through momentum [9] and translation-invariant strategies [10]. TIP-IM [11] employs targeted optimization to generate adversarial identity masks effective against black-box FR models. However, this approach introduces perceptible noise within the face bounding box, thereby degrading perceptual quality.

Makeup-based approaches [14]–[17] embed adversarial perturbations into visually plausible styles using reference images or text prompts. Although [15] addresses both impersonation and dodging, most methods are optimized for impersonation and remain largely unexplored under face-swapping deepfake scenarios. Moreover, these methods offer limited spatial control and high computational cost.

2.2 Face Manipulation↩︎

Face manipulation models aim to generate realistic facial images while controlling attributes such as expression or style. Recent approaches create and leverage semantic segmentation masks to enable structure-aware generation [23], [24]. They synthesize faces conditioned on semantic masks and reference images, allowing localized style transfer. While these models attempt to preserve identity, slight changes can arise during style adaptation. [25] extends this direction by generating identity-diverse images from segmentation masks in a structure-guided manner and optional text prompts for style.

2.3 Face-Swapping Deepfakes↩︎

Face-swapping deepfakes synthesize realistic identity manipulations by disentangling identity features from style (e.g., pose, texture) and recombining them across subjects. Earlier methods such as FaceShifter [26] and SimSwap [22] employ encoder–decoder architectures with FR-guided identity injection, while recent models like UniFace [20] introduce explicit identity and style branches to enhance controllability and fidelity.

3 Methodology↩︎

We propose \(\textit{\boldsymbol{Phantom}}\), a structured adversarial framework for facial privacy protection that proactively defends against black-box FR and face-swapping deepfakes. As illustrated in Fig. 1, \(\textit{\boldsymbol{Phantom}}\) integrates two complementary optimization strategies: a latent-constrained adaptive target synthesis strategy that guides embedding shifts along identity-discriminative directions, and a spatial-constrained masked adversarial attack that restricts perturbations to semantically meaningful regions. Together, these components enable spatially focused and identity-aware perturbations while maintaining high visual fidelity.

3.1 Problem Formulation↩︎

We consider a practical threat scenario where user face images are collected from public sources, such as social media or online platforms, and are exploited without consent. Such images may be used by adversaries for unauthorized face recognition (FR) or identity manipulation via face-swapping deepfakes. Our goal is to proactively defend against both threats by generating visually plausible but identity-disruptive variants.

Let \(x_u\) and \(t_u\) denote the original and target face images of user \(u\), respectively, and let \(x_u^p\) be its protected version. Given a black-box FR model \(f(\cdot)\), we denote the embedding (i.e., the latent representation) of an image \(x\) as \(\mathbf{z}_x = f(x)\). Our goal is to generate \(x_u^p\) such that identity inference is suppressed by disrupting \(\mathbf{z}_{x_u^p}\), while remaining visually similar to \(x_u\). To this end, we define a unified objective that supports both dodging (maximizing deviation from \(\mathbf{z}_{x_u}\)) and impersonation (minimizing distance to \(\mathbf{z}_{t_u}\)), formulated as:

\[\label{eq:general95problem} \begin{array}{c} \displaystyle \max_{x_u^p} \; \mathcal{L}_{adv} = D(\mathbf{z}_{x_u^p}, \mathbf{z}_{x_u}) - D(\mathbf{z}_{x_u^p}, \mathbf{z}_{t_u}) \\[0.8em] \displaystyle \text{s.t.} \quad L_p(x_u^p, x_u) \leq \epsilon \end{array}\tag{1}\]

Here, \(D(\cdot)\) is a similarity metric (e.g., cosine or Euclidean), and \(L_p(\cdot)\) denotes the perceptual distance between \(x_u^p\) and \(x_u\), bounded by a threshold \(\epsilon\) to ensure imperceptibility.

3.2 Adaptive Target Synthesis↩︎

Motivated by the entangled nature of identity, style, and geometry in the latent space of FR models, we seek to isolate identity-discriminative directions while preserving non-identity factors. To this end, we synthesize a target image \(t\) that aligns with the source \(x\) in style and geometry but differs subtly in identity. As depicted in Fig. 2, conventional target selection strategies (e.g., random or fixed) yield perturbations that traverse mixed directions in the latent space, including identity (\(\delta^i = z_t^i - z_x^i\)), style (\(\delta^s = z_t^s - z_x^s\)), and geometry (\(\delta^g = z_t^g - z_x^g\)), which are not optimized solely for identity shift. In contrast, our synthesized \(t\) constrains \(\delta^s\) and \(\delta^g\) near zero by construction, thereby encouraging perturbations aligned predominantly along the identity axis \(\delta^i\). Note that visual similarity between \(x\) and \(t\) is not a concern; rather, it is expected and acceptable, as the objective is not to maximize visual divergence but to extract a semantic direction \(\delta^i\) for identity-axis perturbation. Unlike unconstrained directions that may entangle identity with other attributes, this targeted guidance allows \(\textit{\boldsymbol{Phantom}}\) to suppress identity-discriminative features more effectively with minimal perturbations.

To synthesize a target \(t\) disentangled from \(x\) only in terms of identity, we repurpose face manipulation models. Given an input \(x\), we first extract a semantic mask \(m = G_S(x)\) using a segmentation model \(G_S\). This mask is reused without modification to retain spatial layout. A mask-guided face generation model \(G_F\) then combines \(x\) and \(m\) to produce the synthesized target: \[\label{eq:atsyn} t \gets G_F(x, G_S(x))\tag{2}\] Although \(G_F\) is originally intended for tasks such as retouching or style transfer, it introduces subtle identity shifts while maintaining structure. We reinterpret these changes as semantically aligned traversals in latent space that serve as directional anchors for identity-axis optimization.

3.3 Masked Adversarial Attack↩︎

We identify an optimal perturbation region that maximizes protection against both FR models and face-swapping while minimizing perceptual degradation. Since the spatial placement of adversarial perturbation critically impacts both effectiveness and visual quality, selecting the effective region is essential.

To this end, we empirically evaluate various perturbation scopes including the entire image, full face region, non-face areas (e.g., background, hair), and localized facial components (e.g., eyes, nose, mouth). As detailed in the Supplementary Material, perturbations confined to the face region yield the best trade-off, offering strong protection with minimal artifacts. In contrast, applying perturbations to irrelevant or overly localized regions reduces effectiveness or introduces visible distortions. Based on this analysis, we adopt the face region as our selective region for adversarial optimization.

We formulate a proactive adversarial attack strategy that suppresses identity features in facial images, thereby degrading their utility for downstream recognition or swapping tasks. \(\textit{\boldsymbol{Phantom}}\) combines adaptive target synthesis with spatially constrained perturbation to generate identity-obfuscating perturbations confined to semantically meaningful regions. This enables effective protection under black-box conditions, including unseen FR models or deepfake generators, while preserving image fidelity. The overall objective consists of identity-oriented and quality-preserving terms: \[\label{eq:loss} \begin{align} \mathcal{L}_{total} = \lambda_{id}\lambda_{ori}\mathcal{L}_{ori} + \lambda_{id}\lambda_{tgt}\mathcal{L}_{tgt} + \lambda_{qual}\mathcal{L}_{qual} \end{align}\tag{3}\] \(\mathcal{L}_{ori}\) and \(\mathcal{L}_{tgt}\) represent cosine similarity losses against the original and target embeddings, computed using an ensemble of surrogate FR models to improve transferability. \(\mathcal{L}_{qual}\) enforces perceptual fidelity using pixel-wise MSE loss, and helps bound the perturbation strength by limiting the maximum per-pixel deviation to \(\epsilon\).

We apply per-iteration clamping using a binary mask to constrain adversarial updates within the predefined region. Specifically, a binary mask \(B_u \in {0,1}^{H \times W}\) is constructed based on the selected facial region. During optimization, we compute the gradient \(g\) of the total loss with respect to the adversarial perturbations \(n_{adv}\) and apply a Hadamard product with the mask: \[\label{eq:hadmard} \begin{align} g^* = B_u \odot g, \quad \text{s.t.} \quad g = \nabla_{n_{adv}} \mathcal{L}_{total} \end{align}\tag{4}\] This yields compact, task-aware perturbations that are both visually natural and effective against unseen FR models or deepfake generators.

While \(\textit{\boldsymbol{Phantom}}\) primarily targets dodging scenario (e.g., avoiding unauthorized FR models or deepfakes), it can be naturally extended to impersonation settings. In such cases, adaptive target synthesis is not required. Instead, a specific identity image can be directly selected as \(t_u\) to steer the embedding toward a desired target identity. The rest of the masked adversarial optimization framework remains unchanged.

The full algorithm of \(\textit{\boldsymbol{Phantom}}\) is provided in the Supplementary Material.

4 Experiments↩︎

4.1 Experimental Settings↩︎

We set \(iter{=}200\), \(\epsilon{=}16\), and use the Adam optimizer with a learning rate of \(5\mathrm{e}{-4}\), \(\beta_1{=}0.9\), and \(\beta_2{=}0.999\). Loss weights are configured as \((\lambda_{id}, \lambda_{qual}){=}(1.0, 1.0)\); for dodging, \((\lambda_{ori}, \lambda_{tgt}){=}(1.0, 0.5)\), and for impersonation, \((0.0, 1.0)\). We use \(G_S\) and \(G_F\) from [23], which preserve both style and geometry. To assess robustness under different synthesis characteristics, we also evaluate \(G_F\) from [25], which preserves geometry only.

Following [14], [16], [17], we use the CelebA-HQ [18] and LADN [19] datasets. For CelebA-HQ, we select 1,000 images from [14] and use 930 image pairs with corresponding counterparts. One pair is randomly selected per image based on official pair information. For LADN, we evaluate on the same 334 images used by existing methods to ensure a fair comparison.

Following [14]–[17], we evaluate \(\textit{\boldsymbol{Phantom}}\) against four FR models with different backbone architectures: IRSE50 [27], IR152 [28], FaceNet [29], and MobileFace [30]. Before feeding images into these FR models, we detect faces using RetinaFace [31] implemented in the InsightFace [21].

We evaluate our approach against face-swapping deepfake attacks using three state-of-the-art tools: INSwapper [21], deployed in Picsi.Ai [32], UniFace [20], and SimSwap [22]. Each tool utilizes a different pretrained swapper model with unique resizing configurations in pre-processing. See the Supplementary Material for details.

We compare our approach against representative benchmark approaches. Perturbation-based methods include MI-FGSM [9], TI-DIM [10], and TIP-IM [11]. Makeup-based methods include AMT-GAN [14], CLIP2Protect [15], DiffAM [16], and RMT-GAN [17]. All methods adopt the ensemble strategy from [14] for fair comparison. For the dodging scenario, methods without explicit support for dodging-specific optimization (e.g., AMT-GAN [14], DiffAM [16]) are evaluated using their official implementations and recommended hyperparameters from the original papers. Details in the Supplementary Material.

To assess protection effectiveness, we report Rank-1/5 protection (R1-P/R5-P) and Protection Success Rate (PSR) under FAR@0.01. R1-P/R5-P measure how well protection methods prevent the ground-truth pair from appearing in the top-N predictions on the FR model, while PSR quantifies how often a deepfake image fails to match the true identity under a fixed FAR. For PSR, the model-specific thresholds are 0.241 (IRSE50), 0.167 (IR152), 0.409 (FaceNet), and 0.302 (MobileFace). The Attack Success Rate (ASR) is defined as \((1 - \text{PSR})\) for a target identity. For image quality, we use PSNR, SSIM [33], and FID [34]. Higher R1/R5 and PSR indicate stronger protection; higher PSNR and SSIM, and lower FID indicate better visual quality.

In all experiments, we evaluated on an NVIDIA A100 GPU with 80GB of memory.

4.2 Dodging Scenario Evaluation↩︎

We evaluate \(\textit{\boldsymbol{Phantom}}\) in a black-box setting using four unseen FR models. As shown in Tab. 1, \(\textit{\boldsymbol{Phantom}}\) achieves an average protection of 95.0% in R1-P and 89.6% in R5-P, outperforming existing methods by 9.1% and 18.2%, respectively. This confirms strong identity obfuscation across diverse FR architectures. AMT-GAN and DiffAM show limited performance due to their impersonation-oriented design. Tab. 2 reports SSIM, PSNR, and FID scores, showing that \(\textit{\boldsymbol{Phantom}}\) preserves perceptual quality (SSIM: 0.973, PSNR: 33.646, FID: 18.705) while maintaining strong protection.

We further assess robustness under face-swapping pipelines, where identity and style are explicitly disentangled. Tab. 3 shows adversarial perturbations embedded in peripheral regions or stylistic forms (e.g, makeup), as in prior methods, often struggles to persist through the face swap, leading to reduced protection effectiveness. In contrast, \(\textit{\boldsymbol{Phantom}}\) constrains perturbations to identity-relevant facial regions and optimizes them along identity-aware axes, maintaining their effect even after identity-style disentanglement. This allows \(\textit{\boldsymbol{Phantom}}\) to consistently outperform baselines across face-swapping pipelines. Specifically, we observe substantial PSR improvements of 27.8%, 25.6%, and 16.6% over the best-performing baseline on CelebA-HQ, and 22.4%, 20.2%, and 17.4% on LADN against UniFace, INSwapper, and SimSwap, respectively. Notably, the largest gains against UniFace, known for its dedicated identity disentanglement module, highlight the robustness of our latent- and spatial-constrained strategies. Meanwhile, PSR scores on INSwapper are relatively lower than on other pipelines. Nonetheless, \(\textit{\boldsymbol{Phantom}}\) consistently maintains superior protection across all deepfakes and datasets, as shown in Fig. 3.

These results reaffirm a key limitation of impersonation-driven protection such as AMT-GAN and DiffAM: increasing similarity to a target identity does not ensure sufficient separation from the original one. Face swap makes this gap more evident, underscoring the need for dodging-oriented protection that explicitly maximizes identity separation.

To evaluate the robustness of post-processing transformations, we apply five levels of JPEG compression, Gaussian blur, and resizing to the protected images, and evaluate cosine similarity using black-box FR models. \(\textit{\boldsymbol{Phantom}}\) consistently exhibits minimal degradation, with cosine similarity increasing by no more than 10% under all types of distortion. Notably, in the resizing, the similarity loss remains below 1% across all scales. This robustness stems from applying perturbations at a low-resolution stage and spatially aligning them with the original image, making them less vulnerable to downstream transformations while preserving their spectral effect. These results highlight the robustness of our method against common post-processing distortions. See Supplementary Material for visual examples and detailed results.

4.3 Impersonation Scenario Evaluation↩︎

We evaluate \(\textit{\boldsymbol{Phantom}}\) under the impersonation setting by fixing a target identity and disabling adaptive synthesis. As shown in Tab. 4 and Tab. 5, \(\textit{\boldsymbol{Phantom}}\) consistently outperforms impersonation-optimized baselines on both CelebA-HQ and LADN in terms of ASR and perceptual quality. Notably, on LADN, it achieves at least a 10.2% absolute PSR gain over all compared methods. These results indicate that our spatially constrained masked optimization generalizes effectively to impersonation scenarios without architectural changes, surpassing baselines tailored for impersonation. Compared to the dodging setting, \(\textit{\boldsymbol{Phantom}}\) exhibits notably higher visual fidelity under impersonation, suggesting that embedding target-specific identity cues requires milder perturbations than suppressing the source identity.

4.4 Ablation Study↩︎

We investigate how perturbation placement affects protection and visual quality by disabling target-guided optimization (\(\lambda_{tgt}=0\)) and varying spatial constraints.

Tab. 6 compares four configurations: full face bounding box (baseline), non-face region, face region via post hoc masking, and our masked adversarial attack that constrains optimization to the selective face region. The baseline achieves strong protection (R1-O: 85.8%) but degrades visual quality (FID: 35.1). Conversely, non-face perturbation yields minimal protection (R1-O: 0.0%) despite better quality, confirming that most discriminative features lie in facial regions. Post hoc masking yields similar visual quality but leads to reduced protection (R1-O: 72.1%).

Our approach achieves a better trade-off (R1-O: 84.6%, FID: 24.6) by constraining gradients during optimization, not after. This enforces spatial alignment and compact perturbations, preserving both robustness and visual fidelity. These results highlight the importance of integrating spatial priors directly into the optimization process.

We evaluate \(\textit{\boldsymbol{Phantom}}\) under four adversarial target strategies: Fixed, a shared target across all images adopted in most existing methods; AT-Sel, an adaptive selection approach based on landmark and identity similarity as proposed in RMT-GAN; AT-Syn, our adaptive target synthesis strategy that leverages a generative model from [23]; and AT-Syn(*g), a variant of AT-Syn without style conditioning, following [25].

As shown in Tab. 7, synthesis-based targets yield lower NME and variance, indicating superior geometry consistency, whereas selection-based targets lack such control and exhibit higher variability. In particular, AT-Syn can remain close to the source and within recognition bounds by design. Rather than generating a drastically different visual identity, AT-Syn defines a semantically meaningful direction in latent space to guide perturbation learning. Leveraging this structured signal, \(\textit{\boldsymbol{Phantom}}\) generates identity-disruptive perturbations more effectively through adversarial amplification. As shown in Tab. 8, AT-Syn achieves superior perceptual quality over all alternatives, including its geometry-only variant, without sacrificing protection. Fig. 4 further illustrates that AT-Syn yields cleaner outputs, highlighting the benefit of structure- and style-aware synthesis in guiding effective and visually coherent perturbations.

Fig. 5 illustrates that as \(\epsilon\) increases, facial distortion becomes more pronounced, yet protection against face swap improves owing to enhanced identity obfuscation. This trade-off highlights that \(\textit{\boldsymbol{Phantom}}\) allows controllable balancing between visual fidelity and protection strength. See the Supplementary Material for detailed results w.r.t. \(\epsilon\) and iteration.

To further analyze robustness under imperfect masking (e.g., due to occlusion or extreme pose), We partition the evaluation set into two splits: \(\mathcal{D}_{sts}\) (imperfect masking stress set, at least one missing face element) and \(\mathcal{D}_{std}\) (reliable masking standard set, the remainder). \(\mathcal{D}_{sts}\) exhibits greater perceptual drift and weaker protection, consistent with minor spatial misalignment and slightly contaminated geometry cues that yield mildly off-identity targets and localized visible noise. Nevertheless, our method outperforms baselines because most facial regions remain covered, keeping the noise budget on salient ROIs, and the synthesized target is better aligned in style and geometry than random or fixed targets, helping preserve identity-axis guidance. See the Supplementary for qualitative and quantitative analyses.

5 Conclusion↩︎

References↩︎