1 Introduction↩︎

Photorealistic 3D face avatars are rapidly becoming reusable digital assets. Modern pipelines reconstruct detailed 3D heads from short videos or even single images, enabling talking-head synthesis, virtual telepresence, personalized media, and immersive applications[1]–[3]. Unlike a static photograph, a 3D face model is persistent: it can be rendered under arbitrary viewpoints and lighting conditions long after its creation. At the same time, Vision–Language Models (VLMs), including both commercial foundation models such as GPT-4 or Gemini [4], [5] and open source models like VideoLLaMA3 or LLaVa-NeXT [6], [7], have evolved into open-ended semantic reasoners capable of inferring attributes such as age, ethnicity, gender, and expression from any rendered image without task-specific training[8], [9]. Together, these developments create a structural privacy risk: once a 3D face avatar is released, every possible rendering becomes analyzable by powerful semantic models, exposing sensitive attributes beyond simple identity recognition [10].

Existing defenses are fundamentally mismatched to this setting. Most face anonymization and adversarial cloaking methods operate in 2D image space, protecting only a specific rendering rather than the underlying 3D asset. A new viewpoint instantly bypasses such defenses. While recent adversarial work has explored perturbations in 3D representations, these approaches primarily target identity detectors and do not address identity-preserving semantic manipulation in VLM embedding space [11]–[14].

To this end, we propose 3D FaceShell, a framework that steers VLM interpretations of 3D face avatars while preserving the underlying facial identity and appearance as highlighted in Fig. 1. Our objective is to alter specific attributes inferred by VLMs from a 3D face avatar while preserving the underlying facial identity and geometry. Unlike generic 3D objects, facial avatars are designed to maintain a stable and recognizable digital likeness for legitimate uses such as telepresence, animation, and personalized media. Any intervention must therefore balance two competing requirements: it must be strong enough to redirect high-level semantic interpretation in the VLM embedding space, yet subtle enough to preserve geometric structure and identity-defining appearance across viewpoints. Naive perturbations that visibly distort shape or degrade texture undermine the asset’s utility. The challenge is thus not merely to attack a model, but to manipulate semantic inference in a way that remains perceptually faithful and multi-view consistent.

To realize this objective, 3D FaceShell operates directly in the parameter space of 3D Gaussian Splatting (3DGS) face avatars. Starting from a pre-optimized facial representation, we introduce a set of auxiliary Gaussians positioned near the facial surface, forming a learnable Gaussian shell. The baseline Gaussians remain fixed, separating adversarial perturbations from underlying geometry. This approach enables semantic manipulation while, by design, never deforming facial structure. The auxiliary shell is optimized to produce subtle, spatially distributed perturbations that are minimally perceptible in rendered views yet sufficient to shift VLM embeddings toward a targeted semantic interpretation. During training, multiple rendered views are passed through a pretrained VLM, and the shell parameters are updated via multi-view embedding alignment against a pose-consistent, attribute-altered target. This design ensures that the semantic manipulation is both view-consistent and visually inconspicuous.

Through extensive evaluation on reconstructed celebrity face avatars and multiple opaque-box VLMs, we show that 3D FaceShell consistently increases attribute injection and mismatch rates while maintaining high perceptual similarity and identity consistency. Compared to state-of-the-art 2D semantic attack baselines, 3D FaceShell achieves stronger cross-view robustness and substantially better preservation of facial structure, demonstrating that semantic steering and visual fidelity need not be mutually exclusive.

We summarize our contributions as follows:

• We introduce and formalize the problem of manipulating VLM perception of semantic face attributes for 3D face avatars under strict identity and geometric preservation constraints. Defending against VLM inference for 3D avatars is entirely new, differing fundamentally from prior 3D adversarial attacks that target face detectors and more general adversarial attacks on VLMs that do not preserve face quality or identity.

• We propose 3D FaceShell, a novel additive Gaussian shell framework that injects spatially distributed, minimally perceptible perturbations into 3DGS face representations and optimizes them through multi-view embedding alignment in the vision encoder space to manipulate VLM perception. This design enables view-consistent semantic steering while leaving the baseline geometry untouched.

• We demonstrate strong cross-view robustness and improved identity preservation compared to state-of-the-art 2D semantic attack baselines across multiple opaque-box VLMs, showing that semantic manipulation and perceptual fidelity can be achieved simultaneously in 3D.

2 Related Work↩︎

2.1 3D Gaussian Splatting↩︎

The introduction of 3DGS has established a highly efficient paradigm for real-time radiance field rendering[15]. By utilizing millions of 3D Gaussians characterized by learnable opacity, scale, rotation, and spherical harmonics, 3DGS achieves state-of-the-art view synthesis without the massive computational bottlenecks of earlier methods [16]–[19]. Given this explicit editability and rendering speed, 3DGS has been widely adopted for modeling dynamic and highly detailed human face and body avatars. Recent frameworks [1]–[3] enable the generation of full body or face avatars from 2D video or images. However, as these 3D head avatars become increasingly common and easily shareable digital assets, the underlying geometry and identity textures are left inherently exposed to malicious downstream processing and automated semantic parsing by VLMs[20]–[23]. This danger necessitates privacy defenses for 3D face avatars.

2.2 Adversarial Attacks on 3D Models↩︎

As the use of 3D representations has become more common, the vulnerabilities of 3D representations to adversarial manipulation have become a critical security domain [24], [25]. Discrete 3D structures, such as point clouds, are highly susceptible to targeted perturbations either through point shifting or malicious point generation [26]. However, adversarial attacks in the 3D domain are not limited to explicit point sets. Implicit representations like Neural Radiance Fields (NeRFs) are vulnerable to targeted perturbations that hallucinate or obscure objects during view synthesis[27]. Recently, this threat has been explicitly extended to 3DGS. Recent works inject adversarial noise in the 3DGS space to fool object detectors or vision encoders [24], [28]. Unlike these methods which shift overall semantic representations in object detectors or vision encoders for objects, we aim instead to shift VLM understanding of a more complex domain, faces, targeting specific attributes for transfer rather than broader semantics of the 3D representation.

Attacks on 3D Face Avatars. Biometric authentication systems are very frequently targeted by adversarial attacks leading to the development of specialized physical and 3D face attacks that bypass conventional defenses [11]–[14]. These methods, however, focus on attacks utilizing physical 3D masks or traditional methods of 3D representation like point clouds. In contrast, our 3D FaceShell proposes an attack on 3DGS face avatars, utilizing exclusively additive noise which leaves the underlying face representation undisturbed.

2.3 Adversarial Attacks on VLMs↩︎

Due to their extensive cross-modal training, foundation VLMs possess powerful reasoning capabilities but remain highly susceptible to visual adversarial attacks[29]. Images with adversarial noise applied to them can bypass safety guardrails and execute multimodal jailbreaks, coercing models into generating harmful or unintended text [30], [31]. The threat landscape includes both targeted attacks on specific commercial systems [32] along with targeted and untargeted transferable frameworks designed to disrupt generalized vision-language alignment across a variety of VLMs [33]–[37]. 3D FaceShell differs from these by moving the problem of attacking VLMs with adversarial noise into the 3D space and focusing on shifting the attributes of a face representation while still maintaining a high level of detail and realism to retain the perceptual and identity similarity relative to the source face.

3 Problem Definition↩︎

We represent a 3D face using 3DGS as a set of \(N\) anisotropic Gaussians \(\mathcal{G} = \{g_i\}_{i=1}^{N}\), where each Gaussian \(g_i\) is parameterized by its spatial mean \(\mathbf{x}_i \in \mathbb{R}^3\), covariance \(\Sigma_i \in \mathbb{R}^{3 \times 3}\), and opacity \(\alpha_i \in \mathbb{R}\). Let \(R(\mathcal{G}; \pi)\) denote the differentiable renderer that produces a 2D image from \(\mathcal{G}\) under camera parameters \(\pi\). Simultaneously, we consider a pretrained VLM \(f_{\theta}(\cdot)\), composed of a visual encoder \(\mathrm{VE}(\cdot)\) followed by a large language model \(\mathrm{LLM}(\cdot)\): \(f_{\theta}(x) = \mathrm{LLM}(\mathrm{VE}(x)),\) where \(\theta\) denotes fixed model parameters.

Then, given a source 3D face \(\mathcal{G}_s\) with ground-truth semantic description \(c_{gt}\), our objective is to learn an additive perturbation \(\delta\) in the Gaussian parameter space, yielding a modified 3D representation \(\mathcal{G}_t = \mathcal{G}_s + \delta\), such that rendered views \(R(\mathcal{G}_t; \pi)\) are semantically aligned with a predefined target description \(\mathcal{T}_{tar}\) (targeted attack), while preserving the original facial identity and geometric structure. Under a white-box setting with access to \(f_\theta\), the optimization problem is formulated as: \[\min_{\delta} \; \mathcal{L}_{adv}\big(f_\theta(R(\mathcal{G}_s + \delta; \pi)), \mathcal{T}_{tar}\big) + \lambda \, \mathcal{L}_{reg}(\mathcal{G}_s, \mathcal{G}_s + \delta), \label{objective}\tag{1}\] where \(\mathcal{L}_{adv}\) drives semantic alignment with the target description in the VLM output space, and \(\mathcal{L}_{reg}\) constrains the perturbation to preserve geometric fidelity, identity consistency, and photorealistic rendering quality.

4 Method: 3D FaceShell↩︎

We propose 3D FaceShell, an identity-preserving framework for learning additive adversarial perturbations directly in the parameter space of 3DGS facial representations to induce targeted semantic misalignment in pretrained VLMs.

The proposed framework consists of four key components: (a) Multi-attributed target image generation, which synthesizes pose-aligned target faces with semantically opposing attributes to guide embedding manipulation; (b) Gaussian shell initialization, where a secondary set of auxiliary Gaussians is introduced to parameterize controlled additive perturbations without modifying the baseline geometry; (c) Multi-view adversarial training, which aligns rendered views of the perturbed 3D model with target embeddings under stochastic augmentations for robustness; and (d) Face-aware regularization mechanisms, which constrain perturbations in identity-sensitive regions and suppress unnatural luminance artifacts. Together, these components enable semantically effective yet structurally consistent adversarial manipulation of 3D faces in VLM embedding space.

4.1 Multi-Attributed Target Image Generation↩︎

To construct supervision for the adversarial objective defined in Sec. [problem95definition], we synthesize target images that reflect the desired semantic manipulation \(c_{tar}\). Given an input rendering \(R(\mathcal{G}_s, \pi)\), we first extract its pose representation using OpenPose[38], producing a structural pose map \(\mathbf{P}\) that preserves geometric configuration while disentangling appearance. We then condition a pretrained text-to-image diffusion model on \(\mathbf{P}\) together with a text prompt \(\mathcal{T}'_{tar}\) encoding a set of desired facial attributes. This process is illustrated in Fig. 2. The target prompt is constructed using a multi-attribute composition strategy, where selected attributes are deliberately chosen to be semantically different to those of the original face. While the user may explicitly specify a subset of attributes to modify, the remaining facial attributes are automatically initialized with values opposite to the source description. This multi-attribute formulation encourages a stronger shift in the VLM embedding space, as aligning multiple semantic factors simultaneously produces a more pronounced deviation toward \(\mathcal{T}_{tar}\) compared to single-attribute manipulation, analogous to multi-task supervision. Finally, the synthesized image \(I_{tar}\) serves as a semantic anchor for guiding the optimization of the Gaussian shell parameters.

Thus, we reformulate the VLM text-injection objective into a target image injection objective, replacing purely text-driven alignment with embedding-space supervision via a synthesized target image. If \(x = R(\mathcal{G}_s; \pi)\) denote the original rendering and \(x_\delta = R(\mathcal{G}_s + \delta; \pi)\) the perturbed rendering, then we re-formulate Eq. 1 as: \[\min_{\delta} \; \mathcal{L}\big(f_\theta(x_\delta), f_\theta(I_{\mathrm{tar}})\big) \quad \text{s.t.} \quad I_{\mathrm{tar}} \neq x,\] where \(\mathcal{L}\) denotes a similarity measure in the VLM embedding space. The perturbation \(\delta\) is optimized such that the embedding of the rendered adversarial image aligns with that of the target image, while remaining visually and geometrically consistent with the original face. This formulation of 3D FaceShell explicitly drives the VLM representation of the perturbed face toward a pose-consistent yet semantically divergent target embedding.

4.2 Gaussian Shell Initialization↩︎

For the adversarial noise training, we initialize from a pre-optimized 3DGS representation \(\mathcal{G}_s\) consisting of \(N_{\text{base}}\) Gaussians while keeping all baseline parameters fixed throughout optimization to preserve geometry and identity. We instantiate a secondary set of \(N_{\text{shell}}\) auxiliary Gaussians, referred to as a Gaussian shell to parameterize the additive perturbation \(\delta\). Each Gaussian shell is initialized by randomly sampling the spatial mean of a baseline Gaussian and applying a small uniform perturbation \(\epsilon \sim \mathcal{U}([-0.01, 0.01]^3)\), ensuring close spatial proximity to the original surface. The covariance, rotation, and spatial parameters of the shell Gaussians are frozen after initialization. Also, optimization is restricted to their appearance attributes, specifically color and opacity, which together define the additive perturbation \(\delta\) in the rendered space. This constrained parameterization enforces that adversarial manipulation operates purely through localized appearance and density modulation, yielding semantically effective yet geometrically non-destructive perturbations of the underlying 3D face.

4.3 Multi-View Adversarial Noise Training↩︎

At each optimization step of 3D FaceShell, the perturbed 3D representation \(\mathcal{G}_s + \delta\) is rendered into \(K\) differentiable views \(\{x_k\}_{k=1}^{K}\) using a predefined camera set \(\Pi = \{\pi_k\}_{k=1}^{K}\), where each camera \(\pi_k\) is parameterized by azimuth \(\theta_k\), elevation \(\phi_k\), radius \(r\), and field of view \(\psi\). Formally, \[x_k = R(\mathcal{G}_s + \delta; \pi_k)\] The adversarial objective of 3D FaceShell is computed over a fixed training subset of \(\Pi\) to enforce cross-view consistency of the perturbation in 3D space. We apply stochastic augmentations to each rendered view prior to encoding to prevent overfitting to specific viewpoints. Specifically, we apply a set of augmentations, denoted by \(\mathcal{A}(\cdot)\), simulate 8-bit image discretization via quantization noise and apply random resized crops and color jittering. The augmented rendered views \(x_k\) and the synthetic multi-attribute target image \(I_{\mathrm{tar}}\) are processed by the visual encoder \(\mathrm{VE}(\cdot)\) of the pretrained VLM \(f_\theta\) to obtain their corresponding feature embeddings: \(z_k = \mathrm{VE}(\mathcal{A}(x_k))\) and \(z_{\mathrm{tar}} = \mathrm{VE}(\mathcal{A}(I_{\mathrm{tar}}))\). Then, we optimize \(\delta\) to align these feature embeddings to guide the attribute transfer through image injection. In practice, the perturbation \(\delta\) is optimized using features extracted from \(E\) different visual encoders, corresponding to the visual backbones of multiple VLMs. This multi-encoder supervision encourages consistent feature alignment across diverse representations, improving robustness and enabling the learned perturbations to generalize in a model-agnostic manner. Thus, the feature alignment loss aggregates over all training views as \[\mathcal{L}_{\mathrm{feature}} = \frac{1}{EK}\sum_{e=1}^{E}\sum_{k=1}^{K} \Big(1 - \mathrm{sim}\big(z_k, z_{\mathrm{tar}}\big)\Big), \label{eq:feature95loss}\tag{2}\] where \(\mathrm{sim}(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}^\top \mathbf{b}}{\|\mathbf{a}\|_2 \|\mathbf{b}\|_2}\) denotes cosine similarity. The overall optimization jointly minimizes \(\mathcal{L}_{\mathrm{feature}}\) and regularization terms on the shell parameters, yielding semantically targeted yet geometrically consistent adversarial perturbations across viewpoints.

4.4 Face-Aware Regularization Mechanisms↩︎

Finally, we introduce face-aware regularization terms to preserve identity and photorealism of the underlying 3D face. While \(\mathcal{L}_{\mathrm{feature}}\) drives semantic embedding shifts, the following constraints suppress unnecessary perturbations and ensure that the learned noise remains localized and structurally consistent.

4.4.0.1 Landmark Consistency.

In order to protect identity-critical regions, we extract fixed binary masks \(M\) corresponding to sensitive facial areas (e.g., eyes, nose, and lips) from the baseline renders. Let \(I_k\) and \(I^{\mathrm{base}}_k\) denote the current and baseline renders at view \(\pi_k\), respectively. We penalize pixel-level deviations within the masked regions across all training views: \[\mathcal{L}_{\mathrm{landmarks}} = \frac{1}{K} \sum_{k=1}^{K} \left\| M \odot (I_k - I^{\mathrm{base}}_k) \right\|_2^2, \label{eq:landmark95loss}\tag{3}\] where \(\odot\) denotes element-wise multiplication. This constraint discourages distortions in identity-sensitive areas while allowing controlled modifications elsewhere.

4.4.0.2 Chroma-Constrained Texture Regularization.

Texture-based adversarial perturbations often manifest as high-frequency luminance artifacts. We mitigate this effect and encourage smooth semantic transfer by mapping the additive texture parameters of the auxiliary shell Gaussians directly into the YUV color space, emphasizing our regularization in the luminance channel. Let \((Y_i, U_i, V_i)\) denote the YUV decomposition of the explicit texture noise applied to the \(i\)-th shell Gaussian, out of \(N_{\textrm{shell}}\) total auxiliary Gaussians. We define: \[\mathcal{L}_{\mathrm{chroma}} = \frac{1}{N_{\text{shell}}} \sum_{i=1}^{N_{\text{shell}}} \left( \lambda_{Y} Y_i^2 + U_i^2 + V_i^2 \right), \label{eq:chroma95loss}\tag{4}\]

4.5 Overall Optimization of 3D FaceShell↩︎

The optimization of 3D FaceShelljointly balances semantic embedding manipulation and face-aware regularization in the 3D Gaussian shell parameter space as illustrated in Fig. 3. At each iteration, gradients from the multi-view feature alignment loss are backpropagated through the differentiable renderer to update only the shell parameters (color and opacity), while the baseline Gaussians remain frozen. The overall objective combines latent feature matching with structural and photometric constraints: \[\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{feature}} + \lambda_{\mathrm{chroma}} \mathcal{L}_{\mathrm{chroma}} + \lambda_{\mathrm{landmarks}}\mathcal{L}_{\mathrm{landmarks}}. \label{eq:total95loss}\tag{5}\] where \(\lambda_{\mathrm{chroma}}\) and \(\lambda_{\mathrm{landmarks}}\) are the strengths of landmark consistency and luminance suppression, respectively. Together, these losses constrain the adversarial optimization to produce semantically effective yet visually coherent perturbations, preserving facial identity and structural realism across viewpoints.

5 Experiments↩︎

5.1 Implementation Details↩︎

For the adversarial feature alignment in 3D FaceShell, we use a multi-encoder training with \(E=2\) distinct vision encoders: SigLIP (ViT-SO400M-Patch14)[39] and CLIP (ViT-Large-Patch14-336)[40]. For stable optimization, the weight of the chroma-constrained texture regularization is dynamically annealed using a cosine progression schedule. If \(p = \frac{i}{T}\) denote the normalized training progress at iteration \(i\) out of \(T\) total iterations, and define the cosine progression factor \(C = \frac{1}{2}(1 - \cos(\pi p))\). The iteration-dependent weight is then given by \(\lambda_{\mathrm{chroma}} = \lambda_0 C\), where \(\lambda_0\) is a constant scaling factor. Additionally, we further regularize the optimization and prevent opacity-driven geometric artifacts by imposing a progressively tightening upper bound on the shell Gaussian opacity. The maximum allowable opacity is dynamically clamped as \(O_{\max} = -0.5 - 2.5C.\) As training progresses, this schedule gradually pushes Gaussian shells toward increased transparency, suppressing perturbations that do not meaningfully contribute to the semantic objective. This progressive clamping acts as an additional geometric regularizer, ensuring that the adversarial signal remains minimally intrusive while preserving structural fidelity of the underlying 3D face.

Hyperparameters. Unless otherwise specified, rendered images are generated at a resolution of \(384 \times 384\) with a field of view \(\psi = 50^\circ\) and a fixed camera radius \(r = 2.7\). We use \(K=5\) fixed camera poses for optimization: frontal \((270^\circ, 0^\circ)\), slight left \((255^\circ, 0^\circ)\), slight right \((285^\circ, 0^\circ)\), slight up \((270^\circ, 15^\circ)\), and slight down \((270^\circ, -15^\circ)\). To evaluate cross-view generalization, we introduce a novel validation view at \((285^\circ, 15^\circ)\), which is excluded from optimization and used exclusively for evaluation. We optimize the color and opacity parameters of \(N_{shell}=50,000\) shell Gaussians using the Adam optimizer for 10,000 total iterations, setting the learning rate to 0.1 for both parameter groups. For our face-aware regularization mechanisms, the landmark consistency loss weight is fixed at \(\lambda_{\mathrm{landmarks}}=500\). In the chroma-constrained texture regularization, the luminance penalty is enforced with \(\lambda_Y=10.0\) while we scale our overall chroma-constrained texture loss with \(\lambda_{\mathrm{chroma}}=0.4\). Finally, during the multi-view adversarial noise training, the stochastic augmentations \(\mathcal{A}(\cdot)\) applied to the rendered views consist of simulated 8-bit quantization noise (uniformly sampled between ± 2/255), a random resized crop with a scale factor between 0.8 and 1.0, and random color jittering adjusting brightness and contrast by up to 10%.

5.2 Evaluation Settings↩︎

Dataset. In order to evaluate our method and the competing baselines, we construct a benchmark consisting of 100 celebrity faces curated from the IMDB-WIKI dataset [41]. For each subject, the facial attributes eye color, hair color, hair type, and expression are manually annotated, while age, gender, and ethnicity are obtained from publicly available metadata corresponding to the time the source photograph was captured. Each 2D face image is then converted into a 3D Gaussian Splatting (3DGS) representation using FaceLift [1]. The resulting benchmark, comprising 3D faces paired with curated attribute annotations, will be publicly released to facilitate further research in this direction.

Metrics. We assess the capabilities of each attack method to successfully transfer target attributes onto rendered views of each source 3D face avatar. We provide the rate at which a given VLM outputs target attributes when provided with a rendered view of the source face (Injection Rate) and the rate at which a given VLM outputs an attribute different from the ground truth attribute of a given source face when provided with a rendered view of that source face (Mismatch Rate). Importantly, we assess the quality of the outputs for each method using LPIPS [42] to measure the perceptual distance between the synthesized images and the ground truth references. Finally, we utilize ArcFace [43] to determine the identity similarity between the generated face views produced by each method and the faces in the ground truth images.

Selection Criteria. To determine the optimal stopping point for our adversarial optimization and ensure consistent perceptual fidelity, we evaluate the training progression on a validation set consisting of 20 separate faces. As illustrated in Fig. 4, the optimization process reveals a clear trade-off between the VLM injection accuracy and the perceptual quality (LPIPS) over the course of 10,000 training iterations. To ensure our method balances high visual fidelity with maximum semantic manipulation, we apply a strict threshold-based selection criterion for our final evaluations. For each source face, we extract the rendering from the iteration that yields the highest injection accuracy, strictly bottlenecked by the requirement that the LPIPS score must remain below 0.165. In edge cases where the optimization fails to produce any samples under this 0.165 threshold, we default to selecting the iteration that achieved the absolute lowest LPIPS score overall. This guarantees that the final perturbed 3D avatars remain perceptually grounded and structurally consistent, preventing unbounded visual degradation while still capturing the peak adversarial transfer rate.

Models. We evaluate all methods on four black-box VLMs: VideoLLaMA3 [6], LLaVA-NeXT [7], LLaVA-OV [44], and BLIP-2 [45]. These models are selected due to their strong multimodal reasoning capabilities and the diversity of their visual encoders. In particular, BLIP-2 employs a CLIP-based visual encoder, whereas the others utilize SigLIP-based visual encoders. Baselines. As there are no other methods for adversarial semantic editing in the 3DGS space, we benchmark 3D FaceShell against six state-of-the-art 2D adversarial attacks targeting VLMs: AnyAttack [34], V-Attack Ensemble [33], AdvDiffVLM [36], SSA-CWA [32], M-Attack [37], and AttackVLM [35]. We selected these specific baselines because they represent a comprehensive cross-section of the most effective modern 2D attack paradigms, including diffusion-based, self-supervised, and highly transferable embedding-space perturbations.

5.3 Quantitative Results↩︎

State of the Art Comparison. As shown in Table ¿tbl:tab:main95results?, 3D FaceShellachieves competitive performance in terms of both Injection Rate (IR) and Mismatch Rate (MR) across all evaluated vision-language models. While aggressive 2D baselines such as M-Attack and V-Attack obtain slightly higher mismatch rates overall and higher injection rates on certain models (e.g., BLIP-2 and LLaVA-NeXT), the semantic performance gap remains small. For instance, 3D FaceShelltrails the strongest baseline (M-Attack) by only a 3.9% relative difference in LLaVA-OV MR and by 10.3% relative difference compared to V-Attack on VideoLLaMA3 MR. Moreover, 3D FaceShellsurpasses both M-Attack and V-Attack in absolute Injection Rate on VideoLLaMA3 and LLaVA-OV. Importantly, the marginal semantic gains of these aggressive 2D approaches come at the cost of significant degradation in visual quality and identity preservation. In contrast, 3D FaceShellmaintains substantially higher perceptual and structural fidelity, achieving a +71.5% improvement in LPIPS compared to V-Attack and a +60.7% improvement over M-Attack. Similarly, 3D FaceShellimproves Identity Similarity by +88.9% and +56.2% relative to V-Attack and M-Attack, respectively. This distinction is critical, as preserving facial identity and geometric structure is essential for realistic and minimally intrusive adversarial manipulation.

Multi-Attribute vs Single-Attribute Transfer. Table 1 presents a quantitative comparison of Transfer Rate and Mismatch Rate on VideoLLaMA3 when modifying all targeted semantic attributes simultaneously versus isolating a single attribute and keeping the rest constant. Our results indicate that shifting all attributes at once yields superior performance for most individual attributes. For instance, transferring all attributes simultaneously results in a +50.7% relative increase in the transfer rate for hair color and a +96.7% relative increase in the transfer rate for expression compared to isolated single-attribute attacks. Additionally, Gender and Ethnicity transfer rates see relative increases of +3.2% and +9.7%, respectively. This result confirms that inducing a compounded semantic shift via multi-attribute target image injection provides a stronger gradient signal, thereby improving the attack success rate across individual attributes.

Commercial Model Evaluation. Table ¿tbl:tab:gemini95results? extends our primary evaluation to the commercial Gemini-2.5-Flash[5] VLM, comparing our method against our 2D adversarial attack baselines. Our 3D FaceShell successfully generalizes to this closed-source model, achieving a competitive injection rate and mismatch rates. Crucially, while our method observes a relatively low difference in semantic manipulation metrics, falling short of V-Attack’s injection rate by a relative 13.4% and M-Attack’s by a relative 28.3%, it yields a disproportionately massive improvement in visual fidelity. Specifically, 3D FaceShell achieves an LPIPS score of 0.1499, representing an impressive 62.9% relative improvement over V-Attack and a 60.7% improvement over M-Attack. Similarly, 3D FaceShell achieves an identity similarity score of 0.7629, structurally outperforming V-Attack[33] by 45.2% and M-Attack[37] by 56.2%. This demonstrates that 3D FaceShell maintains semantic competitiveness without relying on the severe, unrealistic face degradation characteristic of 2D baselines. It should be noted that we additionally tried to evaluate each of the methods on GPT-4o[4] but this model largely refused to answer questions about face attributes despite prompt engineering efforts.

r0.48

Cross-View Generalization. We assess the necessity of our multi-view adversarial noise training in Table [tbl:tab:view95results]. When comparing our multi-view training strategy against a model trained exclusively on a single frontal view, it is evident that multi-view training contributes heavily to successful transfer onto novel views. The model trained only on the front view experiences a severe drop in performance on the validation view, whereas our multi-view approach maintains robust generalization. Specifically, multi-view adversarial noise training achieves a +55.6% higher relative Injection Rate on the unseen validation view for VideoLLaMA3 compared to the single-view baseline.

5.4 Qualitative Results↩︎

Fig. 5 displays the large improvement in identity preservation and visual clarity of our method compared to state-of-the-art 2D methods of semantic shifting. While baseline 2D attacks introduce noticeable pixelated noise, severe color distortions, or structural warping that immediately compromise the photorealism of the face, 3D FaceShell introduces far less intrusive noise. By confining the optimization to our learnable Gaussian shell and heavily regularizing the perturbations, our approach ensures the rendered outputs look naturally persistent. The qualitative comparisons confirm that 3D FaceShell successfully balances the dual objectives of manipulating VLM semantic interpretations while leaving the underlying human-recognizable geometry and textures perceptually undisturbed.

5.5 Ablation Studies↩︎

Impact of Losses and Augmentation. In Table [tbl:tab:loss95results] we conduct an ablation study analyzing the impact of our loss functions and augmentations. Removing the data augmentations leads to a drastic drop in both Injection Rate and Mismatch Rate across all VLMs. For instance, causing a 60.9% relative drop in VideoLLaMA3 Injection Rate and a 46.2% relative drop in LLaVA-OV Injection Rate. This demonstrates that augmentations are strictly necessary to prevent the adversarial shell from overfitting to specific rendered views. Furthermore, removing the chroma loss (\(\mathcal{L}_{\mathrm{chroma}}\)) or the landmark loss (\(\mathcal{L}_{\mathrm{landmarks}}\)) significantly degrades the incredibly important perceptual quality and identity similarity of the rendered images compared to the source images. Quantitatively, disabling the chroma loss causes a 24.1% relative drop in identity similarity and a 26.3% decrease in perceptual similarity. Similarly, disabling the landmark loss drops identity similarity by 17.6%. This confirms that the face-aware regularization mechanisms are vital for achieving the optimal balance between a high rate of semantic transfer and identity preservation. Finally, omitting dynamic opacity clamping (“No Clamp") yields marginal semantic gains, such as a relative 7.9% increase in LLaVA-NeXT Injection Rate, but introduces unacceptable visual artifacts, causing a 25.2% relative reduction in perceptual similarity and a 5.4% drop in identity similarity.

Multi-Encoder Supervision. Additionally, Table [tbl:tab:encoder95results] highlights the benefit of utilizing multiple visual encoders during the feature alignment step. Utilizing multiple encoders in our training architecture produces more effective overall semantic attribute shifting across the black-box VLMs than using either SigLIP or CLIP in isolation. For example, ensembling the models yields a +108.3% relative increase in VideoLLaMA3 Injection Rate compared to using CLIP alone, and a +26.0% relative increase in BLIP-2 Injection Rate compared to using SigLIP alone. This demonstrates that ensembling diverse representation spaces during optimization prevents the Gaussian shell from overfitting to a single model’s feature space, ultimately yielding more robust and transferrable adversarial perturbations.

6 Conclusion↩︎

In this work, we introduced 3D FaceShell, a novel privacy defense framework designed to protect 3D face avatars from unauthorized semantic inference by VLMs. Our method remains robust through multiple rendered viewpoints by operating directly in the 3D Gaussian space and successfully transfers face attribute semantics without significantly degrading visual clarity or destroying the identity of the source face. By introducing a heavily regularized, learnable auxiliary Gaussian shell, we successfully inject perturbations that manipulate VLM understanding while leaving the underlying baseline geometry and identity-defining appearance completely untouched.

Our extensive evaluations across multiple state-of-the-art black-box VLMs demonstrate that 3D FaceShell achieves highly competitive targeted attribute transfer and mismatch rates. Crucially, it accomplishes this while maintaining significantly higher perceptual quality and identity similarity than current state-of-the-art 2D semantic manipulation baselines. Ablation studies further validate that our multi-attribute synthesis, multi-view adversarial training, and face-aware regularization mechanisms are strictly essential for achieving robust, view-consistent semantic steering.

While our current framework successfully defends static 3D face avatars, a promising direction for future work involves extending this mechanism into the 4D Gaussian Splatting space. Adapting our learnable auxiliary shell to maintain temporal consistency could safeguard fully animatable and dynamic portrait avatars [46] against continuous semantic inference in video formats.

Acknowledgments↩︎

This material is based upon work supported by the Center for Identification Technology Research and the National Science Foundation under Grant No. 2601332. This work is also supported in part by the National Science Foundation (IIS-2245652). We would also like to thank Charlotte Vision Lab members for our valuable discussions.

7 Appendix↩︎

In Section 7.1, we provide additional ablation studies, detailing the performance impact of calculating our landmark consistency loss directly in the 3D space and analyzing the effect of varying the total number of initialized auxiliary shell Gaussians. Furthermore, we showcase the robustness of 3D FaceShell to postprocessing, results for 3D FaceShell on an additional dataset with a new 3D reconstruction pipeline, failure case analysis, and a visualization of our average noise intensity. Finally, in Section 7.2 we present a user study which provides further insight into the quality of our rendered images compared to other selected methods.

7.1 Additional Ablation Studies↩︎

Table ¿tbl:tab:ablation953dlm? explores an alternative formulation of our face-aware regularization by shifting the landmark loss calculation from the 2D rendered space directly into the 3D Gaussian parameter space. To achieve this, we first create 2D facial landmark masks (covering the eyes, nose, and lips) for the source face renders. We then map these 2D masks to the 3D auxiliary shells via a perspective projection pipeline:

1. Perspective Projection: The 3D spatial means of the shell Gaussians are converted to homogeneous coordinates and multiplied by the camera’s full projection matrix. We apply perspective division to transform these points into Normalized Device Coordinates (NDC) bounded between -1.0 and 1.0.

2. Image Space Mapping: The NDC coordinates are scaled by the camera’s resolution to calculate exact 2D continuous pixel coordinates \((u, v)\) in the image space.

3. Bounds Checking: We strictly filter out any Gaussians that project behind the camera plane or fall outside the 2D image boundaries.

4. Mask Intersection: For the remaining valid projections, we sample the 2D MediaPipe mask at the discrete \((u, v)\) pixel indices. If a shell Gaussian projects onto an active landmark pixel in any of the training views, it is permanently flagged as a “landmark shell”.

Instead of penalizing 2D pixel distortions, this variant computes the loss directly on the flagged 3D shells by severely penalizing their opacity and explicit color deviations from the baseline. While this direct 3D-space landmark loss slightly improves the accuracy metrics across all evaluated models, for instance, increasing the LLaVA-NeXT[7] injection rate by 7.4%, and the BLIP-2 injection rate by 2.2%, it ultimately degrades perceptual fidelity as well. Identity similarity is degraded by 3.8% and LPIPS is degraded by 2.2%. Ultimately because of the importance of identity preservation and image fidelity in this task it remains best to perform the landmark loss in the 2D space.

Table ¿tbl:tab:ablation95scale? analyzes the structural and semantic impact of initializing different quantities of auxiliary shell Gaussians. We compare our standard configuration of 50,000 Gaussians against variants with 25,000 and 75,000 Gaussians. The results indicate that increasing the number of shell Gaussians yields a significant accuracy improvement that definitively warrants the small trade-off in visual fidelity. For example, scaling the auxiliary shells from 50,000 to 75,000 provides rather large improvements in semantic manipulation. We observe a 7.0% relative increase in the injection rate on VideoLLaMA3[6] and a 14.1% relative increase in the LLaVA-NeXT[7] mismatch rate among improvements in all other accuracy related metrics as well. These substantial semantic gains easily justify the very minor penalty incurred with LPIPS and identity similarity degrading by only 1.9% and 3.2% respectively.

Table 2 evaluates 3D FaceShell under 4 real-world distortions: (i) JPEG compression (15% quality); (ii) resizing to \(196\!\times\!196\); (iii) a screen-photography simulation pipeline comprising Gaussian blur, contrast/brightness scaling, a high-frequency grid overlay, and additive Gaussian noise; and (iv) rendering-level perturbations via exposure modulation, color temperature shifts, and Gaussian scale variation. Across all settings, 3D FaceShell shows consistent robustness, outperforming the 2D SoTA VAttack[33].

Table 3 reports results on 20 identities from APPA-REAL [47], a dataset of user-uploaded (non-celebrity) faces, reconstructed via LGM[48], a general-purpose image-to-3D model. 3D FaceShell maintains strong performance across this real-world dataset and alternative 3D reconstruction pipeline, demonstrating generalization beyond the evaluation setting of the main paper.

To further analyze failure cases Table 4 details the number samples achieving each possible injection and mismatch count (0-7 attributes). 3D FaceShell exhibits near-zero failure: mismatch occurs in all 100 samples, and target attribute injection fails completely in only 1 case demonstrating consistent adversarial effectiveness across the attribute space.

Figure 6 plots spatial LPIPS and identity similarity averaged across all samples at varying optimization iterations. Early iterations exhibit broad structural distortion; late iterations converge to globally minimized perturbation. At intermediate stages, 3D FaceShell concentrates perturbations in semantically non-critical facial regions, disrupting VLM interpretation while enforcing low LPIPS deviation over identity-critical areas, thus preserving subject identity throughout optimization.

7.2 User Study↩︎

Table 5 presents a user study which compares our method with three others. M-Attack[37] and V-Attack[33] were chosen for their strong injection rate and mismatch rate across a variety of models while AttackVLM[35] was chosen for its strong performance in terms of identity preservation and LPIPS. This study was conducted with 13 participants who were each presented with 10 sets of images. Each set contained a source face image alongside a noised version of that source face generated by each of the four methods. The names of each method were anonymized and the position of each method was randomized for each set. Participants were asked to rank each method relatively from 1st (best) to 4th (worst) in two metrics for each set of images. The first metric was the identity preservation of the noised image and the second was the perceptual quality of the noised image. Our results show that users strongly agree that the results generated by 3D FaceShell are consistently much better in terms of perceptual quality and identity preservation compared to the other methods provided. Our method achieved a dominant average rank of 1.33 for perceptual quality, outperforming the next best method, M-Attack[37], at 2.67, and an average rank of 1.61 for identity preservation, significantly leading the next best method, AttackVLM[35], at 2.42.

References↩︎