1 Introduction↩︎

Vision-Language Models (VLMs) such as CLIP [1], pre-trained on large-scale image-text pairs, enable strong cross-modal understanding and zero-shot generalization, and are now widely applied across vision and multimodal tasks [2]. Despite its success, CLIP is highly vulnerable to adversarial perturbations: even imperceptible input distortions [3] can cause severe prediction errors. Such fragility poses critical safety risks in security-sensitive applications, making adversarial robustness a key challenge for reliable deployment.

Adversarial training [4], [5] is a well-studied strategy for improving model robustness. When extended to VLMs like CLIP, methods such as Adversarial Fine-Tuning [6]–[8] and Adversarial Prompt Tuning [9], [10] have achieved notable progress in strengthening adversarial resistance. However, these approaches rely on task-specific annotated datasets, making training costly and less accessible. Optimization on limited data may also weaken generalization and zero-shot transferability. To address these limitations, recent studies have explored test-time defenses that require no additional training, broadly categorized as prompt-based and transformation-based approaches. Prompt-based defenses [11], [12] adapt textual prompts for each instance, effectively mitigating attacks but substantially increasing inference latency. Transformation-based methods, such as counterattack perturbation [13] and Gaussian noise injection [14], offer a simple and efficient way to enhance adversarial robustness by modifying inputs, yet they may degrade performance on clean images.

Given their effectiveness and computational efficiency, transformation-based methods have emerged as a promising approach for test-time defense. Prior studies [15]–[18] have shown that various image transformations can mitigate adversarial effects. However, the underlying mechanism of how and why the adversarial effects are alleviated remains unexplored, limiting further progress in defense design. To address this, we analyze the latent feature space to investigate how diverse transformations alter image features and thereby mitigate adversarial effects. As shown in Fig. 1 (a), when applying various transformations to an input image, the transformed features of a clean image tend to scatter around the original feature, whereas those of an adversarial image consistently shift toward a specific direction, presenting a skewed pattern. To quantify this phenomenon, we further introduce a Directional Bias (DB) score to measure the directional concentration of transformed features. As shown in Fig. 1 (b), the DB-score exhibits a clear bimodal distribution, effectively distinguishing adversarial from clean ones, as adversarial images consistently exhibit high and concentrated scores.

The above observation prompts a key question: what does the direction of transformed features represent? Recall that adversarial attacks work by shifting features away from original class centers, thereby inducing misclassification. We therefore hypothesize that this dominant direction could be anti-parallel to the adversarial shift, pointing features back toward their correct class centers. Building on this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework for VLMs that leverages this specific direction, referred to as Defense Direction, to uncover discriminative features. To capture robust Defense Direction, DBD applies a wide range of transformations across spatial, pixel, and frequency domains to obtain diverse augmented features, and then uses entropy-based filtering to retain high-quality ones. Leveraging the DB-score to distinguish between adversarial and clean inputs, we propose a two-stream feature reconstruction strategy to enhance test-time defense: for high DB-score examples, adversarial features are linearly shifted along the Defense Direction to restore correct representations; while for low DB-score examples, the average transformed features are used as test-time augmentation for stabilizing representations.

We conduct extensive experiments across ten fine-grained classification datasets and five ImageNet-OOD datasets. The results demonstrate that our method not only preserves performance on clean images but also achieves substantial improvements over previous state-of-the-art defenses on adversarial examples across all datasets. Remarkably, as shown in Fig. 1 (c), the classification accuracy on adversarial images even surpasses that on clean images. This counterintuitive result justifies that the generation of adversarial examples guided by ground-truth labels implicitly encodes directional priors about the true decision boundary, which we exploit to achieve effective defense.

Our main contributions are as follows: (1) To the best of our knowledge, we are the first to show that adversarial perturbations implicitly encode directional priors of the true decision boundary, which can be reliably estimated using multiple transformations. (2) We propose Directional Bias-guided Defense (DBD), a test-time framework that leverages these directional priors through Defense Direction estimation and a two-stream reconstruction strategy based on the proposed DB-score, enabling effective and efficient defense. (3) We validate DBD on 15 datasets, demonstrating superior adversarial robustness while preserving zero-shot performance on clean images. In some cases, our method even surpasses the performance on clean images when evaluated on adversarial images.

2 Related Works↩︎

Vision-Language Models (VLMs).  CLIP [1], trained on large-scale image-text pairs, has become a cornerstone vision-language model (VLM) with strong zero-shot generalization and cross-modal reasoning [2]. Building on this paradigm, ALIGN [19] and BLIP-2 [20] further scale or refine the alignment of image-text pairs, while LLaVA [21] extends VLMs toward instruction-following and conversational tasks. By aligning modalities in a shared embedding space, these models provide powerful task-agnostic representations. However, prior studies [22], [23] have shown that VLMs are highly vulnerable to adversarial attacks, posing a critical barrier to their deployment in safety-sensitive applications.

Adversarial Attacks and Defenses.  Adversarial perturbations are small but carefully crafted input distortions that can drastically mislead deep neural networks [3], [24]–[27]. Early works proposed gradient-based attacks such as FGSM [28], iterative methods like PGD [4], and optimization-based approaches such as CW [29]. More recent efforts have introduced adaptive attacks such as AutoAttack (AA) [30], a robust benchmark combining four attacks: the score-based black-box Square [31], the minimal-\(\ell_p\)-perturbation FAB [32], APGD-CE (using cross-entropy loss), and APGD-DLR (using difference-of-logits-ratio loss). To counter adversarial threats, defenses have been extensively explored. Adversarial training [4], [5], [33], [34] optimizes models on perturbed examples to enhance robustness. Input purification [15], [18] transforms inputs toward the clean distribution. Recent diffusion-based purification methods [35], [36] show promise in removing perturbations but often incur high computational cost.

Adversarial Robustness of VLMs.  For VLMs such as CLIP, several extensions of adversarial training [6]–[10] have been proposed to enhance robustness. TeCoA [6] examines the effect of fine-tuning and visual prompt tuning on the zero-shot adversarial robustness of VLMs. Adversarial Prompt Tuning methods, including APT [9] and AdvPT [37], focus on optimizing textual prompts without modifying model parameters. However, these methods rely on annotated data and may weaken generalization, motivating test-time defenses that require no additional training. Prompt-based test-time methods such as R-TPT [11] and TAPT [12] adapt prompts on a per-instance basis, achieving reasonable robustness at the cost of substantial inference overhead. Transformation-based methods mitigate adversarial attacks by modifying input images. For example, TTC [13] generates counterattack perturbations for adversarial images, and AOM [14]injects Gaussian noise into inputs. These approaches can improve robustness in practice but often degrade performance on clean images. Our method exploits directional bias in latent feature space to reconstruct features under diverse transformations, enhancing adversarial robustness while preserving clean performance, and achieving an efficient balance between robustness and computational cost.

3 Method↩︎

3.1 Preliminaries↩︎

Zero-shot classification of CLIP.  CLIP [1] is a VLM that projects images and texts into a shared embedding space and measures their relationships using cosine similarity. For zero-shot classification, CLIP consists of two pre-trained encoders: a visual encoder \(\mathcal{E}_v\) and a text encoder \(\mathcal{E}_t\). For an \(C\)-class classification task, given an image \(x_{test}\) and a set of class names with prompts \(T_c, c \in [1, C]\), CLIP computes text features: \({\boldsymbol{f}}_{t_c}=\mathcal{E}_t(T_c)\), for each class \(c\), and image feature \({\boldsymbol{f}}_v=\mathcal{E}_v(x_{test})\). The prediction probability for class \(c\) is calculated as:

\[P_{\text{CLIP}}(y=c \mid x_{\text{test}}) = \frac{\exp(\text{cos}({\boldsymbol{f}}_{t_c}, {\boldsymbol{f}}_v)/t)}{\sum_{c'=1}^C \exp(\text{cos}({\boldsymbol{f}}_{t_{c'}}, {\boldsymbol{f}}_v)/t)}, \label{eq1}\tag{1}\]

where \(\text{cos}(\cdot, \cdot)\) is the cosine similarity between the features, and \(t\) is a temperature parameter that controls the sharpness of the distribution. The final classification decision is determined by selecting the class with the highest probability:

\[\hat{y} = \mathop{\mathrm{arg\,max}}_{c \in [1,C]} P_{\text{CLIP}}(y=c \mid x_{\text{test}}),\]

where \(\hat{y}\) represents the predicted class label.

Adversarial attacks for CLIP.  Despite its strong zero-shot performance, CLIP is particularly sensitive to small adversarial perturbations [3]. Following recent SOTA test-time method R-TPT [11], we consider a threat model where the attacker has full access to the vanilla CLIP model, but no knowledge of the defense mechanism. This reflects real-world deployment: foundation models like CLIP have publicly available weights, while test-time defenses are typically deployed privately. In this setting, adversarial examples are crafted against CLIP as:

\[\delta = \mathop{\mathrm{arg\,max}}_{\delta^\prime} \mathcal{L}(\text{CLIP}(x+\delta^\prime, T), y), \quad \text{s.t. } \|\delta^\prime\|_p \leq \epsilon , \label{eq:adv-attack}\tag{2}\]

where \(y\) is the ground-truth label of input image \(x\), \(T\) is a set of class names with prompts, \(\mathcal{L}\) is a loss function (typically cross-entropy loss), and \(\epsilon\) is the attack budget controlling the magnitude of perturbations to remain imperceptible.

3.2 DBD for VLMs↩︎

We propose Directional Bias-guided Defense (DBD), a test-time framework for defending VLMs against adversarial attacks. The core idea of DBD is to leverage multiple input transformations to construct diverse reference features, analyze their directional bias relative to the original features, and use this property to guide feature reconstruction. The overall framework consists of three main components: input transformations & feature filtering, Directional Bias (DB) computation, and two-stream feature reconstruction, as illustrated in Fig. 2.

Image Transformations & Feature Filtering.  Since individual transformations have inherent drawbacks, relying on a single transformation may produce unreliable features. For example, random cropping is stochastic and may capture mostly background, while filtering can excessively blur important details. To improve robustness, we apply a diverse set of transformations to generate multiple feature candidates, which leverages complementary strengths across transformations to both preserve task-relevant information and disrupt adversarial noise.

We construct an image transformation library covering diverse transformations across three domains: (1) Spatial domain: including random cropping, scaling, and flipping. These geometric operations alter object position, size, and orientation, thereby disrupting the structured alignment of adversarial perturbations and weakening their effect. (2) Pixel domain: including bit-depth compression (quantization), JPEG compression-decompression, and additive Gaussian noise. These pixel-level modifications distort or overwrite fine-grained perturbations, making them less effective in misleading the model. (3) Frequency domain: including Gaussian, mean, and median filtering. By smoothing or suppressing high-frequency components, these filters reduce adversarial noise while largely preserving the semantic content of the image.

Transformed features exhibit varying quality across different transformations, so we apply a feature filtering step to select the most informative and reliable representations. Following common practice in test-time adaptation [38], [39], we use the entropy of the model’s prediction as a quality metric. Specifically, given \(n\) transformed images, we pre-compute their features \({\boldsymbol{f}}_i(i=1,2,...,n)\) and classification probabilities with CLIP, then calculate the entropy for each: \[E_i = -\sum_c p_{i,c} \log p_{i,c},\] where \(p_{i,c}\) denotes the predicted probability of the \(i\)-th transformed image feature for class \(c\). We then select the \(k\) transformed image features with the lowest entropy as high-quality features for subsequent processing: \[\mathcal{F}_{\mathrm{ref}} = \{ {\boldsymbol{f}}_i \mid i \in \mathcal{I}_k \}, \quad \mathcal{I}_k = \mathop{\mathrm{arg\,min}}_{i \subset \{1, \dots, n\}, |i|=k} \sum_{j \in i} E_j.\]

Directional Bias (DB) Computation.  After applying multiple transformations and feature filtering, we obtain a set of high-quality transformed image features. Visualization (Fig. 1 (a)) shows that for clean images, the transformed features exhibit a dispersed pattern around the original feature. In contrast, for adversarial inputs, they consistently shift toward a specific direction, presenting a directional bias pattern. This occurs because transformations partially mitigate adversarial perturbations, aligning the features closer to the clean feature distribution.

Given a set of transformed features \(\mathcal{F}_{\mathrm{ref}} = \{{\boldsymbol{f}}_i \mid i=1,\dots,k\}\) and the original feature \({\boldsymbol{f}}_0\), we define the direction vectors for each transformed feature \({\boldsymbol{f}}_i\) as unit vectors \({\boldsymbol{d}}_i = ({\boldsymbol{f}}_i - {\boldsymbol{f}}_0)/\|{\boldsymbol{f}}_i - {\boldsymbol{f}}_0\|_2\), and compute the Defense Direction \({\boldsymbol{d}}_{\mathrm{def}}\) as \[\bar {\boldsymbol{d}}= \frac{1}{k} \sum_{i=1}^k ({\boldsymbol{f}}_i - {\boldsymbol{f}}_0), \quad {\boldsymbol{d}}_{\mathrm{def}} = \frac{\bar {\boldsymbol{d}}}{\|\bar {\boldsymbol{d}}\|_2},\] where \(\|\cdot\|_2\) denotes the \(\ell_2\) norm (Euclidean distance). The DB-score is computed as the average cosine similarity between each direction and the Defense Direction: \[S_{\mathrm{db}} = \frac{1}{k} \sum_{i=1}^k \langle {\boldsymbol{d}}_i, {\boldsymbol{d}}_{\mathrm{def}} \rangle,\] where \(\langle \cdot, \cdot \rangle\) denotes the inner product. As shown in Fig. 1 (b), the DB-score exhibits a clear bimodal distribution for clean and adversarial images, allowing simple thresholding to separate them.

Two-stream Feature Reconstruction.  Adversarial perturbations shift image features away from the true classification region, thereby inducing misclassification. Intuitively, the Defense Direction \({\boldsymbol{d}}_{\mathrm{def}}\) could be anti-parallel to the adversarial shift, pointing features back toward their correct classification region. Guided by this intuition, we reconstruct more robust features by linearly shifting the input feature along the Defense Direction. However, due to stochasticity or imperfections in the transformations, the computed Defense Direction may be inaccurate. To reduce the negative impact of inaccurate Defense Direction, we propose a two-stream reconstruction strategy based on the DB-score: (1) High DB-score stream: a high \(S_{db}\) indicates a likely adversarial image and a more reliable Defense Direction. In this case, we shift the feature along the Defense Direction to enhance its distinction from the original. (2) Low DB-score stream: a low \(S_{db}\) suggests a likely clean image with less reliable direction. For these examples, we use the average of transformed features as test-time augmentation for stabilizing representations. Formally, we introduce a threshold \(\tau\) on DB-score \(S_{db}\): \[\hat{\boldsymbol{f}}= {\boldsymbol{f}}_0 + l \cdot {\boldsymbol{d}}_{\mathrm{def}}, \quad l = \begin{cases} \|\bar {\boldsymbol{d}}\|_2, & S_{db} \le \tau \\ \lambda \cdot\|\bar {\boldsymbol{d}}\|_2, & S_{db} > \tau \end{cases},\] where \(\hat{\boldsymbol{f}}\) is the reconstructed feature and \(\lambda\) is a hyperparameter controlling the magnitude of the feature shift. In practice, we use the distance from the average transformed features to the original feature \(\|\bar {\boldsymbol{d}}\|_2\) as a reference for the shift magnitude, which is then scaled by \(\lambda\). Finally, we use the reconstructed image feature \(\hat{\boldsymbol{f}}\) to compute the predicted classification result, as given in Eq.@eq:eq1 . Our DBD is summarized in Algorithm 4 in the appendix.

4 Experiments↩︎

4.1 Experiment setup↩︎

Datasets.  Following prior works [9], [11] on the adversarial robustness of CLIP, we evaluate our proposed test-time DBD on ten fine-grained classification datasets and five ImageNet-based out-of-distribution(OOD) benchmarks. The fine-grained datasets span diverse domains: general objects (Caltech101 [40]), animals (Pets [41]), plants (Flower102 [42]), vehicles (Cars [43], Aircraft) [44], textures (DTD [45]), satellite imagery (EuroSAT [46]), human actions (UCF101 [47]), scenes (SUN397 [48]), and food (Food101 [49]). For ImageNet-OOD evaluation, we use ImageNet [50] and four established variants: ImageNet-A [51], ImageNet-V2 [52] , ImageNet-R [53], and ImageNet-S [54]. Since our method targets test-time adversarial robustness, we do not require access to any training sets.

Implementation details.  We use official pre-trained CLIP backbones (ResNet-50 [55], ViT-B/32, and ViT-B/16 [56]) as the base models. Adversarial images are generated with PGD [4] under the \(L_{\infty}\) norm constraint. Following prior works [9], [11], we evaluate two threat levels. For low-strength attack, we use PGD-10 with \(\epsilon = 1/255\) on CLIP-ResNet50; for high-strength attack, we use PGD-100 with \(\epsilon = 4/255\) on CLIP-ViT-B/32 and CLIP-ViT-B/16. The step size for all attacks is \(\alpha = \epsilon / 4\). For our DBD, we apply \(n = 31\) transformations per input, yielding 32 images including the original, and then select \(k = 16\) transformed image features via entropy-based filtering. The DB-score threshold is \(\tau = 0.8\), and the feature shift magnitude is set to \(\lambda = 2.5\). Both are estimated from ImageNet validation set (50k images). Experiments are conducted in PyTorch on RTX 3090 GPUs.

Baselines.  We compare DBD with several existing methods, including adversarial fine-tuning on ImageNet (TeCoA [6]), adversarial prompt tuning on downstream datasets with 16 shots (APT [9]), test-time prompt tuning (R-TPT [11]), test-time input transformation method (TTC [13]), and the original CLIP [1] models. Except for APT, which uses few-shot tuning , all other methods operate in a zero-shot setting. Baseline results are obtained from official reports or reproduced using official code.

4.2 Main Results↩︎

Results on fine-grained datasets.  We evaluate the adversarial robustness of DBD on ten fine-grained classification datasets, with results summarized in Table 1. Under the CLIP-ViT-B/32 and PGD-100 (\(\epsilon=4/255\)) setting, the original CLIP model demonstrates strong zero-shot classification performance (63.7%) but is almost entirely vulnerable to adversarial attacks (0.0%). Adversarial fine-tuning (TeCoA) improves defense to 11.4%, but at the cost of reduced zero-shot performance on clean images (63.7% \(\to\) 32.8%). Adversarial prompt tuning (FAP) further enhances robustness (0.0% \(\to\) 22.0%) while mitigating the drop in clean accuracy (63.7% \(\to\) 56.1%), though its few-shot setting introduces data dependency. Test-time input transformation method (TTC) achieves moderate robustness on CLIP-ViT-B/32 (0.0% \(\to\) 27.2%) but are sensitive to model architecture, with only 12.5% on CLIP-ViT-B/16. Previous state-of-the-art test-time prompt tuning method (R-TPT) attains better robustness (0.0% \(\to\) 38.1%) and largely preserves zero-shot performance on clean images (63.7% \(\to\) 60.2%). In contrast, ours DBD maintains zero-shot performance close to the original CLIP across all three backbones (even exceeding it on CLIP-ViT-B/32) and achieves substantially higher robustness on adversarial examples, reaching 93.8% on CLIP-ViT-B/16, significantly surpassing both R-TPT and the performance on clean examples.

Results on ImageNet-OOD datasets.  Results on ImageNet-OOD benchmarks are summarized in Table [tab:imagenet-ood]. The original CLIP model demonstrates strong robustness to distribution shifts but remains highly vulnerable to adversarial attacks. Notably, our DBD method not only preserves but slightly improves zero-shot classification on clean images (e.g., 52.7% \(\to\) 55.0% on CLIP-ViT-B/32), which we attribute to the combination of diverse image transformations and entropy-based feature filtering that produces more robust and accurate features. Across all three backbones, DBD substantially outperforms the previous state-of-the-art R-TPT on adversarial examples, achieving up to 97.7% on CLIP-ViT-B/16, a performance that even surpasses the classification accuracy on clean examples.

Discussion.  Remarkably, the classification accuracy on adversarial images significantly exceeds that on clean images. This counterintuitive result suggests that the generation of adversarial examples guided by ground-truth labels implicitly encodes directional priors about the true decision boundary. Our method leverages this by applying multiple image transformations to estimate the Defense Direction , then linearly shifting features along it to reconstruct robust representations.

4.3 More Analysis↩︎

Analysis under PGD attack with pseudo-label.  The previous experiments demonstrate that when PGD generates adversarial examples using ground-truth labels, our method achieves robust accuracy which significantly exceeds clean accuracy. To further validate its effectiveness without relying on ground-truth labels, we consider a pseudo-label setting. Specifically, we use CLIP’s own predictions on clean images as pseudo-labels to guide PGD in generating adversarial examples. As shown in Table 2, under this setting the vanilla CLIP model remains highly vulnerable. In contrast, our method consistently outperforms R-TPT and achieves robust accuracy comparable to clean accuracy (e.g., 67.5% \(\to\) 66.8% on CLIP-ViT-B/16). This indicates that our method is able to fully leverage the directional prior information carried by adversarial examples.

Analysis under various attacks.  To demonstrate the generality of our method, we evaluate DBD and baseline methods under additional adversarial attacks, including FGSM [28], CW [29], AutoAttack (AA) [30], and four component attacks of AA: Square Attack [31], targeted FAB [32], untargeted APGD-CE, and targeted APGD-DLR. We evaluate various attacks on the same six fine-grained datasets in Table 2, with the results summarized in Table 3. DBD consistently demonstrates robust defense performance across all attack types, significantly outperforming R-TPT. Notably, under AA on CLIP-ViT-B/16, DBD achieves an average robust accuracy of 69.8%, surpassing the average clean accuracy of 67.5%.

Geometric Verification of the Proposed Defense Direction.  To validate our hypothesis that the Defense Direction genuinely points toward the correct decision region, we conduct a geometric analysis by measuring the cosine similarity between the estimated Defense Direction and two critical reference directions. The first is the Clean Direction, defined as the vector from the adversarial image’s feature to its corresponding clean (unperturbed) counterpart’s feature. The second is the Class Centroid Direction, defined as the vector from the adversarial image’s feature to the centroid of features belonging to the true class (computed using correctly classified clean images).

Table 4 presents the average cosine similarities across multiple fine-grained datasets under the PGD-100 (\(\epsilon\)=4/255) attack on CLIP-ViT-B/16. The results demonstrate that our proposed Defense Direction exhibits extremely high similarity (\(\approx0.95\)) with the Clean Direction and substantial similarity (\(\approx0.90\)) with the Class Centroid Direction. These high cosine similarities provide strong geometric evidence that the Defense Direction indeed aligns closely with both the clean feature direction and the correct class centroid, supporting our hypothesis that the Defense Direction points back toward the correct decision region.

4.4 Ablation Study↩︎

Ablation of the feature shift magnitude factor \(\lambda\).  To evaluate the impact of the shift magnitude factor \(\lambda\) on the performance of DBD, we test three attack settings across 15 datasets. The average robust accuracy of all is shown in Fig.3 (a). When \(\lambda=0\), the reconstructed feature reduces to the original feature, yielding nearly no defense. With \(\lambda=1.0\), the reconstructed feature becomes the average of transformed features, achieving an average robust accuracy of 34.8%. At \(\lambda=2.5\), DBD reaches 92.1% accuracy, demonstrating that linearly shifting the original feature along the Defense Direction reconstructs robust features that align with the correct class. This highlights the reliability of the Defense Direction identified by DBD.

Ablation of the DB-score threshold \(\tau\).  DBD employs a DB-score threshold \(\tau\) to distinguish high-score examples (likely adversarial) from low-score examples (likely clean). We evaluate the effect of \(\tau\) under three attack settings across 15 datasets, reporting the overall mean accuracy obtained by averaging both clean and adversarial performance across all datasets and attack settings (Fig.3 (b)). The results show that setting \(\tau=0.8\) yields the best overall performance. At this threshold, DBD successfully reconstructs robust features for the majority of adversarial images, while simultaneously avoiding inaccurate shifts in most clean examples, thereby achieving a well-balanced trade-off between robust and clean accuracy. Additional results of the detection performance based on the DB-score threshold \(\tau\) are shown in Fig.6 of the Appendix.

Ablation of DBD mechanisms.  We conduct ablation experiments on all DBD components, with results average over 15 datasets and three attack settings (Table 5). Our analysis reveals four key findings: (1) Using a single type of image transformation with feature shifting provides strong adversarial defense but reduces clean image performance. (2) Aggregating multiple transformations without linear shifting gives the best clean accuracy but weak defense. (3) Adding linear shifting on top of multiple transformations substantially improves adversarial robustness, at the cost of some clean performance. (4) Applying the DB-score-based threshold to handle high-score and low-score examples separately achieves the best trade-off between clean accuracy and adversarial robustness.

5 Conclusion↩︎

In this work, we found that adversarial examples exhibit a strong directional bias under multiple input transformations, in contrast to the dispersed behavior of clean examples. Building on this observation, we proposed Directional Bias-guided Defense (DBD), a test-time framework that leverages the directional bias to reconstruct robust features through Defense Direction estimation and a two-stream reconstruction strategy based on the proposed DB-score. Experiments on 15 datasets under three attack settings demonstrate that DBD achieves state-of-the-art adversarial robustness while preserving zero-shot performance on clean images. Remarkably, robust accuracy even surpasses clean accuracy, highlighting that adversarial perturbations implicitly encode directional priors about the true decision boundary. We believe that our work sheds light on new perspectives for training-free defenses in VLMs.

Acknowledgments↩︎

This work was supported by the National Defense Technology Basic Scientific Research Project (Grant No. JSZL2023416A001).

6 Appendix↩︎

6.1 Algorithm↩︎

We provide a detailed pseudocode of our proposed DBD in Algorithm 4.

6.2 Implementation details↩︎

Transformations.  In our experiments, DBD applies \(n = 31\) transformations per input, yielding 32 images including the original. We provide more details on various image transformations used in DBD in Table 6. (1) For spatial domain transformations, each transformation includes random cropping, scaling, and flipping. Due to the stochastic nature of these operations, we perform 16 transformations per input. (2) For pixel domain transformations, we implement bit-depth reduction, compressing the original 8-bit image to 3 bits using three quantization methods: floor, round, and ceil. In addition, we apply JPEG compression-decompression with three quality levels (50, 60, 75) to increase diversity. Random Gaussian noise with amplitude \(\gamma = 0.1\) is also added, with six stochastic realizations per input. (3) For frequency domain transformations, we apply Gaussian, mean, and median filtering with kernel size 5.

In the ablation study (Table 5 and Fig. 5), the resulting average features from Random Crop-Resize-Flip (16 transformations), Bit-depth Reduction (3 operations), JPEG Compression (3 operations), Gaussian Noise (6 operations), and Image Filtering (3 filters) are used for classification.

Text Prompts.  For text prompts, we use a mix of hand-crafted prompts and GPT-3-generated prompts provided by CuPL [57], and average text features over multiple prompts per class.

Attacks.  All attacks are implemented using the torchattack [58] library. For PGD and FGSM attacks, we follow the baseline R-TPT [11] configuration by using cross-entropy loss in untargeted mode. For the CW attack [29], we use a learning rate of 0.01 with the Adam optimizer. For AutoAttack (AA, [30]), we use the standard mode, which includes four components: untargeted APGD-CE (1 restart), targeted APGD-DLR (10 target classes), targeted FAB (10 target classes, [32]), and Square Attack (5000 queries, [31]). To remain consistent with the baseline setup, EOT was not applied. When verifying these 4 components, we keep the parameters unchanged.

6.3 Datasets↩︎

We evaluate our method on 10 fine-grained classification datasets and 5 ImageNet-OOD datasets. Table 7 summarizes their detailed information, including their content, number of categories, number of images and corresponding hand-crafted prompt.

6.4 Experiments↩︎

6.4.1 Detailed results↩︎

Detailed results under PGD attack with pseudo-label.  Due to space constraints, Table 2 in the main text reports only a subset of the results. Here we provide the complete version in Table [tab:pgd-p2], which includes results on six fine-grained datasets and three attack settings.

Detailed results under various attacks.  We evaluate DBD and baseline methods under additional adversarial attacks, including FGSM [28], CW [29], and AutoAttack (AA) [30]. AA is a stronger, ensemble-based attack that combines targeted FAB [32], Square Attack [31], untargeted APGD-CE (Auto-PGD with Cross-Entropy loss), and targeted APGD-DLR (Auto-PGD with Difference of Logits Ratio loss), incorporating both gradient-based and gradient-free strategies to comprehensively challenge model robustness. We also separately evaluate defense performance against each of these four constituent attacks. For our DBD, we set \(\tau=0.6\) and \(\lambda=2.5\) (with \(\lambda=5\) for FGSM). We assess the effectiveness of various attacks on six fine-grained datasets across three attack settings, and detailed results are shown in Table 8.

6.4.2 More ablation studies↩︎

Detailed analysis of various image transformations.  We ablate the individual image transformations used in DBD to assess their standalone effects (as Fig.5). JPEG compression-decompression preserves clean accuracy while providing moderate defense against low-strength attacks, though its effectiveness diminishes as attack strength increases. Random crop-resize-flip slightly improves clean accuracy and shows robustness across different attack strengths and model backbones. Adding Gaussian noise can yield strong defense in certain cases but substantially degrades clean accuracy. By combining multiple transformations, DBD leverages their complementary strengths to generate more reliable features across diverse scenarios and model variants; moreover, integration of transformations mitigates the risk of defenses being circumvented by attacks tailored to a single transformation.

More ablation study on the DB-score threshold \(\tau\).  We evaluate the detection performance of adversarial examples using the DB-score. Specifically, we assess the effect of threshold \(\tau\) under three PGD attack settings across 15 datasets, reporting both the mean detection accuracy and mean F1-score averaged over all datasets and attack settings. Results are presented in Fig.6. The results show that \(\tau\) = 0.8 achieves near-optimal detection performance, and the metric remains stable in its neighborhood. This supports our choice and confirms that the DB-score provides a reliable signal for distinguishing adversarial from clean inputs.

Adaptive Attack with BPDA and EOT.  While our primary evaluation based on the threat model described in Section 3.1, we additionally evaluate against adaptive, defense-aware attacks to assess potential vulnerabilities. Based on PGD attack, we implement BPDA (Backward Pass Differentiable Approximation, [59]) combined with EOT (Expectation Over Transformation,  [18]) to approximate gradients through our differentiable DBD pipeline. During attack optimization, non-differentiable components are replaced with identity functions in the backward pass, while gradients are averaged over multiple stochastic forward passes.

We evaluate on Caltech101 dataset using strong adaptive attacks: PGD-10 (\(\epsilon\)=1/255) against CLIP-ResNet50 and PGD-100 (\(\epsilon\)=4/255) against CLIP-ViT-B/16. Results in Table 9 show that when attackers explicitly optimize through the full DBD pipeline, robust accuracy degrades significantly (to 50.79% and 1.29%, respectively). Notably, these adversarial images generated buy adaptive attacks also perform substantially worse against the original CLIP model compared to standard attacks, suggesting overfitting to the defense mechanism.

Critically, under the adaptive attack (PGD-100), the average DB-score remains high (0.89), with approximately 81% of adversarial samples exceeding our detection threshold (\(\tau\)=0.80). This reveals an important insight: the directional bias pattern persists under adaptive attacks, but the estimated direction is manipulated to point away from the true class. In other words, while adaptive attacks can subvert the defense functionality by distorting the Defense Direction, they cannot eliminate the underlying directional bias signal, making such attacks still detectable through our DB-score metric.

Analysis of attacks using different loss functions.   To evaluate the robustness of our method in more practical and challenging settings where ground-truth labels are unavailable during inference, we consider margin-based adversarial attacks, which does not rely on true class labels but instead maximizes the difference between the highest non-target logit and the target logit. We conduct PGD-100 attacks (\(\epsilon = 4/255\)) using both cross-entropy (CE) loss and margin loss, with either ground-truth labels or model-generated pseudo-labels, on the Caltech101 dataset. Results are shown in Table 10. Notably, DBD achieves 99.76% robust accuracy under margin-loss attacks with ground-truth guidance, on par with its performance under CE-loss attacks (99.39%). More importantly, when using pseudo-labels (i.e., no access to ground truth at test time), DBD maintains strong robustness (94.24% vs.% under CE-loss), demonstrating that the directional bias exploited by our method is not specific to a particular attack loss formulation. In contrast, the vanilla CLIP model collapses under all attack variants (\(<\)​3% robust accuracy). These results confirm that DBD’s defense mechanism generalizes across different adversarial objectives and remains effective in realistic label-free scenarios.

Analysis of Correct vs. Incorrect Clean Predictions.   We separate samples into those correctly classified by CLIP on clean images versus those misclassified, and evaluate robust accuracy under PGD-100 attacks (\(\epsilon = 4/255\)) on CLIP-ViT-B/16. Results are shown in Table 11. The near-perfect robust accuracy on initially correct samples demonstrates that DBD effectively preserves accurate predictions by leveraging directional information from adversarial perturbations. More importantly, the high robust accuracy on initially misclassified samples (up to 95.83%) shows that DBD genuinely corrects errors by exploiting the directional prior embedded in the attack. This dual capability explains why our overall robust accuracy exceeds clean accuracy, revealing an important insight: ground-truth guided attacks inadvertently leak information about the correct class direction.

Analysis of inference efficiency.   We compare DBD’s inference efficiency with baseline methods. Table 12 reports the training time of APT [9] and the inference time of test-time defenses R-TPT [11], TTC [13], and DBD on UCF101 using CLIP-ViT-B/32. Training-time defenses incur substantial training costs, while test-time defenses avoid this but increase inference overhead. DBD achieves a favorable balance between robustness and efficiency, outperforming R-TPT in both accuracy and inference speed.

Use of LLMs↩︎

In this work, we used ChatGPT to assist in polishing the writing of this paper, focusing primarily on improving clarity, grammar, and style. The model was not involved in the generation of ideas or experimental designs. All the concepts, analyses, and conclusions presented are entirely our own.

References↩︎

---

1. Corresponding author↩︎