1 Introduction↩︎

Vision-Language Models (VLMs) [1]–[5] like CLIP [6], pre-trained on massive image-text pairs, have achieved remarkable zero-shot generalization. Despite this success, VLMs remain highly vulnerable to adversarial perturbations: imperceptible noise can cause severe performance degradation [7]–[10], posing security risks in real word applications.

Existing efforts to enhance the robustness of VLMs generally fall into two categories. The first line of work utilizes adversarial training [11]–[14], which aims to immunize models by explicitly integrating adversarial examples into the optimization loop. While effective, these approaches typically incur prohibitive computational costs due to on-the-fly attack generation and require access to task-specific labeled data, thereby undermining the scalability and zero-shot flexibility inherent to foundation models. The second line of work explores test-time prompt tuning [15]–[17], an efficient paradigm that adapts learnable prompt contexts or predictions during inference without modifying model parameters. However, most existing test-time methods [18], [19] primarily rely on sample-level confidence (e.g., entropy) to filter augmented views. These methods treat augmentations as isolated data points, overlooking the intrinsic distributional structure and latent semantics. Consequently, they fail to distinguish between confident adversarial mispredictions and true semantic consistency, limiting their effectiveness under attack.

To address the above limitation, we revisit how adversarial perturbations interact with test-time augmentations in VLMs. A typical white-box attack is crafted on the original input image to maximally disrupt image-text matching based on its holistic representation. This can drastically shift the attacked image embedding in the feature space and make samples from different classes highly entangle as shown in Figure 1 (a). Importantly, this adversarial effect is not equally persistent across augmentations. Geometric transformations such as random cropping and flipping change the spatial correspondence of pixels, so the perturbation pattern optimized for the original configuration becomes partially mismatched after transformation. Empirically, different views exhibit heterogeneous behaviors: while some views remain strongly influenced by the attack, others still produce confident and semantically consistent predictions (Figure 1 (b)). This observation suggests that robustness should be built by exploiting the distribution of augmented views and the semantic relations it contains, rather than relying solely on sample-level confidence heuristics.

Motivated by the above observation, we propose \(\texttt{RITA}\), a robust test-time framework that shifts from sample-level matching to distribution-level alignment. Instead of relying on a single image embedding, \(\texttt{RITA}\) models the visual input as a discrete distribution over augmented views. To bridge these visual features with textual prototypes, we formulate the alignment objective as an Optimal Transport (OT) [20] problem. As illustrated in Figure 1 (c), this formulation enables us to evaluate the global geometric correspondence between the visual distribution and textual representations, mitigating the influence of adversarial outliers to rectify semantic misalignment. Moreover, test samples arrive as a continuous stream, providing additional information beyond a single image. To leverage this property, \(\texttt{RITA}\) incorporates a dynamic cache mechanism that progressively accumulates reliable semantic views, and uses them to further refine distribution alignment online. As demonstrated in Figure 1 (d), this progressive adaptation significantly enhances zero-shot robustness and maintaining competitive performance on clean data.

Extensive experiments on multiple standard benchmarks under diverse adversarial attacks demonstrate that \(\texttt{RITA}\) significantly enhances zero-shot robustness while preserving competitive performance on clean data. Our contributions are summarized as follows:

• We propose \(\texttt{RITA}\), a robust test-time prompt adaptation framework that leverages augmented views to rectify adversarial misalignment at the distribution level.

• We formulate cross-modal semantic alignment as an Optimal Transport problem, complemented by a dynamic cache for progressive refinement.

• We conduct comprehensive evaluations, showing that \(\texttt{RITA}\) consistently outperforms existing test-time adaptation methods.

2 Related Work↩︎

The vulnerability of VLMs (e.g., CLIP [6]) to adversarial perturbations remains a critical challenge [8], [21]–[23]. Attacks have evolved from uni-modal perturbations [24], [25] to multi-modal strategies like Co-Attack [26] that disrupt cross-modal alignment. To enhance robustness, prior defenses utilize training-time strategies, primarily adversarial contrastive tuning [11]–[13]. However, these methods typically require expensive re-training and labeled data, limiting practicality. Consequently, inference-time robustness has emerged to secure models without weight updates, such as diffusion purification [27], [28] and optimization-based methods [29], [30]. Notably, Test-Time Prompt Tuning approaches like TAPT [18] and R-TPT [19] adapt prompts using unlabeled data via contrastive learning or entropy minimization. Nevertheless, these paradigms typically treat visual inputs as isolated points, failing to exploit the underlying distributional geometry of adversarial samples. In contrast, we propose \(\texttt{RITA}\), a test-time prompt tuning framework that shifts from point-level alignment to distribution-level modeling.

Optimal Transport (OT) [20] provides a principled way to compare probability distributions by accounting for the geometry of the underlying feature space. With efficient solvers such as Sinkhorn [31], [32], OT has been widely used in generative modeling [33], structural alignment [34], and domain adaptation [35]. In vision–language learning, OT has also been applied to reduce semantic misalignment, including few-shot learning [36], distribution calibration [10], [37]–[39], and prompt learning [40]–[42]. For example, PLOT [40] aligns image features with multiple prompts via OT-based matching to capture diverse semantics, while ALIGN [41] further introduces hierarchical/token-level transportation for fine-grained cross-modal alignment [43], [44]. AWT [45] similarly formulates image-text distance as an OT problem to model semantic correlations in the joint space. However, these approaches primarily focus on representation enhancement in clean settings. They rely on undistorted visual manifolds and do not account for the severe structural perturbations caused by adversarial attacks. Differing from these approaches, \(\texttt{RITA}\) repurposes OT for adversarial defense, aiming to reconstruct the distribution-level correspondence disrupted by attacks. This enables inference-time correction of structural misalignment without parameter updates, ensuring robust deployment.

3 Method↩︎

An overview of \(\texttt{RITA}\) is illustrated in Figure 2. We first introduce the preliminaries in Sec. 3.1, then detail our distributed feature modeling and dynamic distribution alignment in Sec. 3.2 and 3.3, respectively. Finally, we provide a theoretical justification in Sec. 3.4.

3.1 Preliminary↩︎

Test-time prompt tuning. Test-Time Prompt Tuning (TPT) [16] improves the zero-shot generalization of CLIP [6] by adapting textual prompts at inference time, without accessing labeled data or updating CLIP parameters. Given a test image \(x_t\), TPT constructs a set of \(N\) stochastic augmentations \(\mathcal{X}_t=\{x_t^{n}\}_{n=1}^{N}\). For each augmented view \(x_t^{n}\), the visual representation is \(\mathbf{x}_t^{n}=\Phi_{\mathrm{img}}(x_t^{n}),\) where \(\Phi_{\mathrm{img}}(\cdot)\) denotes the CLIP image encoder. On the text side, the prompt for class \(k\) is formulated as \(z_k=\{\omega_1,\omega_2,\dots,\omega_L,c_k\},\) where \(c_k\) is the token embedding of the class name, and \(\omega=\{\omega_\ell\}_{\ell=1}^{L}\) are learnable context vectors shared across classes. The textual representation is \(\mathbf{z}_k=\Phi_{\mathrm{text}}(z_k),\) with \(\Phi_{\mathrm{text}}(\cdot)\) being the CLIP text encoder. For each view \(x_t^n\), the prediction probability is computed by feature matching: \[\label{eq:zero95shot} p(k \mid x_t^n; \omega) = \frac{\exp \big( \cos(\mathbf{x}_t^n, \mathbf{z}_k) / \tau \big)}{\sum_{j=1}^K \exp \big( \cos( \mathbf{x}_t^n, \mathbf{z}_j ) / \tau \big)},\tag{1}\] where \(\tau\) is the temperature parameter and \(K\) is the number of categories. Specifically, TPT employs a confidence selection strategy to filter out unreliable augmentations. It selects a subset \(\mathcal{S}\) of low-entropy views to compute the aggregated prediction \(\bar{p}(k \mid x_t; \omega) = \frac{1}{|\mathcal{S}|} \sum_{n \in \mathcal{S}} p(k \mid x_t^n; \omega)\). The prompt \(\omega\) is then optimized by minimizing the entropy of this aggregated distribution \(\bar{p}\): \[\omega^{*} = \operatorname*{argmin}_{\omega} \Big( -\sum_{k=1}^K \bar{p}(k \mid x_t; \omega) \log \bar{p}(k \mid x_t; \omega) \Big).\]

Optimal Transport (OT) [20] provides a principled way to measure the discrepancy between two probability distributions. Consider two discrete distributions in the feature space, \(\mathbb{P} = \sum_{n=1}^{N} a^n \delta_{\mathbf{x}^n}\) and \(\mathbb{Q} = \sum_{m=1}^{M} b^m \delta_{\mathbf{z}^m}\), where \(\delta_{\mathbf{v}}\) denotes the Dirac delta function at location \(\mathbf{v}\), and \(\mathbf{a} \in \Delta_N, \mathbf{b} \in \Delta_M\) are probability vectors. Given a cost matrix \(\mathbf{C} \in \mathbb{R}^{N \times M}\), where \(\mathbf{C}_{nm}\) measures the transport cost from \(\mathbf{x}^n\) to \(\mathbf{z}^m\), the entropy-regularized OT distance is defined as: \[d_{\mathrm{OT}}(\mathbb{P},\mathbb{Q};\mathbf{C}) = \min_{\mathbf{T} \in \Pi(\mathbf{a}, \mathbf{b})} \langle \mathbf{T}, \mathbf{C} \rangle - \lambda h(\mathbf{T}),\] where \(\Pi(\mathbf{a}, \mathbf{b}) = \{ \mathbf{T} \in \mathbb{R}_+^{N \times M} \mid \mathbf{T}\mathbb{1}_{M}=\mathbf{a}, \mathbf{T}^{\top}\mathbb{1}_{N}=\mathbf{b} \}\) is the transport polytope, \(h(\mathbf{T}) = -\sum_{n,m} T_{nm}\log T_{nm}\) is the entropic regularization term, and \(\lambda \geq 0\) controls the regularization strength. This formulation enables efficient computation via the Sinkhorn algorithm.

3.2 Distributed Features Modeling↩︎

In this section, we present our core strategy for robust test-time inference. Moving beyond vulnerable holistic image embeddings, we model the adversarial image and textual prompts as discrete distributions and align them structurally using optimal transport.

Adversarial attacks aim to degrade model predictions by introducing small, carefully crafted perturbations. In a white-box setting, given a clean image \(x\) and its ground-truth label \(y\), the adversarial example \(x'\) is generated by maximizing the cross-entropy loss within an \(\ell_p\)-norm constraint: \[x' = \operatorname*{argmax}_{\|x' - x\|_p \leq \epsilon_{\mathrm{adv}}} \mathcal{L}_{\mathrm{CE}}\Big(p(\cdot \mid x'; z), y\Big),\] where \(\epsilon_{\mathrm{adv}}\) denotes the perturbation budget, and \(p(\cdot \mid x'; z)\) represents the probability distribution over \(K\) classes as defined in Eq. 1 . Typically, we approximate this optimization using the iterative Projected Gradient Descent (PGD) [8].

While adversarial attacks distort alignment at the global representation level, relying on single-point embeddings is insufficient to capture the semantic variations. To alleviate this, we construct sets of diverse visual and textual representations and model their correspondence at the distribution level.

Specifically, given an adversarial image \(\hat{x}_t\), we apply data augmentations to obtain \(N\) views \(\{\hat{x}_t^n\}_{n=1}^{N}\), producing visual features \(\{\hat{\mathbf{x}}_t^n\}_{n=1}^{N}\). Instead of a single text prototype, for each class \(k\), we construct \(M\) prompts \(\{z_k^{(m)}\}_{m=1}^{M}\) using learnable context vectors, yielding textual features \(\{\mathbf{z}_k^{m}\}_{m=1}^{M}\). We model the adversarial image and the \(k\)-th class prototype as discrete distributions: \[\mathbb{P}_{t} = \sum_{n=1}^{N} \frac{1}{N} \, \delta_{\hat{\mathbf{x}}_t^n}, \quad \mathbb{Q}_{k} = \sum_{m=1}^{M} \frac{1}{M} \, \delta_{\mathbf{z}_k^{m}}.\]

Here, we assign uniform weights to both modalities, i.e., \(\mathbf{a}_t=\frac{1}{N}\mathbb{1}_N\) and \(\mathbf{b}_k=\frac{1}{M}\mathbb{1}_M\), such that the marginal constraints of OT are satisfied. We then measure the distribution-level alignment between the visual distribution \(\mathbb{P}_t\) and the textual prototype distribution \(\mathbb{Q}_k\) via entropy-regularized optimal transport: \[d_{\mathrm{OT}}(\mathbb{P}_t, \mathbb{Q}_k; \mathbf{C}_{t,k}) = \min_{\mathbf{T}_{t,k} \in \Pi} \left( \langle \mathbf{T}_{t,k}, \mathbf{C}_{t,k} \rangle - \lambda h(\mathbf{T}_{t,k}) \right),\] where \(\mathbf{T}_{t,k}\in\mathbb{R}_+^{N\times M}\) is the transport plan, and the cost matrix is defined by the cosine distance, \(\mathbf{C}_{t,k}(n,m)=1-\cos(\hat{\mathbf{x}}_t^{\,n}, \mathbf{z}_k^{\,m})\). Intuitively, a smaller OT distance indicates better alignment between the test image and class \(k\) at the distribution level. Accordingly, we predict the label by selecting the class with the minimum transport cost: \[\hat{y} = \operatorname*{argmin}_{k\in[K]} d_{\mathrm{OT}}(\mathbb{P}_t, \mathbb{Q}_k; \mathbf{C}_{t,k}).\]

3.3 Dynamic Distribution Alignment↩︎

While distribution-level alignment in Sec. 3.2 effectively processes individual samples, it treats inference steps in isolation, neglecting the semantic consensus in the continuous test stream. To exploit this temporal information, we introduce a dynamic cache mechanism that accumulates reliable visual features to iteratively refine the alignment online.

Confidence-based cache update. To update the cache with reliable samples, we evaluate the confidence of each individual augmented view. Instead of solving the full global transport problem, we quantify the instance-level alignment by measuring the average affinity between the view \(\hat{\mathbf{x}}_t^n\) and the text distribution of class \(k\): \[p_t^{n}(k) = \frac{\exp\!\left( \frac{1}{M}\sum_{m=1}^{\tau M}\cos(\hat{\mathbf{x}}_t^{\,n},\mathbf{z}_k^{m}) \right)}{\sum_{j=1}^{K}\exp\!\left(\frac{1}{M}\sum_{m=1}^{\tau M}\cos(\hat{\mathbf{x}}_t^{\,n},\mathbf{z}_j^{m})\right)}.\] We compute entropy \(H(p_t^{n})=-\sum_{k=1}^{K}p_t^{n}(k)\log p_t^{n}(k)\) and retain confident views: \[\mathcal{B}_t=\left\{\, (\hat{\mathbf{x}}_t^{\,n},\hat{y}_t^{\,n})\;\middle|\;H(p_t^{n})\le \gamma,\; \hat{y}_t^{\,n}=\operatorname*{argmax}_k p_t^{n}(k)\,\right\}.\] We maintain a class-wise cache \(\{\hat{\mathbf{X}}_k\}_{k=1}^{K}\), where \(\hat{\mathbf{X}}_k\in\mathbb{R}^{N_k\times d}\) stores cached visual features pseudo-labeled as class \(k\) (\(N_k\) is the current cache size). The cache is updated online by prioritizing lower-entropy samples. To bridge the modality gap, we align cached visual features to the textual space of class \(k\) by solving an Orthogonal Procrustes [46] problem: \[\mathbf{W}_k^\ast = \operatorname*{argmin}_{\mathbf{W}^\top \mathbf{W}=\mathbf{I}} \left\| \hat{\mathbf{X}}_k \mathbf{W} - \mathbf{1}_{N_k}\bar{\mathbf{z}}_k^\top \right\|_F^2,\] where \(\bar{\mathbf{z}}_k = \frac{1}{M}\sum_{m=1}^M \mathbf{z}_k^m\) is the mean text embedding for class \(k\). The aligned cached features are \(\tilde{\mathbf{X}}_k = \hat{\mathbf{X}}_k \mathbf{W}_k^\ast\).

Cache-based distribution matching. For each class \(k\), we instantiate a discrete probability distribution over the aligned cached features: \[\tilde{\mathbb{Q}}_k = \frac{1}{N_k}\sum_{j=1}^{N_k}\delta_{\tilde{\mathbf{x}}_k^{\,j}}, \qquad \text{where } \tilde{\mathbf{x}}_k^{\,j}\in \tilde{\mathbf{X}}_k.\] We quantify the discrepancy via the same entropy-regularized OT distance as in Sec. 3.2, but with a cache-specific cost matrix \(\tilde{\mathbf{C}}_{t,k}(n,j)=1-\cos(\hat{\mathbf{x}}_t^{\,n},\tilde{\mathbf{x}}_k^{\,j})\), and denote it as \(d_{\mathrm{cOT}}(\mathbb{P}_t, \tilde{\mathbb{Q}}_k; \tilde{\mathbf{C}}_{t,k})\).

Final inference. We classify the test sample by identifying the category that minimizes the joint transport cost, which integrates both the global prompt alignment and the local cache consensus: \[\label{eq:final95pred} \hat{y} = \operatorname*{argmin}_{k\in[K]} \Big( d_{\mathrm{OT}}(\mathbb{P}_t, \mathbb{Q}_k; \mathbf{C}_{t,k}) +\alpha\, d_{\mathrm{cOT}}(\mathbb{P}_t, \tilde{\mathbb{Q}}_k; \tilde{\mathbf{C}}_{t,k}) \Big),\tag{2}\] where \(\alpha\ge 0\) controls the contribution of the dynamic cache.

3.4 Theoretical Analysis↩︎

Standard methods like TPT [16] and R-TPT [19] optimize centroid alignment via mean pooling and cosine similarity. For \(\ell_2\)-normalized CLIP features, this is equivalent to minimizing the squared Euclidean distance between centroids (\(\Vert\mathbf{x}-\mathbf{z}\Vert^2 = 2(1-\cos(\mathbf{x},\mathbf{z}))\)). To reveal \(\texttt{RITA}\)’s geometric advantage, we analyze alignment using the 2-Wasserstein distance (\(W_2\)). Let \(\mathbb{P}_t\) (\(\boldsymbol{\mu}_x,\boldsymbol{\Sigma}_x\)) and \(\mathbb{Q}_k\) (\(\boldsymbol{\mu}_z,\boldsymbol{\Sigma}_z\)) be the visual and textual distributions. We obtain the following decomposition:

Equation 3 holds exactly for Gaussian distributions and serves as a general lower bound, as derived in Appendix 5. This inequality highlights a critical robustness gap: minimizing only \(\mathcal{L}_{\mathrm{mean}}\) leaves structural variance \(\mathcal{R}_{\mathrm{var}}\) unconstrained, allowing attackers to distort distribution geometry.

4 Experiments↩︎

4.1 Setup↩︎

We evaluate our method on eight image classification benchmarks spanning a wide range of visual domains, including generic object recognition (Caltech101 [47]), texture recognition (DTD [48]), satellite imagery (EuroSAT [49]), human action recognition (UCF101 [50]), as well as several fine-grained classification tasks, namely Pets [51], Cars [52], Flowers [53], and Aircraft [54]. To further assess robustness under distribution shifts, we conduct additional evaluations on ImageNet [55] and four of its variants that share the same label space: ImageNetV2 [56], ImageNet-Sketch [57], ImageNet-A [58], and ImageNet-R [59]. These benchmarks introduce significant variations in image sources, styles, and underlying visual statistics. Detailed analysis of these datasets is provided in Appendix 7.1.

Implementation details. We build all experiments on the official pre-trained CLIP models with two backbones, CLIP-ViT-B/32 and CLIP-ViT-B/16. We generate adversarial examples using PGD [8] under an \(L_\infty\) constraint. We use \(\epsilon=4.0\) with 7 steps for both backbones. At test time, we update only the prompt parameters while keeping the CLIP backbone freezed. The prompt number \(M\) is set to 4 and initialized with the template “a photo of a”. We use AdamW and fix the Test-Time Adaptation (TTA) step at 1 per test sample, with a learning rate of 0.005. We apply standard test-time augmentations for images, including random cropping, resizing, and horizontal flipping. For text, we use an Large Language Model (LLM) to generate class-specific descriptions[45]. For each test image, we sample \(N=64\) augmented views (including the original). For cache construction, We set the entropy threshold to \(\gamma=0.8\). More experiment details are provided in Appendix 6.

We compare RITA with CLIP-based test-time adaptation baselines, including TPT [16], R-TPT [19], C-TPT [15], and MTA [60], as well as the zero-shot CLIP baseline. We also report an Ensemble baseline that averages predictions over augmented views. To further evaluate compatibility with robust pre-trained weights, We incorporate three representative adversarially fine-tuned CLIP models, namely TeCoA [11], PMG [13], and FARE [12]. All methods follow the instance-level test-time adaptation protocol: each test sample is adapted and predicted independently, without access to other test samples.

4.2 Main Results↩︎

We evaluate our method on eight fine-grained benchmark datasets with ViT-B/32 and ViT-B/16 backbones, as presented in Table 1. The results show that \(\texttt{RITA}\) achieves the highest adversarial accuracy on nearly all datasets. Compared to vanilla CLIP, \(\texttt{RITA}\) improves average robustness by 45.0% and 50.9% with ViT-B/32 and ViT-B/16, respectively. This effectively mitigates the severe vulnerability of the baseline under adversarial attacks, which can cause near-zero accuracy on datasets such as Cars and Aircraft. Furthermore, compared to R-TPT, the state-of-the-art test-time adaptation method, \(\texttt{RITA}\) achieves consistent gains in average robust accuracy, outperforming it by 1.8% and 2.2% on the two architectures. Meanwhile, our method also achieves the highest average accuracy on clean samples.This demonstrates that \(\texttt{RITA}\) enhances adversarial robustness while preserving recognition performance in attack-free environments with negligible degradation. Notably, \(\texttt{RITA}\) performs better with ViT-B/16 than with ViT-B/32, aligning with the intuition that a fine-grained ViT backbone yields stronger recognition capabilities.

As an inherently plug-and-play framework, we integrated \(\texttt{RITA}\) with three representative adversarially fine-tuned models, with results summarized in Table [tbl:tab:ablation95finetuning]. \(\texttt{RITA}\) exhibits significant synergy with these robust baselines, substantially enhancing robust accuracy across all datasets without sacrificing clean performance. Notably, when combined with FARE, \(\texttt{RITA}\) achieves a striking 81.9% robust accuracy on Caltech101, representing a 19.0% improvement over the baseline. These results demonstrate that \(\texttt{RITA}\) is a versatile adaptation framework that can seamlessly fortify existing adversarial defense models during inference.

Table 3 further assesses the generalizability of \(\texttt{RITA}\) against CW [61] and DI [25] attacks. \(\texttt{RITA}\) consistently demonstrates superior defense across protocols, achieving a leading average accuracy of 54.0% under CW attacks. Under the more challenging DI attack, \(\texttt{RITA}\) maintains its advantage with 46.2% accuracy, surpassing R-TPT by 2.0%. These results suggest that \(\texttt{RITA}\) enhances the intrinsic robustness of VLMs against diverse adversarial threats via its effective test-time adaptation mechanism.

4.3 Ablation Study↩︎

As illustrated in Figure 3, we conduct sensitivity analyses on the DTD dataset regarding the number of learnable prompts and augmentation views. Figure 3 (a) shows that increasing the number of learnable prompts from 4 to 32 leads to steady performance gains across both settings. Figure 3 (b) assesses the sensitivity of augmentation view scales during inference, where adversarial accuracy peaks at 96 views. Notably, scaling to 128 views results in a slight performance decline, a trend consistently observed in noise-free environments as well. Consequently, selecting a moderate number of augmentation views enables a superior trade-off between efficiency and accuracy.

We investigate the sensitivity of \(\texttt{RITA}\) to the integration coefficient \(\alpha\) during inference on the DTD and ImageNet datasets, as shown in Figure 4. Experimental results confirm that the cache mechanism effectively boosts performance. On both datasets, increasing \(\alpha\) from 0 to 0.1 yields a significant accuracy gain, followed by a slight upward trend that gradually stabilizes as \(\alpha\) further scales. These results indicate that the visual priors stored in the cache not only play a crucial role in correcting predictions under adversarial settings, but also offer effective semantic complementarity during the inference stage. Ablation study of the dynamic cache is provided in Appendix 8.1.

To assess the resilience of \(\texttt{RITA}\) under various attack intensities, we conduct ablation studies on all fine-grained datasets and ImageNet dataset, with the results illustrated in Figure 5. As the perturbation budget \(\epsilon\) increases from 1 to 4, the accuracy declines as expected, indicating the generation of more potent perturbations. Simultaneously, we observe that increasing the number of TTA steps generally yields additional and consistent defensive gains across evaluations.

We report the average robust accuracy across three different backbones on all fine-grained datasets as shown in Figure 6 (a) and the ImageNet dataset as illustrated in Figure 6 (b). The experimental results indicate that all models reach their performance peak at step 2, while the accuracy slightly declines when the steps are increased to 4 across all evaluated datasets. This observation justifies our choice of a small number of iterations, e.g., TTA step = 1, which simultaneously ensures superior robustness and high computational efficiency for real-time inference.

Table [tbl:tab:ablation95running95time] presents the average comparison of running time, clean accuracy, and robust accuracy across various methods on fine-grained datasets. The empirical results indicate that \(\texttt{RITA}\) achieves superior classification performance while maintaining highly competitive inference efficiency compared with existing adaptation methods. Compared to MTA, which exhibits the fastest inference speed, \(\texttt{RITA}\) maintains a substantial lead of 11.8% in robust accuracy. These observations demonstrate that \(\texttt{RITA}\) strikes an excellent balance between computational cost and model robustness, delivering more resilient and precise inference results with minimal additional time latency.

As shown in Figure 7, we visualize semantic alignment quality via KDE curves of the Kullback-Leibler (KL) divergence. Specifically, we measure the divergence between the class-conditional distributions over text prototypes and the ideal one-hot targets. On both DTD and Caltech101 datasets, the curves for augmented features exhibit a pronounced leftward shift and higher concentration in the low-value region compared to the original features. This demonstrates that our cache mechanism effectively rectifies adversarial semantic biases, establishing more deterministic vision-text associations. Extended analysis is provided in Appendix 8.2.

Conclusion & Limitations↩︎

In this work, we propose \(\texttt{RITA}\), a robust test-time prompt adaptation framework that enhances the adversarial robustness of VLMs without requiring retraining or access to labeled data. By modeling augmented visual features and prompt-induced textual prototypes as distributions and aligning them via optimal transport, \(\texttt{RITA}\) corrects cross-modal semantic misalignment caused by adversarial perturbations. Furthermore, a dynamic cache mechanism progressively aggregates reliable semantic cues from the test stream to refine alignment online. Extensive experiments across diverse benchmarks, attack types, and model backbones demonstrate that \(\texttt{RITA}\) consistently improves adversarial robustness while preserving competitive performance on clean data. Our results suggest that distribution-level alignment is a principled and effective paradigm for robust inference in large pre-trained VLMs.

While \(\texttt{RITA}\) demonstrates strong image classification performance, its extension to generative tasks like image captioning remains for future exploration. We believe our distribution-level alignment provides a foundation for adaptation in these broader multimodal scenarios.

Impact Statement↩︎

This work improves the reliability and adversarial robustness of pre-trained VLMs, which is important for their deployment in safety-sensitive applications. The proposed method operates entirely at test time, without modifying model parameters or requiring additional training data, making it lightweight and easy to integrate into existing systems. Nevertheless, it does not eliminate all security risks, and adversarial attacks may continue to evolve. Future work should further investigate robust inference-time defenses and evaluate potential failure modes and misuse risks.

Acknowledgement↩︎

This research is supported by the National Natural Science Foundation of China (No. 62576330) and the National Natural Science Foundation of Anhui (No.2508085MF143).

5 Proof of Theorem 1↩︎

6 Implementation Details↩︎

For fair comparison, all approaches use the same pre-trained CLIP backbone and identical AugMix-based test-time augmentations, without external data, foundation models, or additional knowledge. We report average accuracy on clean samples and adversarial accuracy under PGD attacks with different perturbation budgets for default. Adversarial examples are generated on the original CLIP model, reflecting a realistic threat model.

For most experiments, we adopt a default setting that utilizes class descriptors and sets the subset size \(|\mathcal{S}|=64\). However, to accommodate the distinct characteristics of specific benchmarks, we adjust these parameters for several datasets as follows: 1) For EuroSAT, we disable descriptors for the ViT-B/16 backbone and set \(|\mathcal{S}|=32\) for both ViT-B/16 and ViT-B/32; 2) For ImageNet-A and ImageNet-R, the size of subset \(|\mathcal{S}|\) is 32; 3) For ImageNet-Sketch, descriptors are disabled, and the subset size \(|\mathcal{S}|\) is 32. For all other datasets not explicitly mentioned, the default configuration is maintained. These refinements are designed to better align domain-specific semantic features with the test-time adaptation process.

7 Additional Results↩︎

7.1 Results on ImageNet and ImageNet-OOD datasets.↩︎

Table 5 presents the performance comparison across ImageNet and its four Out-of-Distribution (OOD) variants. The results demonstrate that \(\texttt{RITA}\) maintains a significant robustness advantage even when addressing large-scale general visual tasks. On the standard ImageNet, \(\texttt{RITA}\) achieves state-of-the-art or competitive accuracies across both architectures under both settings; for instance, it reaches 55.1% robustness on ViT-B/16, a substantial leap from the vanilla CLIP’s 0.6%. \(\texttt{RITA}\)’s performance is equally compelling on the more challenging OOD variants, where its average OOD robust accuracy reaches 39.3% and 46.2% in two backbones, outperforming other methods. These findings validate that \(\texttt{RITA}\) not only excels in fine-grained tasks but also provides robust protection against diverse adversarial threats in large-scale general scenarios and under various distribution shifts.

7.2 Evaluation under enhanced adversarial attacks.↩︎

Table 6 presents the performance of various test-time adaptation methods across eight fine-grained datasets under a more stringent adversarial constraint, where \(\epsilon\) is set to 4.0. The experimental results indicate that as the attack intensity increases, the accuracy of the vanilla CLIP drops nearly to zero, whereas \(\texttt{RITA}\) demonstrates exceptional interference resistance. Specifically, our method achieves state-of-the-art results across both backbones, reaching 32.9% and 35.0% in average robust accuracy, respectively. These findings further confirm that \(\texttt{RITA}\) maintains high classification precision while exhibiting remarkable predictive stability, thereby validating its substantial practical value in mitigating complex and intense adversarial risks in real-world scenarios.

7.3 Analysis on an alternative CLIP backbone.↩︎

In Table 7, we further evaluate the performance of \(\texttt{RITA}\) using RN50 as the vision backbone to verify its generalizability across different architectures. The experimental results demonstrate that \(\texttt{RITA}\) exhibits superior robustness across all eight fine-grained classification datasets. Notably, despite RN50 having a relatively weaker baseline representation capability compared to the ViT series, \(\texttt{RITA}\) consistently outperforms other adaptation methods, while maintaining high clean accuracy. These findings provide strong evidence that our method delivers cross-architecture robustness gains and effectively mitigates the vulnerability of diverse vision encoders to adversarial attacks.

7.4 Robustness evaluation under other attacks.↩︎

We conduct experiments under various adversarial attack protocols, including adaptive attack that is aware of augmentation strategies, AutoAttack (AA), and FGSM. As reported in Table 8, \(\texttt{RITA}\) consistently achieves the highest robust accuracy across all eight datasets and all attack types compared to the CLIP and R-TPT baselines. Specifically, under the more rigorous AutoAttack, \(\texttt{RITA}\) maintains an average robust accuracy of 55.4%, outperforming R-TPT by 2.8%. Notably, \(\texttt{RITA}\) shows significant gains on challenging datasets like DTD and EuroSAT across all attack settings. These results demonstrate that the defensive capability of \(\texttt{RITA}\) is not tailored to a specific attack but generalizes well to diverse adversarial threats, confirming its potential for securing VLMs in various hostile environments.

7.5 Generalization to Other VLM Backbones.↩︎

To further validate the architectural agnosticity of \(\texttt{RITA}\), we extend our evaluation to other VLMs, including OpenCLIP and EVA-CLIP. As shown in Table 9, we expanded \(\texttt{RITA}\) to 10 datasets by incorporating SUN397 and Food101. The results consistently demonstrate that \(\texttt{RITA}\) significantly boosts adversarial robustness across all architectures. For instance, when applied to OpenCLIP, \(\texttt{RITA}\) improves the average robust accuracy from 2.4% to 54.4%. On the newly added SUN397 and Food101 datasets, \(\texttt{RITA}\) achieves substantial gains, reaching robust accuracies of 54.0% and 60.7% for OpenCLIP, and 59.2% and 62.4% for EVA-CLIP, respectively. These findings underscore that the robustness gains of \(\texttt{RITA}\) are consistent across diverse VLM backbones and broader dataset distributions, reinforcing its effectiveness as a general test-time adaptation framework.

8 Verification of the importance of the cache mechanism↩︎

8.1 Ablation study of the dynamic cache.↩︎

In Table 10, we conduct an ablation study to specifically validate the significance of the cache mechanism \(d_{cOT}\) within the \(\texttt{RITA}\) framework. The results indicate that while utilizing the cache mechanism in isolation yields lower standard accuracy (39.7% on fine-grained and 20.2% on ImageNet) due to the absence of real-time alignment, it consistently outperforms the vanilla CLIP baseline in robustness metrics, achieving 27.1% and 29.6% respectively, compared to CLIP’s 4.7% and 3.2%. This evidence underscores that the historical priors preserved in the cache serve as an essential reference for stabilizing predictions under adversarial perturbations. Most importantly, the synergy between the cache mechanism and the optimal transport module \(d_{OT}\) leads to peak performance across all metrics, notably boosting fine-grained robustness from 49.4% to 55.0%. This further demonstrates that the cache provides critical semantic supplementation to the multimodal alignment, establishing a more robust and historically-aware inference framework.

8.2 Semantic alignment analysis of the cache mechanism.↩︎

To demonstrate the effectiveness of the cache mechanism, we extract the augmented view features from the DTD and Caltech101 datasets and compare them with the original unaugmented visual features from a "vision-text" alignment perspective. Specifically, for each class, we first compute the class-conditional assignment distribution over all text prototypes, which reflects how the visual features of each class are semantically aligned with the textual features. Subsequently, we measure the KL divergence between this empirical distribution and an ideal one-hot target distribution that assigns all probability mass to the ground-truth class. A lower KL value indicates stronger vision-text alignment.

As illustrated in Figure 8, we present the scatter plots of KL divergence for all classes in DTD and Caltech101. Each point represents the vision-text alignment quality of a specific category, where red dots denote original adversarial views and green dots represent selected augmented views utilized by our cache mechanism. The KL divergence of original views exhibits high variance and remains at an elevated level, reflecting severe adversarial bias. Upon introducing the cache mechanism, the green dots show both a downward shift and reduced dispersion. This indicates that the augmented views effectively calibrate the corrupted feature representations by aggregating historical priors, pulling the model closer to the ideal one-hot distribution. Even for categories where the original KL divergence is particularly high, the augmented features still achieve significant alignment gains. This cross-category consistency provides a granular foundation for the superior robustness of the \(\texttt{RITA}\) framework when encountering diverse adversarial attacks.

9 Algorithm for RITA↩︎

Algorithm 9 summarizes the \(\texttt{RITA}\) framework. \(\texttt{RITA}\) extracts features from augmented views, aligns visual-textual distributions via optimal transport, and maintains a dynamic cache for progressive refinement.

References↩︎