1 Introduction↩︎

Despite significant progress in machine learning that has enabled a broad range of classification tasks [1]–[6], most models are developed and evaluated under a closed-world assumption, where test samples follow the same distribution as the training data. In real-world deployments, however, models frequently operate in open-world settings, where they may encounter classes unseen during training—so-called out-of-distribution (OOD) data. Such OOD inputs can undermine model reliability and, in some cases, severely degrade performance. Therefore, a dependable discriminative model should not only correctly classify in-distribution (ID) samples but also identify OOD samples as unknown. This motivates OOD detection [7]–[9], a capability that is critical for safety in decision-sensitive applications such as autonomous driving [10], medical diagnosis [11], and cybersecurity [12].

This paper studies post-hoc OOD detection, which is often more practical than learning-based approaches that require resource-intensive retraining. Earlier works [13]–[17] largely relied on single-modality information from pre-trained models. More recently, the success of contrastive language–image pre-training (CLIP) [18] has shifted attention toward extending post-hoc OOD detection from unimodal to multimodal settings.

A pioneering line of work [19] introduces a trainable captioner to generate candidate OOD labels for matching OOD images but fails to scale to large-scale datasets, such as ImageNet-1k, that contain a large number of ID classes. On the contrary, MCM [20] treats textual features as concept prototypes for each ID class and measures OOD uncertainty via the scaled distance between an image feature and its nearest ID prototype. This approach helped establish the use of pre-trained vision–language models (VLMs) for post-hoc OOD detection. Nevertheless, MCM relies solely on textual information derived from the ID label space, leaving the text understanding capacity of VLMs under-exploited. To address this issue, NegLabel [21] selects negative labels from in-the-wild lexical resources (e.g., WordNet [22]), based on similarity to the ID label space, thereby strengthening the model’s ability to separate OOD from ID samples.

Despite this progress, a fundamental question remains insufficiently examined: are text embeddings genuinely suitable surrogates for visual class prototypes in zero-shot OOD detection? Existing CLIP-based methods implicitly assume that textual representations of class names are well aligned with the optimal class prototypes in the visual feature space. While CLIP maps images and texts into a shared embedding space, recent evidence suggests that the text and vision representations learned by CLIP-based models remain distinct and separated by a clear margin [23]–[25].

In this paper, we theoretically characterize an intrinsic limitation of text-as-prototype strategies in CLIP-based post-hoc OOD detection. Under an idealized supervised formulation, we derive a closed-form expression for the optimal visual class prototypes and prove that these prototypes must lie in the span of visual features. In contrast, text-derived prototypes generally include components orthogonal to this visual subspace. This mismatch yields a non-vanishing lower bound on the distance between textual prototypes and optimal visual prototypes, which we formalize as a modality gap. Our analysis indicates that this gap is structural and therefore cannot be removed solely through improved prompt engineering or negative-label mining.

To mitigate the modality gap, we propose a principled framework that directly learns class prototypes in the visual feature space. Specifically, we design a pseudo-supervised objective constructed from soft predictions produced by a pre-trained CLIP-based model. We further develop an online optimization procedure that incrementally updates visual prototypes using unlabeled test-time data streams, thereby respecting the post-hoc constraint. Finally, we establish theoretical guarantees on the convergence behavior of the proposed online optimization procedure. Extensive experiments show that our method achieves state-of-the-art performance. In particular, on ImageNet-1K we obtain 12.79% FPR95 and 97.75% AUROC, surpassing AdaNeg [26] by 6.13% and 1.09%, respectively.

2 Preliminary↩︎

Notation. Let \(\mathcal{X}\) and \(\mathcal{Y}\) be the input space and the label space, respectively. Given a random variable \(Y \in \mathcal{Y}\), we write \(\mathbb{P}_Y\) as the marginal distribution defined over \(\mathcal{Y}\), and use \(y\sim\mathbb{P}_Y\) to indicate a sample \(y\) drawn from \(\mathbb{P}_Y\). Considering \(K\)-way classification as a case study, we write \(\mathcal{Y}_{\mathrm{I}}\triangleq\{y_1,\ldots,y_K\}\subset\mathcal{Y}\) as the known ID label space. The joint ID distribution \(\mathbb{P}_{X_{\mathrm{I}} Y_{\mathrm{I}}}\) is a joint distribution defined over \(\mathcal{X} \times \mathcal{Y}_{\mathrm{I}}\). During testing, there are some unknown OOD joint distributions \(\mathbb{P}_{X_{\rm o}Y_{\rm o}}\) defined over \(\mathcal{X}\times\mathcal{Y}_{\mathrm{o}}\), where \(\mathcal{Y}_{\mathrm{o}}\subseteq\mathcal{Y}\setminus\mathcal{Y}_{\mathrm{I}}\) is the unknown OOD label space.

Post-hoc OOD Scoring. Existing methods [14]–[17], [27] adopt a post-hoc strategy to detect OOD data, i.e., given a pre-trained ID classification model \(f\) and a scoring function \(S(\cdot;f):\mathcal{X}\rightarrow\mathbb{R}\), then \(\mathbf{x}\) is detected as ID data if and only if \(S(\mathbf{x};f)\geq \lambda\), for some given threshold \(\lambda\): \[\label{eq1} \small g(\mathbf{x})= \text{ID},~\text{if}~S(\mathbf{x};f) \geq \lambda;~ \text{otherwise},~g(\mathbf{x})=\text{OOD}.\tag{1}\] Typically, \(\lambda\) is chosen to ensure a high fraction (e.g., 95%) of ID data to be correctly classified.

CLIP-based Models adopt a dual-stream architecture [18] with one text encoder \(f_{\mathcal{T}}\) and one image encoder \(f_{\mathcal{X}}\) to map inputs of two modalities into a hyper-spherical space \(\mathbb{S}^{d-1}\triangleq\left \{\mathbf{z}\in\mathbb{R}^{d}|\left \|\mathbf{z}\right \|_2=1\right \}\). Given any image \(\mathbf{x}\in \mathcal{X}\) and any label \(y\in \mathcal{Y}\), the corresponding representation can be extracted as follows: \[\mathbf{z} = f_{\mathcal{X}}(\mathbf{x})\in \mathbb{S}^{d-1},~~\mathbf{r}_{y} = f_{\mathcal{T}}\big(\mathcal{P}(y)\big)\in \mathbb{S}^{d-1},\] where \(\mathcal{P}(\cdot)\) generates the text prompt for the class name associated with label \(y\).

CLIP-based OOD Detectors Studied. CLIP-based models, which are initially proposed for zero-shot ID classification, have recently been extended to zero-shot OOD detection where there is no need to train on ID samples. The pioneering work, MCM [20], treats the prompt of ID labels as concept prototypes and measures the ID-ness of the input image by comparing the similarity between the input image and the concept prototypes in the feature space learned by CLIP-based models, i.e., \[\begin{align} \label{MCM} S_{\text{MCM}}(\mathbf{x};f) &\triangleq \max_{y\in\mathcal{Y}_{\mathrm{I}}} P(y|\mathbf{x};\mathcal{Y}_{\mathrm{I}})\\ &=\frac{\max_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}\cdot\mathbf{r}_{y}/\tau)}{\sum_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}\cdot\mathbf{r}_{y}/\tau)}, \end{align}\tag{2}\] where \(\tau>0\) is a temperature hyper-parameter. Built upon this, NegLabel [21] introduces a \(L\)-sized set of negative labels¹ \(\mathcal{Y}_{\text{neg}}\triangleq\{y_{K+1},\ldots,y_{K+L}\}\) to formulate the OOD scoring function of \(\mathbf{x}\) as the model’s prediction confidence that \(\mathbf{x}\) belongs to \(\mathcal{Y}_{\mathrm{I}}\), i.e., \[\label{NegLabel} \begin{align} \small S_{\text{NegLabel}}(\mathbf{x};f) &\triangleq \sum_{y\in\mathcal{Y}_{\mathrm{I}}} P(y|\mathbf{x};\mathcal{Y}_{\mathrm{I}}\cup\mathcal{Y}_{\text{neg}})\\ & = \frac{\sum_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}\cdot\mathbf{r}_{y}/\tau)}{\sum_{y\in\mathcal{Y}_{\mathrm{I}}\cup\mathcal{Y}_{\text{neg}}} \exp(\mathbf{z}\cdot\mathbf{r}_{y}/\tau)}. \end{align}\tag{3}\]

3 Motivation↩︎

Despite the empirical success of CLIP-based methods for post-hoc OOD detection, we argue that their potential remains fundamentally under-exploited. The key limitation stems from the absence of training data in the zero-shot setting. As a result, existing approaches typically construct class prototypes directly from the text embeddings of class names. While computationally convenient, this practice implicitly assumes that textual prototypes are well-aligned with their optimal counterparts in the visual feature space.

In this work, we argue that this assumption is generally invalid. Even though CLIP embeds images and texts into a shared hyperspherical space, text-derived prototypes often exhibit a systematic mismatch with optimal visual prototypes. We refer to this discrepancy as the modality gap. To make this precise, we first analyze an idealized supervised setting where optimal visual prototypes can be characterized explicitly. This analysis provides a theoretical reference point for understanding the intrinsic limitations of text-only prototypes in zero-shot OOD detection.

Without loss of generality, let \(\mathcal{D}=\left\{\left(\mathbf{x}_i,\hat{y}_i\right)\right\}_{i=1}^N\sim\mathbb{P}_{X\hat{Y}}^N\) where \(\hat{\mathcal{Y}}\subseteq\mathcal{Y}\) denotes a label space of interest with cardinality \(|\hat{\mathcal{Y}}|=M\). The optimal class prototypes w.r.t. \(\hat{\mathcal{Y}}\) can be learned by minimizing the following supervised objective: \[\label{eq4} \mathcal{L}(\mathbf{W};\hat{\mathcal{Y}})=-\sum_{i=1}^N\log\frac{\exp(\mathbf{z}_i\cdot\mathbf{w}_{\hat{y}_i}/\kappa)}{\sum_{\hat{y}\in\hat{\mathcal{Y}}}\exp(\mathbf{z}_i\cdot\mathbf{w}_{\hat{y}}/\kappa)},\tag{4}\] where \(\kappa>0\) is another temperature hyper-parameter and \(\mathbf{W}=\left[\mathbf{w}_{\hat{y}}\right]_{\hat{y}\in\hat{\mathcal{Y}}}\) with \(\mathbf{w}_{\hat{y}}\in\mathbb{S}^{d-1}\) is the class prototype associated with label \(\hat{y}\) in the visual feature space.

According to Theorem 1, one may conclude that each optimal class prototype \(\mathbf{w}^*_{\hat{y}}\) should entirely lie in the visual feature space since it can be expressed as a linear combination of samples from that space. This conclusion will be central to our following analysis of the modality gap.

Remark. Theorem 2 establishes that the gap between textual prototypes and optimal visual prototypes is unavoidable whenever textual prototypes contain components outside the visual feature subspace \(\mathcal{S}\). Since \(\mathbf{w}_{\hat{y}}^{\ast}\in\mathcal{S}\) by Theorem 1, only \(\operatorname{Proj}_{\mathcal{S}}(\mathbf{r}_{\hat{y}})\) can align with \(\mathbf{w}_{\hat{y}}^{\ast}\), while \(\operatorname{Proj}_{\mathcal{S}^{\perp}}(\mathbf{r}_{\hat{y}})\) necessarily contributes to the mismatch. Hence, the lower bound in Theorem 2 vanishes only when each textual prototype lies entirely in \(\mathcal{S}\). As a result, using text embeddings as class prototypes without visual calibration is generally sub-optimal for recovering the optimal visual prototypes.

4 Methodology↩︎

4.1 An online pseudo-supervised learning framework↩︎

Albeit theoretically appealing, it is worth noting that recovering \(\mathbf{W}^*\) by minimizing Eq. (4 ) is technically infeasible in the zero-shot setting where the labeled training dataset \(\mathcal{D}\) is unavailable. While one can have access to unlabeled samples during testing, the absence of ground-truth labels precludes direct optimization of the supervised objective in Eq. (4 ). Fortunately, CLIP-based models exhibit strong zero-shot transfer performance, suggesting that their predictions can serve as a surrogate supervision signal. We thus replace hard ground-truth labels with soft pseudo-labels predicted by CLIP-based models, directly yielding a pseudo-supervised objective \(\hat{\mathcal{L}}(\mathbf{W})=\sum_{i=1}^N\hat{\ell}(\mathbf{W};\mathbf{x}_i,\hat{\mathcal{Y}}),\) where \[\label{eq7} \begin{align} &\hat{\ell}(\mathbf{W};\mathbf{x}_i,\hat{\mathcal{Y}})\triangleq\\ &\quad-\sum_{\hat{y}\in\hat{\mathcal{Y}}} P(\hat{y}|\mathbf{x}_i;\hat{\mathcal{Y}})\log\frac{\exp(\mathbf{z}_i\cdot\mathbf{w}_{\hat{y}}/\kappa)}{\sum_{y\in\hat{\mathcal{Y}}}\exp(\mathbf{z}_i\cdot\mathbf{w}_{y}/\kappa)}. \end{align}\tag{5}\] Given that only a single test sample will be received at each iteration, we propose to update \(\mathbf{W}\) in an online manner. In particular, when the \(i\)-th test sample \(\mathbf{x}_i\) arrives, we have \[\label{eq8} \small \mathbf{w}_{\hat{y}}^{(i)}=\ell_2\left(\mathbf{w}_{\hat{y}}^{(i-1)}-\eta_i \frac{\partial \hat{\ell}(\mathbf{W}^{(i-1)};\mathbf{x}_i,\hat{\mathcal{Y}})}{\partial \mathbf{w}_{\hat{y}}^{(i-1)}}\right),\forall\hat{y}\in\hat{\mathcal{Y}}\tag{6}\] where \(\eta_i=\rho/\sqrt{i}\) is a step size with \(\rho>0\) as a hyper-parameter. For completeness, Theorem 3 provides a theoretical guarantee for the convergence of our proposed online optimization in Eq. (6 ).

4.2 Online Prototype Learning for OOD Detection↩︎

To realize the idea, we, motivated by NegLabel [21], begin by extending the ID label space to include \(L\) additional placeholders. This is essentially equivalent to grouping potential target OOD data into \(L\) OOD pseudo-classes (clusters). For clarity, we denote the corresponding set of OOD pseudo-labels as \(\hat{\mathcal{Y}}_{\mathrm{o}}=\left\{\hat{y}_1,\ldots,\hat{y}_L\right\}\).

Due to the lack of explicit OOD information, NegLabel uses text features of negative labels as OOD pseudo-class prototypes such that \(\hat{\mathcal{Y}}_{\mathrm{o}}=\mathcal{Y}_{\text{neg}}\). However, this strategy suffers not only from the intra-modality discrepancy between negative labels and the true underlying OOD categories, but also from the inter-modality gap between text-derived prototypes and optimal visual prototypes, as discussed in Section 3.

In contrast, as we show later, we directly learn the prototypes of both ID classes and OOD pseudo-classes in the visual feature space via the proposed online optimization framework introduced in Section 4.1.

At high level, our key idea is to update ID class prototypes if an incoming test sample is classified as ID, and to update OOD pseudo-class prototypes otherwise. Since no human annotations indicating ID or OOD are available at test time, we introduce a hard thresholding rule based on the off-the-shelf score \(S_{\text{NegLabel}}(\mathbf{x}_i;f)\) in Eq. (3 ) to identify positive and negative samples from test data streams: \[\label{eq10} \begin{align} \text{Positive}: S_{\text{NegLabel}}(\mathbf{x}_i;f)&\geq\beta, \\ \text{Negative}: S_{\text{NegLabel}}(\mathbf{x}_i;f)&\leq1-\beta, \end{align}\tag{7}\] where \(\beta\in(0.5,1]\). We note that, with Eq. (7 ), test samples whose scores fall within \((1-\beta,\beta)\) are treated as uncertain and are not used to update any prototypes.

Given the \(i\)-th test sample \(\mathbf{x}_i\) (\(i=1,2,\ldots\)) to arrive, we denote by \(c_i^+\) and \(c_i^-\) to represent the cumulative number of positive and negative samples we have ever seen so far respectively such that \[\label{eq11} \begin{align} c_i^+ &= c_{i-1}^+ + \mathbb{1}\left(S_{\text{NegLabel}}(\mathbf{x}_i;f)\geq\beta\right),\\ c_i^- &=c_{i-1}^- + \mathbb{1}\left(S_{\text{NegLabel}}(\mathbf{x}_i;f)\le 1-\beta\right), \end{align}\tag{8}\] with \(c_0^+=c_0^-=0\) and \(\mathbb{1}(\cdot)\) as an indicator function.

For each \(y\in\textcolor{red}{\mathcal{Y}_{\mathrm{I}}}\), the ID class prototypes \(\mathbf{w}_{y}\) is updated as \[\label{eq12} \mathbf{w}_{y}^{(i)}= \begin{cases} \mathbf{w}_{y}^{(i-1)} \quad \text{ if } S_{\text{NegLabel}}(\mathbf{x}_i;f)<\beta,\\ \ell_2\left(\mathbf{w}_{y}^{(i-1)}-\textcolor{red}{\eta^+_i} \frac{\partial \hat{\ell}(\mathbf{W}^{(i-1)};\mathbf{x}_i,\textcolor{red}{\mathcal{Y}_{\mathrm{I}}})}{\partial \mathbf{w}_{y}^{(i-1)}}\right) \text{ otherwise}, \end{cases}\tag{9}\] where the step size is \(\eta_i^+=\rho/\sqrt{c_i^+}\).

Similarly, for each \(y\in\textcolor{orange}{\hat{\mathcal{Y}}_{\mathrm{o}}}\), we can update the OOD pseudo-class prototype \(\mathbf{w}_{y}\) as \[\label{eq13} \mathbf{w}_{y}^{(i)}= \begin{cases} \mathbf{w}_{y}^{(i-1)} \quad \text{ if } S_{\text{NegLabel}}(\mathbf{x}_i;f)>1-\beta,\\ \ell_2\left(\mathbf{w}_{y}^{(i-1)}-\textcolor{orange}{\eta^-_i} \frac{\partial \hat{\ell}(\mathbf{W}^{(i-1)};\mathbf{x}_i,\textcolor{orange}{\hat{\mathcal{Y}}_{\mathrm{o}}})}{\partial \mathbf{w}_{y}^{(i-1)}}\right) \text{ otherwise}. \end{cases}\tag{10}\] where the step size is \(\eta_i^-=\rho/\sqrt{c_i^-}\).

Based on the updated prototypes, we define the online OOD scoring function for \(\mathbf{x}_i\) as \[\label{eq14} S_{\text{ours}}(\mathbf{x}_i;f)= \frac{\sum_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}_i\cdot\mathbf{w}^{(i)}_{y}/\tau)}{\sum_{y\in\mathcal{Y}_{\mathrm{I}}\cup\hat{\mathcal{Y}}_{\mathrm{o}}} \exp(\mathbf{z}_i\cdot\mathbf{w}^{(i)}_{y}/\tau)}.\tag{11}\] where the prototypes are initialized as \[\label{eq15} \begin{align} \mathbf{w}^{(0)}_{y_i}&=f_{\mathcal{T}}\big(\mathcal{P}(y_i)\big),\quad \forall i=1,\ldots,K,\\ \mathbf{w}^{(0)}_{\hat{y}_i}&=f_{\mathcal{T}}\big(\mathcal{P}(y_{k+i})\big),\quad \forall i=1,\ldots,L. \end{align}\tag{12}\] For clarity, we summarize the complete algorithm in Algorithm 1. Finally, we note that the proposed framework can also be applied to the ID-label-only case like MCM; detailed discussions are deferred to Appendix 12.

4.3 Discussion↩︎

We recently found that AdaNeg [26] also performs online updates of class prototypes using unlabeled test data streams. However, our method fundamentally differs from AdaNeg in problem formulation. AdaNeg is motivated by an empirical issue, i.e., negative text labels can be semantically misaligned with the target OOD distribution, and addresses it by constructing adaptive negative prototypes from cached test-image features. While effective, this strategy remains a form of prototype engineering within the standard CLIP paradigm: it insists on treating text embeddings as valid anchors and focuses on refining negative prototypes, rather than questioning the validity of text-derived prototypes more generally.

In contrast, our work targets a structural limitation in CLIP-based post-hoc OOD detection: directly using text embeddings as class prototypes is theoretically unjustified due to an intrinsic modality gap between text-derived prototypes and the optimal visual prototypes. This failure mode is independent of negative-label mining or memory heuristics and therefore cannot be addressed by negative-prototype adaptation alone. We instead replace text-as-prototype with a principled online pseudo-supervised optimization that calibrates prototypes in the visual feature space from unlabeled test streams and comes with convergence guarantees, yielding a more general and theoretically grounded test-time prototype calibration framework. Empirically, we show in Section 5 that these theoretical advantages translate into consistent performance gains across standard benchmarks.

5 Experiments↩︎

Baselines. We compare our method with MSP [28], ODIN [13], Energy [14], Gradnorm [15], Vim [29], KNN [16], VOS [30], NPOS [31], ZOC [19], CLIPN [32], LoCoOp [33], LSN [34], LAPT [35], NegPrompt [36], Mahalanobis  [17], MCM [20], NegLabel [21], GL-MCM [37], DNM [38], InfoOOD [39], AdaNeg [26] and CSP [40].

Implementation Details. Unless otherwise specified, we employ CLIP-B/16 for zero-shot OOD detection. Following prior works [21], [26], we adopt the text prompt of ‘The nice \(<\)label\(>\).’ and use the same \(L=10000\) negative labels selected by [21]. Regarding hyper-parameters, we fix \(\tau=0.01\), \(\kappa = 0.05\), \(\rho = 0.1\) and \(\beta = 0.95\) for all experiments. The reported results of our method are averaged over 10 independent runs.

5.1 Main Results↩︎

Evaluation on ImageNet Benchmark. Following prior work [20], [21], [26], we evaluate our method on the popular ImageNet-1K benchmark [41] in Table 1, where the validation set of ImageNet-1K is designated as the ID dataset while iNaturalist [42], SUN [43], Places365 [44], and Textures [45] are considered as OOD datasets. At test time, all images are resized to \(224\times224\). The baselines range from traditional fine-tuning methods and advanced post-hoc methods from pre-trained VLMs like CLIP. Our method consistently outperforms existing baselines, which highlights its superiority in the zero-shot setting. The significant improvements demonstrate the effectiveness of our method in zero-shot scenarios. Table 2 shows that the empirical advantages of our method over the state-of-the-art still hold under the OpenOOD setup [46], where Near-OOD datasets are SSB-hard [47] and NINCO [48] while Far-OOD datasets are iNaturalist [42], Textures [45], and OpenImage-O [49].

Evaluation on CIFAR Benchmarks We further conduct experiments on CIFAR benchmarks [50] under the OpenOOD setup. To be specific, with CIFAR-10/CIFAR-100 as ID dataset, Near-OOD datasets are CIFAR-100/CIFAR-10 and Tiny-ImageNet [51] while Far-OOD datasets are MNIST [52], SVHN [53], Texture [45] and Places [44]. Experiment results in Table 3 and Table 4 show that our method achieves consistent and notable improvements on both CIFAR-10 and CIFAR-20, demonstrating the generalization of our method.

5.2 Ablation Study↩︎

Hyper-parameters. We evaluate the hyper-parameters most essential to our algorithm design, including \(\kappa\) in Eq. (5 ), \(\beta\) in Eq. (7 ), and \(\rho\) in Eq. (9 ) and Eq. (10 ). The corresponding results are displayed in Figure 2, where one can find that our method is not sensitive to hyper-parameters.

Visual Encoders. In principle, our method is generic to the choice of visual encoder. We evaluate our method with different visual encoder architectures, including ResNet-50, ViT-B/32 and ViT-L/14, and report the corresponding OOD detection results in Table 5. It can be seen that the performance can be improved by more powerful visual encoders. While our method consistently outperforms the most recent AdaNeg regardless of the backbone architecture used, indicating better generalization.

Input Resolution. In principle, our method is generic to the input resolution. We evaluate our method with a larger input size, i.e., 336\(\times\)​336, and report the corresponding OOD detection results in Table 6. On the one hand, the performance of OOD detection can be enhanced by a larger input size. On the other hand, our method consistently outperforms the most recent AdaNeg+NegLabel regardless of input resolution, which implies the better generalization of our method.

Due to space limitation, more ablation studies can be found in Appendix 14 while analysis on time and memory complexity can be found in Appendix 15

5.3 Extensions↩︎

Domain-generalizable OOD Detection. We investigate domain generalizable OOD detection scenarios, where there exist domain shifts in ID data. With ImageNet-1K as a case study, we, following [21], consider ImageNet-A [54], ImageNet-R [55] and ImageNet-S [56] as ID data respectively. The experiment results on four OOD datasets are shown in Table 7. Our method consistently achieves the state-of-the-art performance, which implies the robustness of our method to domain shift.

OOD Detection with Learned Prompt. While this paper, following [21], to use a pre-defined prompts for both negative and ID labels, we show in Table 8 that our method can be made stronger with learned prompts by LAPT [35].

6 Related Work↩︎

6.1 Traditional Out-of-distribution Detection↩︎

The popularity of OOD detection is motivated by the empirical observation [57] that neural networks tend to be over-confident in OOD data. One line of work performs OOD detection by devising post-hoc scoring functions, including confidence-based methods [20], [58], [59], energy-based methods [14], [27], distance-based approaches [16], [17], [60]–[64], gradient-based approaches [15], [65], and Bayesian approaches [66], [67]. Another line of work addresses OOD detection by fine-tuning a pre-trained discrimination model with training-time regularizations that help the model learn ID/OOD discrepancy following the guideline of outlier exposure [68]. For instance, the discriminative model is regularized to produce lower confidence [69]–[71], smaller feature magnitudes [14] or higher energy [72] for outlier points. More recently, some works have considered a practical scenario where the auxiliary outliers can be arbitrarily different from the real OOD data, therefore distributionally augmenting the observed OOD data. Besides, the given OOD samples tend to include unlabelled ID counterparts [73]. Due to this, WOOD [73] formulates learning with noisy OOD samples as a constrained optimization problem while SAL [74] separates candidate outliers from the unlabeled wild data and then trains a binary classifier using the candidate outliers and the labelled ID data.

6.2 CLIP-based Out-of-distribution Detection↩︎

The core of CLIP-based OOD detection lies in how to leverage textual supervision with pre-trained VLMs to assist OOD detection on the visual domain. On the one hand, the pioneering work, MCM [20], defines textual features as concept proto-types for each ID class and uses the scaled distance between visual features and the closest ID prototype to measure OOD uncertainty. Instead of relying on textual information from only ID label space, ZOC [19] applies VLMs to discern OOD instances by training a captioner that generates potential OOD labels. Nevertheless, this captioner often fails to produce effective OOD labels, particularly for ID datasets containing many classes. Differently, NegLabel [21] incorporates additional negative class names mined from available data sources as negative proxies. Considering the nonalignment between target visual OOD distribution and the generated negative textual OOD distribution, AdaNeg [26] leverages the benefits of test-time adaptation to generate adaptive proxies by exploring potential OOD images during testing. More recently, [39] understand CLIP-based post-hoc OOD detection from an information-theoretical perspective while [38] improve negative mining strategy in CLIP-based post-hoc OOD detection from a positive-unlabeled perspective. On the other hand, CLIP-based OOD detection can also be improved by prompt representation learning. In particular, LoCoOp [33] learns ID text prompts by pushing them away from the portions of CLIP local features that have ID-irrelevant nuisances (e.g., backgrounds). CLIPN [32] and LSN [34] design a learnable “no” prompt and a “no” text encoder to capture negation semantics within images. Differently, LAPT [35] initializes prompts with negative labels, followed by tuning prompts with cross-modal and cross-distribution mixing.

7 Conclusion↩︎

This paper theoretically shows that a key assumption behind zero-shot, post-hoc OOD detection with pre-trained vision–language models—that text embeddings of class names can serve as reliable visual class prototypes—is generally invalid due to an intrinsic modality gap between textual prototypes and optimal visual prototypes in the image feature space. Building on this theoretical insight, we propose an online pseudo-supervised framework that respects the post-hoc constraint by learning and updating prototypes directly in the visual feature space from unlabeled test-time streams and soft CLIP predictions, with accompanying convergence guarantees. Extensive experiments across standard benchmarks demonstrate that this modality-aware prototype calibration consistently improves OOD detection performance and achieves state-of-the-art results.

Impact Statement↩︎

This paper presents work aimed at advancing the field of machine learning. While our research may have a range of potential societal implications, we do not believe that any require specific discussion at this time.

8 More Related Work↩︎

Pretraining on large-scale image–text pairs has established vision–language models (VLMs) as a standard backbone for multimodal transfer. Architecturally, existing VLMs can broadly be grouped into two categories: (i) single-stream models, which feed concatenated visual and textual features into a unified transformer, such as VisualBERT [75] and ViLT [76]; and (ii) dual-stream models, which maintain separate visual and textual encoders and learn cross-modal alignment through contrastive image–text pairing, such as CLIP [18], ALIGN [77], SigLIP [78], and FILIP [79]. Among these, CLIP-based models have been widely adopted and have inspired numerous follow-up studies aimed at improving data efficiency and downstream adaptation [80]–[87]. In this paper, we adopt CLIP as the pretrained backbone; however, our method is generally applicable to contrastive vision–language models that learn aligned visual and textual representations.

9 Proof of Theorem 1↩︎

10 Proof of Theorem 2↩︎

11 Proof of Theorem 3↩︎

12 Online Prototype Learning with ID-only labels for OOD Detection↩︎

To realize the idea, we need to first identify positive samples, i.e., those with high confidence of being ID, from the test data stream. To this end, we adopt a hard thresholding rule based on the off-the-shelf \(S_{\text{MCM}}(\mathbf{x}_i;f)\) in Eq. (2 ), i.e., \[\text{Positive}: S_{\text{MCM}}(\mathbf{x}_i;f)\geq\beta,\] where \(\beta\in[0,1]\) is a hyper-parameter.

Given the \(i\)-th test sample \(\mathbf{x}_i\) (\(i=1,2,\ldots\)) to arrive, we use \(c_i^+\) to represent the cumulative number of positive samples we have ever seen so far, such that \[\label{eq46} c_i^+=c_{i-1}^++\mathbb{1}\left(S_{\text{MCM}}(\mathbf{x}_i;f)\geq\beta\right),\tag{19}\] with initialization \(c_0^+=0\) and \(\mathbb{1}(\cdot)\) as an indicator function.

For each \(y\in\mathcal{Y}_{\mathrm{I}}\), the class prototype \(\mathbf{w}_{y}\) is updated as \[\label{eq47} \mathbf{w}_{y}^{(i)}= \begin{cases} \mathbf{w}_{y}^{(i-1)} \quad \text{ if } S_{\text{MCM}}(\mathbf{x}_i;f)<\beta,\\ \ell_2\left(\mathbf{w}_{y}^{(i-1)}-\eta_i^+ \frac{\partial \hat{\ell}(\mathbf{W}^{(i-1)};\mathbf{x}_i,\mathcal{Y}_{\mathrm{I}})}{\partial \mathbf{w}_{y}^{(i-1)}}\right) \text{ otherwise}, \end{cases}\tag{20}\] where \(\eta_i^+=\rho/\sqrt{c_i^+}\).

Finally, we can arrive at the online OOD scoring function for \(\mathbf{x}_i\) as follows: \[\label{eq48} S_{\text{ours}}(\mathbf{x}_i;f)= \frac{\max_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}_i\cdot\mathbf{w}_{y}^{(i)}/\tau)}{\sum_{y\in\mathcal{Y}_{\mathrm{I}}}\exp(\mathbf{z}_i\cdot\mathbf{w}_{{y}}^{(i)}/\tau)},\tag{21}\] with initialization satisfies \(\mathbf{w}_{y}^{(0)}=f_{\mathcal{T}}\big(\mathcal{P}(y)\big)\) for each \(y\in\mathcal{Y}_{\mathrm{I}}\). For clarity, we summarize the complete algorithm in Algorithm 3. Empirical comparison can be found in Table 9.

13 Evaluation Metrics.↩︎

We evaluate the performance of OOD detection with two widely used metrics, including 1) the false positive rate of OOD data is measured when the true positive rate of ID data reaches 95% (FPR95); 2) the area under the receiver operating curve (AUROC) is computed to quantify the probability of the ID case receiving a higher score than OOD case.

14 More Ablation Studies↩︎

Temporal Shift. Since our method dynamically updates class prototypes in a test-time adaptation manner, it is crucial to investigate its stability under temporal shifts, where OOD environments evolve over time. Specifically, we use ImageNet as the ID dataset and assume that OOD datasets change sequentially over time (e.g., I–S–P–T represents OOD dataset transitions from iNaturalist → SUN → Places → Textures). In implementation, we only initialize class prototypes via Eq. (12 ) once at the very beginning rather than when processing a new OOD dataset. As shown in Table 10, our method consistently maintains a large advantage over the AdaNeg+NegLabel baseline, validating its robustness to temporal shifts.

Sample Order. In our experiments, ID and OOD samples are randomly shuffled during testing, corresponding to the “Random Shuffled” scenario. To analyze the impact of sample order, we conducted additional experiments. Specifically, we consider two extreme cases: “ID First” (all ID samples are tested before any OOD samples) and “OOD First” (all OOD samples are tested before any ID samples). As shown in Table 11, while performance does drop under these extreme settings compared to the “Random Shuffled” scenario, our method still significantly outperforms the AdaNeg+NegLabel baseline. This demonstrates the robustness and effectiveness of our method, even when the order of test samples is highly imbalanced.

15 Analysis on Memory and Time Complexity↩︎

We analyze the memory and time complexity of the proposed method and compare it with relevant baselines in Table 12 to clarify its computational overhead and scalability. Note that we omit the computation cost introduced by extracting features from pre-trained CLIP, as our method keeps the same feature extraction procedure as AdaNeg [26].

References↩︎

---

1. By definition, negative labels are those semantically irrelevant/dissimilar to all ID labels.↩︎