1 Introduction↩︎

Modern machine learning models achieve remarkable accuracy on visual tasks such as object detection, medical image analysis, and human action recognition. However, these models typically produce point predictions without quantifying uncertainty, which is a major shortcoming in safety-critical applications. In domains such as medical diagnostics, autonomous driving, or environmental monitoring, reliable uncertainty estimates help identify ambiguous inputs, flag unreliable predictions, and guide human oversight [1], [2].

Conformal prediction (CP) is a statistical framework for uncertainty quantification in machine learning. Given a predictive model and a set of calibration data, CP produces prediction sets or intervals that guarantee a user-defined level of coverage, assuming exchangeability of the data—a weaker assumption than i.i.d [3]. Due to its model-agnostic nature and theoretical guarantees, CP has been adopted in areas where reliable uncertainty estimates are crucial, such as medical imaging and human-in-the-loop systems.

Despite these strengths, a well-known limitation of CP is that it only guarantees marginal coverage (i.e., averaged over the entire data distribution), but not conditional coverage for subgroups (e.g., specific classes or feature regions) [4]. This leads to coverage disparities, where some subpopulations are under- or over-covered. These disparities often stem from the global nature of the calibration step, which assigns equal importance to all calibration examples.

Localized Conformal Prediction (LCP), introduced in [5], addresses this limitation by weighting calibration examples based on their similarity to the test sample. This yields test-specific prediction intervals that adapt to local distributional properties. While LCP retains the marginal coverage guarantees of standard CP, it can also offer improved local coverage under suitable conditions. So far, however, LCP has mainly been studied in regression settings and tested on synthetic datasets.

In this paper, we apply the LCP framework to classification tasks and benchmark it against standard CP on an extensive collection of datasets. Our approach integrates LCP into Vision-Language Models (VLMs) by leveraging their latent representations to define similarity. We show that using a non-linear transformation of the cosine similarity to weight the calibration set leads to improved set sizes and class-conditional coverage compared to traditional CP.

Our contributions are threefold:

1. LCP for Classification: We benchmark LCP for classification tasks using VLMs and evaluate it with different conformal classification scores such as LAC, APS, and RAPS. Additionally, we show that cosine similarity is a suboptimal weighting function for LCP in classification and propose a non-linear transformation to enhance its effectiveness.

2. Extensive Benchmarking: We evaluate our method on nine diverse image classification datasets, showing that LCP significantly improves average prediction set size.

3. Open-Source Implementation: We release the first, to our knowledge, open-source implementation of LCP for classification, compatible with a wide range of datasets and models.

The remainder of this article is structured as follows. Section 2 reviews CP and LCP, including classification-specific conformal scores and an overview of VLMs. Section 3 describes our experimental setup and introduces the proposed non-linear transformation of cosine similarity. In Section 4, we report and analyze the experimental results. Finally, Section 5 concludes the paper and outlines directions for future work.

2 Background↩︎

2.1 Conformal prediction↩︎

Conformal predictions methods generally follow the same basic principle. First, conformal scores \(s_i (k^{*}_i)\) are computed for \(n\) calibration samples, which differ from the training, validation, and test samples, with the knowledge of their ground-truth classes \(k_i^{*}\). Then, the \(1-\alpha\) quantile of these scores are computed as \[q_{\alpha} = \text{Quantile}\left(\{s_i (k_i^{*}) ; 1 \leq i \leq n\} ; \dfrac{n+1}{n} (1 - \alpha)\right)\] for a given error level \(\alpha\). Finally, for a test time sample \(x_{\text{test}}\), all classes \(k\) such that \(s_{\text{test}}(k) \leq q_{\alpha}\) are retained to form the conformal prediction set \(C(\mathbf{x}_{\text{test}})\). One major difference between conformal prediction methods is the way they construct their conformal scores from the prediction vectors \(y_i\), belonging to the \(K-\)simplex \(\Delta_K\), given by the model. In the following, we present four different scores which are ubiquitous in the literature.

2.1.0.1 TopK

The TopK conformal methods uses the rank of the class in the prediction vector as the conformal scores. Formally, let \(P_i\) the permutation function yielding the indexes sorting the components of \(y_{i,k}\) in decreasing order. Then, the TopK conformal score reads as \[s^{\text{TopK}} (k^*_i) = P_i(k^*_i).\]

2.1.0.2 Least Ambiguous set-valued Classifier

The Least Ambiguous set-valued Classifier (LAC) [3], [6], [7] uses conformal scores defined as follow : \[\label{eq:nonconf} s^{\text{LAC}}(k^{*}_i) = 1 - y_{i,k_i^*}.\tag{1}\] Notice how this procedure may return empty sets for prediction vectors close to decision boundaries, a behavior considered undesirable in the literature.

2.1.0.3 Adaptive Prediction Sets

Subsequently, the Adaptive Prediction Sets (APS) method was introduced in [8]. It uses a different conformal score, which is the cumulated sum of the components of the soft label vector \(y_i\), sorted in decreasing order, up to the component corresponding to the ground-truth label. Using the same notations as for TopK, let \(q^{*}\) the rank of the ground truth labels in the sorted prediction vector, i.e. the integer such that \(P_i^{-1}(q^{*}) = k^{*}\). The APS score then reads \[s^{\text{APS}} (k^{*}_i) = \sum_{q = 1}^{q^{*}} y_{i, P_i^{-1}(q)}.\] This procedure improves on the problem of non-empty conformal sets, but comes at the expense of prediction sets much larger than those constructed by the LAC method.

2.1.0.4 Regularized Adaptive Prediction Sets

As a result, regularized APS (RAPS) was introduced in [4]. It introduces two additional parameters, \(k_{\text{reg}} \in \mathbb{N}\) and \(\lambda_{\text{reg}} \in \mathbb{R}\), which penalize excessive set sizes. In detail, the APS score is modified as \[s_i^{\text{RAPS}}(k^{*}) = s_i^{\text{APS}} + \lambda_{\text{reg}}\sum_{q=1}^{q^{*}} \mathbb{1}_{q > k_{\text{reg}}}.\] The parameters \(k_{\text{reg}}\) and \(\lambda_{\text{reg}}\) are estimated from the data by splitting the calibration set into two additional subsets. One is dedicated to the computation of these parameters, the other to the computation of the quantile. We follow the same setup as [4] in our implementation.

2.2 Zero-Shot Prediction with Vision-Language Models↩︎

Generally, a VLM projects both images and textual descriptions into a common latent space, enabling measurement of their similarities. The textual descriptions of target classes provided by the user, so called textual prompts, are transformed into numerical tokens \({\mathbf{c}}_k\). The latter are then mapped by the textual encoder to normalized embeddings \({\mathbf{t}}_k\) on the unit-hypersphere of \(\mathbb{R}^d\), where \(d\) is the latent space dimension. Similarly, the image \(\mathbf{x}_i\) is processed by the visual encoder to produce embeddings \(\boldsymbol{f}_i\) on the same unit-hypersphere. Prediction vectors are then obtained with a softmax transformation: \[\label{zero-shot-prediction} y_{i,k}= \frac{\exp ({\mathbf{f}}_i^T {\mathbf{t}}_k/T)}{\sum_{j}^K \exp ({\mathbf{f}}_i^T {\mathbf{t}}_k/T)}\tag{2}\] where \(T\) is a parameter of the model, typically set to 0.01. The Zero-Shot prediction is then \(\hat{k} = \mathrm{argmax}_{k}~y_{i,k}\). In this paper, we use models trained within the CLIP framework [9]. We investigate four different models, characterized by their visual encoder, either be ViT [10] based (ViT-B/16, ViT-L/14) or CNN [11] based (RN50, RN101). We use the prompts "a photo of a {class}." for all experiments. Notice how \(\boldsymbol{f}_i^{T} \boldsymbol{f}_j\) is an intuitive choice to measure the similarity between images \(\boldsymbol{x}_i\) and \(\boldsymbol{x}_j\). The latter has been used successfully in the few-shot adaptation literature, for instance by Tip-Adapter [12].

2.3 Localized conformal prediction↩︎

We implement the algorithm presented in [5]. For a given test sample \(\mathbf{x}_{\text{test}}\) and its corresponding projection in a latent space \(\boldsymbol{f}_{\text{test}}\), LCP assigns greater importance to calibration examples that are more similar to it, using a localizer function \(H(\boldsymbol{f}_{\text{test}}, \boldsymbol{f}_i) \in [0, 1]\). This yields a test-specific, weighted empirical distribution over conformal scores, resulting in prediction sets that better adapt to local structure in the feature space.

In particular, given a conformal score \(s_i\), \(i =1,\dots, n\) for each calibration example, the weighted empirical distribution of scores is given by:

\[\hat{\mathcal{F}} (\boldsymbol{f}_{\text{test}}) = \sum_{i=1}^n \frac{H(\boldsymbol{f}_{\text{test}}, \boldsymbol{f}_i)}{\sum_{j=1}^n H(\boldsymbol{f}_{\text{test}}, \boldsymbol{f}_j)} \, \delta_{s_i},\]

where \(\delta_{s_i}\) is a point mass at the score \(s_i\).

For classification with label space \(\mathcal{Y} = \{1, \dots, K\}\), a class \(k\) is included in the prediction set for the test point if its test-time score \(s_{\text{test}}(k)\) does not exceed a quantile threshold:

\[C(x_{\text{test}}) = \left\{ k \in \mathcal{Y} : s_{\text{test}}(k) \leq q_{\tilde{\alpha}} \right\},\]

where \(q_{\tilde{\alpha}}\) is the \(1-\tilde{\alpha}\) quantile of the weighted distribution, and \(\tilde{\alpha}\) is chosen to ensure a marginal coverage of \(1-\alpha\). We refer to [5] for additional details regarding the algorithm and technical proofs about the coverage guarantee of LCP.

3 Methods↩︎

3.1 Experimental setting↩︎

3.1.0.1 Datasets

We follow the settings of previous works [13] and use 9 diverse images classification datasets: SUN397 [14] for classification of scenes, Aircraft [15] for aircrafts, EuroSAT [16] for satellite images, StanfordCars [17] for cars models, Food101 [18] for food items, Pets [19] for pet breeds, Flower102 [20] for flowers species, DTD [21] for textures and UCF101 [22] for actions recognition. We exclude Caltech101 [23] from the benchmark, as most backbones achieves an accuracy higher than the target coverage \(1 - \alpha\) for \(\alpha = 0.1\). Details about the number of samples and labels distribution is provided in Supplementary Section 6.

3.1.0.2 Calibration sets generation

We uniformly sample 1000 images from the dataset for calibration, and use the remaining data to measure the metrics exposed in the next Section. Therefore, the calibration have the same label distribution as the dataset (see Supplementary Section 6). We use ten random seeds for each dataset, and use the same calibration / test splits for all methods.

3.1.0.3 Metrics

The CP literature usually wants to balance two objectives: (1) predictive efficiency (i.e., producing small prediction sets); and (2) conditional validity (i.e., maintaining the coverage guarantee across data subgroups).

Efficiency is measured by the mean set size over the set of test samples \(\mathcal{S}_{\text{test}}\): \[\text{Mean Set Size}(\mathcal{S}_{\text{test}}) = \frac{1}{|\mathcal{S}_{\text{test}}|} \sum_{\mathbf{x} \in \mathcal{S}_{\text{test}}} |C(\mathbf{x})|. \label{eq:mean95set95size}\tag{3}\]

To assess conditional validity, we partition the test set by true class labels: \(\mathcal{P} = \{\mathcal{S}_{\text{group}}\}\). For each group, the coverage Cov is: \[\text{Cov}(\mathcal{S}_{\text{group}}) = \frac{1}{|\mathcal{S}_{\text{group}}|} \sum_{(\mathbf{x}, y) \in \mathcal{S}_{\text{group}}} \mathbb{1}_{y \in C(\mathbf{x})}.\]

We follow previous works [24] and use two derived metrics:

• CovGap: average deviation from the target coverage: \[\text{CovGap}(\mathcal{S}_{\text{test}}) = \frac{1}{|\mathcal{P}|} \sum_{\mathcal{S}_{\text{group}} \in \mathcal{P}} \left| \text{Cov}(\mathcal{S}_{\text{group}}) - (1 - \alpha) \right|. \label{eq:covgap}\tag{4}\]

• MCCC: worst-case coverage across groups: \[\text{MCCC}(\mathcal{S}_{\text{test}}) = \min_{\mathcal{S}_{\text{group}} \in \mathcal{P}} \text{Cov}(\mathcal{S}_{\text{group}}). \label{eq:MCCC}\tag{5}\]

3.2 Transformation of Cosine Similarity↩︎

To define the localizer \(H\) in the LCP framework, we apply a normalized sigmoid transformation to the cosine similarity between samples: \[H(\boldsymbol{f}_i, \boldsymbol{f}_j) = \frac{1 + \exp(-m(1 - \tau))}{1 + \exp(-m(\boldsymbol{f}_i^T \boldsymbol{f}_j - \tau))}, \label{eq:sigmoid}\tag{6}\] where \(m > 0\) controls the sigmoid’s steepness, and \(\tau \in [0, 1]\) sets its inflection point. We tune \(m\) and \(\tau\) via 5-fold cross-validation on the calibration set, so as to minimize the average set size. Finally, we take the median of the selected \(m\) values and the mean of the selected \(\tau\) values across folds. In our experiments, we search over \(\tau \in \{0.8, 0.9, 1.0\}\) and \(m \in \{1, 2, \ldots, 30\}\).

4 Results↩︎

4.1 Mean set sizes↩︎

We present results concerning the mean set sizes in Table ¿tbl:tab:results95mean?. To assess the significance of the differences in mean set sizes between local methods and non-local baselines, we perform a paired t-test [25] on the per-fold difference when it succeeds a Shapiro gaussianity test [26], and a Wilcoxon test [27] otherwise. We see that our approach achieves a decrease in mean set sizes for 9 out of 9 datasets for 3 out of 4 backbones compared to the non-local TopK, with statistical significance levels of least 0.01. Conversely, the naive approach fails to decrease the mean set sizes significantly compared to the non-local TopK, and even increases their sizes with statistical significance on several datasets and backbones (e.g. DTD with ViT-B/16, or UCF101 with RN50). This trend is confirmed with other non-local baselines, although not as sharply as with TopK. We see that the naive approach degrades mean set sizes with statistical significance for several other conformal scores, such as APS for Aircraft with ViT-B/16. However, our approach achieves a statistically significant decrease of the mean set size in the latter setting, from 18.11 for the non-local baseline down to 17.43.

= 0.1mm = 0.1mm

Overall, these results prove the effectiveness of our approach and the necessity of transforming the cosine similarities when using LCP. As an illustration, Figure 1 shows examples of sigmoidal transformations obtained with our cross validation strategy exposed in Section 3.2.

4.2 Coverage metrics↩︎

We present results concerning coverage metrics in Table ¿tbl:tab:results95cvg?, namely CovGap (see Eq. 4 ) and MCCC (see Eq. 5 ). Compared to the mean set sizes, it is more difficult to identify trends that hold across datasets and backbones. As a general observation, the local algorithms seem to more closely follow the behavior of the non-local baselines. However, there are still several instances where our approach improves MCCC, e.g., for the LAC conformal score on Pets with ViT-B/16. However, on the same dataset with the same backbone, the MCCC is degraded compared to the non-local baseline when using the TopK conformal score. For the CovGap metric, the results obtained with the local algorithms are almost identical to the non-local baselines across all datasets and backbones. This result suggests that localization of conformal prediction is not sufficient to minimize the CovGap for image classification using VLMs.

5 Conclusion↩︎

In this paper, we presented an extensive benchmarking of localized conformal prediction on a variety of image classification tasks with several VLMs and conformal scores. We showed how the naive approach, using a cosine similarity in the visual latent space (an intuitive choice for VLMs), fails to improve over the non-local baselines, even degrading the mean set size in several cases. We thus introduced a sigmoidal transformation of the cosine similarities, using cross-validation on the calibration samples to tune its hyperparameters. This approach allowed to consistently improve over the non-local baselines across datasets and backbones, often achieving a statistically significant reduction of set sizes. Conversely, we showed that improvements in classical coverage-related metrics were less impacted by the local algorithms, with the coverage gap very closely following the behavior of the non-local baselines. We hope these findings as well as our open source code will contribute to the deployment of localized conformal prediction algorithms in real-world applications.

6 Dataset details↩︎

References↩︎