1 Introduction↩︎

Vision-language models (VLMs) excel at visual question answering (VQA), a setting in which uncertainty can arise from ambiguity in the visual evidence. In this work, we study uncertainty estimation for VLMs in VQA. Reliable uncertainty estimates are essential for detecting unreliable predictions and enabling trustworthy deployment of VLMs. Current approaches to uncertainty estimation (UE) in VLMs include verbalized methods that prompt models to report confidence [1]–[4], logit-based methods derived from output probabilities or token entropy [5], [6], and consistency-based methods that measure agreement across sampled generations [7]–[9]. Another widely used class is entropy-based methods, which estimate uncertainty from the semantic distribution of sampled answers, such as Semantic Entropy (SE) and its variants [10]–[13].

In this paper, we set out to analyze existing UE methods and identify 3 limitations that motivate our approach: (1) Overconfident visual representations can produce highly consistent outputs under sampling, causing entropy-based methods such as SE [10] to underestimate uncertainty. (2) Wording variations among semantically equivalent answers can inflate pairwise semantic distances, leading methods such as SNNE [11] to overestimate uncertainty. (3) Textual perturbations can dominate output variability, causing methods that rely on text paraphrasing or joint text-image perturbations (e.g., VL-Uncertainty [13] and C&U [8]) to reflect prompt sensitivity rather than visual ambiguity.

First, Semantic Entropy [10] estimates uncertainty by sampling multiple outputs under stochastic decoding and computing the entropy of the answer distribution, where diverse answers correspond to high entropy and consistent answers to low entropy. While effective, this approach assumes that epistemic uncertainty manifests through decoding randomness. In VLMs, however, uncertainty can originate from the visual input which SE does not effectively capture (Fig. 1). We show that visually biased images can induce visual embeddings that lead to highly confident predictions (Fig. 3). Under such conditions, repeated stochastic decoding produces little output variation, rendering SE ineffective (Sec. 3.2).

Second, given the resulting answer variants, SE treats answers as categorical labels and estimates uncertainty by counting their frequencies, ignoring semantic distances between categories. Semantic Nearest Neighbor Entropy (SNNE) [11] addresses this by incorporating semantic distances through pairwise aggregation. However, SNNE operates directly on text outputs. Because language is discrete, answers that share the same meaning but differ in wording may appear separated in embedding space. As a result, distance-based aggregation can mistake wording variation for semantic disagreement and inflate uncertainty (Sec. 3.4).

Third, recent work [8], [13] explores input-space perturbations through textual paraphrasing or joint text-image perturbations. As illustrated in Fig. 2, due to prompt sensitivity, text paraphrases can induce non-local, larger semantic shifts than visual perturbations, which typically produce only small, local changes. While such perturbations increase output diversity, the resulting variability is often dominated by textual changes rather than visual evidence, with clustering is largely text-driven (Fig. 5, Sec. 3.3).

To address these limitations, we propose Visual Semantic Entropy (VSE), which perturbs the image to induce alternative answers and measures uncertainty through dispersion among semantic prototypes of the generated responses. First, VSE only perturbs the input image, leaving the text query unchanged, to probe prediction instability that sampling alone fails to capture. Second, VSE clusters semantically similar answers into prototypes so that wording variations do not inflate uncertainty. Uncertainty is measured as the frequency-weighted pairwise distance among prototypes. Finally, by perturbing only the image, VSE avoids modality confounding from textual perturbations and yields uncertainty that better reflects visual ambiguity. Extensive experiments show that VSE is particularly effective on visually adversarial datasets, outperforming methods based on text perturbation and joint image-text perturbations. In summary, our contributions are:

1. Our analysis reveals three failure modes of current VLMs uncertainty estimators: (1) suppressed output variation from overconfident visual embeddings; (2) inflated distances from wording variation; and (3) textual perturbations dominate output variability, causing uncertainty to reflect prompt sensitivity rather than visual ambiguity.

2. We introduce Visual Semantic Entropy (VSE), which perturbs the image, clusters semantically similar answers, and quantifies uncertainty using weighted pairwise distances between prototype embeddings.

3. We present an extensive benchmark across 5 VLMs and 5 datasets, evaluating multiple uncertainty estimation methods and showing that VSE consistently yields more reliable uncertainty estimates.

2 Related Work↩︎

Uncertainty Estimation in VLMs aims to quantify the reliability of model predictions, which is particularly important for visual question answering (VQA), where ambiguous visual evidence can lead to unreliable answers.

Verbalized methods prompt models to report confidence [1]–[4], while logit-based methods derive uncertainty from output probabilities [5], [6]. Both do not explicitly account for visual ambiguity.

Consistency-based approaches, including SelfCheckGPT [7] and C&U [8], estimate uncertainty by measuring agreement between sampled responses and the original answer. C&U additionally generates question paraphrases before probing agreement. However, when visual embeddings are confident, stochastic decoding can yield highly consistent outputs, causing these methods to underestimate uncertainty.

Entropy-based approaches estimate uncertainty from the semantic distribution of sampled answers. Semantic Entropy (SE) [10] computes entropy over outputs sampled under stochastic decoding. Later methods improve semantic representation, including SNNE [11] and KLE [12], which aggregates pairwise distances across sampled answers. However, similar to consistency-based methods, these approaches rely on diversity in generated text under stochastic decoding and may fail to capture visual uncertainty when visual embeddings are confident, as shown in our analysis (Sec. 3.2). Moreover, pairwise distance aggregation can inflate uncertainty due to wording variations across answers(Sec. 3.4).

Input Perturbation. Recent work probes uncertainty through input perturbations. C&U [8] perturbs the question, while VL-Uncertainty [13] perturbs both the image and the text. Our analysis shows that textual perturbations often dominate output variability due to prompt sensitivity, causing uncertainty estimates to reflect linguistic variation rather than visual ambiguity (Sec. 3.3).

Motivated by these limitations, we focus on uncertainty arising from visual ambiguity. Our design probes uncertainty through visual perturbations while avoiding text perturbations that introduce prompt sensitivity, and aggregates predictions at the semantic level to mitigate wording variations.

3 Problem Analysis↩︎

We analyze limitations of existing uncertainty estimators in VLMs and formalize two hypotheses and one proposition that guide our method.

3.1 Problem Formulation↩︎

A visual question instance consists of a question \(q\) and an image \(v\). Given \((q,v)\), a VLM produces an answer \(a = \mathrm{VLM}(q,v)\). Our goal is to estimate the reliability of this prediction. Specifically, we seek an uncertainty estimator \(U\) such that \(\tilde{u} = U(q,v ; \mathrm{VLM})\), where larger uncertainty score \(\tilde{u}\) indicates a higher likelihood that the generated answer \(a\) is incorrect. This is of high interests in scenarios where unreliable predictions must be filtered.

3.2 Confident visual embeddings induce low semantic entropy↩︎

Semantic Entropy estimates uncertainty by sampling multiple outputs under stochastic decoding and computing the entropy of the resulting categorical answer distribution. This approach assumes that epistemic uncertainty manifests as variability induced by decoding randomness. In VLMs, however, predictions are also conditioned on visual embeddings. When these embeddings are highly confident, stochastic decoding produces nearly identical outputs, leading to low SE even if the underlying image is ambiguous.

3.2.0.1 Hypothesis 1. Confident visual embeddings induce low semantic entropy.

If the visual embedding yields a sharply peaked conditional answer distribution, decoding randomness alone cannot generate diverse outputs. As a result, SE may underestimate visual uncertainty.

To test this hypothesis, we utilize the Adversarial split of the VILP dataset [16], which consists of visually ambiguous images. We then filter samples that the VLM answers incorrectly and quantify their visual confidence as follows: Let \(\mathbf{v}_i\in\mathbb{R}^D\) denote the embedding of the \(i\)-th visual token at the final layer and \(W\in\mathbb{R}^{\mathcal{V}\times D}\) the language modeling head with vocabulary size of \(\mathcal{V}\) of the VLM. Each visual token is projected into the vocabulary space using LogitLens  [17]: \[\mathbf{z}_i = W \mathbf{v}_i, \qquad \tilde{\mathbf{z}}_i = \mathrm{softmax}(\mathbf{z}_i).\] We compute the entropy of each visual token distribution and average across visual tokens to obtain the visual entropy \(H_{\text{vis}}\): \[H_{\text{vis}}(x) = \underset{i}{\mathrm{avg}} \left( - \sum_{k=1}^{|\mathcal{V}|} \tilde{\mathbf{z}}_i(k)\log \tilde{\mathbf{z}}_i(k) \right).\] Lower visual entropy \(H_{\text{vis}}\) indicates a more confident visual representation. Based on this metric, we partition samples into two groups: Visually confident (low visual entropy) and Visually uncertain (high visual entropy). Importantly, this metric only reflects the model certainty in the visual input, not the predicted answer. We expect high SE (uncertainty) in both groups, since the model answers these samples incorrectly. However, as shown in Fig. 3-Left, standard SE yields high uncertainty only for visually uncertain samples. Visually confident samples exhibit low SE despite the visual ambiguity, indicating that confident visual embeddings suppress output variability, yielding low semantic entropy.

We then perturb the input image to generate alternative local views while preserving semantic content. As shown in Fig. 3-Right, perturbation induces higher semantic entropy for visually confident samples. We note that the grouping is determined by calculating visual entropy \(H_\text{vis}\) on the original image. More details on experimental settings are included in the Supplementary.

Discussion. These results demonstrate that decoding stochasticity alone fails under confident visual embeddings, while image perturbation induces high semantic entropy for visually ambiguous inputs. This motivates probing uncertainty in the visual input space.

3.3 Text perturbation dominates output semantic shifts↩︎

Beyond perturbing visual inputs, recent work also perturbs the text query [8], [13]. Unlike visual perturbations, which introduce small local changes around the same image representation, textual paraphrases can induce larger semantic shifts. Semantically equivalent sentences may occupy distant regions in embedding space due to the discrete nature of language. As a result, paraphrasing can substantially alter output semantics. This motivates the following hypothesis.

3.3.0.1 Hypothesis 2.Text perturbations dominate output diversity under joint perturbation.

We hypothesize that output diversity is driven primarily by textual variation rather than visual ambiguity. Then uncertainty measured under joint perturbation would mostly reflect prompt sensitivity.

To test this, we analyze several VLMs on the Easy and Adversarial split of the VILP dataset. For each image, we construct an \(L \times M\) perturbation grid, where \(L\) denotes the number of textual paraphrases and \(M\) denotes the number of image perturbations. For each \((l, m)\) pair, we generate a model output and cluster the resulting \(L \times M\) responses based on semantic similarity. This allows us to analyze how cluster membership depends on paraphrase identity or image perturbation.

Purity analysis. To quantify the alignment between clusters and perturbation types, we compute cluster purity. For each cluster \(c\), we define text purity \(P_T(c)\) and image purity \(P_I(c)\) as the fraction of samples in \(c\) originating from the most frequent textual paraphrase and image perturbation. Let \(n_c\) denote the size of cluster \(c\), \(n_c(l)\) the number of samples in \(c\) generated from paraphrase \(l\), and \(n_c(m)\) the number generated from image perturbation \(m\). We define \[P_T(c) = \frac{\max_{l} n_c(l)}{n_c}, \quad P_I(c) = \frac{\max_{m} n_c(m)}{n_c}.\]

If \(P_T(c) > P_I(c)\), most samples in cluster \(c\) originate from a single paraphrase \(l_{\max}\), indicating that cluster membership is primarily determined by the textual input rather than visual perturbations, and vice versa. To obtain an overall text and image purity for each image across clusters, we compute the size-weighted purity: \(P_T = \sum_{c} \frac{n_c}{L \times M} P_T(c)\) and \(P_I = \sum_{c} \frac{n_c}{L \times M} P_I(c)\).

Overall, text purity consistently exceeds image purity (Tab. 1), and the gap widens on the Easy split. Furthermore, we estimate the joint density of \((P_T, P_I)\) across images using kernel smoothing and visualize it as a filled contour plot (Fig. 4). The diagonal \(P_T = P_I\) separates text-dominant (above) from image-dominant regions (below). We observe that the density mass concentrates predominantly above this line, meaning that for most images, cluster assignments align more strongly with paraphrase identity than with visual perturbations. This pattern shows that variability under joint perturbation is largely text-driven rather than visual ambiguity. Additional visualizations and details are in Supp.

To qualitatively examine this effect, we visualize cluster occupancy on the \(L \times M\) perturbation grid. Rows correspond to textual paraphrases and columns to image perturbations. Under joint perturbation, we observe horizontal stripe patterns in the occupancy maps (Fig. 5), indicating that cluster membership is invariant across image perturbations but consistent within a given paraphrase. This pattern confirms that output are primarily conditioned on the textual input.

Discussion. The quantitative purity analysis and qualitative examples support Hypothesis 2: under joint image-text perturbation, output diversity is largely driven by textual variation rather than visual ambiguity. Our finding aligns with Image-DPO [16], where the authors perturb only the image while keeping the question fixed to dismiss model over-reliance on language. In uncertainty estimation, prompt sensitivity leads paraphrasing to induce semantic shifts that inflate uncertainty without reflecting ambiguity in visual information. Hence, restricting perturbations to the visual domain avoids artificial prompt sensitivity and yields more faithful visual uncertainty.

3.4 Linguistic variations inflates distance-based uncertainty↩︎

Other approaches estimate uncertainty by measuring semantic distances among sampled outputs [11], [12]. However, because answers are discrete text, semantically equivalent responses with different wording can be mapped to distant points in embedding space. As a result, distance-based aggregation may yield inflated uncertainty even when there is no true semantic disagreement.

Formally, given \(n\) sampling iterations, the model generates answer variants \(\{a_i\}^n\). A general distance-based estimator computes uncertainty as the average pairwise distance: \[U(q)=\frac{1}{n(n-1)}\sum \mathrm{d}(a,a'), \label{eq:pairwise}\tag{1}\] where \(\mathrm{d}(\cdot,\cdot)\) is a semantic distance function.

3.4.0.1 Proposition 1. Distance inflation under linguistic variations.

Suppose all sampled answers express the same semantic meaning. If wording differences induce non-zero embedding distances, \(\mathrm{d}(a,a') > 0\) for some \(a \neq a'\), then \(U(q) > 0\) despite the absence of semantic disagreement.

3.4.0.2 Intra- vs. Inter-cluster effect.

To separate meaning-level disagreement from wording variation, consider partitioning the answers into semantic clusters, where each cluster groups answers that convey the same meaning. Let \(c\) denote the cluster assignment of \(a\). For a distance metric \(\mathrm{d}(\cdot,\cdot)\), pairwise aggregation can be decomposed as:

\[\begin{align} U(q) = \sum_{k=1}^K \,\mathbb{E}\!\left[\mathrm{d}(a,a') \mid c=c'=k\right] + \sum_{k\neq k'} \,\mathbb{E}\!\left[\mathrm{d}(a,a') \mid c=k, c'=k'\right]. \label{eq:decomposition} \end{align}\tag{2}\] The first term captures variation within clusters, which mainly reflects wording differences. The second term captures variation across clusters, corresponding to genuine semantic disagreement. Because distance-based aggregation sums both terms, uncertainty is influenced not only by disagreement between meanings but also by wording variation within each meaning. Therefore, uncertainty estimation should first group semantically equivalent answers and then measure distances across groups rather than between individual text outputs.

4 Method↩︎

The analysis above shows three limitations of current uncertainty estimators: (1) decoding stochasticity underestimates uncertainty when visual embeddings are overly confident, (2) text perturbations can dominate output variability under joint image-text perturbation, and (3) linguistic variations inflate distance-based estimators. These observations suggest that VQA uncertainty estimation should probe ambiguity in the visual input space, avoid textual perturbations that introduce large semantic shifts, and aggregate outputs at the semantic level.

Guided by these principles, we propose Visual Semantic Entropy (VSE), a method that perturbs the image to elicit alternative visual interpretations (Sec. 4.1), clusters semantically equivalent answers into prototypes, and quantifies uncertainty as the weighted dispersion among prototype embeddings (Sec. 4.2). An overview of VSE is shown in Fig. 6.

4.1 Visual Perturbation↩︎

To probe visual epistemic uncertainty, we perturb the input image while keeping the question fixed. Unlike textual paraphrasing, which may induce non-local semantic shifts, visual perturbations introduce controlled variations around the original image, enabling us to explore local visual variants.

Let \(\mathcal{T}\) denote a visual perturbation operator. Given an input image \(v\), we generate \(M\) perturbed views: \[\begin{align} v_m = \mathcal{T}(v; \xi_m), \quad m = 1,\dots,M, \end{align}\] where \(\xi_m\) parameterizes the perturbation. The operator \(\mathcal{T}\) is designed to preserve high-level semantic content while inducing local variations in pixel space. Each perturbed image \(v_m\) is paired with the same question \(q\) and fed into the VLM to obtain answer samples. Variability across these perturbed views \(v_m\) serves as a probe of visual uncertainty.

Discussion. Prior work [13] progressively increases perturbation magnitude to generate samples at multiple noise levels. In contrast, we fix a small perturbation scale for two reasons. First, uncertainty should quantify ambiguity conditioned on the original input \((q,v)\). Progressively increasing perturbations magnitude may move samples outside the local neighborhood of \(v\), so answer variation reflects changes to the input rather than visual ambiguity. Second, sampling-based estimators aggregate samples uniformly. Mixing perturbation magnitudes inappropriately assigns equal weight to local and non-local, large deviations. By using a single fixed scale, the measured dispersion consistently reflects variability within a local neighborhood.

4.2 Prototype Semantic Aggregation↩︎

Given \(M\) perturbed image variants \(\{v_m\}^M\) and the question \(q\), the VLM produces a corresponding set of answer samples \(\{a_m\}^M\). These samples may differ in wording or semantic meaning. We then perform Prototype Semantic Aggregation (ProtoSem) as follows:

Clustering. We cluster the answer variants \(\{a_m\}\) into \(K\) groups \(\{c_k\}_{k=1}^K\) based on semantic distance base on a function \(\mathrm{d}(\cdot,\cdot)\).

Prototype selection. For each cluster \(c_k\), we select a prototype answer \(p_k\) that minimizes the average distance to other members in the same cluster: \[p_k = \arg\min_{a \in c_k} \sum_{a' \in c_k} \mathrm{d}(a, a').\] This prototype serves as the representative semantic meaning of cluster \(k\).

Prototype dispersion. Let \(w_k=\frac{|c_k|}{M}\) denote the empirical probability of cluster \(k\). We define uncertainty as the expected semantic distance between two independently drawn cluster prototypes: \[\tilde{u} = U(q,v; \text{VLM}) = \mathbb{E}_{k \sim w,\; k' \sim w}\!\left[\mathrm{d}(p_k,p_{k'})\right] = \sum_{k \neq k'} w_k w_{k'} \, \mathrm{d}(p_k,p_{k'}). \label{eq:vpd}\tag{3}\] This measures disagreement among supported semantic meanings: semantic variations are consolidated within each cluster and represented by a single prototype \(p_k\), and each prototype is weighted by its mass \(w_k\). Uncertainty increases only when multiple high-mass prototypes are separated.

Discussion. SE [10] treats answers as discrete categories and ignores semantic proximity, whereas SNNE [11] measures pairwise distances and is sensitive to linguistic variation. VSE instead groups semantically equivalent outputs and compute weighted dispersion over cluster prototypes, eliminating intra-cluster wording inflation while preserving semantic disagreement across clusters.

5 Experiments↩︎

5.1 Setup↩︎

Metrics. Following prior work [10], [11], we evaluate uncertainty estimation using the Area Under the ROC Curve (AUC). Each prediction is labeled as correct or incorrect, and the predicted uncertainty score is used to distinguish between them. AUC measures how well uncertainty scores rank incorrect predictions above correct ones across different thresholds. A higher AUC indicates better alignment between predicted uncertainty and model reliability.

Datasets. We evaluate our method on five benchmarks covering knowledge-based VQA, multimodal reasoning: OKVQA [18], AOKVQA [14],MMVet [19]; and adversarial datasets to expose visual bias: VILP [16], VLM-are-biased [20]. We detail on the dataset and their splits in the Supplementary material.

Other Baselines. We conduct an extensive benchmark to compare VSE against these uncertainty estimation approaches: Verbalized: Verbalized Uncertainty [1] . Logit-based: Confidence measures derived from output probabilities, including AvgEnt and MaxEnt (token entropy), and AvgProb and MaxProb (token probability) [5]. Consistency-based: SelfCheckGPT [7] and C&U [8]. Entropy-based: Semantic Entropy (SE) [10], Semantic Nearest Neighbor Entropy (SNNE) [11], Kernel Language Entropy (KLE) [12], and VL-Uncertainty [13]. Our method, VSE, also belongs to this category.

Models. We evaluate all approaches on five vision-language models that cover a range of architecture: Qwen2.5-VL-7B [21], Gemma3-4B [22], Intern3.5-VL-8B [23], LLaVA-NeXt-8B [24], and Qwen3-VL-8B [25].

Implementation Details. To perturb the image, we add Gaussian noise with standard deviation \(\sigma=20\), introducing small visual variations while preserving semantic content. The initial answer \(a\) is generated using greedy decoding (\(T=0.0\)). After obtaining this initial answer, we sample \(M=10\) additional responses with decoding temperature \(T=1.0\), following the setup of SE [10]. For answer clustering, we use the DeBERTa-v2-xlarge-mnli [26] as the semantic distance function \(\mathrm{d}(\cdot,\cdot)\) and apply hierarchical clustering [27]. Additional hyperparameter analysis can be found in supplementary material.

5.2 Results↩︎

Qualitative Results. We report results on AOKVQA in Tab. 2. Entropy-based methods perform best overall, while logit-based approaches perform poorly across all models. Verbalized uncertainty performs competitively only on Qwen3-VL. Consistency-based methods also show strong performance, particularly C&U with text perturbations. Among entropy-based methods, VSE consistently achieves the best results across all models, reaching AUC scores of 0.783, 0.778, 0.724, 0.798, and 0.792 on Qwen2.5-VL, Gemma3, LLaVA-NeXT, Qwen3-VL, and Intern3.5-VL, and outperforming VL-Uncertainty by up to 9.2% AUC. The same trend holds on MMVet and OKVQA (Tabs. 3 and 4). VSE achieves the best performance across all models, improving over the strongest baselines such as SE (e.g., 0.767 vs. on Qwen2.5-VL in OKVQA) and reaching AUC scores of 0.781 and 0.778 on MMVet for Qwen2.5-VL and Gemma3, respectively. We observe that SE and SNNE perform well on MMVet, suggesting that repeated sampling better captures uncertainty in this benchmark, which primarily evaluates reasoning ability rather than visual ambiguity.

Qualitative Results. We provide qualitative result in Fig. 7. More results are provided in Supplementary.

5.3 Ablation Study↩︎

VSE reflects visual ambiguity more faithfully. We evaluate uncertainty estimation on the visually adversarial datasets VILP and VLM-are-biased (Tab. 5), where images are intentionally designed to be highly ambiguous to probe model biases and test reliance on visual evidence. VSE consistently achieves the best performance across all settings, improving over the strongest baseline by large margins (e.g., \(0.650\) vs.\(0.535\) on VILP with Qwen2.5-VL and \(0.826\) vs.\(0.783\) on VLM-are-biased). In contrast, several existing methods degrade substantially on these datasets, suggesting that they are sensitive to language biases or decoding artifacts. These results highlight that explicitly probing visual variations help to capture uncertainty arising from ambiguous visual inputs.

Effect of Visual Perturbation and Prototype Aggregation. Tab. 6 evaluates two components of our approach. For SE and SNNE, we report the original estimator (Base) and a variant where we add visual perturbation (\(+\mathcal{T}\)). We also evaluate ProtoSem, our prototype semantic aggregation strategy, both with and without visual perturbation.

(1) Comparing Base and \(+\mathcal{T}\) shows consistent improvements for SE and SNNE across all models, with gains up to \(+9.5\%\) AUC, demonstrating that visual perturbations are beneficial for uncertainty estimation.

(2) Comparing methods across \(+\mathcal{T}\) columns (with visual perturbation), ProtoSem consistently achieves the best performance, demonstrating that prototype-based semantic aggregation better captures uncertainty than entropy computed over sampled answers, as used in SE and SNNE.

These results justify the core design of VSE, combining visual perturbation with prototype-based semantic aggregation to improve uncertainty estimation.

6 Conclusion↩︎

We study uncertainty estimation in VLMs for VQA, where ambiguity often arises from the visual input. Our analysis shows that decoding-based estimators can underestimate uncertainty when confident visual embeddings suppress output variation, while text perturbations and wording variations can inflate uncertainty estimates. Motivated by these observations, we propose Visual Semantic Entropy, which probes uncertainty through visual perturbations and aggregates predictions at the semantic prototype level. Experiments across multiple VLMs and VQA benchmarks demonstrate that VSE provides more reliable uncertainty estimates than existing approaches. A limitation of our method is the additional computational cost from perturbation-based sampling, which is shared with other sampling-based uncertainty estimators, as well as sensitivity to the decoding temperature used during generation. Our approach also relies on semantic similarity models to cluster answers, and its performance may depend on the quality of the sematic distance function. While we evaluate VSE across several representative VLMs and benchmarks, extending the analysis to larger model families and additional multimodal tasks remains an important direction for future work. We hope this work encourages future research on uncertainty estimation that explicitly accounts for visual ambiguity.

7 Implementation details↩︎

7.1 Algorithm↩︎

We summarize VSE in Alg. 8.

7.2 Datasets↩︎

The datasets used in this work are:

AOKVQA: A knowledge-intensive VQA dataset where answering questions requires commonsense and world knowledge grounded in the image. It contains 17,056 / 1,145 / 6,702 samples for train/val/test; since test labels are unavailable, we evaluate on the validation split.

MMVet: A challenging benchmark designed to evaluate integrated multimodal reasoning abilities, combining skills such as recognition, OCR, spatial reasoning, and knowledge grounding. The benchmark contains 218 evaluation samples.

OKVQA: A knowledge-based VQA benchmark where answering questions requires external world knowledge beyond what is directly visible in the image, such as object functions, cultural facts, or scientific concepts. The dataset contains 14,055 questions with 9,009 / 5,046 train/test splits, and we report results on the test set.

VILP: A dataset probing language prior bias, where language cues suggest an answer that contradicts the visual evidence. The dataset contains 900 evaluation samples, with 300 Easy images, each of which has 2 Adversarial variants.

VLM-are-biased: A diagnostic benchmark exposing confirmation bias in VLMs through visually misleading examples that encourage reliance on memorized associations. We use the original split of 458 samples.

Together, these benchmarks evaluate uncertainty estimation across diverse VQA settings, with VILP and VLM-are-biased specifically probing visual uncertainty.

7.3 Confident Visual Embeddings↩︎

As illustrated in Fig. 9, we estimate the model’s visual confidence by probing the final-layer visual token representations. Each visual token embedding is projected into the vocabulary space using the language model head via LogitLens [17], producing a token distribution over the vocabulary. We compute the entropy of each projected distribution and average across visual tokens to obtain the visual entropy \(H_{\text{vis}}\), which reflects the model’s confidence in the visual representation. This metric allows us to distinguish visually confident inputs from visually uncertain ones, enabling us to analyze cases where semantic entropy fails to reflect uncertainty despite ambiguous visual evidence. We use the top/bottom 20% percentile to define high and low visual entropy. We additionally test top/bot. 10% & 30% percentiles and also observe increased entropy under perturbations (Fig. 10-Left).

7.3.1 Robustness.↩︎

We further analyze the observation in Sec. 3.2 using Qwen2.5-VL and Gemma3 on VILP and VAB (Fig. 10-Right). Quantitatively, we measure the entropy increase and report the mean \(\Delta H\) with 1-sided Wilcoxon signed-rank test \(p\)-value in Table. 7. We consistently observe positive entropy shifts, with most being statistically significant, suggesting robustness across datasets and models.

8 Visual Ambiguity analysis.↩︎

We understand that visual ambiguity is semantically related, and therefore analyze whether low-level VSE relates to semantic regions in pixel space (feature-level analysis is included in Sec. 11.3.1). We identify semantic regions using SAM3 to localize objects mentioned in the questions and answers in AOKVQA, then apply Gaussian noise either in/outside these regions. The AUC of Inside, Outside and Full-image VSE are 79.1,75.2,78.3 respectively. Moreover, full-image VSE strongly correlates with semantic-region VSE (\(r\)=0.6), suggesting that full-image VSE are driven by semantic regions. Therefore, although Gaussian noise is low-level, VSE reflects visual ambiguity rather than noise-induced instability.

9 Locality of Text and Images↩︎

Image vs.Text Embedding shifts. To examine how different perturbations affect the multimodal representation, we compare the embedding shifts induced by textual paraphrases and image perturbations. For each question, we generate semantically equivalent paraphrases using Gemma3-8B [22] following the prompting strategy of [13]. Image perturbations are produced by adding Gaussian noise with standard deviation 20 to the input image. All perturbed inputs are then encoded into a shared multimodal embedding space using Qwen3-VL-Embedding-8B [15]. For each original image–question pair, we generate eight image perturbations and eight question paraphrases, and compute the cosine distance between their embeddings and the embedding of the original pair. Fig. 11 shows the resulting distance distributions across datasets. Image perturbations consistently induce small embedding shifts, indicating that they explore a local neighborhood around the original query. In contrast, textual paraphrases produce substantially larger shifts, suggesting that paraphrasing moves the representation to distant regions of the embedding space rather than sampling locally around the original conditioning.

Image vs.Text Embedding Space. We further visualize the multimodal embedding space using t-SNE. As shown in Fig. 12, image perturbations remain close to the original embedding, whereas question paraphrases induce substantially larger shifts and disperse across the space.

10 Text perturbation drives cluster assignments↩︎

Such non-local shifts have important implications for uncertainty estimation. When both image and text are perturbed, outputs produced under different paraphrases are likely to occupy distinct regions of the embedding space, even if the visual information remains unchanged. As a result, clustering would group outputs by paraphrase identity rather than by variations in the visual input. To verify this effect, we further provide qualitative examples of the cluster occupancy patterns on the perturbation grid (Fig. 13). We observe horizontal stripe structures where cluster membership is consistent within a paraphrase but invariant across image perturbations. This pattern confirms that cluster assignments are largely driven by the textual input. Hence, these results show that joint image-text perturbation primarily introduces variability through textual paraphrases rather than visual changes. Consequently, uncertainty estimates derived from such perturbations may reflect prompt-induced variability instead of genuine visual ambiguity.

11 Hyperparameters Sensitivity↩︎

11.1 Temperature \(T\) and Number of samples \(M\)↩︎

Fig. 14 analyzes the impact of decoding temperature \(T\) and the number of generated variants \(M\) on uncertainty estimation performance. Increasing \(M\) consistently improves AUC for both Qwen2.5-VL and Gemma3, with the largest gains observed from 5 to around 15-20 samples. Performance largely saturates once \(M\geq 15\), despite the linear increase in computational cost, indicating diminishing returns from additional sampling. Higher temperatures (\(T \in [0.8, 1.0]\)) consistently achieve the best results, while lower temperatures suppress output diversity and limit uncertainty signals. Overall, effective uncertainty estimation benefits from sufficiently stochastic decoding and a moderate number of samples, balancing diversity and computational efficiency. Fig. 15 presents the same ablation in line-plot form for clarity. Increasing the number of variants \(M\) and decoding temperature \(T\) consistently improves AUC, with gains diminishing beyond \(M \geq 15\).

11.2 Augmentation strength \(\xi\)↩︎

We vary the visual perturbation strength from 20 to 500 and observe that moderate noise at 20 yields the best performance, while stronger perturbations slightly reduce AUC (Fig. 16). This falls into the behavior of text-driven perturbations: large textual shifts introduce variability that is not grounded in visual evidence and are therefore harmful. Similarly, overly strong visual perturbations create excessive shifts in the visual space, distorting semantic content rather than revealing genuine ambiguity. In both cases, large shifts degrade uncertainty estimation quality.

11.3 Visual Perturbation Functions \(\mathcal{T}\)↩︎

Table 8 compares different visual perturbation functions for uncertainty estimatio (AUC) on AOKVQA. We evaluate four augmentation types with controlled magnitudes:

• Affine Transform: small geometric perturbations including rotation (\(\pm 10^\circ\)), translation (up to \(5\%\) of image size), scaling within \([0.95, 1.05]\) ratio, and shear (\(\pm 10^\circ\)).

• Color Transform: photometric adjustments implemented via random brightness, contrast, and saturation scaling sampled from \([0.6, 1.4]\), and hue shift sampled from \([-0.02, 0.02]\) in normalized HSV space.

• Gaussian Blur: spatial smoothing using a Gaussian kernel with a small blur radius (\(1\%\) image size) and mild spatial jitter (\(0.1\)).

• Gaussian Noise: additive pixel-wise Gaussian noise with standard deviation \(\sigma = 20\), applied independently to each pixel.

Across both Qwen2.5-VL and Gemma3, Gaussian-based perturbations (noise and blur) achieve stronger AUC compared to affine and color transformations. While the differences are moderate, Gaussian noise yields the best result for Qwen2.5-VL and Gaussian blur performs best for Gemma3. Gaussian-based perturbations (noise and blur) consistently achieve stronger performance across both models. This likely stems from their ability to introduce local, distributed variations that preserve the global semantic structure of the image while probing its immediate neighborhood in representation space. We adopt Gaussian noise as the default perturbation function, as it consistently performs well across models and requires tuning only a single parameter, the noise standard deviation \(\sigma\).

11.3.1 Feature Perturbation↩︎

We also implemented masked patches and feature-level visual-token perturbations, which perform comparably or slightly higher than Gaussian noise. Results on AOKVQA:

As magnitudes are not directly comparable, we ran each method over a strength sweep (mask ratio [10%, 40%], noise scale [2%, 10%]) and report best result. We observe that:

(1) VSE remains strong across perturbation types and not tied to low-level perturbation

(2) Gaussian noise achieves competitive performance, being a simple and practical probe.

We note that all compared methods are blackbox without feature access, and VSE is designed under the same constraint. We agree that feature-level analysis is important and will include it in the paper.

11.4 Distance Function d(,)↩︎

Table 10 compares SNNE and VSE under different semantic similarity metrics, including:

• Cosine distance with all-MiniLM-L6-v2 embedding model [28]

• BERTScore [29],

• DeBERTa-v2-xlarge-mnl [26].

Across both Qwen2.5-VL and Gemma3, VSE consistently outperforms SNNE under stronger semantic metrics. In particular, when using DeBERTa, VSE achieves the highest AUC for both models (0.783 for Qwen2.5-VL and 0.778 for Gemma3). While cosine similarity provides weaker alignment signals, performance improves when adopting contextualized semantic metrics such as
BERTScore and DeBERTa. The gains are more pronounced for VSE, suggesting that clustering-based uncertainty estimation benefits from richer semantic representations. Overall, these results indicate that combining VSE with a strong semantic similarity measure yields the most reliable uncertainty estimates.

12 Complexity↩︎

Complexity of VSE, SE, and SNNE is \(O(M^2)\) . Inference is dominated by VLM sampling, which are all similar, while the aggregation step is negligible: 0.24/0.34/0.21s.

13 Qualitative Examples↩︎

We provide qualitative results where high VSE indicates high uncertainty of wrong answers in Fig. 17; low VSE indicates low uncertainty or confident of correct answers in Fig. 18; and failure cases of VSE in Fig. 19. Percentile indicates the rank of the VSE values within the dataset to reflect a value within a bounded range \([0,1]\) for ease of interpretability. Results are shown for AOKVQA dataset using Qwen2.5-VL model.

References↩︎