1 Introduction↩︎

Medical AI systems have progressed from black-box models that produce predictions without explanation, to MLLMs that generate interpretable reasoning traces alongside their answers. A natural next step in this progression is calibrated self-assessment: a model that not only explains its reasoning but also communicates how much it trusts its own answer. Recent benchmarks show that MLLMs achieve competitive performance on medical knowledge tasks [1] and, in some settings, match or exceed physician-level diagnostic accuracy [2], [3], driving interest in clinical applications such as report drafting, case summarization, and decision support. However, these models consistently express high confidence even when their answers are incorrect, limiting their reliability in clinical use.

A clinician who receives a model-generated suggestion must manually verify the reasoning and check whether it is grounded in the image, a process that negates the efficiency gains the system is meant to provide. A more practical paradigm is one where the model communicates its own uncertainty, analogous to how a human colleague qualifies a recommendation, so that the user can focus verification effort on the cases the model itself flags as uncertain.

Existing verbalized confidence calibration methods have been developed for text-only LLMs and do not account for the multimodal nature of medical image understanding. In this work, we propose a training-based framework for verbalized confidence calibration in Medical VQA, built on three contributions: (1) a \(2\times2\) factorial perturbation design that probes the model’s reliance on visual evidence versus language priors, (2) a composite calibration loss that combines Brier-based calibration with a contrastive alignment term derived from the perturbation conditions, and (3) a top-\(k\) KL divergence regularizer that preserves the answer distribution during confidence training. Across three benchmarks spanning different accuracy regimes and two model architectures, our method outperforms all compared baselines on average ECE, Brier Score, and AUROC.

2 Related Work↩︎

Confidence estimation for LLMs falls into three broad families. Token-probability methods treat output logits directly as confidence scores, using sequence-level perplexity [4] or semantic weighting of answer-relevant tokens [5]. Verbalized confidence methods prompt the model to express certainty in natural language [6]. Consistency-based methods aggregate multiple sampled responses to estimate uncertainty through voting [7], semantic clustering [8], or perturbation-based variance [9]. Top-K Sampling [10] generates multiple answers per question and averages their confidence scores, while SteerConf [11] prompts the model at multiple steering levels and aggregates the responses into a composite score. Beyond these, internal-representation approaches extract confidence signals from hidden states or intermediate layers [12]–[14], and architectural alternatives such as LARS [15] attach dedicated confidence heads to the model. Pre-trained models exhibit calibrated self-knowledge [16], but token probabilities degrade after RLHF alignment [6], hidden-state methods require white-box access to model internals, and consistency methods require multiple inference passes at test time. Among these families, verbalized confidence is the most practical for deployment, as it requires no access to model internals and produces human-interpretable outputs, but the resulting confidence scores are poorly calibrated without further training [17]–[19].

Calibrating verbalized confidence has therefore become an active research direction. Supervision-based (SFT) methods construct proxy targets through token-probability distillation [20], post-fine-tuning self-assessment [21], systematic question-context mutations [22], or clustering of sampled reasoning chains [23]. Reward-based RL methods instead optimize proper scoring rules directly: Rewarding Doubt [24] and LoVeC [25] use logarithmic scoring, while ReCalibrate [26] uses the Brier score. ConfTuner [27] bridges both directions by replacing the SFT cross-entropy loss with a tokenized Brier score, computing the expected squared error under the model’s probability distribution over confidence tokens. Unlike standard cross-entropy, which treats all token mismatches equally, this formulation naturally encodes ordinal structure: the penalty for predicting confidence 3 when the correct value is 2 is smaller than for predicting 7, achieving the theoretical guarantees of proper scoring rules without requiring policy optimization. To preserve answering ability during confidence training, it regularizes the answer tokens with a cross-entropy loss computed on the model’s own outputs.

Despite their differences in training signal and algorithm, these methods share two limitations: they all operate on text-only LLMs without accounting for the visual modality, and they train on a single input condition, so the model never observes what happens when evidence sources are removed or disrupted. To the best of our knowledge, no existing method applies training-based calibration to multimodal models in the medical domain: multimodal work applies perturbation-based fine-tuning to object-level detection in general domains [28] or remains prompt-based [29], and medical methods rely on prompt engineering [30] rather than fine-tuning.

3 Methodology↩︎

We formalize the task as follows. Given a medical image \(\mathbf{x}\), a question \(q\), and \(A\) answer options \(\mathcal{O}\), the model generates a reasoning trace \(r\), an answer \(a \in \mathcal{O}\), and a verbalized confidence score \(c\) drawn from a discrete integer scale, normalized to \(\hat{c} = c/N_A \in [0,1]\). We adopt an answer-first, confidence-second generation order, which prior work shows produces better calibrated outputs [11], [27]. We evaluate confidence quality along two axes: calibration (stated confidence matches empirical correctness probability) and discrimination (correct predictions receive higher confidence than incorrect ones), using Expected Calibration Error (ECE) [31], Brier Score [32], and AUROC [33].

3.1 Factorial Perturbation Design↩︎

In Medical VQA, the ground truth is by definition embedded in the image modality: answering correctly requires interpreting the visual evidence. Textual cues such as option co-occurrence patterns or positional biases often act as dataset-specific shortcuts rather than reflecting visually-grounded diagnostic reasoning. We therefore hypothesize that well-calibrated confidence should reflect the contribution of each evidence source to the prediction, and we design a perturbation scheme that teaches the model to ground its confidence in evidence utilization.

We cross two perturbation axes, image availability (original versus black image) and text integrity (original versus perturbed options), in a \(2 \times 2\) factorial design that yields four conditions per question (see Figure 1). The black-image condition removes all visual evidence while preserving the input structure, probing the model’s reliance on the visual modality rather than degrading it with classical perturbations such as Gaussian noise, which distort diagnostic features in medical images [34], [35]. We discuss the design rationale and provide qualitative examples of model behavior under the black-image condition in Section 6.7 (Appendix). The perturbed-options condition shuffles the option order and replaces all distractors with options from other same-type questions (e.g., disease diagnosis from disease diagnosis), destroying memorized option patterns while keeping the correct answer and question unchanged.

For each sample under each condition, we generate \(G = 10\) independent responses at temperature \(T = 1.0\) and compute the fractional accuracy: \[\hat{a}_j(i) = \frac{1}{G} \sum_{g=1}^{G} y_{i,j}^{(g)}, \label{eq:fractional95acc}\tag{1}\] where \(y_{i,j}^{(g)} \in \{0, 1\}\) is the correctness of the \(g\)-th response for sample \(i\) under condition \(j\). These fractional estimates approximate the true correctness probability \(p_i\) and serve as training targets for the Brier loss, whose optimum is \(\hat{c}_i^* = p_i\), (see Section 6.3).

3.2 Composite Calibration Loss↩︎

During training, the model autoregressively generates a sequence consisting of three segments: a reasoning trace, an answer, and a verbalized confidence token. Following ConfTuner’s tokenized Brier score methodology [27], the confidence token is drawn from a discrete integer scale determined by the model’s available single-token integers and normalized to \([0, 1]\) by dividing by the scale maximum \(N_A\). We distinguish two sets of token positions in the generated output: answer positions \(\mathcal{T}\), spanning the reasoning and answer segments, and the confidence position, where the model produces a softmax distribution \(p_\theta(v \mid i, j)\) over confidence tokens \(v \in \{0, \ldots, N_A\}\). Our calibration losses operate directly on this softmax distribution rather than on a single decoded value, preserving the ordinal structure of the confidence scale through training.

Three calibration losses shape the model’s expressed uncertainty: the Brier loss drives confidence toward the empirical correctness probability, the anchor loss prevents collapse toward extreme values, and the alignment loss ensures that confidence tracks evidence utilization across perturbation conditions. A separate KL divergence regularizer operates on the answer token positions to preserve the model’s answering ability during confidence training.

3.2.0.1 Brier loss with fractional targets.

For sample \(i\) under condition \(j\): \[\mathcal{L}_{\text{Brier}}(i, j)\!\! =\!\! \sum_{v=0}^{N_A} p_\theta(v \mid i, j) \left(\hat{a}_j(i) - v/N_A\right)^2 \label{eq:brier}\tag{2}\] This is the expected Brier score under the model’s confidence distribution, driving probability mass toward the token whose value matches the fractional accuracy target.

3.2.0.2 Anchor loss.

Under the Brier loss, the optimal confidence for a sample with true correctness probability \(p_i\) is \(\hat{c}_i^* = p_i\) (Eq. 10 ). For samples the model cannot reliably answer (\(p_i \approx 0.5\)), the optimum is the midpoint of the confidence scale, representing maximum uncertainty with respect to binary correctness. Under ideal optimization conditions, the Brier loss would induce this behavior, but in practice the model collapses toward extreme confidence values. The anchor loss addresses this by penalizing deviation from \(c_{\text{mid}} = 0.5\), the point equidistant from both Brier targets (\(y=0\) and \(y=1\)). This divides the scale into a negative side (0.0–0.4), where the model expresses belief that its answer is incorrect, and a positive side (0.6–1.0), where it expresses belief in correctness. The anchor acts as a conservative prior: the model starts at maximum uncertainty and must accumulate sufficient calibration signal from the Brier loss to move its confidence in either direction, preventing collapse toward extreme values without evidence: \[\begin{align} &\mathcal{L}_{\text{anchor}}(i, j) = \\ &\sum_{v=0}^{N_A} p_\theta(v \mid i, j) \left(v/N_A - c_{\text{mid}}\right)^2 \end{align} \label{eq:anchor}\tag{3}\]

3.2.0.3 Alignment loss.

The Brier and anchor losses operate on each condition independently. The alignment loss operates on the expected confidence \[\hat{c}_j(i) = \sum_{v=0}^{N_A} p_\theta(v \mid i, j) \cdot v/N_A\] enforcing that confidence differences across conditions track accuracy differences for each sample: \[\begin{align} \mathcal{L}_{\text{align}}(i, j, k) = \left[ \right. & \left(\hat{c}_j(i) - \hat{c}_k(i)\right) \\ & \left. - \left(\hat{a}_j(i) - \hat{a}_k(i)\right) \right]^2 \label{eq:align} \end{align}\tag{4}\] computed over four same-axis condition pairs: \[\begin{align} \mathcal{P} = \{&(\text{V}, \text{VC}),\; (\text{V-TP}, \text{VC-TP}),\\ &(\text{V}, \text{V-TP}),\; (\text{VC}, \text{VC-TP})\}. \end{align} \label{eq:pairs}\tag{5}\] This teaches the model to ground its confidence in evidence utilization: the confidence change when an evidence source is removed should track the empirical accuracy change, reflecting the informational contribution of that source to the prediction.

3.2.0.4 Top-\(k\) KL divergence regularizer.

Since confidence training modifies the model’s shared representations through the LoRA adapters, it can also affect the answer distribution. We constrain this with a top-\(k\) KL divergence on the answer token positions against a frozen copy of the base model: \[\begin{align} &\mathcal{L}_{\text{KL}}(i, j) = \\ &\frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \sum_{v \in \mathcal{V}_k(t)} p_{\text{ref}}(v \mid t) \log \frac{p_{\text{ref}}(v \mid t)}{p_\theta(v \mid t)} \label{eq:kl} \end{align}\tag{6}\] where \(\mathcal{T}\) is the set of generated token positions excluding the confidence token, \(\mathcal{V}_k(t)\) contains the \(k\) highest-probability tokens under the reference model at position \(t\), \(p_{\text{ref}}\) is the frozen base model distribution, and \(p_\theta\) is the fine-tuned model distribution. Restricting the penalty to the top-\(k\) tokens preserves the answer distribution over the candidates that determine the model’s answer selection, while allowing representation changes in the remaining probability mass for confidence learning. Unlike reinforcement learning settings where KL regularization requires keeping a frozen reference model in memory and recomputing reference distributions at each training step, our setting uses fixed training data. This allows us to extract the reference distributions during data generation, which already requires a base model forward pass to produce the training pairs, and cache them offline. The KL term therefore adds no additional model copies or forward passes during training.

3.2.0.5 Complete objective.

We combine the four components as: \[\begin{align} \mathcal{L} = ~&\lambda\, \mathcal{L}_{\text{Brier}} + (1{-}\lambda)\, \mathcal{L}_{\text{anchor}} \\&+ \alpha\, \mathcal{L}_{\text{align}} + \beta\, \mathcal{L}_{\text{KL}} \label{eq:total} \end{align}\tag{7}\] where each term is averaged over all samples and four conditions. The coefficient \(\lambda\) balances calibration against the conservative prior, \(\alpha\) controls the strength of evidence-aware alignment, and \(\beta\) controls answer preservation. We fine-tune the base model using LoRA adapters [36], targeting the query and value projections (3.2M trainable parameters, 0.075% for MedGemma-4B-IT; 2.5M parameters, 0.030% for Qwen2-VL-7B-Instruct).

4 Experiments and Results↩︎

A calibration method that performs well at only one accuracy level may be exploiting the dataset mean accuracy rather than learning genuine calibration. To test this, we evaluate on three Medical VQA benchmarks spanning different difficulty regimes: OmniMedVQA [37] (high-accuracy regime, 59–70% base accuracy depending on architecture), PMC-VQA [38] (mid-accuracy regime, 45–49%), and MedXpertQA [39] (low-accuracy regime, 20–24%). We compare against four baselines: the base model (zero-shot verbalized confidence), Top-K Sampling and SteerConf, which aggregate multiple stochastic passes, and ConfTuner, which fine-tunes verbalized confidence using a tokenized Brier score. We evaluate on two architectures: MedGemma-4B-IT [1] and Qwen2-VL-7B-Instruct [40]. Our method and ConfTuner produce a confidence estimate from a single forward pass; we use greedy decoding for deterministic evaluation. Top-K Sampling and SteerConf require multiple stochastic forward passes per sample; we report mean and standard deviation over 10 independent runs.

4.0.0.1 Loss Hyperparameters.

Table 3 reports the hyperparameter values for the final OmniMedVQA-trained configuration. The search was conducted using a GridSearch approach, in the ranges indicated in Table 3. The resolution of the grid search is \(0.1\) for \(\lambda\), \(0.5\) for \(\alpha\), and \(1.0\) for \(\beta\). We select hyperparameters on the validation set by filtering to the top quartile of accuracy, then choosing the configuration with the highest AUROC, as it depends only on rank ordering and is therefore not tied to the difficulty or accuracy level of the dataset.

For the PMC-VQA-trained pathway, the same grid search and selection criterion were applied within the hyperparameter family identified on the OmniMedVQA pathway. The selected configuration uses \(\lambda{=}0.7\), \(\alpha{=}1.5\), and \(\beta{=}4.0\), with batch size reduced to 1 group (4 samples) and learning rate set to \(1.4 \times 10^{-5}\) (Table 6).

4.0.0.2 Main results.

Tables 1 and 2 report results for MedGemma and Qwen respectively. Our method achieves the best average ECE, Brier Score, and AUROC under both architectures. The Brier Score, which jointly penalizes miscalibration and poor discrimination, is the strongest individual result: our method leads on every benchmark under both architectures, with reductions of 26–43% relative to the next best method per benchmark on MedGemma and 3–18% on Qwen. This consistency across high, mid, and low accuracy regimes indicates that the training objective captures a general relationship between confidence and correctness rather than exploiting a single benchmark’s accuracy distribution. However, the improvement is not uniform across individual metrics: our method does not achieve the lowest ECE on OmniMedVQA or the highest AUROC on MedXpertQA for both MLLMs.

4.0.0.3 The calibration–discrimination trade-off.

On OmniMedVQA, baselines achieve substantially lower ECE (ConfTuner: 0.068 on Qwen, SteerConf: 0.050 on MedGemma) compared to our method (0.136 and 0.184 respectively). This low ECE comes at the cost of discrimination: on the same benchmark, our AUROC reaches 0.926 (MedGemma) and 0.884 (Qwen), while SteerConf remains at 0.693 and 0.705. The mechanism is confidence compression: methods that cluster predictions near the dataset mean achieve low ECE but cannot separate correct from incorrect predictions (Figures 5–9 and Table ¿tbl:tab:ovl?, Appendix 6.6).

The difference is clearer across difficulty regimes. Baseline ECE degrades as accuracy decreases: on Qwen, ConfTuner’s ECE rises from 0.068 to 0.229 to 0.360 across OmniMedVQA, PMC-VQA, and MedXpertQA; SteerConf follows the same trajectory (0.082 to 0.199 to 0.351). Our method produces the narrowest ECE spread under both architectures (0.097–0.184 on MedGemma, 0.058–0.314 on Qwen), indicating that calibration quality holds regardless of the accuracy regime. On MedXpertQA, no method achieves meaningful discrimination: AUROC ranges from 0.48 to 0.51 across all methods and both architectures. At approximately 20% base accuracy, the signal for separating correct from incorrect predictions is insufficient for any verbalized confidence approach.

4.0.0.4 Accuracy preservation.

Our method preserves accuracy relative to the base model under both architectures: OOD accuracy changes are within \(\pm 0.8\) percentage points, while ID accuracy increases by \(+10.0\) (MedGemma) and \(+7.3\) (Qwen) points on OmniMedVQA. The top-\(k\) KL divergence regularizer anchors the fine-tuned distribution to a frozen copy of the base model, providing consistent accuracy preservation across both architectures. The ID accuracy gain reflects the operating point rather than a training objective; whether calibration gains persist under stricter accuracy constraints remains an open question for future work.

4.0.0.5 Cross-dataset transfer.

To test whether the training objective generalizes beyond the source distribution, we train a MedGemma model on PMC-VQA instead of OmniMedVQA and evaluate on all three benchmarks (Table 4; per-benchmark results in Table 7, Appendix). ConfTuner is re-trained on the same PMC-VQA split for a symmetric comparison.

The PMC-trained model achieves the best average ECE (0.123), Brier Score (0.226), and AUROC (0.624), outperforming all baselines on calibration and discrimination. The OmniMedVQA-trained pathway (Tables 1, 2) yields stronger overall results, but the PMC-trained pathway still outperforms all baselines on average, confirming that the training objective is not specific to a single source dataset. Figures 2 and 3 show reliability diagrams on PMC-VQA for both architectures.

4.0.0.6 Component ablation.

Pairwise ablations on both architectures isolate the contribution of each loss component:

• Without alignment (\(\alpha = 0\)): Removing the alignment term causes AUROC to drop sharply (by 20.5% on OmniMedVQA and 12.2% on PMC-VQA) and Brier Score increases by 72.1% on OmniMedVQA (Table 8). ECE improves on OmniMedVQA (9.8%), indicating that confidence values become more conservative but less discriminative. Without the alignment term, confidence scores are well-calibrated but less informative.

• Without KL (\(\beta = 0\)): Removing the KL regularizer causes accuracy collapse on out-of-distribution data: \(-15.0\) percentage points on PMC-VQA and \(-11.3\) points on MedXpertQA, while ID accuracy decreases modestly (\(-2.0\) points) (Table 9). Without the KL constraint, the model overfits to the confidence training signal at the expense of maintaining its question-answering capabilities.

On Qwen, the same component roles hold (Tables 11, 12, Appendix): removing alignment preserves accuracy but drops AUROC from 0.884 to 0.835 on OmniMedVQA, and removing KL degrades accuracy by \(-11.2\) points on OmniMedVQA and \(-16.2\) on PMC-VQA. Removing KL additionally causes confidence format drift: approximately 22% of outputs revert to alternative formats, producing confidence as ratios, on a 0–100 scale, or as linguistic terms instead of the expected integer scale (Table 10), while the full model maintains 100% format compliance. The base model already follows the confidence format from pretraining; without the KL constraint, fine-tuning erodes this ability, showing that the regularizer preserves not only the answer distribution but also the model’s existing instruction-following capabilities.

4.0.0.7 Practical implications.

To examine whether these calibration gains translate to deployment-relevant behavior, we analyze two simplified clinical scenarios in Appendix 6.8: an AI-assisted setting where the model recommends only when confident, and a clinician-assisted setting where low-confidence predictions are routed for human review. Our method adapts its recommendation volume to dataset difficulty and catches a larger share of errors across the full threshold range compared to all baselines.

5 Conclusions↩︎

We presented a training-based framework for verbalized confidence calibration in Medical VQA, built on three contributions: a \(2\times2\) factorial perturbation design that exposes how the model uses visual and textual evidence, a composite calibration loss that combines Brier-based calibration with contrastive alignment across perturbation conditions, and a top-\(k\) KL divergence regularizer that preserves the answer distribution during confidence training.

Our results show that MLLMs can learn meaningful verbalized confidence through targeted fine-tuning. Under both MedGemma and Qwen2-VL, our method achieves the best average ECE, Brier Score, and AUROC across all three benchmarks, with calibration and discrimination improving simultaneously across three accuracy regimes. The learned confidence behavior generalizes under distribution shift: on MedGemma, both training pathways beat all baselines on average metrics, and consistent improvements on Qwen suggest that the gains are not architecture-specific. Accuracy is preserved within practical margins when the KL regularizer constrains the answer distribution. The ablation study shows that each loss component serves a distinct role: the alignment term drives discrimination, while the KL regularizer prevents forgetting and preserves the model’s instruction-following capabilities, including confidence format compliance.

Two directions for future work emerge from these findings. Evaluating on larger model scales and extending to open-ended VQA would test whether the observed gains persist in more clinically realistic settings. On the methodological side, the anchor and alignment terms in our composite loss make explicit properties that the Brier loss encodes only implicitly. A calibration loss where these properties arise from the formulation itself, without requiring separate corrective terms, would be a more principled alternative.

Limitations↩︎

Discrimination on MedXpertQA remains at chance level for all methods across both model architectures. This means verbalized confidence cannot meaningfully distinguish correct from incorrect predictions on questions requiring multi-step clinical reasoning, limiting the practical utility of calibrated confidence in the most challenging diagnostic scenarios. Both models in this work are small-scale (4B and 7B parameters). All experiments use multiple-choice VQA, where the model selects from a fixed set of options rather than generating free-form answers. This constrains the answer space in ways that simplify both prediction and calibration: correctness is binary and unambiguous. In open-ended clinical settings, where answers vary in granularity and partial correctness is common, calibrating verbalized confidence is a fundamentally harder problem that this work does not address.

Ethics Statement↩︎

Uncalibrated Medical AI systems pose a direct safety risk: a model that expresses 90% confidence when it is correct less than half the time actively misleads clinicians. Our method uses only publicly available benchmarks and does not make use of private clinical data. Medical deployment would therefore require further evaluation under real clinical conditions. Calibration quality may vary across patient populations, imaging modalities, and clinical contexts not represented in the evaluation datasets. This work evaluates calibration in a retrospective, multiple-choice setting; translating these gains to open-ended clinical tasks requires further validation. Finally and importantly, calibrated confidence estimation is intended to support clinical decision-making, but not to replace human judgment.

6 Appendix↩︎

6.1 Experimental Details↩︎

Table 5 reports the dataset splits used in our experiments. For OmniMedVQA, we construct a 20,000-sample pool from the open-access subset using quality-aware stratified sampling that preserves modality proportions and excludes low-quality source datasets (e.g., COVID-19-era collections with variable curation). From this pool, 4,000 training samples are randomly drawn; 3,000 test samples are drawn separately from the remaining open-access data using the same stratification. For PMC-VQA, we randomly subsample 4,000 training instances from the official train split and evaluate on a curated subset of 2,000 samples from the test split. MedXpertQA serves exclusively as an out-of-distribution benchmark, using 2,000 samples from its multimodal subset with no training.

Figure 4 shows the prompt template used across all methods and benchmarks. The model receives the medical image, the question, and four answer options, and is asked to produce a rationale, an answer, and a verbalized confidence score on a scale of 1 to 10. For SteerConf, a steering instruction is inserted at the designated position; for all other methods, this field is left empty. Top-K Sampling uses a separate template that requests the model’s two best guesses with individual confidence scores.

None

ConfTuner [27] was originally designed for text-only LLMs. We reimplemented it for multimodal models following the original code repository, training on the same 4,000-sample OmniMedVQA split without the perturbation augmentation used by our method. SteerConf [11] was implemented using the prompts and sampling parameters from the original code repository (\(T{=}0.7\), 5 inference passes per sample). Top-K Sampling follows [10]: the model produces its two best guesses with individual confidence scores (\(k{=}2\)), sampled at \(T{=}0.7\) over 5 passes, and the final confidence is the average across responses. The alternative pair-rank aggregation proposed in the same work performed worse in our setting, so we report the average-confidence variant.

6.2 Training Configuration↩︎

Table 6 reports the hyperparameters for all training configurations. We use the AdamW optimizer with no learning rate scheduler and fine-tune with LoRA adapters [36]. For MedGemma-4B-IT this yields 3.2M trainable parameters (0.075% of 4.3B); for Qwen2-VL-7B-Instruct, 2.5M parameters (0.030% of 8.3B) due to grouped-query attention reducing the value projection dimensions. All configurations train for 3 epochs; we select epoch 2 for both MedGemma pathways and epoch 3 for Qwen based on calibration metrics on the in-distribution validation set. Training runs use an NVIDIA RTX Pro 6000 (Blackwell) GPU, with each epoch taking approximately one hour.

6.3 Accuracy Estimation↩︎

In the multiple-choice setting with \(K\) options, a single response yields a binary correctness label \(y_i \in \{0, 1\}\). A random guesser achieves expected accuracy \(1/K\), so a single correct response does not reliably distinguish genuine knowledge from a random guess. For instance, a model that has no understanding of a particular question and selects uniformly among \(K = 4\) options still answers correctly 25% of the time. To obtain a reliable estimate of whether the model actually knows the answer for a given sample, we generate \(G\) independent responses for each sample under each condition and compute the fraction of correct responses:

\[\hat{a}_j(i) = \frac{1}{G} \sum_{g=1}^{G} y_{i,j}^{(g)} \label{eq:per95sample95acc}\tag{8}\]

where \(y_{i,j}^{(g)} \in \{0, 1\}\) denotes the correctness of the \(g\)-th response for sample \(i\) under condition \(j \in \{\text{V}, \text{VC}, \text{V-TP}, \text{VC-TP}\}\). The resulting estimate \(\hat{a}_j(i) \in \{0, \tfrac{1}{G}, \ldots, 1\}\) provides a fractional accuracy that estimates the model’s true correctness probability \(p_{i,j} = P(y_{i,j} = 1)\) under condition \(j\). Since the \(G\) responses are independent draws at temperature \(T = 1.0\), \(\hat{a}_j(i)\) is an unbiased estimator with variance \(p_{i,j}(1 - p_{i,j}) / G\). We set \(G = 10\), which balances variance reduction against computational cost.

This fractional estimate is the correct training target for the Brier loss. Let \(p_i\) denote the model’s true correctness probability for sample \(i\). The expected Brier loss over the stochastic binary label is:

\[\mathbb{E}_{y_i}\!\left[(y_i - \hat{c}_i)^2\right] = p_i\,(1 - \hat{c}_i)^2 + (1 - p_i)\,\hat{c}_i^2 \label{eq:expected95brier}\tag{9}\]

Differentiating with respect to the predicted confidence \(\hat{c}_i\) and setting the result to zero gives the optimal prediction:

\[\begin{align} \frac{\partial}{\partial \hat{c}_i}\,\mathbb{E}\!\left[(y_i - \hat{c}_i)^2\right] = -2p_i + 2\hat{c}_i = 0 \quad \\ \Longrightarrow \quad \hat{c}_i^* = p_i \label{eq:brier95optimum} \end{align}\tag{10}\]

The Brier loss is therefore minimized when the predicted confidence equals the true correctness probability. Our fractional estimate \(\hat{a}(i)\) directly approximates \(p_i\), providing the model with a training target that is close to this theoretical optimum.

Given that \(G\) inference runs are already necessary to resolve guessing noise, training on \(G\) binary labels is less efficient than training on the single fractional label. Both approaches converge to the same optimum, as the following gradient identity shows:

\[\begin{align} \nabla_{\hat{c}_i} \frac{1}{G}\sum_{g=1}^{G} (&y_i^{(g)} - \hat{c}_i)^2 \;=\; -2\!\left(\hat{a}(i) - \hat{c}_i\right) \;\\&=\; \nabla_{\hat{c}_i}\,(\hat{a}(i) - \hat{c}_i)^2 \label{eq:gradient95identity} \end{align}\tag{11}\]

The equality holds by linearity of the gradient and the definition of \(\hat{a}(i)\). Averaging the Brier loss gradient over \(G\) binary labels produces exactly the same update direction as computing the gradient from a single Brier loss with the fractional target \(\hat{a}(i)\). The two formulations are therefore equivalent in their optimization signal, but the fractional formulation requires only one backward pass instead of \(G\). Since both approaches already require \(G\) forward-only inference passes to obtain the labels, training on the fractional estimate saves \(G - 1\) backward passes, each of which is more expensive than an inference pass due to gradient computation and backpropagation. ConfTuner [27], by contrast, trains on a single binary label from one inference run and does not have access to a fractional estimate of \(p_i\).

6.4 PMC-VQA-Trained Pathway: Per-Benchmark Results↩︎

Table 7 reports the per-benchmark results for the PMC-VQA-trained pathway summarized in Table 4.

6.5 Per-Benchmark Ablation Results↩︎

Tables 8 and 9 report the per-benchmark ablation results summarized in Section 4. The \(\Delta\) column reports the relative change from Full to Ablation; arrows indicate the direction of performance change (\(\uparrow\) = improved, \(\downarrow\) = degraded).

Tables 11 and 12 report the Qwen2-VL ablation results discussed in Section 4. Removing KL causes confidence format drift: the model reverts to producing confidence as ratios, on a 0–100 scale, or as linguistic terms. Table 10 reports the format compliance breakdown on OmniMedVQA. Unparseable samples in both ablation tables are assigned the midpoint value (maximum uncertainty).

6.6 Confidence Distribution Analysis↩︎

Figures 5–9 visualize the confidence distributions underlying the metrics reported in Section 4.

6.6.0.1 Per-sample discrimination.

Figure 5 compares all methods on OmniMedVQA, splitting predictions by correctness. The base model, SteerConf, and ConfTuner concentrate both correct and incorrect predictions in a similar high-confidence region, producing overlapping distributions. Because OmniMedVQA has high accuracy, this high confidence is close to the dataset mean, which yields low ECE. Our method produces more separated distributions, with correct predictions at higher confidence and incorrect predictions at lower confidence, which yields higher AUROC (Tables 1, 2).

Table ¿tbl:tab:ovl? quantifies this separation. We report the mean confidence for correct (\(\mu_{\text{corr}}\)) and incorrect (\(\mu_{\text{incorr}}\)) predictions, their gap (\(\Delta\mu\)), the standard deviation of all scores (\(\sigma\)), and the Overlapping Coefficient (OVL), which measures the intersection area of the correct and incorrect confidence density functions (lower indicates better separation).

On MedGemma, the base model, SteerConf, and ConfTuner all exhibit high OVL (\(\geq\)​0.698), indicating that their correct and incorrect confidence distributions are largely entangled. The corresponding \(\Delta\mu\) values are small (0.077–0.136), confirming that these methods assign similar confidence regardless of correctness. This pattern is consistent with the overconfidence prior of MLLMs: models default to high certainty, and on a high-accuracy benchmark this prior happens to align with the dataset mean, producing low ECE without per-sample informativeness. Our method increases \(\Delta\mu\) to 0.406 and reduces OVL to 0.261, achieving clear separation between correct and incorrect densities.

On Qwen2, a similar pattern holds for the base model (OVL = 0.431, \(\Delta\mu\) = 0.025) and SteerConf (OVL = 0.692). ConfTuner achieves a larger mean gap (\(\Delta\mu\) = 0.460) than our method (\(\Delta\mu\) = 0.318), though our method achieves lower OVL (0.356 vs.), indicating tighter separation around the means. Both methods improve over the base model and SteerConf on this in-distribution benchmark. The difference between ConfTuner and our method becomes apparent under distribution shift: ConfTuner’s ECE rises from 0.068 on OmniMedVQA to 0.229 on PMC-VQA and 0.360 on MedXpertQA, while our method maintains a narrower range (0.058–0.314), suggesting that the separation learned by our method is more robust to changes in the underlying accuracy regime.

6.6.0.2 Adaptivity across difficulty levels.

Figures 6–9 show how each method’s confidence distribution changes across three benchmarks with decreasing accuracy (OmniMedVQA \(\rightarrow\) PMC-VQA \(\rightarrow\) MedXpertQA). The base model, SteerConf, and ConfTuner produce distributions that are largely fixed: confidence remains concentrated at similar values whether dataset accuracy is 0.70 or 0.23. In a deployment setting where difficulty varies across cases, this rigidity means calibration degrades as soon as the input distribution shifts away from the training regime. Our method’s distribution tracks the underlying accuracy, assigning progressively lower confidence as difficulty increases. This is consistent with the narrower ECE spread reported in Section 4.

6.6.0.3 Metric interpretation.

ECE measures whether average confidence matches average accuracy within bins. A method that assigns near-identical confidence to every prediction can achieve low ECE, but such scores do not distinguish correct from incorrect predictions at the sample level. AUROC measures exactly this ranking ability, and the Brier Score captures both calibration and discrimination in a single quantity. In clinical workflows where uncertain predictions should be flagged for review, per-sample ranking is directly actionable, which motivates optimizing for discrimination alongside calibration.

6.7 Visual Evidence Removal Design↩︎

The \(2\times2\) factorial design (Section 3.1) requires a method for removing visual evidence from the input. Noise-based alternatives such as Gaussian blur or mean pixel padding do not guarantee removal of diagnostic content: determining whether a given perturbation intensity actually destroys the clinically relevant signal requires domain expertise or expert annotation, and the answer varies across the diverse imaging modalities in our training data (fundus photography, dermatoscopy, X-ray, ultrasound, histopathology). A black image guarantees complete evidence removal with no domain expertise and no per-modality tuning, making it a simple, modality-agnostic solution. The fractional accuracy under each condition is then measured empirically, so the training signal reflects how the model actually behaves under evidence removal rather than assumptions about the perturbation.

Tables 13 and 14 show representative outputs under the black-image condition. Both models hallucinate visual findings and produce coherent medical reasoning based on language priors. The outputs are structured and fluent rather than degenerate, indicating that the model falls back on textual priors rather than producing artifacts from out-of-distribution visual encoder activations. In some cases the model also answers correctly under the black-image condition, which is expected: certain questions are solvable from language priors alone (e.g., common medical knowledge or option elimination), and the fractional accuracy \(\hat{a}_{\text{black}}\) captures this. The black image serves as a proxy for probing what evidence the model relies on when forming its answer. Alternative proxies such as learned occlusion masks or domain-aware degradation may provide a more targeted signal and are a direction for future work.

6.8 Calibrated Confidence in Practice↩︎

Standard calibration metrics such as ECE, Brier score, and AUROC quantify statistical properties of confidence distributions. To examine what these properties imply in a more applied context, we consider a simplified deployment scenario: a clinical setting where an MLLM assists with diagnostic cases. A deployed model encounters both familiar and novel cases. OmniMedVQA represents in-distribution cases, analogous to a hospital’s own historical records on which the model was trained. PMC-VQA and MedXpertQA represent out-of-distribution cases, analogous to new or previously unseen case types. In addition to the distribution shift, these datasets vary in relative difficulty for the model: OmniMedVQA is the easiest (highest accuracy), PMC-VQA is moderate, and MedXpertQA is the hardest, collectively approximating the range of case complexity a deployed system may encounter.

Before deployment, the system must commit to a confidence threshold that governs its behavior. The choice of threshold determines which cases the model acts on and which it defers. We examine two paradigms below, both dependent on this threshold and therefore on the quality of the underlying confidence estimates. This analysis examines whether the calibration improvements reported in Section 4 translate to observable differences in simplified deployment scenarios, bridging the gap between aggregate metrics and deployment-relevant behavior. As discussed in Section 6.1, the multiple-choice format enables reliable automated correctness assessment, which is necessary for the analysis below.

6.8.1 AI-Assisted Clinician↩︎

In this paradigm the clinician remains the primary decision-maker and the model serves as an advisory tool. For such a system to be practical, the model should provide recommendations selectively, only when it is sufficiently confident, rather than on every case. Unrestricted recommendations increase clinician workload: each suggestion must be verified, and incorrect confident suggestions may introduce bias or require additional effort to rule out. An ideal advisory model would therefore achieve high precision in its recommendations, offering fewer but more reliable second opinions. Calibrated confidence could enable this behavior by allowing the system to suppress recommendations below a confidence threshold, surfacing only cases where the model’s prediction is likely correct. Such selective recommendations could help clinicians catch diagnostic errors, obtain a reliable second opinion on ambiguous cases, or prioritize cases that warrant closer examination.

Figure 10 quantifies this effect at confidence threshold 6 (on a 1–10 scale), chosen as an illustrative high-precision operating point: an advisory system should only surface recommendations it is reasonably certain about. Each bar represents 100 patients and is divided into correct recommendations (green), wrong recommendations (pink), and cases where the model abstains (gray). Figure 11 later confirms that this pattern holds across the full threshold range.

Baselines assign high confidence scores regardless of dataset difficulty. On MedXpertQA, where overall accuracy is approximately 20%, the base model still recommends on nearly all cases, producing 76 wrong recommendations per 100 patients for MedGemma and 80 for Qwen2. Our method reduces these to 25 and 20 respectively by abstaining on cases where its confidence is low. The abstention is not uniform: on OmniMedVQA, where accuracy is higher, our method recommends on approximately 74% of cases (MedGemma) and 47% (Qwen2) with 71 and 45 correct recommendations, while producing only 3 wrong recommendations in both cases.

This behavior emerges directly from calibration. Because our training procedure encourages confidence scores that track actual correctness likelihood, the model naturally scales back its recommendations on harder datasets without any explicit abstention mechanism. An overconfident model lacks this adaptivity: it recommends with similar frequency on OmniMedVQA and MedXpertQA despite a threefold difference in accuracy, exposing clinicians to a high volume of incorrect suggestions on difficult cases. In practice, each wrong recommendation requires the clinician to identify the error and override the suggestion, adding verification effort that may exceed the time saved by correct recommendations. A calibrated model that abstains on uncertain cases reduces this overhead, providing fewer but more time-efficient second opinions.

6.8.2 Clinician-Assisted AI↩︎

In this paradigm the model operates semi-autonomously, handling the majority of cases and routing predictions with confidence below a threshold to a human reviewer. This setup is relevant to high-throughput settings where case volume exceeds clinician capacity, and the model serves as a first-pass filter. For such a safety gate to be functional, low confidence must correspond to cases that actually need review. An overconfident model assigns high scores indiscriminately, rendering the threshold ineffective, and errors pass through undetected regardless of the chosen operating point.

Figure 11 plots the fraction of model errors that fall below each confidence threshold across the full range (1–9), showing that the pattern from Section 6.8.1 is not specific to a single threshold. On MedGemma (rows 1–2), our method reaches higher error catch rates across the full threshold range in both ID and OOD settings. On Qwen2 (row 3), ConfTuner is competitive on the in-distribution dataset at lower thresholds, though our method catches a larger fraction at higher thresholds. The difference is most visible on out-of-distribution datasets, where baseline curves remain near zero at moderate thresholds because overconfident scores prevent the safety gate from triggering.

To illustrate with a concrete operating point, we examine threshold 3, a high-recall setting where the safety net casts a wider net to catch as many errors as possible. On MedGemma (OmniMed-trained) at this threshold, our method flags 10.6% of cases with 91.5% precision (i.e., 9 out of 10 flagged cases are genuine errors) and catches 49.8% of all errors. At the same threshold, SteerConf flags 4.6% at 65.7% precision catching 9.9%, and the base model flags 0.1% catching effectively nothing. The pattern is consistent across architectures: on Qwen2 at threshold 3, our method achieves 92.8% precision while catching 29.2% of errors. Overall, the improved calibration from our training procedure translates to a more practical safety gate, catching a larger share of errors at lower review cost compared to baselines, and suggests higher deployment readiness for threshold-based human-in-the-loop systems.

While this analysis operates within the multiple-choice evaluation setting, it demonstrates a behavioral prerequisite for clinical deployment: a model must be able to reliably modulate its output, abstaining on cases it is likely to answer incorrectly and engaging on cases where it can contribute. Our calibration framework provides the confidence signal necessary to enable this behavior.

References↩︎

---

1. https://github.com/ErenSenoglu/Verbalized-Uncertainty-Calibration-for-MedVQA↩︎