Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models
May 29, 2026
Abstract
Test-time compute (TTC) strategies have emerged as a lightweight approach to boost reasoning in large language models (LLMs). However, their application and benefits for vision-language models (VLMs) remain underexplored. We present a systematic study of TTC across seven VLMs and six benchmarks, specifically analyzing feature-based scoring and majority voting methods. We find that feature heuristics fail and voting yields only modest gains in single-model settings. We theoretically show that this limitation stems from a lack of prediction diversity: when outputs are highly correlated, voting provides little benefit. In contrast, multi-model ensembles offer richer diversity, yet standard majority voting fails to account for varying model capabilities. To address this, we propose Entropy-based TTC (ETTC), which selects the most confident prediction based on predictive entropy. Our method reduces to majority voting in the single-model case, but in model ensembles, it leverages confidence disparities to prioritize stronger models. We prove that ETTC outperforms majority voting under mild assumptions and empirically demonstrate that it consistently surpasses both voting and the best individual model. Crucially, our results show that smaller models can synergistically enhance larger ones, unlocking ensembling gains not achievable with standard strategies.1
1 Introduction↩︎
Vision-Language Models (VLMs) have recently achieved remarkable performance across a range of visual reasoning benchmarks [1]–[6]. At the same time, the large language modeling (LLM) community has developed a family of test-time compute (TTC) strategies, particularly those based on chain-of-thought (CoT) prompting, to improve reasoning without modifying model parameters [7]. These strategies generate multiple outputs per input and then aggregate or rank them to produce more reliable predictions.
In the LLM literature, inference-only TTC methods fall broadly into two paradigms. The first is response selection (often framed as Best-of-N), which scores and selects the most promising reasoning trace. While many of these methods rely on trained reward models, feature-based approaches avoid training by estimating quality through textual heuristics. These include analyzing structural signals such as the presence of specific pivot words [8], [9], confident linguistic tone [10], [11], or the length of the reasoning chain [12], [13]. In contrast, confidence-based (e.g., self-consistency) methods treat the model as a stochastic oracle and improve reasoning reliability by aggregating multiple outputs, typically selecting the most frequent answer across samples via majority voting [7], [14], [15].
Applying TTC to VLMs, however, is far from straightforward. Unlike LLMs, VLMs must first perceive and interpret dense visual signals before reasoning over them. This introduces new challenges: (i) visual perception is inherently error-prone and varies across models [16], [17]; (ii) vision-language alignment remains imperfect, creating subtle inconsistencies [18], [19]; and (iii) textual cues that correlate with correctness in LLMs may not reflect true visual understanding [20], [21]. Therefore, it is unclear whether and when TTC strategies can reliably enhance visual reasoning.
To investigate this, we begin with the single-model (multi-round) setting, where one VLM is queried multiple times with some notion of randomness (3). Our findings reveal that: (1) feature-based methods fail to improve accuracy, showing that linguistic style is a poor proxy for visual reasoning quality; and (2) confidence-based methods such as majority voting provide only modest, but consistent, gains, and only when CoT prompting is used. Without CoT, even aggregation brings no benefit.
Next, we investigate why these gains are so limited. Specifically, we analyze the diversity (formally, the statistical dependency) between predictions and show that the effectiveness of voting decreases as predictions become more correlated (4.1). When model outputs are nearly identical, voting cannot amplify the signal of correctness. Empirically, we confirm this across 7 VLMs and 6 datasets: outputs exhibit weak but nonzero dependency, which explains why voting offers only small improvements in practice (4.2).
These insights point to a deeper limitation: in the single-model setting, diversity arises only from sampling randomness, so the expected skill of the model remains unchanged. By contrast, the multi-model ensemble setting naturally introduces stronger diversity: differences in architecture, training data, and even scale create complementary strengths. This makes ensembles both more realistic in practice and more promising for TTC. Existing methods, such as majority voting, cannot exploit this potential: by treating all models equally, voting risks letting weaker but correlated models dominate the outcome. What is needed is a strategy that adapts to model quality and selectively prioritizes the most reliable predictions.
To address this, we introduce a new strategy for visual reasoning: Entropy-based TTC (ETTC) (5.1). Instead of counting votes, ETTC selects the prediction with the lowest entropy (on the answer distribution from multiple responses), that is, the most confident output distribution. In the single-model setting, ETTC reduces to majority voting, ensuring backward compatibility. But in multi-model ensembles, ETTC diverges from standard voting: it leverages confidence gaps across models, allowing smaller models to assist stronger ones rather than overwhelm them. We theoretically prove that ETTC outperforms majority voting under mild dependence assumptions (5.2), and empirically show that it not only improves over voting but can even surpass the best individual model in the ensemble (5.3). This result is particularly striking: smaller models can enhance larger ones when combined wisely, yielding gains not achievable with voting alone.
In summary, our contributions are:
A systematic theoretical and empirical study of TTC in VLMs, showing that feature cues fail and that majority voting yields only modest CoT-dependent gains (3).
A theoretical analysis linking the effectiveness of voting to prediction dependency, supported by empirical evidence across diverse models and datasets (4).
A new TTC strategy that generalizes majority voting and achieves consistent improvements in multi-model ensembles, often surpassing even the best single model (5).
2 Preparation↩︎
We begin by outlining the models, datasets, prompting templates, baselines, and evaluation settings in our experiments.
2.0.0.1 Models.
We evaluate seven open-source VLMs under two complementary multi-model ensemble configurations. In the similar-size (cross-family) setup, we include four VLMs with comparable parameter sizes but diverse architectures: Qwen2.5-VL-7B-Instruct [4], LLaMA-3.2-11B-Vision [1], Gemma-3-12B-it [3], and Pixtral-12B-2409 [2]. In the same-family (varied-size) setup, we use four models from the Qwen2.5-VL-Instruct family [4], ranging from 3B to 72B parameters (3B, 7B, 32B, 72B), allowing us to study scaling effects within a single model architecture.
2.0.0.2 Datasets.
We experiment on six multiple-choice visual QA benchmarks covering three domains. For mathematical reasoning, we use the testmini split of MathVista [22] and the test set of MathVision [23]. For diagram understanding, we include the test sets of TQA [24] and ScienceQA [25]. For general visual reasoning, we use the validation splits of MMStar [26] and MMMU [27]. All datasets contain multiple-choice questions with \(K\) answer options (\(2 \leq K \leq 9\)). Further statistics, including domain, split size, and option counts, are summarized in 3 of 9.1.
2.0.0.3 Decoding.
We generate responses using stochastic decoding [28] via default settings.2 We adopt two prompting formats: (1) Direct Answer prompting discourages intermediate reasoning and elicits immediate answers; (2) Chain-of-thought (CoT) prompting explicitly encourages step-by-step reasoning, followed by a final answer. We use zero-shot, one-stage prompting for both settings to ensure consistency across models. Full prompt templates are provided in 4 5 of 9.2. Final answers are extracted from the text using regular expressions.
2.0.0.4 TTC Baselines.
To revisit test-time compute strategies for visual reasoning, we evaluate four representative baselines spanning the feature-based selection and confidence-based aggregation paradigms. Three are feature-based Best-of-\(N\) methods that score and rank CoT responses using lexical heuristics, rather than requiring a trained reward model: (1) CoT Pivot Word ranks each response by counting predefined reasoning-related expressions (e.g., “alternatively”) [8], [9]; see the full phrase list in 4 of 9.3. (2) CoT Length prefers longer responses, following prior work suggesting a correlation between length and reasoning quality [12], [13]. (3) Feature-All combines four interpretable features (pivot word count, vague word count, total token count, and lexical diversity) to compute a composite score (see 6). As a confidence-based method, (4) Majority Voting [7], [14], [15] aggregates \(N = 16\) samples and selects the most frequent final answer.
2.0.0.5 Evaluation Settings.
We assess all test-time compute methods under two settings: (1) In the single-model (multi-round) setting, a single VLM is queried \(N\) times per question with stochastic decoding. Test-time compute is used to aggregate these intra-model outputs. (2) In the multi-model ensemble setting, \(M\) distinct VLMs are queried per question (each with multiple samples), introducing both intra- and inter-model variation. This setting allows us to study cross-model complementarity and test whether aggregating weaker models can improve upon the best individual model.
3 Whether TTC Works in Visual Reasoning↩︎
We begin by revisiting whether TTC strategies, widely used in LLMs, improve visual reasoning in VLMs. We evaluate four representative methods across six multiple-choice visual benchmarks and compare their performance under two prompting conditions: direct answering and CoT. Results are averaged across seven VLMs.
3.0.0.1 Direct Answer Prompt: TTC fails without CoT.
The Direct Answer setting tests whether test-time variation alone, without prompting explicit reasoning, can boost accuracy. Since no reasoning chains are produced, only confidence-based methods like majority voting are applicable.
As shown in 1 (left, direct answer), voting provides negligible or no improvement over the greedy baseline (often \(<1\%\)). Although we sample 16 outputs per question with stochastic decoding, the model’s predictions remain mostly identical. This suggests that without CoT prompting, VLMs tend to output the same surface-level answer, showing little diversity in interpretation. Consequently, TTC offers no benefit under direct answering. This aligns with findings in LLMs [7], [14], but the issue is further exacerbated in VLMs by the perception bottleneck: visual content must first be accurately grounded before any meaningful reasoning variation can emerge.
3.0.0.2 CoT Prompt: Confidence helps, features do not.
In contrast, when models are prompted to reason step-by-step, test-time strategies have room to work. This setup enables both feature-based scoring (e.g., using CoT length) and confidence-based aggregation (e.g., majority voting).
As shown in 1 (right, CoT), voting consistently improves performance across all benchmarks, with average gains of 2-4%. This validates the utility of test-time sampling under CoT: the model explores diverse reasoning paths and occasionally corrects itself. However, the improvements are modest, suggesting that sampled CoTs are still highly correlated, a hypothesis we will formally investigate in 4. Meanwhile, feature-based methods fail to provide any consistent gain over vanilla CoT. Their performance often fluctuates slightly around the baseline. This highlights a key difference from LLMs: in VLMs, textual heuristics are poor proxies for reasoning correctness because visual understanding is the bottleneck. If perception fails, even a well-formed CoT cannot save the answer.
3.0.0.3 Takeaway.
TTC can improve visual reasoning, but only under specific conditions. Without CoT prompting, models produce nearly identical outputs, leaving no room for improvement. Even with CoT, gains from voting are modest, and feature-based scoring fails to help, highlighting the unique challenges of visual reasoning where perception quality limits downstream reasoning. This raises a key question: when does TTC actually help? To answer this, we now turn to the analysis of majority voting, focusing on how its effectiveness depends on the statistical dependencies among model predictions.
4 When Does TTC Work?↩︎
Why does test-time compute (TTC), especially majority voting, sometimes fail to improve accuracy in visual reasoning? We address this question by analyzing how the statistical dependency among model predictions influences the effectiveness of voting. To this end, we develop a theoretical framework that quantifies this relationship and support it with empirical evidence.
4.1 Theoretical Insight: TTC Helps with Diverse Predictions↩︎
4.1.0.1 Intuition.
Before diving into formal definitions, the core intuition is simple: if a model makes the exact same mistake every time you ask it a question, taking a vote across multiple attempts will not fix the error. Voting only amplifies accuracy if the model’s responses are somewhat diverse (i.e., not completely dependent on one another) but lean toward the correct answer on average. We formalize this by defining the “dependency” between predictions and proving that as dependency goes up, the benefit of voting goes down.
4.1.0.2 Setup.
Suppose we have a multiple-choice question with \(K\) options and a single true answer, denoted as \(Y \in [K]\). We gather a total of \(U\) predictions, labeled \(X_1, \dots, X_U\). These can come from multiple decoding rounds of a single VLM or from different VLMs in an ensemble.3 We evaluate whether a specific prediction is correct using a binary indicator \(Z_u := \mathbb{I}\{X_u = Y\}\), and define the model’s expected accuracy on a single trial as \(p := \mathbb{E}[Z_u]\).
To represent majority voting mathematically, we count the total votes for each option \(k\), denoted as \(S_k := \sum_{u=1}^U \mathbb{I}\{X_u = k\}\). The final voting prediction is the option with the most votes, \(\widehat{Y}_{\mathrm{MV}} := \arg\max_k S_k\). Finally, we define the overall accuracy of voting as \(A_{\mathrm{MV}}(U) := \mathbb{P}(\widehat{Y}_{\mathrm{MV}} = Y)\), and the net improvement it provides over a single guess as \(\Delta A_{\mathrm{MV}}(U) := A_{\mathrm{MV}}(U) - p\).
4.1.0.3 Dependency metrics.
To measure how heavily the \(U\) predictions rely on each other, we quantify their dependency using two standard statistical metrics: normalized mutual information (NMI) for the raw answer options, and correlation for the correctness indicators.
For any two predicted answers \(X\) and \(X'\), NMI measures the shared information between them, normalized by their individual uncertainty (entropy, \(H\)): \[\mathrm{NMI}(X; X') := \frac{I(X; X')}{\min\{H(X), H(X')\}},\] \[H(X) = -\sum_{k=1}^K \mathbb{P}(X = k) \log \mathbb{P}(X = k).\] For the full set of \(U\) predictions, the average NMI across all pairs is: \[\overline{\mathrm{NMI}} := \frac{2}{U(U-1)} \sum_{u < v} \mathrm{NMI}(X_u; X_v).\]
Similarly, for any two correctness indicators \(Z\) and \(Z'\), we define their statistical correlation (where \(p\) is the single-trial accuracy), and average it across all pairs: \[\rho(Z, Z') \!:=\! \frac{\mathbb{E}[ZZ'] - p^2}{p(1 - p)}, \;\; \overline{\rho} \!:=\! \frac{2}{U(U - 1)} \sum_{u < v} \rho(Z_u, Z_v).\]
Theorem 1. Suppose all prediction pairs \((X_u, X_v)\) share the same dependency level (i.e., \(\overline{\mathrm{NMI}}\) or \(\overline{\rho}\)). Then the voting improvement \(\Delta A_{\mathrm{MV}}(U)\) is monotonically decreasing in both \(\overline{\rho}\) and \(\overline{\mathrm{NMI}}\). In particular: \[\overline{\rho} = 1\;(\text{or } \overline{\mathrm{NMI}} = 1) \Rightarrow\;\Delta A_{\mathrm{MV}}(U) = 0,\] \[\overline{\rho} \!=\! 0\;(\text{or } \overline{\mathrm{NMI}} \!=\! 0),\;p \!>\! \tfrac{1}{K} \Rightarrow\;A_{\mathrm{MV}}(U) \!\to\! 1\;\text{as } U \!\to\! \infty.\]
4.1.0.4 Interpretation.
The formal proof is in 8.1. For practical application, this theorem reveals a powerful boundary condition for test-time compute: voting only improves accuracy when predictions are diverse. If all predictions are identical (correlation equals 1), voting reduces to a single prediction, yielding zero gain. Conversely, if predictions are entirely uncorrelated and individually better than random guessing, voting can aggregate the faint signals to achieve near-perfect accuracy given enough samples. Because NMI and correlation are model-agnostic, they serve as highly practical tools to estimate whether TTC will actually help on a given task, without needing access to ground truth labels.
4.2 Empirical Verification↩︎
To validate our theoretical insight, we structure our empirical evaluation into two parts. First, we determine the practical minimum number of decoding samples required to reliably estimate prediction dependency. Second, we test the core hypothesis: does the benefit of voting genuinely decrease as models become more correlated?
4.2.0.1 Motivation.
Our theoretical analysis assumes a sufficiently large number of decoding samples \(U\) for the benefits of voting to fully materialize. In practice, generating many samples incurs steep computational costs. Therefore, we first investigate how quickly our dependency metrics converge as \(U\) grows, aiming to find the minimal sample size that yields stable estimates.
4.2.0.2 Setup.
We use Qwen-7B to generate \(U=2\) to \(16\) decoded outputs for each example across six visual reasoning datasets. For each subset size \(U\), we compute the average normalized mutual information (\(\overline{\mathrm{NMI}}\)) and average correctness correlation (\(\overline{\rho}\)) between response pairs.
4.2.0.3 Findings.
As shown in 2, both \(\overline{\mathrm{NMI}}\) and \(\overline{\rho}\) stabilize rapidly, flattening out around \(U=12\) across all datasets. Beyond this point, drawing additional samples offers minimal benefit in accurately estimating prediction dependency.
4.2.0.4 Takeaway.
Sampling more than 12 to 16 responses provides diminishing returns. To ensure both statistical stability and computational tractability, we confidently use \(U=16\) in all subsequent experiments.
4.2.1 Does Voting Improvement Decrease with Dependency?↩︎
4.2.1.1 Motivation.
We now test our central theoretical claim: voting is most beneficial when model outputs are diverse, meaning the accuracy gained from voting should measurably shrink as prediction dependency increases.
4.2.1.2 Setup.
We evaluate the voting improvement \(\Delta A_{\mathrm{MV}}(16)\) for seven models across six datasets, utilizing \(U\!=\!16\) decoding samples per query. For each model-dataset pair, we compute the average accuracy improvement alongside its average dependency (measured by both \(\overline{\mathrm{NMI}}\) and \(\overline{\rho}\)).
4.2.1.3 Findings.
3 shows a clear, consistent negative correlation between the improvement gained from voting and both dependency metrics. Smaller models (e.g., Qwen-3B, LLaMA) inherently produce more diverse outputs and therefore reap larger benefits from voting. In contrast, larger or more heavily optimized models (e.g., Qwen-72B, Pixtral) exhibit highly deterministic, low-diversity behavior, resulting in minimal gains from test-time aggregation. Detailed results broken down by dataset are in 6 7 in 10.1.
4.2.1.4 Takeaway.
The effectiveness of majority voting hinges entirely on the diversity of model outputs. As predictions become more deterministic (higher \(\overline{\mathrm{NMI}}\) and \(\overline{\rho}\)), voting offers sharply diminishing returns. This establishes a practical principle for deploying TTC: voting is most beneficial when applied to weaker/smaller models, or in uncertain scenarios (like few-shot tasks or domain shifts) where outputs are naturally more stochastic. Conversely, wrapping large, highly consistent models in a voting ensemble often wastes compute for negligible gain.
5 Beyond Voting: Entropy-Based TTC for Multi-Model Ensembles↩︎
Building on the insight that majority voting benefits from diverse yet independent predictions, we now turn to the more realistic and underexplored multi-model ensemble setting. Compared to multi-round decoding from a single model, which suffers from limited prediction diversity, ensembles of heterogeneous models naturally offer complementary strengths and errors. We first introduce an Entropy-based TTC method (ETTC) designed to better leverage cross-model diversity. Then, we theoretically show that ETTC outperforms majority voting under mild assumptions, and empirically demonstrate that it enables smaller models to reliably enhance larger ones in visual reasoning tasks.
5.1 Entropy-Based TTC (ETTC)↩︎
Our previous analysis showed that the effectiveness of voting depends heavily on prediction diversity. However, majority voting has a deeper limitation in multi-model ensemble settings: it assumes all model responses are equally reliable and votes based solely on frequency, ignoring how confident or capable each individual model is. This oversight is less problematic in the single-model setting, since all predictions come from the same source and share the same expected quality. But in multi-model ensembles, where models vary drastically in size, training, and performance, this uniform treatment becomes a liability. A majority of weaker models can easily outvote a stronger one, even when the strong expert is confidently correct.
5.1.0.1 Intuition.
To fix this, we need a mechanism that listens to the “expert” in the room for any given question. Intuitively, when a capable model knows the correct answer, its output probability distribution will be highly concentrated (low uncertainty). Conversely, when a weaker model guesses, its distribution will be flatter (high uncertainty). Therefore, instead of counting votes, we can select the answer from the model that is most certain. To operationalize this, we introduce Entropy-based Test-Time Compute (ETTC): a simple, model-agnostic method that selects the most confident prediction among multiple sources using normalized predictive entropy as a proxy for uncertainty.
Definition 1 (Entropy-Based Selection Rule). Let \(U\) sources (e.g., different models in an ensemble) each produce a predictive distribution \(p_u(\cdot) \in \Delta^{K-1}\) over \(K\) answer options. We define the normalized entropy of model \(u\) as: \[\widetilde{H}_u := -\frac{1}{\log K}\sum_{k=1}^K p_u(k)\log p_u(k) \in [0,1],\] and its top-1 prediction as \(\hat{y}_u := \arg\max_{k} p_u(k)\). ETTC simply selects the prediction from the least-uncertain source: \[u^\star := \arg\min_{u\in[U]} \widetilde{H}_u, \qquad \widehat{Y}_{\min H} := \hat{y}_{u^\star}.\]
This selection rule prioritizes predictions with lower uncertainty. In contrast to majority voting, which can amplify weak or erroneous signals through sheer numbers, ETTC amplifies precision by trusting the most decisive prediction. Notably, ETTC safely reduces to standard voting in the single-model case (average the predictive distributions over multiple rounds and pick the most probable option). But in the multi-model setting, it diverges: it allows strong models to dominate the decision even when they are in the minority, which is essential for leveraging heterogeneous ensembles.
5.1.0.2 Takeaway.
ETTC replaces raw vote counts with model confidence, providing a more principled and adaptive aggregation strategy. In real-world scenarios where model capabilities vary, ETTC prevents over-reliance on weaker models while fully exploiting the reliability of stronger ones.
5.2 Theoretical Insight: ETTC Outperforms Voting in Ensembles↩︎
In a multi-model ensemble, differences in training data and architecture naturally increase answer diversity. While voting treats all models equally, this can backfire: weaker models may collectively outvote stronger ones, especially when their errors are correlated (e.g., due to shared pre-training data). Our goal is to theoretically prove why ETTC provides a more robust alternative in such correlated scenarios.
5.2.0.1 Intuition.
We base our theory on a simple premise: more confident predictions tend to be more accurate. In other words, if a model assigns a very high probability to the correct answer, its overall entropy will be low.
Assumption 1 (Entropy-Accuracy Monotonicity). For a given input with true label \(Y\), suppose model \(u\) assigns probability \(p_u(Y)\) to \(Y\), and \(\widetilde{H}_u\) is its normalized entropy. Then, for all models \(u, v \in [U]\): \[p_u(Y) > p_v(Y) \quad \Rightarrow \quad \widetilde{H}_u < \widetilde{H}_v.\]
While this strict mathematical relationship may not hold perfectly in every single instance, we find that it holds strongly in aggregate practice across all datasets and models tested (see empirical verification in 8 of 10.2).
Given this assumption, ETTC simply selects the prediction from the most accurate model for that specific question, denoted as \(u^\star\). Let \(c^\star := \Pr(\hat{y}_{u^\star} = Y)\) be the accuracy of this best model. ETTC guarantees performance of at least \(c^\star\). Now, to understand where majority voting fails, we must model prediction dependency. Consider a simple coupling scheme: with probability \(\lambda\), all the non-best models make a correlated error and copy the exact same prediction \(W\) (e.g., due to shared biases). With probability \(1 - \lambda\), their predictions are independent. Let \(\bar{c} := \Pr(W = Y)\) be the accuracy of this correlated “bloc” prediction, and let \(A_{\mathrm{MV}}(0)\) be the baseline accuracy of majority voting if all models were perfectly independent.
| Accuracy (%) | Models | Average | Voting | ETTC | |||
|---|---|---|---|---|---|---|---|
| 2-5 | LLaMA | Pixtral | Gemma | Qwen-7B | |||
| MathVista | 52.04 | 56.03 | 65.03 | 72.08 | 61.30 | 68.33 | 75.93 |
| MathVision | 23.41 | 25.20 | 31.84 | 30.18 | 27.66 | 32.05 | 35.57 |
| TQA | 70.41 | 77.34 | 78.86 | 78.50 | 76.28 | 83.65 | 83.90 |
| ScienceQA | 77.84 | 78.32 | 77.83 | 79.76 | 78.44 | 85.52 | 85.28 |
| MMStar | 46.09 | 50.35 | 53.40 | 56.77 | 51.65 | 59.27 | 60.07 |
| MMMU | 42.87 | 47.65 | 52.49 | 50.53 | 48.39 | 53.66 | 58.63 |
| Average | 52.11 | 55.82 | 59.91 | 61.30 | 57.29 | 63.75 | 66.56 |
Theorem 2 (Superiority of ETTC over Voting). With the setup above and under 1, let \(A_{\min H} := \Pr(\hat{y}_{\min H} = Y)\) be the accuracy of ETTC. Then for any correlation level \(\lambda \in [0,1]\), we have: \[\begin{align} A_{\mathrm{MV}}(\lambda) &= \lambda\,\bar c \;+\; (1 - \lambda)\,A_{\mathrm{MV}}(0), \label{eq:mv95affine} \nonumber\\ A_{\min H} & - A_{\mathrm{MV}}(\lambda) \nonumber \\ &= \lambda(c^\star - \bar c) \;+\; (1 - \lambda)(A_{\min H} - A_{\mathrm{MV}}(0)). \nonumber \end{align}\qquad{(1)}\] In particular, \(A_{\min H} \ge A_{\mathrm{MV}}(\lambda)\) for all \(\lambda\), with strict inequality whenever \(\lambda > 0\) and \(\bar c < c^\star\).
5.2.0.2 Interpretation.
We provide the full proof in 8.2. This result mathematically highlights the fundamental flaw of majority voting in ensembles. Because voting ignores model quality, it is highly vulnerable to correlated errors. As the error correlation \(\lambda\) increases (e.g., multiple weak models sharing the same flaw), voting accuracy degrades and is dragged down toward \(\bar c\), which is substantially lower than the expert model’s accuracy \(c^\star\).
In contrast, ETTC entirely bypasses this failure mode by selecting the single most confident prediction. Under the mild assumption that lower entropy correlates with higher accuracy, ETTC guarantees performance at least as good as the most accurate model, regardless of how many weaker models agree on a wrong answer. Since VLMs heavily share training data and architectures, making their predictions inherently dependent, ETTC offers a structurally safer and more principled aggregation strategy.
5.3 Empirical Verification↩︎
We now evaluate ETTC in practical multi-model ensemble settings. We structure our evaluation to answer three key questions: (1) Can ETTC effectively leverage diverse models of similar sizes across different families? (2) Does it remain effective when scaling models within the same architecture family? (3) How robust is the method to extreme capability gaps and modality shifts?
5.3.1 Similar-Sized Models from Different Families↩︎
5.3.1.1 Motivation.
We first evaluate whether ETTC can better leverage diversity among models of comparable scale but distinct architectural families. In this setting, models offer complementary strengths, but the variance in prediction quality makes standard voting noisy.
5.3.1.2 Setup.
We select four models of similar scale (7B-12B): LLaMA-11B, Pixtral-12B, Gemma-12B, and Qwen-7B. These models produce predictions for each dataset, and we compare voting and ETTC on the exact same set of outputs. Notably, no single model consistently dominates across all tasks, and some models are clearly weaker on specific domains, adding noise to the ensemble.
5.3.1.3 Findings.
In 1, ETTC outperforms voting on five of six datasets, yielding an average accuracy gain of +2.81% (66.56% vs.%). The largest improvements occur on tasks where model performance diverges significantly, such as MathVista and MathVision. In these cases, voting suffers from equal-weighting, allowing the weaker models to dilute the correct signal. In contrast, ETTC adaptively prioritizes high-confidence predictions, effectively aligning with the strongest model for each specific item, and often exceeding the best single model’s standalone performance.
5.3.1.4 Takeaway.
When aggregating diverse but uneven models, ETTC offers a clear advantage over voting: it selectively filters noise from weaker models based on their own uncertainty, making it highly effective in heterogeneous settings.
| Accuracy (%) | Models | Average | Voting | ETTC | |||
|---|---|---|---|---|---|---|---|
| 2-5 | Qwen-3B | Qwen-7B | Qwen-32B | Qwen-72B | |||
| MathVista | 51.94 | 72.08 | 78.58 | 80.58 | 70.80 | 83.15 | 84.44 |
| MathVision | 22.27 | 30.18 | 38.80 | 42.89 | 33.53 | 41.32 | 44.84 |
| TQA | 60.85 | 78.50 | 83.06 | 84.52 | 76.73 | 84.90 | 86.70 |
| ScienceQA | 66.67 | 79.76 | 84.21 | 84.64 | 78.82 | 84.04 | 85.03 |
| MMStar | 41.22 | 56.77 | 56.34 | 62.56 | 54.22 | 61.00 | 63.73 |
| MMMU | 37.41 | 50.53 | 59.04 | 64.18 | 52.79 | 58.63 | 65.34 |
| Average | 46.73 | 61.30 | 66.67 | 69.90 | 61.15 | 68.84 | 71.68 |
5.3.2 Same-Family Models of Different Scales↩︎
5.3.2.1 Motivation.
We then examine whether ETTC remains effective when models share the exact same architecture and training data, but differ vastly in scale. While scaling laws introduce meaningful diversity in capability, the shared inductive biases create high prediction dependency, the exact scenario where our theory suggests voting will struggle.
5.3.2.2 Setup.
We use four models from the Qwen family: 3B, 7B, 32B, and 72B. Each model produces predictions on all datasets, and we compare aggregation methods on their combined outputs.
5.3.2.3 Findings.
As in 2, ETTC outperforms voting on all datasets, achieving an average gain of +2.84% (71.68% vs.%). While overall prediction correlation is higher than in the cross-family setting, the performance variance introduced by scale provides useful diversity. Specifically, smaller models occasionally make correct predictions with higher certainty than larger ones. ETTC successfully detects and leverages these instances, allowing smaller models to override the incorrect predictions of large models. Overall, ETTC consistently surpasses the accuracy of the strongest model (Qwen-72B), whereas voting frequently performs worse than the strongest model due to dilution.
5.3.2.4 Takeaway.
Despite architectural homogeneity, ensembles of different-sized models still benefit immensely from confidence-based selection. ETTC avoids overcounting correlated errors and allows smaller models to meaningfully enhance larger ones, challenging the conventional wisdom that the largest model should dictate test-time performance.
5.3.3 Robustness and Generalization↩︎
To comprehensively validate the reliability of ETTC, we stress-test it by analyzing the specific risk of dilution in heterogeneous ensembles, extending our evaluation to text-only reasoning, and addressing miscalibration via supervision.
5.3.3.1 Robustness to Weak Learners.
In our analysis of same-family models, we combined models with vast capability gaps (e.g., 3B vs.72B). While beneficial for coverage, such ensembles introduce the critical risk of “dilution”, where weaker models drag down the performance of stronger ones [29]–[31]. To investigate whether ETTC can mitigate this risk, we conducted a fine-grained ablation on MathVista evaluating all possible subset combinations of the Qwen family (detailed settings and results in 7 of 10.3).
We find that voting is highly sensitive to this disparity: when a weak learner is combined with a strong expert, voting accuracy drops significantly, as the noise from the smaller model effectively drowns out the expert’s signal. In contrast, ETTC demonstrates remarkable robustness. In the same weak-strong pairings, ETTC not only avoids degradation but actually surpasses the standalone strong baseline. By relying on predictive entropy, ETTC effectively acts as a filter: it accepts the smaller model’s answer only when it is highly confident, while disregarding its uncertain errors. This confirms that ETTC enables “safe” ensembling, allowing the integration of disparate models without compromising the expert’s performance.
5.3.3.2 Generalization to Text-Only Reasoning.
A key question is whether the benefits of ETTC are constrained by the perception bottlenecks inherent to VLMs, or if they reflect a broader property of reasoning models. To address this, we extended our evaluation to the text-only domain using “Thinking” LLMs (Qwen-3-Thinking) on standard reasoning benchmarks (ARC-Easy and MMLU-Pro; see 10.4).
Consistent with our VLM findings, ETTC outperforms voting across diverse ensemble configurations in this setting as well. Notably, in highly heterogeneous ensembles (e.g., combining 4B and 235B models), ETTC improves accuracy by nearly 5% over voting on MMLU-Pro. This result confirms that the correlation between predictive entropy and correctness is a fundamental property of reasoning models, independent of input modality, and that ETTC acts as an effective noise filter in pure language tasks.
5.3.3.3 Supervised Calibration.
Finally, while ETTC is robust in zero-shot settings, it relies on the assumption that model confidence is a reliable proxy for correctness, an assumption that weakens when models are miscalibrated (i.e., “confidently wrong”). To mitigate this, we propose a Supervised ETTC variant that learns to weigh confidence signals based on empirically observed reliability. We train a lightweight logistic regressor using entropy-based features to predict the likelihood of correctness, allowing the system to dynamically downweight low-entropy predictions from unreliable models (see 10.5).
Empirically, this variant consistently yields further improvements over unsupervised ETTC, particularly on challenging benchmarks like MathVision [23] where base model calibration is weaker. By effectively penalizing overconfidence, the supervised approach achieves the highest overall performance across all settings. This highlights that while entropy is a strong zero-shot signal, minimal supervision can significantly enhance reliability by adapting to specific model failure modes.
5.3.3.4 Overall Summary.
Across both ensemble settings (diverse and redundant), ETTC consistently outperforms majority voting without requiring additional training or tuning. These results empirically validate our theoretical findings: when prediction dependency undermines voting, entropy-based selection offers a structurally safer and more adaptive path to test-time improvement.
6 Related Work↩︎
6.0.0.1 TTC in LLMs.
Chain-of-thought (CoT) prompting enables multi-step reasoning [32], [33], while self-consistency (majority voting) improves accuracy by sampling and aggregating diverse reasoning paths [14]. Recent studies demonstrate that optimally allocating test-time compute (TTC) can sometimes rival scaling up model parameters [7]. Advanced TTC methods have also explored self-calibration [34] and entropy-minimization for test-time adaptation [35].
Our Position: While inference-time scaling is well-established for text, its efficacy in the multimodal domain remains underexplored. Many recent adaptive methods also require training stages or model weight updates. In contrast, our work bridges the modality gap by systematically evaluating pure, inference-only TTC strategies, revealing how visual perception bottlenecks fundamentally alter the effectiveness of test-time scaling.
6.0.0.2 Enhancing VLM Reasoning.
To improve VLM reasoning, researchers have adapted visual CoT prompts [36] and developed test-time consistency objectives [37], [38]. Alternatively, post-training methods using Reinforcement Learning from Human Feedback have been proposed to align multimodal reasoning [39], [40].
Our Position: Post-training approaches require substantial annotated data and compute budgets, while prompt-specific strategies often lack generalizability. Instead of retraining, we provide a lightweight, inference-only study across diverse, rigorous visual reasoning benchmarks (e.g., MathVista, MMMU). Crucially, we go beyond measuring performance to diagnose why standard aggregation strategies fail in VLMs through the lens of prediction dependency.
6.0.0.3 Ensembles, Uncertainty, and Correlation.
Classic machine learning theory establishes that ensemble gains depend heavily on prediction diversity (i.e., low error correlation) among members [41], [42]. Deep ensembles are widely used to capture predictive uncertainty [43], [44], and probabilistic aggregation often relies on confidence-weighted “opinion pooling” [45], [46].
Our Position: While classic literature often pools full probability distributions or requires co-training diverse experts, our Entropy-based TTC method is designed for modern, off-the-shelf generative models. Rather than averaging distributions or relying on simple frequency (voting), we use per-item predictive entropy as a zero-shot filter to dynamically route trust to the most reliable model, enabling smaller models to safely augment larger ones without dilution.
7 Conclusion↩︎
We presented a systematic investigation into the transferability of TTC strategies from LLMs to VLMs. Our analysis identifies a critical bottleneck: standard aggregation methods like majority voting are fundamentally limited by the high statistical dependency of VLM predictions. We theoretically and empirically demonstrate that without diversity, voting offers diminishing returns, and in heterogeneous ensembles, it succumbs to noise from weaker models. To overcome these limitations, we proposed Entropy-based Test-Time Compute (ETTC), a method that prioritizes prediction confidence over frequency. ETTC proves to be a robust strategy for leveraging the diversity of multi-model ensembles, enabling significantly smaller models to synergistically enhance larger ones without the risk of dilution. Furthermore, we showed that this confidence-correctness correlation extends beyond vision, improving reasoning in “Thinking” LLMs as well. Ultimately, our work suggests that the future of efficient test-time scaling lies not just in generating more samples, but in intelligently selecting the most reliable signals from diverse model ecosystems.
Acknowledgment↩︎
We thank the reviewers for their constructive feedback. We also thank Jiaoda Li, Yu Fan, Jingwei Ni, and Chenxi Pang for their valuable input during the early stages of this work. Yifan Hou is supported by the Swiss Data Science Center PhD Grant (P22-05).
Impact Statement↩︎
This work aims to improve the reliability and accuracy of Vision-Language Models (VLMs) through inference-time strategies. The impact of our proposed method, Entropy-based Test-Time Compute (ETTC), can be analyzed through three primary lenses: reliability, computational efficiency, and potential risks.
7.0.0.1 Enhancing Reliability in Visual Reasoning.
By leveraging predictive entropy to filter out uncertain predictions, our method significantly improves the accuracy of VLMs in complex domains, such as mathematics and scientific diagram interpretation. This advancement is crucial for deploying VLMs in high-stakes applications (e.g., educational tutoring systems or scientific data analysis) where hallucinated or inconsistent answers can be detrimental. By prioritizing high-confidence predictions, our approach helps mitigate the “stochastic parrot” phenomenon often observed in multimodal generation, anchoring outputs to more deliberate reasoning paths.
7.0.0.2 Democratization and Efficient Utilization.
A key finding of our work is that smaller, less capable models (e.g., 3B or 7B parameters) can synergistically enhance the performance of significantly larger models (e.g., 72B) within an ensemble. This has positive implications for the democratization of AI. It suggests that users or institutions with limited computational resources can still contribute meaningfully to ensemble systems by deploying smaller models. Furthermore, this “small-helps-large” dynamic offers a pathway to improve system performance without solely relying on training increasingly massive, energy-intensive models, potentially extending the useful lifespan of existing open-source weights.
7.0.0.3 Inference Cost and Environmental Impact.
While our method avoids the massive carbon and financial costs of training new models, Test-Time Compute (TTC) strategies inherently increase inference costs compared to standard greedy decoding (due to multiple sampling rounds or ensembling). This leads to higher energy consumption per query. However, the ETTC selection mechanism itself introduces negligible computational overhead beyond the initial inference. Moreover, our results suggest that querying a heterogeneous ensemble (e.g., one large model paired with several small, cheap models) using ETTC can yield performance comparable to much larger, more expensive setups. This potentially offers a more favorable trade-off between accuracy and total energy expenditure.
7.0.0.4 Risks of Over-Reliance on Confidence.
Our method relies on the assumption that lower entropy (higher confidence) correlates with correctness. While we empirically validate this trend, there is a risk that models may be “confidently wrong,” particularly in out-of-distribution scenarios or if the base models are poorly calibrated. If deployed without safeguards, this could lead users to place unwarranted trust in incorrect outputs simply because the system assigned them a high confidence score. To mitigate this, we emphasize that ETTC should be used in conjunction with base models that have undergone rigorous safety alignment and calibration, or paired with supervised variants like the one explored in this work.
8 Theoretical Proofs↩︎
8.1 Proof of 1↩︎
Proof. We provide a theoretical justification for the claim that the improvement from majority voting decreases monotonically with statistical dependency among model predictions. We proceed by defining a simple probabilistic coupling model that controls prediction dependency, and then analyze how the expected voting accuracy varies with this dependency level.
8.1.1 Coupling Model: Copy-or-Independent Sampling↩︎
We assume all \(U\) predictions \(\{X_u\}_{u=1}^U\) are drawn from a shared coupling mechanism that depends on a parameter \(\lambda \in [0, 1]\): With probability \(\lambda\), all predictions are identical copies of a single sample \(X\). With probability \(1 - \lambda\), predictions are sampled independently from a shared categorical distribution \(\pi = (\pi_1, \dots, \pi_K)\) over \(K\) options. Formally, for any pair \((X_u, X_v)\), \[\label{eq:copy-iid-coupling} (X_u, X_v) \sim \begin{cases} (X, X), & \text{with probability } \lambda \\ (X', X''), \quad X', X'' \overset{\text{i.i.d.}}{\sim} \pi, & \text{with probability } 1 - \lambda \end{cases}\tag{1}\] This ensures uniform pairwise dependency, controlled by \(\lambda\).
8.1.2 Lemma: Behavior of Dependency Metrics under Coupling↩︎
We now show that both statistical dependency metrics used in our main theorem, normalized mutual information and correctness correlation, are monotonic in \(\lambda\) under this coupling.
8.1.2.1 (a) Normalized mutual information.
Let \(X, X'\) be two predictions drawn according to the coupling in 1 . Their joint distribution is \[P_\lambda(i,j) = \lambda \cdot \pi_i \cdot \delta_{ij} + (1 - \lambda) \cdot \pi_i \cdot \pi_j,\] where \(\delta_{ij}\) is the Kronecker delta. The marginal distributions remain unchanged as \(\pi\).
Since mutual information \(I(X;X')\) increases with \(\lambda\) (via the convexity of KL divergence), and the marginals are fixed, the normalized mutual information \(\mathrm{NMI}(X; X')\) is also non-decreasing in \(\lambda\): \[\mathrm{NMI}(X; X') = \frac{I(X; X')}{H(X)} \uparrow \text{ as } \lambda \uparrow.\] Hence, the average pairwise NMI \(\overline{\mathrm{NMI}}\) is also monotonic in \(\lambda\).
8.1.2.2 (b) Correctness correlation.
Let \(Z_u = \mathbb{I}\{X_u = Y\}\), where \(Y\) is the correct option. Denote single-trial accuracy as \(p = \mathbb{P}(X_u = Y)\). Then for any pair \((Z_u, Z_v)\): Under the “copy” case: \(\mathbb{P}(Z_u = Z_v = 1) = p\). Under the “independent” case: \(\mathbb{P}(Z_u = Z_v = 1) = p^2\).
Therefore, the covariance is \[\mathrm{Cov}(Z_u, Z_v) = \mathbb{E}[Z_u Z_v] - p^2 = \lambda(p - p^2) = \lambda p(1 - p),\] and the correlation is \[\label{eq:rho-lambda} \rho(Z_u, Z_v) = \frac{\mathrm{Cov}(Z_u, Z_v)}{p(1 - p)} = \lambda.\tag{2}\] Thus, the average correlation \(\overline{\rho} = \lambda\).
8.1.3 Main Proof: Monotonicity of Voting Improvement↩︎
Let \(A_{\mathrm{MV}}(U; \lambda)\) be the expected voting accuracy under dependency level \(\lambda\), and let \(A_{\mathrm{single}} = p\) be the single-trial accuracy.
We decompose voting accuracy by conditioning on the latent sampling regime: \[\label{eq:mv-decomp} A_{\mathrm{MV}}(U; \lambda) = \lambda \cdot A_{\mathrm{MV}}(U;\text{copy}) + (1 - \lambda) \cdot A_{\mathrm{MV}}(U;\text{iid}).\tag{3}\]
In the “copy” case, all predictions are identical, so voting is equivalent to a single trial: \(A_{\mathrm{MV}}(U;\text{copy}) = p\). In the “iid” case, predictions are independent, and voting aggregates \(U\) samples from \(\pi\); here, accuracy improves with \(U\), approaching 1 as \(U \to \infty\) if \(p > \tfrac{1}{K}\). Thus: \[\begin{align} A_{\mathrm{MV}}(U; \lambda) &= \lambda p + (1 - \lambda) A_{\mathrm{MV}}(U; 0), \\ \Delta A_{\mathrm{MV}}(U; \lambda) &:= A_{\mathrm{MV}}(U; \lambda) - p = (1 - \lambda)(A_{\mathrm{MV}}(U; 0) - p). \label{eq:mv-improvement} \end{align}\tag{4}\] The improvement \(\Delta A_{\mathrm{MV}}(U; \lambda)\) is thus a linear function decreasing in \(\lambda\), and since \(\lambda = \overline{\rho}\) (from 2 ) and \(\overline{\mathrm{NMI}}\) increases with \(\lambda\), voting improvement is monotonically decreasing in both.
8.1.4 Corollary (Extremes)↩︎
If \(\lambda = 1\) (i.e., \(\overline{\rho} = 1\) or \(\overline{\mathrm{NMI}} = 1\)), then all predictions are identical and voting offers no improvement: \[\Delta A_{\mathrm{MV}}(U) = 0.\] If \(\lambda = 0\) (i.e., predictions are independent) and \(p > \tfrac{1}{K}\), then: \[A_{\mathrm{MV}}(U) \to 1 \quad \text{as} \quad U \to \infty.\] ◻
8.1.5 Discussion↩︎
This result formalizes an intuitive principle: confidence-based aggregation (e.g., voting) helps only when predictions are sufficiently diverse. High dependency, measured either via correctness correlation or mutual information, reduces the effective information gain from additional samples. Empirical results confirm this trend across VLMs and datasets: voting yields larger gains when dependency is low.
8.2 Proof of 2↩︎
Proof. Setup. Fix a \(K\)-way classification item with true label \(Y\). Let \(u^\star := \arg\max_u p_u(Y)\) be the best model and define \(c^\star := \Pr(\hat{y}_{u^\star} = Y)\). Let \(\mathcal{B} = \{u \ne u^\star\}\) be the set of non-best models, with \(|\mathcal{B}| \ge 2\).
Coupling among non-best models. Introduce a latent variable \(L \in \{\text{copy}, \text{iid}\}\): - With probability \(\lambda\), \(L = \text{copy}\) and all non-best models predict a shared label \(W\); define \(\bar c := \Pr(W = Y)\). - With probability \(1 - \lambda\), \(L = \text{iid}\) and the non-best predictions are drawn independently.
Step 1: Accuracy of ETTC. Under Assumption 1, ETTC selects \(\hat{y}_{u^\star}\), so: \[A_{\min H} = \Pr(\hat{y}_{u^\star} = Y) = c^\star. \label{eq:ettc95acc}\tag{5}\]
Step 2: Accuracy of Voting. By law of total probability: \[\begin{align} A_{\mathrm{MV}}(\lambda) &= \lambda\, \Pr(\widehat{Y}_{\mathrm{MV}} = Y \mid L = \text{copy}) + (1-\lambda)\, A_{\mathrm{MV}}(0). \label{eq:mv95total95prob} \end{align}\tag{6}\] Under \(L = \text{copy}\), all non-best models predict \(W\), forming a majority: \[\Pr(\widehat{Y}_{\mathrm{MV}} = Y \mid L = \text{copy}) = \Pr(W = Y) = \bar c.\] Plugging into 6 , we recover: \[A_{\mathrm{MV}}(\lambda) = \lambda\, \bar c + (1 - \lambda)\, A_{\mathrm{MV}}(0). \label{eq:mv95affine95app}\tag{7}\]
Step 3: Difference and monotonicity. Subtracting 7 from 5 : \[\begin{align} A_{\min H} - A_{\mathrm{MV}}(\lambda) = \lambda (c^\star - \bar c) + (1 - \lambda)(c^\star - A_{\mathrm{MV}}(0)). \label{eq:gap95affine95app} \end{align}\tag{8}\] This gap is nondecreasing in \(\lambda\): \[\frac{d}{d\lambda}(A_{\min H} - A_{\mathrm{MV}}(\lambda)) = A_{\mathrm{MV}}(0) - \bar c \ge 0.\]
Step 4: Dominance threshold. Let \[\lambda^\star = \max\left\{0, \frac{A_{\mathrm{MV}}(0) - c^\star}{A_{\mathrm{MV}}(0) - \bar c} \right\}.\] Then for all \(\lambda \ge \lambda^\star\), ETTC outperforms voting; if \(\bar c < c^\star\) and \(\lambda > \lambda^\star\), the gap is strict. ◻
Remarks. - Since \(u^\star\) is the best model, typically \(\bar c < c^\star\) unless all models perform equally well. - If \(A_{\mathrm{MV}}(0) \le c^\star\), then \(\lambda^\star = 0\): ETTC dominates voting at all dependency levels. - Under the copy-or-independent model, the average correctness correlation among non-best models equals \(\lambda\) (see 8.1), providing a direct link between dependency and the TTC advantage.
9 Experiment Settings↩︎
9.1 Dataset↩︎
| Dataset | Domain | Type | Size | Option Num. |
|---|---|---|---|---|
| MathVista | Math | testmini | 540 | 2–8 |
| MathVision | Math | test | 1,532 | 5 |
| TQA | Diagram | test | 3,285 | 4 |
| ScienceQA | Diagram | test | 2,017 | 2–5 |
| MMStar | General | val | 1,500 | 4 |
| MMMU | General | val | 805 | 2–9 |
We evaluate our methods on six diverse multi-choice benchmarks spanning three domains: mathematical reasoning (MathVista, MathVision), diagram-based QA (TQA, ScienceQA), and general visual understanding (MMStar, MMMU). 3 summarizes key statistics, including dataset size, official split used, and number of answer options. Note that some datasets contain variable numbers of options (e.g., 2 - 9 in MMMU), which adds to the challenge and makes majority voting less stable. This diversity ensures our evaluation reflects a wide range of real-world reasoning settings.
9.2 Prompt↩︎
Figure 4: Example of a direct QA prompt used for evaluating model predictions without reasoning..
Figure 5: Example of a chain-of-thought (CoT) prompt used to elicit intermediate reasoning steps. This format is used when analyzing consistency or measuring correctness under step-by-step reasoning..
To ensure consistency and minimize response variance across models, we standardize the prompting format in all benchmark evaluations. Specifically, we use a direct QA prompt without explanation, and a chain-of-thought (CoT) style prompt when evaluating reasoning performance or conducting consistency analysis. Below, we show two representative examples for comparison. The image and question are kept identical, while only the prompt template changes.
9.3 Baselines↩︎
To better assess the reliability of CoT responses, we include several shallow feature-based baselines. These models predict the correctness of a response using surface-level properties, without access to model internals or gradient signals.
9.3.0.1 Pivot words.
| Reasoning Type | Example Phrases |
|---|---|
| Realization | “wait”, “oh”, “actually”, “I missed something” |
| Verification | “let me doublecheck”, “to verify”, “checking again” |
| Exploration | “what if”, “another way to look at this”, “alternatively” |
| Integration | “now I see how”, “this connects back to”, “putting this together” |
Pivot words are rhetorical expressions that signal shifts in reasoning, such as realization, verification, or synthesis. Prior work [9] suggests that the presence of such expressions often correlates with more deliberate and structured reasoning. We use a curated list of phrases categorized by rhetorical function, shown in 4. These are used as features for correctness prediction (e.g., counting their presence in CoTs).
9.3.0.2 Vague words.
| Reasoning Type | Example Phrases |
|---|---|
| Uncertainty | “maybe"”, “possibly”, “perhaps”, “probably”, “might be”, “could be”, “it seems” |
| Hedging | “somewhat”, “rather”, “kind of”, “sort of”, “generally”, “typically” |
Vague expressions are often used to hedge or express uncertainty, and may correlate with lower confidence or correctness in model reasoning. We group these into two categories, uncertainty and hedging—based on their rhetorical function. See 5.
9.3.0.3 Feature-All.
| Feature | Modeling Method |
|---|---|
| Token Number | Measures the number of tokens in the CoT response. Longer responses may indicate more reasoning steps, though excessive length may signal loops or noise. We vectorize it as \({1}/\text{Token Number}\). |
| Lexical Diversity | Captures vocabulary richness by counting the number of unique tokens. Low diversity often suggests repetition. We vectorize it as \({1}/\text{Vocabulary Size}\). |
| Pivot Word Number | Counts the number of pivot expressions from 4, indicating structured reasoning or correction. We vectorize it as \({1}/\text{Pivot Word Number}\). |
| Vague Word Number | Counts the number of vague phrases from 5, which may reflect uncertainty or low confidence. We vectorize it as \(1 - {1}/\text{Vague Word Number}\). |
We also define a feature set that combines lexical and stylistic signals for each CoT response. Specifically, we consider four interpretable features: response length (token count), lexical diversity (unique token count), number of pivot words, and number of vague words. See 6 for detailed definitions. For prediction, we compute the sum of these feature values for each example, encouraging longer, more expressive, and more structured responses, while penalizing vague expressions. The model response with the highest total score is selected as the final prediction.
10 Supplementary Results↩︎
10.1 Voting Improvement vs. \(\overline{\mathrm{NMI}}\) and Correlation↩︎
While the overall trends in [fig:improvement_nmi:all,fig:improvement_corr:all] are consistent with our theoretical expectations, MathVision stands out as an exception. Specifically, we observe weaker or even inverted correlation between prediction dependency and voting improvement on this dataset. A likely explanation is that MathVision poses significantly higher difficulty for current VLMs, its average accuracy across models is around 30%, which suggests that models are often uncertain or guessing. In such low-performance regimes, prediction behaviors may become erratic or overly stochastic, reducing the reliability of entropy, correlation, and voting-based signals. As a result, the dependency measures may not reflect meaningful error structure, making voting behavior less predictable.
10.2 Empirical Evidence to Support Assumption↩︎
8 shows the relationship between normalized entropy \(\widetilde{H}_u\) and accuracy across multiple models on six benchmarks. We observe a strong inverse correlation between entropy and accuracy, consistent with our Entropy-Accuracy Monotonicity assumption (1). Higher-performing models generally exhibit lower entropy, indicating more confident and reliable predictions.
10.3 Ensemble Robustness Analysis↩︎
| Combination | Min. | Max. | Avg. | Voting | ETTC |
|---|---|---|---|---|---|
| 3B, 7B | 51.94 | 72.08 | 62.01 | 69.81 | 79.26 |
| 3B, 32B | 51.94 | 78.58 | 65.26 | 72.04 | 83.33 |
| 3B, 72B | 51.94 | 80.58 | 66.26 | 73.15 | 84.81 |
| 7B, 32B | 72.08 | 78.58 | 75.33 | 81.48 | 82.78 |
| 7B, 72B | 72.08 | 80.58 | 76.33 | 82.22 | 84.63 |
| 32B, 72B | 78.58 | 80.58 | 79.58 | 84.44 | 84.26 |
| 3B, 7B, 32B | 51.94 | 78.58 | 67.53 | 81.30 | 82.41 |
| 3B, 7B, 72B | 51.94 | 80.58 | 68.20 | 82.22 | 84.44 |
| 3B, 32B, 72B | 51.94 | 80.58 | 70.37 | 83.70 | 84.63 |
| 7B, 32B, 72B | 72.08 | 80.58 | 77.08 | 83.70 | 84.26 |
10.3.0.1 Setup.
To investigate whether including weaker, smaller models in an ensemble degrades performance, we conducted a comprehensive ablation study using the Qwen-2.5-VL family (3B, 7B, 32B, 72B) on the MathVista benchmark. We evaluated all possible pairwise and triplet combinations to simulate diverse ensemble qualities (7).
10.3.0.2 Findings.
The results highlight a critical failure mode of majority voting in heterogeneous ensembles. For instance, when combining the weakest model (3B, \(\sim\)52% accuracy) with the strongest (72B, \(\sim\)80% accuracy), voting performance drops significantly to 73.15%, effectively dragging the strong model down toward the average. This confirms that voting is vulnerable when the ensemble contains models with large capability gaps.
In contrast, ETTC demonstrates remarkable robustness. In the same 3B+72B setting, ETTC achieves 84.81%, not only avoiding the degradation seen in voting but actually surpassing the standalone performance of the 72B model by over 4%. This trend holds across triplet configurations as well; for example, in the {3B, 7B, 72B} ensemble, voting achieves 82.22% while ETTC reaches 84.44%.
10.3.0.3 Takeaway.
These findings demonstrate that ETTC effectively utilizes predictive entropy to “filter” unreliable signals from weaker models while still leveraging their occasional correct, high-confidence predictions. Unlike voting, which requires careful curation of similarly-capable models to avoid dilution, ETTC allows for safe ensembling: users can integrate smaller, cheaper models (like the 3B) to boost larger ones without the risk of degrading overall performance.
10.4 Generalization to Thinking LLMs↩︎
| Dataset | Models | Min. | Max. | Avg. | Voting | ETTC |
|---|---|---|---|---|---|---|
| ARC-Easy | 4B, 30B | 0.9599 | 0.9714 | 0.9656 | 0.9769 | 0.9878 |
| 4B, 235B | 0.9599 | 0.9772 | 0.9686 | 0.9769 | 0.9878 | |
| 30B, 235B | 0.9714 | 0.9772 | 0.9743 | 0.9874 | 0.9895 | |
| 4B, 30B, 235B | 0.9599 | 0.9772 | 0.9695 | 0.9891 | 0.9899 | |
| MMLU-Pro | 4B, 30B | 0.8116 | 0.9412 | 0.8764 | 0.8934 | 0.9408 |
| 4B, 235B | 0.8116 | 0.9431 | 0.8773 | 0.8979 | 0.9467 | |
| 30B, 235B | 0.9412 | 0.9431 | 0.9421 | 0.9504 | 0.9519 | |
| 4B, 30B, 235B | 0.8116 | 0.9431 | 0.8986 | 0.9482 | 0.9482 |
10.4.0.1 Setup.
To verify that our findings are not an artifact of the visual modality or specific to Vision-Language Models (VLMs), we extended our evaluation to text-only reasoning tasks using Thinking LLMs. We employed the Qwen-3-Thinking family (4B, 30B, and 235B parameters) and evaluated them on two established reasoning benchmarks: ARC-Easy (common sense reasoning) and MMLU-Pro (mathematics subset). We tested various ensemble configurations, including combinations of models with vast size discrepancies (e.g., 4B + 235B).
10.4.0.2 Findings.
As shown in 8, the benefits of ETTC generalize robustly to the text domain. Across all 8 ensemble configurations, ETTC consistently outperforms majority voting. Notably, on the MMLU-Pro dataset, aggregating the 4B and 30B models with voting yields 89.34%, significantly underperforming the standalone 30B model (94.12%) due to the noise introduced by the smaller model. In contrast, ETTC achieves 94.08%, effectively recovering the performance of the strong model by filtering out the 4B model’s low-confidence errors. Furthermore, in the most heterogeneous ensemble (4B+235B), ETTC improves upon voting by nearly 5 points on MMLU-Pro (94.67% vs 89.79%), demonstrating its ability to safely leverage small-model compute without diluting the quality of large-model outputs.
10.4.0.3 Takeaway.
These results confirm that the correlation between predictive entropy and correctness is a fundamental property of reasoning models, whether multimodal or text-only. ETTC’s success with “Thinking” models suggests it is a general-purpose, modality-agnostic strategy for enhancing test-time reliability in heterogeneous ensembles.
10.5 Supervised ETTC↩︎
We provide additional details on the supervised variant of ETTC, which learns from a small set of labeled question–model pairs when low entropy is a reliable signal of correctness.
10.5.0.1 Problem setting.
Given \(Q\) questions and \(M\) models, each model \(u\) produces a predictive distribution \(p_{qu}(\cdot)\) over \(K\) options for question \(q\), aggregated over \(U{=}16\) stochastic decoding samples (see 4). The goal is to learn a function that predicts whether a model’s low-entropy output is likely to be correct.
10.5.0.2 Feature construction.
For each \((q,u)\) pair, we compute two features: \[\widetilde{H}_{qu} := -\tfrac{1}{\log K} \sum_{k=1}^K p_{qu}(k)\log p_{qu}(k), \quad \mathrm{RelEnt}_{qu} := \frac{\widetilde{H}_{qu}-\min_v \widetilde{H}_{qv}}{\max_v \widetilde{H}_{qv}-\min_v \widetilde{H}_{qv}}.\] Here \(\widetilde{H}_{qu}\) is the normalized entropy of model \(u\), while \(\mathrm{RelEnt}_{qu}\) contextualizes this entropy relative to other models for the same question. The final feature vector is \((\widetilde{H}_{qu}, \mathrm{RelEnt}_{qu}) \in \mathbb{R}^2\).
10.5.0.3 Labels and classifier.
The binary label is \[Z_{qu} := \mathbb{I}\{\hat{y}_{qu} = Y_q\},\] where \(\hat{y}_{qu}\) is the top-1 prediction and \(Y_q\) is the ground truth. We train a logistic regression classifier to predict \(\Pr(Z_{qu}=1)\) from the entropy features.
10.5.0.4 Training protocol.
To simulate low-resource conditions, we use two-fold cross-validation across questions: each dataset is split into halves, one for training and one for testing, with roles reversed in a second run. This prevents test leakage and mimics scenarios where only limited annotations are available.
10.5.0.5 Inference rule.
At test time, for each \((q,u)\) we compute the adjusted score \[\mathrm{Score}_{qu} := \widetilde{H}_{qu} \cdot (1 - \hat{p}_{qu}),\] where \(\hat{p}_{qu}\) is the predicted correctness probability from the classifier. We then select the model with the lowest score: \[u_q^\star := \arg\min_u \mathrm{Score}_{qu}, \quad \widehat{Y}_q := \hat{y}_{q u_q^\star}.\] This rule penalizes overconfident but unreliable predictions while rewarding trustworthy ones.
| Accuracy % | Similar Size Models | Same Family Models | ||||||
|---|---|---|---|---|---|---|---|---|
| 2-5 (lr)6-9 | Avg. | Voting | ETTC | Sup. ETTC\(_{\Delta}\) | Avg. | Voting | ETTC | Sup. ETTC\(_{\Delta}\) |
| MathVista | 61.30 | 68.33 | 75.93 | 70.80 | 83.15 | 84.44 | ||
| MathVision | 27.66 | 32.05 | 35.57 | 33.53 | 41.32 | 44.84 | ||
| TQA | 76.28 | 83.65 | 83.90 | 76.73 | 84.90 | 86.70 | ||
| ScienceQA | 78.44 | 85.52 | 85.28 | 78.82 | 84.04 | 85.03 | ||
| MMStar | 51.65 | 59.27 | 60.07 | 54.22 | 61.00 | 63.73 | ||
| MMMU | 48.39 | 53.66 | 58.63 | 52.79 | 58.63 | 65.34 | ||
| Average | 57.29 | 63.75 | 66.56 | 61.15 | 68.84 | 71.68 | ||
10.5.0.6 Results.
As in 9, supervised ETTC outperforms both voting and unsupervised ETTC across datasets and ensemble settings. Gains are largest on ambiguous tasks (e.g., MathVision, MMStar, MMMU), where entropy alone is less reliable. Even with only two-fold cross-fitting and no extra supervision, the classifier learns to identify failure modes of entropy selection, making more robust choices and underlining the value of combining entropy with supervised error modeling.
Limitations↩︎
Our study focuses on multiple-choice visual reasoning tasks and assumes access to model confidence scores via output distributions. The proposed methods, especially entropy-based selection, may not directly generalize to open-ended tasks or models lacking probabilistic outputs. Additionally, while our evaluation covers diverse datasets and model ensembles, the gains of supervised entropy-based TTC depend on the quality and availability of annotated examples, which may be costly to obtain in some domains. Lastly, our analysis assumes that entropy correlates with accuracy, which may not hold for all models or tasks.
LLM Usage↩︎
We used ChatGPT as general-purpose assistive tools during the preparation of this paper. Specifically, LLMs were employed for polishing grammar, improving clarity, formatting LaTeX, generating illustrative figures, and debugging minor code snippets. LLMs were not involved in research ideation, experimental design, or the development of theoretical results.
References↩︎
Our code is publicly available here: https://github.com/nanfang-wuyu/Diversity-Matters.↩︎
Note that our theoretical result holds in both conditions.↩︎