1 Introduction↩︎

A sizeable body of work has developed around the identification and quantification of uncertainty in the outputs of transformer-based large language models (LLMs). Accurate uncertainty quantification (UQ) is an essential element in predicting model hallucinations and maintaining user trust. In service of that, UQ research has largely focused on developing and utilizing uncertainty measurement methods that are well-calibrated to model accuracy. A well calibrated measure is one that predicts well the model’s likelihood of generating a valid answer to the given context. Contexts with high certainty should have a low likelihood of being incorrect and vice versa. A subset of UQ work focuses on measures that are able to be calculated at any time during generation, without additional auxiliary generations. This has often been referred to as inference-time uncertainty quantification. Inference-time measures are uniquely useful in that they can provide a constant signal to the user or to external control modules without significant added computation.

Existing research has not considered whether the investigated UQ measures align with human uncertainty. So, while research has investigated measures with significant calibration, the reported values may not correspond with human uncertainty, making the meaning of the values difficult to parse for users.

Simultaneously, a growing body of research has emerged that seeks to identify human-like behaviors in a variety of LLM tasks and contexts. This has included behaviors as varied as theory of mind [1]–[3], strategic preferences [4], [5], and framing effects [6], [7]. This work seeks to synthesize these two research thrusts by identifying uncertainty measures that are simultaneously calibrated and aligned to human uncertainty behavior. In particular, we investigate whether any of the uncertainty measures vary consistently with human uncertainty on a per-question basis. Given the difficulty in reliably quantifying uncertainty for an individual human, we approximate this by comparing model measures against disagreement among groups of human survey respondents.

By evaluating LLM uncertainty alignment as well as calibration in inference-time UQ measures, this paper identifies a set of UQ measures which may be effective and more intuitively interpretable by human users—enabling important advances in LLM-human interaction and prompting the further study of alignment in model signals beyond overt action. This paper specifically contributes to the existing understanding of LLM uncertainty by:

1. Noting that top-p selection in LLM decoding is functionally equivalent to the Bayesian highest density credible set, drawing an important but previously un-noted connection between the fields and inspiring our investigation of top-p as a measure of uncertainty.

2. Observing that many entropy-based inference-time uncertainty measures have significant human uncertainty alignment despite moderate choice selection and no preference ordering alignment.

3. Developing a novel ground-truth distributional calibration measure based on shift in the Jensen-Shannon distance metric to directly evaluate the impact of certainty on answer distribution.

4. Showing that the aligned inference-time measures show evidence of calibration in terms of both correctness correlation and ground-truth distributional calibration.

2 Prior Work↩︎

Uncertainty Quantification in LLMs is a broad field with numerous notions of uncertainty depending on context, task, and available resources. These are covered in a variety of surveys including [8]–[10]. Many of the most successful methods in terms of calibration, like monte-carlo dropout [11], [12], rely on multiple generation steps to quantify uncertainty and cannot be readily adapted to quantify per-token inference-time uncertainty levels. Existing work on inference-time uncertainty quantification typically relies on perplexity [13]–[15], maximum token probability [16]–[19], or entropy methods [18], [20].

Very few works have explicitly investigated the presence of human-like uncertainty responses in LLMs. This work was inspired by preliminary work by [21], which investigated human-similarity on a diverse set of uncertainty measures. Their work was limited in that the dataset consisted of less than 40 items and did not consider whether the measures which were aligned were also calibrated. We expand on those results by drastically expanding the dataset size and narrow our focus exclusively to inference-time measures. Our work is also novel in that it is the only extant work, to our knowledge, that simultaneously evaluates any uncertainty measures for both alignment and calibration. Other related work, including [22], [23], have used LLMs and uncertainty-aware procedures to simulate human group responses, but do not seek to establish human-like uncertainty measures.

3 Inference-Time UQ↩︎

This work is interested in investigating inference-time UQ methods, as inference-time calculation is necessary for model control mechanisms to have a signal to interpret and react. Non-inference-time methods are valuable tools for diagnostics, model comparison, etc., but are ill-suited for time-sensitive application. While the field of LLM UQ research is large and growing, it has focused on measures of uncertainty that have limited or no inference-time capabilities. These commonly include intuitive methods like self-reporting [16], [19], [24]–[31], multi-inference consistency [20], [27], [32]–[34], and ensemble variation [12], [35]–[37]. The remainder of this section will describe the inference-time UQ measures employed here. In all cases, these methods are calculated using the token probability distribution over the vocabulary \(V\) given some context \(c\), \(P(v\in V | c)\).

The simplest inference-time UQ measures rely on relative probabilities of the most probable output token [15], [16], [19]. In this work, we refer to this simple approach as the top-1 probability. Prior work typically does not promote this as a UQ measure, instead utilizing it as a classifier feature [15] or as a basis of comparison [16], [19].

The majority of our candidate measures are entropy-based measures. These measures are based on the Shannon entropy over a probability distribution, \(S(P(X)) = -\sum_{x\in X} P(x)\log(P(x))\). Higher entropy distributions are taken to be indicative of higher uncertainty because entropy increases as the relative probabilities throughout the full distribution approach uniform. Typically, this is measured across the entire probability distribution, which we herein refer to as the total entropy for disambiguity.

We further experiment on entropy calculated over a variety of normalized subsets of the total probability distribution. The simplest method of obtaining this subset is using top-k sampling, in which the \(k\) highest probability tokens are extracted from the total probability distribution. We normalize this subset, \(V^k\) by dividing every token probability by the sum of the entire subset. We choose this over softmax because softmax does not necessarily maintain the relative ratio between individual probabilities, which can drastically affect the entropy calculations. We investigate this measure for five values of \(k\): \(5\), \(10\), \(25\), \(50\), and \(100\). Note that the total entropy is a special case of top-k entropy where \(k=|V|\). Because our datasets are exclusively multiple choice format, as are many common benchmarks, we also measure the uncertainty as the entropy over the normalized probabilities of the target tokens, corresponding to the first \(n\) letters of the capitalized alphabet, where \(n\) is the number of provided answer choices. We call this measure the choice entropy.

We also investigate the other common sampling method, top-p sampling. Top-p sampling, also known as nucleus sampling, extracts the most probable tokens such that the cumulative probability of the extracted tokens is maximized and less than \(p\) and the number of tokens extracted is minimized [38]. Despite its apparent similarity to the highest density credible sets commonly found in Bayesian notions of uncertainty, few prior works have investigated it’s viability for LLM UQ. As in [21], we investigate the size of the resulting token set as a measure of uncertainty. We extend that work by investigating more values for \(p\): \(0.95\), \(0.9\), \(0.75\) and \(0.5\). We also include in our study the entropy over the normalized probabilities of the top-p tokens.

4 Human UQ Alignment↩︎

Alignment refers to how closely the behavior of an AI system conforms to the desired behavior of the user or developer. While this is most commonly discussed in terms of how well the models interpret and conform to instructions or moral imperatives, it can also refer to how closely the model’s behavior matches human behavior in some context. We define herein uncertainty alignment using the latter notion. That is, an uncertainty measure is aligned for a given model if the uncertainty measure correlates well with uncertainty in humans. In this work, we focus on the easier task of correlating with uncertainty among groups of human subjects, as defined by the level of agreement on multiple choice surveys, rather than attempting to measure correlation at the individual level, though this should be explored in future work.

4.1 Dataset↩︎

We use two datasets to investigate UQ alignment. The first is the dataset used in the inspiring work, [21]. This dataset is comprised of \(38\) manually collected and formatted questions originally sourced from Pew Research surveys [39]. This dataset is clearly limited in size and thus diversity, but provides a useful baseline for comparison and is included for completeness.

The second dataset is a collection of 2998 randomly selected questions obtained from the Roper Center for Public Opinion Research database . The exact methods used to sample from this database are detailed in the appendix. All questions were obtained from human surveys performed during the years 2017-2023. Minor keyword filtering was employed to reduce the number of time-sensitive and personal experience questions. A total of 30571 questions were initially retrieved from Roper. We removed questions with invalid response ratios and sampled 3000 questions from the resulting set uniformly at random without replacement. Each question’s answer choices were shuffled to reduce ordering biases, but every model was presented with the same answer choice ordering for consistency. After removal of two additional questions for invalid answer choice counts, the final dataset was comprised of 2998 questions with an average of 3.77 choices per question.

4.2 Methods↩︎

Each model was queried for the full token probability distribution using a consistent prompt template, described in Figure 2. In addition, we use the standard cloze test to determine the model’s chosen answer from the options provided. We extract from the probability distributions each of the uncertainty measures described above in the inference-time uncertainty section. Human group uncertainty is obtained by taking the entropy over the response percentages after normalization.

Across all questions, we measure human agreement in three ways. First, we measure overt agreement based on the response ratios and cloze test results to show how often the models and humans agreed on the best answer. For a more fine-grained analysis, we measure the relative preferential alignment between model and human using the normalized Kendall \(\tau\) distance [40]. This measures the minimum number of pairwise swaps needed to convert the model preference order into the human preference order, normalized by the maximum possible distance. Finally, we measure the correlation across all questions between the human group uncertainty and each of the model uncertainty measures.

4.3 Results↩︎

Table 1 displays the results of the overt agreement and order preference analysis. For overt agreement, models show mild agreement, with all models other than Llama-3.2 1B significantly (\(p<0.01\)) beating the theoretically determined random chance (\(\sim0.265\)) based on a one-sided one-proportion z test [41]. The test is appropriate as only one proportion is used given the precise random proportion is calculated and the proportion is from a binomial distribution.

The relational analysis shows remarkably little agreement between model and human. Every model shows a consistent mean distance of \(\mu=0.476 \pm 0.035\) and standard deviation \(\sigma=0.343\pm0.006\), indicating effectively random and widely distributed distance scores. Together, these indicate that the models show moderate agreement with humans on the top token, but not on overall token preference ordering. This finding is at odds with previous work on strategic preference ordering in LLMs [42], with the primary difference in our approaches being the prompt design and output capture approach. The prior work used a prompting strategy dubbed counterfactual prompting as an alternative to the more common cloze testing used here. In the prior, the output is measured using a consistent canary token whose probability is queried once for each option while in the latter a set of options is presented once and the relative probabilities associated with each option is taken as the relative preference. This might suggest that model alignment is strongly sensitive to specific prompting methodology and preference interpretation. The lack of preference ordering alignment could additionally be explained by the fact that models are trained only on individual target tokens and have no information about what tokens would have been valid but less preferred tokens.

The results for uncertainty alignment, shown in Figure 3, are much more promising. As in the prior work, we see wild variation in correlation across models for every measure. Even still, every model shows significant (\(|r|\geq0.3\)) correlation for all top-k measures, including total entropy. The same is true of choice entropy and top-p entropy for all \(p\geq0.75\), though with progressively more instability as \(p\) decreases. Top-p size, counter to previous studies, shows weak but significant correlation for all \(p\leq0.9\). The only measures that do not show consistent significant correlation are top-1 probability, top-p (\(0.95\)) size and top-p (\(0.5\)) size. It is noteworthy, but currently unexplained that degradation trends are reversed between top-p size and entropy. The top performing measures are identified as those whose correlation exceeds \(r \geq 0.5\) for every model. This set includes choice entropy, total entropy, all top-k entropies, and top-p entropy for \(p\geq0.9\). These models are further evaluated for calibration. It should be noted that all of the top performing measures are significant for all models on both datasets, with the exception of those based on top-p entropy.

5 Calibration↩︎

Calibration is the standard measure by which uncertainty measures are evaluated for LLMs. It refers to the measure’s utility in predicting the model’s likelihood to correctly complete some task. A well-calibrated uncertainty measure should be low when the model is highly likely to answer correctly and it should be high when the model’s likelihood to answer correctly is low or near random chance. We measure calibration on the common mulitple choice question answering benchmark, MMLU [43]. Note that, unlike the Roper and Pew datasets, each MMLU question always has a constant four available answer choices.

5.1 Methods↩︎

Similar to alignment evaluation, each of the MMLU questions are presented to the model using the same prompt template in Figure 2. For each question, the full token probability distribution and cloze test results are recorded. Analysis is split into two phases. In the first phase, a simple measure of calibration is obtained by taking the Spearman correlation between the binary correctness of the cloze test result and the candidate uncertainty measures. Because results can vary wildly within a single model across the various question subjects, we separate by subject during analysis.

5.1.1 Jensen-Shannon Distance Shift↩︎

We provide a more nuanced analysis of the calibration using the shift in Jensen-Shannon distance. The Jensen-Shannon distance (JSD) is a symmetric and finitely-valued extension of the Kullback-Leibler divergence. It is defined as \(JSD(P ||Q) = \sqrt{\frac{1}{2}D(P||M)+\frac{1}{2}D(Q||M)}\), where \(D\) is the K-L divergence function and \(M\) is a mixture distribution of \(P\) and \(Q\). As JSD is a metric, and thus obeys the triangle inequality, we can use it to directly compare the relative distance between two separate pairs of probability distributions over a shared outcome space [44]. That is, if \(JSD(P,P^\prime) < JSD(Q, Q^\prime)\) and if \(P\), \(P^\prime\), \(Q\), and \(Q^\prime\) represent probability distributions over an identical outcome space, this indicates that \(P\) and \(P^\prime\) are more similar to each other than \(Q\) is to \(Q^\prime\). This metric thus provides a similar, but more nuanced, view of the model accuracy, but allows for robust hypothesis testing. Our hypothesis is that a well-calibrated measure should show a significant change in JSD from the correct distribution when the model goes from certain to uncertain.

We leverage the JSD to examine how closely the distribution of answers given by the model matches the distribution of correct answers at high and low entropy. For each uncertainty measure, we standardize the measured certainties to have a mean of \(0\) and standard deviation of \(1\). We assign the questions to high or low certainty by whether the standardized uncertainty is above or below \(0\). Questions with a standardized uncertainty of \(0\) were randomly assigned. From this assignment, we obtain four distributions, \(H_A\), \(H_M\), \(L_A\), and \(L_M\). \(H_A\) is the count of each correct answer choice for the questions with high certainty, while \(H_M\) is the distribution of answers given by the model for those same questions. \(L_A\) and \(L_M\) are defined similarly. Finally, we use a permutation test with random uncertainty level assignments to test whether \(JSD(H_M,H_A) > JSD(L_M, L_A)\) to a significant degree. We dub the resulting calibration metric, \(JSD(H_M,H_A) - JSD(L_M, L_A)\), as the JSD shift. We run the permutation test with \(1000\) permutation iterations.

5.2 Results↩︎

The results of the simple correlation analysis is shown as a heatmap in Figure 4. The ideal measure would appear as bright blue (negative correlation) for all models and subjects. Unlike the alignment case, there is a clear winner in the choice entropy. Across nearly all models and subjects, choice entropy shows mild to moderate correlation with correctness. The primary exceptions are LLaMa 1B, which shows low correlation on all subjects, and a small collection of subjects that show no correlation for any model. In both cases, these are likely indicative of poor performance due to underpowered model or excessive question difficulty.

Outside of choice entropy, all models, with the notable exception of Mistral 0.1 7B Instruct, show negative correlation in most subjects, in particular for top-\(k\) entropy. Qualitatively, it appears that there is a non-linear relationship between calibration and size of \(k\). The calibration appears to peak at \(k=10\) and degrades as \(k\) increases. Mistral 0.1 7B Instruct is noteworthy in showing weak positive correlation for all measures except choice entropy.

The results of the permutation test on the JSD shift test for the LLaMa 3.1 8B Instruct model is depicted in Figure 5, with the JSD shift and significance for every model-measure pair listed in Table 2. This figure is typical of all models except Mistral 0.1 7B Instruct. Each dotted vertical line is one uncertainty measure’s JSD shift. For all models, including Mistral 0.1 7B Instruct, all measures show clear significance, with at most one measure per model visually intersecting the JSD shift distribution for random assignment. As is apparent in Figure 4 and reinforced in Table 2, Mistral 0.1 7B Instruct shows significant but negative JSD shift for nearly every measure, suggesting that the model itself is unusually negatively calibrated. Permutation graphs for all models can be found in the Appendix.

6 Conclusion↩︎

In this work, we found strong initial evidence that many entropy-based uncertainty measures are well-aligned to human uncertainty. This is in spite evidence that the human groups and models rarely agree, both in chosen answer and in answer preference ordering. We identify nine uncertainty measures that show especially strong correlation with human group uncertainty: choice entropy, top-k entropy (for numerous values of k), and top-p entropy (for high values of p). We found that these candidate measures show weak but statistically significant calibration on the MMLU benchmark.

6.0.1 Future Work↩︎

Future work should seek to find measures that are more highly calibrated without sacrificing alignment. Additional fruitful lines of research could include extending the current research into open-ended contexts—including measures similar and dissimilar to the proposed framework below—and investigating whether uncertainty-aware applications based in human-aligned measures show benefits to user experience or task efficacy by directly conducting human studies. Our work also did not investigate a direct relationship between uncertainty and human-LLM answer agreement, in particular whether answer agreement correlates with uncertainty level. Future work should investigate this relationship to a finer degree. Finally, future work should seek to measure uncertainty alignment at the individual level as well as at the human group level.

The most important area of future work is the introduction of highly calibrated and aligned measures—with top-k 10 being the most promising—into agentic and cooperative software to provide users with an intuitive understanding of model confidence. We hope that such a method will increase trust and improve outcomes in human-LLM cooperation contexts.

6.0.2 A Proposed Conceptual Framework for Open Ended Questions↩︎

This paper investigated and found human aligned uncertainty measures in a multiple choice context. However, the measures and approach used here can be extended to open answer generation contexts. Consider a language model asked an open ended question. The model generates a response, \(R\). The model context including \(R\) is then extended with a prompt to the effect, "Evaluate if the answer given is a true, false, or neutral response to the the question". This prompt converts the open ended question response uncertainty problem into a multiple-choice, 3 answer choice question—similar to the 3.77 answers per question on average in the dataset used to establish alignment. The measures evaluated in this paper are then directly applicable with little inferential cost as the precomputed token values do not need to be re-calculated as is necessary in other UQ measure approaches discussed in the related work section. This framework is planned to be evaluated in future work.

7 Limitations↩︎

The most significant limitation to this study is that the experiments were limited to only multiple-choice question contexts. Further research is needed to determine whether the results herein will persist in more open ended contexts. Our work is also limited to a selection of widely used open-weight models with 8 billion or fewer parameters. We are prevented by available compute resources from extending our experiments to larger models, but our results do not show any apparent size dependence for the most aligned and calibrated measures.

While we did not find a model size dependence in top-k measures, this should be evaluated in significantly larger models to determine the relevance to consumer grade large language models.

8 Technical Details↩︎

All experiments were performed using the computing resources at Tennessee Technological University. The high-performance computer was used, leveraging one A100 GPU with 40GB of VRAM. The code was developed in a Python 3 Jupyter environment using the huggingface toolbox [45].

Experiments were performed on both the completion and instruct fine-tuned versions of the following models: LLaMa-3.2 1B and 3B [46], LLaMa 3.1 8B [47], and Mistral 7B versions 0.1 and 0.3 [48]. All experimentation and analysis source code has been released under the MIT license and is publicly available at https://github.com/KyleAMoore/LLM-UQ-Align-and-Calibrate.

The Pew [39] and Roper [49] datasets are copyrighted and were accessed through the official portals. Therefore, we are unable to independently release the dataset. Our Roper data collection process can be replicated using details included in the appendix.

9 Human Comparison Dataset Construction↩︎

The Roper dataset is comprised of \(2998\) questions obtained from the Roper Center for Public Opinion Research iPoll database [49]. These questions were sampled uniformly at random from an initial set of \(30571\) questions pulled from the database. The exact search criteria used to obtain these questions can be found in Figure 6. The initial set was comprised of all results of this search.

Before sampling, the questions were processed for validity and removal of non-response answer choices. Many questions included non-response answer choices for respondants that had no opinion or preferred to not answer the given question for any reason. We decided to remove these, given that the goal of this research is to investigate the behavior of the model when making a decision, not how likely it is to refuse to make a decision. The latter is a well-established behavior [50], and likely affected by uncertainty, but is outside the scope of this work. We also elected to not remove "None of the Above" and similar answers that indicate an active rejection of the given answers rather than a passive lack of preference. This is consistent with the questions pulled from Roper iPoll, many of which included both rejection and refusal options. Refusal answer choices were found using manual inspection of the dataset, identifying the following refusal options:

• don’t know/refused

• don’t know/skipped

• don’t know/skippedrefused

• no answer

• not selected

• not selected/no answer

• not sure/refused

• not sure/skipped

• omit

• refused

• refused/web blank

• skip

• skipped

• skipped on web

• skipped/refused

• skipped/web blank

• web blank

After refusal filtering, remaining human answer choice ratios were re-normalized to sum to \(100\%\). Questions with fewer than two answer choices after removal of refusal options, totaling \(332\) questions, were then removed. Finally, questions for which the total human response ratio sum was invalid were removed from consideration, totaling \(572\) questions. Valid answer ratio sum values are those that fall into the range (\(100 - N\), \(100 + N\)), where \(N\) is the number of answer choice options post-refusal-removal, representing the maximum divergence from \(100\) given worst-case rounding errors. The resulting set of \(29667\) questions were then sampled for \(3000\) questions uniformly at random without replacement. Two questions were found after experimentation time to have anomalous choice counts, likely due to those questions having more than \(26\) available options, which was not accounted for in the initial processing. These were removed and excluded from all analysis, leaving the reported \(2998\) questions. Due to licensing constraints, the final set of questions is not released in the supplementary materials.

10 Additional Figures↩︎

References↩︎