1 Introduction↩︎

Large language models (LLMs) have seen unprecedented adoption in recent years [1], [2], including in language communities besides English [3]. Despite both public awareness and academic research on their limitations, hallucination, the generation of plausible but factually incorrect statements (see e.g. [4] for an overview), still pose an ongoing problem that risks downstream harms and the loss of user trust. One potential solution to this problem is given by confidence estimation (CE), i.e. obtaining numerical values that reflect the probability that a model output is correct. CE has long been studied in machine learning [5]–[9] and has received increasing attention for LLMs [10]–[15]. However, the vast majority of works focus exclusively on English, leaving other languages largely unexplored. This is particularly concerning because models may behave less reliably in languages with fewer training data, making accurate confidence estimation even more important [16]–[18].

In this work, we take a first step toward truly multilingual confidence estimation in LLMs. Our approach is motivated by a key observation: prior work on multilinguality has shown that LLM hidden states capture cross-lingual features [19]–[22], while e.g. [23], [24] have identified uncertainty-related features. We hypothesize that these properties jointly enable cross-lingual transfer of confidence estimates. To test this, we apply a lightweight linear probe that learns a weighted combination of hidden layers to predict answer correctness. We show that the probe, trained on an LLM’s latent representations in a single language, can predict confidence scores for unseen, typologically diverse languages without any target-language supervision (zero-shot): 1 illustrates how correct and incorrect responses are distributed similarly in the learned aggregation of the target LLM’s latent space. The probe can exploit this fact as shown by its predictive distribution on the right, with some degradation as the zero-shot language becomes more dissimilar.

Our contributions are as follows: To the best of our knowledge, we are the first to demonstrate zero-shot cross-lingual transfer of confidence estimation through probing of intermediate LLM representations. We analyze how performance varies across unseen languages of the same and different language families and writing systems, and show that the probe outperforms baselines from several CE method families without retraining on the target language. Through ablations, we show that the most salient information for zero-shot confidence prediction is localized in the upper-middle layers, precisely where prior work has shown LLMs map inputs into a shared representational space. More broadly, our findings suggest a possible path toward multilingual calibration with little or no target-language supervision, where confidence signals learned in higher-resource languages transfer to lower-resource settings. All code is open-source.¹

2 Related Work↩︎

We position our work at the intersection of studies examining the latent space of multilingual LLMs, the probing of model representations to identify whether specific properties are encoded, and the wider landscape of confidence estimation in LLMs.

2.0.0.1 Multilingual Latent Spaces.

Several studies have examined the organization of LLMs’ latent space, finding that layers closest to the input and output are more language-specific. However, there is no consensus on whether multilingual processing involves mapping to a particular language or through shared, language-agnostic regions in the representation space. Some works support the language dominance hypothesis, according to which inputs are mapped into a representation space shaped by the most prevalent training language (usually English), perform computations in that space, and translate the result back into the original language [20], [22], [25]–[27]. [26] show that in the middle layers reason in English while leveraging multilingual knowledge, while [27] identify two types of neurons, namely ones responsible mapping from a language-specific space to a shared semantic space in the early and middle layers, and those operating in reverse in the upper layers. [22] find that semantically meaningful words pass through an English-centric representation space, and [28] extend these findings to writing systems, suggesting that middle layers represent tokens in Latin script. By contrast, [29], [30] indicate that multilinguality is facilitated by shared language-agnostic representations, identifying parameters essential to cross-lingual performance. [31]–[34] discover language-specific neurons primarily in the lower and upper layers, with a gap in the middle layers. [21] further show that the output language is encoded in earlier layers, whereas the concept to be translated emerges in later layers. [35] find no consistently dominant language in the middle layers.

2.0.0.2 Probing Latent Representations.

Probes are lightweight models that map a model’s internal representations to a property of interest [36], [37], and are widely used in mechanistic interpretability to test whether a property is encoded in LLM representations [38], [39]. In multilingual LLM research, probes have been used to study morphosyntax [19], [40], factual knowledge [41], semantics [19], [41], and context [42], with shared features typically emerging in the middle layers. Relevant to our work, probes have been trained on LLM representations in English to detect truthfulness [23], [43]–[45], with [41] finding stronger signals in high-resource than in low-resource languages. While [23], [43] find that truthfulness signals peak in the middle layers, [46] show that more complex properties require deeper layers.

2.0.0.3 Confidence Estimation in LLMs.

As mentioned in the introduction, confidence estimation (CE), i.e. obtaining a score that reflects the probability of a model’s answer to be correct, has recently produced a flurry of new works (see e.g. [47]–[52]). Since some methods are based on the generation process, e.g. through token logits or additional responses, they are immediately applicable to any arbitrary language (although without performance guarantees). Other methods however are trained on a specific language, with unseen languages resulting in a zero-shot prediction problem; this is the setting in which probing approaches fall, which have also been applied to CE [11], [23], [53]–[58]. Importantly however, few works explicitly test performance on non-English and especially lower-resourced languages: [18] show how the calibration of verbalized confidence scores can vary greatly between different languages, with [59] showing the same for confidence based on sequence likelihood. [60] create a multilingual CE benchmark, consisting of questions that are both language-agnostic and specific to languages (and cultures), again demonstrating stark performance differences between methods. More related to our approach, [61] find that the most salient features for multilingual CE lie in the middle layers. However, they restrict themselves to multiple choice question-answering, and do not consider zero-shot prediction.

3 Methodology↩︎

Based on the insights from the literature, we now discuss which representations to consider, verify our assumptions, and define the probe.

3.1 Selected Intermediate Representations↩︎

Our work focuses on decoder-only transformer-based LLMs [62], [63]. Given an input token sequence, the model embeds the token at every time step \(t\) into a high-dimensional space \(\mathbf{x}_t \in \mathbb{R}^D\) and produces a hidden state \(\mathbf{h}_t^{(\ell)}\) by running sequentially through each block of transformer block at layer \(\ell\), before finally generating sequence \(y_1, y_2, ..., y_T\). Each block updates the previous hidden states in a residual way: \[\label{eq:residual-stream} \mathbf{h}_t^{(\ell)} = \mathbf{h}_t^{(\ell-1)} + g_\ell\Big(\mathbf{h}_1^{(\ell-1)}, \dots, \mathbf{h}_{t-1}^{(\ell-1)}\Big),\tag{1}\] where \(g_\ell\) is the model-specific transformation of the \(\ell\)-th block applied to the prefix and current token. This also referred to as the residual stream, serving as a central pathway that carries and incrementally refines each token’s representation. Due to the insights from e.g. [61], we exploit signals from all layers, but focus on two specific time steps: The prompt’s last token (last input token), and the generated answer’s last token (last answer token) hidden states. The former serves as a proxy for the entire input, integrating information from all preceding tokens and capturing the model’s state at the start of generation, and has been widely used in multilingual interpretability and uncertainty studies [20], [21], [30], [40], [41], [44], [64], [65]. The latter represent the model’s final state after generation and may additionally encode the LLM’s confidence in its completed response, and has been shown in other contexts to encode knowledge [66] or truthfulness [67].

3.2 A Cross-lingual Confidence Probe↩︎

Prior work has identified considerable overlap of latent representations for different language representations in the middle layers [19], [20], [29], [61]. We confirm this in our case as well, showing a selection of PCA plots in 2. Since that implies that some layers will carry more generalizable information, we define a probe that learns a weighted average of layer hidden representations \(\mathbf{h}_t^{(\ell)}\) by introducing a learnable parameter \(\mathbf{w} \in \mathbb{R}^L\), using it to compute layer weights \(\alpha_l\) and pooled representation \(\bar{\mathbf{h}}_t \in \mathbb{R}^D\):

\[\label{eq:layer95imp} \alpha_\ell = \mathrm{softmax}(\mathbf{w})_\ell = \frac{\exp(w_\ell)}{\sum_{k=1}^L \exp(w_k)},\tag{2}\] \[\label{eq:repr95pooling} \bar{\mathbf{h}}_t = \sum_{\ell=1}^L \alpha_\ell \, \mathbf{h}_{t}^{(\ell)}.\tag{3}\] This representation allows the probe to select the most informative layers for confidence estimation during training, assuming approximate per-dimension consistency as a side effect of 1 as information propagates through layers. Finally, we use a linear layer with learnable parameters \(\mathbf{v} \in \mathbb{R}^D, b \in \mathbb{R}\) and the sigmoid function \(\sigma(\cdot)\) to obtain a confidence score \(\hat{p}_t\): \[\label{eq:logit} \hat{p}_t = \sigma\big(\mathbf{v}^\top \mathbf{\bar{h}}_t + b\big).\tag{4}\]

While non-linear designs of the probe are possible, evidence from the linear representation hypothesis [38] and other studies [24], [68]–[70], suggests that features are often represented linearly in latent space.

4 Experiments↩︎

4.1 Setup↩︎

4.1.0.1 Datasets.

We base our experiments on two popular question-answering datasets, aiming for parallel data for comparability. Firstly, we use MKQA [71], consisting of open-ended questions across 26 languages, with the original English questions extracted from Natural Questions [72]. We consider short-phrase, entity, numeric, numeric-with-unit, and date question types and exclude questions that are time-sensitive. We considered at most 5237 questions per language, with further details of the selection process in 8.1.1. Secondly, we consider Global-MMLU [73], a parallel multilingual multiple-choice dataset covering 42 languages, derived from the English MMLU dataset [74]. Since multiple-choice evaluation is discriminative and not representative of real-world generative use cases [75], we retain only questions that can be asked in an open-ended manner, using a process we describe in detail in 8.1.2. This results in a final set of 4215 questions per language.

4.1.0.2 Selected Languages.

To capture cross-lingual variation, we select the six languages shown in 1: English (en) is included as it dominates most LLMs’ training data. We further include French (fr) and Spanish (es), two closely related languages that belong to the Romance language genus [76] and also use the Latin script. Polish (pl) and Russian (ru) are chosen to represent the more dissimilar Slavic languages [76], with Russian additionally introducing the Cyrillic script. Finally, Japanese (ja) is different in both linguistic family and script. For our experiments, we choose French over English as a source language: This enables us to test on Spanish as near out-of-distribution and English as a typologically more dissimilar language.²

4.1.0.3 Models.

We test our approach on two strong contemporary autoregressive LLMs that have enjoyed lots of use in academic research and in industry alike, namely Llama 3.1 8B Instruct [77] and Qwen3 8B [78], which is also instruction-finetuned. The models feature 32 and 36 layers, respectively, and have demonstrated considerable multilingual capabilities. We generate answers with a temperature of \(0.7\) for a maximum of \(40\) tokens. All language and dataset prompts are detailed in 15 in 8.2. All experiments where performed on a single A100-80G-PCIe NVIDIA GPU.³

4.1.0.4 Answer Correctness.

We judge answer correctness through using GPT-4.1 mini [79] as a judge, using prompt given in 16 in 8.2, obtaining the accuracies listed in 2. Two things are notable: Accuracies differ between languages due to difference in factual consistency [80], [81], and performance is strongest in English, with a degradation of performance roughly in accordance with the typological distance to it.

4.1.0.5 Evaluation Metrics.

We evaluate confidence estimates using commonly used metrics, including expected calibration error (ECE; [6], [7]) using \(10\) bins, and Brier score [82], measuring how well confidence scores correspond to the probability of correctness (calibration). We also measure how well confidence scores distinguish between correct and incorrect responses (discrimination) by recording the area under the receiver operating characteristic (AUROC) and the precision-recall curve (AUPR).

4.2 Zero-shot Confidence Estimation Results↩︎

We first evaluate the zero-shot CE performance for the probe in isolation. We include a majority class baselines (majority), which in our case refers to the negative class, and a prior-probability baseline, which samples binary correctness labels from a Bernoulli distribution, with its probability \(p\) corresponding to the training accuracy. We also train another probe on only data from the target language, which we will refer to as the oracle probe.

4pt

4.2.0.1 Results.

We show detailed results for Qwen 3 8B in 3 and visualize the overall metric trends in 3, with the complementary results for Llama 3.1 8B given in 7.1. 3 shows that the majority and prior probability baseline can be well-calibrated due to ECE not being a proper scoring rule [83], but fail at discriminating correct from incorrect predictions. Expectedly, the probe performs well when tested on its training language French. Unsurprisingly, some metrics like AUROC and AUPR tend to degrade as the probe is tested on more dissimilar languages, as exemplified by the difference to the oracle. Interestingly though, we can still see that other metrics like Brier score and ECE often stay close to or even outperform the oracle in some instances. This indicates that multilingual transfer of features is sometimes not only possible, but also beneficial, although some major performance drops are observed in specific instances. These trends become especially visible in 3, where we observe that while discrimination degrades, calibration appears to be mostly unaffected. This implies that probes are not so much affected by the covariate shift in the multilingual latent space, but the concept shift in the conditional label distribution [84], as the accuracy of the target LLM decreases for languages such as Russian or Japanese. Lastly, it appears that the last query token seems to yield more useful model representations for calibration, the last answer hidden states appear to produce better discrimination. This makes sense, as the model latents will not have encoded any information about the actual generated answer at the end of the query.

4.3 Comparison To Other Estimators↩︎

We also compare our probe to other common CE methods,⁴ namely length-normalized sequence likelihood (Seq. Likelihood); P(True) [11], which asks whether a response was true or false and considers the probability of the “true” token, verbalized confidence (Verbalized Conf.; [85]–[87]), which asks for confidence from \(0\)-\(100\) in the target language; as well as the mass-mean probe by [88], which is not trained but directly derived from the centroids of correct and incorrect predictions.⁵ Prompts for P(True) and verbalized confidence are shown in 8.2. Note that besides the mean-mass probe, no other baseline is similarly zero-shot in the sense of having an explicit distributional shift between training and test, but are intuitive choices for confidence estimation on unseen languages.

{#fig:[ width=“80%”}

4.3.0.1 Results.

[tab:baseline-comparison-qwen3_8b] list results of the last token answer hidden states for selected languages, with all results in 7.2. We can see that compared to the other confidence estimators, the cross-lingual probe holds up reasonably on AUROC, but degrades on calibration as typological distance to the source language (fr) grows. However, in some cases, such as Japanese on MKQA, it is outperformed by the mass-mean probe, which often is better calibrated but usually worse in discrimination. The fact that both probes are able to identify generalizable information in latents confirms our hypothesis of language-agnostic confidence signals. It also suggests that calibration can likely profit from further investigation into carefully choosing the right probe training data and architecture and potentially post-hoc re-calibration through techniques such as temperature scaling [7] or isotonic regression [89].

4.4 Understanding Layer Importance↩︎

To understand what makes the probe work, we can first look at the learned layer weights in 4, with results for Llama 3.1 8B found in 7.3. While the weight distribution appears more uniform for last query token latents, the probe overall seems to learn to assign higher weights to upper-middle and top layers. To investigate whether these weights causally affect performance, we apply the interventional framework by [90]: We assess the importance of layer \(\ell\) by measuring the performance gap on a metric \(\mathcal{P}\) between a model \(\mathcal{M}\) and a variant \(\mathcal{M}^{\setminus \ell}\), where only the component \(\ell\) is changed:

\[\label{eq:delta95ablation} \Delta(\mathcal{M}, \ell) := \mathcal{P}(\mathcal{M}) - \mathcal{P}(\mathcal{M} ^{\setminus \ell}),\tag{5}\] where positive values of \(\Delta\) indicate that the component \(\ell\) contributes to a higher metric score. We apply this framework in two ways: Firstly by setting specific weights \(\alpha_\ell = 0\) and renormalizing the distribution (weight ablation) and secondly by replacing representations from a specific layer with a dummy (representation ablation). ⁶ Further details on these ablations are described in 7.4.

4.4.0.1 Results.

We show a selection of results in 5, with the rest given in 7.4. We can observe that removing middle layers leads to a less performant probe, with only the ablation of later layers yielding positive results in some cases, likely due to the probe learning language-specific information about the source that is not useful in a zero-shot setting. Measuring Spearman’s \(\rho\) between layer weights and ablation impact per metric in [fig:spear-qwen-answer], we can see correlation between layer weights and performance metrics that is preserved in zero-shot settings. The effect seems less consistent on calibration. When plotting the layer ablation impact separately on few-shot vs. zero-shot separately in 6, we can identify that middle-layer ablation has a stronger effect in zero-shot settings on calibration, while the effect on discrimination seems the remain the same in both settings. Lastly, we demonstrate in 7.5 that while learnable layer weights reveal the most salient language-agnostic signals for zero-shot confidence estimation, the probe can also learn to extract necessary information directly from the latents, likely due to the information redundancy in transformer layers visualized in 2. However, comparisons against learned weights yield inconsistent results.

5 Discussion↩︎

In our work, we demonstrated that language-general spaces in multilingual LLMs found by prior work (e.g. [20], [22], [26]) carry useful signal for confidence estimation. While [61] show that the best representations for multilingual confidence estimation lie indeed in middle layers, something that our ablations in 4.4 strongly confirm, we are the first to demonstrate that this information can be used zero-shot on unseen languages. In 4.2, we observed a gentle degradation of discrimination for confidence estimates as the similarity to the source language decreases, but overall stable calibration across most conditions. Another finding that stands out is that the time step from which hidden representations are extracted matters: While the last token of the input, often used in related probing works (e.g. [20], [21], [30], [40]) was better calibrated on other languages, this was not true for the source language itself; indeed, the lesser-used last token of the answer [45], [66] also achieved better discrimination. This could be explained by considering that the query latent states might encode information about question difficulty, while the last answer states provide definitive information about the actual answer given.

6 Conclusion↩︎

In this work, we presented the first evidence that language-agnostic latent spaces in multilingual LLMs contain correctness signals that can be exploited to perform CE on unseen languages. While performance degrades on languages dissimilar from the training language, we nevertheless demonstrated promising results in this challenging setting. Interestingly, we also found learned layer weights to mostly attend to language-general layers identified by prior work. It remains especially interesting to explore how to make this approach more robust to unseen language families and scripts by training on multiple source languages, identifying the most critical latent representations, and applying common post-hoc calibration techniques. Overall, we believe that these findings open up exciting new research directions towards truly multilingual confidence estimation with little or no supervision, especially for underserved languages.

Limitations↩︎

We acknowledge the following limitations: Firstly, our work focuses on open-source models of 8 billion parameters; nevertheless we expect our approach to still be applicable, as evidence suggests that larger models also encode multilingual / language-agnostic information that can be used for correctness predictions in their hidden representations [92], [93].

Secondly, while our language selection considers languages from different language families and script, it still mostly focuses on European languages and languages that are all relatively high-resource, with zero-shot performance likely degrading for languages that are less related, use a different script, or are less presented in the LLM’s training data, based on observed dissimilarity of latent representations for more dissimilar languages [94], [95].

Thirdly, we focus on short-form QA, whereas recent works also have started to consider reasoning [96]–[100] and long-form uncertainty quantification [101]–[104]. While our approach might still be applicable in these cases, longer generations probably require a more careful selection of the time step of hidden representation.

Additionally, linear probes may be insufficient in some cases, as self-attention and feed-forward blocks can substantially rotate and mix representation dimensions [62]. Furthermore, linearly combining layer representations may fail to capture complex cross-layer interactions that are relevant for uncertainty estimation.

Ethical Consideration↩︎

We foresee no immediate ethical issues with this work.

Acknowledgments↩︎

This work is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053) and the HELLENiQ ENERGY’s "Proud of Youth" Scholarship. Computational resources were provided by the Cirrus UK National Tier-2 HPC Service at EPCC (http://www.cirrus.ac.uk), funded by the University of Edinburgh and EPSRC (EP/P020267/1). We thank Rochelle Choenni for her help in conceptualizing this project.

7 Additional Results↩︎

7.1 Confidence Estimation Results↩︎

4pt

4pt

4pt

This section contains complete results for the confidence estimation results in 4.2. 4 shows the results with French as the source language for Llama 3.1 5 6 gives both sets of results, but with English instead of French used as the source language.

7.2 Comparison to Other Estimators↩︎

4pt

4pt

4pt

4pt

This section contains the full results for the comparison with other confidence estimators in 4.3. This includes first the full results for all languages, including the few-shot setting in French, in 7 8 for Qwen 3 and Llama 3.1 respectively. In 4.3 we chose to compare other estimators to our probe trained on last answer token hidden representations, since other methods also, implicitly or explicitly, have access to full generation of the model. To also benchmark the probe trained on the last input token hidden representations, we provide 9 8.

7.3 Additional Layer Weight Plots↩︎

The corresponding layer weight plots for 4 in 4.4 for Llama 3.1 are given in 8.

7.4 Additional Ablation Results↩︎

In this section we provide more information on the ablation methodology in 4.4, and include additional results.

7.4.0.1 Weight Ablation.

We selectively make layers unavailable to the probe and quantify the impact of this change on the uncertainty estimation task using a sliding window approach. Recall the computation of layer weights in 2 . To ablate a window \(W\) of \(k\) contiguous layers, we intervene on the probe’s layer-mixing logits by setting \(\{w_\ell : \ell \in W\}\) to \(-\infty\):

\[\alpha_\ell = \mathrm{softmax}(\mathbf{w'})_\ell = \frac{\exp(w_\ell')}{\sum_{k=1}^L \exp(w_k')},\] where \[w_\ell' = \begin{cases} -\infty, & \ell \in W \\[6pt] w_\ell, & \ell \notin W. \end{cases}\]

7.4.0.2 Representation Ablation.

Recall that the probe computes a logit as \(z_i = \sum_{\ell=1}^L \alpha_\ell s_\ell + b\) (3 4 ), where \(s_{i, \ell} = \mathbf{v}^\top \mathbf{h}_{i}^{(\ell)}\) denoted here. To ablate a window \(W\) of \(k\) contiguous layers, for each example \(i\), we switch its score \(s_{i,\ell}\) with the one of another example \(s_{j,\ell}\), selected by independently sampling it from the test set. This is done to remove useful information for specific test sample, but to not introduce corrupting information that might unfairly distort probe performance. The logit of an example \(i\) after ablation of window \(W\) is therefore:

\[z_i^{(W)} \;=\; \sum_{\ell \notin W} \alpha_\ell\, s_{i,\ell} \;+\; \sum_{\ell \in W} \alpha_\ell\, \tilde{s}_{i,\ell} \;+\; b,\] with \[s_{i,\ell} = \mathbf{v}^\top \mathbf{h}_{i}^{(\ell)}, \quad \tilde{s}_{i,\ell} = {s}_{\pi(i),\ell},\] where \(\pi(\cdot)\) a permutation function defined on the natural numbers up to the cardinality of the test set, mapping each index to another index sampling uniformly without replacement and independently for each layer.

7.4.0.3 Results.

Due to the sheer amount of ablation results, we only focus on a selection of additional results here, namely the corresponding plot to 5 using the representation ablation in 9, and complementary results for Llama 3.1 in 10.

7.5 Uniform Probe Weights↩︎

We conduct additional experiments fixing the layer weights in 2 to uniform and only learning the linear layer parameters. We show the performance differences for both model in 11 12. The comparison remains overall inconsistent, but we attribute the fact that the uniform probe is able to achieve compare results to two factors: First of all, the informational redundancy in transformer layers helps to preserve information in the pooled representation even when weights are uniform, and additionally the large amount of parameters in the linear layer (\(4096\) given the size of hidden representations) helps to learn useful features for the task. Nevertheless, we see our results and works such [61] as evidence that future, more robust works will likely rely on specific layers.

8 Reproducibility Details↩︎

8.1 Data Pre-Processing↩︎

In the following, we describe the exact preprocessing used for MKQA and Global-MMLU. The final number of examples per dataset, model and language can be found in 11.

8.1.1 MKQA↩︎

The MKQA dataset [71] (GitHub repository) comprises 10000 questions parallel across 26 languages, each annotated with one or more answer types (binary, date, entity, long answer, number, number with unit, short phrase, and unanswerable). We retain only questions of type date, entity, number, number with unit, and short phrase. Binary questions are excluded due to limited token generation, while long-answer and unanswerable questions are excluded because no ground-truth answers are provided in the dataset. Duplicate questions are also removed.

We further exclude questions with time-sensitive ground-truth answers that are no longer reliable. To identify such cases, we employ an LLM-as-a-judge approach using GPT-4.1 with the prompt shown in Figure 13. We use default parameters, except for a temperature of 0 and a maximum of 5 generated tokens. Filtering is performed on English questions and then propagated to other languages. As a result, the final dataset contains 5237 questions per language (date: 943, entity: 3347, number: 258, number with unit: 254, short phrase: 435).

None

8.1.2 Global-MMLU↩︎

For our experiments, we use the test split of the Global-MMLU dataset [73].⁷ The dataset spans 42 languages with 14042 multiple-choice questions across 57 subjects. Following [75], we identify and retain only those questions that are sufficiently self-contained to be posed in an open-ended format, using the following procedure applied to the English questions.

We first apply a RegEx-based filtering step to remove questions that are inherently tied to the multiple-choice format. Specifically, we remove questions that (1) contain keywords indicative of multiple-choice framing, such as "following", "which of these combinations", or "options", (2) contain fill-in-the-blank placeholders (e.g., three or more consecutive underscores), (3) reference "Statement 1" and "Statement 2" simultaneously, as these questions require evaluating paired assertions, or (4) ask the model to identify a true or false statement, as indicated by truth-value keywords ("true", "false", "correct", "incorrect"), unless the question also contains the word "following", in which case it is already caught by criterion (1).

We further use GPT-4.1 to score how well each question can be answered without the multiple-choice options, on a scale from 1 to 10 (see 13 for the full prompt). 12 outlines the score distribution across the questions. We retain only questions with a score of 8 or above, as these are sufficiently self-contained to be posed as open-ended questions. After these preprocessing steps we are left with 4215 questions. For all other languages, we retain the translations corresponding to these questions. Note that with this procedure we do not rephrase any question, ensuring that no additional wording bias is introduced during conversion to open-ended format.

8.1.3 Normalization & Splitting↩︎

Sun et al. [105] showed that in transformer-based LLMs using pre-layer normalization (e.g., Llama), hidden-state variance grows exponentially with depth, hindering deeper-layer training. To address this and ensure that probe weights capture informative signal rather than magnitude differences, we apply per-layer z-score standardization with scikit-learn’s StandardScaler.

For each feature \(x\), we compute the standardized value \(x' = (x - \mu)/\sigma\), where \(\mu\) and \(\sigma\) are the training-set mean and standard deviation, centering features to zero mean and unit variance. The fitted scaler is applied unchanged to the validation and test splits of the training language, and to the test sets of all zero-shot languages. We do not further normalize, as residual magnitude differences may be informative.

Given that both datasets are parallel and class ratios vary by language, we stratify splits to prevent information leakage and ensure balanced evaluation. For each training language, hidden states are divided into train/validation/test sets (0.7:0.15:0.15) while preserving correct/incorrect ratios. Evaluation samples are drawn from the test sets so that these ratios match across all languages.

8.2 LLM Prompting↩︎

The system instructions used to prompt Llama 3.1 8B Instruct across languages are in 13.

8.3 Answer Correctness Assessment↩︎

We use GPT-4.1 to evaluate answer correctness, using default parameters and the prompt shown in Figure 16.

8.4 Confidence Probe Implementation Details↩︎

The layer weights \(\mathbf{w}\) were initialized to zero (uniform weighting across layers), the hidden-dimension weights \(\mathbf{v}\) were sampled from the normal distribution \(\mathcal{N}(0,1)\) (to break symmetry and enable diverse feature weighting), and the bias \(b\) was set to zero (no offset prior to training). Probes were trained with binary cross-entropy loss with logits and \(L_2\) regularization for \(v\): \[\label{eq:loss95function} \begin{align} \mathcal{L}(\mathbf{y}, \mathbf{z}) ={}& -\frac{1}{N} \sum_{i=1}^N \Big[ w_{+} \, y_i \log \sigma(z_i) \\ &\qquad\qquad\quad + (1 - y_i)\log\!\left(1 - \sigma(z_i)\right) \Big]\\ &\qquad\qquad\quad + \lambda \lVert \mathbf{v} \rVert_2^2 \end{align}\tag{6}\] where \(N\) is the number of training examples, \(y_i \in \{0, 1\}\) is the ground-truth correctness label, \(z_i \in \mathbb{R}\) is the logit produced by the probe, and \(\sigma(\cdot)\) is the sigmoid function. The parameter \(w_{+} \in \mathbb{R}^+\) is the correct-class weight to address class imbalance, computed from the ratio of negative to positive examples in the training set.⁸ \(\lambda \in \mathbb{R}^+\) is the \(L_2\) regularisation applied only to \(\mathbf{v}\), with no regularisation applied to \(\mathbf{w}\) or \(b\).

8.5 Hyperparameter Tuning↩︎

We tuned hyperparameters with Ray Tune using the ASHA scheduler for inter-trial early stopping, optimizing the scalar objective val_loss (minimized). Each trial sampled from a log-uniform search space: learning rate \(\texttt{lr} \in [10^{-5}, 10^{-1}]\) and L2 regularization on the neuron weights \(\mathbf{v}\), \(\texttt{l2\_lambda} \in [10^{-4}, 3 \times 10^{-1}]\). The batch size and maximum epochs were fixed at \(4\) and \(1000\), respectively. During each trial, we trained with AdamW using selective weight decay (L2 applied only to \(\mathbf{v}\); none for layer weights \(\mathbf{w}\) or bias terms), and reported the mean training loss and validation loss at every epoch to Ray Tune. Validation loss was computed on a fixed held-out validation split. To address class imbalance, we used BCEWithLogitsLoss with a positive-class weight proportional to the ratio of negative to positive examples in the training set. Trials were stopped early by ASHA (max_t=1000, grace_period=30, reduction_factor=2) when underperforming, while in-trial early stopping with patience \(=20\) halted training if val_loss failed to improve. After \(50\) trials, Ray Tune selected the best configuration according to the minimum validation loss. The selected hyperparameters are reported in Table 14.

References↩︎

---

1. The code repository can be found under https://github.com/AthinaKyriakou/shared-doubt.↩︎

2. Our main results with English as a source are included in 6 5 in 7.1.↩︎

3. Details on probe training are given in 8.4.↩︎

4. We only show results for last answer time step for comparability, see 7.2 for last query time step results.↩︎

5. Compared to [88], we do not use any patching, but apply their probe from their section 5.1 directly to the latents of correct and incorrect generations.↩︎

6. Due to the redundancy and representational similarity in adjacent transformer layers [91], we perform the ablations using a sliding window of size \(3\).↩︎

7. https://huggingface.co/datasets/CohereLabs/Global-MMLU↩︎

8. For more information, see the pos_weight parameter of PyTorch’s BCEWithLogitsLoss class.↩︎