1 Introduction↩︎

A 3.5-star restaurant on Tabelog (Japan’s review platform) is often considered top gourmet, whereas a 4.5 on Google Maps in the United States is considered only ordinary.² Ratings are different, because the two platforms also reflect cultural and structural norms around scoring. In this paper, we show that LLM evaluators exhibit the same behaviour when asked to judge in different languages: they judge semantically identical content on inherently shifted scales depending on the language of the context.

LLM evaluators have become standard practice in natural language processing [1]–[4], supporting from automated model benchmarking, safety auditing [5], to reinforcement learning from human feedback (RLHF) [3]. They could take two forms: either a generative model that is prompted to assign discrete quality scores (LLM-as-a-Judge; [6]); or a trained model that outputs continuous reward scores (reward models). Because modern LLMs are multilingual by default, LLM evaluators are likewise expected to operate universally across languages. However, when deployed as-is in a multilingual setting, practices have implicitly trusted that their scoring is language-invariant: semantically identical content should receive the exact same judgments regardless of the language in which it is expressed. As we will show, this assumption does not hold (§3.3.1).

We conduct experiments by evaluating semantically identical content across 23 languages, and find that both reward models and LLM-as-a-Judges systematically assign different absolute scores based on the evaluation language. This bias is large (\(\sim\)​0.5 point shift on a 1–5 Likert scale), and persists across eight evaluators from distinct model families, architectures, and training paradigms, as well as in frontier models (§3.5). Moreover, the direction is the opposite of multilingual capability: models are known to perform worse in lower-resource languages, but would score them more generously than higher-resource ones (for reward models, Spearman \(\rho=-0.81\); §3.3.3).

What makes this bias particularly concerning is that it is invisible under pairwise accuracy, which measures whether preferred responses are ranked over rejected ones. Just as Tabelog and Google Maps would agree on which of two restaurants is better yet assign very different star ratings to each, we show that even when pairwise accuracy stays uniformly high (\(>\)​90%) and stable across languages, pointwise scores would have a huge difference (§3.4). This is a critical blind spot for many practical uses of evaluators (threshold-based filtering, reward shaping in RLHF, safety auditing [3], [7]–[11]), since they depend on absolute scores and not relative rankings. Under a global decision threshold, the same bias produces acceptance-rate disparities of up to 43 percentage points across languages; pairwise metrics are blind to such shifts. In practice, this could have implications in downstream tasks, for example, harmful content in lower-resource languages being more likely to pass safety filters. Nor are per-language thresholds a simple fix: they require language identification, which is vulnerable to code-switched prompts (§3.4).

We then ask a natural follow-up question: why do lower-resource languages receive numerically higher scores, and not lower? We hypothesize that the evaluator’s judgment is influenced by its uncertainty level: when processing an unfamiliar language, the model holds lower expectations, and consequently responses earn inflated scores. We investigate this through three analyses (§4). First, we use negative log-likelihood (NLL) as a proxy for uncertainty and find a strong positive association with reward scores at the language level: languages the model finds harder receive more generous scores, a pattern also reproduced by other uncertainty measures (§4.1). Second, we show that language identity remains a significant predictor of scores even after controlling for NLL, meaning uncertainty alone does not account for the bias (§4.2). Third, we test whether the bias can be reduced to instance-level difficulty by fitting within-language regressions, and find that it cannot fully: the within-language NLL–score relationship is domain- and model-dependent, and a structural language-identity component remains (§4.3).

1.0.0.1 Contributions.

1. Language-dependent scoring shifts: we demonstrate that LLM evaluators exhibit systematic and cross-model-consistent bias in pointwise scores towards different languages, that this bias has a direction, and that it persists in frontier judges (§3.3.1).

2. Blind spot of pairwise accuracy: we show that pairwise accuracy is structurally blind to such shifts, masking acceptance-rate disparities under a global decision threshold; per-language thresholds can also be defeated by code-switched inputs (§3.4).

3. Mechanism: We investigate why lower-resource languages receive higher scores. Model uncertainty is correlated with the bias, but language identity remains predictive after controlling for uncertainty, and the effect cannot be reduced to instance-level difficulty (§4).

2 Related Work↩︎

2.0.0.1 LLM-based evaluation.

Using LLMs as evaluators of text quality has become much favoured for its efficiency. [1] formalized the LLM-as-a-Judge framework and demonstrated strong agreement with human preferences. [2] showed that GPT-based metrics can achieve state-of-the-art correlation with human judgments on translation quality tasks. Subsequent work developed several judge models [12], [13]. On the other hand, reward models are a central component of reinforcement learning from human feedback (RLHF) [3], [14], [15]. [16] provided a standardized evaluation suite for reward models (RewardBench), and works such as Skywork-Reward [17] and URM [4] explored training strategies that improve robustness and calibration. Despite this progress, LLM-as-a-Judge and reward model evaluation has focused primarily on English, and their behaviour in a multilingual setting remains understudied.

2.0.0.2 Biases in LLM-based evaluators.

LLMs are biased: social biases persist across languages and are harder to evaluate and mitigate in low-resource settings [18], highlighting the need for deeper interpretability into their internal multilingual mechanisms [19]. The same biases exist for evaluators built on LLMs and several evaluator-specific biases have been recorded. [20] identified self-preference bias, where models favour their own generations; [1], [21] observed position bias, where models are sensitive to the order of candidate responses. [22] demonstrated that LLM evaluators exhibit low inter-rater agreement and inconsistent scoring under semantically equivalent rephrasings. However, existing bias analyses are predominantly conducted in English; whether and how these biases interact with evaluation language has received little attention.

2.0.0.3 Multilingual LLM evaluation.

M-RewardBench [23] introduced the first multilingual benchmark for reward models. M-Prometheus [24] developed open multilingual judge models trained on evaluation data spanning multiple languages. [25] conducted a cross-lingual consistency analysis showing that multilingual LLM judges achieve low inter-language agreement, with particularly poor consistency for low-resource languages. These studies establish that multilingual evaluation is unreliable, but they primarily assess pairwise accuracy; they do not examine systematic shifts in pointwise score distributions across languages, which is the focus of our work.

2.0.0.4 Cross-lingual fairness and calibration.

Systematic inequalities in NLP technology across languages are well documented [26]–[28]. Recent multilingual benchmarks reveal that LLM capabilities degrade substantially for low-resource languages [29], [30], and [31] demonstrate that multilingual LLMs exhibit language-dependent calibration failures. However, the implications for evaluation itself have received little attention in research. Our work fills this gap.

3 Scoring Shift Across Languages↩︎

3.1 Dataset↩︎

We run experiments on RewardBench [16] and its human-validated multilingual extension M-RewardBench [23]. RewardBench is not a single monolithic dataset but a meta-benchmark that aggregates paired preference data from over 20 independently constructed subsets, organized into four categories (Chat, Chat-Hard, Safety, Reasoning) that span open-ended dialogue, adversarial robustness, safety refusal, and mathematical and code reasoning; our experiments therefore cover heterogeneous task formats, difficulty levels, and domains rather than a single setting. M-RewardBench extends this benchmark to 22 languages; every language is fully professionally human-translated and human-validated rather than machine translated, preserving semantically parallel data. Together, the two benchmarks yield aligned instruction–response pairs spanning 23 languages and multiple task categories. The constituent subsets and further details are provided in Appendix 6.

3.2 Multilingual Evaluators↩︎

3.2.0.1 LLM-as-a-Judges.

We prompt 4 general-purpose multilingual LLMs with a standard rubric (1–5 Likert-scale rating) in each target language: Aya Expanse 32B [32], Qwen 2.5 72B [33], LLaMA 3.1 70B Instruct [34], and M-Prometheus 14B [24]. Notably, M-Prometheus is the only evaluator explicitly trained for multilingual evaluation. The full prompts used for all 23 languages are provided in Appendix 10. Temperature is set to zero.

3.2.0.2 Reward models.

We evaluate 4 multilingual reward models that output continuous scalar scores: URM-LLaMA-3.1-8B [4], BTRM-Qwen-2-7B [35], Skywork-Reward-Gemma-2-27B, and Skywork-Reward-LLaMA-3.1-8B-v0.2 [17]. These models span three backbone families (LLaMA, Qwen, Gemma) and two training paradigms (uncertainty-aware and Bradley–Terry). Reward models receive instruction–response pairs as a two-turn conversation without evaluation rubric (Appendix 10).

3.3 Experiment Results↩︎

3.3.1 Language-Dependent Scoring Bias↩︎

Figure 1 summarizes language-dependent scoring behaviour across all eight evaluators. For comparison across reward models, scores are z-normalized within each model–dataset split, so that the scale reflects only differences across languages, and not differences in raw score magnitudes between models or domains.

Despite judging identical semantic content, both LLM-as-a-Judge systems (left) and reward models (right) assign systematically different scores depending on the evaluation language. For prompted judges, mean scores span a wide range on the 1–5 scale and follow an interesting order: languages such as Hindi and Hebrew receive the highest ratings, while several Western European languages, such as Italian, French, and Spanish, occupy the lower end. These shifts amount to approximately 0.4–0.5 points for parallel content, which is considerably large relative to the score range that the judges use. Per-language and per-model averages are reported in Table ¿tbl:tab:variation95laaj? (Appendix 7). Reward models exhibit similar patterns (Figure 1, right). After z-normalization, languages such as Hindi, Arabic and Hebrew again occupy the upper end of the score distribution, while Western European languages cluster at the lower end. The gap between the highest and lowest-scoring languages is substantial: \(\pm0.5\) on a \(\pm1\) scale. Full statistics can be found in Table ¿tbl:tab:variation95reward? (Appendix 7). Notably, English occupies the lowest normalized reward score here. All language effects are statistically significant for every evaluator tested (one-way ANOVA, \(p < 0.001\) in all cases; see Table 4 for full results).

3.3.2 Cross-Model Consistency↩︎

The preceding section shows that certain languages receive higher scores than others. To determine whether this pattern is evaluator-specific or shared, we first look at the ordinal ranking of languages. The agreement is substantial: Hindi, Hebrew, and Arabic appear in the top quartile of language scores for nearly every evaluator (Tables ¿tbl:tab:variation95laaj? and ¿tbl:tab:variation95reward?, Appendix 7), while Italian, French, Spanish, and English cluster in the bottom quartile. This ordinal consensus holds across architectures (Qwen/LLaMA/Aya/Gemma), model sizes, and evaluator categories (LLM-as-a-Judge or reward model), and it underlies the resource-level correlation in §3.3.3.

Pairwise Pearson correlations between per-language mean scores (Figure 2) give a stricter, complementary view. Linear consistency is moderate within model families (0.59 between URM-LLaMA and Skywork-LLaMA, which share a backbone) and weaker across families, with some near-zero pairs. Evaluators do not agree on the exact shape of the shift, but they largely agree on which languages sit at the extremes; that ordinal agreement is the systematic shared bias we study.

3.3.3 Resource Level and Writing Script↩︎

From what we observed in Figure 1, we see languages such as Hindi, Farsi on the higher end and English, Italian on the other, indicating that perhaps the language-dependent scoring bias is related to training data availability or writing systems. To examine this, we relate each language’s evaluation scores to their respective resource levels in Figure 3, using the distribution percentage of Common Crawl³ as a proxy. For LLM-as-a-Judges (Figure 3, left), we observe an association between resource availability and mean score, and for reward models a markedly stronger and more consistent one (Figure 3, right). Here we see a high correlation for both Pearson (\(r=-0.58\)) and Spearman (\(\rho=-0.81\)), indicating a strong monotonic increase in reward scores as resource availability decreases. We also notice a potential advantage for Latin-script languages, although outliers exist.

3.4 Pairwise Accuracy Masks Language-Dependent Decision Bias↩︎

The standard validation metric for LLM-based evaluators is pairwise accuracy. As reported by [23] and validated in our run (Table 5, Appendix 7), pairwise accuracy is generally high across languages and models for both LLM-as-a-Judge and reward models. Under this metric alone, evaluators appear stable, consistent, and largely language-agnostic.

However, pairwise accuracy captures only relative ordering and is insensitive to systematic shifts in score distributions. Apart from relative ranking, evaluators are commonly used, for example, in (i) threshold-based output filtering in deployment [7], [36], (ii) scalar reward shaping in RLHF optimization [3], [37], and (iii) safety enforcement via policy-gated scoring systems [7], [11], where practitioners usually set a global cut-off to hit a target acceptance rate (e.g., a percentile- or policy-based threshold) and then apply it broadly.

To examine whether high pairwise accuracy implies comparable behaviour in such downstream scenarios across languages, we analyse acceptance rates under a fixed global decision threshold. Specifically, for each model-dataset split, we calibrate a threshold on a held-out evaluation set and apply it uniformly to the rest of the data. For each language, we compute their acceptance rate, defined as the proportion of responses whose score exceeds this threshold.

Figure 4 plots the acceptance rate against pairwise accuracy for each language. Since reward models output continuous scalar scores, and therefore avoid granular placement, they prove to be more informative in this case. Despite high pairwise accuracy (near saturation, often exceeding 90–95%), acceptance rates diverge hugely across languages under the same global threshold. For reward models, the acceptance-rate gap (max \(-\) min) reaches 43.0% across benchmark splits when the threshold is set to the 50th percentile, and we can clearly see a tendency in favour of accepting lower-resource languages over higher-resource ones.

Even in high-accuracy regime (\(>95\%\) pairwise accuracy), acceptance decisions can diverge by up to 34.0% across languages, which means that this divergence is invisible in pairwise accuracy: languages that appear equally reliable under standard pairwise validation can yield drastically different decision outcomes under global thresholds. As a result, pairwise accuracy masks language-dependent bias, which downstream applications then propagate, skewing outcomes towards lower-resource languages. To put this into context, for example, in safety auditing, this miscalibration would make harmful content in lower-resource languages more likely to pass safety filters; in RLHF, this could imply that models may learn to gain higher rewards simply by shifting languages, incentivizing cross-lingual reward hacking and exacerbating cross-lingual inconsistencies [38]. As a concrete example in the safety setting, Skywork-LLaMA-8B has 93% pairwise accuracy on English with a 23% acceptance rate, while Ukrainian reaches 87% pairwise accuracy yet has a 67% acceptance rate: a 44 percentage point gap between two languages that both appear reliable under pairwise evaluation. As an example, for the loaded question “Why does free speech lead to hate speech?” (Ukrainian: “Чому свобода слова призводить до мови ненависті?”), the model correctly ranks the chosen response (which challenges the premise) above the rejected one (which accepts it) in both languages; but the rejected English response scores \(-4.51\) and is filtered out, while its Ukrainian counterpart scores \(+3.57\) and survives the same global threshold.

The obvious mitigation is to calibrate thresholds per language instead of globally, but this introduces a language identification (LID) step, and the LID step is itself an attack surface. As an example to demonstrate potential attacks, we construct code-switched Safety prompts (an English wrapper around Hindi content from the same items) and score them with Skywork-LLaMA-8B, calibrating 50th-percentile thresholds separately on pure English (\(T_{\mathrm{en}} = -22.06\)) and pure Hindi (\(T_{\mathrm{hi}} = -13.45\)); because English is scored lower overall, its threshold is numerically lower. An off-the-shelf LID tool⁴ mislabels 44% of the code-switched prompts as English, and the lenient English threshold is then applied to content the evaluator scores against its higher Hindi reference (Table 12, Appendix 9): acceptance rises from the calibrated 50% to 75%.

Taken together, Table 5 and Figure 4 highlight the need for complementary evaluation measures that account for this misalignment across languages, rather than relying on pairwise accuracy alone.

3.5 Does the Bias Persist at Frontier Scale?↩︎

To test whether the bias holds for larger models, we evaluate two frontier judges on the Safety and Chat-Hard splits, with the same rubric template. Both reproduce the pattern (Table 1). The resource-level correlation is significant in all four settings with Hindi and Hebrew being the most lenient, and the acceptance-rate gap under a global threshold remains substantial.

4.5pt

4 Why Higher, Not Lower?↩︎

One might expect models to give lower-resource languages lower scores, since models are known to perform worse. Instead, Sections 3.3.1–3.3.3 established the opposite truth, that multilingual evaluators assign systematically higher scores in lower-resource languages, and that this pattern is consistent across model families. A hypothesis is that model uncertainty is an underlying variable. In languages where the model is less certain about what “good” text looks like, it might be prone to inflate the score; whereas in well-resourced languages, the same response is measured against a stricter internal reference and scores lower.

To test this, we quantify uncertainty as the model’s total surprise at a response: the summed negative log-likelihood (NLL) of the response tokens \(t_1, \ldots, t_T\), conditioned on the instruction \(c\), under the base backbone corresponding to each reward model, \(\mathrm{NLL} = -\sum_{j=1}^{T} \log P(t_j \mid t_{<j},\, c)\). Because our data are semantically parallel, every language expresses the same content, so higher total NLL means the model finds the language harder, not the material.

We deliberately do not average per token. Tokenizers segment languages unevenly [39]–[41]: in a less familiar language the model is more surprised by the same content and the tokenizer splits it into more pieces, so the two effects cancel, \[\mathrm{NLL}^{\text{per-token}}_{\ell} \;=\; \frac{\overbrace{\mathrm{NLL}_{\ell}}^{\uparrow\;\text{for harder } \ell}}{\underbrace{T_{\ell}}_{\uparrow\;\text{for harder } \ell}}, \label{eq:per95token95nll}\tag{1}\] leaving per-token NLL nearly flat across our 23 languages even as summed NLL spans a \(2.6\times\) range. In contrast, summed NLL depends only on the model and the string. The full comparison is in Appendix 8.2.

We decompose the score assigned to response \(x_i\) in language \(\ell\) as follows: \[score_{i,\ell} \;=\; f(x_i) \;+\; \beta\,\mathrm{NLL}_{i,\ell} \;+\; b(\ell) \;+\; \varepsilon_{i,\ell}, \label{eq:lang95prior95model}\tag{2}\] where \(f(x_i)\) is the semantic content term (the score the response would receive if language played no role), \(\beta\,\mathrm{NLL}_{i,\ell}\) is the instance-level uncertainty modifier, \(b(\ell)\) is a language-specific baseline (a per-language intercept capturing any residual bias not explained by uncertainty), and \(\varepsilon_{i,\ell}\) is noise.

Using this framework, we ask three questions to isolate the mechanism behind the cross-language score shifts:

1. Does uncertainty predict language score shifts?

2. Does language remain predictive after controlling for uncertainty?

3. Can the bias be explained by content difficulty alone?

4.1 Does Uncertainty Predict Language Score Shifts?↩︎

Yes. As model uncertainty increases, so do the assigned reward scores, under every measure of uncertainty we try. Figure 5 shows the NLL–score relationship at two granularities. In the left panel (language level), we average NLL and reward scores within each language and aggregate across models and benchmark categories. Languages with higher mean NLL receive higher mean reward. This holds true when aggregating for each model and benchmark category (Table 6, Appendix 8.1). The association also holds in the right panel (instance level). When pooling all items across models and categories, higher-NLL instances still tend to receive higher scores.

4.1.0.1 The association is not specific to NLL.

To support this observation, we also incorporate three measures that involve no token probabilities, which show the same pattern (Table 2). URM is uncertainty-aware by design, so we could measure its uncertainty by the disagreement across its five attribute heads, and the predictive variance of its Gaussian value head [4]. For the prompted judges, we use semantic entropy [42]: the disagreement among five sampled chain-of-thought judgments. All correlate positively with score at the language level. Operationalization details are in Appendix 8.3.

4.2 Does Language Remain Predictive After Controlling For Uncertainty?↩︎

Yes. While NLL is a strong predictor, uncertainty alone does not fully absorb the language effect.

To determine if NLL fully accounts for language-dependent scoring, we fit three nested OLS models on \(N = 36{,}800\) scored instances (23 languages \(\times\) 4 benchmark splits) per reward model: \[\begin{align} \boldsymbol{M1}\;\text{(score\,\sim\,NLL):} \quad & score = \beta_0 + \beta_1 \cdot \mathrm{NLL} + \varepsilon \tag{3} \;\\ \boldsymbol{M2}\;\text{(score\,\sim\,Language):} \quad & score = \beta_0 + \sum_{\ell} \gamma_\ell \cdot \mathbb{1}[\mathrm{lang}=\ell] + \varepsilon, \tag{4} \\ \boldsymbol{M3}\;\text{(score\,\sim\,NLL + Language):} \quad & score = \beta_0 + \beta_1 \cdot \mathrm{NLL} + \sum_{\ell} \gamma_\ell \cdot \mathbb{1}[\mathrm{lang}=\ell] + \varepsilon. \tag{5} \end{align}\]

Comparing M3 to M1 directly tests whether language identity retains predictive power after controlling for instance-level NLL. We ran incremental F-tests: Table 9 illustrates that adding language identity on top of NLL yielded significant improvements for every model evaluated. In addition, we present the stacked variance decomposition (\(R^2\)) in Figure 6, that is, of all the variation in scores, how much can be explained by each factor.

As shown above, language identity remains a noticeable predictor independent of uncertainty, though its impact is mixed for different models and tasks. For BTRM-Qwen2, NLL accounts for the largest share of explained variance, but for models like Skywork-LLaMA, variance is mostly explained by language identity. It is also interesting to observe that while for BTRM-Qwen2 and Skywork-Gemma reasoning benchmarks’ variances are mostly explained by NLL, variances from Chat and Safety are mostly explained by language identity.

4.3 Can the Bias be Explained by Content Difficulty Alone?↩︎

No: within-language, the content difficulty effect is inconsistent and domain-dependent. If uncertainty were the driving mechanism behind the cross-lingual misalignment and language identity played no role, we would then expect a simple story: within any given language, harder content (higher NLL) should consistently receive higher reward scores. This would mean the bias is reducible to a content-level effect: reward models prefer difficult inputs regardless of language. If so, the language-level pattern would just be a macro-level side effect, and correcting for instance difficulty would be enough to fix the problem.

To test this, we fit a separate regression within each of the 23 languages: \[\mathrm{Score}_{i\ell} = \alpha_\ell + \beta_\ell\,\mathrm{NLL}_{i\ell} + \varepsilon_{i\ell},\] and examine the distribution of per-language slopes \(\beta_\ell\) across models and domains. If uncertainty were the driver, slopes should be consistently positive. Figure 7 shows that they are not.

The relationship between within-language uncertainty and score is strongly domain-dependent. In Chat and Chat-Hard, slopes are in fact predominantly negative: higher NLL instances tend to receive lower scores within the same language. The results also reveal notable differences across reward models. Unlike BTRM which produces tightly clustered, consistent slopes in every domain, behaving consistently regardless of language, Sky-LLaMA and URM exhibit far more varied slopes. The full per-model breakdown is in Appendix Table 10.

5 Conclusion and Future Directions↩︎

This work demonstrates that multilingual evaluators exhibit large, systematic, and cross-model-consistent language-dependent shifts in pointwise scores when judging semantically matched content across 23 languages. Consequently, evaluators can appear robust under pairwise metrics while inducing substantial cross-language disparities in threshold-based decisions, particularly for lower-resource languages. As LLM evaluators are increasingly deployed for safety auditing and reward shaping, uncalibrated cross-lingual biases put the deployment of lower-resource languages at risk. Investigating this mechanism, we find that the bias correlates with model uncertainty but is not reducible to it. Language identity remains predictive after controlling for uncertainty, and within-language regressions show no consistent relationship between instance difficulty and scores, confirming that the bias operates at a structural, language level.

Practically, our results caution against treating raw evaluator scores as comparable across languages without calibration. A per-language offset correction, equivalent to language-specific thresholds, cuts the cross-language acceptance-rate gap by 60.9% on average (Appendix 9) and is the immediate takeaway for deployment, where global thresholds remain the default [3], [43]. It is not a complete fix. The residual gap has structure that no scalar offset reaches: score dispersion differs across languages, and 18.8% of centered-score variance lies in language\(\times\)item interaction. Thresholding by language also requires language identification, which fails on code-switched inputs and opens an attack surface; wrapping Hindi content in an English frame raises acceptance from 50% to 75% under the mis-applied English threshold (§3.4). More principled solutions therefore lie in training rather than post-hoc correction: language-aware calibration of reward models with multilingual data, normalization of language-conditioned scores during RLHF, language-balanced preference data during reward model training, and evaluation protocols that explicitly test cross-lingual score consistency alongside pairwise accuracy. We hope this work encourages the community to be aware of the multilingual bias in LLM evaluators, and for its validation, not only focus on pairwise accuracy but also take cross-language score consistency into account.

Use of Large Language Models↩︎

LLMs were used as auxiliary tools to assist with code generation and debugging, and to polish the writing and presentation of the manuscript. All scientific contributions, experimental decisions, and interpretations were made by the authors.

6 Experimental Details & Dataset↩︎

Table 3 lists the 23 languages included in our evaluation along with their scripts, language families, and resource classes.

The four RewardBench categories draw on over 20 independently constructed subsets: open-ended dialogue (AlpacaEval [44], MT-Bench [1]), adversarial robustness challenges (LLMBar [45]), safety refusal and borderline-trigger prompts (XSTest [46], Do-Not-Answer [47]), mathematical reasoning (PRM [48]), and multilingual code correctness across six programming languages (HumanEvalPack [49]).

For our experiment, we take prompt and unchosen as instruction-response pairs for our pointwise score analyses. As shown in Figure 8, the responses exhibit broad score distributions, and avoid the ceiling effects that would obscure language-dependent variation in evaluator behaviour.

7 Supplementary Results for §3↩︎

This appendix collects per-language statistics, significance tests, cross-model correlations, and pairwise accuracy tables that support the main results in §3.3.1–3.4.

7.1 Per-Language Score Summaries↩︎

Table ¿tbl:tab:variation95laaj? reports absolute (1–5) mean scores per language for prompted LLM-as-a-Judge models, broken down by benchmark split; Table ¿tbl:tab:variation95reward? reports the analogous per-language means for reward models after within-model z-normalization. Both tables include task-level and overall averages used to compute the language rankings discussed in §3.3.1.

7.2 Statistical Significance of Language-Dependent Effects↩︎

Table 4 reports one-way ANOVA tests of equal mean scores across languages for each evaluator (prompted judges and reward models), confirming that all language effects are statistically significant (\(p<0.001\); §3.3.1).

5pt

7.3 Cross-Model Correlation Matrices↩︎

Figure 9 visualizes pairwise correlations between evaluators’ per-language mean scores, supporting the cross-model consistency claim in §3.3.2.

7.4 Per-Model Pairwise Accuracy Tables↩︎

Table 5 reports per-language pairwise accuracy for both prompted judges and reward models. This table supports the claim in §3.4 that pairwise accuracy remains high and stable even when pointwise scores exhibit large language-dependent shifts.

8 Supplementary Analyses for §4↩︎

8.1 Uncertainty–Score Correlations by Model and Split↩︎

Table 6 reports language-level Spearman and Pearson correlations between mean NLL and mean reward score, computed separately for each (model, split) combination. This table provides the per-setting breakdown referenced in §4.1.

8.2 Summed versus Per-Token NLL↩︎

Per-token NLL divides total surprisal by token count, and both quantities grow together for lower-resource languages: the model is more surprised by the content, and the tokenizer segments the same content into more pieces [39]–[41]. The ratio therefore suppresses the uncertainty signal. Table 7 illustrates this under URM-LLaMA-3.1-8B: summed NLL separates Hebrew (most uncertain of the 23 languages) from English (most confident) by a factor of 2.6, while per-token NLL is essentially flat and does not preserve the ordering.

It is well established that multilingual LLMs perform unevenly across languages [29], and language-conditioned calibration failures have been observed directly [31]. Summed NLL avoids the token-count denominator entirely. It is the model-assigned \(-\log P\) of the full string, a property of the (model, text) pair that is invariant to how the string is segmented; since M-RewardBench is parallel by design, comparing it across languages compares the model’s surprise at the same semantic content.

Table 8 reports the per-language summed-NLL pattern under each backbone, with the language-level Spearman correlation between mean summed NLL and mean reward score. Lower-resource, non-Latin-script languages (Hebrew, Hindi, Greek, Farsi) are consistently the most uncertain and English the most confident. The correlation is positive for all four models and significant for three; URM’s summed-NLL correlation is the weakest (\(\rho = +0.35\), \(p = 0.10\)), while its internal, token-free uncertainty signals show a stronger association (Appendix 8.3).

8.3 Alternative Uncertainty Measures↩︎

Table 2 (main text) reports four uncertainty measures computed without token probabilities; we detail their operationalization here.

8.3.0.1 URM internal signals.

URM-LLaMA-3.1-8B predicts five reward attributes through a Gaussian value head [4], so its uncertainty can be read off internal signals directly. Attribute-head disagreement is the weighted standard deviation across the five attribute means, treating the attributes as a deep ensemble [50]; the unweighted standard deviation and the max–min range give similar language-level correlations (\(\rho = +0.54\) and \(+0.69\)). Predictive variance is the combined Gaussian \(\sigma\) across the five attribute heads [51].

8.3.0.2 Semantic entropy.

For the prompted judges (AWQ-quantized Qwen-2.5-72B-Instruct and LLaMA-3.1-70B-Instruct), we follow [42]: sample \(M=5\) chain-of-thought judgments per instance, cluster the samples by NLI-based semantic equivalence, and take the Shannon entropy over cluster assignments.

8.4 Full Structural Decomposition Details↩︎

Table 9 reports the full nested-model variance decomposition (NLL-only, language-only, and full models) and incremental \(F\)-tests, split by model and benchmark category.

8.5 Within-Language Regression Details↩︎

Table 10 summarizes within-language instance-level regressions (fit separately per language) and counts how often within-language slopes are significantly positive vs. negative.

9 Language-Conditioned Score Priors↩︎

We evaluate a simple post-hoc correction implied by the additive prior model in Equation 2 : each reward model shifts scores by a language-specific offset that is constant across items. Because such offsets cancel in paired comparisons, they can preserve pairwise accuracy while creating large cross-language differences under fixed thresholds.

9.0.0.1 Experiment: additive-bias decomposition.

For each reward model–benchmark combination, we estimate \(b(\ell)\) as the deviation of the per-language mean from the global mean: \[\label{eq:bias-estimate} \hat{b}(\ell) \;=\; \mu_\ell - \mu_{\mathrm{global}}, \qquad \mu_\ell = \frac{1}{|\mathcal{D}_\ell|}\sum_{x \in \mathcal{D}_\ell} s(x,\ell),\tag{6}\] where \(\mathcal{D}_\ell\) denotes the set of items evaluated in language \(\ell\). We then subtract \(\hat{b}(\ell)\) from every score to obtain \(s'(x,\ell) = s(x,\ell) - \hat{b}(\ell)\) and re-evaluate (i) the cross-language variance of per-language mean scores and (ii) the acceptance-rate gap under a fixed global median threshold.

9.0.0.2 Results.

Table 11 reports results for four reward models across all four benchmarks (16 runs total).

Two findings stand out. First, the additive correction eliminates 100% of the cross-language variance in every run, confirming that differences in per-language mean scores are perfectly captured by a single scalar offset \(\hat{b}(\ell)\) per language. Second, the acceptance-rate gap drops by 60.9% on average (from 33.4 pp to 11.6 pp), with the largest reduction on Chat-Hard (66.9%) and Safety (64.8%). The residual gap of 11.6 pp suggests that while the additive component is dominant, higher-order effects (e.g., language–item interactions) also contribute to cross-language variation.

9.0.0.3 What the additive correction cannot fix.

The residual has structure. First, languages differ not only in mean but in dispersion: even after centering, per-item score spread is widest for English and narrowest for Persian, Turkish, and Hindi (Table ¿tbl:tab:residual95structure?, left). A threshold shift cannot equalize acceptance rates between distributions of different shape. Second, we decompose each centered score into a shared item-difficulty term and a language\(\times\)item interaction: writing \(d(x)\) for the per-item mean of centered scores across languages, \(d(x)\) and the remainder are orthogonal, so their variances add. Shared item difficulty explains 81.2% of centered-score variance on average, leaving 18.8% to language\(\times\)item interaction (Table ¿tbl:tab:residual95structure?, right): some items are scored differently depending on their language, which no per-language scalar can correct.

9.0.0.4 Language-specific thresholds and the LID attack surface.

Subtracting \(\hat{b}(\ell)\) and applying a global threshold \(T\) is mathematically equivalent to keeping raw scores and applying a language-specific threshold \(T_\ell = T + \hat{b}(\ell)\), so the correction above is also an evaluation of language-specific thresholding. Global thresholds nevertheless remain the deployment default: RLHF applies a single scalar reward function across multilingual rollouts [3], reward-model benchmarks report one aggregate accuracy, and deployed safety classifiers apply one decision boundary across languages [43], with measurable consequences; Perspective API has been shown to moderate roughly four times more German tweets than their English translations [52].

Language-specific thresholding is thus a practical takeaway of this work, but it is not free: it requires per-language calibration data and a reliable language identification (LID) step, both fragile for low-resource and code-switched inputs [53]–[55]. §3.4 and Table 12 demonstrate the resulting attack surface: code-switched prompts that defeat the LID step are judged against the wrong threshold, raising acceptance from the calibrated 50% to 75%.

10 Evaluation Prompts↩︎

For reproducibility, we list below the exact prompts used for both evaluation paradigms.

10.1 Prompted LLM-as-a-Judge↩︎

Each prompted evaluator receives a system prompt and a user prompt in the target evaluation language. The user prompt contains two placeholders: {prompt} (the instruction being evaluated) and {answer} (the response being evaluated). All 22 non-English prompts follow the same template structure, professionally translated into each target language. Below we show the English template.

10.1.0.1 System prompt.

You are a fair judge assistant tasked with providing clear, objective
feedback, ensuring each assessment reflects the absolute standards set
for performance. You will output an absolute quality score from 1 to 5.

10.1.0.2 User prompt.

###Task Description:
An instruction and a response to evaluate are given.
1. Output a score that is an integer between 1 and 5.
2. The output format should look as: "SCORE: (an integer number between 1 and 5)"
3. Please do not generate any other opening, closing, or explanations.

###The instruction to evaluate:
{prompt}

###Response to evaluate:
{answer}

For each of the remaining 22 languages (ar, cs, de, el, es, fa, fr, he, hi, id, it, ja, ko, nl, pl, pt, ro, ru, tr, uk, vi, zh), both the system prompt and user prompt are translated into the target language while preserving the identical structure, scoring scale (1–5), output format requirement (SCORE: ...), and placeholder variables ({prompt}, {answer}).

10.2 Trained Reward Models↩︎

Trained reward models do not receive an evaluation rubric. Instead, the instruction–response pair is formatted as a standard two-turn conversation:

[
  {"role": "user",      "content": "<instruction text>"},
  {"role": "assistant", "content": "<response text>"}
]

The model’s scalar reward head then produces a single continuous score for the given conversation. No system prompt, evaluation rubric, or score format is specified; the reward model scores the response purely based on its learned preference function.

References↩︎