1 Introduction↩︎

Reliable evaluation remains the bottleneck in aligning Large Language Models (LLMs). Human evaluation, while the gold standard, does not scale to the iterative development cycles that modern alignment pipelines demand. The field has therefore converged on the LLM-as-a-Judge paradigm [@zheng2023judging], in which another LLM (typically a frontier model) acts as a referee, scoring outputs along one or more quality attributes. Subsequent work has trained open judges to match this behaviour [@kim2024prometheus] and standardised rubric-based evaluation protocols [@li2023alpacaeval; @dubois2024alpacafarm; @ye2024flask]. A single judge, however, is a single point of statistical failure. The systematic biases its backbone exhibits, e.g., position, verbosity, self-enhancement, sycophancy, and refusal artefacts, are by now well documented [@wang2023large; @panickssery2024llm; @saito2023verbosity; @stureborg2024large]; they propagate uncorrected to every score, and the cost-quality profile of the resulting evaluation is fixed to that of the single model.

A natural remedy is to evaluate by committee. The LLM Jury, instantiated by the Panel of LLM Evaluators (PoLL) of [@verga2024replacing], ensembles smaller, diverse, cheaper backbones and reports the arithmetic mean of their scores as the consensus—sufficient, in their experiments, to match or exceed a single large judge. Related multi-model evaluators include peer-rank discussion [@li2024prd], multi-agent debate [@chan2024chateval], and deeper/wider judge networks [@zhang2024wider]; these vary the panel structure but inherit PoLL’s aggregation rule. PoLL is the optimal aggregator precisely when judge errors are light-tailed and centered on the truth, in which case averaging \(N\) judges contracts the variance at the parametric rate \(1/N\) (Proposition 1, §3.5); 11 (§6.5) shows the clean-baseline parameter-efficiency this delivers empirically.

1.0.0.1 The problem: Byzantine failures, not Gaussian noise.

Real LLM judges fail in ways that are nothing like Gaussian noise. A judge that produces malformed JSON triggers a parser fallback to the all-zeros score, dropping a single observation onto the boundary of the score space. A judge with sycophancy bias rates every response near the maximum, flattening genuine quality differences. A judge that handles one attribute well may catastrophically mis-score another, producing a vector that is plausible per axis yet jointly anomalous. A judge whose parser hallucinates can emit values entirely outside the bounded score scale. These four failure modes—mode collapse, sycophancy, cross-attribute confusion, and heavy-tailed hallucination—are all biased point masses far from the truth, not symmetric perturbations of it, and each occurs in real deployments at non-trivial rates: in our corpus, parser-failure alone reaches \(33\%\) on the smallest judge (Gemma-4B) for HelpSteer 3 multilingual prompts, with mean rates of \(3.4\%\) on HelpSteer 3 and \(0.6\%\) on HelpSteer 2 across the \(13\)-judge panel (2, §3.4).

This is the regime the classical robust-statistics literature [@huber1964robust; @tukey1960survey; @small1990survey; @vardi2000multivariate; @minsker2015geometric; @lugosi2019sub] and Byzantine-robust optimisation literature [@blanchard2017machine; @yin2018byzantine; @elmhamdi2018hidden; @acharya2022robust; @acharya2025geometric] identify as the wrong regime for PoLL-style aggregation. The Huber \({\epsilon}\)-contamination model (Assumption 2) admits all four failure modes as specific instantiations of the contamination distribution \(Q_{i}\) (zeros, inverted, bimodal-random, and cauchy-far; mapped explicitly in §3.4 and evaluated in §6.2–6.4), and a direct calculation (Proposition 2, §3.6) shows that under any positive contamination rate PoLL’s conditional bias grows linearly with the corruption shift and is unbounded over the corruption class, regardless of \(N\): the \(1/N\) variance reduction that motivates juries cannot rescue an aggregator whose bias is itself unbounded.

1.0.0.2 Overview of our approach.

We propose RoPoLL (Robust Panel of LLM-as-Judge), a drop-in replacement for the arithmetic-mean aggregation step of PoLL with a robust mean estimator. Among classical candidates—the coordinate-wise median (CoMed), the trimmed mean, and the geometric median (GM)—only GM is simultaneously tuning-free (no contamination-rate hyperparameter unlike the trimmed mean), joint-distance preserving (operates on Euclidean distance over the full score vector unlike CoMed, which decouples coordinates and misses cross-attribute structure of Example 1), and attains the optimal \(1/2\) breakdown point (Definition 8, Proposition 3); the comparison is developed in detail in §4.1. We instantiate RoPoLL with the geometric median (Definition 7), computed via the modified Weiszfeld iteration (Algorithm 4, §4.3) at \(O(Nd\log(1/{\epsilon}))\) per query. CoMed and the trimmed mean serve as empirical baselines in §6.

1.0.0.3 Contributions.

• Formalisation. We give the first formal treatment of LLM jury aggregation as a robust mean-estimation problem (§3): we model the LLM-as-Judge pipeline as a Markov kernel (Definition 4), define the LLM Jury (Definition 5), and characterise judge failures as Byzantine faults under the Huber contamination model (Assumption 2). Proposition 2 shows that PoLL admits unbounded bias under this model.

• Algorithm and theory. We propose RoPoLL (§4) and establish its theoretical guarantees (§5): a finite-sample upper bound \(\|\hat{{\mathbf{y}}}_{\mathrm{GM}}-{\mathbf{y}}^{\star}\|_{2} \leq C_{\alpha+\beta}\rho\) with explicit absolute constants (Theorem 1), a correlated-jury extension under the equicorrelated-indicator hypothesis (Lemma 3, with empirical indicator-correlation \(\bar\gamma_{W} \in [0.45, 0.53]\) measured on our judge panels, 6.10), and an information-theoretic minimax lower bound (Theorem 2) that matches on the parametric rate \(\sigma\sqrt{d/N}\) and differs on the breakdown floor by a \(\sqrt{d}\) statistical–computational gap, attributed to GM’s polynomial-time tractability relative to the (intractable) Tukey halfspace median.

• Large-scale empirical validation. We evaluate \(13\) open-weight LLM judges spanning four model-size tiers (\(4\) B–\(675\) B parameters) on three benchmarks with complementary ground-truth sources: HelpSteer 2 [@wang2024helpsteer2], HelpSteer 3 [@wang-etal-2025-helpsteer3], and UltraFeedback [@cui2024ultrafeedback]. Under systematic adversarial injection at contamination rates up to \(50\%\) (§6), RoPoLL outperforms PoLL by up to three orders of magnitude on biased heavy-tailed (cauchy-far, 1) and cross-dimensional (bimodal-random, 9) attacks; a \(3\)-judge RoPoLL committee at \(38\) B total parameters beats Mistral-Large-3 (\(675\) B) by \(1.31\times\) on HelpSteer 2 under \(30\%\) bimodal-random corruption (an \(18\times\) parameter advantage, §6.3). A Noisy-GT control (§6.7) rules out the obvious confound that the RoPoLL premium is paid against benign Gaussian imprecision rather than against biased contamination.

• Open release of the judge-output corpus. ² We release the full \(13\)-judge \(\times\) three-benchmark output corpus—approximately \(28\mathrm{K}\) scored \((\text{judge}, \text{sample})\) cells of parsed attribute scores, per-call latencies, and reference labels underlying every figure in §6 (§6.8). To our knowledge this is the first standardised corpus of LLM-jury outputs; follow-up work on judge calibration, alternative aggregators, or new corruption families can be benchmarked against this fixed substrate without re-running the inference cost.

1.0.0.4 Paper organisation.

§3 formalises the problem setup, including the LLM Jury (Definitions 4–5), the Huber contamination model (Assumption 2, with the empirical natural-failure-rate calibration of 2), and the unbounded-bias result for PoLL (Proposition 2). §4 develops the RoPoLL methodology: the choice of geometric median over coordinate-wise median and trimmed mean (§4.1), structural properties of GM (Proposition 3), and the Weiszfeld iteration (Algorithm 4). §5 states the finite-sample upper bound (Theorem 1), its correlated-jury extension (Lemma 3), and the matching minimax lower bound (Theorem 2). §6 presents the benchmark evaluation organised by corruption type; §2 situates the work in the LLM-as-Judge, robust-statistics, and Byzantine-distributed-learning literatures; §7 concludes with scope, limitations, and follow-up directions. The released corpus and its inter-judge correlation structure, including the empirical \(\bar\gamma_{W}\) measurement, are documented in §6. Full proofs and a 2D synthetic visualisation gallery are deferred to Appendices 8–9.1.

2 Related Work↩︎

LLM-as-Judge evaluation and per-judge biases. The LLM-as-Judge paradigm was established by [@zheng2023judging] (MT-Bench, Chatbot Arena), demonstrating that strong models such as GPT-4 can serve as reliable proxies for human annotators. Subsequent work has extended the paradigm along several axes: open-source judges with fine-grained rubrics [@kim2024prometheus]; automated frameworks for instruction-following models [@li2023alpacaeval; @dubois2024alpacafarm]; and skill-level evaluation [@ye2024flask]. A parallel literature documents systematic biases of single judges—position, verbosity, self-enhancement, sycophancy, and prompt-format sensitivity [@wang2023large; @panickssery2024llm; @saito2023verbosity; @stureborg2024large]. These findings motivate the use of diverse judge panels but treat each judge in isolation; no prior work analyzes the aggregation step or its failure modes.

Jury and panel evaluation. [@verga2024replacing] introduced the Panel of LLM Evaluators (PoLL), our direct predecessor: a diverse committee of smaller backbones aggregated by the arithmetic mean. Their work established the practical value of LLM juries but did not analyze robustness; the mean aggregator is used without justification, and no failure modes are considered. [@zhang2024wider] studied how panel width and depth affect evaluation fairness, again without robustness guarantees. The key gap across this literature is the absence of any analysis of catastrophic failure modes or formal robustness properties of the aggregation rule. Our Proposition 2 closes this gap: under any positive contamination rate PoLL [@verga2024replacing] admits unbounded bias regardless of \(N\).

Multi-agent debate and structured aggregation. A distinct family of multi-judge methods produces aggregated judgments through structured interaction rather than independent scoring. [@li2024prd] propose peer-rank discussion among judges, in which each judge sees others’ scores and updates its own; [@chan2024chateval] propose multi-agent debate, in which judges argue over a verdict before consensus. These methods change the joint distribution of \((\hat{{\mathbf{y}}}_{1}, \ldots, \hat{{\mathbf{y}}}_{N})\)—they introduce dependence by design, breaking Assumption 3—and trade independence for deliberation-driven error reduction. Whether they exhibit the same Byzantine-failure mode as PoLL is an open question. The Huber-contamination analysis of this paper does not directly apply to such interactive aggregators, but the corruption-class diagnosis (point masses far from the truth) likely transfers, suggesting robust extensions of debate-based aggregation as a future direction. Majority voting in mathematical reasoning [@cobbe2021training] is a related but coarser ensemble technique on binary correctness; the analogue of Proposition 2 for vote-based aggregation on \(\{0, 1\}\) outputs is the standard \(\alpha < 1/2\) Byzantine threshold.

Calibration as a complementary paradigm. A separate line of work removes judge bias at the source via per-judge calibration on a labeled validation slice [@zheng2023judging]. Calibration assumes a stationary, recoverable bias and trades worst-case guarantees for average-case efficiency; RoPoLL assumes nothing on the corruption distribution and pays a constant-factor insurance premium to bound the worst case. The two are complementary: RoPoLL can aggregate calibrated scores, and the calibration-RoPoLL composition—together with extensions to heterogeneous, correlated, and dependent juries—is left to future work.

Robust statistics and the geometric median. The Huber contamination model [@huber1964robust] and the breakdown point [@tukey1960survey] are the classical framework for estimation under arbitrary corruption. The geometric median attains the optimal \(1/2\) breakdown for any translation-equivariant estimator [@lopuhaa1991breakdown; @small1990survey; @vardi2000multivariate]; in high dimensions, [@minsker2015geometric] established sub-Gaussian concentration for the geometric median of means—the result Theorem 1 adapts to contaminated juries—and [@lugosi2019sub] developed sub-Gaussian mean estimators with optimal dimension dependence. Recent applications to ML pipelines include block-coordinate GM descent for robust training [@acharya2022robust] and GM Matching for robust subset selection [@acharya2025geometric]; [@acharya2025robust] surveys robust learning from noisy data. Our setting differs from this literature on three axes: (i) low dimension (\(d \in \{4, 5\}\) evaluation attributes, so the \(\sqrt{d/N}\) rate is dominated by constants and the \(1/(1-2\alpha)\) contamination factor is the load-bearing dependence); (ii) structured contamination (\(Q_{i}\) arises from specific LLM failure modes—parser fallback, sycophancy, refusals, cross-attribute confusion—which inform the four empirical corruption types in §6); and (iii) heterogeneous workers (per-judge \(\sigma_{i}, \alpha_{i}\) vary across the panel, outside the i.i.d. regime that the classical robust-statistics literature targets). Among broader alternatives in the robust-aggregation toolbox, the half-space (Tukey) median attains the optimal breakdown \(1/2\) in any dimension but is NP-hard to compute and prohibitive at \(d \geq 5\) [@small1990survey]; median of means [@lugosi2019sub] targets heavy-tailed data rather than Huber contamination concentrated in a minority of judges; the geometric median’s tuning-free \(1/2\) breakdown, joint-distance objective, and \(O(Nd\log(1/{\epsilon}))\) cost make it the right default for the small-\(d\), small-\(N\), heterogeneous-worker, one-shot regime that LLM juries occupy. A systematic empirical comparison against the broader family is left to future work (§7).

Byzantine-robust distributed learning. The connection between robust aggregation and Byzantine fault tolerance has been worked out in distributed optimization: Krum [@blanchard2017machine], coordinate-wise median and trimmed mean as gradient aggregators [@yin2018byzantine], and Bulyan [@elmhamdi2018hidden]. This literature targets \(N\) from tens to thousands of workers, with adversarial perturbations composed across thousands of training rounds. The LLM-jury setting shares the mathematical structure but differs operationally on three axes: (a) small \(N\) (juries operate at \(N \in \{3,\dots,13\}\) where every judge is materially expensive, requiring tight finite-sample guarantees); (b) per-sample heterogeneity (the contamination indicator \(Z_{i}\) is conditional on the prompt-response pair \(x\), not per round); (c) no iterative learning loop (LLM-jury aggregation is one-shot at evaluation time, so the per-instance bias bound matters directly rather than its cumulative effect across rounds). These differences explain why our analysis emphasizes finite-sample distribution-free guarantees over the corruption class (Theorem 1); the heterogeneity of the worker pool, judge correlation, and explicit dependence (in debate-based methods) are left to future work and have no direct analogue in the Byzantine distributed-learning literature.

To our knowledge, we are the first to formalize LLM jury aggregation as a robust estimation problem, prove finite-sample contamination guarantees in this setting, and evaluate robustness systematically against both natural and adversarial judge failures at scale.

3 Problem Setup↩︎

We evaluate a system agent \({\mathcal{M}}: {\mathcal{P}}\to {\mathcal{R}}\) that maps prompts to responses. For each evaluation instance \(x = (p, r) \in {\mathcal{X}}\triangleq {\mathcal{P}}\times {\mathcal{R}}\) the goal is to estimate a vector of attribute scores describing how good the response \(r\) is for the prompt \(p\).

3.1 System Agent and Reward Space↩︎

Let \({\mathcal{P}}\) and \({\mathcal{R}}\) denote the spaces of admissible natural-language prompts and responses, and define the instance space \({\mathcal{X}}\triangleq {\mathcal{P}}\times {\mathcal{R}}\). The model under evaluation is the system agent \({\mathcal{M}}: {\mathcal{P}}\to {\mathcal{R}}\), \(p \mapsto {\mathcal{M}}(p)\). Given a prompt \(p\), the realized response is \(r = {\mathcal{M}}(p)\) and the evaluation instance is \((p, r) \in {\mathcal{X}}\). Any stochasticity in the underlying generation procedure is immaterial for the development below, which is carried out conditional on the realized pair \((p, r)\).

Definition 1 does not impose a common structure across the coordinate spaces, and typical instantiations include bounded scalars (\({\mathcal{Y}}^{(k)} = [0, K_{k}]\)), categorical or ordinal labels (\({\mathcal{Y}}^{(k)} = \{c_{1}, \ldots, c_{L}\}\)), and free-form text (\({\mathcal{Y}}^{(k)} = {\mathcal{R}}\)). For ease of exposition we specialize throughout the paper to the homogeneous bounded-scalar setting: there exists \(K > 0\) with \({\mathcal{Y}}^{(k)} = [0, K]\) for every \(k \in [d]\), so \[\label{eq:scalar95reward95space} {\mathcal{Y}}\;=\; [0, K]^{d} \;\subset\; \mathbb{R}^{d}.\tag{1}\]

3.2 Reference Protocol, Rubric, and Parser↩︎

Because \({\mathbf{y}}^{\star}\) is unobservable, evaluation must proceed through an observable reference protocol.

3.3 LLM Jury and Aggregation Function↩︎

The central question of this work is: which \({\mathcal{A}}\) remains accurate when judges fail in arbitrary, possibly adversarial ways? To answer it formally, we adopt the classical contamination model from robust statistics.

3.4 Huber Contamination Model and Companion Assumptions↩︎

3.4.0.1 Concrete instantiations of \(Q_{i}\).

The unrestricted \(Q_{i}\) admits, as special cases, every LLM-judge failure mode reported in the single-judge bias literature: mode collapse (\(Q_{i} = \delta_{{\mathbf{0}}}\), the parser-fallback vector emitted on malformed JSON or safety refusals); sycophancy (\(Q_{i} = \delta_{K \cdot {\mathbf{1}}}\), near-maximum scores assigned indiscriminately, [@wang2023large; @stureborg2024large]); anti-correlated Byzantine attacks (\(Q_{i} = \delta_{K{\mathbf{1}}- {\mathbf{y}}^{\star}}\), mirror-image scores); cross-attribute confusion (\(Q_{i} = \mathrm{Unif}\{0,K\}^{d}\), each coordinate plausible per axis but jointly anomalous, matching the cross-dimensional failure mode of Example 1); and heavy-tailed adversaries (\(Q_{i}\) Cauchy or otherwise unbounded, modelling parser hallucinations of arbitrarily large scores). The four synthetic regimes evaluated in §6 (zeros, inverted, bimodal-random, cauchy-far) instantiate these four \(Q_{i}\) choices respectively; \(\alpha_{i}\) encodes the per-judge unreliability and is expected to decrease with backbone capacity.

3.4.0.2 Empirical grounding.

Naturally-occurring parser failures (the \(Q_{i} = \delta_{{\mathbf{0}}}\) instantiation above) are not hypothetical: across our \(13\)-judge \(\times\) benchmark grid (2), mean failure rates span \(0.59\%\) on HelpSteer 2 and \(3.38\%\) on HelpSteer 3, with the smallest judge (Gemma-4B) failing on \(33\%\) of HS 3 multilingual signed-preference samples. The deployment regime is therefore dataset-dependent across one to two orders of magnitude in \(\alpha\), and the distribution-free contamination class \(\{Q_{i}\}\) is the right object of study.

3.5 Observation Model and Variance Reduction↩︎

Collecting the assumptions above, the complete observation model for a single evaluation instance is \[\label{eq:full95model} \boxed{ \hat{{\mathbf{y}}}_{i} \;=\; (1 - Z_{i})\,({\mathbf{y}}^{\star} + \boldsymbol{\epsilon}_{i}) \;+\; Z_{i}\, \boldsymbol{\eta}_{i}, \qquad i = 1, \ldots, N, }\tag{4}\] where \(Z_{i} \sim \mathrm{Bernoulli}(\alpha_{i})\) are independent latent corruption indicators, \(\boldsymbol{\epsilon}_{i} \sim P_{i} - {\mathbf{y}}^{\star}\) is zero-mean and \(\sigma_{i}^{2}\)-sub-Gaussian (Assumption 4), and \(\boldsymbol{\eta}_{i} \sim Q_{i}\) is the arbitrary corruption noise, independent of \(\boldsymbol{\epsilon}_{i}\) and \(Z_{i}\). The statistician observes only \(\{\hat{{\mathbf{y}}}_{1}, \ldots, \hat{{\mathbf{y}}}_{N}\}\) and has no access to \(\{Z_{i}\}\) or \(\{Q_{i}\}\).

The canonical jury aggregator is the arithmetic mean adopted by PoLL [@verga2024replacing]: \[\label{eq:mean95aggregation} \hat{{\mathbf{y}}}_{\mathrm{mean}} \;\triangleq\; \frac{1}{N}\sum_{i=1}^{N}\hat{{\mathbf{y}}}_{i}.\tag{5}\] On clean juries the mean enjoys the parametric variance-reduction rate.

The proof is a direct application of linearity of expectation and bilinearity of covariance and is given in Appendix 8.1.

A controlled synthetic validation (8 (a) of 8, §5.1) confirms that empirical MSE on an equicorrelated Gaussian jury matches the closed-form prediction \(\tfrac{1 + (N-1)\gamma}{N}\,d\sigma^{2}\) across the full range \(\gamma \in [0, 0.95]\). The corollary has a direct implication for jury design: for any \(\gamma > 0\) the effective jury size \(N_{\mathrm{eff}}\) saturates at \(1/\gamma\), so adding more judges past \(N \approx 1/\gamma\) buys essentially nothing. With \(\gamma\) in the moderate range \([0.3, 0.5]\) characteristic of diverse but non-orthogonal LLM backbones, \(N_{\mathrm{eff}}\) saturates already at \(N \approx 2\)–\(3\), motivating the three-judge committees throughout §6.

3.6 Fragility of PoLL↩︎

The next result shows that the \(1/N\) variance-reduction rate of Proposition 1 is irrelevant the moment any contamination is present.

Proposition 2 is the central impossibility motivating RoPoLL: variance reduction over \(N\) judges is irrelevant when the bias of the aggregator is unbounded over the corruption class. The construction in the proof scales the corruption mean linearly with \(N\), exactly cancelling the \(1/N\) averaging—so increasing the jury size cannot fix the problem. We therefore seek an aggregator that simultaneously (i) matches the \(O(\sigma\sqrt{d/N})\) rate of the mean in the clean case and (ii) has bounded error under arbitrary contamination with \(\alpha < 1/2\). The geometric median, introduced in §4, achieves both.

4 Robust Panel of LLM Judges↩︎

Proposition 2 forces us to abandon the arithmetic mean: under contamination its bias is unbounded over the corruption class regardless of jury size \(N\). We therefore propose RoPoLL, a drop-in replacement for the PoLL aggregation step that swaps the arithmetic mean for a robust mean estimator. The framework is agnostic to the choice of estimator; we instantiate it with the geometric median, motivated below.

4.1 Choosing the Robust Estimator↩︎

Three classical robust mean estimators are natural candidates: the coordinate-wise median (CoMed), the trimmed mean, and the geometric median (GM).

4.1.0.1 Coordinate-wise median.

The coordinate-wise median applies the univariate median per dimension, solving the separable problem \[\label{eq:coord95median} \hat{{\mathbf{y}}}_{\mathrm{Med}} \;=\; \mathop{\mathrm{arg\,min}}_{{\mathbf{z}}\in \mathbb{R}^{d}} \sum_{i=1}^{N} \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{1},\tag{6}\] whose \(k\)-th coordinate is the univariate median \(\mathrm{med}_i(\hat{y}_i^{(k)})\). The geometric median operates on joint Euclidean distance instead. The distinction matters when corruptions are structured across dimensions, as the following example illustrates.

The picture in Example 1 extends to a class of corruptions that are bounded per-coordinate but jointly anomalous: random vertices of the score hypercube \(\{0, K\}^{d}\), mixtures of extreme corner values, or any corruption whose marginals look plausible but whose joint structure is not. The empirical bimodal-random class (§6.3), in which each corrupted slot has each coordinate independently drawn from \(\{0, K\}\) with equal probability, is the canonical instance: per-coordinate the marginal is \(\frac{1}{2}(\delta_{0}+\delta_{K})\), indistinguishable from plausible scoring; jointly, the corrupted vector sits at a random hypercube corner, far from \({\mathbf{y}}^{\star}\) in \(\ell_{2}\). Because each coordinate-wise estimator must commit per coordinate without seeing the joint pattern, its per-coordinate bias is \(\Omega(\alpha)\) on this class (a one-dimensional Le Cam two-point argument under the symmetric corruption marginal, [@huber1964robust], Thm. 5.1), and the per-coordinate errors compose in \(\ell_{2}\) with a \(\sqrt{d}\) factor; 9 (and the §6.3 sweep) is the empirical analogue.

4.1.0.2 A note on the geometric median’s optimality.

The geometric median we use is the polynomial-time robust default at the small jury sizes \(N \leq 5\) characteristic of LLM panels; it is not the theoretically optimal estimator at large \(N\). Geometric median-of-means [@lugosi2019sub; @hopkins2020mean] achieves the parametric rate \(\sigma\sqrt{d/N}\) and a tighter breakdown-floor scaling of order \(\sigma\sqrt{\alpha}\) by aggregating block means before applying the geometric median. At \(N \leq 5\), however, MoM degenerates: \(K = N\) blocks of size 1 gives plain GM, and any coarser blocking lacks per-block concentration. We therefore use plain GM throughout.

4.1.0.3 Trimmed mean.

The \(\beta\)-trimmed mean discards the \(\beta\) fraction of points farthest from the sample mean and averages the remainder. It requires choosing \(\beta\), which in turn requires knowledge of the contamination rate \(\alpha\): if \(\beta < \alpha\), corrupted points survive trimming and the bias of Proposition 2 returns; if \(\beta > \alpha\), competent points are discarded, inflating variance.

4.1.0.4 Geometric median.

The geometric median attains the same \(1/2\) breakdown as the trimmed mean, operates on joint Euclidean distance, and is tuning-free—it requires no knowledge of the contamination rate. We therefore instantiate RoPoLL with the geometric median; empirical comparisons against CoMed and the trimmed mean are reported in §6, with the head-to-head against the trimmed mean on heavy-tailed corruption shown in 1.

4.2 The Geometric Median: Definition and Properties↩︎

The geometric median has a long history in location theory dating to Fermat (1643) and Weber (1909); its modern robustness analysis is due to [@lopuhaa1991breakdown; @small1990survey; @vardi2000multivariate; @acharya2022robust; @acharya2025geometric]. We collect the structural properties we will use.

4.3 The Weiszfeld Iteration↩︎

The geometric median has no closed form for \(d \geq 2\) [@bajaj1988algebraic]; we compute it via the modified Weiszfeld iteration [@weiszfeld1937point; @vardi2000multivariate].

4.3.0.1 Derivation.

At any non-data point \({\mathbf{z}}\neq \hat{{\mathbf{y}}}_{i}\) (for all \(i\)), the gradient of \(F({\mathbf{z}}) = \sum_{i} \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}\) is \(\nabla F({\mathbf{z}}) = \sum_{i=1}^{N} ({\mathbf{z}}- \hat{{\mathbf{y}}}_{i})/\|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}\). Setting \(\nabla F({\mathbf{z}}) = {\mathbf{0}}\) and rearranging gives the fixed-point \[\label{eq:weiszfeld95fixedpoint} {\mathbf{z}} \;=\; \frac{\sum_{i=1}^{N} \hat{{\mathbf{y}}}_{i} / \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}}{\sum_{i=1}^{N} 1 / \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}}.\tag{7}\] A modified weight \(1 / \max(\|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}, \eta)\) for small stability parameter \(\eta > 0\) handles the singularity when the iterate coincides with a data point [@vardi2000multivariate], yielding Algorithm 4.

Each iteration is a reweighted mean in which points far from the current consensus receive small weights, automatically downweighting corrupted judges.

4.3.0.2 Convergence and cost.

[@vardi2000multivariate] prove that the modified Weiszfeld iteration converges to the unique geometric median at a linear rate whenever the data are not collinear: \(\|{\mathbf{z}}^{(t)} - \hat{{\mathbf{y}}}_{\mathrm{GM}}\|_{2} \leq \rho^{t} \|{\mathbf{z}}^{(0)} - \hat{{\mathbf{y}}}_{\mathrm{GM}}\|_{2}\) for some \(\rho \in (0, 1)\). The number of iterations to reach tolerance \({\epsilon}\) is therefore \(O(\log(1/{\epsilon}))\), and each iteration costs \(O(Nd)\), giving total cost \(O(Nd\log(1/{\epsilon}))\). For a typical LLM jury (\(N \leq 20\), \(d \leq 5\), \({\epsilon}= 10^{-8}\)) this is microseconds on a modern processor—negligible relative to the seconds of GPU time per judge inference. A full convergence analysis is in Appendix 8.4.

6pt

5 Theoretical Guarantees↩︎

5.1 Finite-Sample Error Bound↩︎

We bound \(\|\hat{{\mathbf{y}}}_{\mathrm{GM}} - {\mathbf{y}}^{\star}\|_{2}\) under our Huber model in two steps: a deterministic geometric lemma about the geometric median (Lemma 1, due to [@minsker2015geometric]), and a probabilistic lemma that controls the sub-Gaussian cluster radius (Lemma 2). Plugging the cluster radius into the geometric lemma yields Theorem 1, our finite-sample upper bound on RoPoLL. Full proofs are in Appendix 8.

This is purely deterministic: a multiplicative bound between the geometric median and any target \(z\) in terms of how concentrated the inputs are around \(z\). The constant \(C_{\alpha}\) is sharp and diverges as \(\alpha \to 1/2\), matching the breakdown point of GM. 6 illustrates the geometric core of the proof: under the contradiction hypothesis \(\Delta = \|x_{*}-z\|_{2} > C_{\alpha}r\), the cluster ball subtends a narrow cone at \(x_{*}\), forcing the \((1{-}\alpha)k\) cluster points to lie inside it—and a balance of unit-vector subgradients (the first-order optimality condition for the geometric median) produces the contradiction.

The slack \(\beta\) controls a trade-off: a larger \(\beta\) permits a smaller cluster radius \(\rho\) (since fewer competent samples need to be inside) but augments the effective contamination threshold from \(\alpha\) to \(\alpha + \beta\) in the geometric step.

The proof is a one-line combination: applying Lemma 1 with \(k = N\), \(z = {\mathbf{y}}^{\star}\), \(r = \rho\), and effective threshold \(\alpha' = \alpha + \beta < 1/2\), the count bound from Lemma 2 is exactly the hypothesis of Lemma 1, so the geometric lemma gives \(\|\hat{{\mathbf{y}}}_{\mathrm{GM}} - {\mathbf{y}}^{\star}\|_{2} \leq C_{\alpha+\beta}\,\rho\). Full details are in Appendix 8.7.

5.1.0.1 Interpretation.

The bound has two components. The geometric constant \(C_{\alpha+\beta}\) depends only on the contamination rate (and slack); it diverges as \(\alpha + \beta \to 1/2\), encoding the breakdown point. The cluster radius \(\rho\) depends only on the noise scale \(\sigma\) and the dimension \(d\); it does not shrink with \(N\). This reflects the breakdown-point character of plain GM: under arbitrary \(Q\) in the Huber class, the asymptotic-\(N\) floor is set by the cluster radius, not by sample averaging. The bound is distribution-free over the corruption class \(\{Q_{i}\}\). 5 illustrates the two-step geometry. For comparison with the matching minimax lower bound (Theorem 2), see §5.2.

5.1.0.2 Synthetic validation.

8 validates the i.i.d.theory on controlled synthetic data: panel (a) confirms the closed-form clean-jury MSE of Corollary 1; panel (b) shows the geometric median converging to the predicted breakdown floor \(\sigma\alpha\sqrt{d}/(1-\alpha)\) as \(N\) grows under worst-case Huber contamination, while PoLL stays above the floor (Proposition 2); panel (c) confirms the \(\sqrt{d}\) scaling of the cluster radius \(\rho\) (Lemma 2) at fixed \(N\).

5.1.0.3 Beyond i.i.d.: equicorrelated juries.

Theorem 1 assumes conditional independence (Assumption 3). Real LLM juries trained on overlapping corpora violate this: inter-judge correlation \(\bar\gamma \in [0.3, 0.7]\) is typical ([fig:corpus_judge_corr,sec:appendix_dataset_correlation]). We close §5.1 by extending Theorem 1 to this regime: the breakdown structure (\(C_{\alpha+\beta}\) and \(\rho\)) survives unchanged; only the high-probability event weakens, from exponential in \(N\) to polynomial in the effective jury size \(N_{\mathrm{eff}} = N/(1+(N-1)\bar\gamma_{W})\) familiar from Corollary 1.

The proof replaces the Hoeffding step of Lemma 2 with a Chebyshev bound on \(\sum_{i}W_{i}\) under the bounded-covariance hypothesis; the cluster radius \(\rho\) and the geometric constant \(C_{\alpha+\beta}\) are unchanged because Lemma 1 is purely deterministic. The hypothesis is on the indicator correlation \(\bar\gamma_{W}\), which we estimate directly from the empirical co-incidence of cluster events \(\{W_{i} = 1\}\) on our \(13\)-judge experimental panels ([sec:appendix_gamma_W,tab:empirical_gamma_W]): across three cluster-radius calibrations (50th/70th/90th percentiles of pooled per-sample deviations), \(\bar\gamma_{W} \in [0.45, 0.53]\) on HelpSteer-2 and UltraFeedback, in line with the inter-judge score correlations \(\bar\gamma \in [0.49, 0.71]\) of 15. At \(N{=}3\) this gives \(N_{\mathrm{eff}} \in [1.45, 1.58]\); the breakdown floor and geometric constant are unaffected, and the high-probability event remains non-trivial for the moderate slack \(\beta\) used in practice. Full details are in Appendix 8.8.

5.2 Minimax Lower Bound↩︎

We close the section with a matching information-theoretic minimax lower bound that exposes a \(\sqrt{d}\) statistical–computational gap: RoPoLL is rate-optimal up to a dimensional constant on the breakdown floor, the price for \(O(Nd\log(1/{\epsilon}))\) tractability via the Weiszfeld iteration. No estimator can do better on the parametric variance term; the breakdown floor, in turn, is matched up to a dimensional constant only by the (intractable) Tukey halfspace median.

7 illustrates the modulus-of-continuity construction: when the two Gaussians at \({\mathbf{y}}_{0}\) and \({\mathbf{y}}_{1}\) are TV-close enough, their Huber neighborhoods overlap at a common distribution \(F\) that is consistent with both truths—making the two centers indistinguishable from any number of i.i.d.samples.

5.2.0.1 Comparison with the upper bound.

Theorems 1 and 2 match exactly on the parametric rate \(\sigma\sqrt{d/N}\). On the breakdown floor they differ by a \(\sqrt{d}\) factor: the upper bound scales as \(C_{\alpha}\sigma\sqrt{d}\) while the lower bound scales as \(\sigma\alpha/(1-\alpha)\). This is not a slack in the analysis but a real statistical–computational gap. The minimax-optimal estimator on the breakdown floor is the Tukey halfspace median [@tukey1975mathematics; @donoho1992breakdown], whose exact computation is NP-hard for \(d \geq 3\) [@johnson1978densest; @aloupis2006geometric]; the smoothed-depth estimator of [@chen2018robust] matches the \(\sigma\alpha\) floor in sub-exponential time. The geometric median is the polynomial-time alternative: it shares the optimal \(1/2\) breakdown point but pays a \(\sqrt{d}\) price for \(O(Nd\log(1/{\epsilon}))\) tractability via the Weiszfeld iteration (§4.3). For LLM juries the trade is favourable: \(d\) is small (1–5 in our benchmarks) so the \(\sqrt{d}\) overhead is at most \(\sim 2.2\times\), and at \(N = 3\) the variance term \(\sigma\sqrt{d/N}\) dominates the breakdown floor on every regime we test (§6.6). RoPoLL is therefore an efficient robust estimator, matching the minimax breakdown point at the small price of a dimensional constant on the contamination floor.

5.2.0.2 Scope of the i.i.d.assumptions.

Theorems 1 and 2 are stated under the i.i.d.baseline (Asm 3, identical \(\sigma_{i}\)). Lemma 3 relaxes independence to equicorrelation, covering the inter-judge correlation \(\bar\gamma \in [0.3, 0.7]\) measured in our experiments. Two further deviations remain out of scope: per-judge heterogeneity (\(\sigma_{i}, \alpha_{i}\) varying across the \(4\)–\(675\) B parameter range) and explicit dependence by design (peer-rank discussion [@li2024prd], multi-agent debate [@chan2024chateval], judge networks [@zhang2024wider]). The empirical evaluation in §6 holds these axes fixed and isolates the effect of contamination type and rate; the heterogeneous-jury extension—together with side information such as per-judge calibration on a labeled validation slice—is the subject of follow-up work.

6 Experiments↩︎

We evaluate RoPoLL against PoLL (the arithmetic-mean baseline of [@verga2024replacing]) and the coordinate-wise Median on three reward-model benchmarks under a per-case corruption pipeline that exposes the corruption-type dependence predicted by Theorem 1 and Example 1.

6.1 Setup↩︎

6.1.0.1 Datasets.

We use three popular reward model benchmarks with complementary ground-truth sources. HelpSteer 2 [@wang2024helpsteer2] contributes \(1{,}000\) samples drawn uniformly at random from the validation split, each rated on a \(0\)–\(4\) Likert scale across five attributes (helpfulness, correctness, coherence, complexity, verbosity) by trained human annotators. HelpSteer 3 [@wang-etal-2025-helpsteer3] contributes its full \(2{,}017\)-sample multilingual validation split; the native chosen-vs-rejected preference is converted to a scalar overall_preference target on \([-4, 4]\) by re-scoring both responses on the HelpSteer 2 rubric and taking the signed difference. UltraFeedback [@cui2024ultrafeedback] contributes \(1{,}000\) samples scored on a \(1\)–\(5\) scale across four attributes (helpfulness, honesty, instruction following, truthfulness) using GPT-4 as the reference annotator.

6.1.0.2 Judges and juries.

We score every sample with \(13\) open-weight judges spanning \(4\)–\(675\) B parameters at temperature \(0\) under a shared structured rubric: Mistral-Large-3 (\(675\) B), DeepSeek-V3.1 (\(671\) B), Qwen3-235B, Qwen3-32B, Nemotron-30B, Gemma-27B, Magistral-Small (\(24\) B), Ministral-14B, Gemma-12B, Nemotron-12B, Nemotron-9B, Ministral-8B, and Gemma-4B. From these we curate four three-judge committees that trade size against compute: Medium \(\approx 89\) B (Qwen3-32B, Nemotron-30B, Gemma-27B), Mixed \(\approx 53\) B (Qwen3-32B, Gemma-12B, Nemotron-9B), Small \(\approx 38\) B (Ministral-14B, Gemma-12B, Nemotron-12B), and Tiny \(\approx 21\) B (Nemotron-9B, Ministral-8B, Gemma-4B). The choice of \(N = 3\) is not arbitrary: under the equicorrelated jury model of Corollary 1 the effective jury size \(N_{\mathrm{eff}} = N/(1 + (N-1)\gamma)\) saturates at \(1/\gamma\) as \(N \to \infty\), so for the moderate inter-judge correlation \(\gamma \in [0.3, 0.5]\) characteristic of diverse but non-orthogonal LLM backbones, \(N_{\mathrm{eff}}\) saturates already at \(N \approx 2\)–\(3\) (8 (a) of 8, §5.1)—a prediction corroborated by the empirical diminishing-returns knee at \(N = 3\) in 12. We compare PoLL (arithmetic mean), the coordinate-wise Median, and RoPoLL (Algorithm 4) on these committees; all three operate on the same three (possibly corrupted) score vectors per sample.

6.1.0.3 Per-case corruption protocol.

Rather than injecting extra adversarial judges into a fixed pool, we hold the jury size at three and corrupt individual (sample, judge) cells at a per-case rate \(r \in \{0\%, 10\%, 20\%, 30\%, 40\%, 50\%\}\), matching the realistic failure pattern in which a judge occasionally emits a bad score on a specific input. The sweep range is calibrated against the natural-failure characterization of 2 (§3.4): naturally-occurring rates span \(0.59\%\) on HelpSteer 2 to \(33\%\) on HelpSteer 3 multilingual, so \(r \in [0, 50\%]\) covers the natural regime and stress-tests beyond. We consider four corruption types covering distinct adversarial regimes: (i) zeros, where every corrupted slot is replaced by \({\mathbf{0}}\) (the parser-failure fallback); (ii) inverted, where corrupted slots are replaced by \(K \cdot {\mathbf{1}}- {\mathbf{y}}^{\star}\) (the worst-case anti-correlated Byzantine attack); (iii) bimodal-random, where each coordinate of the corrupted slot is independently set to \(0\) or \(K\) with equal probability (the cross-dimensional failure mode of Example 1); and (iv) cauchy-far, where each corrupted slot is drawn as \({\mathbf{y}}^{\star} + 10 + 2(s_{\max}\!-\!s_{\min})\!\cdot\!{\mathbf{t}}\) with \({\mathbf{t}}\) component-wise standard Cauchy (a biased heavy-tailed Byzantine attack with undefined mean and variance). Pre-existing parser failures from real judges are dropped at \(r = 0\%\) and replaced with adversarial vectors at \(r > 0\%\), so the effective observed corruption rate at \(r > 0\) is \(f + (1 - f)\,r\) with \(f\) the naturally occurring parser-failure rate. We report RMSE against the reference labels; per-judge calibration breakdowns (MAE, mean bias) on the UltraFeedback rubric dimensions are in Appendix 9.2.

6.2 Heavy-Tailed Corruption↩︎

The cauchy-far attack is the empirical analogue of the adversarial choice in Proposition 2: each corrupted slot has an unbounded first moment, and a single contaminated judge can in principle drag PoLL by an arbitrary amount. The teaser (1, queued in §1) confirms this empirically. On the Medium jury, PoLL’s RMSE exceeds RoPoLL’s by one to three orders of magnitude at every \(r \geq 10\%\) on all three benchmarks. The largest gap occurs on HelpSteer 2 at \(r = 40\%\), where PoLL’s RMSE reaches \(\approx 4{,}951\) while RoPoLL holds at \(\approx 9.2\)—a ratio of \(\approx 540\times\). The coordinate-wise Median is competitive with RoPoLL here (full three-method comparison in 13, §6.6): under heavy-tailed Byzantine attacks any robust aggregator beats the mean, exactly as the theory predicts.

6.3 Cross-Dimensional Corruption↩︎

The bimodal-random attack drives each coordinate of a corrupted score independently to an extremum—each corrupted score is plausible per coordinate but jointly anomalous, the failure mode predicted by Example 1. 9 plots, for each benchmark, the \(13\) individual judges (gray circles) against their parameter count alongside the four RoPoLL committees (coloured stars) at their aggregate parameter count, both evaluated under identical \(30\%\) bimodal-random corruption. On HelpSteer 2 and UltraFeedback, all four RoPoLL committees sit visibly below the individual-at-30% scaling trend. The headline number: on HelpSteer 2, the Small committee at \(38\) B reaches RMSE \(= 1.18\), beating Mistral-Large-3’s \(1.55\) at \(675\) B—a \(1.31\times\) accuracy advantage at \(18\times\) fewer parameters. On the harder HelpSteer 3 (signed-preference target), the Medium committee at \(89\) B still matches DeepSeek-V3.1 (\(671\) B) at RMSE \(= 1.85\).

6.3.0.1 Compute-matched comparison.

The single-judge-vs-committee framing above understates the case for robust aggregation, because any 3-judge committee (\(114\) B–\(267\) B forward-pass compute) beats a single \(675\) B judge at evaluation cost. The fair compute-matched comparison is RoPoLL versus PoLL on the same committee. On the Small committee at \(30\%\) bimodal-random, RoPoLL (RMSE \(1.18\)) beats PoLL (RMSE \(\approx 1.45\)) on the same three judges by \(\approx 19\%\) at identical inference cost (13, full grid): the parameter-efficiency advantage holds because the geometric median’s joint-distance objective extracts more signal from the same forward passes than PoLL. Robust aggregation, not the ensemble itself, is what delivers the win.

6.4 Bounded Mean-Preserving Corruptions: Zeros and Inverted↩︎

The zeros and inverted attacks place corrupted scores at fixed bounded points on the score scale. On a bounded scale PoLL is mean-preserving under uniform-rate corruption when the corrupted point happens to lie at the scale midpoint—an empirical accident, not a property of the mean as an estimator—which makes these the regimes where RoPoLL and PoLL should be hardest to separate. 10 plots the parameter-efficiency view under \(30\%\) zeros corruption on the same axes as 9. The headline: even in this favourable-to-the-mean regime, all four RoPoLL committees sit at or below the individual-at-30% scaling line on every benchmark, and the gap to PoLL remains positive but small (\(\leq 0.3\) RMSE for the Medium jury across the full corruption sweep—see 13). RoPoLL is therefore not a universal replacement for the mean: when the corruption is bounded and happens to be mean-preserving, the two are within an insurance premium of each other. The argument for RoPoLL as a default is that the practitioner does not get to choose which regime the next corruption falls into.

6.5 Clean-Baseline Parameter Efficiency↩︎

A natural concern about a robust aggregator is that the robustness costs accuracy in the absence of corruption. Theorem 1 predicts a small insurance premium at \(\alpha = 0\) (the geometric constant \(C_{\beta} \to 1\) as \(\beta \to 0\)); 11 cashes this empirically by plotting the four RoPoLL committees against the \(13\)-judge individual scaling line at \(r = 0\%\). The Medium, Mixed, and Small committees sit below the individual scaling line on every benchmark; Tiny is roughly on-trend. The clean-case insurance premium is at most \(+6.4\%\) relative RMSE across the full grid (median \(+0.9\%\)), so the cost of using RoPoLL when corruption happens to be absent is a small fraction of the gains it delivers when corruption is present.

6.6 Jury-Size Ablation and Corruption-Type Dependence↩︎

Two practical questions remain: how many judges does RoPoLL actually need, and is the geometric median always the right choice? 12 answers the first by sweeping the jury size \(N \in \{1, 2, 3, 4\}\) across the four committee tiers under three corruption types at the clean baseline (\(r = 0\%\), left column) and under heavy contamination (\(r = 30\%\), right column). 13 answers the second by reporting the full three-method PoLL/Median/RoPoLL comparison across every (dataset, corruption type, rate) cell.

6.6.0.1 Jury ablation.

RMSE drops sharply from \(N = 1\) to \(N = 3\) and levels off thereafter on every tier and corruption type, both clean and at \(30\%\) contamination; the marginal benefit of a fourth judge falls within the standard-deviation band, confirming the Corollary 1 prediction that the three-judge committees sit at the knee of the cost–accuracy frontier.

6.6.0.2 Corruption-type ablation.

Under zeros and inverted PoLL is mean-preserving on the bounded score scale and tracks RoPoLL within \(\pm 0.3\) RMSE; under bimodal-random mean-preservation fails and RoPoLL stays below PoLL at every \(r \geq 10\%\); under cauchy-far the gap reaches one to three orders of magnitude (see 13). The coordinate-wise Median tracks RoPoLL closely on heavy-tailed and bounded-symmetric corruptions but lags RoPoLL on bimodal-random, where the cross-dimensional structure (Example 1) is invisible to a per-coordinate median. At the clean baseline, RoPoLL pays a small insurance premium (\(+0.01\%\) to \(+6.4\%\) relative RMSE). RoPoLL is a robust default for high-penalty regimes.

6.7 Noisy-GT Control: Systematic Bias, Not Imprecision↩︎

A natural concern about robust aggregation is that the “insurance premium” might be paid on a phantom—if real judge failures are imprecise but unbiased rather than systematically wrong, robustness machinery is unnecessary. We test this directly with a Noisy-GT adversary that injects \(\hat{{\mathbf{y}}}_{\mathrm{noisy}} = \mathrm{clip}({\mathbf{y}}^{\star} + \boldsymbol{{\epsilon}}, 0, K)\) with \(\boldsymbol{{\epsilon}} \sim {\mathcal{N}}({\mathbf{0}}, 0.8^{2}{\mathbf{I}})\) in place of the adversarial vectors of §6.1. Empirically, all three aggregators improve as the Noisy-GT injection rate increases on both HelpSteer 2 and UltraFeedback, with PoLL slightly preferred (because averaging unbiased noise is statistically optimal under Gaussian errors). This rules out the most obvious confound: the RoPoLL premium reported in §§6.2–6.4 is paid against biased contamination, not imprecision. The full per-model and per-dimension breakdowns supporting this control are in Appendix 9.2.

6.8 Released Corpus↩︎

To support reproduction and follow-up work, we release the full \(13\)-judge \(\times\) three-benchmark output corpus that drives every figure in this section. For each \((\mathrm{judge},\,\mathrm{sample})\) cell the corpus contains the parsed score vector \(\hat{{\mathbf{y}}}_{f}(x) \in \mathbb{R}^{d}\) produced by the deterministic parser \(\phi\) (Definition 4), the per-call latency, and the reference label \({\mathbf{y}}^{\mathrm{ref}}_{j}\) (Definition 2); parser-failure cells are recorded as the all-zeros fallback vector. The corpus totals approximately \(28\mathrm{K}\) scored \((\mathrm{judge},\,\mathrm{sample})\) cells (Table 2), enabling exact reproduction of every reported figure without re-running the inference cost.

4pt

6.8.0.1 Per-attribute score distributions.

14 plots the score distributions per attribute (per-dataset). HelpSteer 2 and UltraFeedback have substantial mass at the score extremes (parser fallback at \(0\); sycophantic judges concentrating at the maximum), motivating the zeros and inverted corruption types used in §6. HelpSteer 3, which reduces a five-attribute pair of responses to a single signed-preference scalar, is well-centered on \(0\) with light tails, consistent with the cancellation of per-attribute biases under the signed-difference reduction.

6.9 Inter-Judge Correlation Structure↩︎

15 shows the pairwise Pearson correlation between every judge pair in the \(13\)-judge pool, averaged over attributes. Empirical mean off-diagonal correlations are \(\bar\gamma_{\mathrm{HS2}} = 0.49\), \(\bar\gamma_{\mathrm{HS3}} = 0.49\), and \(\bar\gamma_{\mathrm{UF}} = 0.71\). These directly support the assumption \(\gamma \in [0.3, 0.5]\) used in §6.1 to motivate the choice \(N = 3\): substituting the measured \(\bar\gamma\) into the saturation law \(N_{\mathrm{eff}}^{\infty} = 1/\gamma\) (Corollary 1) yields \(N_{\mathrm{eff}}^{\infty} \approx 2.0\) on HS 2 and HS 3 and \(\approx 1.4\) on UltraFeedback, so the empirical diminishing-returns knee at \(N = 3\) in 12 sits at or just past the saturation point predicted by the corpus’s actual correlation structure.

The UltraFeedback correlation \(\bar\gamma_{\mathrm{UF}} = 0.71\) is notably higher than the HelpSteer correlations. This reflects the fact that UltraFeedback’s reference labels are themselves GPT-4 annotations [@cui2024ultrafeedback], so judges trained on similar rubric distributions converge to similar scores; the HelpSteer benchmarks use trained-human annotators (HelpSteer 2) or pairwise human preferences (HelpSteer 3), producing more genuine inter-judge variation. This is consistent with the smaller RoPoLL/PoLL gap observed on UltraFeedback in §6: when judges already agree, the difference between the mean and the geometric median is small.

6.10 Empirical Indicator Correlation \(\bar\gamma_{W}\)↩︎

Lemma 3 bounds the failure probability of the RoPoLL cluster event in terms of the indicator correlation \(\bar\gamma_{W} = \mathrm{mean}_{i \neq j}\frac{\mathrm{Cov}(W_{i},W_{j})}{\sqrt{\mathrm{Var}(W_{i})\mathrm{Var}(W_{j})}}\) of the cluster indicators \(W_{i} = \mathbb{1}\{Z_{i}=0,\;\|\hat{{\mathbf{y}}}_{i}-{\mathbf{y}}^{\star}\|_{2}\leq\rho_{p}\}\). This is in principle a finer object than the inter-judge score correlation \(\bar\gamma\) of §6.9: \(\bar\gamma\) measures the linear correlation between raw score vectors, while \(\bar\gamma_{W}\) measures the co-incidence of two judges both being competent and within the cluster ball. We estimate \(\bar\gamma_{W}\) directly on the experimental panels.

6.10.0.1 Estimation procedure.

For benchmark \(b \in \{\mathrm{HS2}, \mathrm{UF}\}\): (i) for each \((\text{judge } i, \text{sample } s)\) cell, compute the \(\ell_{2}\) deviation \(\delta_{i}^{(s)} = \|\hat{{\mathbf{y}}}_{i}^{(s)} - {\mathbf{y}}^{\star,(s)}\|_{2}\) (parser-failure cells contribute \(W_{i}^{(s)} = 0\)); (ii) select a cluster radius \(\rho\) as the \(p\)-th quantile of pooled deviations \(\{\delta_{i}^{(s)}\}_{i,s}\); (iii) form \(W_{i}^{(s)} = \mathbb{1}\{\delta_{i}^{(s)} \leq \rho\}\); (iv) compute the mean off-diagonal Pearson correlation of the rows of \(W \in \{0,1\}^{N \times S}\). We report \(\bar\gamma_{W}\) at three radii (\(p \in \{0.50, 0.70, 0.90\}\)) to show stability under the calibration choice.

8pt

6.10.0.2 Implication for Lemma 3.

The role of Lemma 3 is structural: it shows that the geometric-breakdown structure (\(C_{\alpha+\beta}\) and the cluster radius \(\rho\)) of Theorem 1 is preserved when the i.i.d. assumption is replaced by an equicorrelated-indicator hypothesis with \(\bar\gamma_{W} \in [0, 1]\). The probability event delivered by the Chebyshev step, \(\Pr[\cdot] \geq 1 - 1/(\beta^{2} N_{\mathrm{eff}})\), is informative in the large-\(N_{\mathrm{eff}}\) regime (e.g., a hypothetical jury of \(N=10\)–\(30\) judges with \(\bar\gamma_{W} \approx 0.2\) gives \(N_{\mathrm{eff}} \in [3.6, 7]\) and \(\beta = 0.2\) gives a non-trivial bound) but degenerates at small \(N_{\mathrm{eff}}\), including the practical \(N_{\mathrm{eff}} \approx 1.5\) of our \(N=3\) panels. This is a fundamental limit of variance-only concentration at small \(N\), not a slack in the analysis: with only \(\sim 1.5\) effective independent samples, no concentration argument can deliver a tight high-probability bound, regardless of the estimator. A Bernstein-type bound under bounded-covariance martingale structure (Remark 20) would replace \(\beta^{-2}\) with \(\exp(-c\beta^{2}N_{\mathrm{eff}})\) but does not materially help at \(N_{\mathrm{eff}} \approx 1.5\). The practical value of Lemma 3 for our small-\(N\) regime is therefore the structural guarantee, not the quantitative probability: the breakdown floor and the geometric constant are independent of this concentration argument and are the load-bearing quantities for jury-aggregation deployment.

6.11 Practical Recommendation↩︎

Use RoPoLL as the default jury aggregator: the clean-case insurance premium is small (\(\leq 6\%\) relative RMSE, §6.5) and the threat to LLM juries is biased contamination rather than imprecision (§6.7). The jury size is not a fixed prescription but follows from the saturation law of Corollary 1 (\(N_{\mathrm{eff}}\) saturates at \(1/\gamma\)): the cost–accuracy knee sits at whatever \(N\) reaches that ceiling for the inter-judge correlation \(\gamma\) of the judge pool at hand. For the diverse open-weight pools studied here (\(\gamma \approx 0.49\)–\(0.71\), §6.9) the knee falls at \(N \approx 3\) (12); a more orthogonal pool (smaller \(\gamma\)) would push it higher, and a redundant one lower. A controlled 2D synthetic visualisation of three representative failure modes, the per-model and per-dimension breakdowns, and the full extra-metrics tables are in Appendices 9.2 and 9.1.

7 Conclusion↩︎

We recast LLM-jury aggregation as a robust mean-estimation problem, showed that PoLL [@verga2024replacing] admits unbounded bias under any positive contamination (Proposition 2), and proposed RoPoLL: replace the mean with the geometric median. Theorem 1 gives a finite-sample upper bound; Lemma 3 extends it to equicorrelated juries; Theorem 2 provides a matching minimax lower bound that aligns on the parametric rate and exposes a \(\sqrt{d}\) statistical–computational gap on the breakdown floor (the price of GM’s polynomial-time tractability vs.the intractable Tukey halfspace median). Empirically (§6), RoPoLL reduces PoLL’s RMSE by orders of magnitude on heavy-tailed and cross-dimensional attacks while paying \(\leq 6.4\%\) in clean-baseline relative RMSE, and a \(3\)-judge RoPoLL committee at \(38\) B beats Mistral-Large-3 (\(675\) B) by \(1.31\times\) on HelpSteer 2 under \(30\%\) bimodal-random corruption: robust aggregation, not the ensemble itself, delivers the win.

7.0.0.1 Scope and limitations.

The theory holds the i.i.d.baseline of Assumptions 2–5 fixed and partially relaxes independence via Lemma 3; per-judge heterogeneity (\(\sigma_{i},\alpha_{i}\) varying across \(4\)–\(675\) B), explicit dependence by design [@li2024prd; @chan2024chateval; @zhang2024wider], and tightening the contamination-constant gap between Theorem 1 and Theorem 2 at finite \(\alpha\) remain open. Empirically, the corruption sweep is synthetic injection at a single rubric serialisation and temperature \(0\), so it does not probe prompt-format sensitivity [@wang2023large; @stureborg2024large]; the Noisy-GT control (§6.7) rules out the obvious confound that RoPoLL’s premium is paid against benign imprecision rather than biased contamination, but a large-scale evaluation against naturally occurring judge failures on a downstream alignment task remains the most important next step. A systematic comparison against the broader robust-aggregation toolbox (median-of-means, smoothed Tukey depth, learned calibration) is also left open.

7.0.0.2 Outlook.

The corruption-class diagnosis transfers verbatim to any pipeline where heterogeneous workers produce biased point-mass errors at low rate—reward-model ensembles for RLHF, synthetic-data filtering juries, crowd annotation—suggesting the geometric median as a candidate default beyond LLM juries.

The appendix collects the deferred proofs and supporting material for the body of the paper. The roadmap of formal results is in Table 1 (§5); the appendix sections that follow are organised as Appendix 8 (proofs for §3, §4, and §5, including the matching minimax lower bound), Appendix 9.2 (per-model and per-dimension breakdowns supporting §6), and Appendix 9.1 (controlled 2D synthetic visualisation of five representative failure modes).

8 Complete Proofs and Full Theoretical Development↩︎

8.1 Proof of Proposition 1 (Variance Reduction)↩︎

8.2 Proof of Proposition 2 (Unbounded Bias of PoLL)↩︎

For convenience we recall the statement: under Assumption 2 and finite first moments \({\mathbf{\mu}}_{i}^{Q} \triangleq \mathbb{E}_{Q_{i}}[\hat{{\mathbf{y}}}_{i}]\), the mean’s conditional bias is \[\label{eq:appendix95mean95bias95formula} \mathbb{E}\!\left[\hat{{\mathbf{y}}}_{\mathrm{mean}} \mid {\mathbf{y}}^{\star}\right] - {\mathbf{y}}^{\star} \;=\; \frac{1}{N}\sum_{i=1}^{N}\alpha_{i}\!\left({\mathbf{\mu}}_{i}^{Q} - {\mathbf{y}}^{\star}\right),\tag{9}\] and is not uniformly bounded over the corruption class as long as \(\alpha > 0\), regardless of \(N\).

8.3 Proof of Proposition 3↩︎

8.4 Weiszfeld Iteration: Full Derivation, Convergence, and Cost↩︎

For completeness, this subsection gives the full derivation, convergence statement, and cost analysis for the Weiszfeld iteration sketched in §4.3.

8.4.0.1 Derivation.

At a non-data point \({\mathbf{z}}\neq \hat{{\mathbf{y}}}_{i}\) for all \(i\), the gradient of the GM objective \(F({\mathbf{z}}) = \sum_{i} \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}\) is \[\label{eq:gm95gradient} \nabla F({\mathbf{z}}) = \sum_{i=1}^{N} \frac{{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}}{\|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}}.\tag{12}\] Setting \(\nabla F({\mathbf{z}}) = {\mathbf{0}}\) and rearranging gives the fixed-point equation 7 of §4.3, \[{\mathbf{z}} \;=\; \frac{\sum_{i=1}^{N} \hat{{\mathbf{y}}}_{i} / \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}}{\sum_{i=1}^{N} 1 / \|{\mathbf{z}}- \hat{{\mathbf{y}}}_{i}\|_{2}},\] which is the Weiszfeld iteration \({\mathbf{z}}\leftarrow T({\mathbf{z}})\). When the current iterate coincides with a data point \(\hat{{\mathbf{y}}}_{j}\), the denominator \(\|{\mathbf{z}}- \hat{{\mathbf{y}}}_{j}\|_{2} = 0\) creates a singularity; the modified step of [@vardi2000multivariate] replaces the weight by \[\label{eq:modified95weight} w_{i}^{(t)} = \frac{1}{\max\!\left(\|{\mathbf{z}}^{(t)} - \hat{{\mathbf{y}}}_{i}\|_{2},\; \eta\right)}\tag{13}\] for a small stability parameter \(\eta > 0\), recovering Algorithm 4.

8.4.0.2 Convergence.

[@vardi2000multivariate] prove that the modified Weiszfeld iteration converges to the unique geometric median at a linear rate whenever the data are not collinear: there exists \(\rho \in (0, 1)\) depending on the data configuration with \(\|{\mathbf{z}}^{(t)} - \hat{{\mathbf{y}}}_{\mathrm{GM}}\|_{2} \leq \rho^{t} \|{\mathbf{z}}^{(0)} - \hat{{\mathbf{y}}}_{\mathrm{GM}}\|_{2}\). The number of iterations to reach tolerance \({\epsilon}\) is therefore \(O(\log(1/{\epsilon}))\).

8.4.0.3 Cost.

Each iteration computes \(N\) Euclidean distances in \(\mathbb{R}^{d}\) and one weighted average, costing \(O(Nd)\) arithmetic operations. With \(O(\log(1/{\epsilon}))\) iterations the total cost is \(O(Nd \log(1/{\epsilon}))\). For typical LLM juries (\(N \leq 20\), \(d \leq 5\), \({\epsilon}= 10^{-8}\)) this amounts to a few hundred floating-point operations—microseconds on any modern processor, while a single LLM judge invocation costs seconds of GPU time. The aggregation step is computationally negligible relative to the inference cost of the jury.

8.5 Proof of Lemma 1↩︎

For convenience, we recall Lemma 1: let \(x_{1}, \ldots, x_{k} \in \mathbb{R}^{d}\) and let \(x_{*}\) be any minimizer of \(z \mapsto \sum_{j=1}^{k} \|z - x_{j}\|_{2}\); fix \(\alpha \in (0, 1/2)\), \(r > 0\), \(z \in \mathbb{R}^{d}\). If \(|\{j : \|x_{j} - z\|_{2} \leq r\}| \geq (1-\alpha)k\), then \(\|x_{*} - z\|_{2} \leq C_{\alpha}\,r\) with \(C_{\alpha} = (1-\alpha)/\sqrt{1-2\alpha}\).

8.6 Proof of Lemma 2↩︎

For convenience, we recall Lemma 2: under Assumptions 2–5, for any slack \(\beta \in (0, 1/2 - \alpha)\), with probability at least \(1 - \exp(-N\beta^{2}/2)\), at least \((1-\alpha-\beta)N\) of the \(N\) judge outputs lie within distance \(\rho = \sigma\big(C_{1}\sqrt{d} + \sqrt{(1/c)\log(2(1-\alpha)/\beta)}\big)\) of \({\mathbf{y}}^{\star}\), where \(C_{1}, c > 0\) are absolute constants from the sub-Gaussian-norm tail bound derived in Step 1 below.

8.6.0.1 Note on heterogeneous parameters.

Assumptions 2 and 4 are stated per-judge (\(\alpha_{i}\) and \(\sigma_{i}\)). Throughout this proof we read \(\alpha\) as the global mean contamination \(\alpha = (1/N)\sum_{i}\alpha_{i}\) from Assumption 5 (which we are entitled to do because Hoeffding in Step 2 only sees \(\sum_{i}\mathbb{E}W_{i}\); per-judge heterogeneity averages out at this aggregation step), and \(\sigma\) as \(\sigma = \max_{i}\sigma_{i}\) (the worst-case sub-Gaussian parameter, used in Step 1 to bound every \(i\) simultaneously).

The proof is in three stages: (1) control the tail of one competent sample’s deviation \(\|\hat{{\mathbf{y}}}_{i} - {\mathbf{y}}^{\star}\|_{2}\) at probability \(p\) via a covering-net argument; (2) count, via Hoeffding, how many of the \(N\) judges fall inside the resulting ball; (3) pick \(p\) and a Hoeffding slack \(u\) so that the count exceeds \((1-\alpha-\beta)N\) with the claimed probability.

8.6.0.2 On the choice of competent-component assumption.

The sub-Gaussian assumption is one of four natural choices for the competent component, ordered from weakest to strongest. Each gives a different cluster-radius bound; sub-Gaussian is the choice that delivers Lemma 2. We record the alternatives for context.

8.7 Proof of Theorem 1↩︎

For convenience, we recall Theorem 1: under Assumptions 2–5, fix any slack \(\beta \in (0, 1/2 - \alpha)\); with probability at least \(1 - \exp(-N\beta^{2}/2)\), \(\|\hat{{\mathbf{y}}}_{\mathrm{GM}} - {\mathbf{y}}^{\star}\|_{2} \leq C_{\alpha+\beta}\,\rho\), where \(\hat{{\mathbf{y}}}_{\mathrm{GM}}\) is any geometric median of the \(N\) judge outputs (Definition 7), \(C_{\alpha+\beta} = (1-\alpha-\beta)/\sqrt{1-2(\alpha+\beta)}\) is the geometric-breakdown constant of Lemma 1 evaluated at \(\alpha+\beta\), and \(\rho = \sigma(C_{1}\sqrt{d} + \sqrt{(1/c)\log(2(1-\alpha)/\beta)})\) is the cluster radius of Lemma 2 (\(C_{1}, c > 0\) absolute constants).

8.8 Proof of Lemma 3↩︎

For convenience we recall Lemma 3: under Assumptions 2, 4, 5 and the equicorrelated-indicator assumption (replacing Asm. 3) \(\mathrm{Cov}(W_{i},W_{j}) \leq \bar\gamma_{W}\sqrt{\mathrm{Var}(W_{i})\mathrm{Var}(W_{j})}\) for \(i \neq j\), with \(\bar\gamma_{W} \in [0,1]\), the RoPoLL bound \(\|\hat{{\mathbf{y}}}_{\mathrm{GM}}-{\mathbf{y}}^{\star}\|_{2} \leq C_{\alpha+\beta}\rho\) holds with probability at least \(1 - 1/(\beta^{2}N_{\mathrm{eff}})\), where \(N_{\mathrm{eff}} = N/(1+(N-1)\bar\gamma_{W})\).

The proof follows the same skeleton as Lemma 2 (per-sample tail \(\to\) count-bound) combined with Lemma 1 (deterministic geometric step), but replaces the Hoeffding count-bound (which required independence) with a Chebyshev count-bound on the variance of \(\sum_{i}W_{i}\) under the bounded-covariance hypothesis. The deterministic geometric step (Lemma 1) and the per-sample sub-Gaussian tail (Step 1 of Lemma 2’s proof) are correlation-free and carry through unchanged.

8.9 Proof of Theorem 2↩︎

Theorem 1 provides an upper bound on the error of the geometric median. A natural question is whether this rate can be improved by any estimator. The following result shows that, in the parametric regime, it cannot.

For convenience we restate Theorem 2: under the observation model 4 with \(N\) judges in \(\mathbb{R}^{d}\), homogeneous contamination rate \(\alpha < 1/2\), and \(\sigma^{2}\)-sub-Gaussian competent noise (Assumptions 2, 3, 4, 5), there exists a universal constant \(c > 0\) such that \[\label{eq:minimax95lower} \inf_{\hat{{\mathbf{y}}}}\; \sup_{F \in {\mathcal{F}}_{\alpha,\sigma}}\; \mathbb{E}_{F}\!\left[\big\|\hat{{\mathbf{y}}} - {\mathbf{y}}^{\star}\big\|_{2}\right] \;\geq\; c\,\sigma\!\left( \sqrt{d/N} \;+\; \frac{\alpha}{1 - \alpha} \right).\tag{45}\]

8.9.0.1 Comparison with the upper bound.

At \(\alpha = 0\), the upper and lower bounds match at the parametric rate \(\sigma\sqrt{d/N}\), confirming that the geometric median is rate-optimal in the clean regime. On the breakdown floor the upper bound (Thm 1) scales as \(C_{\alpha}\sigma\sqrt{d}\) while the lower bound scales as \(\sigma\alpha/(1-\alpha)\); the gap is a \(\sqrt{d}/\alpha\) factor. This is not slack in the analysis but a real statistical–computational gap. The minimax-optimal estimator on the breakdown floor is the Tukey halfspace median [@tukey1975mathematics; @donoho1992breakdown], whose exact computation is NP-hard for \(d \geq 3\) [@johnson1978densest; @aloupis2006geometric]; the smoothed-depth estimator of [@chen2018robust] matches the \(\sigma\alpha\) floor in sub-exponential time. The geometric median is the polynomial-time alternative: it shares the optimal \(1/2\) breakdown point but pays a \(\sqrt{d}\) price for \(O(Nd\log(1/{\epsilon}))\) tractability via the Weiszfeld iteration. For LLM juries the trade is favourable: \(d\) is small (1–5 in our benchmarks) so the \(\sqrt{d}\) overhead is at most \(\sim 2.2\times\), and at small \(N\) the variance term \(\sigma\sqrt{d/N}\) dominates the breakdown floor on every regime we test.

9 Additional Experiments↩︎

9.1 Synthetic 2D Simulation: Visual Intuition↩︎

For pedagogical intuition we instantiate the observation model 4 in \(d = 2\) dimensions with score range \([0, K]\) and visualize five representative failure modes. A jury of \(N\) judges evaluates a single instance with latent reward \({\mathbf{y}}^{\star} \in [0, K]^{2}\). Each competent judge (\(Z_{i} = 0\)) draws from a tight isotropic Gaussian centered on \({\mathbf{y}}^{\star}\); each corrupted judge (\(Z_{i} = 1\)) draws from a corruption distribution \(Q_{i}\) specific to the failure mode. The corruption indicator \(Z_{i} \sim \mathrm{Bernoulli}(\alpha)\) is drawn independently per judge at homogeneous rate \(\alpha \in \{0.10, 0.30, 0.40\}\). We compare the arithmetic mean and the geometric median (computed via Algorithm 4). In every figure, the gold star marks \({\mathbf{y}}^{\star}\), blue dots are competent judge outputs, red crosses are corrupted outputs, and the orange square and purple triangle mark the arithmetic mean and the geometric median, respectively.

9.1.0.1 Mode collapse (\(Q = \delta_{{\mathbf{0}}}\)).

The corrupted judge outputs the zero vector on every attribute—the canonical parser-fallback failure mode (Remark 2). 16 shows the mean pulled toward the origin as \(\alpha\) grows, while the geometric median remains anchored to the competent cluster.

9.1.0.2 Inverted (\(Q = \delta_{K \cdot {\mathbf{1}}- {\mathbf{y}}^{\star}}\)).

The worst-case anti-correlated Byzantine adversary (17). This is the sharpest visual demonstration of the breakdown-point advantage: the corrupted locus and \({\mathbf{y}}^{\star}\) lie on opposite sides of the score space, so at \(\alpha = 0.30\) the mean has already crossed the midpoint while the geometric median remains within the competent cluster.

9.1.0.3 Biased dimension.

Partial competence: correct on one attribute, catastrophically wrong on the other (18). This is the synthetic counterpart of bimodal-random (§6.3) and the picture of cross-dimensional corruption from Example 1: each corrupted score is plausible per coordinate but jointly anomalous, and the geometric median’s joint-distance objective resists the off-axis pull that fools per-coordinate alternatives.

9.1.0.4 Random hypercube corners.

The canonical instance of the cross-dimensional class: each corrupted score lands at a vertex of \(\{0, K\}^{d}\) chosen uniformly at random (19). The per-coordinate marginal \(\frac{1}{2}(\delta_{0}+\delta_{K})\) is indistinguishable from plausible scoring; jointly, every corrupted vector sits at a corner far from \({\mathbf{y}}^{\star}\) in \(\ell_{2}\). This is the “random vertex” generalisation of biased dimension above and exactly the bimodal-random class evaluated empirically in §6.3.

9.1.0.5 Sycophantic.

A real-world failure mode in which corrupted judges always rate near the top of the scale—the “everything is great” bias (20). The corrupted cloud sits in the upper-right corner of \([0, K]^{d}\); the arithmetic mean drifts diagonally toward it while the geometric median stays anchored to the competent majority near \({\mathbf{y}}^{\star}\). This complements mode collapse (corruption at the lower-left extremum) at the opposite extreme of the score scale.

9.1.0.6 Summary.

Across all three failure modes, the arithmetic mean acquires a bias proportional to \(\alpha\) and aligned with the corruption locus, while the geometric median remains close to \({\mathbf{y}}^{\star}\) as long as \(\alpha < 1/2\), in agreement with Theorem 1. The complementary Noisy-GT control (§6.7) confirms that this advantage is paid only against biased contamination: when the corruption is benign Gaussian noise, the geometric median does not sacrifice accuracy.

9.2 Per-Model and Per-Dimension Calibration Breakdowns↩︎

The figures in §6 aggregate across rubric dimensions and report the Medium jury’s RMSE. This subsection records the underlying per-model and per-dimension calibration breakdowns on UltraFeedback that motivated the curated three-judge committees of §6.1.

9.2.0.1 Judge set.

The calibration analysis in this subsection includes three closed-API reference judges (Claude Opus, Sonnet, and Haiku  4.5) in addition to the \(13\) open-weight judges of §6.1. The closed-API judges are reference points only — they are not used in any RoPoLL committee — and are included here to contextualise the open-weight calibration patterns.

9.2.0.2 Per-dimension MAE.

21 reports the mean absolute error for each judge against the UltraFeedback rubric dimensions (Helpfulness, Honesty, Instruction Following, Truthfulness). Qwen3 32B and Mistral-Large-3 lead with sub-\(0.75\) MAE across all four dimensions; the Claude family lies near the bottom of the calibration ranking despite strong ranking ability (22 below explains why).

9.2.0.3 Per-dimension mean bias.

22 reports the signed mean bias \(\mathbb{E}[\hat{y}_{i}^{(k)} - y^{\star,(k)}]\) for each (judge, dimension) cell. Two systematic patterns emerge. The Claude family shows uniformly negative bias across all four dimensions (\(-0.5\) to \(-0.8\) on Truthfulness)—a systematic under-scoring tendency. Smaller open-weight models (Magistral Small, Gemma 4B, Nemotron 9B) show uniformly positive bias of comparable magnitude. Qwen3 32B and Qwen3 235B are closest to zero across all dimensions, consistent with their leading MAE. The bias direction is precisely the contamination structure Proposition 2 formalises: mixing systematically over-scoring and under-scoring judges leaves the arithmetic mean’s bias bounded only by the worst per-judge displacement; the geometric median is robust to such mixed-direction biases because the joint subgradient balance does not weight per-coordinate sign.

---

1. Corresponding author.↩︎

2. Dataset released at https://github.com/aws/RoPoLL.↩︎