1 Introduction↩︎

LLM-as-a-judge evaluation has become a standard methodology for scalable assessment of language model outputs [1]. To mitigate single-model biases, researchers have turned to multi-model panels — ensembles of diverse LLMs that vote on evaluation items — with the expectation that cross-model diversity yields something approaching independent assessment [2]. The intuition is compelling: if models from different providers, trained on different data, make different errors, then majority voting should be robust.

This intuition draws on the logic of the Condorcet Jury Theorem [3]: if each voter is better than chance and votes independently, majority-vote accuracy increases monotonically with panel size and approaches certainty. The practical appeal is clear — adding more judges should always help, and a panel of 9 should be far more reliable than any single judge.

But is it? We evaluate a 9-judge panel spanning 7 model families on three natural language inference (NLI) benchmarks — MNLI, SNLI, and AlphaNLI — each with 100 human annotations per item (§3). The panel provides negligible or negative lift over its single best member. On MNLI, the panel (72.0%) barely edges the best judge (Qwen3-32B, 71.8%) by 0.2pp — within noise; on SNLI, the best judge dominates (Claude Sonnet 4.5, 84.2% vs.panel 77.7%); and on AlphaNLI, an abductive reasoning task with a different label set, the pattern persists (91.2% vs.%). These results are impossible under the independence assumption but expected when errors are highly correlated. Recent work has documented such correlated errors across LLMs on standard benchmarks [4], and conceptual arguments suggest that shared training paradigms should induce dependence [5]. However, no prior work has quantified the effective independence of LLM judge panels in a way that directly connects to majority-vote reliability, using a ground truth rich enough to validate the measurement.

We address this gap with three contributions:

1. A diagnostic framework for LLM judge panels. We combine the Kish effective sample size (\(n_{\text{eff}}\)) — a measure of how many truly independent votes a panel contains — with a Condorcet null model that simulates what majority-vote accuracy would be if judges voted independently. Applied to a 9-judge, 7-family panel on three ChaosNLI datasets [6], we find \(n_{\text{eff}}\approx 2.0\)–\(2.5\): the panel contains roughly two independent votes worth of information. The accuracy shortfall relative to this independent prediction (the Condorcet gap) is 8–22 percentage points (pp; permutation \(p < 10^{-4}\)).

2. A severe independence deficit, stable across tasks, prompts, and temperatures. The deficit is remarkably consistent: \(n_{\text{eff}}\approx 2.0\)–\(2.5\) across three NLI datasets, three prompt variants, two temperature settings, and a pairwise preference task (RewardBench; \(n_{\text{eff}}= 2.0\)) — despite panel accuracy ranging from 69% to 93%. Across all conditions, the panel fails to meaningfully outperform the single best judge. The scaling curve shows that adding judges beyond 5 yields negligible benefit, with effective independence asymptoting at roughly 2.3–3.1 (varying by dataset).

3. A negative result: aggregation cannot overcome correlation. Established aggregation methods — Dawid-Skene EM [7] and accuracy-weighted voting — close at most 11% of the Condorcet gap across all four datasets, even with oracle access to gold labels. The bottleneck is correlated inputs, not the algorithm: no weighting scheme can extract a third independent vote from \(\sim\)​2.2 effective votes of information.

These results have direct practical implications: paying for 9 opinions but receiving the informational equivalent of \(\sim\)​2 is a substantial inefficiency. The marginal value of additional judges is near zero, and unanimous panel agreement is far less diagnostic than it appears. The path forward is not larger ensembles of similar models, but diversification of the underlying reasoning — models that genuinely differ in how they process information.

2 Related Work↩︎

2.0.0.1 LLM-as-a-judge.

[1] established the LLM-as-a-judge paradigm, and subsequent work has revealed systematic biases [8]–[10] and raised measurement-theory concerns about validity and reliability [11], [12]. Our work goes beyond cataloguing individual biases to quantify the structural dependence among judges — a more fundamental constraint on panel reliability.

2.0.0.2 LLM judge panels.

[2] proposed PoLL (Panel of LLM Evaluators), demonstrating that panels of smaller, diverse models outperform single large judges across six datasets. Importantly, PoLL compares panels to the average individual judge, where panels naturally win by diversifying away individual quirks. Our finding — that the best individual outperforms the panel — does not contradict PoLL but reveals a different phenomenon: when judges are highly correlated, majority voting dilutes the best judge’s signal with redundant weaker votes. Our work complements PoLL by showing that the panel’s effective information content is far lower than the raw panel size suggests.

The Trust-or-Escalate framework [13] provides provable guarantees using single-model confidence to decide when to escalate to human review. Our approach differs fundamentally: we use cross-model disagreement and show that this disagreement is itself unreliable due to correlated errors.

2.0.0.3 Correlated errors and scaling limits.

Most closely related to our work, [4] conducted a large-scale study of error correlation across 350+ LLMs, finding that models agree on wrong answers 60% of the time on some benchmarks. [14] demonstrated what they term the “artificial hivemind” effect: LLMs produce strikingly homogeneous outputs on open-ended tasks, both within and across model families, and LLM judges are poorly calibrated on items where human annotators disagree — our \(n_{\text{eff}}\approx 2.0\)–\(2.4\) provides a precise quantification of this qualitative insight. [15] proved a complementary theoretical result: when the judge is no more capable than the evaluated model, no debiasing method can reduce the required ground-truth data by more than half, establishing a fundamental scaling ceiling. We build on this body of work by (a) measuring effective independence in the LLM-as-a-judge setting using Kish \(n_{\text{eff}}\), (b) quantifying the Condorcet gap that correlation creates, and (c) showing that established aggregation methods cannot close this gap.

2.0.0.4 Condorcet Jury Theorem and ensembles.

The Condorcet Jury Theorem [3] underpins much of the intuition behind ensemble methods [16], [17]: diverse, independent voters collectively outperform individuals. [18] showed that independence is necessary, not just sufficient — correlated voters can perform worse than individuals. In the LLM setting, [5] applied the Condorcet Jury Theorem to sentiment analysis ensembles, finding marginal improvement consistent with a lack of independence. [19] formalized this via an information-theoretic error floor. Our work empirically validates these theoretical concerns, providing the first item-level measurement of effective independence in a judge evaluation setting.

2.0.0.5 Statistically principled aggregation.

[20] proposed CARE, a confounder-aware aggregation framework that models inter-judge correlations, reducing aggregation error by up to 25%. The crowdsourcing literature offers related methods: the Dawid-Skene model [7] estimates annotator error rates via EM, and [21] extended this to learning from noisy crowds. Where these methods propose solutions (better aggregation), our paper provides both the diagnosis — quantifying how much independence is actually present — and a negative result: even with oracle access to gold labels, established aggregation methods close at most 11% of the Condorcet gap (§5.4), suggesting that the problem is structural rather than algorithmic.

2.0.0.6 Human label variation and ChaosNLI.

Human disagreement on NLI items is systematic, not mere noise [22], [23]. [6] created ChaosNLI with 100 annotator labels per item, built on MNLI [24], providing the richest available ground truth for studying disagreement patterns. [25] showed that single LLMs fail to capture the distributional properties of human disagreement on ChaosNLI. We extend this line by testing whether multi-model panels can capture human disagreement patterns, finding that correlated errors severely limit their ability to do so.

3 Methodology↩︎

3.1 Datasets↩︎

We use ChaosNLI [6], which provides 100 annotator labels per item. Our primary dataset is ChaosNLI-MNLI (1,599 MNLI items; [24]), with labels entailment (e), neutral (n), or contradiction (c). We replicate on ChaosNLI-SNLI (1,514 SNLI items; [26]) as a same-task robustness check, and on ChaosNLI-AlphaNLI (1,532 abductive NLI items; [27]) as a cross-task replication with a different label set (2-class: hypothesis 1 vs.hypothesis 2) and reasoning type (abductive rather than textual entailment). From each dataset, we sample 1,000 items using entropy-stratified sampling (equal proportions from low, medium, and high human-entropy terciles) with seed 42.

The gold standard for each item is the majority vote of 100 annotators (tie-breaking details in Appendix 17). Human entropy (Shannon entropy, base-2) ranges from 0.00 to 1.58 bits on MNLI/SNLI and 0.00 to 1.00 bits on AlphaNLI (lower maximum due to 2-class), providing rich ground truth against which to validate panel behavior. We present MNLI results in the main body and report SNLI and AlphaNLI replication results in §4.5.

3.2 Judge Panel↩︎

Our panel consists of 9 judges from 7 model families (Table 1). All judges use temperature 0.0 and receive a standardized NLI classification prompt (Appendix 7). Rare parse failures (\(<\)​0.1% of all judgments; 21 of 28 from Llama 4 Maverick, with 5 from Gemini 2.5 Pro and 2 from Claude Sonnet 4.5) are handled via deterministic hash-based random assignment to {e, n, c} to avoid systematic bias. With an odd number of judges, majority-vote ties are eliminated on 2-class tasks (AlphaNLI, RewardBench); on 3-class tasks (MNLI, SNLI), the rare remaining ties (0.4–1.1% of items) are broken via deterministic SHA-256 hashing of the item index and vote sequence, ensuring reproducibility (Appendix 17).

3.3 Effective Sample Size (\(n_{\text{eff}}\))↩︎

We measure effective independence using two complementary approaches.

3.3.0.1 Kish design-effect \(n_{\text{eff}}\).

For each judge, we construct a binary error vector \(\mathbf{e}_j \in \{0,1\}^{1000}\) where \(e_{j,i} = 1\) if judge \(j\) disagrees with the ChaosNLI majority label on item \(i\). We compute the pairwise phi coefficient \(\phi_{jk}\) between all \(\binom{9}{2} = 36\) judge pairs, then apply the Kish formula: \[n_{\text{eff}}= \frac{k}{1 + (k-1)\bar{\phi}} \label{eq:kish}\tag{1}\] where \(k\) is the number of judges and \(\bar{\phi}= \frac{1}{\binom{k}{2}}\sum_{j<k}\phi_{jk}\) is the mean pairwise correlation [28]. For binary error vectors, the phi coefficient reduces to the Pearson product-moment correlation, which is the quantity the Kish formula requires [28]. Alternative association measures (e.g., Cohen’s kappa) conflate prevalence with dependence; phi isolates the linear dependence that directly degrades majority-vote performance. This formula assumes exchangeability (approximately equal pairwise correlations); we validate this assumption against the eigenvalue method below.

3.3.0.2 Eigenvalue \(n_{\text{eff}}\).

As a robustness check that does not assume exchangeability, we compute \(n_{\text{eff}}^{\text{eigen}} = k / \lambda_{\max}\), where \(\lambda_{\max}\) is the largest eigenvalue of the \(k \times k\) phi correlation matrix [29]. Under perfect independence, \(\lambda_{\max} = 1\) and \(n_{\text{eff}}^{\text{eigen}} = k\); under perfect correlation, \(\lambda_{\max} = k\) and \(n_{\text{eff}}^{\text{eigen}} = 1\).

3.3.0.3 Bootstrap confidence interval.

We resample the 1,000 items with replacement 10,000 times, recomputing \(n_{\text{eff}}\) (Kish) for each resample, and report the 2.5th–97.5th percentile interval. This captures uncertainty over the item sample for these specific judges on ChaosNLI; it does not generalize to other datasets or judge panels.

3.4 Condorcet Null Model↩︎

To translate \(n_{\text{eff}}\) into a concrete accuracy gap, we construct a Condorcet null model that simulates what majority-vote accuracy would be if judges voted independently with the same error characteristics. Crucially, this tests conditional independence: whether judges vote independently given the item’s gold label and difficulty level. Some correlation from shared item difficulty is expected and accounted for; the gap measures dependence beyond what difficulty explains.

3.4.0.1 Confusion-matrix calibration.

For each judge \(j\), we estimate the 3\(\times\)​3 confusion matrix \(P(\hat{y} = c' \mid y = c, j)\) from their labels on the 1,000 items. This captures class-specific error patterns (e.g., a judge that confuses entailment with neutral more than with contradiction).

3.4.0.2 Item-aware simulation.

We stratify items into three difficulty bins by human entropy (terciles at the 33rd and 67th percentiles) and estimate per-judge, per-bin confusion matrices. We report results for 3 bins (terciles) as the default; §4 reports sensitivity to 10 bins, and split-half cross-validation confirms minimal overfitting (Appendix 11). For each item, we run 10,000 Monte Carlo simulations: sample each judge’s vote independently from the appropriate bin-specific confusion matrix given the item’s gold label, compute majority vote, and record accuracy. This yields a predicted accuracy for each panel-entropy bin under the independence assumption, accounting for shared item difficulty.

3.4.0.3 Condorcet gap.

The Condorcet gap is the difference between predicted (independent) and actual majority-vote accuracy, computed as a weighted average across panel-entropy bins (weighted by bin size). A negative gap indicates that actual accuracy falls below the independent prediction. We compute a 95% bootstrap confidence interval by resampling items 1,000 times and re-estimating the full pipeline for each resample.

3.5 Statistical Tests↩︎

3.5.0.1 Permutation omnibus test.

To test whether the observed \(\bar{\phi}\) is significantly above chance, we conduct a stratified permutation test (10,000 permutations). Within each human-entropy stratum, we independently shuffle each judge’s error vector, breaking inter-judge correlations while preserving per-judge error rates and the difficulty structure. We compute \(\bar{\phi}\) on each permuted dataset and report the fraction of permuted statistics that equal or exceed the observed value.

3.5.0.2 Per-bin binomial tests.

For each discrete panel-entropy value, we conduct a one-sided binomial test of whether actual accuracy is significantly below the Condorcet prediction, with Wilson score confidence intervals. These per-bin tests are exploratory; the stratified permutation test serves as our primary omnibus significance test.

4 Results↩︎

4.1 Effective Independence↩︎

The 9-judge panel yields \(n_{\text{eff}}= 2.18\) with 95% bootstrap CI \([2.07, 2.31]\) (Table 2). The eigenvalue estimate (\(n_{\text{eff}}^{\text{eigen}} = 2.16\)) closely matches, validating the Kish exchangeability assumption for this panel. The mean pairwise phi is \(\bar{\phi}= 0.391\) (\(\sigma = 0.111\), range: \([0.161, 0.603]\)), and the independence ratio is \(n_{\text{eff}}/ k = 24.2\%\).

The error distribution across items (Figure 1) reveals the signature of correlated errors: 290 items (29%) have all 9 judges correct and 51 (5.1%) have all 9 wrong — far more than any independence model predicts (\(<\)​1). Over-prediction of contradiction accounts for 51% of all-wrong confusions despite comprising only 16.5% of gold labels (Appendix 16).

4.2 Condorcet Gap↩︎

Majority-vote accuracy is 72.0%, compared to the Condorcet prediction of approximately 94% for the item-aware model. The weighted Condorcet gap is 22.0 percentage points (95% CI: \([19.5, 24.1]\)pp). Only 6.8% of this gap is attributable to shared item difficulty; with 10 difficulty bins, the explained fraction rises to 13.5% on MNLI but 66–87% remains unexplained across all datasets. Split-half validation (ratio = 1.00) and the permutation test (\(p < 10^{-4}\)) confirm this is not overfitting.

The per-bin breakdown (Appendix 9, Table 6) shows the gap is significant (\(p < 0.05\)) in 8 of 12 discrete panel-entropy levels. Even for unanimous items (panel entropy = 0, \(n = 319\)), accuracy is 90.9% — not the 99.99% that Condorcet would predict for 9 independent voters each with \(\sim\)​68% accuracy.

4.3 Permutation Test↩︎

The permutation omnibus test yields \(p < 10^{-4}\) (0 of 10,000 permutations reached the observed \(\bar{\phi}= 0.391\); permutation null: mean \(= 0.060\), SD \(= 0.005\), \(z = 65.6\)). This decisively rejects the null hypothesis that the observed inter-judge correlation is attributable to shared item difficulty alone.

4.4 Scaling: \(n_{\text{eff}}(k)\) vs.\(k\)↩︎

Figure 2 shows how \(n_{\text{eff}}\) varies with panel size \(k\) across all \(\binom{9}{k}\) subsets (full data in Appendix 13). The empirical curve closely tracks the Kish prediction \(n_{\text{eff}}(k) = k / (1 + (k-1) \cdot 0.391)\), with a hard asymptote at \(1/\bar{\phi}\approx 2.6\). The diminishing returns are severe: the first 5 judges contribute 90% of the achievable independence (\(n_{\text{eff}}= 1.96\) vs.). Adding judges 6–9 provides only \(+0.22\) effective votes.

4.5 Cross-Dataset Replication↩︎

To test whether the independence deficit generalizes beyond a single NLI source corpus, we replicate the full analysis on 1,000 ChaosNLI-SNLI items [26] (same 3-class task, different corpus) and 1,000 ChaosNLI-AlphaNLI items [27] (2-class abductive reasoning, fundamentally different task type). Table 3 summarizes the comparison.

3pt

The core finding replicates across all three datasets: \(n_{\text{eff}}\) remains in the narrow 2.2–2.5 range despite panel accuracy ranging from 72% to 89%, and the best individual judge matches or outperforms the panel in every case. On MNLI, the panel edges the best judge by a negligible 0.2pp; on SNLI and AlphaNLI, the best individual wins convincingly (\(-\)​6.5pp and \(-\)​2.5pp). The panel underperforms relative to the Condorcet prediction, which already accounts for each judge’s individual accuracy — the gap is driven by correlated errors, not merely vote dilution. The AlphaNLI result is particularly noteworthy: a 2-class abductive reasoning task with fundamentally different cognitive demands, yet \(n_{\text{eff}}= 2.48\). Condorcet gaps decrease with base accuracy (22.0pp \(\to\) 14.0pp \(\to\) 7.6pp), as expected when higher accuracy leaves less room for correlated errors.

Krippendorff’s \(\alpha < 0.667\) on all datasets, indicating only moderate agreement by annotation standards [30]. Human annotator panels achieve roughly \(2\times\) higher \(n_{\text{eff}}\) (\(4.0\)–\(5.8\) vs.LLMs’ \(2.2\)–\(2.5\)), suggesting the deficit is specific to LLM judges.¹

Split-half cross-validation confirms no overfitting (ratios 0.96–1.00), and \(n_{\text{eff}}\) stabilizes by \(N \approx 200\) (Appendix 12).

4.6 Robustness to Prompt, Temperature, and Task↩︎

To test whether the independence deficit is a prompt or decoding artifact, we re-run all 9 judges on the same 1,000 MNLI items under four variants: (1) reframed wording, (2) reversed label order, (3) chain-of-thought reasoning, and (4) temperature \(T = 0.5\). We also evaluate on 1,000 RewardBench items [31] — a pairwise preference task with deterministic gold labels, sampled via proportional stratified sampling across four categories, using the official MT-Bench pairwise judge prompt with A/B position randomization.

3pt

Table 4 shows that \(n_{\text{eff}}\) is remarkably stable. Varying prompt wording, label ordering, and temperature has essentially no effect: \(n_{\text{eff}}\) ranges from 2.15 to 2.18, with overlapping 95% bootstrap CIs. The reversed-label variant rules out position bias [8] as a driver: the near-identical \(n_{\text{eff}}\) (2.15 vs.) confirms that correlation is robust to label ordering. Chain-of-thought actually increases correlation (\(\bar{\phi}= .456\), \(n_{\text{eff}}= 1.94\)) — shared reasoning amplifies shared errors. This stability rules out prompt engineering artifacts.

On RewardBench — a binary pairwise preference task with deterministic gold labels — \(n_{\text{eff}}= 1.99\) [1.83, 2.20]. The smaller Condorcet gap (6.8pp vs.–22pp on MNLI) reflects higher panel accuracy (92.7%), but \(\phi = 0.44\) confirms high error correlation regardless of task type. Same-family correlation is larger on RewardBench (\(+0.109\)) than MNLI (\(+0.047\)). All 9 judges show residual A-preference despite the anti-bias prompt; the NLI results, structurally immune to position effects, confirm that correlation is not a position-bias artifact.

5 Analysis and Discussion↩︎

5.1 Leave-One-Out: Which Judges Matter?↩︎

Leave-one-out analysis (Appendix 14, Table 9) reveals that herding is systemic: \(\Delta n_{\text{eff}}\) ranges narrowly from \(-0.13\) to \(+0.02\) across judges, with no single model driving the effect. Removing DeepSeek-V3 or Mistral Large 3 increases \(n_{\text{eff}}\) (their errors are most correlated with the panel), while removing Llama 4 Scout decreases it the most.

Most strikingly, removing Gemini 2.5 Pro — highly correlated with Claude (\(\phi = 0.60\)) and GPT-4o (\(0.52\)) — increases accuracy by 1.3pp (95% CI \([+0.1, +2.6]\)), and 6 of 9 removals improve accuracy. The three judges whose removal hurts (Maverick, Scout, Qwen3) include the two most individually accurate. That adding voters can hurt is theoretically predicted under positive correlation [18], but has not previously been demonstrated in the LLM judge setting.

5.2 Stratified and Subset Analyses↩︎

When stratified by gold NLI class (Appendix 14, Table 10), herding is present across all three classes: \(n_{\text{eff}}\) ranges from 1.85 (contradiction, \(\bar{\phi}= 0.482\)) to 2.40 (neutral). Even on the 179 “easy” items (17.9%) where \(\geq\)​80% of human annotators agree, \(n_{\text{eff}}= 2.67\) — higher than the full set but far from 9, ruling out the explanation that herding is merely a response to item ambiguity. Panel entropy correlates with human entropy (\(\rho_s = 0.301\)) and predicts majority-vote correctness (\(r_{pb} = -0.342\)). Among unanimous items (\(n = 319\)), accuracy is 90.9%; with any disagreement (\(n = 681\)), it drops to 63.1%. The 9.1% error rate on unanimous items is dramatically higher than the \(\sim\)​0.02% that independence would predict.

5.3 Same-Family vs.Cross-Family Correlation↩︎

Same-family pairs (OpenAI-OpenAI: \(\phi = 0.437\); Meta-Meta: \(\phi = 0.435\)) are only slightly more correlated than the cross-family mean (\(\bar{\phi}_{\text{cross}} = 0.389\), difference = 0.047). The three highest-correlated pairs are all cross-family: Claude \(\times\) Gemini (\(\phi = 0.603\)), GPT-4o \(\times\) Claude (\(\phi = 0.588\)), and Mistral \(\times\) DeepSeek (\(\phi = 0.564\)). When restricted to one judge per family (7 judges, selecting the best in each), \(n_{\text{eff}}\) decreases to 1.93 — a selection effect where the best judges concentrate errors on the same hard items (full matrix in Appendix 8). Family diversity alone does not recover independence.

5.4 Does Smarter Aggregation Help?↩︎

A natural question is whether the Condorcet gap can be closed by replacing naïve majority voting with more sophisticated aggregation. We test three established methods: (1) Dawid-Skene EM [7], which estimates per-judge confusion matrices and true label posteriors via expectation-maximization without access to gold labels; (2) accuracy-weighted voting, which weights each judge by their individual accuracy (using 5-fold cross-validation to avoid label leakage); and (3) Markowitz-optimal weighting, which selects weights to minimize correlated error via the inverse phi correlation matrix (also cross-validated). The latter two methods use gold labels for weight estimation — giving them an oracle advantage that would be unavailable in practice.

2.5pt

Table 5 shows the results. On MNLI, accuracy-weighted voting (5-fold CV) achieves 72.2% — a gain of just 0.2pp over majority vote, closing less than 1% of the 22.0pp Condorcet gap. Dawid-Skene actually underperforms majority vote on MNLI (70.7%), illustrating that unsupervised EM can misestimate error rates when judges are highly correlated. On AlphaNLI, Dawid-Skene closes 10.5% of the gap — the best stable result across all four datasets. On SNLI, AlphaNLI, and RewardBench, the best individual judge outperforms every aggregation method, including those with oracle access. Note that identifying the best individual also requires oracle access to gold labels; the comparison highlights that even oracle-informed weighting cannot overcome correlation. Markowitz-optimal (phi-optimal) weighting closes 20.6% of the gap on RewardBench but underperforms majority vote on AlphaNLI, illustrating the instability of correlation-based weighting (Appendix 15).

With only \(\sim\)​2.2 effective independent votes, no weighting scheme can extract a third independent perspective — including calibrated soft voting [32], [33]. Our oracle-access stable methods close at most 11% of the gap. Confounder-aware methods [20] face the same structural limit: same-family pairs are only marginally more correlated than cross-family (\(+\)​0.047; §5.3), and the three highest-correlated pairs are all cross-family.

6 Conclusion↩︎

We have applied the Kish effective sample size framework to LLM judge panels, providing the first measurement that directly connects inter-judge correlation to majority-vote reliability via Condorcet theory. The independence deficit (\(n_{\text{eff}}\approx 2.0\)–\(2.5\)) is stable across three NLI datasets, three prompt variants, two temperature settings, and a pairwise preference task (RewardBench), confirming that the correlation is structural rather than an artifact of any particular experimental choice. Adding judges does not help: the panel matches or underperforms the best individual judge across all conditions. Established stable aggregation methods close at most 11% of the Condorcet gap (unstable correlation-aware weighting reaches 21% on one dataset but hurts on others), confirming that the bottleneck is in the inputs, not the algorithm.

These results have direct practical implications. Paying for 9 opinions but receiving the informational equivalent of \(\sim\)​2 is a substantial inefficiency: a 5-judge panel already captures 90% of achievable independence. Unanimous panel agreement carries far less weight than it appears — our data show a 9.1% error rate on unanimous items, vs.\(\sim\)​0.02% under independence. We recommend computing \(n_{\text{eff}}\) as a standard panel diagnostic: if \(n_{\text{eff}}/ k < 0.5\), results should be treated with caution.

Our findings complement [15] and [14]. The path forward requires models that genuinely differ in how they process information — not merely different brand names on similar architectures. The Kish formula makes progress measurable: halving \(\bar{\phi}\) from 0.39 to 0.20 would raise \(n_{\text{eff}}\) from 2.2 to 3.5, closing roughly half the Condorcet gap. Whether architecturally diverse models, specialist fine-tuning, or hybrid human-LLM panels can achieve this remains an open question.

Limitations↩︎

6.0.0.1 Classification tasks.

Our results are replicated across three ChaosNLI NLI datasets and a pairwise preference task (RewardBench), with consistent \(n_{\text{eff}}\approx 2.0\)–\(2.5\). The cross-task replication strengthens generalizability, but all four remain classification or binary preference tasks. The degree of inter-judge correlation may differ on open-ended generation evaluation or code review, where output structure differs fundamentally.

6.0.0.2 Gold standard validity.

The 100-annotator majority label is our ground truth, but for high-entropy items, the majority label may represent a plurality preference rather than a “correct” answer. Appendix 10 reports distributional alignment metrics confirming the same pattern without reducing labels to binary accuracy.

6.0.0.3 Snapshot in time.

Our results reflect a snapshot of current frontier models. Future models may exhibit lower correlation, but the framework (\(n_{\text{eff}}\) and Condorcet gap) remains applicable.

6.0.0.4 Condorcet model calibration.

The confusion matrices are estimated from the same items on which we measure the gap. Split-half cross-validation (Appendix 11) yields overfitting ratios of 0.96–1.00 across the three datasets, confirming negligible overfitting.

6.0.0.5 Prompt and decoding choices.

We test four prompt variants (including chain-of-thought) and two temperature settings (§4.6), finding \(n_{\text{eff}}\) stable in the 1.94–2.18 range. Chain-of-thought actually increases correlation (\(n_{\text{eff}}= 1.94\)). More radical prompt reformulations — such as few-shot exemplars or persona-based prompting — could in principle alter the correlation structure. We use RewardBench [31] rather than the more recent RewardBench 2 [34] because the latter uses LLM-derived gold labels for several categories (e.g., GPT-4o and Claude consensus for Factuality), which would introduce circularity when evaluating LLM judges drawn from the same model families.

6.0.0.6 Bootstrap CI interpretation.

Our bootstrap CI captures uncertainty over items for these specific judges. It does not account for judge selection uncertainty — a different panel might yield a different \(n_{\text{eff}}\).

Ethics Statement↩︎

This work uses publicly available benchmark data (ChaosNLI) and commercial LLMs. No human subjects were recruited for this study. The ChaosNLI annotations were collected by [6] and are publicly released. Our findings highlight limitations of LLM judge panels, which we believe serve the public interest by encouraging more careful deployment of automated evaluation systems. Claude (Anthropic) was used for writing assistance.

Data Availability↩︎

ChaosNLI-MNLI is publicly available via HuggingFace (metaeval/chaos-mnli-ambiguity); ChaosNLI-SNLI and ChaosNLI-AlphaNLI are available from the ChaosNLI GitHub repository [6]. RewardBench is available via HuggingFace (allenai/reward-bench; [31]).

7 NLI Classification Prompt↩︎

All judges receive the following prompt at temperature 0.0:

Given the following premise and hypothesis, determine the relationship between them.

Premise: {premise}

Hypothesis: {hypothesis}

What is the relationship? Reply with ONLY one word: “entailment”, “neutral”, or “contradiction”.

We use the canonical NLI label vocabulary (entailment, neutral, contradiction) rather than crowdsourcing-style phrasing (“definitely true”, “might be true”, “definitely not true”) because LLMs are trained primarily on NLI benchmark data that uses these terms; non-standard labels risk introducing an additional source of variance unrelated to the underlying judgment.

Parse failures are handled by deterministic hash-based random assignment to {e, n, c}, ensuring reproducibility and avoiding systematic bias.

7.1 RewardBench Pairwise Preference Prompt↩︎

For RewardBench evaluation, we use the default pairwise judge prompt from the RewardBench codebase [31], originally from MT-Bench [1]. The RewardBench codebase provides several prompt variants for different judge architectures (e.g., Prometheus, OffsetBias, Atla); we use the default MTBENCH_V2 template, which is the standard for generative LLM-as-judge evaluation on the RewardBench leaderboard. The system message instructs judges to act as impartial evaluators, provide a short explanation, and output [[A]] or [[B]]. The user message presents the prompt and both responses with structured delimiters. All judges use temperature 0.0 and max_tokens=4096.

8 Phi Correlation Matrix↩︎

Figure 3 shows the full \(9 \times 9\) pairwise phi correlation matrix, ordered by hierarchical clustering. Notable high-correlation pairs: Claude Sonnet \(\times\) Gemini 2.5 Pro (\(\phi = 0.603\)), GPT-4o \(\times\) Claude Sonnet (\(\phi = 0.588\)), Mistral Large 3 \(\times\) DeepSeek-V3 (\(\phi = 0.564\)). Notable low-correlation pair: Gemini 2.5 Pro \(\times\) Llama 4 Scout (\(\phi = 0.161\)).

9 Condorcet Gap Visualization↩︎

Figure 4 shows the per-bin Condorcet gap. The vertical distance between the Condorcet prediction (red) and actual accuracy (blue) visualizes the independence deficit: the gap is largest for moderate-entropy items where independent voting should still yield high accuracy but actual voting fails.

10 Distributional Alignment Analysis↩︎

Our main analyses use binary accuracy (match to the 100-annotator majority label). Since ChaosNLI involves 3-way classification with genuine human disagreement, we verify our findings using distributional metrics that do not reduce labels to right/wrong.

For each item, we compare the panel’s label distribution (9 votes over {e, n, c}) to the human distribution (100 annotations) using total variation (TV) distance and symmetric KL divergence. Table 7 reports results stratified by human-entropy tercile.

TV distance correlates strongly with human entropy (\(\rho_s = 0.434\), \(p < 10^{-46}\)): the panel’s distributional misalignment grows with item ambiguity. This parallels the Condorcet gap finding (§4.2) but does not depend on reducing the 3-class problem to binary accuracy. On low-entropy items where accuracy is a clean metric (mean TV \(= 0.192\)), the panel is reasonably well-calibrated; on high-entropy items (mean TV \(= 0.345\)), the panel’s distribution diverges substantially from the human distribution — consistent with correlated judges collapsing onto a single class rather than reflecting the human spread.

11 Split-Half Condorcet Validation↩︎

To verify that our Condorcet gap estimate is not inflated by overfitting, we perform split-half cross-validation. We split the 1,000 items into two halves (500 each), stratified by human-entropy tercile to preserve difficulty balance. We estimate per-judge, per-bin confusion matrices on half A, simulate the Condorcet prediction on half B, and vice versa.

The cross-validated weighted Condorcet gap is 21.9pp for MNLI, closely matching the in-sample estimate of 22.0pp (overfitting ratio = 0.997). Across all three datasets, the overfitting ratios are 0.997 (MNLI), 0.960 (SNLI), and 1.000 (AlphaNLI) — all near unity, confirming that with 1,000 items and 9-parameter confusion matrices per judge, the gap estimate is stable and not an artifact of in-sample fitting.

12 Sample Size Convergence↩︎

A natural concern is whether 1,000 items provide a stable estimate of \(n_{\text{eff}}\). Figure 5 shows \(n_{\text{eff}}\) computed on entropy-stratified subsamples of increasing size (\(N \in \{100, 200, 300, 400, 500, 750, 1000\}\)). For each \(N < 1{,}000\), we draw 100 independent stratified subsamples and report the mean \(n_{\text{eff}}\) with 2.5th–97.5th percentile bands; for \(N = 1{,}000\) (the full dataset), we report the bootstrap CI (10,000 resamples). Using multiple independent subsamples per \(N\) avoids the non-monotonic artifacts that a single random draw can produce. The estimate stabilizes by \(N \approx 200\)–300, and the interval narrows substantially by \(N = 500\). This confirms that our 1,000-item sample provides a reliable measurement.

13 Scaling Curve Data↩︎

14 Supplementary Analysis Tables↩︎

Tables 9 and 10 provide the full leave-one-out and per-class stratification data referenced in §5.

15 Aggregation Method Details↩︎

Table 11 shows the full aggregation comparison including phi-optimal (Markowitz) weighting, which is omitted from the main table due to instability. Phi-optimal weighting selects judge weights via the inverse phi correlation matrix to minimize correlated error (analogous to Markowitz portfolio optimization). While it achieves the highest accuracy on MNLI (72.4%) and RewardBench (94.1%, closing 20.6% of the gap), it hurts performance on AlphaNLI (86.2%, below MV), suggesting overfitting to the correlation structure. Cross-validated estimates reduce this instability but confirm the same conclusion: no method meaningfully closes the Condorcet gap.

3pt

16 All-Wrong Item Analysis↩︎

Table 12 breaks down the 51 items where all 9 judges are wrong.

                    |

                    |

                    |

17 Tie-Breaking↩︎

17.0.0.1 Gold-label ties (100 annotators).

With 100 annotators, ties are possible: 28 items across our three 1,000-item datasets have exactly tied top labels (e.g., 49n/49c/2e). The ChaosNLI dataset resolves these via Python Counter insertion order — an undocumented artifact, not an intentional design choice. We verified that our results are robust: flipping all 28 tied items to the alternative label changes panel accuracy by at most \(\pm\)​0.7pp and \(n_{\text{eff}}\) by at most 1.3%, both well within bootstrap confidence intervals.

17.0.0.2 Majority-vote ties (9 judges).

With 9 judges (an odd number), ties are impossible on 2-class tasks (AlphaNLI, RewardBench). On 3-class tasks (MNLI, SNLI), ties can still occur in 3-3-3 or 4-4-1 configurations, but are rare (MNLI: 11 ties or 1.1%; SNLI: 4 ties or 0.4%). These are broken via deterministic SHA-256 hashing of the item index and vote sequence, ensuring reproducibility and avoiding insertion-order bias. Monte Carlo simulations use random tie-breaking (seeded RNG) to correctly average over tie outcomes. Tie frequencies: MNLI 11/1,000 (1.1%; 4 at 3-3-3, 7 at 4-4-1), SNLI 4/1,000 (0.4%; all 4-4-1), AlphaNLI 0/1,000 (0.0%; ties impossible with 9 judges on a 2-class task), RewardBench 0/1,000 (0.0%).

References↩︎