1 Introduction↩︎

Modern LLM leaderboards rank models by percentage-point gaps on shared-prompt benchmarks. A leaderboard saying “model \(A\) scores \(78.3\%\), model \(B\) scores \(77.5\%\)” converts a \(0.8\)-point gap into headlines and product decisions; but the assertion that \(A\) is meaningfully better than \(B\) is a statistical claim about the gap, not the gap itself. On the four-task Open LLM Leaderboard v1, \(11\) of \(40\) displayed pairwise rankings do not meet the paired-test resolution target at \((\alpha,1{-}\beta){=}(0.05,0.8)\); on the MMLU-Pro top-\(10\), \(4\) of \(9\) adjacent-rank pairs remain unresolved at the actual benchmark size \(N{=}12{,}032\). Concretely, the displayed gap between gemma-7B and Llama-3-8B on HellaSwag is \(\hat{\delta{=}}{+}0.46\) pp at \(n{=}10{,}042\): significant by asymptotic \(\chi^2_1\) (\(p{=}0.049\)), not by the exact conditional binomial (\(p{=}0.054\)), with a paired-bootstrap \(95\%\) CI on \(\bar D\) containing zero. This is the kind of claim a resolution diagnostic surfaces. Whether a leaderboard has the resolution to support these claims depends on its size, on the within-prompt structure of the data, and on how many other adjacent gaps share the table.

The natural inference is a paired hypothesis test: for binary accuracy, McNemar’s test on the discordant pairs with the corresponding required-\(N\) [1], [2]; for graded scores, a paired-\(t\) or paired bootstrap. The machinery is classical. What is missing is a resolution-reporting protocol that says, given the size and structure of a benchmark, what gaps can be distinguished from sampling noise at conventional Type-I and Type-II error. Power calculators in common use focus on unpaired comparisons [3], and there is no standard for what to publish alongside a headline gap.

We treat shared-prompt LLM benchmarks as paired hypothesis-testing problems and derive a resolution-reporting framework by inverting level-\(\alpha\), power-\((1{-}\beta)\) tests. The framework yields three quantities: the minimum detectable effect (MDE) at the current \(N\); the required paired sample size \(N^{\star}\) at a target effect; and the resolution ratio \(q=N/N^{\star}\).

The methodological building blocks are classical: Wald inversion, McNemar-Connor required-\(N\), Bonferroni/Holm multiplicity, design-effect cluster correction, and anytime-valid e-processes. This paper contributes three things on top. First, a sharp characterisation of a specific misuse pattern (one new lemma with an explicit constant). Second, an empirical demonstration that the resulting diagnostic flags several displayed rankings as unresolved at the target \((\alpha, 1{-}\beta)\) resolution level. Third, a packaged reporting protocol that exposes each diagnostic entry as a one-line call. Concretely:

New theory.

• A sharp small-effect expansion (1) for the unpaired Cohen-\(h\)-plus-\((1{-}\rho)\) shortcut: the ratio \(n_h/N^{\star}\) deviates from \(\tfrac12\) by at most \(C(p,\rho)\delta^2 + O(\delta^4)\) with an explicit second-order constant \(C(p,\rho)\) that lets practitioners compute the admissible \(\delta^{\star}\) at which the shortcut is reliable (1). The leading-order factor of two is not new; the explicit constant and the uniform convergence on compact admissible sets are.

New empirical findings.

• A calculator-misuse characterisation across five power-analysis tools at the worked example \((p_A,p_B,\rho){=}(0.65,0.60,0.30)\): three of five (Cohen’s 1988 textbook formula, G*Power 3.1, R pwr) silently underestimate required-\(N\) by a factor of two when their per-arm output is post-multiplied by \((1{-}\rho)\) (1). The misuse pattern is easily reproduced on every widely-used calculator we tried; whether it occurs in published LLM-evaluation work is a separate empirical question that we do not claim to settle here.

• A real-leaderboard reanalysis with prospective validation: paired-McNemar required-\(N\) is median \(2.15{\times}\) smaller than the unpaired Gaussian formula of [3] on the same data (IQR \([1.60,2.75]\); 1), and on three real OLL v1 pairs the framework’s \(N^{\star}\) prescription delivers empirical McNemar power \(0.80\pm 0.03\) over \(M{=}1000\) bootstrap trials, with sub-prescription (\(0.8\,N^{\star}\)) and super-prescription (\(1.2\,N^{\star}\)) correctly under- and over-shooting ([tab:prospective]). The diagnostic is calibrated on data, not just in asymptotic theory.

• Real subject-level cluster sensitivity: using MMLU-Pro’s \(14\) subject categories as natural clusters, the unresolved adjacent-pair count rises from \(4/9\) at IID to \(6/9\) (4); two pairs flip from comfortably resolved to \(N^{\star} > 3N\). A category-bootstrap CI on the cluster-corrected unresolved count and LOSO sensitivity (13) both support the flip.

Methodological integration.

• Multiplicity, multi-arbiter, and anytime-valid sequential stress tests applied jointly to a single leaderboard family (§6–§6.5). Each component is classical; the joint pipeline and per-pair verdict table (5) are, to our knowledge, the first end-to-end resolution characterisation of a public LLM leaderboard.

• A pip-installable package, llm-power, exposing the diagnostic for benchmark designers and leaderboard maintainers.

The empirical verdicts in §6 use 6 directly and do not depend on the explicit constant in 1; the lemma’s role is to sharply quantify the well-known factor-of-two pattern in a closed form practitioners can use to pre-screen their tools.

1.0.0.1 Paper roadmap.

§3 defines the inversion and the resolution ratio \(q\). §4 instantiates for paired-binary accuracy and proves the shortcut lemma; §4.1 traces it through five real calculators. §5 reports finite-sample calibration. §6 applies the diagnostic to OLL v1 and MMLU-Pro; §6.3–§6.5 stress-test the verdict under multiplicity, real subject clustering, and anytime-valid sequential testing. The headline empirical finding is 5; the headline theoretical finding is 1.

2 Related work↩︎

2.0.0.1 Paired-binary tests.

McNemar’s test [1] conditions on the discordant-pair count \(b{+}c\). The large-sample required-\(N\) formula is [2]; mid-\(p\) [4] and continuity-corrected variants [5] are the standard small-\(b{+}c\) upgrades.

2.0.0.2 Power for NLP/LLM evaluation.

[6] argued NLP comparisons are routinely underpowered; [7] survey appropriate tests, including McNemar for paired binary outcomes. Methodologically closest is [3], who gave a closed-form required-\(N\) for the unpaired Gaussian-accuracy case. The unpaired formula is appropriate for independent samples; our comparison quantifies the efficiency loss when the evaluation design is actually paired. On the same \(40\) OLL v1 pairs we study (§6.1), the paired McNemar required-\(N\) is a median \(2.15\times\) smaller than Miller’s unpaired formula at the empirical \(\hat{\rho}\) (IQR \([1.60, 2.75]\), range \([1.33, 5.39]\)); the empirical efficiency gain matches the textbook prediction \(1/(1{-}\rho)\) for paired vs.unpaired Wald tests (1, mean residual \(-0.009\), max \(0.062\)).

[8] measure benchmark variance across \(13\) tasks. [9] give a clustered bootstrap for ability-estimation precision; that is an estimator-variance statement, distinct from test power. [10] subsample benchmarks for efficient point-estimate accuracy, also distinct from the required-\(N\) for paired hypothesis tests we address. Construct-validity critiques [11]–[13] are orthogonal: a benchmark can have high construct validity and still be too small for the test at hand.

To our knowledge, prior work treats these elements separately. The present paper integrates paired-difference variance with test inversion, leaderboard-scale multiplicity, and a per-pair resolution diagnostic on a real LLM benchmark, and packages the result as a reusable tool.

3 Methodology↩︎

3.0.0.1 Paired setup.

Two models \(A, B\) are evaluated on the same \(N\) prompts, which we treat as an i.i.d.sample from a target prompt superpopulation; without this, the gap on exactly these items is a fixed quantity and no hypothesis test is needed. Per-prompt scores \(X_i^A, X_i^B\) may be binary, graded in \([0,1]\), or real-valued. Define the paired difference \(D_i = X_i^A - X_i^B\) and the paired-mean estimator \(\hat{\delta} = N^{-1}\sum_i D_i\).

3.0.0.2 Test and inversion.

Under a normal approximation, \[\label{eq:wald} T_N = \hat{\delta}/\widehat{\mathrm{SE}}(\hat{\delta}), \quad \widehat{\mathrm{SE}}(\hat{\delta}) = \sigma_D/\sqrt{N},\tag{1}\] where \(\sigma_D^2 = \mathrm{Var}(D_i)\). For two-sided level-\(\alpha\) tests, \(H_0: \mathbb{E}[D_i] = 0\) is rejected when \(|T_N| \geq z_{1-\alpha/2}\). Under a fixed alternative \(\mathbb{E}[D_i] = \delta\), \[\label{eq:power} 1-\beta = \Phi\!\left(\frac{|\delta|}{\sigma_D}\sqrt{N} - z_{1-\alpha/2}\right) + O(N^{-1/2}),\tag{2}\] where the \(O(N^{-1/2})\) term is the standard finite-sample remainder of the normal approximation [14]; 5 characterises it empirically.¹ Inverting at target power gives \[\begin{align} N^{\star}(\delta;\alpha,\beta) &= \!\left(\!\frac{(z_{1-\alpha/2}{+}z_{1-\beta})\,\sigma_D}{|\delta|}\!\right)^{\!2}\!, \tag{3}\\ \delta_{\mathrm{MDE}}(N;\alpha,\beta) &= \frac{(z_{1-\alpha/2}{+}z_{1-\beta})\,\sigma_D}{\sqrt{N}}. \tag{4} \end{align}\]

For a single pair, \(q\) is a deterministic monotone transform of the squared Wald statistic: \(q = T_N^2/(z_{1-\alpha/2}+z_{1-\beta})^2\), so \(q\ge 1 \Leftrightarrow |T_N|\ge z_{1-\alpha/2}+z_{1-\beta} \approx 2.80\). Per-pair \(q\) therefore carries no information beyond the (Wald-statistic) \(p\)-value; its value-add over a \(p\)-value is interpretive, plus the aggregation layer (multiplicity, clustering, anytime-validity) where the \(q\) scale composes more naturally than \(p\)-values do.

\(q < 1\) does not assert equality of the two models, nor does it overturn a fixed-\(N\) \(p\)-value: it says the benchmark of this size does not have the target resolution for a gap of the displayed magnitude under the stated \((\alpha,1{-}\beta)\) operating point. The HellaSwag boundary pair in §1 illustrates the distinction: it rejects asymptotically at \(p{=}0.049\) yet has \(q\approx \tfrac12\), so a nominally significant gap sits only halfway to the \((0.05, 0.8)\) resolution target. We use \(q\) as the paper’s load-bearing reporting quantity throughout.

3.0.0.3 Prospective vs.diagnostic.

3 has two uses. Plugging in a pre-specified target \(\delta\) gives a prospective sample-size requirement for benchmark designers. Plugging in the observed \(\hat{\delta}\) gives a resolution diagnostic. We do not compute power at the observed effect to argue that a particular ranking is true or false (the misuse criticised by [15]); instead we compute \(N^{\star}(\hat{\delta})\) as a benchmark-design diagnostic: the required \(N\) to detect a gap of the observed magnitude, irrespective of the significance verdict on the current sample.

3.0.0.4 Paired variance and multiplicity.

On shared prompts, \(\sigma_D^2 = \mathrm{Var}(X^A) + \mathrm{Var}(X^B) - 2\,\mathrm{Cov}(X^A,X^B)\). State-of-the-art LLMs solve overlapping subsets of items, so \(\mathrm{Cov}(X^A,X^B)\) is large and independent-proportion variances inflate MDE and \(N^{\star}\). A \(K\)-model leaderboard simultaneously displays up to \(\binom{K}{2}\) pairwise tests; replacing \(\alpha\) with a Bonferroni-, Holm- [16] or BH-adjusted \(\alpha'\) [17] in 3 directly inflates \(N^{\star}\). Family-level resolution is the relevant object whenever a leaderboard summarises many pairs at once.

3.0.0.5 Sequential / anytime-valid extension.

Public leaderboards update continuously, so the relevant \(N\) is a stopping time chosen post-hoc. The same \(\sigma_D^2\) plugs into a confidence sequence [18], [19] or a paired-Bernoulli mixture e-process, replacing \(z_{1-\alpha/2}\) with a time-uniform boundary \(u(n)\) satisfying \(\Pr(\sup_n |T_n| \geq u(n)) \leq \alpha\) under \(H_0\); 6.5 reports an \({\approx}2{\times}\) threshold inflation and one additional unresolved MMLU-Pro pair.

4 Paired-binary instantiation and the shortcut lemma↩︎

For binary accuracy the paired-difference variance is \[\label{eq:vardiff} \sigma_D^2 = p_A q_A + p_B q_B - 2\rho\sqrt{p_A q_A\,p_B q_B}, \;\, q_\cdot{=}1{-}p_\cdot,\tag{5}\] with \(\rho\) the within-pair Bernoulli correlation. Substitution into 3 yields the McNemar-Connor required paired count \[\label{eq:reqn95exact} N^{\star} = \frac{(z_{1-\alpha/2}+z_{1-\beta})^2\,\sigma_D^2}{(p_A-p_B)^2},\tag{6}\] which agrees asymptotically with the \(\chi^2_1\) McNemar test on the discordant counts \((b,c)\sim\mathrm{Binomial}(b{+}c,\tfrac12)\) under \(H_0\) [1], [2].

4.0.0.1 Admissible correlations.

Not every \(\rho\) is achievable for given marginals \((p_A,p_B)\): the Hoeffding bound on Bernoulli correlation pins \(\rho\) to an interval \([\rho_{\min},\rho_{\max}]\) with \[\begin{align} \rho_{\max} &= \sqrt{\tfrac{\min\{p_A(1{-}p_B),\,(1{-}p_A)p_B\}}{\max\{p_A(1{-}p_B),\,(1{-}p_A)p_B\}}},\\ \rho_{\min} &= -\sqrt{\tfrac{\min\{p_A p_B,\,(1{-}p_A)(1{-}p_B)\}}{\max\{p_A p_B,\,(1{-}p_A)(1{-}p_B)\}}}. \end{align}\] 5 requires \(\rho\) to lie in this admissible interval; we restrict all numerical claims accordingly.

4.0.0.2 The unpaired-to-paired shortcut.

A natural temptation, when only an unpaired Cohen-\(h\) calculator is on hand, is to read off the per-arm \(n_{\mathrm{unp}} = (z_{1-\alpha/2}+z_{1-\beta})^2/h^2\) [20] with \(h = 2\arcsin\sqrt{p_A} - 2\arcsin\sqrt{p_B}\), then apply Cohen’s generic \((1{-}\rho)\) paired adjustment [20]: \[\label{eq:reqn95h} n_h = (1-\rho)\,(z_{1-\alpha/2}+z_{1-\beta})^2/h^2.\tag{7}\] This is a natural shortcut when only an unpaired Cohen-\(h\) calculator is to hand (see the comparison of §4.1). 1 is a sharp small-effect characterisation of this failure mode: not a foundational theorem about paired tests, but a tight second-order expansion for a misuse that these tools make easy. The shortcut deviates from \(N^{\star}\) by approximately a factor of two in the close-comparison limit; the deviation of \(n_h/N^{\star}\) from \(\tfrac12\) is \(O(\delta^2)\), not \(O(\delta)\), and the \(\rho\)-dependent term in \(C\) vanishes at \(p=\tfrac12\) (relevant since many benchmark accuracies cluster near \(\tfrac12\)). The technical contribution is the explicit constant \(C(p,\rho)\) in 9 .

Interpretation. The shortcut \(n_h\) is approximately one half of \(N^{\star}\) at small \(\delta\), with the deviation above or below \(\tfrac12\) depending on \((p,\rho)\). The leading-order factor of two follows from \(\mathrm{Var}(X^A-X^B)\) algebra and is not new; the explicit constant and the uniform convergence are what 1 relies on to deliver a usable admissible \(\delta^\star\).

4.0.0.3 When the shortcut matters operationally.

In the close-comparison regime, the shortcut produces approximately \(N^\star/2\) with a deviation that is \(O(\delta^2)\). What changes with \(|\delta|\) is the operational consequence. A verdict flips (resolved vs.unresolved at benchmark size \(N\)) only when \(N^{\star}\) and \(n_h\) straddle \(N\). For cross-tier comparisons where \(\max(N^{\star}, n_h) \ll N\), both return “resolved” and the underestimate is benign. The verdict-changing regime is precisely the close-comparison regime that dominates leaderboard adjacencies: \(17/40\) OLL v1 pairs have \(|\hat{\delta}|\leq 5\) pp (3) and all \(9\) MMLU-Pro top-\(10\) adjacent pairs have \(|\hat{\delta}|\leq 7\) pp ([tab:discord95mmlupro]). The shortcut is material exactly where adjacent-rank claims rest, and benign where cross-tier comparisons would resolve either way.

Proof sketch (full proof in 8). The strategy is to Taylor-expand both \(h^2\) and \(\sigma_D^2\) around the midpoint \(p\), exploiting the symmetry of \(\arcsin\sqrt{\cdot}\) to kill the linear-in-\(\delta\) terms. With \(u = p(1{-}p)\), the result is \(h^2 = (\delta^2/u)\cdot[1 + \delta^2\cdot H_2 + O(\delta^4)]\) where \(H_2 = 1/(12u) + (1{-}2p)^2/(16u^2)\), and \(\sigma_D^2 = 2u(1{-}\rho) + (\delta^2/4)\cdot[\rho(1{-}2p)^2/u - 2(1{-}\rho)] + O(\delta^4)\). Substituting and combining the two \(O(\delta^2)\) corrections, \(n_h/N^{\star} = \tfrac12 - \tfrac{\delta^2}{2}\bigl[(1{+}\rho)(1{-}2p)^2/(16(1{-}\rho)u^2) - 1/(6u)\bigr] + O(\delta^4),\) from which 9 follows. 0◻

4.0.0.4 Numerical check.

Per-cell verification on a \((p,\rho,\delta)\) grid (e6_ratio_heatmap.csv; full details in 8) matches \(C(p,\rho)\) to four significant figures at small \(\delta\). At \(\delta{=}0.05\), \(|n_h/N^{\star} - \tfrac12| \leq 0.0008\); at \(\delta{=}0.20\), \(\leq 0.014\). On the \(40\) OLL v1 pairs (§6.1), the empirical ratio \(n_h/N^{\star}\) has median \(0.5002\), IQR \([0.4999, 0.5035]\), range \([0.487, 0.562]\); the \(7\) close-comparison pairs (\(|\hat{\delta}|\leq 2\%\)) hit \(\tfrac12\) to four decimals.

The structural cause is that \((1{-}\rho)\) is correctly applied to the single-arm variance \(p(1{-}p)\), but the paired-difference variance 5 carries an additional factor of two from \(\mathrm{Var}(X^A){+}\mathrm{Var}(X^B)\). The fix is to use the paired-difference variance directly. We show next that this misuse pattern occurs in widely-used power tools.

4.1 Calculator-misuse comparison↩︎

2pt

1 traces the comparison. Three of the five tools (Cohen’s textbook, G*Power 3.1, R’s pwr) reproduce the shortcut’s factor-of-two underestimate when \((1{-}\rho)\) is applied to a per-arm \(K/h^2\) output, because the paired adjustment is being applied to a single-arm variance rather than to \(\mathrm{Var}(\Delta)\). The remaining two recover the correct \(N^{\star} \approx 1{,}028\): statsmodels.NormalIndPower uses a \(2K/h^2\) per-arm convention that already accounts for both arms, and llm_power computes the paired-difference variance directly. This is not a calculator-correctness issue: every tool returns a defensible quantity. It is a user-error pattern that the tools make easy, and 1 is its quantitative consequence at the leaderboard adjacency regime.

4.1.0.1 Field practice.

The same pattern surfaces in the published literature. The unpaired Gaussian approximation of [3] is the prevalent closed-form required-\(N\) treatment for LLM accuracy comparisons; paired-variance corrections via McNemar–Connor are rarely surfaced, even on data with shared prompts. The NLP power advocacy of [6] and the test-selection survey of [7] flag McNemar’s availability but do not contrast paired and unpaired required-\(N\) on the same data. 1 and 1 together explain why this shortcut is quantitatively material at leaderboard adjacencies.

5 Finite-sample calibration↩︎

3 treats the rejection threshold as exact at any \(N\), but the underlying normal approximation has a finite-\(N\) remainder. We check empirically how big that remainder is. We calibrate five paired-binary test variants on simulated paired-Bernoulli populations: McNemar \(\chi^2_1\), exact conditional binomial, mid-\(p\), continuity-corrected McNemar, and the paired bootstrap of 2. The grid covers marginal accuracies \(p\in\{0.5,0.7,0.9\}\), latent-Gaussian correlations \(\rho_z\in\{0,0.4,0.8\}\) used to generate the Bernoulli pair, and sample size \(n{=}500\). The per-cell \(\delta\) under \(H_1\) is tuned so the McNemar-Connor asymptotic target is exactly \(0.80\) at each cell (Monte Carlo standard error \({\approx}0.6\) pp on Type-I, \({\approx}1.0\) pp on power). 2 reports cell-level Type-I deviations and empirical power; the bootstrap is the variant we recommend when no closed-form \(\sigma_D\) is available.

5pt

Randomisation on discordant signs coincides with the exact conditional binomial in the binary case. The paired-\(t\) / paired-bootstrap row applies to graded data only [14], [21].

5.0.0.1 Empirical Type-I and power.

All five variants are calibrated within \(1.1\) pp of \(\alpha=0.05\). Under \(H_1\) tuned to a \(0.80\) asymptotic target, the asymptotic trio (McNemar \(\chi^2_1\), mid-\(p\), paired bootstrap) achieves median empirical power \(0.79\), while the exact and continuity-corrected variants are \({\approx}3\) pp conservative; the remaining gap is the \(O(N^{-1/2})\) remainder of the normal approximation in 2 . On synthetic Bernoulli and Beta\((4,2)\) graded marginals, the bootstrap tracks the parametric required-\(N\) to within \(4\)–\(6\%\) (9). Recommendation: use 6 with empirical \(\hat{\rho}\) for binary accuracy; use the paired bootstrap of 2 for graded metrics.

6 Results↩︎

We apply the inversion to Open LLM Leaderboard v1 (§6.1) and the OLL v2 MMLU-Pro top-\(10\) (§6.2). All required-\(N\) figures are at \((\alpha,1{-}\beta)=(0.05,0.8)\).

6.1 OLL v1: 40 unique pairwise comparisons↩︎

We pulled per-question \(0/1\) scores for five \(7\)–\(8\)B-parameter open-weights models (Llama-3-8B\(\pm\)Instruct, Mistral-7B-Instruct-v0.2, Gemma-7B\(\pm\)it) on four tasks (ARC-Challenge, HellaSwag, Winogrande, GSM8K) from the EleutherAI lm-evaluation-harness dumps released via the OLL details_* repos. Each task contributes \(\binom{5}{2}=10\) unique pairwise comparisons (not adjacent-rank only), \(40\) comparisons in total.

2 plots \(N^{\star}\) from 6 at the observed \((\hat{p}_A,\hat{p}_B,\hat{\rho})\) against the actual benchmark size; markers above \(y=x\) are unresolved. The \(|\delta|\)-binned breakdown (3) shows the resolution boundary near \(|\delta|\approx 5\%\): every pair with \(|\delta|\leq 2\%\) is unresolved and every pair with \(|\delta|>5\%\) is resolved; the \(2\)–\(5\%\) band is mixed.

4pt

6.1.0.1 Bootstrap uncertainty on \(N^{\star}\).

Plug-in \(N^{\star}\) inherits the sampling uncertainty of \((\hat{p}_A,\hat{p}_B,\hat{\rho})\). We resample prompts with replacement (\(B=500\)), recompute, and report the \(5\)th–\(95\)th-percentile interval. Among the \(7\) OLL v1 close pairs (\(|\delta|\leq 2\%\)), \(4\) have a \(5\)th-percentile \(N^{\star}\) exceeding \(N\) (robustly unresolved); the remaining \(3\) have intervals that span the diagonal because \(\hat{\delta}\) is poorly estimated near zero.

6.1.0.2 Prospective design validation.

To verify that the framework’s \(N^{\star}\) prescription actually achieves the target power on real data, we picked three OLL v1 pairs spanning \(|\delta| \in [6.3, 10.1]\) pp, computed \(N^{\star}\) via 6 at \((\hat{p}_A,\hat{p}_B,\hat{\rho})\), bootstrap-subsampled to \(N^{\star}\) prompts, and ran McNemar at \(\alpha{=}0.05\) over \(M{=}1000\) trials each. Empirical power at \(N^{\star}\) lands at \(0.796\)–\(0.827\), within \(\pm 2.7\)pp of the \(0.80\) target; at \(0.8\,N^{\star}\) and \(1.2\,N^{\star}\) it correctly under- and over-shoots. The framework’s prescription is calibrated on the data, not only in asymptotic theory. Per-pair table in 15.

6.1.0.3 Multi-arbiter agreement on close pairs.

On every \(|\delta|\leq 2\%\) pair, four arbiters (asymptotic McNemar \(\chi^2_1\), exact two-sided conditional binomial, mid-\(p\), and the paired bootstrap of 2) agree on six of seven close pairs (all unanimous fail-to-reject); only the leaderboard-displayed-significant HellaSwag pair flagged in §1 produces a split verdict (\(\chi^2_1\!:\!p{=}.049\) rejects; exact: \(p{=}.054\) does not). Per-pair \(p\)-values in 15.

6.2 MMLU-Pro paired item-level tightening (OLL v2)↩︎

For the largest OLL v2 task with accessible per-item correctness, we pull the gated open-llm-leaderboard/*_details parquets for the top-\(10\) MMLU-Pro models (\(N=12{,}032\) items) and compute \(N^{\star}\) for the \(9\) adjacent-rank pairs. The paired-Bernoulli formula (6 ) and the discordance-form McNemar required-\(N\) agree to within \(1\%\) on every pair (3). Four of nine adjacent-rank pairs are unresolved at \(N=12{,}032\) under the adjacent-rank multiplicity convention \(m=9\). The all-pairs convention \(m=\binom{10}{2}=45\) inflates \(N^{\star}\) by \({\approx}2.14\), but every unresolved pair already has \(q < 1\), so all four verdicts stand.

1 holds tightly on this data: the shortcut \(n_h\) is below \(N^{\star}\) by the predicted factor of two on every pair (median ratio \(0.500\), range \([0.496, 0.500]\), despite \(\rho\) ranging over \([0.45, 0.99]\)).

6.2.0.1 External replication.

11 reports a closed-source frontier panel (4 models, MMLU-Pro \(N{=}1{,}350\) all-four-scored): one adjacent pair (Llama-4-Maverick vs.DeepSeek-V3.2, gap \(2.8\) pp) below resolution, shortcut/\(N^{\star}\) in \([0.496, 0.500]\) replicating 1 on a different model class (illustrative; deployments rotate).

6.2.0.2 Sensitivity to \(\hat{\rho}\) misspecification.

The diagnostic uses the empirical paired-correlation \(\hat{\rho}\), which is itself estimated. Perturbing \(\hat{\rho}\) by \(\pm 0.10\) on every pair (clamped to the admissible interval) moves the OLL v1 unresolved count to \([9, 12]/40\) and the MMLU-Pro count to \([2, 4]/9\); the qualitative leaderboard message survives the perturbation. Three of \(40\) OLL v1 pairs and \(2\) of \(9\) MMLU-Pro pairs flip verdict; in every case a boundary pair with \(|\hat{\delta}|\le 4\) pp, including the HellaSwag pair from §1 (\(\hat{\delta{=}}0.46\) pp, \(\hat{\rho{=}}0.81\)) which becomes resolved at \(\hat{\rho{+}}0.10{=}0.91\). Boundary-case sensitivity is expected.

6.2.0.3 Stress-test sequence.

The next three subsections stress the verdicts of §6 against family-level multiplicity (§6.3), real subject-level clustering (§6.4), and anytime-valid sequential testing (§6.5). Each adjustment strictly tightens the verdict (5); the qualitative leaderboard message, that displayed adjacent gaps in this regime are often under-supported, survives all three.

6.3 Leaderboard-scale multiplicity↩︎

We report multiplicity under two pre-declared families: adjacent-rank claims (family size \(K{-}1\)) and all-pairs leaderboard claims (family size \(\binom{K}{2}\)). Families are declared before observing \(p\)-values.

6.3.0.1 Bonferroni / Holm.

Running the \(40\) OLL v1 pairs at nominal \(\alpha=0.05\) inflates the family-wise error rate. Bonferroni control replaces \(z_{1-\alpha/2}=1.960\) with \(z_{1-\alpha/(2\cdot 40)}\approx 3.227\), multiplying \(N^{\star}\) by \({\approx}2.11\). Šidák yields a near-identical \(2.10\times\) inflation; Holm [16] is bounded above by Bonferroni at every position, so \(2.11\times\) is an upper bound on the Holm-adjusted required-\(N\). Applying \(2.11\times\) to 3, every \(|\delta|\leq 2\%\) pair remains unresolved, the \(2\)–\(5\%\) unresolved fraction rises from \(4/10\) to \(6/10\), and \(16/17\) of the \(5\)–\(15\%\) pairs stay resolved (the worst-case pair at \(r{=}0.65\) flips since \(0.65{\times}2.11{>}1\)); the \(>\!15\%\) bucket stays fully resolved.

6.3.0.2 Benjamini–Hochberg.

Controlling the false-discovery rate at \(0.05\) via Benjamini–Hochberg [17] is less conservative. On these data it effectively coincides with the \(|\delta|\approx 5\%\) resolution boundary (close pairs produce \(p\)-values far from significance, large-gap pairs essentially at zero), so no \(|\delta|>5\%\) rejection is retracted under FDR control.

6.4 Cluster-aware sensitivity↩︎

Real benchmarks contain topic clusters, near-duplicates, and templated subdomains [8], [9] that induce intra-cluster correlation in the per-item paired difference \(D_i\). The cluster-aware required-\(N\) scales the IID estimate by the design effect \(\mathrm{DE} = 1 + (\bar m - 1)\,\mathrm{ICC}(D)\), where \(\bar m\) is mean cluster size and \(\mathrm{ICC}(D)\) is the intra-cluster correlation of \(D\). We compute \(\mathrm{ICC}(D)\) empirically on MMLU-Pro using the dataset’s \(14\) subject categories [8] as natural clusters (\(\bar m \approx 859\) items per category); 4 reports per-pair ICC, DE, and cluster-adjusted \(N^{\star}\) for the \(9\) adjacent-rank top-\(10\) pairs.

6.4.0.1 Real-data verdict.

Median empirical \(\mathrm{ICC}(D) = 0.0010\) across the \(9\) pairs; the distribution is heavy-tailed (range \([-3\!\times\!10^{-4},\, 3.6\!\times\!10^{-2}]\)). Median DE \(=1.88\) but two pairs (rank \(4\) vs \(5\), rank \(5\) vs \(6\)) show large clustering (\(\mathrm{DE} = 31.5\) and \(6.7\)): paired-difference homogeneity is high within categories for these pairs because the model gap is dominated by specific subject domains. Under real cluster correction, the unresolved count rises from \(4/9\) at IID to \(6/9\): rank \(4\) vs \(5\) flips from \(N^{\star}_{\mathrm{IID}}=432\) (well-resolved) to \(N^{\star}_{\mathrm{cluster}}=13{,}621\) (\(>N\)); rank \(5\) vs \(6\) flips from \(5786\) to \(39{,}009\).

6.4.0.2 Bootstrap stability of the verdict.

With only \(K{=}14\) clusters, individual ICC point estimates carry nontrivial uncertainty. We address this by cluster-bootstrapping the verdict: resample the \(14\) categories with replacement (\(B{=}1000\), seed \(42\)) and recompute the unresolved-pair count. We hold \((\hat{p}_A,\hat{p}_B,\hat{\rho})\) at the full-data estimates so the resulting CI isolates cluster-structure uncertainty. The three pairs driving the IID-to-cluster flip have \(5\)th-percentile ICC bounds that are all strictly positive (rank \(4\) vs \(5\) at \([0.021, 0.044]\), rank \(5\) vs \(6\) at \([0.003, 0.010]\), rank \(6\) vs \(7\) at \([0.004, 0.020]\)), so the cluster signal that flips them is unlikely to be a \(K{=}14\) artefact. Across the full bootstrap, the unresolved-pair count is \(5/9\) in \(45\%\) of resamples and \(6/9\) in \(55\%\); \(\Pr(\text{unresolved}\!\geq\!5)=99.9\%\), and only \(1\) of \(1000\) resamples returns to the IID \(4/9\) count. Full per-pair CIs are in 14.

6.4.0.3 Sensitivity, not certification.

§6.4 should be read as a cluster-sensitivity analysis. Three robustness checks support the conclusion despite \(K{=}14\): (i) the cluster-bootstrap above (verdict is at \(5\)–\(6\) unresolved out of \(9\) in \(99.9\%\) of resamples), (ii) a leave-one-subject-out (LOSO) recomputation that drops each MMLU-Pro category in turn (13), which holds the unresolved count at \(6/9\) on \(11\) of \(14\) drops and \(5/9\) on the remaining \(3\), and (iii) the cluster-definition sensitivity below. None of these collapses the result below \(5/9\) except under random clusters (the null check), so the headline \(4/9\to 6/9\) flip is not an artifact of either category sampling or any single high-ICC subject.

6.4.0.4 Cluster-definition sensitivity.

Subject categories are one of many possible clusterings. We rerun the §6.4 pipeline under three alternatives: random clusters of \(K{=}14\) (null check); difficulty quartiles (\(K{=}4\), binned by per-item mean accuracy across the top-\(10\)); and subject sub-clusters (\(K{=}28\), each subject split in half by item parity). Random clusters give \(\mathrm{ICC}{\approx}0\) and revert to the IID \(4/9\) verdict (the null check). Difficulty quartiles produce a stronger cluster signal (median \(\mathrm{ICC}{=}0.019\), max \(0.19\)) and \(9/9\) unresolved; subject sub-clusters give \(5/9\). The unresolved count thus spans \([5, 9]/9\) across non-null definitions, with the headline \(6/9\) at the midpoint.

4pt

6.5 Anytime-valid leaderboard testing↩︎

Public leaderboards update continuously: each new model triggers up to \(K{-}1\) new pairwise tests against the existing entries. A fixed-\(n\) test loses Type-I control when stopping can be decided after seeing the data. We replace the fixed McNemar-Connor threshold \(z_{1-\alpha/2}\) by an anytime-valid threshold (valid simultaneously at every \(n\)) derived from a paired-Bernoulli mixture e-process [18], [19], [22]. Construction, mixture choice, and Type-I/stopping calibration are in 10.

6.5.0.1 Applied to MMLU-Pro adjacencies.

At \(N{=}12{,}032\) the time-uniform threshold inflates \(N^{\star}\) by \(2.15\times\) vs.fixed-\(n\). The numerical proximity to the \(m{=}45\) Bonferroni inflation (\(2.14\times\), §6.2) and the \(m{=}40\) figure (\(2.11\times\), §6.3) is coincidence: anytime-validity gives a time-uniform boundary; Bonferroni is a family-size correction.

Anytime-valid testing flips one extra verdict relative to fixed-\(n\). Five of nine adjacent-rank pairs are unresolved under anytime-validity, vs.\(4/9\) under fixed-\(n\) and Bonferroni-\(9\). The extra pair is rank \(5\) vs.\(6\) (\(\hat{\delta{=}}1.88\)pp, \(\hat{\rho{=}}0.45\)): the fixed-\(n\) exact McNemar \(p\)-value is \(5.8\!\times\!10^{-5}\), comfortably rejecting. The anytime-valid threshold, however, is stricter than a one-shot test at the realised discordance, so the same data fails to cross it. 4 traces sample \(\log e_n\) trajectories under \(H_0\) and \(H_1\).

6.5.0.2 Verdict across procedures.

5 pulls the unresolved counts together. On the larger OLL v1 family (\(m{=}40\)) the anytime-valid threshold-inflation at typical \(N\) coincides with Bonferroni’s; on the smaller MMLU-Pro family (\(m{=}9\)) anytime-validity dominates, flagging one additional pair as sequentially-not-resolvable. The columns of 5 answer subtly different questions and are not directly comparable as more-vs-less-correct: the anytime-valid column scores whether a verdict is resolvable under continuous monitoring of an infinitely-extending stream, which is a stricter criterion than fixed-\(n\) McNemar applied once.

4pt

7 Conclusion↩︎

LLM leaderboard claims should report not only a gap and a \(p\)-value, but the benchmark resolution needed to make that gap detectable under the actual paired design.³ Across OLL v1 and MMLU-Pro, many displayed adjacent gaps fall below this resolution once pairing, multiplicity, clustering, or anytime-validity are made explicit. Clustering is verified by a category-bootstrap and LOSO ([app:icc_bootstrap] [app:loso]).

7.0.0.1 Scope.

The load-bearing claims (the shortcut lemma and the unresolved-pair verdicts) apply to the close-comparison regime where leaderboard adjacencies live. For cross-tier comparisons with large gaps, both paired and unpaired calculators return “resolved” at typical benchmark sizes, and the shortcut’s factor-of-two underestimate is benign. The lemma is sharpest near \(p{=}\tfrac12\), which is where many benchmark accuracies cluster.

7.0.0.2 Limitations.

Our empirics are restricted to binary accuracy, the displayed metric on the headline benchmarks; graded metrics and pairwise-preference leaderboards are handled only methodologically via 2 (validated to \(4\)–\(6\%\) on Beta\((4,2)\) marginals, 9). The MMLU-Pro cluster analysis rests on \(K{=}14\) subject categories; LOSO and a category bootstrap ([app:loso] [app:icc_bootstrap]) support the headline \(4/9 \to 6/9\) flip, but a benchmark with finer-grained natural clusters would tighten the design-effect estimate. The closed-source frontier panel (11) is a snapshot under rotating deployments and replicates 1 only as illustration, not as a durable head-to-head comparison.

7.0.0.3 Future work.

Four directions extend the framework. (i) Generalise 1 away from the equal-marginal midpoint and to multi-arm power. (ii) Derive a PRDS-aware required-\(N\) for leaderboard families with overlapping models, building on [23]. (iii) Plug in a fully clustered-prompt bootstrap with empirically-estimated ICC [9] as the principled refinement of the design-effect estimate. (iv) Instantiate the resolution diagnostic on judge-scored and pairwise-preference (Chatbot-Arena-style) leaderboards via Bradley–Terry e-processes. Construct validity [11]–[13] composes with resolution: a benchmark can be statistically resolvable and still fail to measure what it claims to.

8 Proof of 1↩︎

Let \(\phi(t)=\arcsin\sqrt{t}\), so \(h(p_A,p_B)=2[\phi(p_A)-\phi(p_B)]\) with \(\phi'(t)=1/[2\sqrt{t(1{-}t)}]\) and \(u(t):=t(1{-}t)\). Under the midpoint parameterization \(p_A = p + \delta/2\), \(p_B = p - \delta/2\), expand symmetrically around \(p\): \[\begin{align} \phi(p_A) - \phi(p_B) &= \delta\,\phi'(p) + \frac{\delta^3}{24}\phi'''(p) + O(\delta^5), \end{align}\] the even-order (\(\delta^2\), \(\delta^4\), \(\ldots\)) Taylor terms cancelling by symmetry of \(\phi\) around \(p\). Squaring, \[h^2 = 4\delta^2 \phi'(p)^2\,\bigl[1 + \tfrac{\delta^2}{12}\,\phi'''(p)/\phi'(p) + O(\delta^4)\bigr].\] Direct computation gives \(\phi'(p)^2 = 1/(4u)\) and \(\phi'''(p)/\phi'(p) = 1/u + 3(1{-}2p)^2/(4u^2)\), so \[\label{eq:h2-expansion} h^2 = \frac{\delta^2}{u}\,\Bigl[1 + \delta^2\!\left(\tfrac{1}{12u} + \tfrac{(1{-}2p)^2}{16u^2}\right) + O(\delta^4)\Bigr].\tag{10}\] The \(O(\delta^2)\) correction inside the bracket contributes at the same order as the \(\sigma_D^2\) correction below, so it cannot be dropped.

For the paired-difference variance 5 , expand symmetrically. The sum \(u(p_A)+u(p_B) = 2u(p) + (\delta^2/4)u''(p) + O(\delta^4)\), and since \(u''(t)=-2\), \[u(p_A)+u(p_B) = 2u(p) - \tfrac{\delta^2}{2} + O(\delta^4).\] For the product, write \(u(p_A) \approx u(p) + (\delta/2)u'(p) + (\delta^2/8)u''(p)\) and similarly for \(u(p_B)\), then \[\begin{align} u(p_A)u(p_B) &= \!\bigl[u(p) - \tfrac{\delta^2}{4}\bigr]^2 - \!\bigl[\tfrac{\delta}{2}u'(p)\bigr]^2 + O(\delta^4) \nonumber\\ &= u(p)^2 - \tfrac{\delta^2}{2}u(p) - \tfrac{\delta^2}{4}u'(p)^2 + O(\delta^4), \end{align}\] so \[\sqrt{u(p_A)u(p_B)} = u(p) - \delta^2\!\left(\tfrac{1}{4} + \tfrac{u'(p)^2}{8u(p)}\right) + O(\delta^4).\] With \(u'(p)=1{-}2p\), substitution into 5 gives \[\begin{align} \sigma_D^2 &= 2u(p)(1{-}\rho) \nonumber\\ &\quad + \tfrac{\delta^2}{4}\!\left[\rho\,\tfrac{(1{-}2p)^2}{u(p)} - 2(1{-}\rho)\right] + O(\delta^4). \end{align}\]

The shortcut-to-paired ratio is then \[\begin{align} \frac{n_h}{N^{\star}} &= \frac{(1{-}\rho)\,\delta^2}{h^2\,\sigma_D^2} = \frac{1}{2\,(1 + \delta^2 H_2)(1 + \delta^2 V_2)} \nonumber\\ &= \tfrac{1}{2}\bigl[1 - \delta^2(H_2 + V_2) + O(\delta^4)\bigr], \end{align}\] where the corrections are \[\begin{align} H_2 &= \tfrac{1}{12u} + \tfrac{(1{-}2p)^2}{16u^2},\\ V_2 &= \tfrac{\rho(1{-}2p)^2}{8u^2(1{-}\rho)} - \tfrac{1}{4u}. \end{align}\] Combining, \[H_2 + V_2 = \tfrac{(1{+}\rho)(1{-}2p)^2}{16(1{-}\rho)u^2} - \tfrac{1}{6u}.\]

Taking absolute values, \[\left|\frac{n_h}{N^{\star}} - \frac{1}{2}\right| \leq \frac{1}{2}\!\left|\frac{(1{+}\rho)(1{-}2p)^2}{16(1{-}\rho)u^2} - \frac{1}{6u}\right|\,\delta^2 + O(\delta^4),\] giving \(C(p,\rho)\) as in 9 . Convergence is uniform on any compact \((p,\rho)\) set in which \(u(p)\ge u_0>0\) and \(1-\rho \ge \eta_0 > 0\), because \(C(p,\rho)\) is then bounded above and the \(O(\delta^4)\) remainder uniform. 1 follows by inverting \(C(p,\rho)\delta^2 \le \epsilon\). 0◻

8.0.0.1 Numerical cross-check.

Direct per-cell evaluation on the heatmap grid of e6_ratio_heatmap.csv confirms the leading-order coefficient: the empirical \(|n_h/N^{\star} - \tfrac12|/\delta^2\) matches the closed-form \(C(p,\rho)\) to four significant figures at small \(\delta\) on every cell (\(p \in \{0.5, 0.65, 0.8\}\), \(\rho \in [0, 0.7]\), \(\delta \in [0.005, 0.20]\); median relative error \(0.08\%\) at \(\delta \le 0.05\)). The full inequality \(|n_h/N^{\star} - \tfrac12| \le C(p,\rho)\delta^2 + O(\delta^4)\) from 1 holds with the explicit \(O(\delta^4)\) remainder: at the upper end of the grid (e.g.\((p,\rho,\delta) = (0.8, 0.5, 0.20)\)), the \(O(\delta^4)\) correction adds up to \({\approx}17\%\) on top of \(C\delta^2\), which is the expected behaviour of a small-\(\delta\) Taylor bound. At \(p=0.5\) the \((1{-}2p)^2\) factor vanishes a fortiori, giving the clean sanity check \(C(0.5,\rho) = 1/(12u) = 1/3\) for every admissible \(\rho\) (independent of \(\rho\)). Off-midpoint the constant grows with both \(|1{-}2p|\) and \(\rho\): empirically \(C(0.65, 0.0) \approx 0.31\), \(C(0.8, 0.5) \approx 0.80\), \(C(0.65, 0.9) \approx 0.67\).

9 Synthetic Bernoulli and graded validation↩︎

5 reports bootstrap power against \(n\) on synthetic paired Bernoulli data with \(p_A=0.65\), \(\delta\in\{0.02,0.04,0.08\}\), \(\rho\approx 0.3\) via a latent Gaussian copula, \(N=30{,}000\). The bootstrap crosses the \(0.8\) target within \(5\%\) of \(N^{\star}\) from 6 for all three \(\delta\). 6 repeats the exercise on Beta\((4,2)\) marginals (graded stress test); the bootstrap tracks the paired-\(t\) required-\(N\) to within \(4\)–\(6\%\).

10 Mixture e-process: construction and calibration↩︎

10.0.0.1 Construction.

Conditioned on \(b{+}c\) discordant pairs, the sign sequence is i.i.d.Bernoulli\((\tfrac12)\) under \(H_0\). With a discrete mixture prior \(\nu\) over the alternative discordance probability \(\theta \in (0, \tfrac12) \cup (\tfrac12, 1)\), the e-process is \[\label{eq:eprocess} e_n \;=\; \int \frac{\theta^{b_n}(1-\theta)^{c_n}}{(1/2)^{b_n + c_n}}\,\mathrm{d}\nu(\theta),\tag{11}\] By Ville’s inequality (a martingale maximal inequality), \(e_n\) has \(\Pr(\sup_n e_n \geq 1/\alpha) \leq \alpha\) at any stopping time. The rejection rule “reject the first \(n\) at which \(e_n \geq 1/\alpha\)” is therefore anytime-valid.

10.0.0.2 Mixture choice.

We use a discrete uniform mixture over \(\theta \in \{0.01,\ldots,0.49,\,0.51,\ldots,0.99\}\) with equal weights, for two reasons. First, it is conjugate to the discordant-binomial likelihood, so the integral in 11 is a closed-form sum. Second, it is the natural default in the betting-based formulation of safe testing [24], where the prior plays the role of a uniform prior over wagers. Alternatives that grow faster against a specific point alternative (e.g.GROW, [25]) require knowing the alternative density and are sensitive to misspecification. Empirically, a Beta\((2,2)\) mixture and a discrete two-point mixture at \(\theta \in \{0.4, 0.6\}\) both produce stopping times within \({\approx}8\%\) of the uniform on our calibration pairs.

10.0.0.3 Calibration.

We calibrate the mixture e-process on simulated paired Bernoulli calibrated to two ARC pairs (\(\hat{\delta}=2.4\%, 7.8\%\); \(\hat{\rho}=0.64, 0.54\); \(M{=}600\) trials each). The empirical Type-I rates are \(0.035\) and \(0.043\) (Monte Carlo SE \({\approx}0.9\) pp, both within nominal \(\alpha{=}0.05\)); the \(H_1\) rejection rate is \(97\)–\(100\%\). The expected stopping time is \(1.84\)–\(2.32{\times}\) the fixed-\(n\) McNemar-Connor \(N^{\star}\) on resolved pairs: the time-uniform cost of dropping the pre-specified-\(n\) assumption.

11 Closed-source frontier panel (illustrative)↩︎

We replicate 1 on a closed-source frontier panel as a robustness check on a different model class and evaluation route; we include it as illustrative rather than load-bearing because deployments rotate. Four pay-walled API models (GPT-5.5 and GPT-5.4 on Azure OpenAI; DeepSeek-V3.2 and Llama-4-Maverick-17B-128E-Instruct-FP8 on Azure AI Foundry) were evaluated on a \(3{,}000\)-item random subsample of MMLU-Pro on 2026-05-08 under a \(5\)-shot CoT prompt; dropping items where any model errored leaves \(N{=}1{,}350\) all-four-scored items. Across the \(3\) adjacent-rank pairs the shortcut/\(N^{\star}\) ratio sits in \([0.496, 0.500]\) (median \(0.497\)), replicating Lemma 1’s factor-of-two. The Llama-4-Maverick vs.DeepSeek-V3.2 pair (gap \(2.8\) pp, \(\rho=0.48\)) is unresolved at \(N{=}1{,}350\); the other two are resolved (7). Exact API identifiers, call timestamps, the verbatim \(5\)-shot CoT prompt, and the \(3{,}000\)-item index list (seed \(42\) over the MMLU-Pro test split) are in the artifact under appendix_d/.

12 Raw discordance and pair details↩︎

[tab:discord_oll,tab:discord_mmlupro] list the per-pair raw discordance counts \((b,c)\), marginals, correlation, and both \(p\)-value variants for the seven OLL v1 close pairs (§6.1) and the nine MMLU-Pro adjacent-rank pairs (§6.2). These let a reader verify the headline HellaSwag \(p_{\chi^2}{=}0.049\) vs.\(p_{\mathrm{exact}}{=}0.054\) split (§1, row 4 of [tab:discord95oll]) without re-running the code, and trace each rank pair to its concrete model identifiers.

13 Leave-one-subject-out cluster robustness↩︎

6.4 reports a \(4/9\to 6/9\) unresolved-count flip on MMLU-Pro top-\(10\) adjacent pairs under real subject-level clustering (\(K{=}14\) categories). To confirm the flip is not driven by any single category, we recompute the cluster-corrected unresolved count after dropping each MMLU-Pro category in turn and recomputing the per-pair ICC, design effect, and \(N^{\star}_{\mathrm{cluster}}\) on the remaining items.

[tab:loso] lists the LOSO results: across the \(14\) drops, the unresolved count stays at \(6/9\) for \(11\) drops and at \(5/9\) for three drops (health, law, psychology). No drop collapses the result below \(5/9\), so the cluster-induced flip is not an artifact of a single high-ICC category.

14 Cluster-bootstrap CIs for MMLU-Pro ICC↩︎

6.4 reports a category-bootstrap stability check on the MMLU-Pro cluster-induced verdict flip. [tab:icc95bootstrap] lists the per-pair \(5\)–\(95\%\) bootstrap intervals on \(\mathrm{ICC}(D)\), design effect, and cluster-corrected \(N^{\star}\). Resampling protocol: at each of \(B{=}1000\) iterations (seed \(42\)), draw \(K{=}14\) subject categories with replacement, recompute the one-way ANOVA \(\mathrm{ICC}\) on the paired-difference series \(D_i\) using the resampled cluster structure, derive \(\mathrm{DE} = 1 + (\bar m - 1)\,\mathrm{ICC}^+\), and report \(N^{\star}_{\mathrm{cluster}} = N^{\star}_{\mathrm{IID}}\cdot\mathrm{DE}\). The IID inputs \((\hat{p}_A,\hat{p}_B,\hat{\rho},N^{\star}_{\mathrm{IID}})\) are held fixed at their full-data estimates so the CIs isolate cluster-structure uncertainty (full-bootstrap CIs would only be wider). The “\(\Pr(\text{unres.})\)” column reports the fraction of \(B\) bootstraps in which \(N^{\star}_{\mathrm{cluster}} > N{=}12{,}032\) for that pair.

15 Prospective design validation and multi-arbiter agreement↩︎

[tab:prospective] reports the prospective validation referenced in §6.1; [tab:arbiters] reports the multi-arbiter agreement check on every \(|\hat{\delta}| \leq 2\)pp pair.

16 Reporting checklist↩︎

17 llm-power API↩︎

cohens_h(p1, p2) – Cohen’s \(h\) for two proportions.
paired_bootstrap_delta(scores_a, scores_b, ...) – prompt-bootstrap CI on \(\hat{\delta}\).
bootstrap_power(...) – empirical power against a reference score-matrix.
parametric_required_n_proportions(p1, p2, paired, rho) – shortcut \(n_h\) (Cohen convention).
parametric_required_n_paired_binary(p1, p2, rho) – \(N^{\star}\) from 6 .
required_n_mcnemar(n_ab, n_ba, n_observed) – discordance-form McNemar required-\(N\).
parametric_required_n_paired(mean_diff, sd_diff) – paired-\(t\) required-\(N\) for graded data.

References↩︎

---

1. The exact two-sided power is \(\pi_N(\delta) = \Phi(-z_{1-\alpha/2} - \mu) + 1 - \Phi(z_{1-\alpha/2} - \mu)\) with \(\mu = |\delta|\sqrt{N}/\sigma_D\). The second term dominates at conventional target powers (the first contributes \(<10^{-3}\) at \(1{-}\beta = 0.8\)); 3 inverts the dominant term.↩︎

2. The percentile variant is chosen for simplicity; the studentized bootstrap [14] is asymptotically more accurate. Empirically (2), the percentile bootstrap’s Type-I deviation on our \(5\)-cell grid is within \(0.9\) pp of nominal, so the simpler variant suffices at the \(N\) regime of interest.↩︎

3. Code and raw responses: https://github.com/akotawala10/llm-power.↩︎