¹

1 Introduction↩︎

LLM evaluation increasingly draws on two complementary signals: static benchmark correctness and human preferences in open-ended arenas. Static benchmarks provide ground-truth labels and standardized grading protocols [1]–[3], but can become saturated or contaminated as models improve [4], [5]. Arena-style evaluations, such as Chatbot Arena [6], better reflect open-ended user interactions through Bradley–Terry-style rankings [7], but open-ended preference judgments can involve annotator disagreement, subjectivity, and systematic evaluator biases, making them difficult to interpret without additional structure [8]–[10]. Recent work has begun to connect benchmark and preference-based evaluation [11], [12], but the two settings still provide weakly integrated views of model capability.

Furthermore, both settings are largely model-centric: they aggregate item-level outcomes into overall model scores, treating all items as equally informative. This misses a key insight: ranking signal depends on model-item interaction. Items that are nearly always solved or nearly always failed provide limited separation among the current model pool; the most informative items are those for which small ability differences among evaluated models translate into detectable differences in correctness or preference outcomes. Modeling this item structure can make evaluation more stable, interpretable, and cost-efficient, while also identifying which items are saturated, discriminative, or future-facing.

Toward this end, we propose DualEval, a unified evaluation framework that jointly estimates model abilities and item properties from both static benchmark correctness and arena-style preference signals. Inspired by Item Response Theory (IRT) [13], [14], DualEval places model abilities and item difficulties on a shared latent scale. For static benchmarks, we use a two-parameter logistic IRT model over binary correctness labels. For arena-style data, we use reward-model scores to construct soft pairwise preference targets and fit them through the same latent ability–difficulty structure. This novel joint formulation allows correctness and preference supervision to update shared model and item parameters, yielding not only model rankings but also item-level diagnostic information.

This joint formulation has two key benefits. First, it bridges static and arena evaluation: correctness labels and preference signals can update the same latent model parameters, allowing each source to compensate for the other’s weaknesses. Static benchmarks provide clean, controlled supervision, while arena prompts provide open-ended signals closer to realistic user interactions. Second, DualEval turns evaluation data itself into an object of analysis. The learned item parameters characterize each question’s difficulty and sharpness, identify saturated or low-signal items, and support low-cost subsets that preserve the full-data ranking.

We apply DualEval to four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. The first three domains combine static benchmark labels with reward-model scores for open-ended arena responses, while the generic domain uses arena data only. Our experiments cover 18 frontier LLMs from multiple providers. Arena supervision uses our proprietary scalar reward model, developed internally at LM Arena and validated against held-out human preferences; we also verify robustness using the public Skywork-Reward-V2-Qwen3-8B model. Across domains, DualEval reconstructs both evaluation signals, achieving 88–92% static-label accuracy in the three static-anchored domains and 68–81% decisive-pair agreement on arena comparisons. Compared with static-only and arena-only baselines, the joint model yields more balanced rankings across evaluation sources and remains robust under reward-model substitution. Its learned item profiles further support diagnostic applications: selecting the top 10% of items by Fisher information recovers the full-data ranking with Spearman \(\rho \geq 0.95\) across all static-anchored domains, while residual-based anomaly detection recovers injected contamination with AUROC \(\geq 0.995\) for static-label targets and AUROC \(0.93\)–\(0.98\) for arena-reward targets under a clean-fit reference.

Overall, we reframe LLM evaluation as joint model-item calibration, and our contributions can be summarized as follows:

• We propose DualEval, a novel evaluation paradigm that unifies static benchmark correctness and open-ended preference signals on a shared latent model-item scale, enabling mutually informed frontier model evaluation and item diagnostics.

• We evaluate DualEval on 18 frontier LLMs across coding, math, miscellaneous domain knowledge, and generic everyday user queries, showing that it produces balanced rankings across signal sources, stable bootstrap estimates, and robustness to reward-model choice in static-anchored domains.

• We demonstrate two diagnostic applications: Fisher-information-based benchmark compression, which recovers full-data rankings from small high-signal subsets, and residual-based anomaly detection, which flags (model, item) pairs whose observed outcomes deviate from the fitted model-item expectations.

2 Related Work↩︎

2.0.0.1 Static and open-ended LLM evaluation.

Static benchmarks cover NLI and reasoning [15]–[17], QA [18]–[21], and math [1], [22], [23], but vary substantially in difficulty and discriminability [24]–[26]. Open-ended preference evaluations such as Chatbot Arena [6] and MT-Bench [10] capture user-facing quality through aggregate Bradley-Terry rankings without explicit item modeling. Several efforts integrate the two settings [11], [12].

2.0.0.2 IRT and efficient evaluation.

Item Response Theory [27] has long been used in educational testing and was applied early to NLP for instance-level analysis [28], [29]. Recent work uses IRT-style modeling for benchmark diagnostics [24], [25], label-free evaluation [30], data-efficient evaluation [31], [32], and adaptive testing [33]. These methods generally operate within a single evaluation source, most often objective correctness labels. DualEval instead couples static correctness with reward-distilled open-ended preferences in a shared latent model-item space.

3 Method↩︎

In our DualEval framework, each model \(i\) is assigned an ability parameter \(\theta_i \in \mathbb{R}\), and each task \(q\) is assigned a difficulty parameter \(b_q \in \mathbb{R}\) and a sharpness parameter \(a_q > 0\), parameterized as \(a_q=\exp(k_q)\) with \(k_q \in \mathbb{R}\). Intuitively, \(\theta_i-b_q\) measures how capable model \(i\) is relative to task \(q\), while \(a_q\) is a sharpness parameter that controls how rapidly the predicted success probability changes as model ability moves relative to item difficulty.

3.0.0.1 Static IRT.

For static benchmarks with binary correctness labels, we use a standard two-parameter logistic IRT model: \[p_{i,q}=P(y_{i,q}=1)=\sigma(a_q(\theta_i-b_q)),\] where \(\sigma(\cdot)\) is the sigmoid function. Let \(\mathcal{S}\) denote the set of observed static labels. We define the static loss as: \[\mathcal{L}_{\mathrm{static}} = \mathbb{E}_{(i,q)\in \mathcal{S}} \mathrm{BCE}\!\left(p_{i,q}, y_{i,q}\right).\]

3.0.0.2 Reward-Distilled Pairwise IRT.

For open-ended evaluations, a reward model assigns each model response a scalar score \(r_{i,q}\). Our framework accepts any scalar response-quality scorer; the joint formulation provides robustness to RM noise when static correctness labels are available (Appendix §12). We first standardize rewards globally by \(z_{i,q} = \frac{r_{i,q}-\bar{r}}{\sigma_r}\), where \(\bar{r}\) and \(\sigma_r\) are the mean and standard deviation over all reward observations. For each question \(q\) and model pair \((i,j)\), we define a soft pairwise preference target: \[p^{*}_{ijq}=\sigma(z_{i,q}-z_{j,q}),\] which converts reward gaps into calibrated preference strengths: small gaps yield targets near \(0.5\), while large gaps yield more confident preferences.

Given the shared IRT parameters, we define the latent success probability of model \(i\) on question \(q\) as: \[p_{i,q}=\sigma\!\left(a_q(\theta_i-b_q)\right).\] The predicted preference probability is then: \[\hat{P}(i \succ j \mid q) = \sigma\!\left(\gamma (p_{i,q}-p_{j,q})\right),\] where \(\gamma>0\) is a learned arena-temperature parameter.

3.0.0.3 Tie and Both-Bad Filtering.

Not all reward-derived pairs provide useful relative preference signal. We identify ties using the empirical reward-gap distribution: a pair is marked as a tie if \(|z_{i,q}-z_{j,q}| < \delta_{\mathrm{tie}}\), where \(\delta_{\mathrm{tie}}\) is chosen as the configured percentile of pairwise absolute reward gaps. Ties are excluded from the arena loss.

We also identify “both-bad” pairs, where both models receive low standardized rewards with \(\max(z_{i,q}, z_{j,q}) < \tau_{\mathrm{bb}}\), where \(\tau_{\mathrm{bb}}\) is chosen as the configured percentile of pairwise maximum rewards. Let \(\mathcal{A}_{\mathrm{dec}}\) denote the decisive arena pairs, defined as non-tie and non-both-bad pairs, and let \(\mathcal{A}_{\mathrm{bb}}\) denote the both-bad, non-tie pairs. We define the full arena loss as: \[\begin{align} &\mathcal{L}_{\mathrm{arena}} = \lambda_{\mathrm{pref}} \mathbb{E}_{(i,j,q)\in \mathcal{A}_{\mathrm{dec}}} \mathrm{BCE}\!\left( \hat{P}(i \succ j \mid q), p^*_{ijq} \right) \\ &-\lambda_{\mathrm{bb}} \mathbb{E}_{(i,j,q)\in \mathcal{A}_{\mathrm{bb}}} \left[ \log(1-p_{i,q}) + \log(1-p_{j,q}) \right]. \end{align}\] The first term fits relative preference evidence from decisive pairs. The second term treats both-bad pairs as absolute failure evidence, pushing both models toward low success probability on that question and avoiding conflicting gradients between “both models failed” and “one model was slightly better”.

3.0.0.4 Joint Objective.

The static and arena components share the same latent parameters \((\theta,b,k)\). The full optimization objective is: \[\begin{align} &\mathcal{L} = \lambda_{\mathrm{static}}\mathcal{L}_{\mathrm{static}} + \mathcal{L}_{\mathrm{arena}} \\ &+ \lambda_{\mathrm{reg}} \left( \|\theta\|_2^2 +\|b\|_2^2 +\|k\|_2^2 +(\log \tilde{\gamma})^2 \right). \end{align}\]

Let \(\gamma_0=4\) denote the default arena-temperature scale. We regularize the relative scale \(\tilde{\gamma}=\gamma/\gamma_0\) toward one.

For identifiability, after each optimization step we center model abilities to have zero mean and absorb the same shift into item difficulties. This fixes the translation invariance of the latent scale and keeps reported abilities comparable across fits. Final model rankings are obtained by sorting the learned abilities \(\theta_i\).

4 Experiments↩︎

In this section, we first validate DualEval’s joint fit and ranking behavior (§4.1), then study two practical applications enabled by the learned item profiles: sample-efficient benchmark compression (§4.2) and residual-based anomaly detection (§4.3).

4.0.0.1 Experimental Setup.

We evaluate DualEval on four settings: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. The first three combine challenging static benchmarks with reward-scored open-ended responses; the generic setting uses arena-style data only, serving as an arena-only stress test without static anchors. We select evaluation-relevant arena prompts using LLM judges, with filtering details in Appendix §13. Static benchmarks provide binary correctness labels, while open-ended responses are scored by our proprietary scalar reward model (RM), developed internally at LM Arena and validated against held-out human preferences (Appendix §9). The primary RM uses a Qwen-3-32B backbone with a reward head and was trained on over five million human-preference pairs using a Bradley–Terry objective. To test robustness to reward-model choice, we also repeat the main static-anchored analyses with the public Skywork-Reward-V2-Qwen3-8B RM, which has lower agreement with held-out human preferences (Appendix §12). For each model-question pair, we generate one response under a consistent inference setup (Appendix §13). Table ¿tbl:tab:data95models? lists the benchmark composition and evaluated models. For agentic coding tasks that require a fixed execution harness, we use the same harness for all models: Terminus2 for TerminalBench [34] and mini-SWE-agent v2 for SWE-Bench [35].

4.1 DualEval Results↩︎

Figures 1, 4, 5, and 6 show the learned model abilities and item-difficulty distributions across the four settings, with math, miscellaneous, and generic visualizations provided in Appendix §7.

4.1.0.1 Static and Arena Fit.

DualEval reconstructs both signal types: 88–92% binary accuracy on static labels across the three static-anchored settings, and 68–81% decisive-pair agreement on arena comparisons across all four settings. This arena agreement is comparable to the proprietary reward model’s agreement with held-out human preferences, suggesting that the joint latent structure captures a substantial share of the preference signal rather than merely fitting noise. Reliability diagrams in Appendix §11 show approximate calibration for both static success probabilities and arena preference probabilities, with ECE \(0.04\)–\(0.07\) on static cells and \(0.01\)–\(0.03\) on arena soft pairwise targets, suggesting no systematic biases of over- or under-confidence across domains.

4.1.0.2 Difficulty Structure.

The learned item profiles reveal substantial heterogeneity in difficulty and sharpness. Many items lie far from the main ability cluster, indicating saturation or difficulty beyond the evaluated model pool. We also observe a modality difference: static items show near-zero correlation between difficulty \(b_q\) and log-sharpness \(\log a_q\), whereas arena items show a stronger positive association (\(\rho \approx 0.6\) across domains; Appendix §14). This suggests that difficulty and sharpness contribute differently across evaluation sources, motivating the Fisher-information score in §4.2, which combines both quantities rather than selecting items by difficulty or sharpness alone.

4.1.0.3 Static–Arena Integration.

We next test whether joint calibration can integrate static correctness and arena preference signals without collapsing to either source alone. For each static-anchored setting, we compare DualEval with three baselines: a static-only 2PL IRT model trained on benchmark correctness, an arena-only Bradley–Terry model trained on reward-derived pairwise comparisons, and an arena-only average-reward ranking. We average results over three random question splits. Each method is evaluated by held-out arena decisive-pair accuracy and Spearman correlation with two full-data references: static-only IRT and arena-only Bradley–Terry rankings.

Table 1 shows the expected source-specialization tradeoff. The static-only model aligns best with the static reference but transfers weakly to the arena reference. Arena-only baselines achieve the highest arena pair accuracy and strongest agreement with the arena reference, but align less well with the static reference. DualEval is designed to balance these sources: it preserves nearly the same static-reference agreement as static-only training while improving arena-reference agreement over the static-only baseline. This supports the role of the joint model as an integration mechanism rather than a single-source optimizer.

We also illustrate the stabilizing role of DualEval when arena supervision is noisier. Replacing our proprietary RM with the public Skywork RM substantially changes arena-only rankings: Arena BT’s agreement with the public Arena sub-leaderboards drops to \(\rho=0.51/0.54/0.25\) for coding, math, and miscellaneous, respectively. In contrast, DualEval with the public RM preserves high agreement with the static reference ranking, with Static \(\rho=0.976/0.984/0.963\), and remains closer to the proprietary-RM DualEval ranking than arena-only training does. A direct cross-RM comparison yields Spearman correlations of \(0.963\), \(0.992\), and \(0.835\) for coding, math, and miscellaneous, respectively (Appendix §12). The arena-only generic setting provides the no-static-anchor counterfactual: without static labels, proprietary- and public-RM rankings diverge sharply (\(\rho=0.25\)). Together, these results suggest that static anchoring stabilizes the shared latent scale when preference supervision is noisy.

2.5pt

4.1.0.4 Ablations.

A pair-treatment ablation (Appendix §15, Table 4) confirms the importance of both-bad anchoring: removing the loss component drops both-bad AUC from 0.903 to 0.622 while leaving held-out arena pair accuracy near 0.78. Tie filtering has a smaller effect on pair accuracy but slightly improves both-bad calibration. The full configuration is used in all subsequent experiments.

4.2 Item Informativeness↩︎

A central diagnostic question for any benchmark is which items carry the bulk of the ranking signal between models, and which are redundant. The joint formulation produces item-level parameters that let us answer this question directly: we score each item by its Fisher information about model abilities under the fitted DualEval objective, then quantify how much of the full-data ranking can be recovered from the most informative items alone.

4.2.0.1 Quantifying Informativeness via Fisher Score.

For a static item, \(p_{i,q}=\sigma(a_q(\theta_i-b_q))\), where \(a_q\), \(b_q\), and \(\theta_i\) are the learned item sharpness, difficulty, and model ability. The Fisher information for model \(i\) on item \(q\) is \(\mathcal{I}^{\mathrm{static}}_{i,q}=a_q^2\,p_{i,q}(1-p_{i,q})\), which we average over models and weight by the static loss coefficient: \[S_q^{\mathrm{static}}= \lambda_{\mathrm{static}}\frac{1}{M}\sum_{i=1}^{M}a_q^2\,p_{i,q}(1-p_{i,q}).\]

For an arena item, DualEval compares responses through \(\mu_{ijq}=\sigma(\gamma(p_{i,q}-p_{j,q}))\). We score each pair by the trace of the Fisher information with respect to the two involved ability parameters \((\theta_i,\theta_j)\), which measures local sensitivity of the predicted preference probability to either model’s ability: \[\begin{align} &\mathcal{I}^{\mathrm{arena}}_{ijq} = \mu_{ijq}(1-\mu_{ijq})\gamma^2 \\ &\quad\cdot \Bigl[ \bigl(a_q p_{i,q}(1-p_{i,q})\bigr)^2 + \bigl(a_q p_{j,q}(1-p_{j,q})\bigr)^2 \Bigr]. \end{align}\] We average this quantity over decisive arena pairs for item \(q\): \[S_q^{\mathrm{arena}} = \lambda_{\mathrm{pref}} \frac{1}{|\mathcal{P}_q|} \sum_{(i,j)\in\mathcal{P}_q} \mathcal{I}^{\mathrm{arena}}_{ijq},\] where \(\mathcal{P}_q\) denotes the set of decisive, i.e., non-tie and non-both-bad, model pairs for item \(q\).

Static and arena items are scored in the same units: expected information about the model ability vector. Averaging over arena pairs prevents items from receiving an artificial advantage from \(O(M^2)\) pairwise comparisons.

4.2.0.2 Subset-Recovery Experiment.

For each domain, we select a fraction of items, refit ability parameters on the subset (item parameters held fixed at full-data values), and measure Spearman correlation between the resulting ranking and the full-data DualEval ranking. We compare five subset-selection strategies, including ablations that isolate the contribution of each component of \(S_q\): (i) Top Fisher info (DualEval), selecting items by descending \(S_q\); (ii) Top sharpness \(a_q\), ablating the \(p(1-p)\) frontier weight; (iii) Difficulty-matched, selecting by smallest \(|b_q-\mathrm{median}(\theta)|\), isolating frontier-matched difficulty alone; (iv) Random (DualEval), a uniform random subset; (v) Random (2PL+BT), a uniform random subset refit with a single-source 2PL+BT baseline that lacks item parameters for arena data.

4.2.0.3 Ranking Signal Is Concentrated in a Small Subset.

Figure 2 reveals substantial item-level redundancy in current benchmarks. Across all three static-anchored domains, the top 10% of items by Fisher information recover the full-data ranking at Spearman \(\rho \geq 0.95\), and the top 30% recover it at \(\rho \geq 0.99\). The full Fisher score is close to the sharpness-only ablation but is the most consistent at small fractions, suggesting that sharpness explains much of the signal while local non-saturation still improves item selection. Difficulty matching alone is less reliable, confirming that proximity to the model-ability range is not sufficient without sensitivity to model differences. The 2PL+BT baseline trails the DualEval variants throughout, indicating that item parameters fit jointly across static and arena data carry information missing from disjoint-source rankings.

4.2.0.4 Implications.

This redundancy structure has two practical consequences. First, leaderboard refreshes against established benchmarks can target the high-Fisher subset for substantial reductions in evaluation cost while preserving ranking confidence. Second, the items with lowest Fisher information can be retired without affecting ranking quality, providing a principled diagnostic for identifying saturated or non-discriminative items. Because informativeness is relative to the evaluated model pool, high-signal subsets should be recalibrated as model abilities shift.

4.3 Residual-Based Anomaly Detection↩︎

A second diagnostic question is whether the joint fit can identify suspicious \((\text{model}, \text{item})\) pairs whose observed outcomes are inconsistent with what DualEval expects given the model’s ability and the item’s difficulty. Such residual anomalies are candidates for contamination, leakage, or data-quality issues, and the joint model’s per-pair expectations make them directly computable.

4.3.0.1 Per-Pair Residual Scores.

For a static \((i, q)\) pair we define the standardized residual \[r^{\mathrm{static}}_{i,q} = \frac{y_{i,q}-p_{i,q}}{\sqrt{p_{i,q}(1-p_{i,q})+\varepsilon}}, \label{eq:residual-static}\tag{1}\] where \(y_{i,q}\in\{0,1\}\) is the observed correctness label and \(p_{i,q}\) is a DualEval success probability. Large positive residuals indicate unexpected successes; large negative residuals, unexpected failures. For an arena \((i,q)\) pair, we average the target–prediction gap across decisive opponents of model \(i\) on item \(q\): \[r^{\mathrm{arena}}_{i,q} = \frac{1}{|\mathcal{P}_q^{(i)}|}\sum_{j\in\mathcal{P}_q^{(i)}}\bigl(\sigma(z_{i,q}-z_{j,q})-\mu_{ijq}\bigr), \label{eq:residual-arena}\tag{2}\] where \(z_{i,q}\) is the standardized reward, \(\mu_{ijq}=\sigma(\gamma(p_{i,q}-p_{j,q}))\) is the DualEval-predicted pairwise probability, and \(\mathcal{P}_q^{(i)}\) is the set of decisive opponents of model \(i\) on item \(q\).

4.3.0.2 Synthetic Contamination Protocol.

We inject synthetic contamination into existing evaluation artifacts. For each static-anchored domain, we sample candidate cells from low-probability success regions. For static labels, the candidate pool consists of failed cells \((y_{i,q}=0)\) with the lowest \(\mathrm{acc}^{\mathrm{model}}_i \times \mathrm{acc}^{\mathrm{item}}_q\), where \(\mathrm{acc}^{\mathrm{model}}_i\) is model \(i\)’s average static accuracy in the domain and \(\mathrm{acc}^{\mathrm{item}}_q\) is the fraction of models that solve item \(q\). Flipping these cells creates unexpected successes by weak models on hard items. For arena responses, the candidate pool is the bottom \(30\%\) of cells by standardized reward \(z_{i,q}\). We sample cells uniformly from each pool at rates \(\{0.5\%,1\%,2\%,5\%\}\), with five random seeds per rate. Static contamination flips the correctness label from \(0\) to \(1\); arena contamination adds \(2.0\) to the standardized reward \(z_{i,q}\) before pairwise targets are rebuilt.

We evaluate two detectors. The contaminated-fit detector scores each cell by its signed positive residual under a DualEval model refit on the contaminated data. The clean-reference detector computes the same scores using a DualEval model fit once on the original clean data, while evaluating the contaminated labels or rewards against clean-fit expectations. Injected cells are positives and all other cells are negatives for cell-level AUROC and AUPRC; Appendix §17 details how these detection metrics are computed.

4.3.0.3 Calibrated Residuals Recover Injected Contamination.

Figure 3 reports AUROC and AUPRC across injection rates and domains. The clean-reference detector recovers injected static-label contamination nearly perfectly, with AUROC \(\geq 0.995\) and AUPRC \(0.80\)–\(0.99\), and remains strong on arena-reward contamination, with AUROC \(0.93\)–\(0.98\) and AUPRC \(0.28\)–\(0.83\). This setting is operationally realistic when a benchmark already has a vetted historical calibration and new submissions or data refreshes are screened against it. The contaminated-fit detector, which uses only the corrupted snapshot, remains above the random baseline across all conditions but is less sensitive when flipped labels are absorbed into the refitted latent parameters. The gap between the two detectors highlights the value of preserving clean benchmark snapshots and calibrated historical fits for ongoing leaderboard auditing.

5 Discussion↩︎

DualEval suggests a shift from treating evaluation items as interchangeable test cases to treating them as calibrated measurement instruments. Once items have learned difficulty, sharpness, informativeness, and residual profiles, benchmark maintenance becomes a model-item problem: evaluators can identify saturated items, preserve high-signal subsets, and audit suspicious outcomes rather than relying only on aggregate leaderboard scores.

This perspective also points to richer extensions. A scalar ability parameter is useful for interpretability and ranking, but many evaluation domains contain multiple skills; multidimensional IRT provides a natural path toward modeling specialization across reasoning, coding, factuality, and interaction quality. More broadly, calibrated item profiles could support adaptive evaluation protocols that choose questions based on the current uncertainty about a model, while retaining item-level evidence for why a ranking changed.

6 Conclusion↩︎

We introduced DualEval, a framework that jointly models static benchmark correctness and arena-style preference signals on a shared latent scale. Across four domains, DualEval produces interpretable model rankings while estimating item difficulty, sharpness, and informativeness. Baseline comparisons show that combining static and arena supervision yields a more balanced signal than either source alone, while item-level analyses support sample-efficient evaluation and diagnostic benchmark auditing. Together, these results point toward evaluation pipelines that jointly rank models and calibrate benchmark items, making leaderboards more efficient, interpretable, and grounded in item-level evidence.

Limitations↩︎

Our study has several limitations. First, the primary arena supervision relies on our internal reward model, which limits direct reproducibility even though we describe its architecture, training objective, and training-data scale. We address this by validating the RM against held-out human preferences (Appendix §9) and replicating the main analyses with the publicly available Skywork-Reward-V2-Qwen3-8B model (Appendix §12), but broader validation across multiple public RMs and human preference annotations remains important.

Second, each model-question pair is evaluated with a single generated response. This matches many leaderboard settings but does not capture within-model sampling variability, sensitivity to decoding parameters, or multi-attempt performance. Extending DualEval to repeated responses would allow item sharpness and residual anomalies to separate model capability from response-level stochasticity.

Finally, the current formulation uses a scalar ability parameter within each domain. This keeps rankings interpretable, but future evaluation systems should move toward multidimensional model profiles that capture specialization across skills, task formats, and interaction settings. Such extensions could make item selection adaptive not only to overall model strength, but also to the specific capabilities an evaluator wants to measure.

7 Additional DualEval Ranking Visualizations↩︎

We provide additional examples of DualEval rankings of math, miscellaneous, and generic domains in Figures  4, 5, and 6.

8 DualEval Hyperparameters and Configurations↩︎

All DualEval experiments in this paper use the same hyperparameters unless otherwise noted: Adam optimizer with learning rate \(0.02\), \(2000\) epochs, loss weights \(\lambda_{\text{static}}=\lambda_{\text{pref}}=1.0\) and \(\lambda_{\text{bb}}=0.2\), L2 regularization coefficient \(\lambda_{\text{reg}}=0.01\) applied to \(\theta\), \(b\), \(k\), and \(\log \tilde{\gamma}\), and pair-treatment thresholds derived as the 15^th percentile of \(\max(z_i, z_j)\) for both-bad flagging and the 15^th percentile of \(|z_i - z_j|\) for tie filtering, applied per-domain. The arena temperature is parameterized as \(\gamma = \gamma_0 \cdot \tilde{\gamma}\) with \(\gamma_0 = 4\), and \(\log \tilde{\gamma}\) is L2-regularized toward zero. After each optimizer step, model abilities are zero-mean centered with the same shift absorbed into question difficulties, preserving all differences \(\theta_i - b_q\). All fits use single-precision floating point on CPU; a single domain’s joint fit completes in 10–25 seconds. Arena-only fits on the generic domain use the same configuration with the static loss disabled.

9 Validation of Reward Model with Real Human Preference↩︎

This appendix supports the robustness claims in Section 4 by (i) quantifying how well each reward model agrees with real human preference on the prompts actually used by our pipeline, (ii) visualizing cross-RM agreement of DualEval’s learned model abilities in the static-anchored domains, and (iii) documenting the arena-only generic domain as a no-anchor reference.

9.0.0.1 Reward-model alignment with human preference.

For each domain we identify a subset of arena prompts for which human preference labels are available, and that have a decisive (non-tie) human winner. Sample sizes per domain are reported in Table 2. Both reward models are scored on the same pairs from this filtered subset. Table 2 reports the per-domain pair count and the RM’s accuracy at predicting the human winner. Our proprietary RM is substantially more human-aligned than the public RM in every domain.

6pt

10 Stability of DualEval Rankings↩︎

We assess the stability of DualEval rankings using a question-level cluster bootstrap. For each domain we draw \(B=100\) bootstrap replicates, resampling questions with replacement stratified by source (each benchmark source plus arena), so every replicate has the same per-source \(N\) as the original. Drawn copies of a question receive unique IDs so the IRT fit treats them as independent items–the standard cluster-bootstrap behavior. Reward \(z\)-scores are renormalized per replicate. For each replicate we refit DualEval with the published hyperparameters and record per-model \(\hat{\theta}_i\). We use the same procedure to bootstrap Static 2PL and Arena BT for direct comparison (Static 2PL is omitted for the arena-only generic domain).

Figure 8 shows per-model \(\hat{\theta}_i\) with symmetric 95% CIs (\(\hat{\theta}_i \pm 1.96,\hat{\sigma}_{\text{boot}}\)) for all three methods in each domain. Top-3 and bottom-3 positions are stable to \(\pm 1\) rank across replicates in every domain. Middle-band positions span 4–6 ranks of 95% CI in the three static-anchored domains, indicating that not all adjacent leaderboard positions are statistically distinguishable at the current evaluation size. Generic–being arena-only with 500 prompts contributing \(\binom{18}{2}\) pairwise comparisons per item–produces the tightest CIs across the four domains.

Table 3 reports per-(domain, method) summary statistics. On equivalent data (generic, arena-only), DualEval matches or slightly improves over Arena BT (median \(\hat{\sigma}_\theta = 0.016\) vs \(0.019\)), confirming that the IRT structure adds discriminative information without inflating uncertainty. Against Static 2PL on the static-anchored domains, the joint formulation reduces ranking variance by 30–60%. Against Arena BT on those domains, DualEval carries a modest variance premium that reflects the additional cross-source information being integrated, in exchange for substantially better Static-\(\rho\) (Table 1).

5pt

11 Held-out calibration↩︎

We evaluate calibration of the joint fit’s per-pair predictions on a cell-level 80/20 random holdout (averaged over three random seeds; same fit hyperparameters as §4). On held-out static cells, predicted success probabilities \(p_{i,q}\) match observed binary outcomes with expected calibration error (ECE) of \(0.04\)–\(0.07\) across coding/math/misc domains. On held-out arena pairs, predicted preference probabilities \(\mu_{ijq}\) match the soft pairwise targets \(\sigma(z_{i,q}-z_{j,q})\) with ECE of \(0.01\)–\(0.03\), with reliability diagrams nearly diagonal across the predicted-probability range (Figure 9). For transparency we also report ECE against the binarised pairwise outcome \(\mathbf{1}[\,\sigma(z_{i,q}-z_{j,q}) \geq 0.5\,]\), which is higher (\(0.07\)–\(0.16\)) due to expected under-confidence when soft pairwise targets near \(0.5\) are discretised to hard win/loss labels; this binarisation does not reflect a miscalibration of the underlying joint fit. These calibration results justify treating \(p_{i,q}\) and \(\mu_{ijq}\) as meaningful probability estimates in the downstream Fisher information (§4.2) and residual diagnostics (§4.3).

12 Cross-RM Ranking Agreement↩︎

Figure 10 plots DualEval’s learned model abilities under the proprietary and public RMs against each other, one panel per static-anchored domain. Points fall close to the diagonal: Spearman correlations are \(0.963\) (coding), \(0.992\) (math), and \(0.835\) (misc), consistent with the Static \(\rho \geq 0.96\) entries in Table 1 and indicating that static anchoring stabilizes the ranking against RM-induced noise. The arena-only generic domain provides the no-anchor counterfactual: with no static labels to pin the latent scale, the agreement between the proprietary and public RMs drops to \(\rho = 0.25\), empirically isolating static anchoring as the mechanism behind the joint framework’s cross-RM stability.

13 Evaluation Details↩︎

13.0.0.1 Benchmark sources.

Our static benchmark pool draws from LiveCodeBench [36], MBPP-Plus from EvalPlus [37], SWE-Bench [35], TerminalBench [34], AIME 2025/2026 [38], Humanity’s Last Exam (HLE) [39], OlympiadBench/Olympiad-Math [40], and SimpleQA [41]. Arena prompts are drawn from publicly available LMArena [6] logs and filtered by domain as described below.

13.0.0.2 Prompt sampling and filtering.

Static items are drawn from each source benchmark by seeded stratified sampling, with strata defined by the natural axes of each source: difficulty tier for LiveCodeBench-v6, problem type for AIME and Olympiad-Math, HLE level for HLE-Math, and field for HLE-Misc (Bio./Med., Engineering, Humanities/Social Science, Other); SimpleQA is stratified by topic. MBPP-Plus, SWE-Bench-lite, and TerminalBench-2.0 are taken in full up to the per-source quotas in Table ¿tbl:tab:data95models?, after dropping items shorter than 150 characters. Arena prompts are sampled from LMArena releases: arena-expert-5k for coding, math, and miscellaneous, andarena-human-preference-140k for generic. We first apply a coarse domain pre-filter using the dataset-provided occupational and category tags, then run an LLM judge (gpt-5.4-mini) with a domain-specific rubric that returns an inclusion label and a confidence score: coding requires the answer to depend on runnable code, math requires the core task to be mathematical reasoning or calculation, miscellaneous requires expert domain knowledge outside code/math, and generic requires a non-technical everyday task with no code or math content. Candidates with confidence below the per-domain threshold (\(0.5\) for coding/math/misc, \(0.8\) for generic) are dropped, and the top-\(N\) surviving prompts by judge confidence are retained to match the per-domain arena pool sizes in Table ¿tbl:tab:data95models? (coding \(300\), math \(200\), miscellaneous \(300\), generic \(500\)).

13.0.0.3 Evaluation setup.

For each (model, question) pair we generate a single response with default sampling settings for each provider’s API as of submission (temperature and top-\(p\) are model-default; no system prompt is prepended for arena prompts; a standard “provide a complete, correct solution” instruction is appended for coding/math/misc). Model identifiers are reported in Table ¿tbl:tab:data95models?. For agentic SWE-Bench tasks we route every model through mini-SWE-agent-v2 with a single attempt and the harness’s default rollout budget; for TerminalBench-2.0 we use Terminus2 under identical configuration for all models.

13.0.0.4 Static grading.

Coding tasks are graded by their reference execution harness (test-suite pass for MBPP-Plus and LiveCodeBench; resolved-instance flag for SWE-Bench; harness pass for TerminalBench). Math and miscellaneous items use exact-answer matching after a normalization pass on the final-answer line; ambiguous cases are resolved by an LLM judge with chain-of-thought disabled.

13.0.0.5 Arena prompts.

For coding, math, and miscellaneous we filter LMArena prompts to the target domain via LLM judges using a fixed rubric. The generic domain uses LM Arena’s category-mix without domain filtering. Per-domain arena pool sizes also appear in Table ¿tbl:tab:data95models?.

13.0.0.6 Reward scoring.

Each model response is scored by a single forward pass of the relevant reward model (proprietary or Skywork-Reward-V2-Qwen3-8B for the replication). Rewards are global-standardised per (domain, RM) before pairwise targets are constructed.

14 Difficulty and Sharpness↩︎

Figure 11 plots learned item difficulty \(b_q\) against log-sharpness \(\log a_q\) for every domain, with points colored by benchmark source. Static items show near-zero or weakly negative correlation in every static-anchored domain (\(\rho = -0.00, +0.13, -0.36\) for coding, math, misc), confirming that difficulty and sharpness are largely independent properties of a static item: saturated items can be sharp, hard items can be flat, and Fisher information cannot be inferred from difficulty alone. Arena items, by contrast, show a robust positive coupling (\(\rho \in [+0.58, +0.73]\) across all four domains), reflecting how continuous reward signals propagate both location and spread into the IRT parameters. The joint DualEval formulation accommodates both regimes without imposing a single structural assumption on either, and the contrast itself confirms that the framework’s learned parameters reflect the structure of the underlying outcome modality.

15 Pair-Treatment Ablation↩︎

We ablate the two pair-treatment components introduced in §3: both-bad (BB) anchoring and tie filtering. All variants are evaluated on the coding domain using the same 20% held-out pair set constructed with the full configuration thresholds, so metrics are directly comparable. The full model uses BB flagging with \(\lambda_{bb}=0.2\) and tie filtering. Arena pair accuracy is measured on held-out decisive pairs; BB-AUC measures whether \(1-\min(p_i,p_j)\) ranks BB pairs above non-BB pairs.

4pt

Table 4 shows that BB anchoring is the dominant choice: removing the BB loss drops BB-AUC from \(0.903\) to \(0.622\) while leaving arena pair accuracy near \(0.78\), and removing BB flagging entirely gives essentially the same degradation. Tie filtering has a smaller, complementary effect — it slightly improves BB calibration when combined with the full model (0.903 vs 0.899) but removing it alone barely changes either metric. The full configuration is used in all main-text experiments.

16 Public Leaderboard Rankings↩︎

Table 5 lists the public LMArena Text-leaderboard ranks for the 18 models we evaluate, taken from the matched public Arena Text sub-leaderboards used in Table 2 (Coding, Math, Text-Expert, Text-Overall). These are the external reference rankings against which Table 1’s Public \(\rho\) column is computed. Lower rank is better. Public ranks were retrieved on the same date the paper’s arena prompts were sampled.

5pt

17 Contamination-Detection Metric Details↩︎

For each domain, modality, injection rate, and random seed, the synthetic contamination protocol defines a binary mask \(m_{i,q}\in\{0,1\}\) over model–item cells, where \(m_{i,q}=1\) denotes an injected cell. Each detector assigns a scalar anomaly score \(s_{i,q}\) to every cell using the residual scores defined in §4.3. We then evaluate whether these scores rank injected cells above non-injected cells.

AUROC is computed as the probability that a randomly selected injected cell receives a higher score than a randomly selected non-injected cell, with ties counted halfway: \[\mathrm{AUROC} = \Pr(s^+ > s^-) + \tfrac{1}{2}\Pr(s^+ = s^-),\] where \(s^+\) and \(s^-\) denote scores from injected and non-injected cells, respectively.

AUPRC is the area under the precision–recall curve obtained by thresholding the anomaly scores. For a threshold \(t\), precision and recall are \[\mathrm{Prec}(t)= \frac{\sum_{i,q}\mathbf{1}[s_{i,q}\ge t]m_{i,q}}{\sum_{i,q}\mathbf{1}[s_{i,q}\ge t]},\] \[\mathrm{Rec}(t)= \frac{\sum_{i,q}\mathbf{1}[s_{i,q}\ge t]m_{i,q}}{\sum_{i,q}m_{i,q}}.\] We compute the precision–recall curve over all score thresholds and report the area under this curve. Both AUROC and AUPRC are computed at the cell level and averaged over random injection seeds for each injection rate and modality.

18 Examples↩︎

18.0.0.1 Item-Profile Gallery.

Table 6 walks the four corners of the (difficulty \(b_q\), sharpness \(a_q\)) plane on coding-domain static items. The example items are drawn directly from the full-data DualEval fit; pass counts are observed across the 18 evaluated models. This gallery illustrates qualitatively what the Fisher subset experiment in §4.2 measures quantitatively: ranking signal lives in items where both \(a_q\) is high and the pass rate is between the extremes, and items in the saturated and uniformly hard corners contribute essentially no ranking information.

4pt

18.0.0.2 Anomaly Candidates.

Table 7 lists representative anomaly candidates surfaced by the standardized DualEval residual diagnostic of §4.3, restricted to natural (un-perturbed) responses with \(|r_{i,q}| > 3.5\sigma\). These are candidates, not verified contamination: large residuals can also reflect MC guessing on items with few options, judge artifacts, or genuine but narrow capability not captured by the joint fit. Semantic-preserving perturbation audits are the appropriate follow-up to distinguish memorization from robust capability and are left to future work. The patterns observed across the three picks (number-sequence puzzle archives, specialized MC questions, named MBPP function signatures) are nevertheless consistent with prior exposure as a plausible alternative hypothesis.

4pt

References↩︎

---

1. Correspondence: aaronjli@berkeley.edu↩︎