1 Introduction↩︎

Large language models (LLMs) [1]–[3] have achieved remarkable capabilities across a wide range of open-ended generation tasks. A key driver of this progress is post-training, which aligns pretrained models with human intentions through techniques such as Reinforcement Learning with Human Feedback (RLHF) [4], [5] and Reinforcement Learning with Verifiable Rewards (RLVR) [6], [7]. Despite their differences in reward acquisition, both approaches share a common dependency: reliable reward or preference signals that distinguish better responses from worse ones. Constructing such signals is especially challenging for open-ended tasks, where response quality is multidimensional, context-dependent, and difficult to reduce to a single opaque score. How these signals are defined, produced, and validated has therefore emerged as a central challenge for both evaluation [8], [9] and optimization.

A natural way to make these signals more transparent — inspired by how humans evaluate open-ended work — is to express evaluation criteria as rubrics or checklists [10]–[12]. Rather than collapsing quality into a single opaque judgment, a rubric decomposes evaluation into concrete, verifiable sub-questions. Prior work has shown that rubric- or criteria-conditioned judges can substantially narrow the gap with human raters [13], [14], and that rubric-style process supervision can stabilize reasoning-oriented reinforcement learning [15], [16]. Together, these results position rubrics as a promising interface for both evaluation and optimization. However, existing approaches typically rely on gold responses, expert-written references, large-scale preference datasets, or extensive human annotations to construct effective rubrics — requirements that are costly and frequently unavailable in realistic open-ended settings.

Recent annotation-free approaches [13], [17] reduce this cost by generating rubrics directly from the query. Yet they typically rely on a single generic evaluator to enumerate all quality criteria at once. This single-voiced formulation assumes that one evaluator can recover the full structure of human preference, overlooking the fact that human judgment is inherently distributed across complementary perspectives: users emphasize usefulness, domain experts prioritize correctness, and educators focus on clarity, evidence, and safety. Consequently, single-voiced rubrics tend to omit preference-relevant dimensions — a failure mode we term dimensional blind spots.

Dimensional blind spots lead to two concrete failures. First, they make judging accuracy highly domain-dependent: as shown in Figure 1 (a), single-voiced rubric judgment performs well in technically structured domains but degrades when domain-specific quality criteria are insufficiently covered. Second, they enable rubric hacking: as shown in Figure 1 (b), responses can score well by optimizing for covered criteria while remaining low quality. Table 1 illustrates this mechanism concretely: for a broad NASDAQ-related question, a single-voiced rubric rewards surface-level task completion while overlooking feasibility, epistemic calibration, and unverifiable claims. Additional analyses are provided in Appendix 8.

To address these blind spots, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that recasts rubric generation as multi-role elicitation. Rather than relying on a single generic evaluator, MRRG instructs an LLM to adopt complementary evaluative roles — such as user, domain expert, educator, AI researcher, and linguist — each generating compact, atomic, and verifiable rubric items. These are then pooled and deduplicated into an auditable rubric-based scorer suitable for both preference validation and as a reward model in GRPO-style RLVR. Figure 2 illustrates the overall framework. Our contributions are as follows:

• Dimensional blind spots. We identify dimensional blind spots as a key failure mode of single-voiced rubric generation and demonstrate that they cause domain-dependent judging accuracy and rubric hacking, as illustrated in Figure 1.

• Multi-Role Rubric Generation. We propose MRRG, a training-free and reference-free framework that elicits rubric criteria from multiple complementary roles and aggregates them into an auditable scorer for both LLM-as-a-judge evaluation and RLVR reward modeling.

• Empirical validation. On RewardBench-2, JudgeBench, and PPE, MRRG consistently outperforms the strongest listed baseline across Qwen2.5 and GPT-series models, with gains of 3.1–16.4 percentage points. In GRPO-style RLVR experiments, MRRG improves over the strongest single-voice reward baseline by 1.7 points on BiGGen Bench and 3.4 points on HealthBench-Hard.

5pt

2 Method↩︎

This section develops MRRG in four steps. We first formalize rubric-based scoring as a unified interface that subsumes both LLM-as-a-judge and RLVR reward modeling (Sec. 2.1). We then introduce our multi-role rubric generator, MRRG (Multi-Role Rubric Generation, Sec. 2.2), and describe how the resulting per-role rubrics are pooled into a scalar score (Sec. 2.3) that can be used, without modification, as both a preference validator and an RLVR reward (Sec. 2.4).

2.1 Preliminaries: Rubric-Based Scoring↩︎

Let \(x\in\mathcal{X}\) denote an input prompt and \(y\in\mathcal{Y}\) a candidate response. Every alignment signal aims to approximate the latent human quality function \(q^{\star}:\mathcal{X}\times\mathcal{Y} \to \mathbb{R}\). Classical reward models parametrize \(q^{\star}\) as a scalar network \(r_\theta(x,y)\) trained on preference pairs [4], [5]; LLM-as-a-judge approaches instantiate it as a prompted language model, \(q^{\star}(x,y)\approx G\big({prompt}_{{judge}}(x,y)\big)\) [8]. Both collapse the multi-dimensional structure of \(q^{\star}\) into an opaque scalar.

2.1.0.1 Notation.

Throughout the paper, \(G(\cdot)\) denotes a call to a pretrained language model. Different uses of \(G\) are distinguished by the prompt template provided to the model. We use \({prompt}_{{judge}}\) to denote the per-criterion evaluation template, \({prompt}_{m}\) for role-conditioned rubric generation under role \(p_m\), and \({prompt}_{r}\) for generic rubric generation.

2.1.0.2 Rubric-based scoring.

A rubric-based scorer breaks this collapse by introducing an interpretable intermediate variable. A rubric \(R(x)=\{c_{k},w_{k}\}_{k=1}^K\) is a set of criteria, each \(c_k\) being an atomic, verifiable proposition about the response (e.g., “The answer cites the correct theorem”). Each criterion is evaluated by the same language model to \(s_k=G\big({prompt}_{{judge}}(x,y,c_k)\big)\in\{0,1\}\), and the overall score aggregates criterion-level judgments: \[\label{eq:rubric-score} \begin{align} &S(x,y;R)\;\\&=\;\frac{\sum_{k=1}^{K} w_{k}\cdot G\big({prompt}_{{judge}}(x,y,c_k)\big)}{\sum_{k=1}^{K} w_{k}}, \end{align}\tag{1}\] where \(w_{k}>0\) is the importance of the \(k\)-th criterion. This decomposition makes the signal simultaneously auditable (every contribution \(w_k s_k\) is attributable to a named criterion), verifier-like (each \(s_k\) is a constrained yes/no or graded judgment), and model-agnostic (any sufficiently capable LLM can serve as \(G\)). The central design question thus becomes: how should \(R(x)\) be constructed so as to cover more comprehensive evaluative dimensions and thus be close to \(q^{\star}\)?

2.2 Multi-Role Rubric Generation↩︎

Existing rubric generators [17]–[19] construct initial \(R(x)\) with one or a few LLM calls, e.g., \(R(x)=G\big({prompt}_{r}\big(x,\hat{y}\big)\big)\) with \(\hat{y}=G(x)\) a sample response, where a “universal evaluator” is asked to enumerate every relevant criterion at once. As argued in Sec. 1, this single-voice formulation compresses genuinely distinct viewpoints into a shared prior and could lead to dimensional blind spots.

2.2.0.1 Role-conditioned elicitation.

A straightforward approach to broadening the evaluative dimensions in rubric generation is to repeat the generation process multiple times. Although this can increase the apparent diversity of the resulting rubrics, it remains constrained by the inherent biases and limitations of a single evaluative perspective or role. To address this limitation, we recast rubric generation as a multi-role elicitation problem. Let \(\mathcal{P}=\{p_1,\ldots,p_M\}\) be a small, fixed set of evaluative roles; by default \(\mathcal{P}=\{\mathrm{\small User},\;\mathrm{\small Domain Expert},\; \mathrm{\small Educator},\\ \mathrm{\small AI Researcher},\;\mathrm{\small Linguist}\}\). For each role \(p_m\) we supply a role-specific template \({prompt}_m\) that foregrounds the concerns most salient to that role—e.g., the User template emphasizes problem-solving utility and intent satisfaction, while the Educator template emphasizes evidential support, communication quality and safety. This produces a role-specific rubric \[\label{eq:per-role} \begin{align} R_{m}(x)\;&=\;G\big({prompt}_m(x,\hat{y})\big) \;\\&=\;\{(c_{m,k},w_{m,k})\}_{k=1}^{K_m}, \end{align}\tag{2}\] where \(m=1,\ldots,M\) and \(\hat{y}=G(x)\) is a sample response. Each \(R_m(x)\) is compact (\(K_m\in[3,7]\) in practice) and internally coherent, because the generator only has to reason from one viewpoint at a time. Importantly, \(w_{m,k}\) is produced jointly with \(c_{m,k}\) inside the same role-conditioned call: the role that writes a criterion is also the one best positioned to judge its importance.

2.2.0.2 Criterion format and self-assessed importance.

Each role is also instructed to emit binary criteria paired with a self-assessed importance weight \(w_{m,k}\in\{1,2,3\}\) on a three-point scale: \[\label{eq:weight-scale} w_{m,k}= \begin{cases} 3 & \text{core need}\\ 2 & \text{significant issue }\\ 1 & \text{non-critical polish} \end{cases}\tag{3}\]

2.2.0.3 Selection of Roles.

One might consider selecting a question-specific subset of roles, since the perspectives most relevant to evaluation can in principle depend on the input question. We instead adopt a fixed but carefully chosen role pool \(\mathcal{P}\), motivated by four considerations. (i) Determining the “most relevant” roles for an arbitrary question is itself ill-posed and would likely require an LLM-based role selector. Such a selector could inherit, or even amplify, the same dimensional biases that MRRG aims to mitigate. (ii) The Domain Expert role is already instantiated conditionally on the input question, providing question-aware specialization without exposing the high-level role-selection process to additional model bias. (iii) The default roles cover the principal axes along which humans evaluate open-ended generation: practical utility (User), technical correctness (Domain Expert), communicative clarity (Educator), AI-specific reasoning and model behavior (AI Researcher), and linguistic quality (Linguist). (iv) Keeping the role pool small, with \(|\mathcal{P}|=M\) and \(M=5\) in our experiments, bounds the rubric-generation cost by \(\mathcal{O}(M)\) LLM calls per prompt, independent of dataset size. We study the sensitivity of MRRG to the choice of \(\mathcal{P}\) in Section 3.2.0.1.

2.3 Rubric Deduplication↩︎

The per-role rubrics \(\{R_m(x)\}_{m=1}^{M}\) may contain repeated criteria, since different roles can independently produce the same requirement. In practice, we also observe that LLMs sometimes generate identical criteria within role-specific rubrics, potentially due to hallucinated redundancy. If left untreated, such exact duplicates may be counted multiple times and distort the final rubric-based judgment. To address this issue, we apply a simple deterministic post-processing step.

We first concatenate all role-specific rubrics into a single list, \[\label{eq:concat95rubrics} R_{1:M}(x) = \big[ (c_{m,k}, w_{m,k}) \big]_{m=1,k=1}^{M,K_m}.\tag{4}\] We then scan this list in order and remove criteria whose text exactly matches a criterion that has already appeared. The first occurrence is kept together with its original weight, while subsequent duplicate occurrences are discarded. The final rubric is \[\label{eq:deduplicated95rubric} \widetilde{R}(x) = \big[ (\tilde{c}_{j}, \tilde{w}_{j}) \big]_{j=1}^{N},\tag{5}\] where \(N\) is the number of criteria in the final rubric.

Importantly, this procedure only removes exact duplicates and leaves all non-identical criteria unchanged, including criteria that are semantically similar or potentially in tension with one another. Although one could apply semantic deduplication or consolidation methods, such as using an LLM to merge redundant criteria or resolve conflicting ones, we avoid this design choice for two reasons. First, LLM-based consolidation introduces an additional judgment step, where decisions about redundancy and conflict may themselves be biased and may propagate such bias into the final rubric. Second, merging or removing semantically similar criteria from different roles may inadvertently weaken important evaluation dimensions: when multiple roles independently emphasize similar criteria, this agreement can signal that the criterion is important. Collapsing these criteria into a single item may reduce their effective influence and thus affect the final judgment. We therefore adopt exact-match deduplication as a conservative, deterministic, and reproducible choice. Additional empirical analysis is provided in Appendix 8.3.

2.4 From Rubric Score to Alignment Signal↩︎

The scalar \(S(x,y;\widetilde{R}(x))\in[0,1]\) plugs into both downstream settings without modification.

2.4.0.1 Preference validation (LLM-as-a-judge).

Given two candidates \((y_A,y_B)\) for the same prompt \(x\), we predict the preferred response as \(\arg\max_{i\in\{A,B\}} S(x,y_{i};\widetilde{R}(x))\).

2.4.0.2 Reward modeling for RLVR.

We use \(S(x,y;\widetilde{R}(x))\) directly as the reward inside GRPO-style optimization [7]. For a prompt \(x\) with a rollout group \(\{y_i\}_{i=1}^{G_{\text{roll}}}\), the group-relative advantage of \(y_i\) is \[\label{eq:grpo-advantage} A_{i}\;=\;\frac{S(x,y_{i};\widetilde{R}(x))-\mu_{S}(x)}{\sigma_{S}(x)+\varepsilon},\tag{6}\] where \(\mu_{S}(x)\) and \(\sigma_{S}(x)\) denote the mean and standard deviation of \(\{S(x,y_{i};\widetilde{R}(x))\}_{i=1}^{G_{\text{roll}}}\) taken within the rollout group for prompt \(x\) and \(\varepsilon\) is a small constant for numerical stability. Because \(\widetilde{R}(x)\) is generated once per prompt and cached, the per-rollout reward cost reduces to \(N\) yes/no judge calls, which parallelize trivially across the group.

3 Experimental Results↩︎

We evaluate MRRG along the two downstream settings it was designed for: (i) preference validation, where MRRG acts as an LLM-as-a-judge on benchmark preference pairs (Sec. 3.2); and (ii) reward modeling for RLVR, where the same rubric-based scorer drives GRPO-style policy optimization (Sec. 3.3). Across the experimental part we answer the following questions:

• Can MRRG outperform single-voiced rubric generators and other baselines in both preference validation and RLVR reward modeling?

• Is MRRG robust to the choice of roles, or does it rely critically on any specific role?

• Does MRRG expand the coverage of evaluative dimensions beyond single-voiced generation, or are its gains simply due to producing more criteria?

4.5pt

3.1 Experimental Setup↩︎

3.1.0.1 Rubric generator and judge models.

For preference validation experiments, we instantiate \(G(\cdot)\) with a diverse range of models including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, GPT-oss-20b and GPT-oss-120b for rubric generation and per-criterion judging. And all methods are evaluated on JudgeBench [20], RewardBench-2 [21] and PPE [22]. As for RLVR experiments, we use Qwen2.5-3B-Instruct as base model for GRPO, while GPT-4o is used for rubric generation for different strategies and gpt-oss-120b is used for per-criterion judging. We select \(2000\) questions from Wildchat [23] as training set, and models are evaluated on two widely used open-ended-task datasets: BiGGen Bench [24] and HealthBench-Hard [25].

Detailed description for these two datasets is provided in Appendix 6. Also, full prompt templates for all five roles, and the judge template are provided in Appendix 9.

3.1.0.2 Baselines.

We compare MRRG against three methods: (i) Single-voice rubric generator without sample responses (SVRG w/o SR): the method generates rubrics for questions without sample responses. (ii) Single-voice rubric generators with sample responses (SVRG w/ SR): the method generates rubrics for specific questions with sample responses. (iii) Chasing the Tail[17]: the method generates and refines rubrics by comparing great and diverse responses.

3.2 Preference Validation↩︎

Table 2 shows that MRRG consistently achieves the strongest performance across all three benchmarks and all backbone models, often with substantial margins over the baselines. On the Qwen2.5 series, MRRG improves upon the SVRG w/o SR baseline by more than \(20\) points on RewardBench-2 for the 3B, 7B, and 32B variants, and yields comparable improvements on JudgeBench. On the GPT series, MRRG remains uniformly superior, raising GPT-OSS-120B from \(52.3\) to \(74.5\) on RewardBench-2 and from \(56.5\) to \(74.8\) on JudgeBench. These consistent gains across model scales and families demonstrate that the advantage of multi-role rubric generation is not tied to a particular backbone. They also suggest that MRRG constructs more comprehensive and discriminative rubrics for open-ended evaluation.

We also visualize the rubrics generated by different roles and by the single-voiced process in Figure 3. The visualization shows that role-specific rubrics largely cover the regions occupied by single-voiced rubrics. Meanwhile, they also extend to additional regions where few or no single-voiced rubrics are present, suggesting that the multi-role process captures evaluative dimensions that are underexplored by single-voiced generation. This provides qualitative evidence that MRRG yields more diverse and comprehensive criteria.

3.2.0.1 Ablations.

Table 3 isolates the contribution of different components of MRRG. (i) Role ablation: using rubrics from any single role leads to a clear performance drop compared with MRRG, indicating that no individual perspective is sufficient to construct a comprehensive evaluation rubric. Among the single-role variants, Educator performs best on both RewardBench-2 and JudgeBench, achieving 37.8 and 44.8, respectively. Nevertheless, it still lags behind MRRG by 13.4 points on RewardBench-2 and 10.3 points on JudgeBench, highlighting the importance of aggregating complementary evaluative perspectives. (ii) Repeated single-voiced rubric generation: To examine whether the advantage of MRRG simply comes from generating more criteria, we introduce a repeated single-voiced rubric generation baseline, denoted as RSVRG. Specifically, RSVRG repeats the single-voiced rubric generation process \(5\) times and then applies the same post-processing pipeline as MRRG. As shown in Table 3, RSVRG improves over individual-role variants in some cases, especially when sample responses are enabled, suggesting that repeated sampling can increase rubric diversity to a limited extent. However, RSVRG w/ SR remains substantially below MRRG, with gaps of \(7.3\) points on RewardBench-2 and \(10.0\) points on JudgeBench. This indicates that simply generating more single-voiced rubrics cannot fully substitute for explicitly eliciting rubrics from multiple complementary roles. Overall, these results suggest that the gains of MRRG arise from role-based diversity rather than merely from an increased number of generated criteria.

In addition, we conducted an experiment to compare single-role baselines with leave-one-role-out MRRG variants, as shown in Figure 4. The results show that the most effective single role varies across backbone models, suggesting that no single evaluative perspective is uniformly optimal. In contrast, removing a role from MRRG leads only to moderate performance changes in most cases. Although excluding stronger single roles tends to result in larger drops, the overall degradation remains limited. This indicates that MRRG does not depend critically on any particular role; rather, its effectiveness comes from aggregating complementary perspectives to construct more comprehensive rubrics.

3.3 Reward Modeling for RLVR↩︎

3.3.0.1 Setup.

We plug \(S(x,y;\widetilde{R}(x))\) into GRPO as the per-rollout reward, using the same rubric \(\widetilde{R}(x)\) cached per prompt across the rollout group. The base policy is Qwen2.5-3B-Instruct; we train it on \(2000\) open-ended prompts drawn from WildChat [23].

3.3.0.2 Main results.

Table 4 reports the post-RL policy performance under different reward sources. Using MRRG as the reward consistently yields the best results on both in-domain and domain-transfer evaluations. On BiGGen Bench, MRRG achieves a score of \(63.7\), outperforming the strongest baseline, SVRG w/ SR, by \(1.7\) points. The improvement is even larger on the domain-transfer benchmark HealthBench-Hard, where MRRG reaches \(32.1\) and exceeds the second-best method by \(3.4\) points. Notably, all rubric-based rewards improve over the base policy, while MRRG provides the largest gains across both benchmarks. These results suggest that multi-role rubric rewards provide a more comprehensive and robust training signal than single-voiced alternatives, leading to better generalization under distribution shift. Detailed experimental results are provided in Appendix 8.4.

4 Related Work↩︎

4.1 Reward Modeling and LLM-as-a-Judge↩︎

Scalar reward models. Aligning LLMs with human intent typically relies on a learned reward model trained on Bradley–Terry-style pairwise preferences. InstructGPT [4] and HH-RLHF [5] train a Bradley–Terry head on top of a base language model (LM) using human-labeled pairwise preferences, and use the resulting scalar as the reward for RLHF. PPE [22] subsequently shows that such reward models are brittle proxies of downstream policy quality and proposes a preference-proxy benchmark to measure that gap. LLM-as-a-judge. Other methods like MT-Bench / Chatbot Arena [8] prompts a strong LLM (GPT-4) to directly score or pairwise-compare two responses, and AlpacaFarm [26] uses this LLM-judge signal as a stand-in for human feedback when simulating RLHF.

4.2 Rubric-Based Evaluation and RLVR↩︎

A recent line of work makes evaluation criteria explicit through rubrics or checklists. These methods typically generate rubrics using one or a few LLM calls under a single “universal evaluator” prompt, and the resulting criteria can further be used as verifiable rewards for RLVR. For example, Chasing the Tail [17] emphasizes that rubric generation should account for distinctions among high-quality and diverse responses, and proposes a workflow for constructing rubrics that capture such distinctions. Rubrics as Rewards (RaR) [18] generates question-specific rubrics from the input question and a reference answer, and uses them as rewards for RLVR. OpenRubrics  [19] collects large-scale and diverse question–rubric pairs to train rubric-generation models capable of producing reliable and scalable rubrics.

While these approaches demonstrate the effectiveness of rubric-based evaluation and reward modeling, they often depend on additional supervision, such as reference answers for rubric construction or preference-labeled data for training rubric generators. This reliance can limit their applicability in realistic settings where reference answers or preference annotations are unavailable or expensive to obtain.

4.3 Multi-Role Prompting for Evaluation and Reasoning↩︎

Prior work has shown that role conditioning can substantially affect the knowledge and reasoning patterns elicited from LLMs. In-Context Impersonation [27] shows that asking a LLM to act as a domain expert / a child systematically changes the knowledge it surfaces on QA and bandit tasks; Role-Play Prompting extends role-play to chain-of-thought reasoning and reports gains across MMLU and GSM8K. Multi-Agent Debate [28] runs several LLM instances that iteratively critique and revise each other’s answers, improving factuality on TriviaQA-style tasks; Encouraging Divergent Thinking (EDT) [29] shows that explicitly encouraging divergent viewpoints across agents further reduces mode collapse in reasoning. These methods primarily use multiple roles to produce better answers. In contrast, MRRG generates the evaluation criteria themselves. Moreover, unlike debate-based methods, MRRG does not require iterative interaction, critique, or consensus among agents. Instead, role-specific rubrics are generated independently and then combined through deterministic deduplication.

5 Conclusion↩︎

We presented Multi-Role Rubric Generation (MRRG), a training-free framework that recasts rubric construction as a multi-role elicitation process and pool per-role rubrics into a powerful scorer. The same scorer serves uniformly as an LLM-as-a-judge for preference validation and as a reward inside RLVR, with no architectural change between the two uses. Empirically, MRRG consistently outperforms single-voice rubric generators, and rubric-refinement baselines on JudgeBench, RewardBench-2, and PPE, and it also delivers stronger downstream policies under GRPO on BiGGen-Bench and HealthBench-Hard.

Limitations↩︎

While MRRG mitigates the dimensional blind spots of single-voiced rubric generation, it still has one notable limitation. The pool of roles used to instantiate the multiple voices is predefined and remains fixed across all questions. In practice, however, the set of perspectives that are truly relevant for judging a response is inherently question-dependent: a coding task and a moral-reasoning task, for instance, call for largely disjoint expert viewpoints. Using a static role pool may therefore introduce uninformative voices on some questions while omitting critical ones on others. A natural extension is to incorporate adaptive, question-conditioned role selection — for example, by retrieving or generating roles on the fly based on the input — which is an important direction for future work.

6 Dataset description↩︎

• JudgeBench [20]: Using a novel pipeline that transforms any dataset with ground truth labels and verification algorithms into a corresponding dataset specifically tailored for LLM-based judges, It constructs a challenging dataset consisting of 350 response pairs across four categories:general knowledge, reasoning, mathematics, and coding. Each pair contains one objectively correct response and one objectively incorrect response, with the incorrect response designed to contain subtle errors, making it difficult for LLM-based judges to distinguish between the two.

• RewardBench-2 [21]: The benchmark contains 1865 preference validation data containing the following categories: factuality, precise instruction following, math, safety, focus and ties. Each instance contains at least one correct answer and three wrong answers. For each sample, the judgment is identified as correct only if all the chosen answers obtain higher score than all the rejected answers.

• PPE: [22] Preference Proxy Evaluations is an evaluation benchmark for reward models and LLM-judges, which are used to replicate human preferences for large-scale LLM training and evaluation. PPE contains more than 16000 real human preference data sourced from Chatbot Arena as well as verifiable correctness preference data sourced from reputable LLM benchmarks, such as MMLU-Pro and MATH. PPE provides various metrics across the benchmarks, and includes analysis on which metrics correlated the most with post-RLHF outcomes. PPE is correlated to downstream outcomes to ensure using PPE as a reference evaluation signal when tuning reward models and LLM Judges leads to tangible downstream improvements in post-RLHF LLM performance.

• WildChat: [23] WildChat is a collection of 650K conversations between human users and ChatGPT. The authors collected WildChat by offering online users free access to OpenAI’s GPT-3.5 and GPT-4. The dataset contains a broad spectrum of user-chatbot interactions that are not previously covered by other instruction fine-tuning datasets: for example, interactions include ambiguous user requests, code-switching, topic-switching, political discussions, etc. WildChat can serve both as a dataset for instructional fine-tuning and as a valuable resource for studying user behaviors.

• BiGGen Bench: [24] BIGGEN BENCH evaluates 9 core capabilities of LMs–namely instruction following, grounding, planning, reasoning, refinement, safety, theory of mind, tool usage, and multilingualism–across 77 tasks and 765 instances.

• HealthBench-Hard [25]: A subset of 1,000 HealthBench examples chosen because they are difficult for current frontier models. HealthBench Hard is designed to be a difficult target for future models, with many current models achieving a score of zero on it.

7 Implementation Details↩︎

All experiments were conducted on a system equipped with approximately 500 GB of GPU memory, with GRPO training taking approximately 7 hours. Detailed hyperparameters for GRPO are provided in Table 5.

8 Detailed experimental results↩︎

8.1 Experimental judging results in different domains↩︎

In Figure 1 (a), we report the experimental results of Qwen2.5-7b-Instruct on each domain from JudgeBench. In this subsection, we report the performance of each method on each domain from RewardBench-2 and JudgeBench in Figure 5 and Figure 6. From the figures, we can observe domain-dependent judging accuracy in single-voiced rubric and other single-role rubric caused dimensional blind spots.

8.2 Experimental Setup for Rubric-Hacking Experiments↩︎

Figure 1 (b) reports our rubric-hacking experiments. The experimental procedure is as follows. We first sample \(400\) query prompts and use Qwen2.5-3B-Instruct to generate both single-voiced and multi-voiced rubrics for each query. Given each generated rubric, we then prompt GPT-4o to produce an adversarial answer that attempts to satisfy the rubric while remaining low-quality along evaluative dimensions not covered by the rubric. The prompt template used for this adversarial answer generation is shown in Figure 7. Finally, GPT-4o is asked to evaluate the generated answers on a \(0\)–\(10\) scale using the prompt template in Figure 8.

The motivation behind this experiment is that a more comprehensive rubric should be harder to hack. If a rubric covers a broader set of evaluative dimensions, it becomes more difficult for an adversary to generate an answer that both satisfies the rubric and remains low-quality by exploiting dimensional blind spots. Conversely, if such a low-quality yet rubric-passing answer can be easily generated, this suggests that the rubric fails to capture important aspects of answer quality. Therefore, the quality of these adversarial rubric-passing answers serves as an indirect measure of rubric comprehensiveness and robustness: higher evaluated answer quality indicates that the rubric leaves fewer exploitable blind spots, whereas lower quality indicates greater vulnerability to rubric hacking.

8.3 Ablation on LLM-Based Rubric Consolidation↩︎

In this subsection, we empirically examine whether an additional LLM-based consolidation step benefits MRRG. Specifically, after generating rubrics with the multi-role process, we prompt an LLM to consolidate them by merging semantically overlapping criteria and removing mutually conflicting ones using prompt in Figure 14. This ablation allows us to assess whether semantic rubric refinement improves upon our default exact-match deduplication strategy.

6pt

As shown in Table 6, LLM-based consolidation does not improve MRRG. Instead, MRRG consistently outperforms MRRGC across all backbone models and both benchmarks. On RewardBench-2, MRRG exceeds MRRGC by \(5.0\) to \(15.0\) points, while on JudgeBench the gains range from \(8.0\) to \(14.6\) points. The degradation is particularly pronounced for the GPT-OSS models, where consolidation leads to more than \(12\)-point drops on both benchmarks.

These results suggest that semantic consolidation is not a free improvement for rubric generation. Although consolidation can reduce surface-level redundancy, it may also remove useful role-specific criteria or suppress legitimate differences between evaluative perspectives. Moreover, the LLM used for consolidation must itself decide which criteria are redundant or conflicting, introducing an additional source of subjective bias. We therefore adopt exact-match deduplication as the default aggregation strategy, as it preserves complementary multi-role criteria while remaining deterministic and reproducible.

8.4 Detailed experimental results on BigGen Bench and Health-Bench-Hard↩︎

In this subsection, we provide detailed post-RL policy performance on BiGGen Bench and HealthBench-Hard under different reward sources in Table 7 and 8 respectively.

8.5 More case studies↩︎

Table 9 illustrates a case where the single-voiced rubric fails to separate a well-grounded answer from a superficially plausible one. The query “What is huli?” is inherently ambiguous, since the term may refer to a Hawaiian verb, the food-related expression “huli huli”, or other language-specific meanings. The rejected answer appears fluent and informative, but it adds weakly supported associations such as mischief, idiomatic usage, and unrelated food references. Because the single-voiced rubric mainly checks whether the response defines the term, provides context, and includes examples, it assigns the rejected answer the same score as the chosen answer. By contrast, the multi-voiced rubric includes more discriminative criteria concerning ambiguity resolution, calibrated uncertainty, precise wording, and avoidance of unsupported or tangential details. These criteria allow MRRG to prefer the chosen answer, which presents the main plausible meanings, explains the relevant usage of “huli huli”, and appropriately asks for additional context. This example demonstrates that MRRG better captures factual reliability under ambiguity, where the key challenge is not producing more information, but selecting information that is grounded, relevant, and properly scoped.

9 Prompt template for rubric generation and evaluation↩︎

In this section, we provide all the prompting template in our experiments in the following figures:

• Rubric passing answer generation prompt template used in rubric-hacking experiments are shown in Figure 7.

• Answer evaluation prompt template used in rubric-hacking experiments are shown in Figure 8.

• Rubric generation prompt template for \(\{\mathrm{\small AI Researcher}, \mathrm{\small User},\; \mathrm{\small Domain Expert},\\\ \mathrm{\small Educator},\;\mathrm{\small Linguist}\}\) are shown in Figure 13, 9, 10, 11 and 12 respectively.

• Rubric consolidation prompt template used in LLM consolidation experiments is shown in Figure 14.

• Rubric generation prompt template for single voiced methods are shown in Figure 15 and 16 respectively.

• Criterion judging prompt is shown in Figure 17.

5pt

References↩︎