1 Introduction↩︎

LLM-as-a-Judge has emerged as a promising paradigm for evaluating LLM outputs, mitigating limitations of traditional similarity-based metrics such as BLEU, ROUGE, and BERTScore [1]–[3]. This approach has demonstrated success across diverse applications, including evaluation of natural language generation [4], validation of synthetic data [5], and self-rewarding for LLM alignment [6]. However, LLM judges exhibit biases toward irrelevant heuristics, such as answer length [7] and the order of candidate responses [8], among others.

To address these biases, previous work has explored training specialized LLM evaluators [9]–[11]. However, such methods require substantial computational costs and high-quality training data. An alternative line of research focuses on training-free methods that leverage evaluation rubrics—structured sets of criteria that specify what dimensions to assess and how to score them—to guide LLM judges toward more consistent and interpretable evaluation. Within this direction, several studies employ fine-grained, decomposition-based rubrics. [12] develop a meta-evaluation benchmark with instance-specific rubrics, and several works decompose holistic evaluation into checklists or weighted sub-criteria [13]–[16]. Other efforts curate reusable evaluation metrics across tasks, such as AutoMetric [17]. Despite these efforts, existing approaches typically rely on human-annotated data such as expert-crafted rubric examples and reference answers, limiting their applicability to critical domains such as finance, health, and security, where expert annotations are scarce or unavailable.

This work focuses on improving LLM-as-a-Judge systems from the evaluation rubric perspective. We investigate automatically generating fine-grained rubrics without requiring any human-annotated data (e.g., reference answers, rubric examples, or annotated training data). Specifically, we propose a two-stage approach. First, we introduce a training-free method that directly generates evaluation rubrics at multiple granularities—from dataset-specific to instance-specific—without requiring human-annotated examples or reference answers. Second, we investigate fine-tuning a rubric generator model via preference learning, leveraging reward signals derived from a meta-judge’s evaluation on the generated rubrics. Experiments across four LLMs and four benchmarks demonstrate that our training-free approach achieves competitive performance with several baselines and human-crafted rubrics, while rubrics from our fine-tuned generator further improve the LLM judge, outperforming all existing methods.

Our contributions are as follows:¹

2 Related Work↩︎

2.1 LLM-based Automatic Evaluation↩︎

With advances in LLM fine-tuning methods, recent work has explored various learning-based approaches to improve LLM judge models. Prometheus 2 [10] is an open-source LLM evaluator trained on a human-crafted dataset that includes predefined rubrics and reference answers [9]. GLIDER [11] is another open-source LLM evaluator trained on synthetic data that employs span highlighting to provide explainable, fine-grained evaluations. Hercule [18] is a multilingual LLM evaluator fine-tuned on multilingual data obtained via machine translation. JudgeLRM [19] focuses on improving LLM judges’ reasoning ability by adopting RL-based preference training with a specialized reward function. MetaRewarding [20] introduces a self-improvement framework that leverages LLMs to judge their own judgments and uses that feedback to refine judging ability.

In this work, we demonstrate that a training-free approach yields competitive evaluation results against state-of-the-art LLMs and LLM judges.

2.2 Fine-grained and Dynamic Rubrics for Automatic Evaluation↩︎

Another line of work focuses on training-free approaches to improve LLM judges’ evaluation quality, specifically by improving the quality of evaluation rubrics. FLASK [21] proposes a fine-grained evaluation framework that decomposes response quality into skill-specific dimensions aligned with human judgment, though it relies on a fixed, manually curated skill taxonomy. BiGGen Bench [12] presents a meta-evaluation benchmark with instance-specific rubrics and demonstrates that they are beneficial for LLM judges. RubricHub [16] introduces an automated rubric generation framework grounded in reference answers and predefined principles. AutoMetric [17] proposes a task-specific evaluation framework that generates evaluation metrics with rubrics, retrieves from a curated MetricBank, and optimizes their combination via regression to maximize correlation with human judgment; however, this approach relies on ground truth for both generation and optimization. [15] present an approach that decomposes evaluation into multiple criteria and aggregates scores with dynamic weights; however, their method is limited to pairwise evaluation. CheckEval [14] adopts a checklist-based evaluation method by decomposing high-level rubrics into discrete boolean questions, but relies on human knowledge to select evaluation dimensions and sub-dimensions (e.g., dimension “fluency” with sub-dimensions “grammar” and “readability”). [13] further investigate checklist-based evaluation with different checklist generation policies and observe that such methods tend to work well only in pairwise settings while showing fewer benefits for pointwise evaluation.

These approaches typically rely on human-annotated data such as reference answers or predefined evaluation dimensions. Our method supports both pointwise and pairwise settings without any such data, making it applicable to domains where these resources are unavailable.

2.3 Rubric-based Reward Modeling↩︎

Beyond improving LLM-as-a-Judge systems, recent work has investigated leveraging rubric-based reward signals for RL fine-tuning. Rubrics as Rewards [24] investigates an on-policy reinforcement learning method that leverages generated rubrics as reward signals. [25] address reward hacking by dynamically generating task-specific rubrics rather than relying on static rubrics for reward modeling. AutoRule [26] automatically extracts rules from human preference data to construct rule-based rewards for improving instruction-following ability. However, all of these approaches rely on gold reference answers as a supervised proxy for generating or extracting rubrics. Lastly, checklist-based rubrics have also been adopted for preference tuning. [27] propose replacing traditional reward models with dynamic, instruction-specific checklists evaluated by LLM judges. We are the first to investigate fine-tuning a rubric generator model via reward signals from a meta-judge.

3 Generating Dynamic Rubrics↩︎

We formalize evaluating LLM-as-a-Judge systems with rubrics and describe our approach to generating fine-grained evaluation rubrics without human preference data. We first define the two evaluation settings (Section 3.1) and evaluation benchmarks (Section 3.2), followed by our training-free rubric generation approach (Section 3.3) and the preference learning approach (Section 3.4).

3.1 Task Formulation↩︎

We consider two evaluation settings commonly adopted in the LLM-as-a-Judge paradigm: pairwise and pointwise evaluation.

3.1.0.1 Pairwise Evaluation

Given an instance input \(x\) and two candidate responses \(y_1, y_2\), the judge model \(\mathcal{J}\) decides \(\mathcal{J}(x, y_1, y_2) \in \{1, 2\}\), indicating which response is preferred. We evaluate pairwise judges by measuring human agreement: the percentage of instances where the judge’s preference matches the majority vote of human annotators.

3.1.0.2 Pointwise Evaluation

Given an instance input \(x\) and a single response \(y\), the judge model \(\mathcal{J}\) produces a scalar score \(\mathcal{J}(x, y) \in [a, b]\), where \([a,b]\) denotes the scoring range (e.g., an integer in \([1, 5]\) for holistic scoring, or a continuous value in \([0, 1]\) for criterion-level aggregation). We measure alignment with the ground truth (i.e., human scores) using Spearman and Pearson correlation coefficients computed across all (instance, response) pairs in the dataset. Note that since these correlation metrics are scale-invariant, the judge and human scores need not share the same range.

3.1.0.3 Rubric Granularity

We distinguish two levels of rubric specificity used in this work. A dataset-specific rubric defines a fixed evaluation instruction applied uniformly to all instances within a benchmark or domain, capturing general quality dimensions shared across instances. An instance-specific rubric is generated for each individual instance, providing criteria tailored to the particular instance.

3.2 Benchmarks↩︎

We evaluate on four meta-evaluation benchmarks spanning pairwise and pointwise settings (Table 1).

3.2.0.1 Pointwise Benchmarks

HelpSteer2 [22] contains 1,038 instances scored on five dimensions (helpfulness, correctness, coherence, complexity, verbosity) as integers in \([0,4]\). We use the average of helpfulness, correctness, and coherence as the ground-truth score, as complexity and verbosity lack a monotonic relationship with quality (e.g., higher verbosity is not inherently better or worse). It provides no reference answers or instance-specific rubrics. BiGGen Bench [12] provides human scores (\([1,5]\)) for 695 prompts across 4 models (2,776 instances), along with per-instance scoring rubrics defining what each score level means. It provides both reference answers and instance-specific rubrics.

3.2.0.2 Pairwise Benchmarks

Length-Controlled AlpacaEval [7] contains 321 response pairs with human preference annotations collected from 4 evaluators each; we use the majority vote as the ground-truth label. It provides neither reference answers nor instance-specific rubrics. MT-Bench [23] consists of multi-turn dialogues with 3,355 human preference annotations grouped into 1,204 unique pairs. We take the majority vote and exclude ties, yielding 941 evaluation pairs. We additionally repurpose BiGGen Bench for pairwise evaluation by comparing all model pairs per prompt and using the higher human score as the preferred response, excluding ties. This yields 2,792 pairwise evaluation pairs.

3.3 Training-free Rubrics Generation in Multiple Granularities↩︎

We propose a training-free approach to generate evaluation rubrics at two granularity levels without requiring human-annotated data. We denote the model for generating rubrics as \(\mathcal{G}\); in this setting, \(\mathcal{G} = \mathcal{J}\) (i.e., the judge model itself generates the rubrics), while in Section 3.4, we fine-tune a dedicated \(\mathcal{G}\) separate from the judge.

3.3.0.1 Dataset-specific Rubric

We prompt the judge model to generate a dataset-specific rubric, which is applicable to any instance within a benchmark. The rubric generation prompt contains no instance-specific information; instead, it asks the model to identify universal dimensions that distinguish good responses from poor ones (e.g., instruction adherence, factual accuracy, completeness, clarity). This is a cost-efficient setting as the generated rubric is cached and reused for all evaluation instances.

3.3.0.2 Instance-specific Rubric

For each instance \(x\), we prompt the judge model to generate an instance-specific rubric \(\mathcal{R}(x) = \{c_1, \ldots, c_L\}\) consisting of \(L\) evaluation criteria tailored to the particular instance. The rubric generation is conditioned only on the instance’s input (e.g., the prompt), without seeing any reference responses. This rubric is then injected into the judge prompt, guiding the model to evaluate responses along the specified dimensions. The evaluation on response \(y\) is conditioned on the instance and generated rubric: \[\mathcal{J}(\mathcal{R}(x), x, y).\]

Unlike prior rubric generation methods [16], [17], our approach requires neither reference answers nor human-crafted rubrics as few-shot examples, making it applicable to domains where such resources are unavailable. Appendix 6 and 7 provide full prompt templates and examples of generated rubrics.

3.4 Annotation-free Preference Learning for Rubric Generator↩︎

While the training-free approach produces effective rubrics, its quality is bounded by the generator’s parametric knowledge. Empirically, larger models generate better rubrics but at higher cost. To close this gap, we iteratively fine-tune a small open-source rubric generator via preference learning, still without requiring any human-annotated rubrics or reference answers. Inspired by Meta-Rewarding [20], we leverage reward signals from a meta-judge to improve rubric generators. The training pipeline proceeds in five steps: (1) generate diverse rubric candidates, (2) score them using a meta-judge, (3) construct preference pairs, (4) fine-tune the generator with Direct Preference Optimization (DPO) [28], and (5) iterate by using the fine-tuned model as the new base generator.

We use instances from the evaluation benchmarks themselves (Section 3.2) as training instances. No human annotations (preference labels or scores) are leaked; the only supervision comes from the meta-judge comparing rubric candidates on intrinsic quality dimensions such as correctness, specificity, and coverage. This is functionally equivalent to best-of-\(K\) selection with a meta-judge at inference time, but amortizes the cost by distilling these preferences into the generator offline. Since the reward signal is rubric-intrinsic and never accesses downstream human judgments, the generator cannot learn to exploit evaluation outcomes.

Step 1: Rubric Candidate Generation For each instance \(x\), we generate \(K\) diverse rubric candidates \(\{\mathcal{R}_1(x), \ldots, \mathcal{R}_K(x)\}\) from the base generator \(\mathcal{G}\) using high temperature (\(\tau{=}0.8\)). This diversity is essential for constructing informative preference pairs. We use \(K{=}8\) candidates per instance.

Step 2: Meta-Judge Evaluation We employ a strong meta-judge model \(\mathcal{M}\) (e.g., Claude Sonnet 4) to compare rubric candidates pairwise. For each pair \((\mathcal{R}_i, \mathcal{R}_j)\), the meta-judge determines which rubric is superior for evaluation, considering:

To mitigate positional bias, we evaluate each pair in both orders and count a win only when both orders agree. From \(K{=}8\) candidates (\(\binom{8}{2}=28\) possible pairs), we randomly sample 10 pairs per instance to limit computational cost while ensuring each candidate participates in multiple comparisons.

Step 3: Preference Pair Construction We fit a Bradley-Terry model [29] to the meta-judge outcomes, estimating a strength parameter \(\gamma_i\) for each candidate by maximizing the likelihood of the observed pairwise comparisons (details in Appendix 8). For each instance, we select the candidate with the highest \(\gamma\) as the chosen rubric \(\mathcal{R}_w\) and the lowest as the rejected \(\mathcal{R}_l\), forming a preference pair \((x,\, \mathcal{R}_w,\, \mathcal{R}_l)\).

Step 4: DPO Fine-tuning We fine-tune the rubric generator \(\mathcal{G}_\theta\) using DPO, which directly optimizes the policy to prefer \(\mathcal{R}_w\) over \(\mathcal{R}_l\): \[\mathcal{L}_{\text{DPO}} = -\mathbb{E}\!\left[\log\sigma\!\left(\beta\, r_w - \beta\, r_l\right)\right],\] where \(r_i = \log \frac{\pi_\theta(\mathcal{R}_i \mid x)}{\pi_{\text{ref}}(\mathcal{R}_i \mid x)}\) is the log-ratio between the trained policy \(\pi_\theta\) and the frozen reference policy \(\pi_{\text{ref}}\), and \(\beta\) controls the KL constraint strength.

Step 5: Iterative Training The fine-tuned generator from iteration \(t\) serves as both the base model and the reference policy for iteration \(t{+}1\): \[\mathcal{G}^{(t+1)} = \text{DPO}\!\left(\mathcal{G}^{(t)},\; \mathcal{D}^{(t)}\right),\] where \(\mathcal{D}^{(t)}\) denotes the preference pairs constructed from rubric candidates generated by \(\mathcal{G}^{(t)}\) and scored by the meta-judge. Steps 1–4 are repeated with the updated generator, allowing it to progressively improve beyond the capability of the base model. Full implementation details including the hyperparameters are provided in Appendix 8.

4 Experiments↩︎

We evaluate our approach on four benchmarks (Section 3.2) across four models. We first compare our training-free rubric generation against existing baselines, then evaluate the fine-tuned rubric generator (Section 4.3).

4.0.0.1 Baselines

We compare against Prometheus 2 [10], an open-source LLM fine-tuned specifically for rubric-based evaluation, and three training-free rubric-based methods: DnA-Eval [15] decomposes evaluation into 3 weighted aspect-specific questions and aggregates scores per aspect; CheckEval [14] generates a fixed checklist of 20 binary questions per benchmark and scores responses by the proportion of affirmative answers; RubricHub [16] generates fine-grained binary criteria grounded in a reference response and aggregates via weighted sum. All three methods support pointwise evaluation by producing a numeric score per response. For pairwise benchmarks, we score both responses independently and select the higher-scoring one; ties (equal scores) are resolved by random assignment, equivalent to 50% accuracy on tied pairs.

Additionally, we report results with human-crafted rubrics provided by the benchmark authors. We evaluate at both dataset-specific (all benchmarks) and instance-specific granularity (BiGGen only, Section 3.2). We consider the latter as an oracle upper bound since such fine-grained rubrics are infeasible to obtain at scale in real-world settings. Appendix 9 reports further details of the baseline methods.

4pt

4.1 Experimental Setting↩︎

4.1.0.1 Judge Models

We use four judge models ranging from open-source to proprietary: Llama 3.1 8B and 70B, Qwen3 14B, and Claude Sonnet 4. We report main results with the latter three; extra results with Llama 3.1 8B are in Appendix 10. For pairwise evaluation, we mitigate positional bias [30] by evaluating each pair in both orders and counting disagreements as incorrect.

4.1.0.2 Rubric Generator Model

As discussed in Section 3.3, for training-free rubric generation (Section 4.2), the judge model itself serves as the rubric generator (\(\mathcal{G} = \mathcal{J}\)). For preference learning (Section 4.3), we fine-tune Qwen3 14B as a dedicated rubric generator separate from the judge. Claude Sonnet 4 serves as the meta-judge \(\mathcal{M}\) for collecting rubric preferences. An ablation on different generator models and meta-judges is reported in Appendix 12.

5pt

4.2 Results with Baselines and Training-Free Rubric Generation↩︎

Table 2 presents results comparing our training-free rubric generation against baselines and human-crafted rubrics, reporting human agreement (%) for pairwise benchmarks and Spearman/Pearson correlation for pointwise benchmarks.

4.2.0.1 Existing Baselines

Among the three existing methods, DnA-Eval consistently achieves the strongest results, outperforming CheckEval and RubricHub on most benchmarks—the only exception is MT-Bench. Prometheus 2 (8\(\times\)​7B) underperforms the best training-free methods despite being explicitly trained for rubric-based evaluation, suggesting that general LLMs with high-quality rubrics can surpass specialized LLM judges.

Despite being the strongest baseline, DnA-Eval still falls short of human-crafted dataset-specific rubrics on most benchmarks (e.g., 69.78% vs.% on AlpacaEval, 78.80% vs.% on MT-Bench, both with Qwen3 14B). We attribute this gap to annotation alignment: human-crafted rubrics mirror the exact evaluation protocol used by annotators (e.g., HelpSteer2’s rubric uses helpfulness, correctness, and coherence, the same dimensions averaged to compute the ground-truth score), whereas existing methods generate ad-hoc aspects that may not align with the annotation criteria.

Human-crafted instance-specific rubrics (available only for BiGGen) further outperform all automated methods by a large margin (78.33% vs.%, 0.584 vs. with Claude Sonnet 4), highlighting the value of fine-grained, expert-crafted rubrics. However, such rubrics require domain expertise that is expensive and often infeasible to collect at scale.

4.2.0.2 Our Approach

Our training-free rubric generation achieves competitive and even superior results to the existing methods without any training data, human-crafted rubrics, or reference answers. Generated instance-specific rubrics consistently outperform dataset-specific across all judge models, again confirming that fine-grained rubrics work better even without reference answers. Notably, our approach achieves state-of-the-art results on AlpacaEval and HelpSteer2 with all models.

The majority of our results are not statistically significantly worse than the best existing method (reported in Table 2 without an asterisk (\(^\ast\)); bootstrap test, \(p > 0.05\)). The only significant gaps appear on BiGGen with human-crafted instance-specific rubrics, which is considered an oracle upper bound. Appendix 11 reports more details including the significance testing.

4.3 Results with Fine-tuned Rubric Generator↩︎

We apply the preference fine-tuned rubric generator exclusively for instance-specific rubric generation. Dataset-specific rubrics, in principle, would benefit less from preference learning since their quality is already high with the base models. We train separate generators for pairwise and pointwise evaluation, as the two settings demand different rubric properties: pairwise rubrics must highlight dimensions that differentiate between two responses, while pointwise rubrics must define criteria for absolute scoring. Table [table:preference95learning] presents results using rubrics from the fine-tuned Qwen3 14B generator over two iterations.

The fine-tuned generator consistently improves over training-free rubric generation across all benchmarks. Many of these improvements are statistically significant (with an asterisk \(^\ast\), e.g., 83.69% vs.% on MT-Bench and 76.96% vs.% on BiGGen with Claude Sonnet 4). Importantly, the rubrics from the fine-tuned generator outperform most existing methods on every benchmark. Again, the only exception is the remaining gap to the oracle on BiGGen, though it narrows with stronger judges (4.23 to 1.37 percentage points from Llama 3.1 70B to Claude Sonnet 4).

Notably, the fine-tuned Qwen3 14B generator outperforms Claude Sonnet 4 at rubric generation despite being a much smaller model: rubrics from Qwen3 14B achieve 83.69% on MT-Bench and 76.96% on BiGGen (with Claude as judge), compared to 81.62% and 74.89% when Claude generates its own rubrics (Table 2). Moreover, the fine-tuned Qwen3 14B paired with Qwen3 14B as judge (72.34% AlpacaEval, 82.87% MT-Bench) outperforms Claude Sonnet 4 as both generator and judge (71.65%, 81.62%), demonstrating that overall evaluation quality can be decoupled from judge capability. This suggests that under limited computational budgets, investing in a specialized rubric generator can yield greater returns than scaling up the judge model alone.

Furthermore, the iterative training procedure (Section 3.4) yields gains across both iterations. Compared to the base Qwen3 14B generator, the first iteration improves pairwise agreement by +1.15 percentage points on average, and the second iteration further increases this to +1.53 (Table 3). We observe a similar trend on pointwise evaluation (Spearman improved by +0.017 after the first iteration, +0.030 after the second). However, the marginal gain diminishes from Iteration 1 to Iteration 2 (+1.15 vs.+0.38 for pairwise; +0.017 vs.+0.013 for Spearman), suggesting potential performance convergence. We do not explore further iterations due to computational constraints.

Finally, we conduct several ablation studies, including fine-tuning a rubric generator jointly on both pairwise and pointwise instances, using a different model as the meta-judge for preference collection, and fine-tuning a smaller rubric generator model (Llama 3.1 8B). Due to space limitations, these ablations are reported in Appendix 12.

4.3.0.1 Qualitative Analysis

We compare rubrics produced by Qwen3 14B as the base, Iteration 1, and Iteration 2 models (Table 3). Rubric length and criteria count remain stable across iterations, indicating that improvement is qualitative rather than structural. The key shift is in rubric framing: the base model uses prescriptive criteria (“The response clearly explains …”) while Iteration 2 reframes around evaluation dimensions (“Accuracy of …”, “Clarity and coherence in …”). Prescriptive criteria drop from 47.2% to 24.9% for pairwise rubrics, with an even sharper decline for pointwise (38.6%\(\to\)​0.9%). The same-instance divergence between base and Iteration 2 is only 0.498 (i.e., fewer than half of the words are shared), confirming that preference learning substantially rewrites rubrics rather than merely paraphrasing them. This reframing provides stronger discriminative signal for pairwise comparison, explaining the larger gains on pairwise benchmarks. The meta-judge statistics further support this: the best-candidate win rate increases from 0.30 (base) to 0.53 (Iteration 1), indicating that stronger rubric candidates emerge as training progresses. We provide extra analysis and three case studies to further explain the observed performance gains in Appendix 13.

5 Conclusion↩︎

We presented an approach to improve LLM-as-a-Judge through automatically generated evaluation rubrics, requiring no human-annotated data or reference answers. Our training-free rubric generation achieves competitive performance with existing methods across four benchmarks. We further proposed a preference learning method to iteratively fine-tune a rubric generator via meta-judge feedback, consistently outperforming all existing baselines. Notably, a fine-tuned Qwen3 14B generator produces higher-quality rubrics than Claude Sonnet 4, demonstrating that evaluation quality can be decoupled from judge model scale through specialized rubric generation.

Future work includes incorporating reference answers when available to improve rubric specificity, evaluating cross-domain generalization of the fine-tuned generator, and exploring alternative reward signals beyond meta-judge preferences.

Limitations↩︎

Our approach requires multiple LLM calls per evaluation instance (rubric generation followed by judgment), increasing computational cost compared to direct scoring. The preference learning procedure relies on a capable meta-judge for reward signal collection, which may limit accessibility in resource-constrained settings. We explored only two iterations due to computational constraints; the sustained improvement trend suggests further iterations may yield additional gains, but potential saturation or degradation remains unexplored. We do not investigate the trade-off between rubric generator models and LLM judge models, which would be beneficial for practical usage with limited resources. We train and evaluate on the same benchmarks; an out-of-domain evaluation (training on one benchmark and testing on another) would better test generalization, but few meta-evaluation benchmarks with reliable human judgments exist beyond the ones we use. We do not compare the fine-tuned generator against best-of-\(K\) selection at inference time (generating \(K\) rubrics and selecting the best via meta-judge), which would isolate the benefit of amortizing preferences into the generator versus paying the cost at inference. We only evaluate under pairwise and pointwise settings, omitting reference-based evaluation where the judge compares a response against a gold answer. This is partly because reference answers are scarce in many real-world datasets, but extending our rubric generation approach to the reference-based setting remains future work. Finally, our evaluation is limited to mostly general domain benchmarks. An extension to domain-specific benchmarks (e.g., medical, legal, finance) would be valuable.

Ethics Statement↩︎

We used AI-based assistance to improve the language and clarity of this paper, and as a coding aid during implementation. All research ideas, experimental design, analysis, and conclusions are entirely the work of the authors.

6 Prompt Templates↩︎

We list all prompt templates used in our system. Each box shows the system and user messages sent to the model.

Rubric Generation Prompts

Judge Prompts

Meta-Judge and Reward Signal Collection Prompts

7 Example Rubrics↩︎

We provide examples of generated rubrics at both granularity levels. Figure 1 shows a dataset-specific rubric applicable to any instance, and Figure 2 shows an instance-specific rubric tailored to a particular task.

None

None

8 Implementation Details↩︎

8.0.0.1 Model Inference

Llama 3.1 8B Instruct, Qwen3 14B, and Llama 3.1 70B Instruct are served on a locally hosted NVIDIA H100 GPU (94GB) via vLLM [31]. Claude Sonnet 4 is accessed via the AWS Bedrock API. All judge and rubric generation calls use greedy decoding (temperature 0) with a maximum of 256 output tokens for judging and 1024 tokens for rubric generation. During reward signal collection, rubric candidates are sampled with temperature 0.8 to encourage diversity.

8.0.0.2 Preference Fine-tuning

We fine-tune both Llama 3.1 8B Instruct and Qwen3 14B using QLoRA [32] with 4-bit quantization via DPO. The same hyperparameters are used for both models (Table 4).

8.0.0.3 Meta-Judge Configuration

We use Claude Sonnet 4 as the meta-judge model. For each instance, we generate \(K{=}8\) rubric candidates with temperature \(\tau{=}0.8\) and sample up to 10 pairwise comparisons. Positional bias is mitigated by evaluating each pair in both presentation orders.

8.0.0.4 Bradley-Terry Ranking

We fit a Bradley-Terry model to the meta-judge outcomes. The model assumes each candidate \(\mathcal{R}_i\) has a latent strength \(\gamma_i > 0\), with the probability of \(\mathcal{R}_i\) being preferred over \(\mathcal{R}_j\) given by: \[P(\mathcal{R}_i \succ \mathcal{R}_j) = \frac{\gamma_i}{\gamma_i + \gamma_j}.\] We estimate \(\gamma_1, \ldots, \gamma_K\) by maximizing the log-likelihood of the observed outcomes: \[\mathcal{L} = \sum_{(i,j) \in \mathcal{O}} \log \frac{\gamma_i}{\gamma_i + \gamma_j},\] where \(\mathcal{O}\) is the set of observed preferences (each entry \((i,j)\) indicates \(\mathcal{R}_i\) was preferred over \(\mathcal{R}_j\) by the meta-judge). The candidate with the highest \(\gamma\) becomes the chosen rubric and the lowest becomes the rejected, forming a preference pair for DPO training.

8.0.0.5 Compute

All fine-tuning runs use a single NVIDIA H100 GPU (94GB). Each run completes in approximately 1–2 hours for 3 epochs.

9 Baseline Method Details↩︎

9.0.0.1 Prometheus 2

Prometheus 2 [10] is a family of open-source LLMs (7B and 8\(\times\)​7B MoE) fine-tuned specifically for evaluation tasks. The models are trained on a large corpus of evaluation data encompassing both pairwise and pointwise formats, with rubric-conditioned scoring. Given an instance and a rubric, Prometheus 2 directly produces a score (pointwise) or preference judgment (pairwise) without requiring multi-step decomposition. We report results with the 8\(\times\)​7B variant using both dataset-specific and instance-specific rubrics (where available).

9.0.0.2 DnA-Eval

Decomposition and Aggregation [15] decomposes evaluation into multiple aspect-specific questions with importance weights. For each instance \(x\), the method generates \(K\) evaluation aspects \(\{a_1, \ldots, a_K\}\) and corresponding importance weights \(\{w_1, \ldots, w_K\}\) (with \(\sum_k w_k = 1\)) using the judge model itself. For pairwise evaluation, each response is scored on each aspect independently on a 1–10 scale, yielding aspect scores \(\{s_k^{(1)}\}\) and \(\{s_k^{(2)}\}\), and the final preference is determined by the weighted aggregation: \[\hat{y} = \arg\max_{i \in \{1,2\}} \sum_{k=1}^{K} w_k \cdot s_k^{(i)}.\] For pointwise evaluation, the judge score is \(\sum_k w_k \cdot s_k\). We use \(K{=}3\) aspects following the original paper.

9.0.0.3 CheckEval

CheckEval [14] decomposes evaluation into a fixed checklist of binary (Yes/No) questions. A set of \(N\) questions \(\{q_1, \ldots, q_N\}\) is generated once per benchmark (not per instance) and applied uniformly to all items. For each response, the judge answers all \(N\) questions, and the score is computed as the proportion of affirmative answers: \[s = \frac{1}{N}\sum_{n=1}^{N} \mathbb{1}[\text{answer}(q_n) = \text{Yes}].\] For pairwise comparison, both responses are scored independently and the higher-scoring response is preferred. We use \(N{=}20\) questions.

9.0.0.4 RubricHub

RubricHub [16] generates fine-grained, weighted binary criteria grounded in a reference response. The pipeline consists of four stages: (1) generate a reference response \(r\) for instance \(x\); (2) generate a set of rubric criteria \(\{(t_i, d_i, w_i)\}_{i=1}^{M}\) (each with title \(t_i\), description \(d_i\), and importance weight \(w_i \in [1, 10]\)) conditioned on both \(x\) and \(r\); (3) grade the candidate response on each criterion independently, producing binary verdicts \(b_i \in \{0, 1\}\); and (4) aggregate via weighted sum: \[s = \frac{\sum_{i=1}^{M} w_i \cdot b_i}{\sum_{i=1}^{M} w_i}.\] References and rubrics are cached per instance so that repeated instances share the same evaluation criteria.

10 Additional Results with Llama 3.1 8B as Judge↩︎

Table 5 reports baselines and our training-free rubric generation results with Llama 3.1 8B as the judge model.

11 Statistical Significance Testing↩︎

We test statistical significance between our approach and the best existing method for each benchmark–judge combination using the following procedures.

11.0.0.1 Pairwise Benchmarks

For pairwise benchmarks, each method produces a binary per-instance outcome (correct or incorrect agreement with the human majority vote). We apply a paired bootstrap test with 10,000 resamples: in each iteration, we resample instances with replacement and compute the accuracy difference between the two methods. The two-sided \(p\)-value is computed as twice the proportion of bootstrap samples where the observed difference reverses sign. We additionally report McNemar’s test with continuity correction as a complementary check on paired binary outcomes.

11.0.0.2 Pointwise Benchmarks

For pointwise benchmarks, we compare Spearman correlations via a paired bootstrap procedure. In each of 10,000 iterations, we resample (instance, score) pairs with replacement and recompute both methods’ correlations with human scores. The \(p\)-value is derived from the proportion of bootstrap samples where the correlation difference reverses sign (two-sided).

Results marked with \(^\ast\) in Table 2 indicate cases where our method is statistically significantly worse than the best existing method at \(p < 0.05\).

12 Ablation Studies↩︎

We ablate three design choices. The first two use the Llama 3.1 8B rubric generator (Iteration 1) evaluated with Claude Sonnet 4 as the judge (Table 6); the third varies the generator model size.

12.0.0.1 Joint Pairwise–Pointwise Training

In our main experiments, we train separate rubric generators for pairwise and pointwise evaluation. Here we ablate this choice by training a single model on the combined preference data. Joint training degrades pairwise performance (70.09\(\to\)​68.85 on AlpacaEval, 82.14\(\to\)​81.41 on MT-Bench) while showing marginal improvement on pointwise BiGGen (0.434\(\to\)​0.441 Spearman). This suggests that pairwise and pointwise rubrics require different properties—pairwise rubrics must highlight discriminative dimensions, while pointwise rubrics need calibrated scoring anchors—and mixing both objectives dilutes the training signal for each.

12.0.0.2 Alternative Meta-Judge

Our main experiments use Claude Sonnet 4 as the meta-judge \(\mathcal{M}\) for collecting rubric preferences. We ablate this choice by substituting Llama 3.1 70B as the meta-judge. The weaker meta-judge substantially degrades AlpacaEval performance (70.09\(\to\)​67.50) while slightly improving MT-Bench (82.14\(\to\)​82.55) and BiGGen (74.25\(\to\)​74.46). Overall, using a stronger meta-judge provides more reliable reward signals, particularly on benchmarks where subtle quality differences matter (e.g., AlpacaEval’s open-ended instructions).

12.0.0.3 Generator Model Size

We additionally ablate the rubric generator model size by fine-tuning Llama 3.1 8B (vs.Qwen3 14B in the main paper). Table 7 reports results with the fine-tuned 8B generator evaluated across three judge models. The 8B generator consistently underperforms the Qwen3 14B generator (Table [table:preference95learning]), which we attribute to the smaller model’s limited capacity for generating diverse, high-quality rubric candidates during reward signal collection.

13 Qualitative Analysis and Case Studies↩︎

13.1 Analysis of Rubric Framing Shift↩︎

The key qualitative shift across iterations is from prescriptive criteria to dimension-labeled criteria. Prescriptive criteria (e.g., “The response clearly explains...”, “The answer includes...”) describe what a good response should contain, functioning as binary checklists. Dimension-labeled criteria (e.g., “Accuracy of ...”, “Clarity and coherence in ...”) frame evaluation around abstract axes along which responses can be compared.

We quantify this shift by counting criteria that begin with abstract dimension labels versus prescriptive “The response...” patterns. For pairwise rubrics, prescriptive criteria drop from 47.2% (base) to 39.5% (Iteration 1) to 24.9% (Iteration 2), while dimension-labeled criteria rise from 13.8% to 13.1% to 26.4%. The trend is even more pronounced for pointwise rubrics: prescriptive criteria decline from 38.6% to 33.5% to 0.9%.

This reframing explains why preference fine-tuning improves pairwise evaluation more than pointwise. Dimension-labeled criteria guide the judge to compare responses along specific axes, providing clear discriminative signal for relative judgments. Prescriptive criteria are more binary—either both responses satisfy the criterion or neither does—offering less signal for distinguishing quality differences between two responses. For pointwise scoring, however, abstract dimensions without calibrated score-level anchors make consistent absolute scoring harder, explaining the smaller gains on pointwise benchmarks.

13.2 Case Studies: Rubric Evolution Across Iterations↩︎

We present representative examples showing how fine-tuning transforms rubric framing and improves judge accuracy. All pairwise case studies use Claude Sonnet 4 as judge; the base rubric is generated by Claude Sonnet 4 itself (training-free), while the Iteration 1/2 rubrics are from the fine-tuned Qwen3 14B generator. The pointwise case study uses Claude Sonnet 4 as judge with rubrics from the base Qwen3 14B generator vs.the fine-tuned Qwen3 14B (Iteration 2).

None

None

None

References↩︎

---

1. Data, code and models available at https://github.com/wang-zijie/generating_dynamic_rubric↩︎