1 Introduction↩︎

Reward modeling is a core component of LLM post-training [1]–[3]. In particular, pointwise reward modeling is attractive, as they assign each response an independent scalar score [4], enabling efficient evaluation and direct policy optimization. Yet in non-verifiable tasks such as open-ended instruction following and chat, a single holistic score is often unreliable [5]–[8]. This has motivated rubric-based reward modeling, which decomposes evaluation into explicit criteria and aggregates criterion-level judgments into a pointwise reward score.

However, rubric-based reward modeling introduces a new challenge: a reliable evaluator requires not only high-quality criteria, but also a judge that can apply them consistently. A common solution is to prompt frontier LLMs to generate rubrics and judgments [9]–[12]. While effective, this paradigm largely outsources evaluation to external models, making reward estimation expensive, difficult to control, and hard to deploy as a standalone reward model.

Recent learning-based approaches take steps toward trainable rubric evaluators, but they remain limited in three important ways. First, several methods train either the rubric generator or the judge, without jointly optimizing the two components altogether [13], [14]. Second, many approaches rely on frontier LLMs as the judge to provide supervision for reward modeling [15]–[17], preserving the dependence on external evaluators. Third, the supervision most commonly available for alignment is pairwise preference data, which does not directly specify either absolute pointwise scores or criterion-level labels. This mismatch makes it unclear how to train a rubric-conditioned pointwise reward model from preferences alone. Compounding this issue, hard Boolean aggregation over rubric criteria often collapses distinct responses into tied scores, weakening the discriminative signal needed for both reward modeling and downstream policy optimization [18].

Motivated by challenges above, we introduce Rubric-ARROW¹, an alternating training scheme for pointwise rubric reward models from pairwise preferences. Rubric-ARROW treats rubric-based evaluation as a two-module learning problem: a rubric generator proposes criteria, and a rubric-conditioned judge assigns criterion-level scores that are aggregated into a pointwise reward. Crucially, both modules are optimized with reinforcement learning using only pairwise preference data in the RL stage, avoiding reliance on frontier-LLM-labeled judge supervision. To obtain informative rewards, we replace hard Boolean aggregation with a probability-based scoring rule that yields finer-grained criterion scores and reduces ties. We then couple this scoring rule with phase-specific preference rewards and an alternating RL procedure, allowing the rubric generator and judge to iteratively improve each other. The resulting reward model is deployable as a standalone pointwise evaluator and can be used directly for downstream policy optimization with standard algorithms such as DPO [19] and GRPO [20].

In summary, our contributions are as follows:

• We present Rubric-ARROW, a pointwise rubric reward modeling framework for non-verifiable domains that introduces a probability-based scoring rule and an alternating RL procedure to jointly train a rubric generator and a rubric-conditioned judge, with the RL stage relying on pairwise preference data alone.

• We provide theoretical analysis establishing the preference consistency of our judge-side reward, variance reduction from opposite-side averaging, and phase-wise convergence guarantees for our designed alternating training pipeline.

• Extensive experiments on reward modeling benchmarks and downstream policy post-training show that Rubric-ARROWconsistently outperforms strong baselines by 3.0%.

2 Related Work↩︎

2.0.0.1 Reward modeling and LLM judges.

Reward modeling is a central component of RLHF and LLM post-training [2]. Early methods typically learn scalar reward models from pairwise preferences using Bradley–Terry-style objectives [4], [21]. More recent works shifted toward generative and judge-based evaluators, including models that produce rationales or chain-of-thought judgments [10], [22]–[26], and LLM-as-a-judge methods for open-ended evaluation [27]–[30]. Such judges can be pointwise, scoring responses independently, or pairwise, comparing responses directly. While pointwise judges are efficient and directly reusable as reward models, their scores are often too coarse to capture fine-grained preference differences in subjective tasks.

2.0.0.2 Rubric-based reward modeling.

Rubric-based reward modeling improves open-ended evaluation by replacing holistic judgments with structured criteria or checklists [9], [11], [31]. Prior work has explored rubric extraction and synthesis [13], [16], rubric-guided policy learning [11], [14], [15], [17], and rubric-conditioned judges [32], [33]. Most related are pointwise rubric reward models [18], [34], which often depend on frontier LLMs for rubric or judge supervision and use hard aggregation over criteria. Rubric-ARROW differs by jointly training the rubric generator and judge from pairwise preferences during alternating RL, while using probability-based criterion scores to obtain more discriminative pointwise rewards.

3 Preliminaries↩︎

We consider a pointwise rubric reward model for non-verifiable tasks. Given an instruction-response pair, the evaluator first generates a rubric and then queries a rubric-conditioned judge to produce criterion-level judgments, which are aggregated into a scalar score.

3.0.0.1 Rubric generator.

For an instruction \(x\), the rubric generator \(\pi_r\) with parameters \(\theta_r\) samples a rubric set \(\boldsymbol{r}=\{r_{k}\}_{k=1}^{K}\sim\pi_r(\cdot \mid x;\theta_r)\), where \(K\) is the number of rubric items and \(r_k\) denotes the \(k\)-th rubric item.

3.0.0.2 Rubric-conditioned judge.

Given an instruction \(x\), a candidate response \(y\), and a rubric set \(\boldsymbol{r}\), the judge \(\pi_j\) with parameters \(\theta_j\) produces criterion-level judgments \(\boldsymbol{a}=\{a_{k}\}_{k=1}^{K}\) with \(a_{k}\in\{0,1\}\) and corresponding rationales \(\boldsymbol{e}=\{e_{k}\}_{k=1}^{K}\), drawn as \((\boldsymbol{a},\boldsymbol{e})\sim\pi_j(\cdot \mid x, y, \boldsymbol{r};\theta_j)\). Here, \(a_k=1\) if \(y\) satisfies \(r_k\) and \(0\) otherwise.

3.0.0.3 Pointwise score.

The evaluator maps \((x,y)\) to a scalar score by aggregating the criterion-level judgments. We use the weighted average \[s(x,y) = \frac{ \sum_{k=1}^{K} w_{k} a_{k} }{ \sum_{k=1}^{K} w_{k} }, \label{eq:pointwise-score}\tag{1}\] where \(w_k>0\) is a predefined prior weight assigned to rubric item \(r_k\).

4 Method↩︎

We propose a framework that trains a pointwise rubric reward model in two stages: a brief SFT warm-up followed by alternating RL on pairwise preference data alone. The RL stage alternates between optimizing the rubric-conditioned judge and the rubric generator, using phase-specific rewards derived from pairwise preference data. The remainder of this section introduces a probability-based scoring rule that yields smoother pointwise scores, the phase-specific RL objectives, the alternating GRPO procedure, and finally how the learned evaluator drives downstream policy training.

4.1 Probability-based scoring↩︎

The pointwise score in Eq. 1 aggregates Boolean judgments \(a_k \in \{0,1\}\), which can lead to many tied scores in practice. To obtain a fine-grained signal, we follow prior LLM-based evaluators that exploit output distributions rather than only hard decisions [6], [12]. Specifically, for each rubric item \(r_k\), let \(p_k^{\mathrm{true}}\) and \(p_k^{\mathrm{false}}\) denote the judge’s probabilities of the “true” and “false” tokens at the verdict position. We define the criterion-level score as the probability margin \(d_k = p_k^{\mathrm{true}} - p_k^{\mathrm{false}}\). The probability-based pointwise score is then obtained by replacing \(a_k\) with \(d_k\) in Eq. 1 . This preserves the original aggregation structure while converting discrete decisions into continuous, more discriminative reward signals.

4.2 Phase-specific RL objectives↩︎

Unlike prior RL-based judge training methods that continuously rely on frontier LLMs to synthesize judge labels in the RL loop, our RL stage uses only widely available pairwise preference data. Let \(\mathcal{D}=\{(x_i,y_i^1,y_i^2,o_i^\star)\}_{i=1}^{N}\) be a preference dataset with \(o_i^\star\in\{1,2\}\) indicating the preferred response.

Pairwise preference data does not directly supervise a pointwise rubric reward model because the evaluator scores each response independently. In our alternating RL stage, we therefore use different rollout estimators for the judge and the generator phase, while both phases aim to make the pointwise evaluator recover the human preference ordering.

4.2.0.1 Judge-side preference-consistency reward.

Fix the rubric generator \(\pi_r\), and let \(q_i=+1\) if \(o_i^\star=1\) and \(q_i=-1\) otherwise. For each response \(y_i^b\), \(b\in\{1,2\}\), the current judge produces \(n\) stochastic evaluation rollouts, yielding scores \(\{s_{i,m}^{b}\}_{m=1}^{n}\) and rollout averages \(\bar{s}_i^{b} = \frac{1}{n}\sum_{m=1}^{n} s_{i,m}^{b}\). We compare each rollout on one side against the opposite side’s average score. For the \(m\)-th rollout of \(y_i^1\), \[R_{i,m}^{1} = \mathbb{I}\!\left[q_i\bigl(s_{i,m}^{1}-\bar{s}_i^{2}\bigr)>0\right], \label{eq:r1}\tag{2}\] and for the \(m\)-th rollout of \(y_i^2\), \[R_{i,m}^{2} = \mathbb{I}\!\left[q_i\bigl(\bar{s}_i^{1}-s_{i,m}^{2}\bigr)>0\right], \label{eq:r2}\tag{3}\] When \(o_i^\star=1\), the first reward encourages judge rollouts of \(y_i^1\) to score above \(\bar{s}_i^2\), while the second encourages judge rollouts of \(y_i^2\) to score below \(\bar{s}_i^1\); the direction reverses automatically when \(o_i^\star=2\). The corresponding judge-side pair reward is \[R_i = \frac{1}{2n}\sum_{m=1}^{n}\left(R_{i,m}^{1}+R_{i,m}^{2}\right) \in [0,1], \label{eq:group-reward}\tag{4}\] and the judge-side population objective is \[J_j(\theta_j;\theta_r) = \mathop{\mathrm{\mathbb{E}}}_{(x_i,y_i^1,y_i^2,o_i^\star)\sim \mathcal{D}}\left[R_i\right]. \label{eq:judge-objective}\tag{5}\] This phase converts pairwise preference labels into a pointwise supervision signal for the judge, even though each response is scored independently.

4.2.0.2 Rubric generator-side preference-ranking reward.

Fix the judge \(\pi_j\). For each instruction \(x_i\), the rubric generator samples \(n\) rubric rollouts \(\{\boldsymbol{r}_{i,m}\}_{m=1}^{n}\). Each sampled rubric is then used by the frozen judge to evaluate both responses under the same rubric, producing \(s_{i,m}^{1}\) and \(s_{i,m}^{2}\). We assign each sampled rubric the reward \[R_{i,m}^{(r)} = \mathbb{I}\!\left[q_i\bigl(s_{i,m}^{1}-s_{i,m}^{2}\bigr)>0\right] -\lambda \Omega(\boldsymbol{r}_{i,m}), \label{eq:generator-rollout-reward}\tag{6}\] where \(\Omega(\boldsymbol{r}_{i,m})\) penalizes unnecessarily long or semantically redundant rubrics; the exact length-aware bonus structure is given in Appendix 8.4. The corresponding generator-side population objective is \[J_r(\theta_r;\theta_j) = \mathop{\mathrm{\mathbb{E}}}_{(x_i,y_i^1,y_i^2,o_i^\star)\sim \mathcal{D}} \left[\frac{1}{n}\sum_{m=1}^{n} R_{i,m}^{(r)}\right]. \label{eq:generator-objective}\tag{7}\] The generator is rewarded for producing concise, non-redundant rubrics that enable the frozen judge to rank the preferred response higher. We optimize the judge and generator in alternating update phases until convergence.

4.3 Alternating optimization↩︎

4.3.0.1 SFT warm-up.

Before alternating RL, we initialize both modules with a brief supervised stage. We adopt the high-quality rubrics from OpenRubrics [13] as-is and generate judge labels for each (instruction, response) pair with GPT-5-mini. We then filter the warm-up data to keep only pairs whose chosen response receives a higher aggregated pointwise score than the rejected response, and use the resulting data to SFT both the rubric generator \(\pi_r\) and the rubric-conditioned judge \(\pi_j\). After this one-time warm-up, the alternating RL stage uses only pairwise preference data and no further frontier-LLM annotation.

4.3.0.2 Judge phase.

We fix \(\pi_r\) and optimize \(\pi_j\) with GRPO on \(J_j(\theta_j;\theta_r)\). For each preference pair, the GRPO group contains the \(2n\) rollout rewards \(\{R_{i,m}^{1},R_{i,m}^{2}\}_{m=1}^{n}\). Since \(\pi_r\) is frozen, the rubrics for each training instance can be pre-generated once and reused throughout the judge update, which substantially reduces wall-clock cost.

4.3.0.3 Rubric-generator phase.

We then freeze the judge \(\pi_j\) and optimize the rubric generator \(\pi_r\) with GRPO on \(J_r(\theta_r;\theta_j)\). For each preference pair, the GRPO group consists of the \(n\) sampled rubrics and their rewards \(\{R_{i,m}^{(r)}\}_{m=1}^{n}\). The reward is assigned to each rubric rollout, as a rubric defines the evaluation criteria for both responses and is scored by whether the frozen judge ranks the preferred response higher under those criteria.

4.4 Policy Training↩︎

After training, the pointwise rubric reward model serves as a standalone evaluator for policy optimization. In the offline setting, we use it to construct DPO training pairs [19] by scoring candidate responses for each instruction and selecting the highest- and lowest-scoring responses as the chosen and rejected samples, respectively. In the online setting, we use the pointwise score as the reward signal for GRPO [20].

5 Theoretical Analysis↩︎

We summarize four theoretical properties of our method and defer all proofs to Appendix 9. The first two results analyze the judge-side reward \(J_j\). The last two are phase-wise GRPO guarantees: they apply to either a judge phase or a generator phase with the other module held fixed.

5.1 Preference consistency of the judge-side reward↩︎

Fix an instruction \(x\) and two candidate responses \(y^+\) and \(y^-\) such that the human prefers \(y^+\) over \(y^-\). Let the rollout-level scalar scores satisfy \(S^+=\mu^+ + \xi^+\) and \(S^- = \mu^- + \xi^-\), and define \(\Delta:=\mu^+-\mu^-\). Let \(J_n\) denote the judge-side preference recovery probability induced by our rollout comparison rule.

Theorem 1 states that the judge-side reward is calibrated for recovering the human preference.

5.2 Variance reduction from opposite-side averaging↩︎

Theorem 2 shows that opposite-side averaging improves judge reliability by reducing comparison variance.

5.3 Optimization guarantees↩︎

For the next two results, let \(b\in\{j,r\}\) denote a phase of the alternating algorithm: \(b=j\) corresponds to judge updates with \(\theta_r\) frozen, and \(b=r\) corresponds to generator updates with \(\theta_j\) frozen. Let \(J_b\) be the corresponding population objective (\(J_j\) or \(J_r\)), and let \(L_{G,b}\) be the one-step on-policy GRPO surrogate for that phase. The effective rollout group sizes are \(G_j=2n\) for the judge phase and \(G_r=n\) for the generator phase. Here \(T\) denotes the number of GRPO updates performed in phase \(b\), \(\theta_b^t\) is the active-module parameter at phase step \(t\), and \(\theta_{-b}\) denotes the frozen parameter of the other module.

6 Experiment↩︎

6.1 Datasets and Experiment Settings↩︎

Training data. We train Rubric-ARROWon the general-domain pairwise preference data from OpenRubrics [13]. We divide the dataset into equally sized, disjoint subsets and assign one subset to each outer iteration of the alternating procedure.

Backbone and variants. Both the rubric generator and the judge are initialized from Qwen-3-8B [35]. At inference time, Rubric-ARROWfollows the two-stage pointwise rubric-judging pipeline described in Section 4. We additionally report two variants: a voting@5 ensemble, where five independent rubric-judging runs are aggregated by majority voting; and Rubric-ARROWw/o RL, a pointwise rubric reward model that uses only the SFT warm-up checkpoint without the alternating RL stage, included to isolate the contribution of RL.

Baselines. For reward-model evaluation, we compare Rubric-ARROWwith generative judges (RM-R1 [22], RRM [23], JudgeLRM [10]), prompted Qwen-3-8B rubric+judge pipelines, and trained rubric-based baselines (Rubric-RM [13], Rubric-ARROW w/o RL, RIFL [17]), with black-box APIs included for reference. For downstream policy training on Qwen2.5-7B-Instruct [36], we compare against scalar reward models (Skywork [21], ArmoRM [4]), AI feedback data (UltraFeedback [37]), and rubric/checklist baselines (RLCF [9], Rubric-RM, Rubric-ARROW w/o RL, RIFL). Full baseline descriptions are given in Appendix 8.3.

Evaluation benchmarks and metrics. We evaluate Rubric-ARROWin two settings. For reward-model evaluation, we use RewardBench (Chat/Chat-Hard) [38], FollowBench [39], PPE-IFEval [40], InfoBench [41], IFBench [42], RM-Bench [43], RewardBench2 (Precise-IF/Focus) [44], and HelpSteer3 [45]. For downstream policy evaluation, we use IFEval [40], InfoBench [41], IFBench [42], Arena-Hard [46], AlpacaEval2 [47], Creative Writing Benchmark v3 [48], and WildBench [49]. Evaluation construction and metric details are deferred to Appendix 8.2.

6.2 Performance of Rubric-ARROW↩︎

Table 1 shows that Rubric-ARROW provides a strong white-box pointwise reward model for open-ended evaluation. Among white-box methods, Rubric-ARROW achieves the best average performance, improving over the strongest rubric baseline Rubric-RM from 70.1 to 73.1, and further to 75.6 with voting@5. The gains are most pronounced on instruction-following and chat-oriented benchmarks, where Rubric-ARROW-voting@5 achieves the best results on Chat Hard, PPE-IFEval, InfoBench, and IFBench. The improvement over the no-RL variant further confirms the benefit of our alternating RL procedure, indicating that jointly optimizing the rubric generator and judge yields more discriminative reward signals. Notably, Rubric-ARROW-voting@5 also outperforms prompting-based judges, suggesting that learned rubric-conditioned evaluation is more reliable than direct prompting.

We next study robustness under best-of-\(N\) selection on 100 WildBench prompts from instruction-following and chat categories, using frontier-LLM scores under the dataset rubric as the oracle reference. As shown in Fig. 1, Rubric-RM and Skywork begin to degrade after \(N=8\), suggesting sensitivity to reward overoptimization. In contrast, Rubric-ARROW continues to closely track the oracle upper bound, indicating that its rewards remain well aligned when used for offline selection.

6.3 Ablation Study↩︎

We perform two ablation studies to better understand where the improvements of Rubric-ARROW come from. First, we remove the length penalty to examine whether controlling rubric verbosity is important for reliable rubric-conditioned judging. Second, we evaluate probability-based scoring against a random-probability control to verify whether the model-derived confidence scores provide meaningful tie-breaking signals.

6.3.0.1 Length penalty.

Removing the length penalty consistently hurts performance, reducing the average score from 73.1 to 72.3 and voting@5 from 75.6 to 74.3 (Table 2). The drop is most visible on stricter instruction-following benchmarks such as Chat Hard and PPE-IFEval, suggesting that length regularization helps prevent overly verbose or redundant rubrics and improves reward accuracy.

6.3.0.2 Probability-based scoring.

Table 3 shows that probability-based scoring breaks ties without sacrificing correctness. On InfoBench and IFBench, it resolves 98/140 and 68/115 tied cases while preserving most originally correct predictions (305/312 and 234/255). In contrast, a random-probability control resolves a similar number of ties but corrupts many correct decisions, preserving only 160/312 and 130/255. Thus, the improvement comes from meaningful judge confidence rather than arbitrary tie-breaking.

6.4 Performance of offline RL↩︎

We investigate whether the gains of Rubric-ARROW generalize to downstream offline policy learning.

6.4.0.1 Instruction-Following Evaluation.

Table 4 and Fig. 2 show that policies trained with Rubric-ARROW-derived rewards yield consistent instruction-following gains. On IFEval, DPO via Rubric-ARROW lifts the average score to 80.7, outperforming both Rubric-RM and the Rubric-ARROW variant without RL, while remaining competitive on InfoBench (83.6 with DPO, 83.7 with IterDPO). The advantage is more pronounced on IFBench (Fig. 2): Rubric-ARROW reaches 37.4 with DPO and 36.7 with IterDPO, outperforming Rubric-RM under both settings.

6.4.0.2 Human preference alignment evaluation.

Rubric-ARROW-trained rewards also transfer to preference-alignment benchmarks (Table 5, Table 6). On Arena-Hard and AlpacaEval, DPO via Rubric-ARROW reaches an average of 50.7 and IterDPO further improves it to 53.0, achieving the best results on Arena-Hard (both vanilla and style-controlled) and on AlpacaEval length-controlled. On WildBench, IterDPO via Rubric-ARROW reaches the best macro score of 55.2 (53.6 with DPO), surpassing IterDPO via Rubric-RM by 2.2% relative.

6.4.0.3 Creative writing evaluation.

On the Creative Writing Benchmark v3 (Fig. 3), policies trained with Rubric-ARROW lead all compared methods: DPO via Rubric-ARROW reaches 39.8 and IterDPO further improves the score to 40.5. Rubric-ARROW-based optimization also outperforms creative-writing baselines such as RaR, Rubric-based RL, and RuscaRL, as well as Rubric-RM-based DPO and IterDPO. This indicates that Rubric-ARROW-derived rewards generalize beyond instruction-following and preference benchmarks to open-ended generation.

6.5 Performance of online RL↩︎

We evaluate Rubric-ARROW in an online RL setting by directly optimizing Qwen2.5-7B-Instruct with GRPO using different reward models. As shown in Table 7, GRPO via Rubric-ARROW delivers the best overall average, improving from 56.1 for the base model to 65.4. It also outperforms reward baselines such as RM-R1, RIFL, and Skywork. Compared with the Rubric-ARROW variant without RL, Rubric-ARROW further improves the average score from 63.6 to 65.4. These results indicate that Rubric-ARROW provides a stronger and more reliable reward signal for online policy optimization.

6.6 Efficiency Comparison↩︎

We evaluate inference efficiency on 100 samples with vLLM (Table 8): Rubric-ARROW-8B processes them in 28.35s, an order of magnitude faster than reasoning-based reward models (RRM-7B and RM-R1-7B/14B, all \(>\)​170s) and substantially faster than Rubric-RM-8B (105.12s). Although JudgeLRM-7B is slightly faster (25.71s), it produces only a direct judgment without rubric-conditioned signals. Rubric-ARROW thus preserves interpretability at a fraction of the reasoning-based inference cost.

6.7 Case Study↩︎

We qualitatively analyze failures of a baseline reward model on a challenging example. Table 12 in Appendix 8 presents a comparison instance about the difference between birding and bird watching. RIFL produces several surface-level rubrics (e.g., conciseness, clear organization, neutral tone, and whether a direct comparison is made), and its judge marks the inaccurate Resp B as True on most of them; the response wins on aggregate even though the judge separately flags its factual inaccuracy. However, Rubric-ARROW produces a focused set of hard-rule rubrics centered on accurately addressing the question, and the judge correctly marks Resp A as True and Resp B as False on the central direct-comparison rubric, yielding the right preference.

7 Conclusion↩︎

We propose Rubric-ARROW, an alternating framework that trains a pointwise rubric reward model for non-verifiable domains. After a brief SFT warm-up, its alternating RL stage couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and alternating GRPO to jointly train the rubric generator and the rubric-conditioned judge using only pairwise preference data, with no further frontier-LLM annotation. Theoretical analysis establishes preference consistency, variance reduction from opposite-side averaging, and phase-wise convergence guarantees for alternating GRPO. Empirically, Rubric-ARROW attains the best average across reward-modeling benchmarks and yields the best downstream policies under both offline (DPO, IterDPO) and online (GRPO) training.

8 Experiment↩︎

8.1 Implementation Details↩︎

Tables 9 and 10 summarize the hyperparameters used for Rubric-ARROW and policy model training, respectively. In rubric-based scoring, we assign a weight of 3 to each Hard Rule and a weight of 1 to each Principle when computing the final score. We train the GRPO models using the ms-swift library² [54], and conduct DPO and IterDPO training with LLaMA-Factory³ [55]. In total, Rubric-ARROW is trained for three alternating RL iterations. The inference-time sampling parameters are reported in Table 11. For baseline methods, we use the sampling configurations specified in their official implementations or original papers.

8.2 Evaluation Details↩︎

8.2.0.1 Pairwise construction for FollowBench and InfoBench.

For these two benchmarks, we convert the original single-response evaluation into a pairwise comparison setting: for each prompt, two responses are sampled from a fixed source model (Qwen-3-8B or Qwen-3-14B), and constraint violations are identified using each benchmark’s official verifier.

8.2.0.2 Splits and metrics.

We follow each benchmark’s official splits and scoring rules and report accuracy, win rate, or the benchmark-specific metric.

8.3 Baselines↩︎

8.3.0.1 Reward-model evaluation baselines.

We compare Rubric-ARROWwith state-of-the-art white-box generative judge and reward models, including RM-R1 [22], RRM [23], and JudgeLRM [10]. We also include black-box references for context: Claude-3.5-Sonnet\(\footnote{\url{https://www.anthropic.com/news/claude-3-5-sonnet}}\), Gemini-2.5-Flash [56], and API-based rubric+judge pipelines (pairwise, pointwise, and direct-judge variants, with GPT-4.1-Mini\(\footnote{\url{https://openai.com/index/gpt-4-1/}}\) as the rubric generator and Gemini-2.5-Flash-Lite as the judge). To isolate the effect of learned rubric-conditioned pointwise scoring, we further compare with prompted Qwen-3-8B rubric+judge pipelines (pairwise and pointwise) and with trained rubric-based baselines Rubric-RM [13], Rubric-ARROW w/o RL, and RIFL [17].

8.3.0.2 Downstream policy training baselines.

For downstream policy training, we provide training signals for Qwen2.5-7B-Instruct [36] using different reward models. Skywork [21] and ArmoRM [4] are scalar Bradley–Terry reward models trained on pairwise human preferences; for Skywork we use the Skywork/Skywork-Reward-V2-Llama-3.1-8B checkpoint. UltraFeedback [37] provides multi-aspect AI feedback annotations used as a preference dataset. RLCF [9] is a checklist-based reward signal that converts explicit constraints into a scalar reward. We also compare against rubric-based reward models Rubric-RM, Rubric-ARROW w/o RL, and RIFL.

8.4 Length-Aware Rubric Reward↩︎

The rubric-generator reward \(R_{i,m}^{(r)}\) in Eq. 6 combines preference consistency with a length-aware bonus. Concretely, let \[\mathcal{C}_i = \{m \in \{1,\ldots,n\} : q_i(s_{i,m}^1 - s_{i,m}^2) > 0\}\] denote the set of preference-consistent rubric rollouts for pair \(i\), and let \[K_{i,m} = |\boldsymbol{r}_{i,m}|\] denote the number of rubric items in rollout \(m\). When \(\mathcal{C}_i \ne \emptyset\), we further define \[K_i^{\min} = \min_{m \in \mathcal{C}_i} K_{i,m}, \qquad \bar K_i = \frac{1}{|\mathcal{C}_i|} \sum_{m \in \mathcal{C}_i} K_{i,m}.\] The rubric-generator reward used in practice is \[R_{i,m}^{(r)} = \begin{cases} 1.1, & \text{if } m \in \mathcal{C}_i,\;K_{i,m} = K_i^{\min},\;\bar K_i \ge 5, \\ +1, & \text{if } m \in \mathcal{C}_i, \\ -1, & \text{otherwise.} \end{cases} \label{eq:generator-reward-detail}\tag{8}\] If \(\mathcal{C}_i = \emptyset\), all rubric rollouts receive reward \(-1\). This length-aware bonus rewards the shortest rubric set among those that correctly recover the human preference, while the condition \(\bar K_i \ge 5\) prevents the generator from being rewarded for producing overly short rubrics.

8.5 Full Version of Policy Model Performance on IFBench and Creative Writing Benchmark v3↩︎

Figs. 4 and 5 report the full versions of the IFBench and Creative Writing v3 policy-model comparisons abbreviated in the main text, including the two baselines that were omitted from the main-body figures for readability.

8.6 Case Study↩︎

Table 12 expands the case study from the main text, showing the prompt, the two candidate responses, and the full rubric and judge output for both RIFL and Rubric-ARROW.

9 Proofs of Theoretical Results↩︎

This appendix provides the technical assumptions and full proofs omitted from the main text.

9.1 Auxiliary setup↩︎

For the judge-side results, define \[J_n = \frac{1}{2}\mathbb{P}\bigl(S^+>\bar{S}_n^-\bigr) + \frac{1}{2}\mathbb{P}\bigl(\bar{S}_n^+>S^-\bigr), \label{eq:appendix-Jn}\tag{9}\] where \(\bar{S}_n^+\) and \(\bar{S}_n^-\) are the averages of \(n\) i.i.d. copies of \(S^+\) and \(S^-\), respectively.

For the optimization results, fix a phase \(b\in\{j,r\}\) and hold the other module \(\theta_{-b}\) constant throughout the phase. The phase-specific population objective is \[J_b(\theta_b;\theta_{-b}) = \begin{cases} J_j(\theta_j;\theta_r), & b=j,\\ J_r(\theta_r;\theta_j), & b=r. \end{cases}\] Let \(L_{G,b}(\theta_b;\theta_{-b})\) denote the one-step on-policy GRPO surrogate for phase \(b\), and let the GRPO iterates satisfy \[\theta_b^{t+1} = \theta_b^t + \eta_t g_t^{(b)}, \qquad t=0,1,\dots,T-1, \label{eq:appendix-phase-update}\tag{10}\] where \(g_t^{(b)}\) is the stochastic gradient estimator for the active phase.

We use the following assumptions for Theorems 3 and 4.

9.1.0.1 A1. Phase-wise smoothness and upper boundedness.

For each phase \(b\in\{j,r\}\), \(L_{G,b}(\theta_b;\theta_{-b})\) is \(L_b\)-smooth in \(\theta_b\) when \(\theta_{-b}\) is fixed, and \(L_{G,b}(\theta_b;\theta_{-b})\le L_{G,b}^{\star}\).

9.1.0.2 A2. Unbiased phase-wise gradient estimator.

At each step, \[\mathop{\mathrm{\mathbb{E}}}[g_t^{(b)}\mid \theta_b^t] = \nabla_{\theta_b} L_{G,b}(\theta_b^t;\theta_{-b}). \label{eq:appendix-unbiased}\tag{11}\]

9.1.0.3 A3. Bounded conditional variance.

There exists a constant \(\sigma_b^2\) such that \[\mathop{\mathrm{\mathbb{E}}}\!\left[\left\lVert g_t^{(b)}-\nabla_{\theta_b}L_{G,b}(\theta_b^t;\theta_{-b}) \right\rVert^2 \mid \theta_b^t\right] \le \frac{\sigma_b^2}{G_b}, \label{eq:appendix-var}\tag{12}\] where \(G_j=2n\) for the judge phase and \(G_r=n\) for the generator phase.

9.1.0.4 A4. Step size.

We use \[\eta_t=\frac{\eta_0}{\sqrt{T}}, \qquad \eta_0\le \frac{1}{L_b}. \label{eq:appendix-stepsize}\tag{13}\]

9.1.0.5 A5. Phase-wise surrogate-to-true gradient gap.

For each phase \(b\in\{j,r\}\), there exist nonnegative constants \(B_{\mathrm{grp}}^{(b)}\), \(B_{\mathrm{stale}}^{(b)}\), and \(B_{\mathrm{clip}}^{(b)}\) such that \[\begin{align} \notag&\sup_{\theta_b} \mathop{\mathrm{\mathbb{E}}}\left\lVert \nabla_{\theta_b}J_b(\theta_b;\theta_{-b})-\nabla_{\theta_b}L_{G,b}(\theta_b;\theta_{-b}) \right\rVert^2 \\&\le \frac{B_{\mathrm{grp}}^{(b)}}{G_b} + B_{\mathrm{stale}}^{(b)} + B_{\mathrm{clip}}^{(b)}. \label{eq:appendix-gradient-gap} \end{align}\tag{14}\]

9.2 Proof of Theorem 1↩︎

9.3 Proof of Theorem 2↩︎

9.4 Proof of Theorem 3↩︎

9.5 Proof of Theorem 4↩︎

10 Prompts↩︎

We report the prompts used in our experiments in this section. For baseline methods, we follow the prompts provided by their official implementations and original papers.

Your job is to look at a conversation and a set of rubric items, and score the last turn (i.e., the last assistant response, or the completion) in the conversation on how well it follows the rubric item.

# Conversation
<<conversation>>

# Rubric item
<<rubric_item>>

# Instructions
Return a json object. For each rubric item i (starting from 1), keys must be exactly "explanation_i" and "criteria_met_i" for each i and it includes two top-level fields in the JSON object:
- The "explanation_i" field should be a string explaining why the response does or does not meet the criteria of the rubric item.
- The "criteria_met_i" field should be a boolean indicating (true/false) whether the response meets the criteria of the rubric item. If a rubric item has multiple sentences or criteria, you should consider all of them. If any of the criteria is not met, the answer should be false. Only return true is all of the criteria are met.
- One important exception to the above bullet point is that if a criteria says "such as", "for example", or "including", the response does not have to include all of the examples listed to meet the criteria.

# Final Output Format (a single JSON object, not an array)
{
  "explanation_1": "...",
  "criteria_met_1": true/false,
  "explanation_2": "...",
  "criteria_met_2": true/false,
  ... repeat this pattern for every rubric item i in order (i = 1, 2, 3, ...)
}

# Final instruction
Return just the json object. Do not include any other text in the response.

References↩︎