1 Introduction↩︎

Reinforcement learning depends on the reward signal’s ability to discriminate among policy outputs of varying quality [1]–[5]. In verifiable domains such as math and code, programmatic checkers provide this signal directly [6], [7] but open-ended instruction following has no analogous checker. Bradley-Terry reward models [8]–[10] provide only coarse discrimination, leading to reward hacking and plateaued performance on complex benchmarks [11]–[14]. Recent work conditions LLM judges on instance-specific rubrics [12], [15], [16], recovering fine-grained discrimination and generalizing zero-shot to novel domains [17].

The discriminative power of the LLM judge, however, comes with a precondition: the prompt itself must induce rollouts that differ in quality. We refer to such prompts as challenging for the current policy. When a prompt is too easy for the current policy, all rollouts succeed; when too hard, all rollouts fail. In either case, the rubric judge has nothing to discriminate, and the reward signal collapses regardless of judge quality. This is a property of the policy-prompt pair, not of the prompt alone. Yet existing pipelines for non-verifiable RL with LLM judges draw training prompts from static corpora [12], [15], [16], leaving the supply of challenging prompts to chance.

A natural response is to adapt prompts to the current policy so they reliably induce discriminative rollouts, a property we call policy-adaptiveness. Recent work in non-verifiable RL pursues this but does not fully exploit the discriminative capabilities of LLMs. One line uses scalar reward-model scores [18], inheriting the coarse-discrimination problem that motivated rubric-equipped judges in the first place. Another uses LLM judges with generic, prompt-agnostic rubrics [19], which lack the fine-grained discrimination instance-specific rubrics provide [16], [17], [20].

We propose LLM-as-a-Tutor, a method that adapts prompts to the current policy by exploiting the discriminative capabilities of LLMs that prior work has left unused. Detecting whether a prompt is challenging for the current policy is a fundamentally pairwise question, namely whether two rollouts from the policy differ in quality. Pairwise judgment is the mode in which LLMs are consistently sharper and more human-aligned than pointwise scoring [21]–[23]. We extend the role of the LLM from judge to tutor, in which a single LLM plays two roles: an examiner that compares rollout pairs to detect non-challenging prompts, and a generator that augments them.

Concretely, the tutor operates as follows. Each seed prompt is paired with a base rubric, generated once by the tutor, that serves as the reward signal during training. At each iteration, the examiner samples a pair of rollouts under the current policy and assesses whether they differ in quality. If they do not, the generator appends an additional constraint to the prompt and corresponding criteria to the rubric in a single call, while leaving the seed prompt and base rubric otherwise intact. In complex instruction following, difficulty is naturally instantiated as the number of requirements a model must simultaneously satisfy [24], so this modification monotonically increases difficulty while preserving the source distribution.

We evaluate LLM-as-a-Tutor on three complex instruction-following benchmarks, FollowBench [24], AdvancedIF [13], and InfoBench [14], using Qwen3-1.7B [25] as the policy and Qwen3-8B as the tutor. Our method achieves the highest average score, outperforming both baselines that train on fixed prompts (base rubrics, WildChecklists [12], policy-adaptive rubrics) and baselines that modify prompts (Evol-Instruct [26], EVA [18]). Notably, policy-adaptive rubrics also underperform on average, suggesting that for some prompts no rubric refinement can recover discriminative signal and the prompt itself must be changed. Ablations confirm that the gains come from policy-aware adaptation: applying constraints unconditionally, at random, or against a stronger model’s responses all underperform targeted appending, and appending outperforms rewriting prompts. Analysis shows that the tutor’s pairwise judgments correlate with empirical difficulty, that each appended constraint amplifies the discriminative signal carried by the base rubric, and that the modification rate rises monotonically with policy scale.

Our contributions are as follows:

• We identify policy-prompt mismatch as a precondition that limits rubric-equipped reward judges in non-verifiable RL.

• We propose LLM-as-a-Tutor, which extends the LLM’s role from judge to tutor by detecting non-challenging prompts via pairwise comparison [21]–[23] and augmenting them with atomic constraints.

• Our method outperforms both policy-unaware and policy-adaptive baselines on three instruction-following benchmarks, establishing prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

2 Related work↩︎

2.1 Reward variance as a learnability signal↩︎

A long-standing principle in RL is that learning requires reward variation across sampled trajectories: when all trajectories from a state receive identical returns, the policy gradient carries no signal about which action was preferable. This motivates curriculum methods that select tasks at the frontier of agent competence [2], [27], and the same idea has recently been adapted to LLM RL by filtering prompts whose empirical pass rate is too high or too low [1] or by sampling prompts in proportion to per-prompt reward variance [28]. A parallel line of self-play methods [29]–[31] co-evolves task generators with solvers under moderate-difficulty objectives so that neither side of the success distribution collapses. All of these assume binary, verifiable rewards, whereas we target rollout-level discriminative variance under continuous, rubric-based rewards in non-verifiable instruction following.

2.2 Rubric-based non-verifiable RL↩︎

Standard non-verifiable RL relies on scalar reward models trained on preference data, but these signals are coarse and brittle—prone to reward hacking, biased by surface features such as response length, and unstable under optimization [11]. Building on rubric-based fine-grained evaluation [17], [20], recent work instead repurposes instance-specific rubrics as reward signals for on-policy RL: RLCF [12], Rubrics as Rewards [16], and AdvancedIF [13] all train against per-prompt checklists scored by LLM judges, while OpenRubrics [32] tackles the complementary problem of scaling rubric construction itself. A subsequent direction makes the rubrics adaptive rather than fixed: DR Tulu [15] maintains rubrics that co-evolve with the policy during training, continuously tailoring criteria to its current behavior so that the reward signal remains discriminative as the model improves.

2.3 Policy-adaptive methods for RL↩︎

A line of work adapts the training distribution to the policy’s current capability during RL. DR Tulu [15] evolves rubrics rather than prompts, while EVA [18] and LSP [19] evolve prompts but rely on scalar RMs or generic, prompt-agnostic rubrics, forgoing the fine-grained signal of instance-specific rubrics. Our method extends policy-adaptive prompt evolving to the rubric-based setting, using instance-specific rubric judges with pairwise saturation checks [21] and appending atomic constraints rather than rewriting prompts, monotonically raising empirical difficulty while preserving the seed distribution.

3 Method↩︎

3.1 Preliminaries↩︎

3.1.0.1 Rubric-based reward.

We consider RL for non-verifiable instruction-following tasks, where for each prompt \(x \sim \mathcal{D}\), a prompt-specific rubric \(R(x) = \{(r_k, w_k)\}_{k=1}^{K}\) defines a set of natural-language criteria \(r_k\) with non-negative weights \(w_k\) satisfying \(\sum_{k=1}^{K} w_k = 1\). We refer to \(R(x)\) as the base rubrics, since it captures broad quality criteria for \(x\) independent of any particular policy. The reward score \(s\) for a response \(y\) given a target prompt \(x\) is computed as a weighted sum of per-criterion judgments from an LLM judge \(\mathcal{J}\): \[s(x, y) = \sum_{k=1}^{K} w_k \cdot \mathcal{J}(y \mid x, r_k), \label{eq:rubric95reward}\tag{1}\] where \(\mathcal{J}(y \mid x, r_k) \in [0, 100]\)² denotes the judge’s score for how well \(y\) satisfies criterion \(r_k\). Following prior work [12], [15], \(\mathcal{J}\) evaluates each criterion independently, producing a separate judgment per criterion. Training on a fixed prompt corpus with reward computed solely from the base rubrics \(R(x)\) constitutes our baseline.

3.1.0.2 Policy optimization with GRPO.

For each prompt \(x\), we sample \(G\) rollouts \(\{y^{(i)}\}_{i=1}^{G}\) from the old policy \(\pi_{\theta_{\text{old}}}\) and optimize the policy \(\pi_\theta\) by maximizing the clipped surrogate objective [5], [6] \[{!}{ \displaystyle \mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y^{(i)}|} \sum_{t=1}^{|y^{(i)}|} \min\!\left( \rho^{(i)}_{t}(\theta)\, A^{(i)},\, \mathrm{clip}(\rho^{(i)}_{t}(\theta), 1-\varepsilon, 1+\varepsilon)\, A^{(i)} \right) - \beta\, \mathbb{D}_{\text{KL}}[\pi_\theta \,\|\, \pi_{\text{ref}}] \right] } \label{eq:grpo95objective}\tag{2}\] with per-token importance ratio \(\rho^{(i)}_{t}(\theta) = \pi_{\theta}(y^{(i)}_{t} \mid x, y^{(i)}_{<t}) / \pi_{\theta_{\text{old}}}(y^{(i)}_{t} \mid x, y^{(i)}_{<t})\), clipping range \(\varepsilon\), and KL coefficient \(\beta\) regularizing toward a frozen reference policy \(\pi_{\text{ref}}\). Following [6], the per-rollout advantage \(A^{(i)}\) is broadcast to every token of \(y^{(i)}\) and computed in a group-relative manner, \[A^{(i)} = \frac{s(x, y^{(i)}) - \mu(x)}{\sigma(x)}, \label{eq:grpo95advantage}\tag{3}\] where \(\mu(x)\) and \(\sigma(x)\) are the mean and standard deviation of \(\{s(x, y^{(i)})\}_{i=1}^{G}\) (with a small constant added to \(\sigma(x)\) for numerical stability). The advantage \(A^{(i)}\) provides a meaningful learning signal only when the rollouts within a group exhibit sufficient spread in score.

3.1.0.3 Discriminative signal as a property of the prompt–policy pair.

Whether a prompt induces rollouts of varying quality is a property of the prompt paired with the current policy, not of the prompt alone. When the pair is misaligned, the rollouts within a group offer little contrast for the advantage to exploit, regardless of the quality of \(R(x)\) or \(\mathcal{J}\). Such misalignment arises naturally when prompts are drawn from static corpora [33] and tends to worsen as the policy improves under RL [1], [18], [28]. Closing this gap requires adapting the prompts themselves to the policy’s evolving capability.

3.2 LLM-as-a-Tutor↩︎

3.2.0.1 Pipeline overview.

The tutor first generates a base rubric \(R(x)\) for each prompt, capturing general requirements derived from \(x\). Then, at the start of each training iteration, the tutor processes every prompt in a single call: it samples two rollouts from the current policy and judges whether they are indistinguishable in quality. If they are, the tutor appends an atomic constraint to \(x\) along with a matching rubric criterion. Discriminative prompts pass through unchanged. The resulting prompts and rubrics then feed into the GRPO update, where the LLM judge scores each rollout against every rubric criterion and the weighted sum serves as the reward. Each adaptation only appends to the prompt, so any constraints added in earlier iterations remain in place and can be extended further as the policy improves. The result is a per-prompt curriculum whose pace is set by the policy’s own behavior.

We instantiate this tutor as a single model \(\mathcal{T}\) playing two roles. The first role is a pairwise judge of discriminativeness. The second is a generator of atomic constraints. We describe each role in turn below. The full prompt templates used for the tutor model are provided in . .

3.2.0.2 Pairwise discriminativeness judgment.

The tutor decides whether \(x\) is discriminative for \(\pi_\theta\) by comparing two rollouts from \(\pi_\theta\). We use a pairwise comparison rather than independent scoring since LLMs are consistently sharper and more human-aligned in pairwise comparisons than in pointwise scoring [21]–[23]. Specifically, the tutor samples two rollouts \(y^{(1)}, y^{(2)} \sim \pi_\theta(\cdot \mid x)\) and makes a binary judgment on whether they are indistinguishable in quality; if so, \(x\) is declared non-discriminative for the current policy. The judgment considers whether the two rollouts agree in quality, in their overall approach, and in the absence of meaningful weaknesses (tutor’s prompt template in . ). We verify in . that prompts judged non-discriminative indeed have higher mean reward and lower variance across the full rollout group than those judged discriminative.

3.2.0.3 Constraint-based prompt adaptation.

When \(x\) is non-discriminative, the tutor invokes its second role and produces an atomic constraint \(c\) together with rubrics \(R_c = \{(r_{c,j}, w_{c,j})\}_{j=1}^{m}\) scoring adherence to it. The constraint \(c\) is atomic in the sense of imposing a single requirement along a dimension left unspecified by \(x\). The constraint is appended to form the adapted prompt and rubric set, \[\tilde{x} = x \oplus c, \qquad \tilde{R}(x) = R(x) \cup R_c, \label{eq:adapted}\tag{4}\] with the weights of \(R(x)\) and \(R_c\) jointly renormalized to sum to one. Discriminative prompts are left untouched: \((\tilde{x}, \tilde{R}(x)) = (x, R(x))\). The reward and GRPO objective from . are then applied unchanged with \((x, R(x))\) replaced by \((\tilde{x}, \tilde{R}(x))\), so that adaptation is the sole locus of change relative to the baseline.

The key property of this append-only design is that adaptation monotonically increases difficulty: any response that satisfies \(\tilde{x}\) must also satisfy \(x\), making \(\tilde{x}\) no easier than \(x\) by construction. Strategies that rewrite the prompt offer no such guarantee, and we show in . that a rewriting variant indeed fails to raise empirical difficulty. Retaining \(R(x)\) in \(\tilde{R}(x)\) also keeps the original criteria of \(x\) active in the reward. As \(\tilde{x}\) accumulates constraints and the policy must satisfy them simultaneously, scoring well on \(R(x)\) itself grows increasingly demanding, leaving the base rubrics to continue to carry discriminative signal across iterations ( . ).

3.2.0.4 Difficulty escalation across iterations.

The tutor is invoked at the start of every training iteration, where an iteration denotes any chosen adaptation interval (e.g., every epoch, every \(N\) train steps) so that prompts adapted in earlier iterations can be revisited as the policy improves. Let \(x^{(t)}\) denote the prompt presented to the policy at iteration \(t\), with \(x^{(0)} = x\). If \(x^{(t)}\) is non-discriminative for \(\pi_{\theta_t}\), the tutor appends a new atomic constraint \(c^{(t)}\) to form \(x^{(t+1)} = x^{(t)} \oplus c^{(t)}\), with \(\tilde{R}(x^{(t+1)})\) accumulating the corresponding rubrics; otherwise \(x^{(t+1)} = x^{(t)}\). Since each adaptation monotonically increases difficulty, the sequence \(\{x^{(t)}\}\) is non-decreasing in difficulty for any policy, while the trigger for escalation is set by the current \(\pi_{\theta_t}\). Prompt difficulty thus escalates in step with the policy’s improving capability, realizing the dynamic adaptation that a fixed prompt corpus cannot provide.

4 Experiments↩︎

4.1 Experimental setup↩︎

We train Qwen3-1.7B-Thinking [25] as the policy, with Qwen3-8B-Thinking serving as both tutor and judge. We train for 3 epochs on randomly sampled 4K prompts from WildChat [33], and we instantiate the tutor’s adaptation interval as one full epoch. We evaluate on three instruction-following benchmarks: FollowBench [24], AdvancedIF [13], and InfoBench [14]. Detailed training and evaluation configurations are provided in . .

4.2 Baselines↩︎

We compare our method against baselines that differ in how they treat the seed prompts \(x\) during training, grouped into three categories.

4.2.0.1 No prompt modification.

These methods train directly on the fixed WildChat prompts. Base rubrics use \(\mathcal{R}(x)\) as the reward; WildChecklists [12] uses high-quality rubrics generated offline; and policy-adaptive rubrics, motivated by DR Tulu [15], additionally inject criteria targeting the policy’s current weaknesses at each iteration, while still leaving \(x\) unchanged. As a non-RL reference, we also include Distillation [34], [35], a supervised-fine-tuned variant of the policy trained on responses sampled from the Qwen3-8B tutor for the same prompts; this uses the tutor as a teacher rather than as an examiner-generator.

4.2.0.2 Policy-unaware prompt modification.

This category rewrites prompts to be harder independently of the policy. Evol-Instruct [26] applies predefined operations (e.g., adding constraints, deepening) to increase prompt complexity in a single offline pass, controlling for whether harder prompts alone improve learning, independent of policy-awareness.

4.2.0.3 Policy-adaptive prompt modification.

These methods adapt prompts so they remain challenging for the current policy. EVA [18] selects prompts via scalar reward-gap signals and rewrites them in a single step. Our EVA re-implementation matches the original method’s prompt-evolution mechanism but adopts our training stack (GRPO, Qwen3-8B evolver, rubric-based reward) for a controlled comparison. In contrast, our LLM-as-a-Tutor detects non-discriminative prompts via pairwise quality examination and appends atomic constraints to them rather than rewriting. Further details for each baseline are provided in . .

4.3 Main results↩︎

2.5pt

4.3.0.1 LLM-as-a-Tutor outperforms both policy-unaware and prior policy-adaptive baselines.

As shown in . , LLM-as-a-Tutor achieves the highest average score across three instruction-following benchmarks and outperforms all baselines on five out of six metrics. The gain over the policy-unaware baseline confirms the value of adaptation itself, while the gain over prior policy-adaptive methods indicates that adapting at the level of prompts provides a richer training signal that rubric-based RL alone cannot capture.

4.3.0.2 Policy-adaptive methods outperform policy-unaware ones.

Policy-unaware methods (Base rubrics, WildChecklists) yield modest improvements over the untrained policy, while methods that adapt to the policy at training time (Policy-adaptive rubrics, EVA) deliver larger gains and reach the same average (51.04) despite adapting to the rubric or prompt respectively. In contrast, Evol-Instruct, which rewrites prompts to be harder independently of the policy, underperforms even Base rubrics (50.24 vs.), confirming that increased difficulty without policy-awareness can degrade training.

4.3.0.3 Prompt adaptation outperforms rubric adaptation.

LLM-as-a-Tutor outperforms Policy-adaptive rubrics though both methods adapt to the current policy. The two methods differ in what they can change: rubrics determine how rollouts are scored, while prompts determine what rollouts the policy produces. When a prompt fails to elicit quality gap among rollouts, no rubric—regardless of how well it is adapted—can produce a discriminative reward signal. In such cases, modifying the prompt itself can restore variance among rollouts.

4.4 Ablation on policy-adaptiveness↩︎

4.4.0.1 Policy-adaptive constraint addition outperforms non-adaptive variants.

. compares our method against heuristic variants that share the same append-based modification but differ in how target prompts are chosen. Applying constraints to all prompts (Always) or to a random 28% (Random, matching the average constraint-added ratio across three epochs) both fall short of our adaptive variant, indicating that constraint addition is effective only when targeted at the right prompts. Examining a larger Qwen3-8B model’s rollouts instead of the policy’s own (Wrong) also underperforms, despite augmenting a larger fraction of prompts (47% vs.our 28%), showing that the relevant signal is the current policy’s behavior, not difficulty alone.

4.5 Ablation on prompt adaptation strategies↩︎

4.5.0.1 Append outperforms other prompt modification methods.

. compares three strategies for modifying a prompt once it is judged non-discriminative. Reset replaces the previous constraint with a new one each epoch, while Rewrite regenerates the entire prompt as a more challenging one. Neither guarantees monotonically increasing difficulty: discarding earlier constraints or rewriting from scratch can produce a prompt no harder than before. Rewrite has an additional drawback in that it drifts the prompt away from the original task distribution, weakening the alignment between the training signal and the instruction-following objective. Append (Ours), in contrast, adds a new constraint while keeping previous ones, so each adaptation monotonically increases difficulty while preserving the seed distribution. Append achieves the best performance on all three benchmarks.

5 Analysis↩︎

5.1 Analysis of the tutor’s behavior↩︎

5.1.0.1 Policy scaling drives denser constraint generation.

. shows the ratio of prompts augmented with constraints as the policy scales from 0.6B to 4B, with the tutor fixed at 8B. The ratio grows monotonically with policy scale, from 8.1% to 40.5%. As the policy scales up, seed prompts become relatively easier and the policy produces high-quality rollouts with little variation among them, leading the tutor to judge more prompts as non-challenging. As a result, the tutor augments more prompts as the policy grows stronger, without relying on a fixed schedule.

5.1.0.2 Model-based decision correlates with empirical difficulty.

. compares the per-prompt reward mean and standard deviation between prompts the tutor identifies as challenging and those it identifies as non-challenging. Non-challenging prompts have a higher mean reward (90.76 vs.) and a lower standard deviation (12.96 vs.), meaning the policy already produces high-quality and similar rollouts on them. Challenging prompts show the opposite pattern, with lower mean and higher variance among rollouts. This separation shows that the tutor’s decision effectively tracks the policy’s empirical performance rather than being arbitrary.

5.1.0.3 Appended constraints amplify the discriminative power of base rubrics.

. reports the per-prompt reward mean and standard deviation evaluated on the base rubrics only, as the number of appended constraints grows from 0 to 3. The mean tends to decrease as more constraints are appended: the policy finds it harder to satisfy the same base criteria once additional constraints must be satisfied simultaneously, consistent with the append-only design that monotonically raises difficulty. The standard deviation tends to increase over the same range, suggesting that the base rubrics continue to discriminate among rollouts rather than saturating as the policy improves.

5.2 Qualitative analysis↩︎

3 illustrates how a single atomic constraint converts a saturated prompt into one that induces discriminative rollouts. On the original prompt, all rollouts score \(100\) on the base rubrics (\(\sigma_{\text{base}} = 0\)): the prompt is too easy for the current policy, every rollout succeeds, and the prompt no longer provides an informative reward signal. Appending one clause that demands a specific named policy (highlighted in red) breaks this saturation and fractures the rollouts with a clear quality gap. For example, Rollout A engages the new requirement directly, inserting an explicit [Company Policy Name] citation and scoring \(100\); Rollout B retreats to a generic “consult your supervisor” deflection without naming any policy and scores \(60\). Quantitatively, the mean of the base rubrics drops from \(100\) to \(92.5\) and the standard deviation rises from \(0\) to \(14.9\), introducing variance that restores an informative reward signal. This example concretely illustrates our method’s mechanism: the tutor’s pairwise judgment identifies a saturated policy-prompt pair, and a targeted atomic constraint reinstates a discriminative reward signal without rewriting the seed task. We provide additional qualitative samples in . .

6 Discussion↩︎

6.0.0.1 LLM-as-a-Tutor as a potentially stronger form of distillation.

Rubric-based RL succeeds because fine-grained reward signals nudge the policy toward responses it is already capable of producing but does not by default [12], [16]. LLM-as-a-Tutor extends this principle to the prompt distribution, using the tutor’s discriminative capability to identify where the policy requires harder prompts and to construct them. We believe this has the potential to constitute a new form of distillation. Unlike SFT distillation, which is bounded by the teacher’s outputs [34]–[36], our tutor produces no target response and instead shapes the training signal so the student discovers the capability through RL. The student’s ceiling is therefore not the tutor’s ability to answer the prompt, but only its ability to discriminate between two answers, a regime in which LLMs are consistently sharper [21]–[23].

6.0.0.2 Generalizing the LLM-as-a-Tutor framework beyond constraint-additive tasks.

Our choice to append atomic constraints aligns with how complex instruction following is operationalized in benchmarks such as FollowBench [24], where difficulty is defined by the number of constraints a response must simultaneously satisfy. This alignment is by design, since constraint count is the natural notion of difficulty for non-verifiable instruction following. The broader pipeline, however, is task-agnostic, and different domains admit different modification templates: added reasoning steps for multi-step reasoning [7], additional edge cases for code, tighter sourcing requirements for factual QA [15]. The underperformance of the rewrite variant reflects the harder nature of the rewrite operation, which must add complexity along axes other than constraint count while preserving the source distribution, and we expect this trade-off to become more favorable as tutor models grow more capable.

7 Conclusion↩︎

We introduce LLM-as-a-Tutor, which addresses the policy–prompt mismatch in rubric-based non-verifiable RL by reusing the judge LLM to also adapt the training prompts. When seed prompts fail to elicit quality variance among rollouts, the tutor appends targeted constraints, yielding a self-calibrating curriculum that tracks the policy’s capability without external difficulty schedules. Across three instruction-following benchmarks, our method consistently outperforms both policy-unaware and prior policy-adaptive baselines, establishing prompt adaptation as a missing axis of policy-awareness in non-verifiable RL, complementary to rubric adaptation. We believe this framework can extend naturally beyond instruction following to any domain with an appendable notion of difficulty, such as reasoning steps, edge cases, or sourcing requirements, pointing toward a task-agnostic recipe for shaping training signal as policies grow stronger.

8 Limitations↩︎

8.0.0.1 Scope of evaluated policy–tutor configurations.

Our main results in . are reported for a single policy–tutor pair (Qwen3-1.7B policy with Qwen3-8B tutor), and we do not run the full benchmark suite across additional model families or larger policy sizes due to compute constraints. The mechanism behind LLM-as-a-Tutor, however, is not specific to this pair: . sweeps the policy from 0.6B to 4B with the tutor held fixed and shows that the constraint-addition ratio rises monotonically with policy capability (8.1% \(\rightarrow\) 25.8% \(\rightarrow\) 40.5%). The adaptation procedure thus tracks policy capability rather than locking onto a particular checkpoint, suggesting the method should generalize to other policies whose capability falls within the tutor’s discriminative range.

8.0.0.2 Dependence on tutor capability.

The method assumes the tutor can perform pairwise discrimination and atomic constraint generation reliably; a weaker tutor produces noisier saturation judgments and less targeted constraints, as the Wrong baseline in . illustrates when judgments are made on a stronger model’s rollouts rather than the policy’s. Because pairwise judgment and constraint generation are both reasoning-heavy subtasks, improvements in tutor reasoning capability translate directly into stronger adaptation, and the same pipeline should yield correspondingly stronger results as more capable open models become available.

8.0.0.3 Additional inference cost from tutor invocations.

LLM-as-a-Tutor introduces additional tutor calls during training. Per training prompt, our method requires two sample rollouts plus a single tutor invocation for the pairwise saturation judgment, and one additional call for constraint generation when the prompt is judged saturated. For comparison, the existing rubric-based RL pipeline already consumes 8 policy rollouts and \(8 \times N\) judge invocations per prompt, where \(N\) is the number of rubric criteria (typically 5–10). The tutor’s cost is therefore a small fraction of the per-step inference budget already required for rubric-based reward computation.

9 Broader impacts↩︎

Our work improves the instruction-following capability of language models through policy-adaptive prompt modification in non-verifiable RL. The positive impact is more reliable instruction following: models that better track user-specified constraints are more useful for downstream applications such as writing assistance, document drafting, and other open-ended tasks where rubric-based quality matters. The training-time mechanism is also general; it could be combined with safety- or helpfulness-oriented rubrics to improve adherence to those objectives.

The same mechanism, however, has dual-use potential. An adaptation procedure that elicits stronger compliance with arbitrary instructions can equally elicit stronger compliance with harmful ones if the seed corpus or rubrics target undesirable objectives. Because LLM-as-a-Tutor is task-agnostic—it does not encode any notion of what the appended constraints should optimize for—it inherits whatever values are implicit in the seed prompts and the tutor’s judgments. We mitigate this in our experiments by using a public, non-adversarial prompt corpus (WildChat) and rubrics aimed at general response quality, but practitioners applying our method should ensure both the seed distribution and the tutor model are aligned with their intended deployment context. We do not release model checkpoints trained with this method, which limits direct misuse risk from this work.

10 Experimental setup↩︎

10.1 Training↩︎

We implement all experiments using VeRL [37] framework. Training hyperparameters are summarized in . .

6pt

10.2 Evaluation↩︎

We evaluate all policy models on three instruction-following benchmarks (FollowBench, AdvancedIF, InfoBench) using a unified vLLM-based generation pipeline [38]. Because LLM-based judges introduce non-trivial scoring stochasticity, we report the mean over \(5\) independent evaluation runs for every benchmark. For FollowBench, we evaluate on the content, situation, style, format, and mixed categories, excluding the example category. For AdvancedIF, we restrict evaluation to the single-turn complex instruction-following split. Otherwise, we follow the standard evaluation protocol for each benchmark, including the metric calculation. Sampling parameters and the judge model used for each benchmark are summarized in . .

6pt

10.3 Licenses for existing assets↩︎

All assets used in this work are publicly available and used in compliance with their respective licenses for non-commercial academic research. Qwen3-1.7B and Qwen3-8B [25] are released under the Apache 2.0 License. WildChat [33] is released under ODC-BY 1.0. FollowBench [24] is released under the Apache 2.0 License. AdvancedIF [13] is released under CC-BY-NC-4.0; we use it solely for evaluation in a non-commercial research context. InfoBench [14] is released under the MIT License. The gpt-5-mini judge is accessed through the OpenAI API in accordance with OpenAI’s usage policies; we do not redistribute judge outputs.

10.4 Compute resources↩︎

All experiments are run on NVIDIA H100 GPUs. Each main result in . corresponds to a single training run on 4\(\times\)H100, taking approximately one day end-to-end (rollout generation, tutor invocation, GRPO updates, and per-epoch tutor adaptation). Baselines that share our GRPO setup (Base rubrics, WildChecklists, Policy-adaptive rubrics, EVA) consume comparable compute per run. Evaluation is run on the same hardware via vLLM [38], with judge calls served by the gpt-5-mini API; running all three benchmarks across five evaluation runs for one method costs approximately $15 in API spend under OpenAI’s flex tier.

11 Baseline details↩︎

11.0.0.1 SFT-distilled.

We fine-tune the Qwen3-1.7B policy on responses generated by the Qwen3-8B tutor for the same WildChat prompts used in our main experiments. For each prompt, we sample a single response from Qwen3-8B and apply standard cross-entropy fine-tuning over the (prompt, tutor-response) pairs. Hyperparameters are summarized in . .

6pt

11.0.0.2 Evol-Instruct.

Evol-Instruct [26] serves as our policy-agnostic prompt adaptation baseline: prompts are rewritten offline using only the seed prompt itself, without consulting the policy. Each WildChat seed prompt undergoes four sequential evolution rounds; at each round, we uniformly sample one of five mutation methods (four in-depth operators—Add Constraints, Deepening, Concretizing, Increased Reasoning Steps—and one in-breadth operator) and ask Qwen3-8B to rewrite the prompt under the corresponding template.³ The output of the final round replaces the original prompt one-to-one. Because our setup uses rubric-based RL, we then regenerate the base rubric on each rewritten prompt so that the reward signal stays aligned with the modified prompt.

11.0.0.3 EVA.

EVA [18] selects prompts via a reward gap derived from a scalar reward model: for each prompt \(x\), responses \(\{y_i\}_{i=1}^K\) sampled from the policy are scored by a scalar reward model \(r\), and the gap \(\max_i r(x, y_i) - \min_i r(x, y_i)\) serves as the informativeness signal. Selected prompts are then modified by complete rewriting with Evol-Instruct templates, without reference to policy rollouts.

We re-implement EVA following the algorithmic specification in the original paper, with three deviations driven by our infrastructure and the rubric-based RL setup. First, we train under GRPO rather than DPO [39]. Second, while the original paper uses gemini-1.5-pro to evolve prompts, we replace it with Qwen3-8B for consistency with the tutor used by our method and other baselines. Third, the original paper derives the reward from a scalar reward model (Skywork/Skywork-Reward-V2-Llama-3.1-8B), whereas we use our rubric-based reward by regenerating the base rubric after each prompt evolution. All other components, including the informativeness-based prompt selection and the prompt evolution templates, follow the original implementation. These deviations are summarized in . .

6pt

11.0.0.4 Policy-adaptive rubric.

We include policy-adaptive rubric generation as a baseline that adapts only the rubric, isolating the contribution of rubric-level adaptation from the prompt-level adaptation used in our main method. As in LLM-as-a-Tutor, the tutor first judges whether the current rubric discriminates among the policy’s rollouts; if not, it generates additional criteria targeting the specific weaknesses observed in those rollouts. These adaptive criteria are appended to the base rubric at each training iteration; the prompt used for adaptive rubric generation is provided in . . In contrast to . , which augments both the prompt and the rubric, here only the rubric is adapted while the prompt is left unchanged (\(\tilde{x} = x\), \(\tilde{R}(x) = R(x) \cup R_{adaptive}\)).

12 Prompt templates↩︎

This section lists the prompt templates used in our pipeline. Slots wrapped in {{...}} are filled at runtime with the corresponding field.

12.1 Judge prompt↩︎

Our judge prompt is adapted from RLCF [12], with minor modifications. The full template is shown below.

You are evaluating a model's response to an instruction against a specific criterion. Your task is to score how well the response satisfies that criterion, given the context of the instruction.

Score the response on a scale of 0 to 100 using the following anchors. For example, the instruction might be "What is a good vegan substitute to meat for someone allergic to soy and gluten? Provide a single-sentence response consisting of an answer followed by a factually detailed and humorous one-sentence explanation", and the criterion might be "Is the explanation factually detailed?". Your selection should be based on the response and the criterion, using the following rating scale:

- 100: The response represents an optimal solution that expertly balances all relevant aspects of the instruction. For the example above (about the vegan substitute), and the criterion above (about factual detail), an example 100-point response is "Mushrooms, because they can be easily caramelized and browned, they are rich in the glutamates which lead to incredible umami flavors, they naturally are completely free of soy and gluten, and they don't look cute as babies". This response is richly detailed and factual, and though it fails to be humorous, it is still a 100-point response on the factual detail criterion.

- 75: The response very effectively addresses the criterion but has room for minor improvements. The response should be unconditionally acceptable (at a professional level) but may not be absolutely perfect. There are no mistakes that critically undermine the criterion. An example 75-point response to the example criterion above is "Mushrooms - they are rich in the glutamates that lead to incredible umami flavors and they don't look cute in the slightest while alive.". This response has one interesting fact but could be more detailed.

- 50: The response adequately fulfills the criterion but contains notable flaws or missed opportunities for improvement. The response should still be functionally acceptable. The response contains at most one minor inadequacy or inaccuracy related to the criterion but there are no mistakes that critically undermine the criterion. An example 50-point response to the example criterion above is "Mushrooms, because they can be easily caramelized and browned, they're universally beloved by sophisticated palates, and they don't look cute in the slightest while alive." The statement that they're universally beloved by people with sophisticated palates, while potentially true, is vague and not objective.

- 25: The response fulfills the key condition specified by the criterion and demonstrates awareness of the key condition but fails to execute them effectively. The text may contain non-critical inaccuracies or irrelevant information. However, if there is even one element that critically undermines the core purpose specified in the criterion (even if that element seems minor in isolation), the score should be 0 (not 25). An example 25-point response to the example criterion above is "Mushrooms, because they can be easily caramelized and browned, they are absolutely brimming with protein, and they don't look cute in the slightest while alive." The claim that mushrooms are "absolutely brimming with protein" is factually inaccurate.

- 0: The response fails to meet the criterion or provides no information that could be utilized to answer the criterion. If the response contains a critical error relevant to the criterion, return a 0. For the criterion about the vegan substitute, an example 0-point response is "Mushrooms, because they make you question why you ever thought a dead animal could compare to this vegan delight." While funny and engaging, this response contains zero factual detail about mushrooms, critically violating the criterion.

Your score can be any integer between 0 and 100 (not just the ones listed above). If you are totally confused, return -1.

Evaluate the response ONLY against the stated criterion. Do not penalize or reward aspects of the response unrelated to it.

<instruction>
{{instruction}}
</instruction>

<response>
{{response}}
</response>

<criterion>
{{requirement}}
</criterion>

After examining the instruction, the response, and the criterion:

- Briefly justify your score in up to 100 words.
- Conclude with the score using the following format:
  <score>{integer between 0 and 100 or -1}</score>

12.2 Tutor prompts↩︎

12.2.1 Base rubric generation↩︎

Given a seed prompt, the tutor generates a list of weighted criteria that constitute the base rubric.

You are responsible for developing criteria for judging arbitrary responses to instructions. You will be given an instruction, and your task is to write a set of rubric criteria for evaluating responses to the instruction.

---

# Task Guidelines

## Step 1 - Analyze the instruction

Analyze the instruction to understand the key quality dimensions needed to evaluate potential responses to the instruction. Identify the task scope and the explicit constraints specified in the instruction.

## Step 2 - Write rubric criteria

Write the rubric criteria in the form of complete questions. Example: 'Does the response include at least three examples?'

Each criterion question must follow these four principles:
- **Self-contained**: The criterion can be evaluated independently, without reading other criteria or needing extra context beyond the instruction and response.
- **Atomic**: Each criterion judges exactly one thing. Do not bundle multiple requirements into a single criterion.
- **Non-redundant**: No two criteria overlap or measure the same underlying quality.
- **Diverse**: Collectively, the criteria should cover a range of dimensions (e.g., correctness, depth, clarity, structure, style) rather than clustering on one aspect.

For each criterion question, assign an **importance score** between 0 and 100.
- 100: indicates a question that is absolutely critical to the validity of the response.
- 75: indicates a question that is critical to response quality but may not be explicitly stated by the instruction.
- 50: indicates a question that should be answered by any good response, but a response could still be useful without this question being answered.
- 25: indicates a question that is a preference but not a requirement.
- Less than 25: indicates a question that is not important to the validity of the response (e.g. a soft nice-to-have).

---

# Input

## Instruction

<instruction>
{{instruction}}
</instruction>

---

# Output Format

<analysis>
Summary of your analysis of the instruction.
</analysis>
<rubric>
    <criterion>Criterion question 1</criterion>
    <importance>integer score between 0 and 100</importance>
    ...
    <criterion>Criterion question N</criterion>
    <importance>integer score between 0 and 100</importance>
</rubric>

12.2.2 Constraint generation prompt↩︎

Given a prompt and a pair of policy rollouts, the tutor decides whether the prompt is saturated for the current policy; if so, it appends a single targeted constraint.

You are evaluating an instruction and two model responses. Your task is to determine whether the model has completely mastered this instruction -- leaving no room for quality improvement -- and therefore needs an additional constraint to create a meaningful learning signal.

The default answer is NO. Only say YES when the evidence clearly shows the model has hit its quality ceiling on this instruction.

---

# Step 1 - Evaluate each response independently

For each response, assess:
**Accuracy**: Is the information correct? Are there any factual errors, hallucinations, unsupported claims, or uncertain guesses?
**Completeness**: Does it address everything the instruction asks for? Is anything missing or treated superficially?
**Depth**: Does it go beyond surface-level treatment? Would a knowledgeable reader find it thorough, or would they want more?
**Execution quality**: Is it well-organized, clear, and appropriate in tone for the task?

Summarize the quality of each response. Be honest and critical -- look for genuine weaknesses, not just things you could add.

# Step 2 - Compare the two responses

Now compare them:

**Quality gap**: Is one response noticeably better than the other? If so, the model produces variable quality on this instruction -- there is already a gradient to learn from.
-> `<decision>no</decision>`
**Approach divergence**: Do the responses take meaningfully different approaches, structures, or perspectives? If so, the model is still exploring how best to handle this instruction -- variance already exists.
-> `<decision>no</decision>`
**Content convergence**: Do both responses arrive at essentially the same answer with the same structure and coverage? If so, this is a sign the model has settled on one approach and produces it consistently.

# Step 3 - Make the decision

Say YES only when ALL of the following are true:
- Both responses are **high quality** -- accurate, complete, well-executed. Neither has meaningful weaknesses.
- Both responses **converge** -- similar approach, structure, and content. They are nearly interchangeable.
- There is **no gradient** -- you cannot meaningfully say one is better than the other, because both are equally strong.

If ANY of the following is true, say NO:
- Either response has factual errors, hallucinations, or incorrect information.
- Either response is superficial, generic, or lacks depth on a topic that warrants it.
- Either response misses something the instruction asked for.
- One response is clearly better than the other.
- The responses take substantially different approaches or cover different ground.
- Either response shows uncertainty (hedging, listing multiple possibilities, "it could be...").

When in doubt, say NO.

# Step 4 - Write the constraint (only if decision is yes)

## 4a. Identify what to target

Both responses do everything well -- what dimension is entirely absent? Target only something neither response attempted.

## 4b. Choose a dimension

Select ONE:
- **Content**: Modify scope, add subtasks, mandate components.
- **Situation**: Impose a scenario, condition, edge case, or role.
- **Style**: Require a specific tone, voice, or persona.
- **Format**: Require a specific structure, length limit, or presentation style.

## 4c. Write the constraint

Write a single, atomic constraint that is:

- **Challenging**: Targets something neither response attempted.
- **Natural**: Reads as a seamless continuation of the instruction. Uses a connector (e.g., "Additionally,"). No meta-language.
- **Additive**: A genuinely new demand, not a restatement of anything already in the instruction.
- **Specific**: Concrete and evaluable.
- **Factually grounded**: Only references things present in the instruction or responses.
- **Atomic**: One requirement only.

The constraint must NOT be a generic filler ("include an example", "add a case study") or a quality baseline ("ensure accuracy").

# Step 5 - Write rubric criteria for the constraint
# (only if decision is yes)

Write rubric criteria that cover ONLY the new constraint from Step 4 -- not the original instruction. Write **at most 2 criteria**. If the constraint can be fully captured in 1 criterion, use only 1.

Each criterion must be a complete question (e.g., "Does the response include at least three examples?") and follow these principles:
- **Self-contained**: Evaluable independently, without needing other criteria or extra context.
- **Atomic**: Judges exactly one thing.
- **Positively framed**: A "yes" answer means the response satisfies the criterion.

For each criterion, assign an **importance score** (0-100):
- 100 = absolutely critical to validity.
- 75 = critical to quality but not explicitly stated.
- 50 = expected of any good response.
- 25 = preference, not a requirement.

---

# Input

## Instruction

<instruction>
{{instruction}}
</instruction>

## Model Responses

<response_1>
{{response_1}}
</response_1>

<response_2>
{{response_2}}
</response_2>

---

# Output Format

<analysis>
Step 1: Quality assessment of each response -- strengths and genuine weaknesses.
Step 2: Comparison -- quality gap? approach divergence? content convergence?
Step 3: Decision -- has the model fully mastered this instruction with no room to improve?
Step 4 (if yes): What is absent? Chosen dimension and rationale.
Step 5 (if yes): Rubric criteria for the constraint.
</analysis>
<decision>yes or no</decision>
<constraint>
The single constraint to append (required only when decision is yes).
</constraint>
<rubric>
    <criterion>Criterion question 1</criterion>
    <importance>integer score between 0 and 100</importance>
    <criterion>Criterion question 2 (optional)</criterion>
    <importance>integer score between 0 and 100</importance>
</rubric>

12.2.3 Policy-adaptive rubric generation↩︎

You are evaluating an instruction, two model responses, and an existing rubric. Your task is to determine whether the existing rubric fails to discern the two responses -- catching none of the observable failures -- and therefore needs an additional criterion. The default is NO; say YES only when the evidence clearly shows the rubric cannot discern the responses.

---

# Step 1 - Spot concrete, observed failures

For each response, list specific, instruction-relevant **failures** -- missing required elements, internal contradictions, malformed structure, misinterpretations, obvious quality defects -- that you can verify directly from the text. Tag which response(s) exhibit each.

**Self-verifiability (hard constraint).** Only count failures confirmable from the response text and the instruction alone -- not anything requiring external knowledge, factual lookup, or domain expertise (e.g., "the cited paper does not exist", "the historical date is wrong"). If unsure, drop.

If neither response has a self-verifiable failure, the decision will be NO.

# Step 2 - Compare against the existing rubric

**Existing gradient**: Do one or more existing criteria already produce a different answer on response 1 vs response 2? If so, the rubric is already discriminating -- there is already signal to learn from.
-> `<decision>no</decision>`

**Any covered failure**: For each failure from Step 1, ask whether an existing criterion -- even approximately -- would catch it. If **at least one** is already covered, the rubric is doing meaningful work. Leave it alone.
-> `<decision>no</decision>`

**No coverage at all**: Does the rubric fail to catch a single observed failure, and produce no differentiating signal between the two responses? Only this case warrants a new criterion.

# Step 3 - Make the decision

Say YES only when ALL are true:
- At least one concrete, self-verifiable failure identified.
- **None** of the failures is caught -- even approximately -- by any existing criterion.
- **No existing criterion** produces a different answer between the two responses.
- The uncaught failure(s) represent a class of problem any plausible response could exhibit -- not a quirk of these samples.

Say NO if ANY is true (when in doubt, say NO):
- No self-verifiable failures found.
- At least one failure is already caught (or nearly caught) by an existing criterion.
- Existing rubric already produces a different answer on at least one criterion.
- Failure requires external knowledge, or is specific to these samples rather than a class of problem.
- Uncertain whether any criterion catches any failure.

# Step 4 - Write new criteria (only if decision is yes)

## 4a. Select targets

From the uncovered failures, select the most load-bearing class(es). Target **at most 2 criteria**; use 1 if sufficient.

## 4b. Generalize each failure into a criterion

For each selected failure, write **one criterion question** that detects this *class* of problem in **any** response to this instruction.

Each criterion must satisfy ALL of these:

- **Grounded.** Cites a specific failure from Step 1. If you cannot cite one, drop it.
- **Substantive, not surface.** Targets a meaningful quality dimension (content, reasoning, coverage, depth, accuracy) -- not formatting, typography, specific word choice, or length counts. Ask what underlying quality the failure manifests and write around that. Do **not** write criteria like "Does the response use bullet points?", "Does the response include 'therefore'?", or "Is the response at least 500 words?" unless the instruction demands such form.
- **Generalized.** Does not reference entities or phrases appearing **only** in the sample responses. *Exception:* items the instruction itself enumerates (named characters, required sections, listed keywords) may be referenced by name.
- **Self-contained.** Evaluable from the instruction and a single response alone.
- **Atomic.** Judges exactly one thing.
- **Reachable.** A competent, good-faith response could reasonably satisfy it. No extreme demands ("lists 50 examples", "cites 10 peer-reviewed sources") unless the instruction requires them.
- **Natural to the task type.** Any structural or formal demand must appear in the instruction or be a natural expectation of the task type (a story has a narrative arc; a financial report shows numbers). Do **not** impose structure the instruction does not require and the task does not naturally imply.
- **Non-trivial.** Could plausibly separate a good response from a mediocre one. "Any reasonable attempt passes" is too weak -- avoid vague checks like "addresses the topic" or "is in English" that any competent response would pass.
- **Self-verifiable.** A judge reading only the instruction and one response can score it without external knowledge.
- **Complete question.** E.g., "Does the response include a clearly labeled summary section?"
- **Positive polarity.** Higher score clearly means a better response. Avoid "A or B" listings (bad: *"...list only beginner exercises, or include advanced variations?"*; good: *"...restrict its list to beginner exercises?"*). When the instruction forbids X, phrase around successfully avoiding X (bad: *"Does the response use passive voice?"*; good: *"Does the response avoid passive voice throughout?"*).

## 4c. Assign importance (0-100)

**100** critical to validity * **75** critical to quality, not explicit in instruction * **50** expected of any good response * **25** preference * **<25** soft nice-to-have.

---

# Input

## Instruction

<instruction>
{{instruction}}
</instruction>

## Existing Rubric

<existing_rubric>
{{existing_rubric}}
</existing_rubric>

## Model Responses

<response_1>
{{response_1}}
</response_1>

<response_2>
{{response_2}}
</response_2>

---

# Output Format

<analysis>
Step 1: Self-verifiable failures, tagged with which response(s) exhibit them.
Step 2: For each failure, whether an existing criterion covers it; whether any criterion already discriminates.
Step 3: Decision -- is it difficult to discern the responses using the existing rubric?
Step 4 (if yes): For each new criterion, name the underlying failure and confirm it is substantive, natural to the task type, non-trivial, reachable, self-verifiable, and generalized.
</analysis>
<decision>yes or no</decision>
<rubric>
    <criterion>Criterion question 1</criterion>
    <importance>integer score between 0 and 100</importance>
    <criterion>Criterion question 2 (optional)</criterion>
    <importance>integer score between 0 and 100</importance>
</rubric>

12.2.4 Rewrite variant of LLM-as-a-Tutor↩︎

You are evaluating an instruction and two model responses. Your task is to determine whether the model has completely mastered this instruction -- leaving no room for quality improvement -- and therefore needs more challenging instruction to create a meaningful learning signal. If needed, you will REWRITE the original instruction so that the new instruction is asking for more difficult tasks.

The default answer is NO. Only say YES when the evidence clearly shows the model has hit its quality ceiling on this instruction.

---

# Step 1 -- Evaluate each response independently

For each response, assess:

**Accuracy**: Is the information correct? Are there any factual errors, hallucinations, unsupported claims, or uncertain guesses?

**Completeness**: Does it address everything the instruction asks for? Is anything missing or treated superficially?

**Depth**: Does it go beyond surface-level treatment? Would a knowledgeable reader find it thorough, or would they want more?

**Execution quality**: Is it well-organized, clear, and appropriate in tone for the task?

Summarize the quality of each response. Be honest and critical -- look for genuine weaknesses, not just things you could add.

# Step 2 -- Compare the two responses

Now compare them:

**Quality gap**: Is one response noticeably better than the other? If so, the model produces variable quality on this instruction -- there is already a gradient to learn from. -> `<decision>no</decision>`

**Approach divergence**: Do the responses take meaningfully different approaches, structures, or perspectives? If so, the model is still exploring how best to handle this instruction -- variance already exists. -> `<decision>no</decision>`

**Content convergence**: Do both responses arrive at essentially the same answer with the same structure and coverage? If so, this is a sign the model has settled on one approach and produces it consistently.

# Step 3 -- Make the decision

Say YES only when ALL of the following are true:
- Both responses are **high quality** -- accurate, complete, well-executed. Neither has meaningful weaknesses.
- Both responses **converge** -- similar approach, structure, and content. They are nearly interchangeable.
- There is **no gradient** -- you cannot meaningfully say one is better than the other, because both are equally strong.

If ANY of the following is true, say NO:
- Either response has factual errors, hallucinations, or incorrect information.
- Either response is superficial, generic, or lacks depth on a topic that warrants it.
- Either response misses something the instruction asked for.
- One response is clearly better than the other.
- The responses take substantially different approaches or cover different ground.
- Either response shows uncertainty (hedging, listing multiple possibilities, "it could be...").

When in doubt, say NO.

# Step 4 -- Identify the dimension to push (only if decision is yes)

## 4a. Pick the dimension

Both responses do everything well -- along which dimension is the task being satisfied too easily? Target the dimension where asking for more would force the model past its current ceiling.

## 4b. Choose a dimension

Select ONE:
- **Content**: Tighten scope, deepen coverage, or demand more rigorous subject matter.
- **Situation**: Constrain the scenario, condition, edge case, or role.
- **Style**: Demand a specific tone, voice, or persona.
- **Format**: Demand a specific structure, length limit, or presentation style.

The harder version must be:
- **Challenging**: Pushes the dimension past what either response attempted.
- **Specific**: Concrete and evaluable.
- **Factually grounded**: Only references things present in the instruction or responses.

# Step 5 -- Rewrite the instruction (only if decision is yes)

Rewrite the instruction over from scratch, asking for a harder version of the same task pushed along the dimension you chose in Step 4.

If decision is no, leave the `<rewrite>` block empty.

---

# Input

## Instruction

<instruction>
{instruction}
</instruction>

## Model Responses

<response_1>
{response_1}
</response_1>

<response_2>
{response_2}
</response_2>

---

# Output Format

<analysis>
Step 1: Quality assessment of each response -- strengths and genuine weaknesses.
Step 2: Comparison -- quality gap? approach divergence? content convergence?
Step 3: Decision -- has the model fully mastered this instruction with no room to improve?
Step 4 (if yes): Which dimension is being satisfied too easily? Chosen dimension and how the rewrite will push it.
Step 5 (if yes): Brief note on the rewrite approach.
</analysis>
<decision>yes or no</decision>
<rewrite>
The full rewritten instruction (required only when decision is yes). It is the same task asked in a harder form along the chosen dimension, written from scratch as one coherent prompt.
</rewrite>

13 Additional qualitative examples↩︎

References↩︎