1 Introduction↩︎

Large language models (LLMs) can be used in two complementary modes: as generators that produce candidate responses, and as validators that assess response correctness or felicity. These two modes are crucial for usages of LLMs such as self-refinement [1] and backtracking during chain-of-thought [2]. However, these two modes exhibit divergent behavior, reflecting an underlying inconsistency: even frontier LLMs may generate responses with high probability but judge them to be incorrect, or vice-versa. Accordingly, past work has examined training LMs to be explicitly generator-validator (G-V) consistent in the hope that this will also make them more accurate [3]–[5]. But formalizing this turns out to be surprisingly difficult. Past approaches close the gap on G-V correlation [5], but these approaches risk contributing to pathological behaviors like suppression of correct responses or contributing to further inconsistency through contradictory training signals.

This work aims to improve G-V consistency through two contributions. First, we axiomatize G-V consistency based on a model of a rational probabilistic agent responding to prompts. Although LLMs are not rational agents, this model allows us to examine what the relationship between a generator and a validator should be in cases where the generator may prefer some possible responses over others, but a validator finds them all to be likely. We derive a theoretical relationship between generator and validator probabilities, which implies adjusting generator scores by subtracting off incorrect probabilities, or how likely a response is to be sampled when an incorrect response is requested. Since this value reflects the base likelihood of the response, we call the final quantity the frequency-corrected generator score.

Second, we use this relationship in a training objective called Frequency-corrected Learning of Ordered Rank Alignment (FLORA). This objective encourages rank-alignment of validator scores with frequency-corrected generator scores, representing an approximation of our theoretical G-V consistency for rational agents. We fine-tune LLMs using this objective function in combination with standard losses to ensure generator and validator correctness.

Our experiments focus on tasks where validators outperform generators. Our goal is to observe that the generator improves without degradation of the validator. We particularly focus on generator AUROC, or ensuring that the frequency-corrected generator distribution correctly ranks the set of positive responses above the set of negative responses, even down into the tail of the distribution, unlike past work which focuses on the head [4]. This is a strong check of consistency and of generator correctness. We evaluate on three tasks: instruction-following, coding, and eliciting taxonomic knowledge. Two of these tasks are long-form (several sentences, for IFEval, or Python code), unlike previous work which focused on short answers of one or a couple words [4], [5] Across these tasks and three LLMs, we find that our method closes the G-V gap more strongly than previous work, and leads to improved discriminability of correct and incorrect completions. Concretely, FLORA improves generator AUROC by up to \(+7.3\)pp and generator-validator correlation by up to \(+27\)pp over the strongest prior alignment method, while preserving or improving validator quality.

2 Background and Motivation↩︎

The extent to which LLMs have consistent internal processing is an important scientific question [6], and the mismatch between generation and validation is an important kind of inconsistency to measure and repair. Much of what a model “knows” never shows up in its sampled outputs, because it lives in discriminative judgments rather than in fluent continuations [7]; this is problematic for evaluations targeting model knowledge [8], [9]. Better aligning generators with an LLM’s own validation capability is also important for reward modeling [10] and reranking applications such as mathematical reasoning [11], [12], code generation [13], and factual question answering.

[5] assert that the log-odds of the generator and validator should correlate. However, there are cases where this relationship cannot hold, arising from two main sources: (a) the role of response frequency and (b) aleatoric uncertainty (multiple correct responses).

2.0.0.1 Motivating Example

A well-known issue with using LLMs to generate responses is surface form ambiguity [14]. Given the question “What do you call it when blood flow to the heart is suddenly blocked?”, heart attack and myocardial infarction are both correct answers. A good validator should score them both highly, yet it would be odd to expect a generator to score both highly, since heart attack is the more common expression.

This becomes more complex when questions have multiple possible correct answers beyond paraphrases. Consider a question that admits several valid answers, e.g., “What’s an example of a noble gas?” (Figure 1). There are 7 possible correct answers to this question, plus potential different surface forms of elements (e.g., “Helium” vs.  “He”). It is unlikely that a generator will actually generate Helium and Oganesson with equal probability when given this question, despite both being correct. Generator scores conflate the correctness of an utterance in response to a prompt with the frequency of that utterance.

There is no single “perfect” generator in this setting. One view is that a model should tend towards a uniform distribution over possible options [15], while another view is that the model’s distribution over valid answers should match the estimated corpus-frequency distribution [16]. A perfect validator, in contrast, should certainly say “Yes” with probability near 1 to each correct option and “No” to incorrect options.

Figure 2 shows this effect in practice on a long-form generation task from IFEval. The validator is imperfect, but most correct responses (green) outscore most incorrect responses (red) on the validator log-odds (y-axis). However, there are incorrect responses that have much higher generator score than correct responses, partially due to being short and generic and sticking close to the prompt.

This example shows that generator and validator scores should not generally be expected to match, or even to correlate linearly, without controlling for competition and frequency effects. Next we derive a relationship between them which accounts for both frequency effects and aleatoric uncertainty.

3 Problem Formulation↩︎

3.0.0.1 Problem Setup

We assume a prompt \(x\) and a set of possible responses \(\mathcal{Y}\).² For each \(y \in \mathcal{Y}\), we assume that there is a validity label \(v(x, y) \in \{0, 1\}\). We assume that validity is binary and unambiguous. We consider an agent, which is an LLM for the purposes of this work, operating in two modes: a generator \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}\), and a validator \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(v {=} 1 \mid x, y)}\). For an LLM, \(p_V\) and \(p_G\) are implemented with different prompts.

3.0.0.2 Properties of Consistency

We would like to choose \(p_V\) and \(p_G\) to be consistent, under the intuition that improving consistency will also improve accuracy. We can straightforwardly understand \(p_V\) as the LM’s current “belief” that \(y\) is a correct response to \(x\). However, since there are in general many correct responses, \(p_G\) additionally has to model competition between these to act as a generator.

We define \(\mathbf{v} = \{0,1\}^{|\mathcal{Y}|}\) as a vector of validities associated with each possible response string, assuming that every possible response is either correct or incorrect, even though the agent may be uncertain as to which. Let \(\mathbf{v}_y \in \{0,1\}\) denote the validity assigned to a particular string \(y\). In the context of Figure 1, a correct \(\mathbf{v}\) should assign 1 to the 7 noble gases and all ways of writing them down and 0 to all other strings. Figure 3 shows possible \(\mathbf{v}\) values in green.

Assume that for a prompt \(x\), we have a known and fixed \(\mathbf{v}\). We define an agent to be consistent with its beliefs \(\mathbf{v}\) if (1) its validator returns exactly the responses in \(\mathbf{v}\); (2) its generator only places mass on valid responses: \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}> 0\) iff \(\mathbf{v}_y = 1\). We call this a support constraint motivated by the maxim of quality [17]: conditional on its latent belief state \(\mathbf{v}\), the agent never generates a response it believes to be incorrect. However, some options (e.g., xenon and krypton) may be assigned very low probability.

In practice, agents have uncertainty over \(\mathbf{v}\). We model this as a distribution \(p(\mathbf{v} \mid x) > 0\) reflecting the agent’s belief over possible response sets given \(x\). Like in real LLMs, we assume that every \(\mathbf{v}\) has nonzero (but possibly tiny) probability mass. (Note that this distribution is a theoretical construct and cannot be materialized in practice.) We can now define consistency with respect to this distribution \(p(\mathbf{v} \mid x)\). A validator is consistent if it satisfies the following relationship: \[\label{eq:v-definition} \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}:= p(v_y = 1 \mid x) = \sum_{\mathbf{v} : v_y = 1} p(\mathbf{v} \mid x)\tag{1}\]

We represent generation through the use of a distribution \(\pi(y \mid \mathbf{v}, x)\) that conditions on \(\mathbf{v}\). \(\pi\), although intractable to fully represent in practice, effectively describes how mass should be distributed among the correct options of \(\mathbf{v}\). To obey the support constraint, \(\pi\) only assigns nonzero probability to items \(i\) where \(\mathbf{v}_i = 1\). But within these items, \(\pi\) governs which surface form to use for a given meaning, splitting probability over surface form realizations [14], [15], [18], and which response to express when multiple are deemed correct. We define the generator as a mixture model over the \(\mathbf{v}\): \[\label{eq:g-definition} \begin{align} \textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}&:= \sum_{\mathbf{v}} p(\mathbf{v} \mid x) \, \pi(y \mid \mathbf{v}, x) \\ &\phantom{:}= \sum_{\mathbf{v}:v_y=1} p(\mathbf{v} \mid x) \, \pi(y \mid \mathbf{v}, x) \end{align}\tag{2}\] This process is depicted in Figure 3.

3.1 Deriving a G-V relationship↩︎

We can now relate the generator and the validator we have defined so far.

We first define one more quantity \(\pi'(y \mid \mathbf{v}, x)\). This is a policy for selecting a response when asked for an incorrect one, with the related constraint that it never generates an answer it believes to be correct, i.e., \(\pi'(y \mid \mathbf{v}, x) = 0\) when \(v_a = 1\). From that, we can define \[\label{eq:g-prime-definition} \begin{align} \textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}&:= \sum_{\mathbf{v}} P(\mathbf{v} \mid x) \, \pi'(y \mid \mathbf{v}, x) \\ &\phantom{:}= \sum_{\mathbf{v}:v_y=0} p(\mathbf{v} \mid x) \, \pi'(y \mid \mathbf{v}, x) \end{align}\tag{3}\]

3.1.0.1 Proof Sketch

We give intuition for the proof and a full proof in Appendix 9.

The conditional expectation of the policy for picking a correct response from among the set of correct answers given that \(a\) is valid is:

\[\label{eq:pi-expectation} \begin{align} &\mathbb{E}\big[\pi(y \mid \mathbf{v}, x) \mid v_y=1, x \big] \\ &\quad = \frac{\sum_{\mathbf{v}: v_y = 1} P(\mathbf{v} \mid x)\, \pi(y \mid \mathbf{v}, x)}{P(v_y = 1 \mid x)} \\ &\quad = \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}}{\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}} \end{align}\tag{4}\]

Similarly, \[\label{eq:pi-prime-expectation} \begin{align} &\mathbb{E}\big[\pi'(y \mid \mathbf{v}, x) \mid v_y=0, x \big] \\ &\quad = \frac{\sum_{\mathbf{v}: v_y = 0} p(\mathbf{v} \mid x)\, \pi'(y \mid \mathbf{v}, x)}{P(v_y = 0 \mid x)} \\ &\quad = \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}}{1- \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}} \end{align}\tag{5}\]

Since \(0<\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}<1\) the main result follows from Eqn 4 and 5 .

3.1.0.2 Intuition

First, we note that \(0<\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}<1\) is a mild condition in practice. We might reasonably expect that an agent backed by a Transformer produces a distribution \(p(\mathbf{v} \mid x)\) with support everywhere due to the nature of softmax; no set of answers is ruled out structurally. Our needed condition follows from this.

In the limit of zero epistemic uncertainty, these probabilities become very small and \(p\) places all probability mass on one configuration \(\mathbf{v}^{\star} \in \{0,1\}^{|A|}\). In this case, the validator probability \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}\) will approach 0 or 1, and the generator probabilities become the policies at configuration \(\mathbf{v}^{\star}\): \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}= \pi(y \mid \mathbf{v}^{\star}, x)\) and \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}= \pi'(y \mid \mathbf{v}^{\star}, x)\).

Finally, the value of \(r(y,x)\) is very important to reasoning about the generator-validator relationship. The numerator of \(r\) is the expected value of the probability of a response being generated as a incorrect response conditioned on it being marked as incorrect in \(\mathbf{v}\). This involves marginalizing over all possible configurations of \(\mathbf{v}\) where \(v_y = 0\). We can think of this as saying: in aggregate, what fraction of the \(\pi'\) mass is assigned to \(y\) when it’s a valid option? Intuitively, this is related to how frequent we are to say \(y\): a string that’s simply more frequent will be uttered with higher probability in a larger number of contexts. The denominator of \(r\) is similar, but for the case of a correct response.

3.2 Mapping Rational Agent Behavior to LLM Behavior↩︎

LLMs are not rational agents of the sort in the previous section. However, we can nevertheless seek to impose the regularity we derived in Equation ?? to their behavior if we can approximate \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}}\), \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}}\), and \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}}\). To do so, \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}\), \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}\) and \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}\) can be elicited from an LLM as follows. Given template function \(T_V: x,y \rightarrow \text{prompt}_V(x,y)\) mapping questions and answers to prompts, and template functions \(T_G: x\rightarrow \text{prompt}_G(x)\) and \(T'_G: x\rightarrow \text{prompt}_G'(x)\), asking for correct and incorrect answers to a given question, respectively: \[\begin{align} \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}&\approx P_{\mathrm{LLM}}\bigl(\text{Yes} \mid T_V(x,y)\bigr), \\ \textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}&\approx P_{\mathrm{LLM}}\bigl(y \mid T_G(x)\bigr), \\ \textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}&\approx P_{\mathrm{LLM}}\bigl(y \mid T'_G(x)\bigr). \end{align}\]

There are two sources of error in these approximations. First, LLMs do not have an internally consistent belief \(p(\mathbf{v} \mid x)\) about the answers to a problem. Second, even if they did, these prompts do not necessarily elicit rational responses to these.

Nevertheless, we argue that these approximations are useful. Empirically, most of the mass for LLMs when prompted for Yes/No answers lies on the Yes and No tokens, so at least LLMs can follow these instructions.³. \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}\) follows closely from how LLMs are trained to answer questions or follow instructions. On the other hand, \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}\) is likely less well-approximated, since asking for incorrect responses is not where post-training focuses its efforts; the fidelity of this proxy is an empirical question, especially for smaller or non-instruction-tuned models.

Letting \(s_V(y \mid x) := \log \frac{\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}{1-\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}\) be the validator log-odds, we then have \[\begin{align} s_V(y \mid x) = {}& \log \textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}- \log \textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}\\ & {} + \log r(y \mid x) \end{align} \label{eq:main-2}\tag{6}\]

Unfortunately, \(\log r(y \mid x)\) depends on conditional expectations over quantities that we cannot directly observe. Intuitively, what this term measures is the ratio of the probability of generating \(y\) when prompted for a wrong response (and the model thinks \(y\) is incorrect) and the probability of generating \(y\) when prompted for a correct response (and the model thinks \(y\) is correct). Assuming that \(\pi\) and \(\pi'\) are both capturing the frequency of \(y\), and this frequency is not biased by response correctness, this ratio can be approximated as 1.

As a result, disregarding \(r\), Eqn 6 motivates the following adjusted generator score: \[s_{\text{Adj}}(y \mid x) \;:=\; \log \textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}- \log \textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}. \label{eq:adj-score}\tag{7}\] By construction, \(s_{\text{Adj}}\) tracks the validator log-odds \(s_V\) up to the residual \(\log r\), and so should correlate better with \(s_V\) than the raw generator score \(\log p_G\) does. In the next section, we operationalize \(s_{\text{Adj}}\) as a training objective for our model.

4 Training for Frequency Correction↩︎

4.0.0.1 Data Condition

We consider a training setting where we have a dataset \(\mathcal{D} = \{(x_i,\{(y_{ij},v_{ij})\})\}\). That is, each \(x_i\) is paired with a set of outputs \(\{(y_{ij},v_{ij})\}\) where the \(y_{ij}\) are the outputs themselves and the \(v_{ij}\) are correctness labels. We operate in a semi-supervised setting where \(v_{ij}\) may not be available for all \((x_i,y_{ij})\) pairs.

As our training involves imposing consistency, we operate over pairs of points (e.g., similar to hydrogen and Rn in Figure 1). We define a labeled set \(\mathcal{T} = \{((x_i,y_i,v_i),(x_j,y_j,v_j))\}\) of pairs with veracity labels. We define an unlabeled set \(\mathcal{U} = \{((x_i,y_i),(x_j,y_j))\}\) of pairs without labels.

Furthermore, throughout this work we require \(x_i = x_j\); we sample points within the same prompt. We also require \(|p_V(y_j \mid x_j) - p_V(y_i \mid x_i)| > \delta\) to avoid noise from too-close validator pairs, following past work [5], [7].

4.0.0.2 Training Objective

We use a loss designed to do two things: (1) teach the generator to rank pairs in the same way as the validator, and (2) use the labeled examples to improve the discriminability of generator and validator scores. We train using the following loss, where we let \(y_j = y^+\) and \(y_i = y^-\) denote the higher- and lower-ranked completions under the validator: \[\begin{align} \mathcal{L} =& \mathcal{L}_{\text{pref}}(x_i,y_i,x_j,y_j) + \notag \\ &\lambda_G (v_i \mathcal{L}_\text{NLL}(y_i \mid x_i) + v_j \mathcal{L}_\text{NLL}(y_j \mid x_j)) + \notag \\ & \lambda_V (\mathcal{L}_\text{NLL}(v_i \mid x_i, y_i) + \mathcal{L}_\text{NLL}(v_j \mid x_j, y_j)) \label{eq:loss} \end{align}\tag{8}\]

The \(\mathcal{L}_{\text{pref}}\) loss is always active, while \(\mathcal{L}_\text{NLL}\) terms are only active for labeled examples. The generator terms only apply to positive examples (i.e., \(v_i = 1\) or \(v_j = 1\)) and the validator terms are only present for labeled examples, which have ground truth Yes/No completions.

4.0.0.3 Preference Loss

We use a pairwise logistic preference loss given by \[\small \mathcal{L}_{\text{pref}}(x, y^-, y^+) = -\log \sigma\!\left( s(y^+ | x) - s(y^- | x) \right)\] where \(\sigma\) is the logistic sigmoid and \(s_G(y \mid x)\) is the generator’s log-likelihood of completion \(y\) given prompt \(x\), \[s(y \mid x) = \sum_{t=1}^{|y|} \log P_{\text{LLM}}(y_t \mid x, y_{<t}).\]

To account for completion frequency, we consider two empirical estimators of the adjusted score \(s_{\text{Adj}}\) from Eqn 7 (§3.1). The first, NegFC, directly substitutes a negative-prompt elicitation for \(p_{G'}\): \[s_{\text{Neg}}(y \mid x) = s_G(y \mid x) - \log P_{\text{LLM}}(y \mid T'_G(x)).\] The second, Unconditional FC, uses the unconditional log probability [19] as a cheaper proxy, \[s_{\text{PMI}}(y \mid x) = s_G(y \mid x) - \log P_{\text{LLM}}(y),\] which captures a similar frequency-penalization effect without requiring an elicitation prompt \(T'_G\) for incorrect answers.

4.0.0.4 Sampling strategy

Finally, we ensure that the preference losses are consistent with the labels. For any pair \((x,y^-), (x,y^+)\) where \(s_V(y^+ \mid x) > s_V(y^- | x) + \delta\), the preference loss will push the generator score of \((x,y^-)\) down and \((x,y^+)\) up; this is incorrect if \(y^-\) is a positive example or \(y^+\) is a negative example. In addition, if \(y^-\) is positive the preference loss and generator \(\mathcal{L}_\text{NLL}\) will compete in opposite directions. To avoid these problems, we only keep a labeled \((x,y^-)\) when it is negative, and similarly we only keep a labeled \((x,y^+)\) when it is positive.

5 Experimental Setup↩︎

5.1 Tasks and Datasets↩︎

We evaluate on three datasets which we adapted from previous work to test our method on instruction following, coding, and conceptual knowledge. We use IFEval [20] for instruction following, HumanEval [13] for Python coding, and a mix of datasets to probe for knowledge of taxonomic category relations (hyponymy) [21]–[26]. Each dataset has a different notion of correctness, detailed below. These datasets are described here, with further details on datasets given in Appendix 11. Prompt templates for each of these tasks are shown in Appendix 12.

5.1.0.1 Instruction Following

This task uses prompts from IFEval [20], e.g., “Write a poem that’s at least 350 words about the beauty of eucalyptus trees and their many uses.” which specify explicit content and format constraints. A correct response should follow all constraints in the prompt, while an incorrect response violates one or more. We generate, using GPT-4o mini, correct responses via paraphrases of the original prompt and incorrect responses by introducing violations (e.g., missing required elements or breaking constraints). Labels are assigned using rule-based checks for verifiable constraints together with LLM-as-a-judge evaluations for content correctness where deterministic verification is not possible. We use 79 prompts for the training set and 20 for the test set.

5.1.0.2 Coding

This task uses prompts from the HumanEval [13] benchmark of Python programming exercises. Each prompt asks to complete a Python function to solve a given problem. Here correct responses are those which pass all the unit tests, while incorrect ones fail one or more. We use various LLMs to generate correct and incorrect responses. To stress-test our method with examples that are correct but unlikely, we systematically convert the variables of all the correct solutions to uppercase.

5.1.0.3 Hyponymy

This task consists of giving exemplars to categories, i.e., answering “An example of a X is a __” for various nouns X. We evaluated on the ten categories used in the original experiments of [21], supplemented with GPT-5-generated incorrect completions in order to obtain a balanced test set. We decided on a cutoff between items that are members of the category while those that are not. While there is some subjectivity in this decision, our cutoff is validated by the fact that LLMs with validator prompts can successfully distinguish between the positive and negative classes (Table 5).

For all three tasks, we evaluate on a held-out set of prompts. For IFEval and HumanEval, we randomly split the data. For Hyponymy we train on a different set of categories than the ten in [21]. Further details of the dataset composition and construction process is given in Appendix 11.

5.2 Target Models for G-V Alignment↩︎

As we defined the dataset at the beginning of Section 4, our target models for G-V consistency do not necessarily have to be those from which responses were generated. In fact, having a range of responses (as we generated for each dataset) that we require G-V consistency on allows us to stress-test each model beyond the mode of the distribution that it generates.

We use a range of models including those in the Gemma family, namely Gemma-2-9b-it (G2-9b-it) and Gemma-4-31b-it (G4-31b-it), and Qwen family, namely Qwen-3.5-9B (Q3.5-9b). For each task, we select models in an appropriate performance range: for instance, on IFEval, we only use larger instruction-tuned models because smaller models are generally not capable enough at the task. For each task and model combination, we only proceed with models where validator AUROC is over 65%); this eliminates Gemma-2-9b-it from the HumanEval experiments.

5.3 Evaluation↩︎

For all methods, we evaluate both using the raw generator scores \(s_G(y \mid x) = \log P_{LLM}(y \mid x)\) as well as the two corrected forms: frequency-corrected scores \(s_{\text{PMI}}= s_G(y \mid x) - \log P_{LLM}(y)\), and negative prompting-corrected scores \(s_{\text{Neg}}= s_G(y \mid x) - s_G'(y\mid x)\). All metrics are computed independently for each prompt, and averaged over prompts.

5.3.0.1 Accuracy

We evaluate the validator and generator performance at discriminating correct and incorrect answers through AUROC scores of the associated classifiers: \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}> \tau_G\) and \(s_V(y \mid x) > \tau_V\) for thresholds \(\tau_G\) and \(\tau_V\). Each defines an ROC curve and we refer to the corresponding AUROC scores as \(\mathbf{ROC_G}\) and \(\mathbf{ROC_V}\). For the generator, we use one of several different possible generator scores: \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}}\), \(s_{\text{PMI}}\) or \(s_{\text{Neg}}\). We additionally measure the validator accuracy at \(\tau_V =0\) in order to measure how well-calibrated the validator is. We focus primarily on \(\mathbf{ROC_G}\); our focus in this work is on tasks where the validator is stronger than the generator and can be used to improve the generator’s performance.

5.3.0.2 Correlations

Following [5], we measure the Pearson correlation \(\rho\) between the validator and (possibly-corrected) generator scores over the full set of candidate answers.

6pt

5.4 Baselines↩︎

We compare our method against the following baselines in addition to the Base (untrained) model:

5.4.0.1 SFT

We sample pairs of completions as in RankAlign, but use a Supervised Fine-Tuning (SFT) loss, i.e., only the negative log-likelihood terms from Eqn. 8 .

5.4.0.2 Consistency FT

[4] SFT only on the subset of examples where generator and validator agree. We use the version from [5], which measures agreement via \(\mathbb{1}_{l_G(z, y_A) > t_G} = \mathbb{1}_{l_V(z, y_A) > t_V}\) where \(t_G\) and \(t_V\) are thresholds set as the average generator and validator scores over the dataset.

5.4.0.3 RankAlign

We use the preference loss of RankAlign [5]. Compared to our method this (1) does not apply any frequency correction during training; (2) does not enforce sampled pairs to have the same prompt during training, (3) does not use NLL terms.

6 Results↩︎

6.1 Frequency Correction Helps at Test Time↩︎

Frequency correction improves generator–validator consistency even before training.

We first ask whether frequency correction, applied at test time, improves the generator’s ability to discriminate correct from incorrect answers and its consistency with the model’s own validator. Table 1 compares the raw generator score \(s_G\) with the two corrected variants introduced in Section 4: NegTC (\(s_{\text{Neg}}\)), which subtracts the log probability when using the negative prompt, and Unconditional FC (\(s_{\text{PMI}}\)), which subtracts the unconditional log probability of the completion.

Both corrections improve ROC\(_G\) and \(\rho\) over the raw generator score on all tasks. This indicates that some amount of apparent G-V gap goes away when using these corrected terms, indicating that they form a better starting point for improving G-V consistency and generator discriminability.

6.2 Main Results↩︎

FLORA outperforms prior consistency alignment methods on both ROC\(_G\) and \(\rho\).

Table 2 compares methods on ROC\(_G\) and \(\rho\) for IFEval, HumanEval, and Hyponymy datasets. All methods are evaluated with the per-task best eval-time correction identified in Section 6.1. Full results with standard errors are given in Table 4 in Appendix 10.

4pt

FLORA achieves a higher \(\text{ROC}_G\) than the other baselines in five out of six settings (tying RankAlign on Hyponymy with Qwen-3.5-9B), with gains ranging from +0.9 to +7.3pp over the next-best method. G-V consistency improves by a wider margin, with \(\rho\) +27pp over the next-best method on both IFEval with Gemma-2-9b-it (FLORA-PMI: 62.8 vs. 35.5 for Base) and HumanEval (FLORA-Neg: 76.3 vs. 49.0 for Consistency FT). On the two Hyponymy settings, FLORA is competitive on both metrics, trailing RankAlign by at most 1.4pp on \(\rho\).

6.2.0.1 FLORA Preserves Validator Quality

Improvements in G-V consistency do not come at the cost of validator quality.

A standard concern with preference-style training is likelihood displacement [27], [28]: the model may appear better aligned only because its validator has degraded. To rule this out, we measure the validator accuracy and AUROC. Table 5 shows the validator accuracies per-method (\(\text{ROC}_V\) and \(\text{Acc}_V\)).

6.2.0.2 The effect of frequency correction

Table 3 ablates the frequency correction term applied during training. Here the “w/o typ.corr.” variant trains on the raw generator log-likelihood. Both variants apply the same eval-time correction, so the comparison isolates the effect of frequency correction during training. Removing it hurts both ROC\(_G\) and \(\rho\) on every task, with the largest drops on HumanEval (\(-6.3\)pp ROC\(_G\), \(-16.1\)pp \(\rho\)).

6pt

6.3 Qualitative Analysis of Alignment↩︎

Across many prompts, we observe cleaner consistency between generator scores and validator scores after applying our method. Figure 5 illustrates a case of this for a prompt in IFEval and a set of possible completions. X-axes in the left panel are the raw generator scores, while x-axes in the right two panel show the corrected generator scores \(s_{\mathrm{PMI}}\). For the base model (left), the relationship between generator scores and validator scores is noisy: some high-likelihood generations receive low validator scores and vice versa, indicating a misalignment between what the generator prefers and what the validator deems correct. Using frequency correction for the generator scores at test time improves both generator-validator correlation and generator ROC (middle panel). After training, (right panel) the relationship between generator and validator scores becomes even more structured, with a clearer trend in which higher generator scores correspond to higher validator scores.

7 Related Work↩︎

7.0.0.1 Generator-Validator Gap.

Generator-validator gaps have been discussed in a number of contexts, including [3], [4], and [5]. Our work builds on these notions and is the first to derive a principled relationship between generator probabilities and validator log-odds, which [5] treated heuristically.

7.0.0.2 Preference Learning.

The ranking-based preference objective we use is similar to DPO [29]. In our case, the validator produces preferences for the generator, broadly related to self-rewarding LMs [10]. However, rather than a general post-training method, we view FLORA as a mechanism to achieve consistency in an LLMs’ predictions.

7.0.0.3 Frequency correction.

Past work has explored similar ideas in frequency correction. [14] correct for frequency in multiple-choice settings. Their domain-conditional PMI is similar to our PMI objective, but it is motivated as a heuristic. [30] uses the idea of “anti-experts” for controlled text generation, which gives rise to a similar form (subtracting log probabilities from the anti-expert). [31] then extend this idea to contrasting strong and weak language models. However, none of these approaches targets consistency per se.

7.0.0.4 Theoretical Formulation.

We are not the first to analyze LLM behavior by decomposing output distributions over a latent variable. [32] formulate in-context learning as posterior inference over a latent document concept, and [33] similarly adopt a latent variable formulation to unify in-context learning and activation steering. Our derivation in §3.1 uses a similar idea, but applies it to a different question: the discrepancy between a single model’s generator and validator distributions on a given prompt.

8 Conclusion↩︎

In this paper, we presented FLORA, a method for improving LLMs via validator-to-generator alignment. We derive a relationship between generator probabilities and validator log-odds based on a model of how a rational agent might generate a response. This relationship suggests a frequency-based correction factor. When trained to align validator and generator with this factor, we see improved results across taxonomic categorization (Hyponymy) and two long-form generation problems, IFEval and HumanEval.

Limitations↩︎

One limitation of this work is a gap between the theoretical motivation and practical implementation. We rely on an assumption that the intractable \(r\) term in Equation \(\ref{eq:main}\) is close to 1. Although we believe this is a reasonable assumption as discussed in the text, it is difficult to validate this as \(r\) is intractable to estimate even on relatively simple examples. Nevertheless, we see that our theoretically-derived training objective performs well empirically, lending credence to our method.

A second limitation is the focus on evaluating fixed sets of outputs. We focus on evaluating the tails of the model’s distribution: samples which are potentially low likelihood but which we still believe the model should rank correctly. We believe this setting is relevant, as LLM judges are frequently asked to reckon about fixed responses from other models, and various kinds of self-consistency or best-of-N approaches use a related kind of scoring. However, we do not demonstrate an impact on the 1-best answers produced by greedy decoding.

Furthermore, our framework assumes that correctness is a binary notion. Relaxing this to accommodate partially-correct answers would be an interesting direction for future.

Finally, we note that evaluating within-prompt rather than across a diverse set of prompts prevents us from evaluating directly on the datasets used to measure the G-V gap in [5], because most prompts in that work only have a small set of completions. To still enable a comparison, we re-implement RankAlign as a baseline and evaluate it on our datasets alongside FLORA (Tables 4, 5).

Acknowledgments↩︎

We thank Jacob Andreas for insightful discussion around this work. This work was supported by NSF CAREER Award IIS-2145280, NSF grant IIS-2433071, the NSF AI Institute for Foundations of Machine Learning (IFML), and the NSF under Cooperative Agreement 2421782 and the Simons Foundation grant MPS-AI-00010515 awarded to the NSF-Simons AI Institute for Cosmic Origins — CosmicAI, https://www.cosmicai.org/. We also thank members of the TAUR Lab for helpful feedback.

9 Proof of Theorem 1↩︎

9.0.0.1 Step 1: Rewrite \(p_G\) and \(p_{G'}\).

By definition, \[\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}= \sum_{\mathbf{v}} p(\mathbf{v} \mid x)\,\pi(y \mid \mathbf{v}, x).\] Split the sum into \(\{\mathbf{v} : v_y = 1\}\) and \(\{\mathbf{v} : v_y = 0\}\). By the support constraint on \(\pi\), every term in the second sum vanishes, so \[\label{eq:pg-restricted} \textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}= \sum_{\mathbf{v} : v_y = 1} p(\mathbf{v} \mid x)\,\pi(y \mid \mathbf{v}, x).\tag{9}\] By the same reasoning, \[\label{eq:pgprime-restricted} \textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}= \sum_{\mathbf{v} : v_y = 0} p(\mathbf{v} \mid x)\,\pi'(y \mid \mathbf{v}, x).\tag{10}\]

9.0.0.2 Step 2: Express with conditional expectation.

For any function \(f(\mathbf{v})\) and set \(A \subseteq \{0,1\}^{|\mathcal{Y}|}\) with \(P(\mathbf{v} \in A) > 0\), \[\mathbb{E}\!\left[f(\mathbf{v}) \mid \mathbf{v} \in A\right] = \frac{\sum_{\mathbf{v} \in A} p(\mathbf{v} \mid x)\,f(\mathbf{v})}{\sum_{\mathbf{v} \in A} p(\mathbf{v} \mid x)}.\] Applying this with \(f(\mathbf{v}) = \pi(y \mid \mathbf{v}, x)\) and \(A = \{\mathbf{v} : v_y = 1\}\), the fact that \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}> 0\) ensures that the denominator is greater than 0, as the denominator equals \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}\) by 1 . The numerator equals \(\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}\) by 9 . Hence \[\label{eq:pi-cond-exp} \mathbb{E}\!\left[\pi(y \mid \mathbf{v}, x) \mid v_y = 1, x\right] = \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}}{\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}.\tag{11}\] The analogous calculation with \(f(\mathbf{v}) = \pi'(y \mid \mathbf{v}, x)\) and \(A = \{\mathbf{v} : v_y = 0\}\), using \(\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}< 1\) to ensure \(1 - \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}> 0\), yields \[\label{eq:pi-prime-cond-exp} \mathbb{E}\!\left[\pi'(y \mid \mathbf{v}, x) \mid v_y = 0, x\right] = \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}}{1 - \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}.\tag{12}\]

9.0.0.3 Step 3: Take the ratio.

The denominators on the right-hand sides of 11 and 12 are strictly positive by the hypothesis \(0 < \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}< 1\). The numerators are nonnegative as sums of nonnegative terms. Dividing 12 by 11 , \[\frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}/ (1 - \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)})}{\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}/ \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}} = \frac{\mathbb{E}\!\left[\pi'(y \mid \mathbf{v}, x) \mid v_y = 0, x\right]}{\mathbb{E}\!\left[\pi(y \mid \mathbf{v}, x) \mid v_y = 1, x\right]}\] The left-hand side simplifies to \[\frac{\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}{1 - \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}} \cdot \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}}{\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}},\] and rearranging gives \[\frac{\textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}}{1 - \textcolor[RGB]{34,139,34}{\boldsymbol{p_V}(y \mid x)}} = \frac{\textcolor[RGB]{30,100,200}{\boldsymbol{p_G}(y \mid x)}}{\textcolor[RGB]{30,100,200}{\boldsymbol{p_{G'}}(y \mid x)}} \cdot r(y, x). \qedhere\]

10 Additional Results↩︎

Full results on generator ROC (ROC\(_G\)) and Pearson correlation (\(\rho\)) are given in Table 4. Each entry is the average score across prompts, while \(\pm\) indicates the standard error.

4pt

The full results on validator accuracy (Acc\(_V\)) and validator AUROC (ROC\(_V\)) are given in Table 5.

4pt

11 Dataset construction Details↩︎

11.1 IFEval↩︎

IFEval [20] is an instruction-following benchmark designed to evaluate whether model outputs satisfy explicit, verifiable constraints specified in a prompt. To construct our dataset, we sample and filter prompts from IFEval to ensure they are well-formed, English-only, and support the generation of sufficiently diverse outputs. For each prompt, we generate candidate responses by introducing prompt variations, including equivalent rephrasings (to produce positive samples) and violations of content or format constraints (to produce negative samples). This yields a set of responses with nontrivial separability. Ground-truth labels are assigned using a combination of IFEval-style rule-based verification for format constraints (e.g., keyword counts, structural requirements) and prompt-induced correctness assumptions for content constraints (e.g., relevance or factual consistency). A response is labeled as correct only if it satisfies both criteria. We further divide the dataset into in-domain and out-of-domain splits, where the former evaluates performance on seen prompts with held-out responses, and the latter measures generalization to entirely unseen prompts.

11.2 Hyponymy↩︎

Hyponymy is the relationship between a higher-level category and it’s exemplars (e.g., (furniture, table)). We adopted the data used by [21], consisting of ten classes, each with  30 example items. These examples ranged from very prototypical (e.g., table) to less usual (e.g., lamp), with the least prototypical not arguably not really belonging to the category at all. Since most of the examples were positive (actual examples of a category), we used GPT-5 to sample additional examples.

In order to train with a disjoint set of categories, we combined the datasets from [22]–[26], in order to have a diverse set. Then we manually removed the categories which overlapped with the ten in [21].

11.3 HumanEval↩︎

We generated solutions to the 164 Python problems in the HumanEval benchmark, requiring at least 10 passing and 10 failing solutions per problem. We used a diverse set of LLMs to generate solutions from: gpt-3.5-turbo, gpt-4, gpt-4o, gpt-4o-mini, gpt-4.1, gpt-5, and gpt-5.5, as well as Llama-3.1-8B-Instruct, deepseek-coder-1.3b-instruct, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.3, and OLMo-2-0425-1B-Instruct. Each generation was prompted under one of three difference persona strategies: normal, beginner, unusual, refactorable.

Finally, we filtered the solutions that were too short or too long (less than 10 characters or more than 900 characters). 82 problems were randomly sampled to be the held-out test set.

12 Prompts↩︎

13 Dataset Licenses↩︎

We summarize the licenses of the datasets used in this work and confirm that our use is consistent with their intended terms.

13.0.0.1 HumanEval.

HumanEval [13] is released by OpenAI under the MIT License (see https://github.com/openai/human-eval). The MIT License permits use, modification, and redistribution, including for research purposes such as ours, provided the original copyright and license notice are retained. Our derived dataset of model-generated solutions is built on top of these problems and is used solely for non-commercial research.

13.0.0.2 IFEval.

IFEval [20] is released by Google Research under the Apache License 2.0 (see https://github.com/google-research/google-research/tree/master/instruction_following_eval and https://huggingface.co/datasets/google/IFEval). Apache 2.0 permits use, modification, and redistribution for research and other purposes, subject to the standard attribution and notice requirements, which we follow.

13.0.0.3 Hyponymy stimuli (Rosch, 1975).

The category–exemplar lists we use for the Hyponymy task originate from the published norms in [21], which appear as tables of example items within the article itself. We are not aware of an explicit data license accompanying the original paper; our use of these short lists of category exemplars for non-commercial academic research, with full citation, is consistent with fair use as commonly applied to stimulus materials reported in published psychology research. The additional category datasets used to construct disjoint training categories ([22]–[26]) are likewise drawn from items reported in published academic articles or accompanying supplementary materials; THINGSplus [23] in particular is released under CC BY 4.0. We use only the textual category–exemplar pairs and cite each source.

References↩︎