A Unifying Lens on Reward Uncertainty in RLHF


Abstract

Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is pessimism: penalizing rewards in regions where the RM is uncertain. However, standard scalar RMs provide no principled notion of uncertainty. We argue that the right object is a distributional reward model \(p(r\mid x,y)\). Under either a Bayesian inference or a KL-distributionally robust optimization (KL-DRO) lens, the KL-regularized RLHF objective admits a closed-form effective reward \(\tilde{r}(x,y) = \pm\beta\log\mathbb{E}_p[e^{\pm r/\beta}]\). The pessimistic branch unifies the prior heuristics for RM ensemble aggregation: mean aggregation, worst-case optimization (WCO), and uncertainty-weighted optimization (UWO) all emerge as limits or truncations of this single expression. This also clarifies the implicit assumptions of each existing rule.

1 Introduction↩︎

Reinforcement learning from human feedback (RLHF) is the dominant paradigm for aligning large language models with user intent  [1][4]. Pairwise preferences are first used to fit a reward model (RM) and a policy is then optimized against the RM with a KL penalty to a reference policy, via policy-gradient methods such as PPO [5] or GRPO [6]. Because the RM is an imperfect proxy for human preferences, policy optimization can overoptimize into out-of-distribution regions, producing high RM scores without improving true response quality. This is an instance of Goodhart’s law [7][10] and has been documented across forms including sycophancy and [11], [12] length bias [13].

A natural mitigation is pessimism: when the RM is uncertain, a lower effective reward should be assigned. Prior work in RLHF instantiates this idea by training ensembles of RMs and aggregating their outputs in different ways. [14] introduce three rules: mean aggregation (averaging members), worst-case optimization (WCO, taking the minimum across members), and uncertainty-weighted optimization (UWO, mean minus a tunable multiple of the empirical variance). [15] make a similar case for ensembles, and follow-up work has explored adversarial [16] and robustness-driven [17] variants. These methods empirically help, but are typically presented as heuristics with little principled guidance on which to prefer and how their hyperparameters relate.

We argue here that to properly handle uncertainty, one should replace the scalar RM with a distributional reward model \(p(r \mid x, y)\). Once the RM is distributional, a single principle—either marginalization over reward uncertainty through a Bayesian inference lens [18], [19] or KL-distributionally robust optimization (KL-DRO) [20], [21]—yields a closed-form effective reward to substitute into the standard KL-RL objective. The aggregation rules used in practice emerge as truncations or limits of this expression. This perspective also connects RLHF pessimism to distributional reinforcement learning [22][25] and to offline-RL pessimism [26][28].

1.0.0.1 Contributions.

  • A unifying lens. From either a Bayesian inference or KL-DRO perspective, the RLHF objective with reward uncertainty admits a closed-form effective reward \(\tilde{r}(x,y) = \pm\beta\log\mathbb{E}_p[e^{\pm r/\beta}]\) (Sec. 3).

  • Unification of prior heuristics. Mean, WCO, and UWO emerge as the \(\beta\to\infty\) limit, \(\beta\to 0\) limit, and Gaussian truncation, respectively, of the pessimistic branch (Sec. 3.5).

  • Implementation guidance. We discuss what each heuristic implicitly assumes about the reward distribution \(p(r\mid x,y)\), and what is needed (calibration, posterior families) to instantiate the framework with a real RM (Sec. 4).

2 Background and Related Work↩︎

2.0.0.1 Reward overoptimization.

[7] established that the proxy RM score and a stronger “gold” RM score (which is used to model the true objective we seek to optimize, i.e., how a human would score the response) diverge as PPO progresses, the canonical signature of reward hacking. [8] formalize reward hacking, and [9] situate it among broader AI safety concerns. Empirically, hacked policies exhibit sycophancy [11], length inflation [13], and other surface artifacts that the proxy RM cannot distinguish from true quality [29].

2.0.0.2 Ensemble pessimism.

[14] introduce Mean, WCO, and UWO aggregation over RM ensembles and find that pessimistic variants reduce reward hacking. [15] reach similar conclusions for best-of-\(n\) and PPO, while noting that ensembles can share failure modes. Lightweight ensembles share a frozen backbone and diversify only a head [30].

2.0.0.3 Uncertainty-aware reward models.

An alternative to ensembles is to place a posterior directly over a single head. Bayesian last-layer methods [31], [32] yield a closed-form Gaussian posterior over the reward \(\mathcal{N}(\mu(x,y), \sigma^2(x,y))\) at each prompt-response pair. [33] use RM uncertainty to route hard cases to a stronger LLM judge.

2.0.0.4 Inference-as-control and robust RL.

The Bayesian view of KL-regularized RL [18], [19] interprets policies as posteriors under an “optimality” observation. Robust optimization [20], [21] and pessimism in offline RL [26][28] provide the complementary, adversarial framing. Our derivation shows that, in the RLHF setting, these two seemingly disparate views produce the same functional form up to a sign.

3 Theory: A Unifying Lens↩︎

3.1 Setup↩︎

Standard KL-regularized RLHF [2], [4] maximizes \[\label{eq:rlhf} \max_{\theta}\;\mathbb{E}_{x \sim \mathcal{D}}\!\Big[\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)} [r(x,y)] - \beta\, D_{\mathrm{KL}}\!\left(\pi_\theta\|\pi_{\mathrm{ref}}\right)\!\Big],\tag{1}\] with the well-known optimum \(\pi^*(y\mid x) \propto \pi_{\mathrm{ref}}(y\mid x)\exp(r(x,y)/\beta)\).

In the standard pipeline, \(r(x,y) = r_\phi(x,y)\) is a deterministic scalar from a learned RM. We instead treat the reward as a random variable \(r \sim p(r \mid x, y)\), arising, for example, from a deep ensemble or a Bayesian head, and ask: to best mitigate reward hacking, what scalar effective reward \(\tilde{r}(x,y)\) should replace \(r_\phi(x,y)\) in Eq. 1 ?

3.2 Optimistic Branch: Bayesian Marginalization↩︎

Following the inference-as-control view [18], [19], we introduce a binary optimality variable \(O \in \{0,1\}\) with \[\label{eq:inference95as95control} p(O{=}1\mid x,y,r) = e^{r/\beta},\tag{2}\] where \(r\) is shifted so that \(\sup r = 0\). Marginalizing over reward uncertainty gives \[p(O{=}1\mid x,y) = \int e^{r/\beta} p(r\mid x,y)\,dr = M_x(1/\beta),\] where \(M_x(t) := \mathbb{E}_p[e^{tr}]\) is the moment generating function of \(p(r\mid x,y)\). Bayes’ rule with prior \(\pi_{\mathrm{ref}}\) then yields \(\pi^*_{\mathrm{opt}}(y\mid x) \propto \pi_{\mathrm{ref}}(y\mid x)\, M_x(1/\beta)\). Matching to the KL-RL form \(\pi^* \propto \pi_{\mathrm{ref}}\exp(\tilde{r}/\beta)\) identifies \[\label{eq:opt} \boxed{\;\tilde{r}_{\mathrm{opt}}(x,y) = \beta \log \mathbb{E}_{p(r\mid x,y)}\!\left[e^{r/\beta}\right].\;}\tag{3}\] By Jensen’s inequality, since \(\exp\) is convex, \[\tilde{r}_{\mathrm{opt}}(x,y) \;\geq\; \mu(x,y),\] so uncertainty inflates the effective reward. This is the Bayesian-optimal use of an uncertain RM under the inference-as-control likelihood in Eq. 2 in the absence of any robustness concern.

3.3 Pessimistic Branch: KL-DRO↩︎

To mitigate reward hacking we instead want a worst-case effective reward. We let an adversary choose the reward distribution \(Q\), paying a KL cost to deviate from the believed posterior \(p\) [20], [21]: \[\label{eq:dro} \tilde{r}_{\mathrm{rob}}(x,y) = \inf_Q \!\Big\{\mathbb{E}_Q[r] + \beta\, D_{\mathrm{KL}}(Q\,\|\,p(r\mid x,y))\Big\}.\tag{4}\] The parameter \(\beta\) controls the adversary’s reach: large \(\beta\) pins \(Q\) close to \(p\), small \(\beta\) permits arbitrary tilting. Solving the variational problem (see Appendix 6) yields the unique optimizer \[Q^*(r) \;\propto\; p(r\mid x,y)\, e^{-r/\beta},\] which is \(p\) exponentially tilted toward low rewards. Plugging back in, \[\label{eq:rob} \boxed{\;\tilde{r}_{\mathrm{rob}}(x,y) = -\beta \log \mathbb{E}_{p(r\mid x,y)}\!\left[e^{-r/\beta}\right].\;}\tag{5}\] This is the entropic risk measure of \(p(r\mid x,y)\). By Jensen’s inequality, \(\tilde{r}_{\mathrm{rob}}(x,y) \leq \mu(x,y)\): uncertainty deflates the effective reward.

3.3.0.1 Choice of KL coefficients.

The KL coefficient \(\beta\) in Eq. 1 controls the deviation of the policy from the reference, while \(\beta\) in Eq. 4 controls the robustness to reward uncertainty. These play distinct roles and need not coincide. In this work we tie the two for simplicity, yielding the simple coefficient \(\lambda=1 /(2 \beta)\). More generally one may consider \(c \beta\), with \(c\) reflecting the desired degree of pessimism relative to policy regularization.

3.4 Cumulant Expansion↩︎

Let \(K_x(t) := \log M_x(t) = \log\mathbb{E}_p[e^{tr}]\) be the cumulant generating function (CGF) of \(p(r\mid x,y)\), with cumulants \(\kappa_n\) defined by \(K_x(t) = \sum_{n\geq 1} \kappa_n t^n / n!\). Then \(\kappa_1 = \mu\), \(\kappa_2 = \sigma^2\), \(\kappa_3\) is the third central moment, \(\kappa_4 = \mathbb{E}[(r-\mu)^4] - 3\sigma^4\) (excess kurtosis \(\times\sigma^4\)), and so on.

Substituting into Eqs. 3 and 5 : \[\label{eq:cumulants} \tilde{r}_{\mathrm{opt/rob}} = \mu \pm \frac{\sigma^2}{2\beta} + \frac{\kappa_3}{6\beta^2} \pm \frac{\kappa_4}{24\beta^3} + \cdots\tag{6}\] Even cumulants (\(\kappa_2,\kappa_4,\ldots\)) flip sign between the two while odd cumulants (\(\kappa_3,\kappa_5,\ldots\)) do not. For symmetric distributions (\(\kappa_{2n+1} = 0\) for \(n\geq 1\)), \(\tilde{r}_{\mathrm{opt}}\) and \(\tilde{r}_{\mathrm{rob}}\) are exact mirror images about \(\mu\).

3.4.0.1 Gaussian case.

If \(p(r\mid x,y) = \mathcal{N}(\mu, \sigma^2)\), all cumulants beyond \(\kappa_2\) vanish, \(K_x(t) = \mu t + \sigma^2 t^2 / 2\), and the series in Eq. 6 terminate exactly: \[\label{eq:gauss} \tilde{r}_{\mathrm{opt}} = \mu + \frac{\sigma^2}{2\beta},\qquad \tilde{r}_{\mathrm{rob}} = \mu - \frac{\sigma^2}{2\beta}.\tag{7}\] Under any Gaussian distributional RM, the principled pessimistic reward is \(\mu - \sigma^2/2\beta\), with the variance coefficient set by the KL coefficient \(\beta\) rather than tuned as a free hyperparameter.

3.5 Unification of Prior Methods↩︎

The aggregation rules of [14] emerge as limits and truncations of \(\tilde{r}_{\mathrm{rob}}\).

3.5.0.1 Mean (\(\beta \to \infty\)).

As \(\beta \to \infty\), the variance term in Eq. 6 vanishes (\(\sigma^2/2\beta \to 0\)), and similarly for all higher orders. Hence \(\tilde{r}_{\mathrm{rob}} \to \mu\). Equivalently, an infinite KL penalty forces the adversary’s distribution \(Q^*\) back onto \(p\), so the worst case coincides with the mean. Mean aggregation is the risk-neutral limit of the pessimistic effective reward.

3.5.0.2 WCO (\(\beta \to 0\)).

As \(\beta \to 0\), the tilted distribution \(Q^*(r) \propto p(r) e^{-r/\beta}\) concentrates all of its mass on \(\inf\,\mathrm{supp}(p)\). For an empirical ensemble \(\{R_i\}_{i=1}^K\) where \(p\) is a uniform mixture of point masses, the infimum is \(\min_i R_i\). Hence \[\lim_{\beta \to 0} \tilde{r}_{\mathrm{rob}}(x,y) \;=\; \min_i R_i(x,y).\] Worst-case optimization is the unbounded-adversary limit of the pessimistic effective reward.

3.5.0.3 UWO (Gaussian truncation).

The UWO rule of [14] is \[R_{\mathrm{UWO}}(x,y) = \hat{\mu}(x,y) - \lambda\,\hat{\sigma}^2(x,y),\] with \(\lambda\) treated as a free hyperparameter. Intuitively, UWO works by penalizing the policy for generating responses for which there is high disagreement among reward models within the ensemble, which helps prevent the exploitation of a single reward model which might be erroneously assigning high rewards to incorrect or low-quality responses. Comparing with the Gaussian case of Eq. 7 , UWO is exactly the pessimistic effective reward under a Gaussian posterior, with \(\lambda = 1/(2\beta)\). UWO’s implicit assumptions are: (i) \(p(r\mid x,y)\) is approximately Gaussian; (ii) the variance coefficient can be decoupled from the KL coefficient \(\beta\).

\begin{table}[t]

Method Formula Recovered as
Mean \(\hat{\mu}\) \(\beta\to\infty\) limit
WCO \(\min_i R_i\) \(\beta\to 0\) limit
UWO \(\hat{\mu}-\lambda\hat{\sigma}^2\) Gaussian, \(\lambda\) free
Ours \(\hat{\mu}-\hat{\sigma}^2/2\beta\) Gaussian, \(\lambda=1/2\beta\)
\vskip

-0.15in \end{table}

Table ¿tbl:tab:unification? summarizes the unification. The take-away is that the choice of aggregation rule is a choice about the assumed shape of \(p(r\mid x,y)\) and the adversary’s KL budget. Mean assumes no uncertainty matters; WCO assumes the worst possible distribution; UWO is the middle ground in a Gaussian approximation. Our framework makes these assumptions explicit and supplies a principled default (\(\lambda = 1/2\beta\)) when working with a Gaussian distributional RM.

3.6 Estimation from a Finite Sample↩︎

In practice \(p(r\mid x,y)\) is approximated by a finite sample \(\{r_i(x,y)\}_{i=1}^K\)—either ensemble members or posterior samples. Let \[\hat{\mu} = \frac{1}{K}\sum_{i=1}^K r_i,\quad \hat{\sigma}^2 = \frac{1}{K-1}\sum_{i=1}^K (r_i - \hat{\mu})^2.\] Two natural estimators of \(\tilde{r}_{\mathrm{rob}}\) arise.

3.6.0.1 Direct log-MGF estimator.

\[\hat{\tilde{r}}_{\mathrm{rob}}^{\,(\text{logMGF})} = -\beta\log\!\left(\frac{1}{K}\sum_{i=1}^K e^{-r_i/\beta}\right).\] This estimator is consistent but biased at finite \(K\): since \(\log\) is concave, Jensen gives \(\mathbb{E}[\hat{\tilde{r}}_{\mathrm{rob}}^{\,(\text{logMGF})}] \geq \tilde{r}_{\mathrm{rob}}\), with bias growing in \(\sigma/\beta\). It is sensitive to outliers when \(\beta\) is small (the \(\exp(-r_i/\beta)\) term blows up for any single low \(r_i\)).

3.6.0.2 Gaussian-truncated estimator.

\[\label{eq:est-gauss} \hat{\tilde{r}}_{\mathrm{rob}}^{\,(\text{Gauss})} = \hat{\mu} - \frac{\hat{\sigma}^2}{2\beta}.\tag{8}\] This is exact when \(p\) is Gaussian and drops higher cumulants otherwise. We note that for small \(K\) (as is typical for RM ensembles, \(K \in \{3,\ldots,10\}\) in [14]), higher cumulants are poorly estimated. The relative error on \(\hat{\sigma}^2\) alone is \(\sqrt{2/(K-1)} \approx 32\%\) for \(K=20\), so we recommend Eq. 8 as a default.

4 Discussion↩︎

The pessimistic effective reward \(\tilde{r}_{\mathrm{rob}}\) is only as informative as the posterior \(p(r\mid x,y)\) used to compute it. Two concrete prerequisites:

(i) A genuine distributional RM. Scalar RMs trained with the Bradley-Terry likelihood [34] provide no uncertainty. Two practical options are: (a) a deep ensemble [14], [30], [35], whose empirical variance approximates \(\sigma^2(x,y)\); and (b) a Bayesian last-layer construction [31], [32], which yields a closed-form Gaussian posterior at the cost of a single forward pass and an \(O(d^2)\) matrix-vector product. The Bayesian construction is especially interesting as its posterior is Gaussian and Eq. 7 applies without truncation.

(ii) Calibration. The variance term \(\sigma^2(x,y)\) only modulates \(\tilde{r}_{\mathrm{rob}}\) usefully if it tracks true epistemic uncertainty. The Bradley–Terry loss collapses soft preferences (\(0.51\) and \(0.99\)) onto the same hard label and degrades calibration. Strictly proper scoring rules [36], in particular the Brier score [37], preserve annotator confidence and are known to improve calibration. Using AI feedback [3], [38], [39] provides soft labels naturally by prompting for a confidence score during AI feedback.

4.0.0.1 Implications for existing methods.

  • When Mean should suffice. The cumulant expansion (Eq. 6 ) shows that pessimism corrections decay as \(1/\beta^{n-1}\). For large \(\beta\) (heavy KL regularization), Mean is essentially optimal as all rules collapse to it.

  • When WCO is appropriate. WCO is the \(\beta \to 0\) limit, suitable when the adversary is essentially unconstrained. In RLHF, this corresponds to a setting where one expects the RM to be arbitrarily wrong in some direction. [15] report that WCO can underperform Mean when the ensemble is too small or correlated, which our framework explains: WCO discards all distributional information beyond the minimum.

  • Why UWO works. UWO is the right shape under a Gaussian posterior. Its empirical success suggests that practical RM ensembles are reasonably symmetric and unimodal. The free \(\lambda\) may be absorbing miscalibration; under the theory-prescribed value \(\lambda = 1/2\beta\), calibration becomes essential.

4.0.0.2 Beyond Gaussian.

The cumulant expansion in Eq. 6 suggests three directions when \(p(r\mid x,y)\) is not well-approximated by a Gaussian. (i) For asymmetric but light-tailed distributions, third- and fourth-cumulant corrections to Eq. 8 can be estimated from larger ensembles. (ii) For genuinely heavy-tailed distributions where the MGF estimator is unstable, quantile-based pessimism (e.g., CVaR) is a robust alternative, connecting to quantile distributional RL [23][25]. (iii) Flexible posterior families, such as mixtures of Gaussians or normalizing flows fit directly to held-out preference data could replace the ensemble or Bayesian linear head.

4.0.0.3 Connections to broader pessimism.

The KL-DRO derivation connects RLHF reward hacking to pessimism in offline RL [26][28], where similar log-MGF / entropic-risk formulations arise from analogous principles. Our contribution is to localize this connection: in RLHF, the relevant random variable is the reward (not the transition dynamics, which are deterministic for an autoregressive policy), and the relevant penalty parameter is the standard KL coefficient \(\beta\) already present in Eq. 1 .

5 Conclusion↩︎

We have given a single derivation that captures the essence of reward uncertainty in RLHF. From either a Bayesian or a KL-DRO starting point, the closed-form effective reward is \(\tilde{r}(x,y) = \pm\beta\log\mathbb{E}_p[e^{\pm r/\beta}]\). The pessimistic branch unifies mean aggregation, WCO, and UWO as limits and truncations, and yields a principled default \(\tilde{r} = \mu - \sigma^2/2\beta\) under a Gaussian posterior in which the variance coefficient is set by the KL coefficient rather than tuned. The framework clarifies what each existing heuristic implicitly assumes and gives a recipe for going beyond Gaussian via higher cumulants or quantile-based pessimism.

Acknowledgements↩︎

We thank Kiante Brantley for initial guidance and helpful discussions during the early stages of this project. OpenAI GPT-5.5 and Refine.ink were used to assist with language editing and improve the clarity and readability of the manuscript

6 Proofs and Derivations↩︎

6.1 Optimistic Effective Reward↩︎

6.1.0.1 Setup.

Let \(\pi_0 = \pi_{\mathrm{ref}}\). For each \((x,y)\), \(r \sim p(r\mid x,y)\). The KL-regularized objective with effective reward \(\tilde{r}\) is \[J(\pi) = \mathbb{E}_\pi[\tilde{r}(x,y)] - \beta\, D_{\mathrm{KL}}(\pi \| \pi_0), \qquad \beta > 0.\]

6.1.0.2 Step 1: Optimality variable.

Introduce \(O \in \{0,1\}\) with \(p(O=1\mid x,y,r) = e^{r/\beta}\) where \(r\) is shifted so \(\sup r = 0\).

6.1.0.3 Step 2: Marginalize.

\[p(O{=}1\mid x,y) = \int e^{r/\beta} p(r\mid x,y)\,dr = M_x(1/\beta),\] where \(M_x(t) := \mathbb{E}_p[e^{tr}]\) is the MGF.

6.1.0.4 Step 3: Bayes’ rule.

With prior \(\pi_0\), \[\pi^*_{\mathrm{opt}}(y\mid x) \propto \pi_0(y\mid x) M_x(1/\beta).\]

6.1.0.5 Step 4: Identify \(\tilde{r}\).

Matching to \(\pi^* \propto \pi_0 \exp(\tilde{r}/\beta)\) gives \[\tilde{r}_{\mathrm{opt}}(x,y) = \beta \log \mathbb{E}_p[e^{r/\beta}].\]

6.1.0.6 Step 5: Jensen.

Since \(\exp\) is convex, \(\mathbb{E}_p[e^{r/\beta}] \geq e^{\mu/\beta}\), so \(\tilde{r}_{\mathrm{opt}} \geq \mu\).

6.2 Pessimistic Effective Reward (KL-DRO)↩︎

6.2.0.1 Step 1: Definition.

\[\tilde{r}_{\mathrm{rob}}(x,y) = \inf_Q \{\mathbb{E}_Q[r] + \beta D_{\mathrm{KL}}(Q\|p)\}.\]

6.2.0.2 Step 2: Lagrangian.

With multiplier \(\lambda\) for \(\int Q = 1\), \[\mathcal{L}(Q,\lambda) = \int Q(r) r\,dr + \beta\!\int Q\log Q\,dr - \beta\!\int Q\log p\,dr - \lambda\!\left(\int Q\,dr - 1\right).\]

6.2.0.3 Step 3: Stationarity.

Using \(\frac{d}{dQ}[Q\log Q] = \log Q + 1\): \[\frac{\delta \mathcal{L}}{\delta Q(r)} = r + \beta(\log Q + 1) - \beta\log p - \lambda = 0,\] giving \(Q^*(r) \propto p(r) e^{-r/\beta}\). The normalization is \(C(x,y) := \mathbb{E}_p[e^{-r/\beta}]\).

6.2.0.4 Step 4: Evaluate the infimum.

\(\log Q^*(r) - \log p(r) = -r/\beta - \log C\), so \[D_{\mathrm{KL}}(Q^*\|p) = -\tfrac{1}{\beta}\mathbb{E}_{Q^*}[r] - \log C.\] Therefore \(\mathbb{E}_{Q^*}[r] + \beta D_{\mathrm{KL}}(Q^*\|p) = -\beta\log C\) (the reward terms cancel exactly), giving \[\tilde{r}_{\mathrm{rob}}(x,y) = -\beta\log\mathbb{E}_p[e^{-r/\beta}].\]

6.2.0.5 Step 5: Jensen.

\(\mathbb{E}_p[e^{-r/\beta}] \geq e^{-\mu/\beta}\); taking \(-\beta\log\) reverses the inequality, so \(\tilde{r}_{\mathrm{rob}} \leq \mu\).\(\square\)

6.3 Cumulant Expansion↩︎

Let \(K_x(t) = \log\mathbb{E}_p[e^{tr}] = \sum_{n\geq 1} \kappa_n t^n / n!\). Then \[\tilde{r}_{\mathrm{opt}} = \beta K_x(1/\beta) = \sum_{n\geq 1} \frac{\kappa_n}{n!\,\beta^{n-1}} = \mu + \frac{\sigma^2}{2\beta} + \frac{\kappa_3}{6\beta^2} + \frac{\kappa_4}{24\beta^3} + \cdots,\] \[\tilde{r}_{\mathrm{rob}} = -\beta K_x(-1/\beta) = \sum_{n\geq 1} \frac{(-1)^{n+1}\kappa_n}{n!\,\beta^{n-1}} = \mu - \frac{\sigma^2}{2\beta} + \frac{\kappa_3}{6\beta^2} - \frac{\kappa_4}{24\beta^3} - \cdots.\] For Gaussian \(p\), \(\kappa_{n\geq 3} = 0\) and the series terminate at \(\tilde{r} = \mu \pm \sigma^2/2\beta\).

6.4 Recovery of Mean and WCO↩︎

6.4.0.1 \(\beta\to\infty\) (Mean).

Each term \(\kappa_n/(n!\beta^{n-1})\) for \(n\geq 2\) vanishes, leaving \(\tilde{r}_{\mathrm{rob}} \to \kappa_1 = \mu\). Equivalently, \(Q^*(r) \propto p(r) e^{-r/\beta} \to p(r)\) as \(\beta\to\infty\).

6.4.0.2 \(\beta\to 0\) (WCO).

For an empirical ensemble \(\{R_i\}_{i=1}^K\) with \(p(r\mid x,y) = \frac{1}{K}\sum_i \delta(r - R_i)\), \[\tilde{r}_{\mathrm{rob}} = -\beta\log\!\left(\frac{1}{K}\sum_{i=1}^K e^{-R_i/\beta}\right).\] Let \(R_{\min} = \min_i R_i\) and factor out \(e^{-R_{\min}/\beta}\): \[\tilde{r}_{\mathrm{rob}} = R_{\min} - \beta\log\!\left(\frac{1}{K}\sum_{i=1}^K e^{-(R_i - R_{\min})/\beta}\right).\]

As \(\beta \rightarrow 0\), all terms in the sum with \(R_i>R_{\min }\) are exponentially suppressed; the sum approaches the count of minimizing members \(m\) divided by \(K\). The logarithm of this sum approaches the constant \(\log (m / K)\), so the overall correction term scales as \(O(\beta) \rightarrow 0\). Hence \(\tilde{r}_{\mathrm{rob}} \to R_{\min} = \min_i R_i\), recovering WCO.

6.4.0.3 Gaussian truncation (UWO).

For \(p = \mathcal{N}(\mu, \sigma^2)\), \(K_x(t) = \mu t + \sigma^2 t^2/2\) exactly, so \(\tilde{r}_{\mathrm{rob}} = \mu - \sigma^2/2\beta\). This matches \(R_{\mathrm{UWO}} = \hat{\mu} - \lambda\hat{\sigma}^2\) with \(\lambda = 1/(2\beta)\).

References↩︎

[1]
P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in neural information processing systems (NeurIPS), 2017.
[2]
L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in neural information processing systems (NeurIPS), 2022.
[3]
Y. Bai et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022, [Online]. Available: https://arxiv.org/abs/2212.08073.
[4]
N. Stiennon et al., “Learning to summarize with human feedback,” in Advances in neural information processing systems (NeurIPS), 2020.
[5]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms.” 2017, [Online]. Available: https://arxiv.org/abs/1707.06347.
[6]
Z. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” 2024, doi: 10.48550/ARXIV.2402.03300.
[7]
L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in International conference on machine learning (ICML), 2023, [Online]. Available: https://arxiv.org/abs/2210.10760.
[8]
J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” in Advances in neural information processing systems (NeurIPS), 2022.
[9]
D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, 2016, [Online]. Available: https://arxiv.org/abs/1606.06565.
[10]
A. Pan, K. Bhatia, and J. Steinhardt, “The effects of reward misspecification: Mapping and mitigating misaligned models,” in International conference on learning representations (ICLR), 2022.
[11]
M. Sharma et al., “Towards understanding sycophancy in language models,” arXiv preprint arXiv:2310.13548, 2023, [Online]. Available: https://arxiv.org/abs/2310.13548.
[12]
E. Perez et al., “Discovering language model behaviors with model-written evaluations,” in Findings of the association for computational linguistics: ACL, 2023.
[13]
P. Singhal, T. Goyal, J. Xu, and G. Durrett, “A long way to go: Investigating length correlations in RLHF,” arXiv preprint arXiv:2310.03716, 2023, [Online]. Available: https://arxiv.org/abs/2310.03716.
[14]
T. Coste, U. Anwar, R. Kirk, and D. Krueger, “Reward model ensembles help mitigate overoptimization,” in International conference on learning representations (ICLR), 2024, [Online]. Available: https://arxiv.org/abs/2310.02743.
[15]
J. Eisenstein et al., “Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking,” arXiv preprint arXiv:2312.09244, 2023, [Online]. Available: https://arxiv.org/abs/2312.09244.
[16]
S. Chakraborty et al., “Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation,” arXiv preprint arXiv:2403.05171, 2024, [Online]. Available: https://arxiv.org/abs/2403.05171.
[17]
Y. Yan et al., “Reward-Robust RLHF in LLMs,” 2024, doi: 10.48550/ARXIV.2409.15360.
[18]
T. Korbak, E. Perez, and C. L. Buckley, RL with KL penalties is better viewed as Bayesian inference,” arXiv preprint arXiv:2205.11275, 2022, [Online]. Available: https://arxiv.org/abs/2205.11275.
[19]
S. Levine, “Reinforcement learning and control as probabilistic inference: Tutorial and review,” arXiv preprint arXiv:1805.00909, 2018, [Online]. Available: https://arxiv.org/abs/1805.00909.
[20]
A. Ben-Tal, L. El Ghaoui, and A. Nemirovski, Robust optimization. Princeton University Press, 2009.
[21]
L. P. Hansen and T. J. Sargent, Robustness. Princeton University Press, 2008.
[22]
M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in International conference on machine learning (ICML), 2017, [Online]. Available: https://arxiv.org/abs/1707.06887.
[23]
W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos, “Distributional reinforcement learning with quantile regression,” in AAAI conference on artificial intelligence, 2018.
[24]
W. Dabney, G. Ostrovski, D. Silver, and R. Munos, “Implicit quantile networks for distributional reinforcement learning,” in International conference on machine learning (ICML), 2018.
[25]
R. Keramati, C. Dann, A. Tamkin, and E. Brunskill, “Being optimistic to be conservative: Quickly learning a CVaR policy,” in AAAI conference on artificial intelligence, 2020.
[26]
Y. Jin, Z. Yang, and Z. Wang, “Is pessimism provably efficient for offline RL?” in International conference on machine learning (ICML), 2021.
[27]
T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal, “Bellman-consistent pessimism for offline reinforcement learning,” in Advances in neural information processing systems (NeurIPS), 2021.
[28]
M. Rigter, B. Lacerda, and N. Hawes, RAMBO-RL: Robust adversarial model-based offline reinforcement learning,” in Advances in neural information processing systems (NeurIPS), 2022.
[29]
A. Wang, I. Arcuschin, and A. Conmy, “Automatically Finding Reward Model Biases,” 2026, doi: 10.48550/ARXIV.2602.15222.
[30]
V. Dwaracherla, S. M. Asghari, B. Hao, and B. Van Roy, “Efficient Exploration for LLMs,” 2024, doi: 10.48550/ARXIV.2402.00396.
[31]
D. Yang et al., RewardUQ: A unified framework for uncertainty-aware reward models,” arXiv preprint arXiv:2602.24040, 2026, [Online]. Available: https://arxiv.org/abs/2602.24040.
[32]
Y. Zhai et al., “Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles,” arXiv preprint arXiv:2401.00243, 2024, [Online]. Available: https://arxiv.org/abs/2401.00243.
[33]
Z. Xu et al., “Ask a strong LLM judge when your reward model is uncertain,” in Advances in neural information processing systems (NeurIPS), 2025.
[34]
R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. The method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952.
[35]
B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,” 2016, doi: 10.48550/ARXIV.1612.01474.
[36]
T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” Journal of the American Statistical Association, vol. 102, no. 477, pp. 359–378, 2007.
[37]
G. W. Brier, “Verification of forecasts expressed in terms of probability,” Monthly Weather Review, vol. 78, no. 1, pp. 1–3, 1950.
[38]
H. Lee et al., RLAIF: Scaling reinforcement learning from human feedback with AI feedback,” arXiv preprint arXiv:2309.00267, 2023, [Online]. Available: https://arxiv.org/abs/2309.00267.
[39]
H. Lee et al., “RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback.” 2024, [Online]. Available: https://arxiv.org/abs/2309.00267.