1 Introduction↩︎

Reinforcement learning from human feedback (RLHF) is one of the main tools used to align powerful AI systems [1]. Alignment is important for numerous reasons, such as minimizing unintentional harm, ensuring safety and control, and increasing public trust and legal compliance [2], [3]. In order to use RLHF for alignment more effectively, researchers require high-quality data from humans that express rational preferences. This begs the question: How should rationality of preferences be defined within RLHF? Standard approaches to RLHF assume that humans provide preferences based only on what actually happened in the trajectories, i.e., the return-based interpretation [4]–[7]. This partial return-based approach does not consider events that could have occurred while taking a risky action or what might happen after the end of the trajectory segment.  [8] show that the regret-based model addresses this by looking at the quality of actions taken, rather than just the summed reward of the observed outcomes.

However, even the regret-based approach may not fully capture the factors that humans may consider when giving preferences. The regret model assumes that humans judge the “goodness" of an action as a function of the optimal advantage function \(A^*\) that the agent should ideally be able to achieve under the assumed MDP, given enough data. But what if humans do not assume optimality when judging action quality, but instead rely on their beliefs about the agent’s capabilities—in other words \(A^{\pi_{\textrm{belief}}}\) for some imagined policy \(\pi_{\textrm{belief}}\) rather than \(A^*\)?

For example, consider the scenario in Figure 1, in which an agent must choose between a shorter cliff-side path to a goal or a longer, safer path far away from the cliff. Path 1 is preferable if the agent can reproduce the trajectory noiselessly and also behave optimally afterwards. However, if the agent is suboptimal, then it risks falling off the cliff when taking this path or afterwards. Conversely, Path 2 is better for a suboptimal agent, but overly conservative for an optimal agent. Suppose that a human labeler provides a preference for Path 1 based on the incorrect belief that the agent will behave optimally post-RLHF. If the agent is trained on these preferences and is suboptimal post-RLHF, this can lead to catastrophic outcomes.

This motivates the questions: Should a rational preference labeler consider agent capabilities? And what is a normative ideal for labeler beliefs about agent capabilities? There are a variety of factors that can affect an agent’s capabilities. Factors such as limited training data, incomplete or noisy data, or policy parameterization could limit the space of learnable policies. This limitation of learnable policies, by extension, limits agent performance. Furthermore, if the agent is a robot, physical limitations may also limit agent performance beyond what a human might assume. Given a fixed set of trajectory pairs, we show that the normative ideal in regret-based RLHF is for labelers to provide preferences with respect to a quantity that is related to the best possible post-RLHF policy under any labeling—as opposed to the standard assumption of absolute optimality.

Our contributions are as follows. (1) We illustrate the importance of beliefs about agent capabilities by showing an example where poor agent-labeler agreement leads to suboptimal post-RLHF policies. (2) We define a normative ideal of beliefs and theoretically investigate the impact of deviations from this ideal on the expected return of the post-RLHF policy. (3) We empirically examine the effect of the magnitude of deviation from the ideal belief by manipulating it directly in synthetic experiments, showing that the highest performance typically results from beliefs closest to our theoretical normative ideal. (4) Finally, we confirm in a human study that human beliefs about the agent’s capabilities affect preferences. Specifically we show that it is possible to change human preferences in a statistically significant manner by changing their beliefs through priming. This study also points to potential paths forward for aligning labeler beliefs with agent capabilities.

2 Preliminaries↩︎

A Markov Decision Process (MDP) [9], [10] is specified by a tuple \((\mathcal{S}, \mathcal{A}, P, \gamma, \mu, r)\). Here, \(\mathcal{S}\) is the set of possible states, and \(\mathcal{A}\) is the set of all possible actions. \(P\) is the transition function \(P: \mathcal{S}\times \mathcal{A}\rightarrow \Delta(\mathcal{S})\), a probability distribution over the next state given the current state and action. \(\gamma\) is the discount factor, \(\mu\) is the start state distribution, and \(r:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\rightarrow \mathbb{R}\) is the scalar reward function. \(r_t\) refers to the reward received at time \(t\) for taking action \(a_t\) at state \(s_t\) and reaching \(s_{t+1}\). Terminal states have a reward of 0, unless otherwise specified, upon reaching them and a reward of 0 forever after. Throughout this paper, \(r\) refers to the ground-truth reward function, from which preferences are generated.

A policy \(\pi: \mathcal{S}\rightarrow \Delta(\mathcal{A})\), specifies the probability of taking an action in a given state. \(V^\pi_r\) is the state-value function for \(\pi\) under the reward function \(r\), defined as: \(V^\pi_r (s) = E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t, a_t, s_{t+1})|s_0=s]\). \(Q^\pi_r\) is the state-action value function: \(Q^\pi_r (s,a) = E_\pi[r(s, a, s')+ \gamma V^\pi_r (s')]\). The advantage function with respect to \(\pi\) is defined as \(A^\pi_r (s, a)= Q^\pi_r (s, a) - V^\pi_r\). \(J^\pi_r = E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t, a_t, s_{t+1})|s_0 \sim \mu]\) denotes the policy’s expected discounted return. We use \(A^\pi\), \(V^\pi\), \(Q^\pi\), and \(J^\pi\) to denote \(A^\pi_r\), \(V^\pi_r\), \(Q^\pi_r\), and \(J^\pi_r\) respectively. An optimal policy \(\pi^*\) is a policy for which \(V^{\pi^*}_r (s) \geq V^\pi_r(s)\) for all states and all policies. We use \(A^*\), \(V^*\), and \(Q^*\) to denote \(A^{\pi^*}_r\), \(V^{\pi^*}_r\), and \(Q^{\pi^*}_r\) respectively.

Trajectories: \(\sigma\) refers to a trajectory with starting state \(s^\sigma_0\). The length of a trajectory \(|\sigma|\) refers to the number of transitions in the trajectory. Each trajectory has \(|\sigma + 1|\) states and \(|\sigma|\) actions. In the context of our setting, trajectories do not contain any reward information as rewards are unknown and are to be inferred from preferences.

Preference Dataset: We have a preference dataset \(\mathcal{D}_{\text{pref}}=\{(\sigma^+_i, \sigma^-_i) \mid i=1, 2, ..., n\}\) where each pair of trajectories \((\sigma^+_i, \sigma^-_i)\) are of equal length and \(\sigma^+_i \succ \sigma^-_i\) denotes that the first trajectory is strictly preferred over the second.

Preference Models: The pairwise preferences in \(\mathcal{D}_{\text{pref}}\) are assumed to be sampled from an underlying distribution \(P(\sigma^+ \succ \sigma^-)\) which is modeled using preference model (e.g. the partial-return model [4] or the regret model [8]) obeying ground truth reward \(r\). The partial-return preference model is given by: \[\begin{align} P(\sigma^+ \succ \sigma^-) = \frac{\exp(\alpha \sum_{{_\sigma^+}}\gamma^t r_t)}{\exp( \alpha \sum_{{_\sigma^+}}\gamma^t r_t) + \exp( \alpha \sum_{{_\sigma^-}}\gamma^t r_t)} \end{align}\] and the regret-based preference model is given by : \[\begin{align} & P^*(\sigma^+ \succ \sigma^-) = \frac{\exp(\alpha\sum_{\sigma^+}\gamma^t A^{*}(s_t, a_t))}{\exp(\alpha\sum_{\sigma^+} \gamma^t A^{*}(s_t, a_t)) + \exp(\alpha\sum_{\sigma_{-}} \gamma^t A^{*}(s_t, a_t))} \label{eq:regretpreference} \end{align}\tag{1}\] where \(\alpha\) is an inverse temperature coefficient.

3 Related Work↩︎

Reinforcement Learning from Human Feedback. RLHF methods traditionally follow a two-step learning framework: (1) learning a reward function using preference data and (2) using RL to perform policy optimization  [4], [6], [7]. Direct methods remove the RL step from this traditional approach by directly learning policies from preferences but make the same preference modeling assumptions as standard return-based RLHF  [11]–[13].

Modeling human beliefs in learning. Prior work has aimed to model human beliefs in learning settings beyond RLHF.  [14] state that the reason demonstrators deviate from near-optimal actions is because of a misunderstanding of the environment dynamics and propose an algorithm to estimate these misunderstandings.  [15] assume that humans do not maintain a correct belief about dynamics when giving preferences and propose a method to infer both the reward function and the human’s beliefs about dynamics.  [16] aim to separate goals from potentially incorrect human beliefs about environmental transition dynamics.  [17] demonstrate that human irrationality, when correctly modeled, can be an asset to reward inference.

Biases in human data collection. While RLHF assumes that humans offering preferences do so rationally and in alignment with their long-term objectives, biases in human survey data have been studied across various fields of research [1], [18], [19]. It is widely recognized that human decisions can be swayed by biases due to irrationality, incomplete, or inaccurate information [20].  [19] state that mistakes are systematic and can be revealed by behavior and show that previously seen small amounts of non representative data can bias preferences. Previous psychology-aware RLHF literature investigate the impact of biasing effects on preferences.  [21] discuss how preferences can be formed at the same time as the process of preference elicitation. Personality, emotions, social connections, and decision biases (such as decoy effects, serial position effects, anchoring effects, framing effects, and group think) can also affect preferences  [22], [23].  [5] discuss the importance of alignment between researcher goals and the labeler’s labels on open ended summarization and sentiment analysis tasks.

4 Theory: Effects of agent-labeler disagreement↩︎

In this section, we aim to bound the impact of agent-labeler disagreement on expected returns of an RLHF-trained policy. We first formally define the notions of agent capability belief and agent-labeler disagreement. We then present an example MDP where agent-labeler disagreement leads to agents learning suboptimal policies. Finally, we provide a formal bound on regret of the post-RLHF policy loss as a function of agent-labeler disagreement.

Under the regret-based model of preferences, the labeler is assumed to provide preferences based on the advantage of actions under the optimal advantage function \(A^*\). However, a real-world agent’s performance after RLHF will not typically be optimal and instead will be limited by the amount of preference data, training time, or the learning algorithm itself. We will show that if humans do, in fact, generate preferences via the regret preference model, it can lead to catastrophically bad post-RLHF policies. This is due to the assumption of optimality following the trajectory segments for which preferences are given. For example, in Figure 1, the actual advantage of choosing Path 1 depends on whether or not the car will behave optimally in future time steps during execution. Thus, when providing preferences, we posit that the quality of an action ought to be related to the agent’s post-RLHF advantage function, rather than assuming optimality.

Formally, we denote the labeler’s estimate of the agent’s post-RLHF policy as \(\pi_{\text{belief}}\). \(Q^{\pi_{\text{belief}}}\) and \(A^{\pi_{\text{belief}}}\) denote the corresponding state-action value function and advantage function respectively. We propose a new preference model, similar to the regret-based preference model, but which aims to explicitly capture how humans might incorporate such beliefs. We call this new preference model the belief-based preference model: \[\begin{align} \label{eqn:beliefBT} & P^{\text{belief}}(\sigma^+ \succ \sigma^-) =\frac{\exp(\alpha\sum_{\sigma^+}\gamma^t A^{\pi_{\text{belief}}}(s_t, a_t))}{\exp(\alpha\sum_{\sigma^+} \gamma^t A^{\pi_{\text{belief}}}(s_t, a_t)) + \exp(\alpha\sum_{\sigma_{-}} \gamma^t A^{\pi_{\text{belief}}}(s_t, a_t))} \end{align}\tag{2}\] where \(\alpha\) is an inverse temperature coefficient. The remainder of the paper will examine the theoretical and empirical consequences of humans providing belief-based preferences under this model, as well as establish evidence via a human study that human preferences are, in fact, influenced by beliefs about agent capabilities. In order to do so, we must first formally define agent capabilities, as well as agent-labeler disagreement.

We assume that the human has a belief about the capabilities of the agent, which they use to generate preferences. We define this belief over capabilities in terms of a Q-function that represents the quality of an imagined agent’s policy:

In other words, preferences are generated based on the assumption that, after the completion of the preferred partial trajectory, the agent will follow \(\pi_\text{belief}\), whose state-action value function is \(Q^{\pi_{\text{belief}}}\).

One might then ask: what \(\pi_\text{belief}\) might a human imagine when providing preferences? The standard regret-based model assumes that the human imagines optimal behavior, but our human study in Section 6 demonstrates that preferences are subject to priming effects, indicating that the human’s beliefs about agent capabilities may be more closely related to the perceived current policy of the agent, or a prior over similar agents formed from past experiences, or perhaps an estimated performance ceiling for such agents.

Thus, another key question is: What is the ideal \(Q^{\pi_{\text{belief}}}\) for a human to hold? Assume that the human is asked to provide preferences over a fixed, finite set of pairs of partial trajectories. The normative ideal is for the human to provide preferences in a manner that results in the highest expected return of the post-RLHF policy. We denote any policy that achieves this maximum as \(\pi_{\text{post}}^*\). Note that this policy is not necessarily optimal for the MDP; it is simply the best that can be achieved after an RLHF step under any assignment of preferences over the fixed set of trajectory pairs. Further, there exist a set of beliefs that lead to such maximizing policies when preferences are generated according to them. We denote these as \(Q^{\pi_\text{belief}^{*}}\). This optimal belief may require oracle knowledge to generate (e.g. environmental dynamics or learning rule), but it is the normative ideal.

We stress that \(Q^{\pi_\text{belief}^{*}}\) is a theoretical reference point: we use it to define agent-labeler disagreement and to derive the bound in Section 4.2, and do not propose computing or estimating it in practice. The direction of interpretation also matters: \(Q^{\pi_\text{belief}^{*}}\) is not derived from \(\pi_{\text{post}}^*\); rather, it represents accurate beliefs about the agent’s capabilities, and it is labelers holding such beliefs that yields preferences recovering \(\pi_{\text{post}}^*\).

Finally, note that if an infinite number of (noiseless) preferences are to be given and any policy is representable by the agent, this implies that the labeler ought to have a belief of optimal behavior, normatively; in this case, \(\pi_{\text{post}}^*\) is only suboptimal when a finite number of preferences are given. However, if preferences are noisy or the agent cannot represent an optimal policy (e.g. a weak function approximator), then this may not be true.

We now define agent-labeler disagreement, which captures the more realistic scenario of a human labeler providing preferences under some agent capability belief \(Q^{\pi_\text{belief}}\) that is different from normative ideal, \(Q^{\pi_\text{belief}^{*}}\).

4.1 A case study↩︎

Here we present an example to illustrate the effect of agent-labeler disagreement. Consider the simple MDP shown in Figure 2, which is adapted from an example in  [8] to capture the risk of poor decision-making after an action.¹ \(s_0\) is the start state, and arrows show deterministic transitions towards the three terminal states on the far right. All rewards are 0 unless otherwise noted.

Suppose a labeler is asked to provide preferences between two partial trajectories: \((s_0, a_{\text{risk}}, s_{\text{risk}})\) and \((s_0, a_{\text{safe}}, s_\text{safe})\), and their beliefs are such that \(A^{\pi_{\text{belief}}}(s_0, a_{\text{risk}}) > A^{\pi_{\text{belief}}}(s_0, a_{\text{safe}})\). Then, under the belief-based preference model, they would tend to label \((s_0, a_{\text{risk}}, s_{\text{risk}})\succ (s_0, a_{\text{safe}}, s_{\text{safe}})\). If the agent-labeler disagreement is such that \(A^{\pi_{\text{belief}}^*}(s_0, a_{\text{risk}}) \leq A^{\pi_{\text{belief}}^*}(s_0, a_{\text{safe}})\)—which would occur if \(\pi_{\text{belief}}^*(s_{\text{risk}},a_{\text{lose}})>0.5\)—then obeying the preference above results in losing more often than winning during execution, resulting in a learned policy that has lower expected return than taking \(a_{\text{safe}}\).

Conversely, suppose the labeler’s beliefs are such that \(A^{\pi_{\text{belief}}}(s_0, a_{\text{risk}}) < A^{\pi_{\text{belief}}}(s_0, a_{\text{safe}})\). Then under the belief-based preference model, they would prefer \((s_0, a_{\text{safe}}, s_{\text{safe}})\) over \((s_0, a_{\text{risk}}, s_{\text{risk}})\). Obeying this preference results in a return of 0 instead of a higher return during execution if \(A^{\pi_{\text{belief}}^*}(s_0, a_{\text{risk}}) > A^{\pi_{\text{belief}}^*}(s_0, a_{\text{safe}})\). Thus, we can see that agent-labeler disagreements can lead to suboptimal preferences and resulting policies.

4.2 Formal bounds on impact of agent-labeler disagreements↩︎

We now bound the impact on the expected return of the RLHF-trained policy under agent-labeler disagreements. We work in the tabular, pairwise-comparison setting standard in theoretical analyses of RLHF [24], [25], which admits a clean characterization of how belief disagreement translates into performance loss. We first examine a setting where the labeler provides a preference under an agent-labeler disagreement at a single state-action pair, and later discuss how this analysis extends to multiple disagreements.

Next, we show how this proof extends to multiple agent-labeler disagreements.

5 Empirical Study of Impact of Agent-Labeler Disagreements on Policy Learning↩︎

In the prior section, we bounded the effects of agent-labeler disagreement for simple tabular policies without function approximation or generalization. In this section, we aim to empirically study this phenomenon for a broader class of policy representations that can exhibit generalization, as well as limit the set of representable policies. As noted earlier, when infinite noiseless preferences are given, then the normative ideal is for the labeler to give preferences with respect to an optimal policy, as long as the agent can represent and execute an optimal policy. However, if maximum agent performance is limited in some manner—for example, due to a weak function approximator, regularized learning rule, or noisy actuators that don’t faithfully execute the commanded action—then the normative ideal is to give preferences with respect to the best achievable policy.

To study the impact of agent-labeler disagreement in a more realistic setting in which the agent is limited, we design an experiment using synthetic data in a gridworld domain. The agent can only represent epsilon-greedy policies for a particular value of epsilon. This policy noise is intended to serve as a stand-in for the various types of performance-capping limitations discussed above, and has the advantage of being easy to systematically manipulate. We model the labeler’s beliefs very simply as a belief over epsilon; they assume an optimal policy within the representable epsilon-greedy class. Thus, agent-labeler disagreement arises due to incorrect beliefs about the value of epsilon. We hypothesize that performance of the post-RLHF policy will degrade as the labeler’s belief over epsilon diverges from the true epsilon.

We first collect 100 random trajectories (enough to guarantee a nearly optimal-within-class post-RLHF policy) of the agent traversing the gridworld environment that terminate either in the goal state or in the cliff state. We denote \(\epsilon\) as the \(\epsilon\)-greedy noise impacting the post-RLHF policy of the agent and \(\epsilon'\) as the \(\epsilon\)-greedy noise assumed by the labeler and determining the labeler’s preferences. The values of \(\epsilon\) and \(\epsilon'\) may be different. We denote the labeler’s belief about agent’s capability \(Q^{\pi_{\text{belief}}^{\epsilon'}}(s,a)\). We denote the post-RLHF agent capability as \(Q^{\pi_{\text{post}}^{\epsilon}}(s,a)\). We vary the values of \(\epsilon\) and \(\epsilon'\) to show the impact of agent-labeler disagreements on the post-RLHF policy’s expected return.

We use \(Q^{\pi_{\text{belief}}^{\epsilon'}}(s,a)\) to generate synthetic preference data using the belief-based preference model shown in Equation 2 . We use these preferences to train the agent using Contrastive Preference Learning (CPL) [13], a scalable algorithm for regret-based RLHF.

width=.6

Table ¿tbl:tab:gridworld95results? shows the average returns of the post-RLHF policies. The highest return generally comes when the \(\epsilon'\)-greedy noise assumed by labeler is closest to the \(\epsilon\)-greedy noise impacting the agent’s policy. We see that for lower values of epsilon (\(\epsilon = 0 \;\&\; 0.1\)), preferences given under the optimal advantage function (\(\epsilon' = 0\)) provide the highest returns. However, this strategy becomes a hindrance for effective RLHF as the action noise increases, as seen when \(\epsilon = 0.3 \;\&\; 0.5\). This is due to the fact that the post-RLHF policy learns to walk near the dangerous cliffs, often falling off, since the labeler assumed too low of a noise level. Walking near the cliffs is only optimal when the agent has a very low epsilon value. These results serve as a proof of concept for how agent-labeler disagreements can arise in practice and how they can be exacerbated by agent limitations that may be unknown to the labeler.

6 Agent-Labeler Disagreements in Human Preference Collection↩︎

Now that we have established that agent-labeler disagreements pose a problem in RLHF, in this section, we demonstrate that human beliefs about agent capabilities do, in fact, significantly affect the preferences given by human labelers. This is done via a human study in which priming effects are used to change participants’ beliefs about the agent’s capabilities. Additionally, this result indicates that it is possible in theory to change human preferences to be better aligned with agent capabilities.

Domain: We choose a self-driving car domain, simulated using the CARLA driving simulator [28] to collect trajectories to use for human preference collection. Since the rules and trade-offs involved in driving are something that many people already have a strong intuition for, the choice of self-driving car setup allows us to keep the filtering of the participants minimal.

Priming: Before beginning data collection, we randomly assign participants to one of three conditions: (1) no priming (control condition), (2) safe priming (confidence increasing condition), and (3) unsafe priming (confidence decreasing condition). Both safe and unsafe priming videos start from the same starting state to avoid unintended biasing effects. Participants in the safe priming group are shown a video where the car obeys all traffic laws, avoids obstacles, and successfully overtakes a slower car, whereas the participants in the unsafe priming group are shown a video where the car drifts across lanes and crashes into obstacles.

Preference elicitation: Participants are asked to provide preferences over 15 randomly chosen pairs of video data. All data pairs on which preferences are collected show legal driving. Within each pair, the car is shown to be driving on the same path for the same amount of time. In each pair, one video shows faster and more time saving behavior, which if executed imperfectly (during or after the trajectory) could lead to failure states, such as hitting a pedestrian, another car, or breaking traffic laws; the other shows more safety-conscious behavior that requires more time to execute, but for which it is easier to avoid failure states. Each pair of trajectories is designed to make the participants consider both: (a) how a learning agent might behave in the same states shown in the trajectories and (b) what the agent might do after the end of each trajectory.

All data pairs are randomly shuffled, and the order in which each pair of data is shown is also randomly shuffled to avoid ordering bias during preference collection. In each question, both videos in each pair are presented next to each other.

Instructions for the participants: Participants in the control group are given a standard preference collection instruction explaining that the objective of the survey is to improve a self-driving car using their preferences over behavioral data. We chose to frame this language for this group to indicate that their preferences were not meant to be a rating of the trajectories themselves, but rather a learning signal for improving a self driving car’s policy. This was designed to lead them to reflect less on the trajectories themselves and more on their beliefs about how a learning agent might behave in similar or future states. The participants in the safe and unsafe priming groups are given a similar instruction with an addition of a video showcasing the car’s driving behaviors. To encourage the participants to think deeply about the car’s capabilities, they are then asked about which skills they think would be useful for the car to learn to become a more effective driver.

Post-processing of collected data: To maintain a high standard while finalizing the data, we construct a participant filtering step that ensures that participants are paying attention when providing preferences. First, we show a pair of videos: one in which the car drives normally and one in which the car crashes into a guardrail while making a turn. Additionally, we show a pair of videos in which one car drives normally and the other runs a red light. We remove all preference data from participants who do not strongly prefer the attention check videos that show good driving, while also reporting that they were extremely confident.

In total, we collected data from 259 participants. The data filtering and quality checks allowed 146 (46 safe priming, 50 unsafe priming, 50 no priming) participants’ data to be used for analysis. We further balanced the data in each group to avoid unintended impacts on the statistical analysis by randomly removing 4 entries from both the unsafe and no priming groups. The final dataset contains 46 participants in each of the three groups.

To the best of our knowledge, there is not a standard nonparametric statistical test that can test for statistical significance when there are: (1) more than two groups; (2) for which the data is Likert; (3) and for which each participant gives repeated measures data. Instead, we used the average of each participant’s responses, rather than individual question responses, to create independent data that is amenable to standard statistical tests. To reduce noise due to participant confusion, uncertainty, or misunderstandings, we used only those responses where participants reported extremely high confidence in their answers.

We briefly justify these analysis choices. Averaging repeated responses. A natural alternative is a mixed-effects ordinal regression, such as a cumulative link mixed model (CLMM), which can directly model repeated ordinal measures. However, CLMMs rely on the proportional-odds assumption—that priming shifts responses uniformly across all thresholds of the Likert scale. In our setting priming may plausibly affect the extremes of the scale differently from its middle, and we did not have strong grounds to assume proportional odds holds; moreover, with 46 participants per group after filtering, the effective sample size is small for stably estimating the random-effects variance components of a CLMM. Averaging each participant’s responses instead yields a single independent summary statistic per participant, which enables the Kruskal–Wallis test [29]—a well-established nonparametric test with minimal distributional assumptions. This discards some within-participant information, but we preferred a test whose assumptions we could be confident about over a more powerful test whose assumptions may be violated. Retaining only high-confidence responses. In our experience with online platforms such as Prolific, responses given with low self-reported confidence frequently reflect participant disengagement or rushed responding rather than genuine uncertainty about the preference—an effect compounded here, where participants must watch and compare two videos before answering. The confidence filter thus complements the attention-check filtering described above as a quality-control measure. Robustness. Our analysis is conservative by design: the Bonferroni correction used in the Dunn’s test is among the most stringent multiple-comparison corrections available, and the Kruskal–Wallis test makes minimal distributional assumptions. We still observe significance at the 5% level (Table ¿tbl:tab:dunnstest?, \(p=0.015\) between safe and unsafe priming), and a less conservative analysis would be expected to only strengthen these results.

Qualitative results:

Immediately after priming, we asked participants to provide free-text feedback directed specifically at the car’s capabilities—namely, which skills they think would be useful for the car to learn to become a more effective driver. Because this question is directed at the agent’s capabilities rather than driving risk in general, the responses serve as a probe of participants’ beliefs about the agent. To quantify these responses, we used GPT-4 [30] to conduct a sentiment analysis for each response and provide a 1–10 star rating, using the following prompt:

“Analyze the following feedback about a car’s driving skills and rate its safety from 1 to 10 stars. Are there any skills this car need to learn to be a more effective driver? Only consider the presence of any safety concerns and suggestions for improvement with small consideration for factors unrelated to safety: Feedback: [PARTICIPANT’S FEEDBACK] Provide a structured analysis with reasons for the assigned score.”

GPT-4 was the strongest model available at the time our study was conducted. We chose an LLM over human sentiment annotators for greater consistency, as LLM-based annotations have been shown to achieve higher internal consistency than human annotators for sentiment analysis tasks [31], [32]; rating the safety sentiment of short free-text responses on a numerical scale is straightforward relative to the benchmarks studied in that work.

Figure 4 shows that immediately after priming, participants primed with the unsafe priming had lower sentiment scores regarding the car’s capabilities than the participants in the safe priming group, suggesting that priming had an impact on their beliefs about the agent specifically. This establishes the first link in the causal chain we aim to demonstrate—that priming shifts beliefs about agent capabilities. The Kruskal–Wallis and Dunn–Bonferroni analysis of the Likert preference data below establishes the second link—that these shifted beliefs in turn affect labeling behavior.

Quantitative results:

l0.5

width=0.9

Next, we analyzed the Likert data associated with the preferences of each condition group. We used the non-parametric Kruskal-Wallis test for statistical analysis due to the ordinal nature of the data [29]. The Kruskal-Wallis test resulted in a p-value of 0.033, indicating a statistically significant difference between at least two priming groups at the 5% level. In order to detect which of the two priming groups differ, we used the Dunn’s test with Bonferroni correction [33]. The results of the Dunn-Bonferroni test are shown in Table ¿tbl:tab:dunnstest? and show statistically significant (\(p=0.015\)) differences between safe and unsafe priming groups at the 5% level. Cliff’s delta (d = 0.31) between unsafe vs safe priming shows that participants primed with unsafe priming were influenced toward more safe responses at a higher rate than participants primed with safe priming with a small effect size [34]. Participants in the no priming group did not exhibit statistically significant differences in their responses compared to either the unsafe or safe priming groups, as shown in Table ¿tbl:tab:dunnstest?. These results suggest that human preferences are influenced by their beliefs about agent capabilities. To mitigate this bias, practitioners should actively attempt to strengthen agent-labeler agreement during data collection.

6.1 Recommendations For Researchers↩︎

Our results show that having good agent-labeler agreement is important for effective policy learning in regret-based RLHF. While it is difficult to reason about an agent’s post-training performance prior to training, our results suggest potential best practices for practitioners:

(1) Inform labelers directly about known limitations: Practitioners may choose to inform labelers about agent limitations prior to preference collections. For example, this may include known data or training limitations, non-obvious restrictions on agent capabilities (e.g. an unusually limited turning radius on a vehicle), or assessments of projected agent strengths and weaknesses. However, the success of such an approach depends heavily on the ability of labelers to make sense of such information, which may require varying levels of technical expertise.

(2) Online preference collection with intermittent priming: Practitioners may choose to use online preference collection interleaved with RLHF training on the collected preferences. Here, labelers’ beliefs about the agent’s capability may be continuously updated and aligned via priming using data from the agent’s most current policy. This approach may also demonstrate to labelers how their preferences influence policy learning.

Both recommendations reduce to the same mechanism: exposing labelers to the agent’s actual behavior before they provide preferences. Our human study is a proof of concept that this is feasible and effective—a single priming video shifted preferences significantly (Table ¿tbl:tab:dunnstest?)—and the online loop of recommendation (2) adds only one step to a standard pipeline: generating demonstrations, which requires merely running the current policy. These rollouts need not come from a post-RLHF policy; the agent’s policy at any training stage is informative about its current capabilities. What constitutes an informative demonstration is domain-specific (e.g., for a language-model agent), but the mechanism of calibrating beliefs through exposure to behavior is general.

Truthful calibration versus manipulation. Because priming can shift preferences, calibration must be distinguished from manipulation: the difference is whether the demonstrations are representative of the agent’s actual on-policy behavior. Truthful calibration shows unfiltered rollouts reflecting real capabilities and limitations; the same interface could instead curate a misleading picture (e.g., hiding failure modes) to steer labels toward a preferred outcome. We therefore recommend sampling demonstrations representatively from the current policy, as non-representative behavior undermines both labeler autonomy and the reliability of the reward signal.

7 Conclusion↩︎

Current methods of preference collection do not account for participants’ beliefs about the capabilities of the agents learning from their preferences. We show that the preferences collected under agent-labeler disagreements can lead to suboptimal policies both theoretically and empirically and make preliminary suggestions on how to minimize such disagreements. Future work may include developing improved priming strategies to ensure human beliefs are as accurate as possible, as well as algorithmic advances that mitigate the impact of incorrect beliefs on RLHF.

Limitations and open directions. Each part of our analysis suggests a natural next step. Theorem 1 is proved in a clean tabular setting with noiseless preferences; Appendix 8 relaxes the noiseless assumption, and extending it to function approximation—where a belief perturbation can propagate across states through shared parameters—is a promising direction for sharper guarantees. Empirically, our synthetic experiments isolate the effect under controlled conditions and our human study shows that priming shifts both beliefs and preferences; validating gains in a full RLHF pipeline driven end-to-end by human labels, which would require a substantially larger preference budget, is an exciting avenue for follow-up work.

Acknowledgments↩︎

This work has taken place in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (IIS-2437426) and Open Philanthropy.

Scott Niekum holds concurrent appointments as an Associate Professor at the University of Massachusetts Amherst and as an Amazon Scholar. This paper describes work performed at the University of Massachusetts Amherst and is not associated with Amazon.

8 Proofs↩︎

Proof of the corollary 1:

8.1 A visitation-weighted bound for multiple disagreements↩︎

The bound in Corollary 1 is deliberately assumption-free: it replaces every disagreement by the largest one and every visitation probability by its upper bound of \(1\). When the visitation distribution of the perturbed policy is available, the same argument yields a tighter, visitation-weighted bound that makes explicit how disagreements at different states interact through the MDP.

This form makes the interaction between disagreements transparent: a disagreement at a rarely-visited state \(s_i'\) (small \(d_\mu^{\pi_{\text{post}}^\delta}(s_i')\)) contributes proportionally little to the performance loss, whereas a disagreement on the agent’s main path is weighted heavily. Corollary 1 is recovered by bounding each \(d_\mu^{\pi_{\text{post}}^\delta}(s_i') \leq 1\) and each \(\delta_i \leq \max_j \delta_j\), together with \(\sum_i d_\mu^{\pi_{\text{post}}^\delta}(s_i') \leq 1\).

8.2 A generalized bound for noisy preferences↩︎

Theorem 1 assumes noiseless preferences (\(\alpha \to \infty\)), which yield a deterministic post-RLHF policy. We now relax this assumption and derive a bound for finite \(\alpha\), where preferences are noisy and the post-RLHF policy is stochastic. As noted in Remark 2, this bound recovers Theorem 1 in the \(\alpha \to \infty\) limit. We emphasize that the quantities below (in particular the advantage magnitude \(A_{\max}\)) serve a purely analytical role and are not intended to be computed in practice.

As \(\alpha \to \infty\), the factor \(\Delta p \to 1 - p\), so \(\pi_{\text{post}}^\delta(a' \mid s') \to 1\) and the softmax policy collapses to the deterministic action switch of Theorem 1; the analysis then reduces to that theorem, and under its assumptions the bound simplifies to \(\delta/(1-\gamma)\), which avoids the dependence on \(A_{\max}(s')\). We note that the generalized bound above is intended to give qualitative intuition for how performance degrades under relaxed assumptions rather than to be a tight characterization.

9 Additional Details About Human Study↩︎

In this section, we provide more details about our human study and the demographics of participants.

9.1 Design Decisions↩︎

We chose self-driving cars as the domain on which to collect our preference dataset because the rules and trade-offs of driving are something people understand without high level knowledge about any language or formal higher education and allowed for minimal filtering of participants. We conducted these preference collection experiments with prior IRB approval.

9.2 Data Collection UI↩︎

Figure 5 shows the format of the survey that participants saw when giving their responses.

9.3 Example of a Pair of Trajectories Shown to Human Labelers↩︎

All trajectories shown to participants show legal driving. Each pair of trajectories show the car driving along the same route for the same amount of time so participants can easily distinguish the tradeoff between safety and time savings. For example, one pair of scenarios shows a legal overtake of a slow car in a single lane road, as shown in Figure 6. While it is legal and safe to perform the overtake, doing so requires trust in the car’s abilities to successfully return back to its lane without any accidental swerving or any other mishaps. The pair shows one trajectory where the car does not attempt this overtake and instead patiently stays behind the slow car, as shown in Figure 6 (a); the other shows the car overtaking successfully and getting further along the road in the same amount of time, as shown in Figure 6 (b). This allows room for the subject to interpret the quality of the car’s driving for the overtake and be subjective in their belief in the car’s abilities.

9.4 Demographics↩︎

Figure 7 (a) shows the distribution of driver experience among the participants of our survey and Figure 7 (b) shows participant’s willingness to ride in a self driving car. While we did not have specific filters for years of driving experience, most of the participants report a significant number of years of driving experience, meaning that they are well versed in the domain. A large portion of participants from all three priming groups reported that they would be willing to ride in a self driving car. However, out of all participants who reported that they would be unwilling to ride in a self driving car, the largest group of participants was the unsafe priming group.

10 GridWorld Experimental Details↩︎

In this section, we provide additional details about the GridWorld domain and the hyperparameters used in the learning algorithm. For this analysis, we used the following python libraries: numpy [35], math and random from the standard python library [36], and scipy[37].

10.1 Gridworld Domain↩︎

The gridworld domain consists of a 7x7 grid of cells. We number the coordinates from 0 to 6. The agent always starts in the cell numbered (6,6). From there, the agent may move in any of the four cardinal directions. Any movement by the agent that results in the agent moving outside the boundaries of the environment results in the agent staying in its original state with an additional reward of -1. The states (0, 6), (3, 6), and (4, 6) are terminal states and result in a reward of +200, -200, and -200 respectively. The episode terminates when the agent either reaches one of the terminal states or when the number of states that the agent has visited reaches 1,000.

10.2 Hyperparameters↩︎

Here, we will discuss the hyperparameters used in the gridworld experiment for the Contrastive Preference learning algorithm:

11 Disclosure about LLM usage↩︎

We used LLM to format the content of the paper and to make any stylistic changes to the paper.

References↩︎

---

1. Note that decision-making itself may not appear to be a risk from the agent’s perspective, since the agent controls its own actions at subsequent states. However, even deterministically chosen actions can be risky from the labeler’s perspective because the labeler is only able to influence the agent at the current state.↩︎