1 Introduction↩︎

Reinforcement learning (RL) in decision making has demonstrated promising capabilities in many practical scenarios [1], [2]. However, designing dense and comprehensive reward signals for RL requires significant human efforts and extensive domain knowledge [3], [4]. Moreover, agents can easily exploit designed rewards to achieve high returns in unexpected ways, leading to reward hacking phenomena [5]. RLHF [6]–[8] serves as a powerful paradigm that transforms human feedback into guidance signals. It eliminates the need for manual reward function design, playing an important role in scenarios such as robotics control [9] and language model training [10]. Beyond the reward modeling, the integration of diverse human feedback into reinforcement learning [11], [12] has become the spotlight, as it can align the agent’s behavior with the complex and variable intentions of humans [13], [14].

Despite the increasing significance of RLHF, the scarcity of universal tools and unified benchmarks specifically designed for RLHF hinders a comprehensive study into learning from various types of feedback. Several RLHF benchmarks [15]–[17] for comparative feedback have been proposed, but they all utilize predefined hand-engineered reward functions from the environment as scripted teachers, conducting experiments under conditions that simulate absolute expert and infinite synthetic feedback. However, in reality, scenarios akin to unbiased expert feedback are scarce. There is a growing use of crowdsourced annotations [10], [18] as a substitute to cope with the increasing scale of feedback. A significant issue is general unreliability and confusion among large-scale crowdsourced annotators, leading to cognitive inconsistency and even adversarial attacks [11], [15], [19]. Therefore, studying the RLHF algorithm design with human teachers instead of script teachers is more compatible with the practical problem. Moreover, the absence of reliable annotation tools that can support diverse types of feedback across environments, as well as efficient pipelines for crowdsourcing, compounds the challenges of conducting research on human feedback.

To bridge the gap, we introduce Uni-RLHF, a comprehensive system implementation for RLHF, comprising of three packages: 1) universal annotation platform supporting diverse human feedback for decision making tasks, 2) large-scale and reusable crowdsourced feedback datasets, and 3) modular offline RL baselines from human feedback. Uni-RLHF offers a streamlined interface designed to facilitate the study of learning across a variety of feedback types. Uni-RLHF offers a consistent API for different types of feedback. It uses a query sampler to load and visualize a set of available segments from the datasets. The annotators then provide appropriate feedback labels, which are transformed into a standardized format. Uni-RLHF is compatible with mainstream RL environments and demonstrates strong scalability.

Crowdsourced label collection is often challenging and costly. When gathering human feedback labels instead of script labels, various factors such as different experimental setups, and the number of annotators can introduce significant variability in final performance. Therefore, a standardized data collection process and reliable feedback datasets can greatly alleviate the burden on researchers. We have established a systematic pipeline including crowdsourced annotation and data filters, providing 30 feedback datasets (\(\approx\)​15M steps) of varying difficulty levels that can be reused.

Furthermore, we provide Offline RLHF baselines that integrate mainstream reinforcement learning algorithms and different feedback modeling methods. We then evaluate design choices and compare the performance of baseline algorithms on the collected datasets. We empirically found that Uni-RLHF achieves competitive performance with well-designed manual task reward functions. This offers a solid foundation and benchmark for the RLHF community.

Our contributions are as follows:

• We introduce a universal RLHF annotation platform that supports diverse feedback types and corresponding standard encoding formats.

• We develop a crowdsourced feedback collection pipeline with data filters, which facilitates the creation of reusable, large-scale feedback datasets, serving as a valuable starting point for researchers in the field.

• We conduct offline RL baselines using collected feedback datasets and various design choices. Experimental results show that training with human feedback can achieve competitive performance compared to well-designed reward functions, and effectively align with human preferences.

2 Related Work↩︎

Reinforcement Learning from Human Feedback (RLHF) leverages human feedback to guide the learning process of reinforcement learning agents [20]–[22], aiming to address the challenges in situations where environment information is limited and human input is required to enhance the efficiency and performance [23], [24]. RLHF covers a variety of areas such as preference-based RL [7], inverse RL [25], [26] and active learning [27] with various scopes of feedback types. Comparative feedback [7], [28]–[30] facilitate a pairwise comparison amongst objectives, culminating in a preference for trajectories, and further transform these rankings into the targets of the reward model. The TAMER [8], [31]–[33] and COACH [34], [35] series frameworks furnishes quantifiable evaluative feedback (binary right/wrong signals) on individual steps. Despite the excellent generality of comparison and evaluation, the power of expression is lacking [36]. Recent works [37], [38] have employed visual explanation feedback encoding task-related features to further broaden the channels of human-machine interaction. [39], [40] enhances learning efficiency through the incorporation of a saliency map provided by humans. Besides, language [41] and relative attribute [36] corrective guidance furnishes a richer semantic mode of communication, thereby aligning with user intentions with greater precision.

Benchmarks and Platforms for RLHF. Though diverse types of feedback are widely proposed and applied, a prominent limitation is that there are lack of generalized targeted benchmarks for deploying and evaluating different types of human feedback. [42] proposed a DERAIL benchmark for preference-based learning, but it’s limited to simple diagnostic tasks. [28] considered more complex robotic tasks and designed simulated teachers with irrationality. [16] evaluated offline preference-based RL on parts of D4RL [43] tasks, but these studies primarily focus on synthesizing simulated feedback using scripted teachers. [44] experimented with small batches of expert feedback and found high disagreement rates between synthesis and human labels, suggesting synthetic labels may be misleading. Besides, [45] and [11] suggested using real human feedback can improve the learning of human-aligned reward models. In a novel approach, we use large-scale crowdsourced labeling for systematic analysis and experimentation across multiple environments. Another key limitation is the lack of universal platforms for multiple feedback types annotation. While some researchers have open-sourced their tools [12], [44], they are suitable only for individual, small-scale annotations and specific feedback types, presenting significant challenges for scaling up to a large number of environments. [46] provided labeling tools for active learning which are limited in low dimension tasks. A concurrent work, RLHF-Blender [47], implemented an interactive application for investigating human feedback, but it lacks scalability for more environments and has not yet been evaluated in a user study.

3 Universal Platform for Reinforcement Learning with Diverse Feedback Types↩︎

To align RLHF methodologies with practical problems and cater to researchers’ needs for systematic studies of various feedback types within a unified context, we introduce the Uni-RLHF system. We start with the universal annotation platform, which supports various types of human feedback along with a standardized encoding format for diverse human feedback. Using this platform, we have established a pipeline for crowdsourced feedback collection and filtering, amassing large-scale crowdsourced labeled datasets and setting a unified research benchmark. The complete system framework is shown in 1.

3.1 Implementation for Multi-feedback Annotation Platform↩︎

Uni-RLHF employs a client-server architecture, built upon the Flask¹ back-end service and the Vue² front-end framework. It provides an interactive interface for employers and annotators, supporting multi-user annotation. Its intuitive design enables researchers to focus on the task, not the tool. As illustrated in 1 (upper left), Uni-RLHF packs up abstractions for RLHF annotation workflow, where the essentials include: interfaces supporting a wide range of online environments and offline datasets, a query sampler that determines which data to display, an interactive user interface, enabling annotators to view available trajectory segments and provide feedback responses and a feedback translator that transforms diverse feedback labels into a standardized format.

Environment and Datasets Interface. The Uni-RLHF supports both online and offline training modes. During online training, Uni-RLHF samples data from the buffer periodically and utilizes its built-in render function to present the segments through the user interface. This facilitates asynchronous training of the RL agent and the feedback model. The complete training details are shown in 9. In offline mode, we gather access to several widely recognized reinforcement learning datasets. Some representative tasks from these environments are visualized in 1. Furthermore, Uni-RLHF allows easy customization and integration of new offline datasets by simply adding three functions \(\texttt{load\_datasets, sample}\) and \(\texttt{visualize}\).

Query Sampler. The query sampler component determines how to sample the data presented to the user interface. Choosing a suitable query sampler to annotate can help improve model performance [16]. We propose four configurable types of sampling methods:

• Random: Random sampling is the random selection of a specified number. We regard random sampling as the most common baseline, and this simple approach can achieve competitive performance to active learning sampling in numerous environments [15].

• Disagreement: Disagreement sampling is primarily applicable for comparative feedback. We calculate the variance of the queries using the probability \(p\) that trajectory 1 is superior to trajectory 2 according to the feedback model which encourages the selection of informative queries exhibiting maximum divergence [7].

• Schedule: Schedule sampling prefers sampling near on-policy query from the replay buffer during online training, which adapts to the gradually improving policy performance and mitigates the issue of query-policy misalignment [52].

• Customization: Customized sampling allows for task-specific and customizable metrics sampling that facilitates targeted queries.

Interactive User Interface. Uni-RLHF provides users with a clean visual interface, offering an array of interactive modalities for various feedback types. The user interface consists of a segment display, task description, feedback interaction, and additional information module. The feedback interaction supports different levels of granularity, including selection, swiping, dragging, and keyframe capturing. Notably, we also support focused framing of images, which can be applied to the visual feedback, and the annotations can be saved as the PASCAL VOC [53] format. See our homepage for more visualization.

Feedback Translator. The feedback translation component receives various feedback from the user interface, which is encoded into consistent feedback labels based on the standardized feedback encoding framework proposed in 3.2. Post-processed labels are then matched with queries and linked to a shared storage space, such as an SQL database. This configuration allows the application to scale, enabling multiple annotators to carry out tasks without interference. This setup facilitates large-scale distributed human experiments.

3.2 Standardized Feedback Encoding Format for Reinforcement Learning↩︎

To capture and utilize diverse and heterogeneous feedback labels from annotators, we analyze a range of research and propose a standardized feedback encoding format along with possible training methodologies. We examine the standard interaction for an RL agent with an environment in discrete time. At each time step \(t\), the agent receives state \(\mathbf{s}_t\) from the environment and chooses an \(\mathbf{a}_t\) based on the policy \(\mathbf{a}_t \sim \pi\left(\cdot \mid \mathbf{s}_t\right)\). Then the environment returns a reward \(r_t\) (optional) and the next state \(\mathbf{s}_{t+1}\). We discuss five common feedback types, explaining how annotators interact with these types and how they can be encoded. Additionally, we briefly outline the potential forms and applications of reinforcement learning that integrate various forms of human feedback. We introduce the encoding format in this section and refer to 11 for the corresponding training methods.

Comparative Feedback. Comparative feedback is the most commonly used feedback type for reward modeling because of its simplicity and universality. Crowdsourcing tends to prefer relative comparisons over absolute evaluations. Users are presented with either two trajectory segments for comparison or multiple objectives for ranking. We define a segment \(\sigma\) as a sequence of time-indexed observations \(\{\mathbf{s}_{k}, ..., \mathbf{s}_{k+H-1}\}\) with length \(H\). Given a pair of segments \((\sigma^0, \sigma^1)\), annotators give feedback indicating their preference. This preference, denoted as \(y_\text{comp}\), can be expressed as \(\sigma^0\succ\sigma^1\), \(\sigma^1\succ\sigma^0\) or \(\sigma^1=\sigma^0\). We encode the labels \(y_\text{comp} \in \{(1, 0), (0,1), (0.5, 0.5)\}\) and deposit them as triples \((\sigma^0,\sigma^1, y_\text{comp})\) in the feedback datasets \(\mathcal{D}\). Following the Bradley-Terry model [54], comparative feedback can be utilized to train reward function \(\widehat{r}_\psi\) that aligns with human preferences. See training details in 11.1.

Attribute Feedback. In many cases, evaluating the quality of outcomes using a single scalar value is challenging due to the multifaceted nature of numerous problems. For instance, stable robotic movement requires consideration of multiple factors, including speed, stride length, and postural pattern [55]. Similarly, autonomous driving necessitates the simultaneous consideration of speed, safety, and rule-adherence [56], [57]. Thus an assessment mechanism that evaluates attributes from multiple perspectives is needed to guide the agent towards multi-objective optimization. Inspired by [36], we have extended comparative feedback into attribute feedback from multi-perspective evaluation. First, we predefine \(\boldsymbol{\alpha} = \{\alpha_1, \cdots, \alpha_k\}\) represents a set of \(k\) relative attribute about agent’s behaviors. Given a pair of segments \((\sigma^0, \sigma^1)\), The annotator will conduct a relative evaluation of two segments for each given attribute, which is encoded as \(y_\text{attr} = (y_1, ..., y_k)\) and \(y_i \in \{(1, 0), (0,1), (0.5, 0.5)\}\). We then establish a learnable attribute strength mapping model that translates trajectory behaviors into latent space vectors. This approach simplifies the problem to a binary comparison and we can distill the reward model similar to comparative feedback. See training details in 11.2.

Evaluative Feedback. In this feedback mode, annotators observe a segment of a segment \(\sigma\) and assign a numerical score or other quantifiable judgment. Evaluative feedback can offer informative absolute evaluations of samples and be less demanding than comparative feedback. Some studies provide feedback to the agent for a specific state, rather than the entire trajectory, potentially leading to a reward-greed dilemma. Therefore, we define \(n\) discrete ratings for trajectory levels [58]. As shown in the right figure, three ratings are available: good, bad, and average. Thus we receive corresponding human label \((\sigma, y_\text{eval})\), where \(y_\text{eval} \in \{0, ..., n-1\}\) is level of assessment. See training details in 11.3.

Visual Feedback. Visual feedback offers a strong way for annotators to interact with agents naturally and efficiently. Annotators identify key task-relevant objects or areas within the image, guiding the agents toward critical elements for decision-making via visual explanation. We define the feedback as \((\sigma, y_\text{visual})\), where \(y_\text{visual}=\{\text{Box}_1, ..., \text{Box}_m\}\). \(\text{Box}_i\) is a the bounding box position \((x_\text{min}, y_\text{min}, x_\text{max}, y_\text{max})\). To reduce the burden on

the annotators, similar to [59] and [37], we provide an object detector that pre-highlights and tracks all objects in the environment, and the annotator only needs to deselect irrelevant objects. We use the detector for atari provided by [60] as an example. As shown in the right figure, all objects including people, a key, monsters, and ladders are pre-framed, so the annotations only need to remove interference targets like ladders. Given an RGB image observation \(o_t \in \mathbb{R}^{H \times W \times 3}\), the saliency encoder maps the original input to the saliency map \(m_t \in [0, 1]^{H \times W}\) which highlights important parts of the image based on feedback labels. Then the policy is trained with inputs that consist of the RGB images and the saliency predictions as an extra channel. See training details in 11.5.

Keypoint Feedback. Frequently, only a few decision points are important in long-horizon tasks [61]. As illustrated in the right figure, the agent (red arrow) must pick up the key and unlock the door to reach the target location (green circle). Guidance towards key observations such as picking up the key and opening the door can reduce redundant exploration and improve learning efficiency. We annotate key time steps \(y_\text{key} = \{t_1, t_2, ..., t_n\}\) from segment \(\sigma\), where \(n\) is the number of keypoints. Subsequently, we use this feedback to train a keypoint predictor and use historical trajectories to infer key states, which serve as guidance for the agent. See training details in 11.4.

3.3 Large-scale Crowdsourced Annotation Pipeline↩︎

To validate the ease of use of various aspects of the Uni-RLHF, we implemented large-scale crowdsourced annotation tasks for feedback label collection, using widely recognized Offline RL datasets. After completing the data collection, we conducted two rounds of data filtering to minimize the amount of noisy crowdsourced data. Our goal is to construct crowdsourced data annotation pipelines around the Uni-RLHF, facilitating the creation of large-scale annotated datasets via parallel crowdsourced data annotation and filtering. We aim to foster collaboration and reduce the high cost of feedback from RLHF research, facilitating wider adoption of RLHF solutions through reusable crowdsourced feedback datasets. See 14 for details of datasets.

Datasets Collection Workflow. To generate label and conduct primary evaluations, we assembled a team of about 100 crowdsourced workers, each of whom was paid for their work. These annotators come from a broad array of professions, ranging in age from 18 to 45 and boasting diverse academic backgrounds. None had prior experience in reinforcement learning or robotics research. We aim to simulate a scenario like practical applications of low-cost and non-expert crowdsourced annotation. Using comparative feedback annotation as an example, each annotator is assigned a task along with instructions about the task description and judgment standards. To enhance the understanding of task objectives, experts (authors) provide five example annotations with detailed analysis. We refer to 13 for full instructions.

Ex-ante Filters. Collecting large-scale feedback often results in suboptimal and noisy data [62]. To improve learning performance under human irrationality and bias,

we adopt an ex-ante approach. During the feedback collection phase, we incorporate a small amount of expert validation set and discard labels that fall below the accuracy threshold. Furthermore, we check labels during the annotation process and summarize the overall problems to optimize the task description. This process aligns the intent of crowdsourcing with experts. Another approach ex-post methods is label noise learning [63], [64], which improves performance by understanding and modeling the generation mechanism of noise in the samples. This method is orthogonal to our filtering mechanism.

To verify the effectiveness of each component in the annotation pipeline, we initially sampled 300 trajectory segments of the left-c task in SMARTS for expert annotation, referred to as oracle. We had five crowdsourcing instances, each annotating 100 trajectories in three distinct settings. naive implies only seeing the task description, +example allows for viewing five expert-provided annotation samples and detailed analysis, and +filter adds filters to the previous conditions. The experimental results, displayed in 7, revealed that each component significantly improved annotation accuracy, ultimately achieving a 98% agreement rate with the expert annotations.

4 Evaluating Benchmarks for offline RLHF↩︎

Finally, we conducted numerous experiments on downstream decision-making tasks, utilizing the collected crowdsourced feedback datasets to verify the reliability of the Uni-RLHF system. Notably, due to the wide applicability and low crowdsourcing training cost of comparative feedback, we conducted a more detailed study of comparative feedback in 4.1, systematically establishing comprehensive research benchmarks. We then performed additional experimental verification of attribute feedback in 4.2 and keypoint feedback in 8.5 as a primer. We encourage the community to contribute and explore more extensive baselines and usage for additional feedback types within the Uni-RLHF system.

4.1 Evaluating Offline RL with Comparative Feedback↩︎

Setups. As shown in 1 (bottom), we used crowdsourcing comparative feedback for reward learning (see 11.1 for more training details). Then we used the trained reward model to label reward-free offline datasets and train offline RL policy. Because the reward learning process and offline RL were fully decoupled, our methods can simply be combined with any offline RL algorithms. We used three of the most commonly used offline RL algorithms as the backbone, including Implicit Q-Learning (IQL, [65]), Conservative Q-Learning (CQL, [66]) and TD3BC [67]. We adopt implementations of these algorithms from the CORL [68] library and reuse the architecture and hyperparameters for different environments. To ensure comprehensive evaluation and benchmarking that accurately reflects the capabilities of the offline RLHF algorithm across a wide range of tasks and complexities, we selected a wide range of tasks of varied types, encompassing locomotion, manipulation, and navigation tasks from D4RL [43], as well as visual control tasks from D4RL_Atari [48] and realistic autonomous driving scenarios from SMARTS [49]. More environment and dataset details can be found in 6. As baselines, we evaluated preference modeling via three reward model architectures, denoted as MLP, CNN and TFM (Transformer) structure (see 1 bottom right). MLP and CNN architectures are similar to [7], [28]. MLP is for vector inputs, while CNN is for image inputs. Following the [44], we also implemented a similar transformer architecture to estimate the weighted sum of non-Markovian rewards. We refer to 7 for more details about RM architectures.

4.1.1 D4RL Experiments↩︎

We measured expert-normalized scores \(=100 \times \frac{\text{ score }- \text{ random score }}{\text{ expert score-random score }}\) underlying original reward from the benchmark. We used Oracle to represent models trained using hand-designed task rewards. In addition, we assessed two different methods of acquiring labels: one is crowd-sourced labels obtained by crowd-sourcing through the Uni-RLHF system, denoted as CS, and the other is synthetic labels generated by script teachers based on ground truth task rewards, which can be considered as expert labels, denoted as ST. The performance is reported in ¿tbl:tab:d4rl32results?, we focus on the performance of algorithms that use crowdsourced labels because they best match the realistic problem setting.

Based on these results, we can draw several valuable observations. Firstly, among the three algorithmic foundations, IQL demonstrated the most consistent and highest average performance across environments when utilizing the same crowd-sourced datasets. This is attributable not only to the high performance of the IQL baseline but also to the consistency of IQL’s performance. When comparing the results of IQL-CS and IQL-Oracle, we found that IQL demonstrates competitive performance against Oracle in most of tasks and even surpasses Oracle in some cases. On the contrary, CQL occasionally experiences a policy collapse, leading to a drastic performance drop, as exemplified in the application of the CQL-CS-MLP method to the hopper-m-r. Secondly, After comparing the results of CS-TFM and CS-MLP (marked as blue), we empirically found that CS-TFM exhibits superior stability and average performance compared to CS-MLP, particularly in the context of sparse reward task settings. We hypothesize that this is due to TFM’s modeling approach and its ability to capture key events, which align more closely with human judgment. Thirdly, our experiments show that crowd-sourced labels (CS) can achieve competitive performance compared to synthetic labels (ST) in most environments. Especially with CQL and TD3BC backbone, we can observe a significant improvement in some cases. This suggests that while synthetic labels are not subject to issues such as noise and irrationality, they are prone to shortsighted behaviors and cannot provide flexible and variable evaluations based on a global view like humans.

4.1.2 Atari Experiments↩︎

r7.5cm

We conducted experiments in five image-based Atari environments, with detailed results presented in Table 2. We collected 2,000 pairs of annotated comparative feedback for each task and trained a reward model for the CNN structure. Next, we chose the CQL of discrete version by d3rlpy³ codebase and verified the performance of Oracle, ST and CS with the same hyperparameters. Across all tasks, we found that models trained using crowd-sourced labels consistently outperformed those trained with synthetic labels, achieving performance levels comparable to the Oracle. We hypothesize that this superior performance of crowd-sourced labels could be due to their ability to capture nuanced details within the game environment, details that may prove challenging to encapsulate through simple integral reward functions. This potentially results in more accurate evaluations as compared to those derived from synthetic labels.

4.1.3 SMARTS Experiments↩︎

r6cm

As shown in 3, we studied three typical scenarios in autonomous driving scenarios. The reward function design is particularly complex because it requires more extensive expertise and balancing multiple factors. We carefully designed the task rewards incorporating eight factors, which includes some events and multiple penalty (see ¿tbl:tab:smarts32reward32design? for details). We empirically demonstrate that we can achieve competitive performance simply by crowdsourced annotations, compared to carefully designed reward functions or scripted teachers. We provide a multi-indicator metrics for evaluation (speed and comfort) and visualisation in 8.1. The model using human feedback showed a more conservative and rule-following policy which was consistent with human intentions.

4.2 Evaluating Offline RL with Attribute Feedback↩︎

Setups. We trained an attribute-conditioned reward model on attribute feedback datasets for multi-objective optimization (see 11.2 for more training details). To utilize this reward model for training, we adopted an attribute-conditioned TD3BC approach. Specifically, based on the original version of TD3BC, we concatenated the target relative attribute strengths to the inputs of both the policy and value networks. During training, we first evaluated the attribute strengths of the trajectories in the datasets, then obtained the rewards corresponding to each time step on the trajectories using the reward model, and finally used evaluated attribute strengths and rewards to train TD3BC.

Results. In our experiments, we trained TD3BC on the walker environment from DMControl, of which pre-defined attributes are (speed, stride, leg preference, torso height, humanness) (See 13.2 for more details of attribute definition and feedback collection). We selected three sets of target attribute strengths, representing objectives of making the walker move faster, have a larger stride, or have a higher torso. During training, we validated the algorithm’s performance on these three sub-tasks and showed the learning curve in 8. The results showed that using the attribute-conditioned reward model could indeed achieve multi-objective training. We further tested the trained model’s ability to switch behaviors. We ran the model for 1000 steps and adjusted the target attributes every 200 steps, recording the walker’s speed and torso height in 9. The results showed that the policy trained using the attribute-conditioned reward model could adjust its behavior based on the given target objectives, achieving the goal of multi-objective optimization.

5 Conclusion, Challenge, and Future Directions↩︎

To alleviate the difficulties of widely adopting RLHF solutions in applications, we propose a comprehensive system Uni-RLHF. We first implement the platform that supports multiple feedback types along with a feedback standard encoding format. Subsequently, we carry out large-scale crowd-sourced annotation creating reusable offline feedback datasets. Finally, we integrate mainstream offline reinforcement learning algorithms with human feedback, evaluate the performance of baseline algorithms on the collected datasets, and establish a reliable benchmark. Although We attempted to cover a wide range of factors in RLHF, there are some limitations. For example, we conduct large-scale experiments with only the most common Comparative Feedback and we lack deep analyses of human irrationality. We summarize some future research directions in 12.

Acknowledgments↩︎

This work is supported by the National Natural Science Foundation of China (Grant Nos. 92370132), the National Key R&D Program of China (Grant No. 2022ZD0116402) and the Xiaomi Young Talents Program of Xiaomi Foundation. The authors thank Peilong Han for his help in building and deploying the platform.

6 Environment and Datasets Details↩︎

In this section, we describe the details of all environments and corresponding tasks.

6.1 The Details of the Datasets for D4RL↩︎

Gym Mujoco tasks consist of the HalfCheetah, Hopper, and Walker2d datasets from the D4RL [43] offline RL benchmark. Walker2d is a bipedal robot control task, where the goal is to maintain the balance of the robot body and move as fast as possible. Hopper is a single-legged robot control task where the goal is to make the robot jump as fast as possible. Halfcheetah is to make a simulated robot perform a running motion that resembles the movement of a cheetah while trying to maximize the distance traveled within a fixed time period. We use the medium, medium-expert, and medium-replay datasets in our evaluations, which consist of (a mixture of different) policies with varying levels of optimality. We use the v2 version of the environment in our experiments.

AntMaze. involves controlling an 8-DoF ant quadruped robot to navigate through mazes and reach a desired goal. The agent receives sparse rewards of +1/0 based on whether it successfully reaches the goal or not. We study each method using the following datasets: large-diverse, large-play, medium-diverse, medium-play, umaze-diverse, and umaze. The difference between diverse and play datasets is the optimality of the trajectories they contain. The diverse datasets consist of trajectories directed towards random goals from random starting points, whereas the play datasets comprise trajectories directed towards specific locations that may not necessarily correspond to the goal. We used the v2 version of the environment in our experiments.

Adroit contains a series of continuous and sparse-reward robotic environments to control a 24-DoF simulated Shadow Hand robot to twirl a pen, hammer a nail, open a door, or move a ball. This domain was selected to measure the effect of narrow expert data distributions and human demonstrations on a sparse reward, high-dimensional robotic manipulation task. There are three types of datasets for each environment: expert, human, and cloned. We use the v1 version of the environment in our experiments.

6.2 The Details of the Datasets for Atari↩︎

We evaluate offline RLHF algorithms on five image-based Atari games (Pong, Breakout, Qbert, Boxing and Enduro), using the dataset released by D4RL_Atari⁴. The dataset for each task was categorized into three levels of difficulty: mixed denotes datasets collected at the first 1M steps, medium denotes datasets collected between 9M steps and 10M steps, and expert denotes datasets collected at the last 1M steps. We evaluate the algorithms using the medium difficulty dataset because the expert dataset only requires good imitation performance without reward function learning.

6.3 The Details of SMARTS Environments and Datasets Collection↩︎

Scalable Multi-Agent Reinforcement Learning Training School (SMARTS)  [49] is a simulation platform designed for deep learning applications in the domain of autonomous driving, striving to provide users with realistic and diverse driving interaction scenarios.

In this study, we selected three representative and challenging driving scenarios from the Driving SMARTS 2022 benchmark [69]. Each scenario corresponds to a distinct driving situation and demands specific driving skills for appropriate handling. Within these scenarios, the infrastructure is hierarchically organized into road, lane, and waypoint structures. At the highest level, the transportation network is structured as a graph where each node represents an individual road. Each road can further encompass one or multiple lanes, and subsequently, each lane is subdivided into a series of waypoints.

Upon environment initialization, given a start point and goal, the simulation environment generates a road-level global driving route for the agent. The timestep for these scenarios is uniformly set at 0.1s. At each timestep, the agent, using the global route, road features, and the dynamic states of the ego vehicle and neighboring vehicles, decides on actions that lead the ego vehicle to accelerate, decelerate, or change lanes. The environment then updates the states of both the ego vehicle and other vehicles in the environment based on the agent’s decision and other influential factors, such as the behavior of other vehicles, ensuring real-time dynamic information provision for the agent.

The specifics of the three selected scenarios within SMARTS are described as follows:

• cruise: The ego vehicle is tasked to travel on a three-lane straight road. During its journey, it must adjust its speed and change lanes when necessary to avoid potential collisions, ensuring a successful arrival at the end of a specified lane.

• cut-in: The ego vehicle is tasked to travel on a three-lane straight road where it encounters distinct challenges compared to the standard cruise scenario. Environmental vehicles in this setting may display more aggressive behaviors like lane-changing or sudden deceleration, aiming to obstruct the ego vehicle’s progression. As a result, the ego vehicle must adeptly navigate these challenges to ensure a safe arrival at its designated goal.

• left-c: The ego vehicle is tasked to commence its journey from a single-lane straight road, progress towards an intersection, and execute a left turn. Following the turn, it is expected to merge into a designated lane on a two-lane straight road.

Datasets collection. In each of the three driving scenarios, we employed online reinforcement learning algorithms to train two agents, each designed specifically for the respective scenario. The first agent demonstrates medium driving proficiency, achieving a success rate ranging from 40% to 80% in its designated scenario. In contrast, the second agent exhibits expert-level performance, attaining a success rate of 95% or higher in the same scenario. For dataset construction, 800 driving trajectories were collected using the intermediate agent, while an additional 200 were gathered via the expert agent. By integrating the two datasets, we compiled a mixed dataset encompassing 1,000 driving trajectories.

Task Reward Function Design. To direct the ego vehicle efficiently to its goal while prioritizing safety, We went through a number of tweaks and iterations, and eventually established a multifaceted reward function, marked as Oracle in 3.

This function is consistently applied across the three distinct driving scenarios. The function encompasses several components: distance-traveled reward, reach/near goal reward, mistake event penalty, lane-selection reward, and time penalty. The specifics of each component are detailed below and displayed in the ¿tbl:tab:smarts32reward32design?:

• Distance-Traveled Reward: This reward promotes the forward progression of the ego vehicle towards its goal. It is quantified by the distance the vehicle travels along the lane that directs it towards the goal.

• Reach/Near Goal Reward: This is a sparse reward, allocated exclusively at the terminal step. Should the ego vehicle successfully reach the goal, a reward of +20 is granted. Furthermore, if the ego vehicle concludes its task within a 20m proximity of the goal, a "near-goal" reward is dispensed, determined as the lesser value between \(dis\_to\_goal / 5\) and 10. This structure accentuates the importance of closely approaching the goal, fostering an active drive towards the destination.

• Mistake Event Penalty: This penalty addresses undesirable actions undertaken by the ego vehicle. Specifically, a significant penalty of -40 is levied if the ego vehicle collides with another vehicle. For individual transgressions, such as coming into contact with road edges or road shoulders, or instances of driving against the flow of traffic, a penalty of -6 is imposed.

• Lane-Selection Reward: This encapsulates both a reward and a counteracting penalty. The positive reward, valued at +0.3, is conferred whenever the ego vehicle operates within the lane that leads it to the goal, thus encouraging correct lane-changing conducive to the goal’s direction. Conversely, a penalty of -1 aims to inhibit unwarranted or frequent lane-changing actions, imposed on each instance the ego vehicle transitions between lanes.

• Time Penalty: To emphasize efficiency and motivate the vehicle to swiftly reach its destination, a consistent minor penalty of -0.3 is applied at every timestep.

7 Reward Model Structure and Learning Details↩︎

We implemented the structure of the three reward models with the following implementation details:

MLP: We use a three-layer neural network with 256 hidden units each using ReLU as an activation function. To improve the stability in reward learning, we use an ensemble of three reward models and constrain the output using the tanh function. Each model is trained by optimizing the cross-entropy loss defined in 2 using Adam optimizer with the initial learning rate of 0.0003.

CNN: We make the following changes to the MLP structure: 1) We process the image input using a 3-layer CNN with kernel sizes \((8, 4, 3)\), stride \((4, 2, 1)\), and \((32, 64, 64)\) filters per layer. 2) We empirically demonstrate that the tanh activation function exhibits instability in visual environments and is prone to gradient vanishing, so we do not use the activation function for the output. 3) We use a lower learning rate of 0.0001.

Transformer: For our experiments, we modified the publicly available codebase of Preference Transformer (PT)⁵ and re-implemented our transformer-based reward model. Unlike other structures, PT is better able to capture key events in the long-term behaviors of agents through a preference predictor \(P\left[\sigma^1 \succ \sigma^0\right]\) based on the weighted sum of non-Markovian rewards: \[P\left[\sigma^1 \succ \sigma^0 ; \psi\right]=\frac{\exp \left(\sum_t w\left(\left\{\left(\mathbf{s}_i^1, \mathbf{a}_i^1\right)\right\}_{i=1}^H ; \psi\right)_t \cdot \hat{r}\left(\left\{\left(\mathbf{s}_i^1, \mathbf{a}_i^1\right)\right\}_{i=1}^t ; \psi\right)\right)}{\sum_{j \in\{0,1\}} \exp \left(\sum_t w\left(\left\{\left(\mathbf{s}_i^j, \mathbf{a}_i^j\right)\right\}_{i=1}^H ; \psi\right)_t \cdot \hat{r}\left(\left\{\left(\mathbf{s}_i^j, \mathbf{a}_i^j\right)\right\}_{i=1}^t ; \psi\right)\right)}\] As shown in 1 (bottom right), PT consists of Causal Transformer and Preference attention layer components. In all experiments, we use causal transformers (GPT architecture) [70] with one layer and four self-attention heads followed by a bidirectional self-attention layer with a single self-attention head. Given a trajectory state-action segment \(\sigma\) of length \(H\), the embedding is generated by a linear layer. Then the embedding is fed into the causal transformer network and produces the embedding of the output \(\left\{\mathbf{x}_t\right\}_{t=1}^H\) where the \(t\) th output corresponds to \(t\) of the inputs. Then, the preference attention layer receives the hidden embedding from the output of causal transformer \(\left\{\mathbf{x}_t\right\}_{t=1}^H\) and generates rewards \(\hat{r}_t\) and importance weights \(w_t\). We define the output \(z_i\) as the attention weight values of the \(i\) -th query and the keys based on the self-attention mechanism. \[z_i=\sum_{t=1}^H \operatorname{softmax}\left(\left\{\left\langle\mathbf{q}_i, \mathbf{k}_{t^{\prime}}\right\rangle\right\}_{t^{\prime}=1}^H\right)_t \cdot \hat{r}_t\] Then \(w_t\) can be defined as \(w_t=\frac{1}{H} \sum_{i=1}^H \operatorname{softmax}\left(\left\{\left\langle\mathbf{q}_i, \mathbf{k}_{t^{\prime}}\right\rangle\right\}_{t^{\prime}=1}^H\right)_t\). Unlike the reward model with MLP structure, PT requires a trajectory as input to generate rewards. Following the codebase of PT, when predicting \(r_t\), we take the trajectory of length \(H\) up to time \(t\) as input and choose the corresponding last output as \(r_t\).

Hyperparameters for training transformer structure reward models are shown in ¿tbl:tab:transformer32hyperparameters?.

8 Additional Analysis and Evaluation Experiments↩︎

8.1 Additional Analysis and Visualisation of SMARTS Experiments↩︎

In order to evaluate the performance of the model from multiple perspectives, we have introduced two additional metrics: (1) the average driving speed of the vehicle, to assess the operational efficiency, which is the larger the better, and (2) the comfort level of the vehicle, characterized by the sum of the vehicle’s linear jerk (the rate of change of linear acceleration) and angular jerk (the rate of change of angular acceleration), which is the smaller the better.

Next, we provide visualization of an instance of the left turn scene in the SMARTS environment. 10 presents an instance of a left-turn-cross scenario in which the CS model succeeds, while the Oracle model fails due to a collision. In this scenario, the ego vehicle is required to start from a single-lane straight road, make a left turn at the cross intersection, proceed onto another two-lane straight road, and ultimately reach a goal located at the end of one of these two lanes. When navigating the intersection, if there is a stopped or slow-moving neighboring vehicle in front of the ego vehicle, awaiting its turn to pass through, the ego vehicle must decelerate to prevent a collision with the vehicle ahead. The first row of the 10 shows the driving trajectory of the ego vehicle controlled by the Oracle model. The second row displays the trajectory of the ego vehicle under the control of the CS model. The figure reveals that the Oracle model fails to timely decelerate the ego vehicle, resulting in a collision with the vehicle ahead, whereas the CS model opts to stop and wait for the vehicle in front, eventually passing through the intersection smoothly.

8.2 Analysis and Visualization of the Learned Reward Model↩︎

As shown in 11, we evaluate the quality of reward model in the training process. Experiments show that the reward model can align with human intent and achieve a high and stable prediction accuracy. The results in 12 demonstrate that the trained reward model is able to accurately predict step reward for each state and has a strong correlation with the carefully hand-designed reward function.

8.3 Ablation of Different Query Sampler in Offline RLHF↩︎

To better validate the role of different samplers, we collected new crowdsourcing labels and conducted the experiments to demonstrate the performance of different sampling methods. In each experiment, we collect \(N_\text{query}\) = 100 annotation labels (comparative feedback) using a different sampler separately and use IQL-MLP-CrowdSourced method for simplicity. Specifically, we consider the following query sampler:

• Random Sampler: Randomly select \(N_\text{query}\) pairs of trajectory segments from the dataset.

• Disagreement Sampler: Firstly, we generate \(N_\text{init} = 20\) pairs of trajectory segments randomly, then train the initial reward models and measure the variance across ensemble of reward models prediction. Then we select the most \(N_\text{iter} = 40\) high-variance segments. After two iterations, we obtain \(N_\text{query}\) = 100 annotation labels and the final reward model is trained.

• Entropy Sampler: We modify the customization interface to achieve the entropy sampler, which measure the entropy of preference prediction. The other experiments process is consistent with the disagreement sampler.

5 shows the normalized scores of IQL-MLP-CrowdSourced with various query sampler. We observe that a suitable label sampler could improve the task performance, this is also consistent with a similar conclusion by [16]. The disagreement sampler outperforms random sampling, however, the training phase requires multiple rounds of iterations, which introduces a little additional sampling cost.

8.4 Human Evaluation for Attribute Feedback Experiments↩︎

RLHF is most useful in domains where only the humans understand the good behaviour like humanness attribute. Therefore, we conducted human evaluation experiments for the agents trained by attribute feedback. We first collected 3 trajectories with different humanness, which were set to [0.1, 0.6, 1] and invited five human evaluators. The human evaluators performed a blind selection to judge which video is most human-like and which video is least human-like. If the result is consistent with the preset labels, it means that the agents can follow the humanness indicator well.

Finally, all people correctly chose the highest humanness trajectory, and only one person chose the lowest humanness trajectory incorrectly. The experimental results confirm that agents through RLHF training are able to follow the abstraction metrics well. We provide video clip in 15 and full videos on the homepage.

8.5 Evaluating Offline RL with Keypoint Feedback↩︎

We trained an keypoint predictor on keypoint feedback datasets in the Maze2d environments (see 11.4 for more training details). Then we adopted an keypoint-conditioned TD3BC approach, where the state of each step treats the keypoint as a subgoal. Compared to the original TD3BC, we concatenated the predicted goal to the inputs of both the policy and value networks. The results are shown in 6. TD3BC achieves a significant improvement over baseline because the agents are able to utilize the information provided by the keypoint as a subgoal.

9 Online RL Training with Real Human Feedback↩︎

As shown in 13, Uni-RLHF accommodates the online asynchronous reinforcement learning training process. Much of the previous work [7], [28], [29] assumes that online RL training and annotation (even with synthetic labels) are performed alternately, which can lead to low training efficiency for practical applications. Uni-RLHF builds asynchronous pipelines that allow training and annotation to occur simultaneously. It employs a sampler to extract data from the replay buffer, which is then stored in the feedback buffer as pending trajectory samples for annotation. Then various annotators can label these samples through the interface without interrupting the training process of the agents. When the number of new annotations reaches a predefined threshold, the reward function undergoes an update, thus relabelling the data stored in the replay buffer.

We validate the effectiveness of the online mode of Uni-RLHF, which allow agents to learn novel behaviors where a suitable reward function is difficult to design. We demonstrate the behaviour of multiple continuous front flip based on the walker environment as shown in 14. For enabling multiple front flip skill, we train the SAC [71] agent with online setting following [28]. First, we pre-train the exploration policy with an intrinsic reward for the first 2e4 steps to collect diverse behaviors. Collecting a greater diversity of trajectories helps to enhance the value of initial human feedback. Then, at every 1e4 steps, we randomly sample 20 pairs of trajectories from the replay buffer into the feedback buffer to wait for the annotation. The annotators are free to label without interfering with the walker training, and every 50 new labels will retrain the reward model and relabel the reward in the replay buffer. Finally, we give totally 200 queries of human feedback for walker front flips experiments and we observe that walker can master the continuous multiple front flip fluently. The other details and the hyperparameters are the same as [28].

10 Full results on the D4RL↩︎

We report complete average normalized scores on the D4RL datasets. The scores are from the final 10 evaluations (100 evaluations for the antmaze domain) with 3 seeds, representing mean ± std. “-r",”-m",“-m-r", and”-m-e" is short for random, medium, medium-replay, and medium-expert, respectively. IQL results are presented in 7, CQL results are presented in 8, and TD3BC results are presented in 9.

11 Details of Standardized Feedback Encoding Format↩︎

11.1 Comparative Feedback↩︎

We define a segment \(\sigma\) as a sequence of time-indexed observations \(\{\mathbf{s}_{k}, ..., \mathbf{s}_{k+H-1}\}\) with length \(H\). Given a pair of segments \((\sigma^0, \sigma^1)\), a human teacher gives feedback indicating which segment is preferred, so the preference \(y_\text{comp}\) could be indicated \(\sigma^0\succ\sigma^1\), \(\sigma^1\succ\sigma^0\) or \(\sigma^1=\sigma^0\), which represents equally preferred. We encode the labels \(y_\text{comp}\) as \(y_\text{comp} \in \{(1, 0), (0,1), (0.5, 0.5)\}\) and deposit them as triples \((\sigma^0,\sigma^1, y_\text{comp})\) in the feedback \(\mathcal{D}\). Subsequently, we treat the preference for trajectories as the training objective to optimize the reward function \(\widehat{r}_\psi\). Following the the Bradley-Terry model [54], we model human preference based on \(\widehat{r}_\psi\) as follows: \[\label{equ:comparative1} P_\psi[\sigma^1\succ\sigma^0] = \frac{\exp ( \sum_{t} \widehat{r}_\psi(\mathbf{s}_{t}^1, \mathbf{a}_{t}^1) )}{\sum_{i\in \{0,1\}} \exp ( \sum_{t} \widehat{r}_\psi(\mathbf{s}_{t}^i, \mathbf{a}_{t}^i) )},\tag{1}\] where \(\sigma^i\succ\sigma^j\) denotes the event that segment \(i\) is preferable to segment \(j\) and \((\mathbf{s}_{t}^i, \mathbf{a}_{t}^i) \in \sigma^i\). In order to align the reward model with human preferences, the update of \(\widehat{r}_\psi\) is transformed into minimizing the following cross-entropy loss: \[\label{equ:comparative2} \mathcal{L}(\psi, \mathcal{D})=-\mathbb{E}_{\left(\sigma^0, \sigma^1, y_\text{comp}\right) \sim \mathcal{D}}\Big[y(0) \log P\left[\sigma^0 \succ \sigma^1\right]+y(1) \log P\left[\sigma^1 \succ \sigma^0\right] \Big],\tag{2}\] where The term \(y(0)\) and \(y(1)\) represents the corresponding value in \(y_\text{comp}\). During the experimental process, the agent can be trained in parallel with the reward model, undergoing iterative updates. It is worth noting that \(y_\text{comp}\) can simply be extended to a finer-grained division of labels including the degree to which A is better than B [72]. For example, \(y_\text{comp}\) = (0.75, 0.25) represents that A is better than B with a smaller degree than (1, 0). Also, the expansion method can simply be trained along the 2 without any changes.

11.2 Attribute Feedback↩︎

First, we predefine \(\boldsymbol{\alpha} = \{\alpha_1, \cdots, \alpha_k\}\) represents a set of \(k\) relative attributes of the agent’s behaviors. Given a pair of segments \((\sigma^0, \sigma^1)\), The annotator will conduct a relative evaluation of two segments for each given attribute, which is encoded as \(y_\text{attr} = (y_1, ..., y_k)\) and \(y_i \in \{(1, 0), (0,1), (0.5, 0.5)\}\). Then we establish a learnable attribute strength mapping model \(\zeta^{\boldsymbol{\alpha}}(\sigma) = \boldsymbol{v}^{\boldsymbol{\alpha}} = [v^{\alpha_1}, \cdots, v^{\alpha_k}] \in [0, 1]^k\) that maps trajectory behaviors to latent space vectors, where \(v^{\alpha_i}\) indicates the corresponding attribute. Specifically, the probability that annotators believe that segment \(\sigma^0\) exhibits stronger attribute performance on \(\alpha_i\) is defined as follows: \[P_\theta^{\alpha_i}[\sigma_0\succ\sigma_1]=\frac{\exp\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_0)_i}{\sum_{j\in\{1,2\}}\exp\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_j)_i},\] where \(\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\tau)_i\) represents \(i\)-th element of \(\hat{\zeta}^{\boldsymbol{\alpha}}_\theta\). To facilitate the multi-perspective human intention of the attribute mapping model, we train \(\hat{\zeta}^{\boldsymbol{\alpha}}_\theta\) using a modified objective: \[\mathcal{L}(\hat{\zeta}^{\boldsymbol{\alpha}}_\theta, \mathcal{D}) = - \sum_{i\in\{1,\cdots,k\}}\sum_{(\sigma_0,\sigma_1,y_i)\in\mathcal{D}} y_i(1)\log P^{\alpha_i}[\sigma_0\succ\sigma_1]+ y_i(2)\log P^{\alpha_i}[\sigma_1\succ\sigma_0]\] Once such an attribute strength mapping model is established, we can normalize the latent space vectors \(\hat{\zeta}^{\boldsymbol{\alpha}}_\theta\) as \((\hat{\zeta}^{\boldsymbol{\alpha}}_\theta- \hat{\zeta}^\alpha_{\theta, \text{min}})/(\hat{\zeta}^\alpha_{\theta, \text{max}} - \hat{\zeta}^\alpha_{\theta, \text{min}})\). We then derive evaluations of the trajectories based on the target attributes \(\boldsymbol{v}_\text{opt}^{\boldsymbol{\alpha}}\) required for the task (relative strength of desired attributes, constrained between 0 and 1), simplifying it into a binary comparison: \[y(\sigma_0,\sigma_1,\boldsymbol{v}^{\boldsymbol{\alpha}}_\text{opt})=\left\{ \begin{array}{cc} (1,0), & \text{if } ||\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_0)-\boldsymbol{v}_\text{opt}^{\boldsymbol{\alpha}}||_2 \le ||\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_1)-\boldsymbol{v}_\text{opt}^{\boldsymbol{\alpha}}||_2\\ (0,1), & \text{if } ||\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_0)-\boldsymbol{v}_\text{opt}^{\boldsymbol{\alpha}}||_2 > ||\hat{\zeta}^{\boldsymbol{\alpha}}_\theta(\sigma_1)-\boldsymbol{v}_\text{opt}^{\boldsymbol{\alpha}}||_2\; \end{array} \right.\] Subsequently, we establish a conditional reward model \(\widehat{r}_\psi(\mathbf{s}_{t}, \mathbf{a}_{t}, \boldsymbol{v})\) using the pseudo-labels obtained from the transformation. We modified equations 1 and 2 to distill the reward model as follows: \[P_\psi[\sigma^1\succ\sigma^0|\boldsymbol{v}^{\boldsymbol{\alpha}}] = \frac{\exp ( \sum_{t} \widehat{r}_\psi(\mathbf{s}_{t}^1, \mathbf{a}_{t}^1, \boldsymbol{v}^{\boldsymbol{\alpha}}))}{\sum_{i\in \{0,1\}} \exp ( \sum_{t} \widehat{r}_\psi(\mathbf{s}_{t}^i, \mathbf{a}_{t}^i) )},\] \[\label{equ:attribute2} \mathcal{L}(\psi, \mathcal{D})=-\mathbb{E}_{\boldsymbol{v}^{\boldsymbol{\alpha}}\sim \mathcal{D}}\mathbb{E}_{\left(\sigma^0, \sigma^1, y_\text{comp}\right) \sim \mathcal{D}}\Big[y(0) \log P\left[\sigma^0 \succ \sigma^1\right|\boldsymbol{v}^{\boldsymbol{\alpha}}]+y(1) \log P\left[\sigma^1 \succ \sigma^0\right|\boldsymbol{v}^{\boldsymbol{\alpha}}] \Big].\tag{3}\]

11.3 Evaluative Feedback↩︎

Following the RbRL (Rating-based Reinforcement Learning) [58], we define \(n\) discrete ratings for trajectory levels, such as good, bad and average. Therefore we receive corresponding human label \((\sigma, y_\text{eval})\), where \(y_\text{eval} \in \{0, ..., n-1\}\) is level of assessment. Let return \(\hat{R}(\sigma) = \sum_{t} \widehat{r}_\psi(\mathbf{s}_{t}, \mathbf{a}_{t})\), we determine rating categories \(0=: \bar{R}_0 \leq \bar{R}_1 \leq \ldots \leq \bar{R}_n:=1\) divided by evaluation distribution. Based on the boundary conditions, we can estimate the probability \(P_\sigma(i)\) of each segment being assigned to the \(i\)-th rating, defined as: \[P_\sigma(i)=\frac{e^{\left(\tilde{R}(\sigma)-\bar{R}_i\right)\left(\bar{R}(\sigma)-\bar{R}_{i+1}\right)}}{\sum_{j=0}^{n-1} e^{\left(\tilde{R}(\sigma)-\bar{R}_j\right)\left(\tilde{R}(\sigma)-\bar{R}_{j+1}\right)}}\] Then we use multi-class cross-entropy loss to train the reward model: \[\label{equ:evaluative2} \mathcal{L}(\psi, \mathcal{D})=-\mathbb{E}_{\left(\sigma, y_\text{eval}\right) \sim \mathcal{D}}\Big[\sum_{i=0}^{n-1} y(i)\log p_\sigma(i)\Big].\tag{4}\]

11.4 Keypoint Feedback↩︎

We annotate key state time steps \(y_\text{key} = \{t_1, t_2, ..., t_n\}\) from segment \(\sigma\), where \(n\) is the number of keypoints. Subsequently, we employ this feedback to train a keypoint predictor \(g_\phi\) and use historical trajectories to inference key states, serving as guidance for the agent. Our predictor \(g_\phi(\cdot)\) aims to predict the most adjacent keypoints and is trained in supervised manner with a regression loss: \[\label{equ:keypoint2} \mathcal{L}(\phi, \mathcal{D})=-\mathbb{E}_{\left(\sigma, y_\text{key}\right) \sim \mathcal{D}}\Big[\sum_{t}\left\|g\left(\mathbf{s}_t\right)-\mathbf{s}_{t_\text{key}}\right\|_2\Big],\tag{5}\] where \(t_\text{key} \in y_\text{key}\) and \(\mathbf{s}_{t_\text{key}}\) is the keypoint closest to the current \(\mathbf{s}_t\). Then the policy are trained with inputs that consist of the states and keypoint predictions as an additional concatenation, marked as \(\pi_\theta(s_t, g\left(\mathbf{s}_t\right))\). As a result, the agents are able to obtain additional explicit goal guidance during training process.

11.5 Visual Feedback↩︎

We define the feedback as \((s_t, y_\text{visual})\), where \(s_t \in \sigma\) and \(y_\text{visual} = \{\text{Box}_1, ..., \text{Box}_m\}\). \(\text{Box}_i\) is a the bounding box position \((x_\text{min}, y_\text{min}, x_\text{max}, y_\text{max})\). Following the [40], we are able to convert the annotated visual highlights into a saliency map using a saliency predictor \(g_\phi\). There have been many saliency prediction model in the literature, e.g. PiCANet [73], Deepfix [74]. We can then use the human feedback labels to bootstrap the saliency predictor \(g_\phi(o_t)\). Given an RGB image observation \(o_t \in \mathbb{R}^{H \times W \times 3}\), saliency encoder map the original input to the saliency map \(m_t \in [0, 1]^{H \times W}\) which highlight important parts of the image based on feedback labels. There are several methods of aggregating additional saliency visual representations. The CNN encoder and the policy are jointly trained with inputs \((o_t, m_t)\) that consist of the RGB images and saliency predictions as an additional channel. Another possible approach is to use the RGB image \(\times\) predicted saliency map as an input. Utilising saliency information enables a more effective focus on relevant and important representations in the image tasks.

12 Some Challenges and Potential Future Directions↩︎

We outline some challenges and potential future directions as follows:

Performance Metrics: Although we performed evaluation experiments of the RLHF method, it is still evaluated using predefined reward signals, which may not be available in the application. How to quantify human intentions and explain real-world factors is an important research direction.

Estimations of Human Irrationality and Bias: Irrationality in human feedback can cause great difficulties in modeling feedback models [15], and future research may focus on reducing the negative effects of noise, e.g. noise label learning [75], and strong model constraints [76].

Imperfect Reward Correction: Modelling environmental reward through human feedback is easily caused by partially correct rewards [77]. How to second-boost reward modeling through reward shaping [78] and domain knowledge is an area ripe for research.

Reinforcement Learning with Multi-feedback Integration: When multiple feedback types are labeled, it is extremely interesting to jointly apply multiple feedback modalities in the same task or to compare different feedback performances.

13 Instructions for Crowdsourcing Annotators↩︎

In this section, we list the full instructions for each feedback mode for crowdsourcing. In order to improve the consistency of all participant’s understanding of the task objectives, for each task, the expert performed 5 annotations as examples and gave a detailed process of analysis. Note that in Atari environments, we also ask the annotators to play for at least ten minutes to better familiarize themselves with the task goals.

13.1 Comparative Feedback↩︎

The crowdsourced annotators will watch each pair of video clips and choose which one of the videos is more helpful for achieving the goal of the agent according to the following task description. If the annotators cannot make a deterministic decision, they are allowed to choose equally.

Options:

(1) Choose the left side of the video on the left is more helpful for achieving the goal of the agent.

(2) Choose equal if both videos perform similarly.

(3) Choose the right side of the video on the right is more helpful for achieving the goal of the agent.

13.2 Attribute Feedback↩︎

The crowdsourced annotators will watch each pair of video clips and determine the relative strength of the performance of the agents in each attribute based on the behavioral attribute definitions. Select the equal option if the annotators believes that the agents behave consistently on an attribute.

13.3 Keypoint Feedback↩︎

14 Details of Annotated Feedback Datasets↩︎

We establish a systematic pipeline of crowdsourced annotations, resulting in large-scale annotated datasets comprising more than 15 million steps across 30 popular tasks. Documentation of the datasets, including detailed information on environments, tasks, number of queries, and length of queries can be found in the following 10.

References↩︎