1 Introduction↩︎

In recent years, the realm of Large Language Models (LLM) [1]–[3] has seen rapid development, injecting vitality into the AI community while driving the advance in various downstream tasks [4]–[6]. Behind these advancements, Alignment techniques ensure that the behavior of LLMs adheres to human values [7], [8]. To reduce human involvement in the alignment process, Reward Models (RM) trained on human preference data can serve as proxies to provide reward signals for subsequent training, e.g., Reinforcement Learning from Human Feedback (RLHF) [9]. As a result, RMs have garnered increasing research interest in recent years [10]–[12].

Figure 1 illustrates an example of RM in the dialogue domain. The goal is to train an LLM-based chatbot following the “3H” principle (Honest, Harmless, and Helpful) [13]. Given two sampled responses generated by LLMs, the RM follows the instruction and ranks the responses according to the aforementioned three dimensions, then selects a better response by LLM-2 that aligned with human values (less harmfulness in this case), which can subsequently be used to optimize the policy model. The ranking process of the RM demonstrates interpretability and traceability. The task instruction, human input, response pairs, and the RM preference can be utilized to optimize the policy LLM in the RL stage.

In this paper, we focus primarily on parameterized RMs in the LLM era, which are used to reflect human preferences. Some surveys [7], [12] have involved the introduction of RMs (See Appendix 8.1 for more details). However, these works lack a systematic organization of RMs or do not include detailed and constructive discussions of RMs. To fill this gap, our main contribution can be summarized as: (1) We present the first comprehensive survey specifically focused on RMs in the LLM era; (2) We systematically review the related works in the field of RMs and introduce an elaborate taxonomy; (3) We discuss the challenges and future directions, which facilitate further research.

The organization of this survey is as follows: We first present the taxonomy of RMs (§2). This section involves preference collection (§2.1), reward modeling (§2.2), and usage (§2.3). Next, we introduce the applications (§3), evaluation benchmarks (§4). Finally, we discuss the challenges that remained in RMs (§5), and propose potential research directions (§6).

2 Taxonomy↩︎

2.1 Preference Collection↩︎

RMs can serve as proxies of humans, where the preferences can originate from different sources, including humans and LLMs. The details are introduced in this section.

2.1.1 Human Preference↩︎

Scaling up model parameters or training data does not guarantee improved alignment with human preferences [14]. In contrast, larger models may still produce hallucinations, harmful outputs, or unhelpful responses [15]. One straightforward approach is to train an RM on human preference data, which subsequently serves as a proxy to provide the training signal during the reinforcement learning phase. Some methods employ human annotators [9], [16] to label pairs of trajectories produced by the interaction between the policy model and the environment. Other works [17] leverage annotators to assign labels to response pairs from LLMs or humans following the collected prompts [18]. On this basis, improving the efficiency and quality of collection requires further investigation.

2.1.1.1 Efficiency.

Some studies have introduced active learning [19] into preference collection. For example, [20] and [21] use an objective of information gain to choose queries. [22] adopts entropy-based sampling methods to select segment pairs. In addition, some approaches [23], [24] leverage data augmentation and sequential pairwise comparison to achieve preference-efficient learning.

2.1.1.2 Quality.

Some works aim to improve the quality from the perspective of annotators, including the introduction of demonstrations [16], active annotator selection [25], user-friendly interfaces [26], [27], and fine-grained goals and rules [17], [28], [29]. Meanwhile, other works focus on the quality of sampled queries, such as selecting diverse batch samples [30], [31] or adopting online collection settings [32] to prevent distribution shift.

2.1.2 AI Preference↩︎

Although collecting preference data from trained human annotators is intuitively suitable for human preference alignment, the high costs [33] may limit its practicality. As the capabilities [34] of LLMs continue to advance, they have demonstrated a high degree of consistency with human judgment [35]. Besides, when AI systems surpass humans in some tasks [36], [37], it’s hard for humans to evaluate the complex behaviors produced by superhuman models [38]. Therefore, AI preferences have garnered increasing research interest and have the potential to become an alternative to human preferences [39].

[10] first introduce RL from AI Feedback (RLAIF) for training a helpful and harmless AI assistant in conversation scenarios, where the RM is trained on a combination of LLM-generated harmlessness preference labels and human-generated helpfulness preference labels. [40] trains an RM on the synthetic comparisons, the quality of which is determined by the model size, the number of in-context shots. [35] directly utilizes off-the-shelf LLMs to provide reward during RL, which can address the out-of-distribution issue between the sampled trajectories from the initial policy and the dataset on which RM trained.

Similar to human preference collection, some subsequent studies attempt to collect scaled and high-quality AI preference pairs. [41] and [42] construct instruction templates to elicit preference. Various LLMs in the model pool are used to generate and evaluate the completions for the instructions. [43] introduces human-defined principles to achieve instructable RM. Other works further integrate AI preferences with human preferences. [44] and [45] enable LLMs to generate synthetic critiques for completions pairs to enhance RMs. In addition, [46] combines LLM-generated responses and human-annotated negative samples to mitigate the problems of noisy positive samples [47].

=[ rectangle, draw=hidden-black, rounded corners, text opacity=1, minimum height=3em, minimum width=5em, inner sep=2pt, align=center, fill opacity=.5, ] =[ my-box, minimum height=1.5em, fill=hidden-blue!90, text=black, align=left, font=, inner xsep=2pt, inner ysep=4pt, ]

2.2 Reward Modeling↩︎

Reward modeling plays a central role in the alignment of LLMs, especially as a foundational component in reinforcement learning frameworks. RMs have been widely adopted in reinforcement learning research as substitutes for directly using environment rewards [48]. They are particularly relevant to inverse reinforcement learning, which focuses on inferring an agent’s underlying reward function from observed trajectory data [49].

2.2.1 Reward Model Type Level↩︎

In this part, we mainly discuss several reward modeling mechanisms of the RMs according to the underlying model types (Figure 3). Following the taxonomy introduced in [50], [51], the mechanisms include discriminative reward, generative reward, and implicit reward.

2.2.1.1 Discriminative Reward.

Discriminative RMs include a base model and an MLP-based reward head (classifier), which outputs a scalar reward for the given input. Sequence Classifiers (Figure 3 (a)) belong to Discriminative RMs, which model the preference for a single response. For example, [52] proposes conditional RM that incorporates preference data across different domains by leveraging conditional system prompts. [53] introduces absolute rewards for actions to augment the Bradley-Terry (BT) model [54] which is well adapted for the binary comparison task. [55] regularizes the hidden states to improve the generalizability of RMs on out-of-distribution (OOD) data.

Another type of Discriminative RMs is Custom Classifiers (Figure 3 (b)), which take comparison pairs as input or output multiple scores. [56] compares each pair of candidates in the pool and define several scoring functions to select the best candidate. [57] optimizes the ensemble of existing metrics to align with human preferences. [58] and [59] leverage multi-objective rewards for modeling diverse preference. In addition, [59] further uses a gating layer to adaptively allocate suitable objectives to the task.

2.2.1.2 Generative Reward.

Unlike discriminative models, generative reward models (Figure 3 (c)) fully leverage the generative capabilities of LLMs to provide preference scores. Some works use general models [60] or train specialized models [61]–[65] to serve as judges, which are available to generate better options of comparison pairs or rate a single response in text format. [66] and [67] extract the next-token probability of the answer indicators as scores. [68] utilizes a trained generative reward model to rewrite the origin response under the minimum editing constraint. The token-level scores can be obtained by contrasting the response pairs. In addition, the Self-Instruct [69] technique can be used to optimize generative reward models. Some works [70]–[72] iteratively train the model with constructed contrasting synthetic preference pairs, reasoning traces (optional), and generated judgments. Generative reward models can be integrated with other LLM-related technologies like Chain-of-Thoughts (CoT) [73] and Retrieve-Augmented Generation (RAG) [74], thereby endowing them with the potential to be applied across broader tasks.

2.2.1.3 Implicit Reward.

Different from explicit RMs, recent studies construct reward-related variables through weaker optimization signals (The variable \(z\) as shown in Figure 3 (d)) to reduce resource costs. DPO [75] and SLiC-HF [76] eliminates explicit reward modeling by defining implicit rewards through generation probabilities, directly optimizing human preference pairs. [77] proves these implicit rewards’ value functions analytically continue their explicit counterparts, enabling automated credit assignment in LLMs. Some subsequent studies aim to improve the robustness of the models. From the perspective of preference data, the design of appropriate data sampling, selection, and filtering strategies [78]–[83] can be utilized to address the quality and distribution issues of preference dataset. Some works [84], [85] attempt to effectively optimize the target policies from multiple responses, while [86] proposes direct reward optimization on single-trajectory data. Other works focus on preference corruption [87], [88] or the preference distribution shift problem [89]. From the perspective of modeling mechanism, recent techniques like token-level optimization [90], [91], reference-free methods [92]–[94], self-play optimization [95]–[97] exhibit practical potential. It should be noted, however, that these methods generally underperform in reward modeling itself compared to explicit optimization results [51].

2.2.2 Reward Granularity Level↩︎

In this subsection, we classify reward mechanisms based on their granularity when functioning as verifiers to solve problems with ground truth. Specifically, an Outcome-level Reward Model (ORM) predicts the probability that a completion results in a correct answer, while a Process-level Reward Model (PRM) assigns a score to each step in the reasoning process.

2.2.2.1 Outcome Level Reward.

For tasks that require more complex reasoning, an ORM can be employed [50], [55], [98]. Typically, the training data for an ORM is constructed differently from standard preference tuning [54]. Specifically, each solution \(s\) is paired with a problem statement or prompt \(p\). The inductive bias appliedin this setup assumes that one completion represents a solution based on whether it is correct or not for the given problem. The ORM \((P \times S \rightarrow \mathbb{R})\) is usually trained using a corss-entropy loss [99], [100] \[\mathcal{L}_{ORM} = -( \hat{y_s} \log{y_s} + (1 - \hat{y_s}) \log{(1-y_s)})\]

2.2.2.2 Process Level Reward.

Despite their proficiency in multi-step reasoning tasks, outcome-supervised methods are still prone to hallucinations, such as reaching the correct answer through an incorrect reasoning path [101]. This indicates the necessity of incorporating process supervision to address these limitations. Additionally, the PRM \((P \times S \rightarrow \mathbb{R^{+}} )\) can be trained using the standard classification loss function below, where \(y_i\) is the prediction score of the PRM and \(\hat{y_i}\) represents the correctness label, and \(N\) is the total number of reasoning step for \(s\). \[\mathcal{L}_{PRM} = - \sum_{i=1}^{N} \hat{y_{s_i}} \log{y_{s_i}} + (1 - \hat{y_{s_i}}) \log{(1-y_{s_i})}\]

Different from the heuristic method [99] that leverage semantically relevance for stepwise verification, [102] introduces the PRM which are trained on human annotated stepwise labels. The PRM evaluates each reasoning step individually and can reduce tracking error and avoid tampering incentives [103]. Moreover, [104] constructs a large-scale stepwise human feedback dataset PRM800K and train a PRM to predict step correctness in the form of tokens.

To further reduce the cost of human annotation, [105] and [106] obtain process-supervised signals based on the Monte Carlo (MC) method. For each step and prefix, the frequency of reaching correct answer within sampled completions can be used to estimate step quality, i.e., Q-value function [107]. Expanding on them, [108] employs an adaption of Monte Carlo Tree Search (MCTS) to construct state-action trees for collecting the PRM training data. In addition, [109] proposes a stepwise discriminator through contrastive learning, where the preference pairs are obtained by aligning LLM-generated incorrect solutions with the reference solution.

Another series of works argue process reward should measure progress and advantages. [110] frames the PRM as a ranking problem to capture inter-dependencies among reasoning steps, which means larger Q-value is more likely to reach correct answer, and a significant gap will exist between correct steps and the first incorrect step. [111] and [112] introduce advantages as PRMs to measure the confidence and likelihood change of generating a correct response. Furthermore, [113] and [114] obtains implicit PRMs from trained ORMs through reward parameterization, which can be used to estimate advantages by calculating token-level rewards.

2.2.2.3 Pros and Cons of different types of RMs.

Currently, ORM tends to be better than PRM in the tasks with flexible processes due to its ease of implementation and generalizability, but it may lead to false positives solutions [101] in the reasoning tasks. PRM has demonstrated its potential in reasoning tasks [108], [115], but there are several considerations that require attention. Manual annotation is expensive and not scalable [116], while automated annotation may not produce satisfactory results. [117] finds that MC estimation hinder the capability of PRMs to identify incorrect steps compared to judge LLMs. Besides, process rewards are difficult to define [114], determining the correctness of intermediate steps and the progress of solving problems is challenging. Moreover, it is often suffers from reward hacking [118], while retraining the RM introduces additional complexity and resource requirements. Finally, although PRM excels at reranking top-N responses or assisting in guided search [119], its computational overhead in large-scale reinforcement learning tasks outweighs its benefits in practical experiments [120]. An overview of the opinions is in Table 1.

2.3 Usage↩︎

In the context of LLMs, RMs serve as critical components that help guide model behavior toward desired outcomes. By defining a structured, quantifiable signal that measures how well a generated response aligns with specific goals or user preferences, RMs enable the tuning and optimization of LLM outputs. This RM utility manifests across multiple stages of the LLM life cycle, including data selection, policy training, and the inference stage. In this subsection, we investigate RM utility from these three perspectives in detail.

2.3.0.1 Data Selection

Some studies utilize RMs to select data for the fine-tuning of LLMs. [121] proposes an SFT-like iterative training method, where an RM is utilized to rank the quality of LLM-generated responses. Data with the highest reward can be used to finetune the LLM. [122] further introduces ranking loss to align the LLM-generated score with the RM-generated score. [123] leverages an RM-filtered dataset to fine-tune LLM towards an offline RL objective. [124] evaluates answers and rationale for correctness by RMs, thereby selecting peference pairs to optimize LLMs via DPO [75] objective.

2.3.0.2 Policy Training.

RMs provide feedback signals that reinforce or penalize certain behaviors [14], ultimately shaping the model’s decision-making policies. To mitigate the issue of low robustness, which arises primarily because the RM often struggles with out-of-distribution generalization [125] and mismatched human judgment, several strategies have been investigated. These include length-controlled reward setting [126]–[128], causal reward modeling [129], [130], Bayesian method [131]–[133], and ensemble [134]–[136].

2.3.0.3 Inference.

RMs can be used to rank multiple outputs to deliver responses that best align with application-specific criteria. As discussed in §2.2.2, RMs can be classified as ORM and PRM. PRMs are often used at the inference stage to evaluate the progress and improve reasoning ability [112]. Some RM-guided tree search frameworks [115], [137]–[139] which have been shown to be able to greatly enhance the reasoning abilities of LLMs. In addition, RMs can also be used to evaluate intermediate decoding steps and dynamically decide whether to invoke a more powerful target model to balance resource utilization and performance [140].

3 Applications↩︎

RMs have found extensive applications across multiple domains. Here, we briefly summarize some key areas where RMs are currently utilized.

3.0.0.1 Dialogue.

RMs help mitigate harmful responses by refining them based on ethical guidelines and user intent [10], [15], [141]–[143]. Meanwhile, some works focus on the professionalism [144], [145] in dialogue, requiring agents to accurately and clearly express complex knowledge. Other works attempt to improve the overall dialogue impression [146], [147], including empathy, enthusiasm, humanlikeness, and so on.

3.0.0.2 Reasoning.

In mathematical reasoning [4], [102], RMs, especially PRM, can provide guidance to LLMs to improve logical consistency by balancing the exploration of various solutions with minimizing errors [104], [105], [108], [148]–[151]. Additionally, RMs have also shown promise in code generation [152] by integrating API calls, improving learning efficiency, and optimizing performance [64], [115], [153]–[156].

3.0.0.3 Retrieve & Recommendation.

RMs can be employed to help align the retrieve process with the preferences of strong LLMs  [157], which include assessing relevance [158], [159], adaptive retrieval [160], and improving the quality of intermediate queries [161]. As for recommendation systems, RMs can be used to capture nuanced user preferences [162], evaluate LLM-generated user preferences [163], and lead to high-quality explanations [164].

3.0.0.4 Other Applications.

Apart from aforementioned applications in the text domain, RMs have demonstrated potential in other modalities, such as text to audio [165]–[167], text to image [168]–[170], text to video [171]–[173]. Moreover, RMs have been explored in some interactive tasks including robotic manipulation [174], [175] and games [176], [177], which become the foundation of artificial general intelligence.

4 Benchmarks↩︎

RM evaluation is crucial because errors in RM can negatively affect the performance of the final policy [178]–[180]. However, the development of general and standardized benchmarks for RM evaluation remains nascent, making it hard to compare and improve RMs. This is due to several challenges: (1) The most direct way to evaluate an RM is to train a full RL policy and observe its performance, which is very costly [178]. (2) RM evaluation is often tied to the performance of the policy trained with it, making it difficult to assess the RM independently [51]. (3) While creating a dataset for evaluation (e.g., annotating a simple pairwise comparison dataset) is relatively easy, RMs are sensitive to changes in input style, domain, or format [181]. This means RM evaluation requires a more comprehensive approach, considering constructing more dynamic, multi-faceted testing, which further compounds the difficulty. Recently, researchers have tried to construct high-quality benchmarks to explore optimizing RMs within different RL policies, LM architectures, training budgets, etc.

4.0.0.1 ORM Benchmarks.

[51] constructs a comprehensive benchmark RewardBench, which contains human-verified prompt-chosen-rejected trios spanning chat, reasoning, safety, and prior test sets, meanwhile providing a toolkit to audit RM behavior. [181] proposes RM-Bench, which includes chat, code, math, and safety annotated data, and conducts large-scale evaluation on publicly accessible RMs. [182] introduces RMB that involves over 49 real-world scenarios, and discusses the generalization defects in previous benchmarks. Specifically, [178] proposes PPE that evaluate RMs on proxy tasks (related to downstream RLHF outcomes) by launching an end-to-end RLHF experiment.

4.0.0.2 PRM Benchmarks.

With the emergence of reasoning research, LMs are adapted to more complex scenarios like math and multi-hop decision-making tasks, therefore PRMs have appeared and been applied. For evaluating PRMs, [183] propose ProcessBench, which consists of a huge number of cases with annotated step-by-step solutions on competition math problems. [116] introduce PRMBench, comprises thousands of designed problems with stepwise labels, evaluating RMs across multiple dimensions.

In addition to aforementioned studies, some recent works evaluate RMs for specific domains or applications, e.g., Vision-Language [184]–[186], Multilingual Settings [187], and Retrieve-Augmented Generation [188]. These benchmarks collectively mitigate the need for a more comprehensive and fine-grained evaluation of RMs, paving the way for more reliable and robust RMs for training stronger LMs.

5 Challenges↩︎

5.1 Data↩︎

High-quality data collection to reflect human preference is the cornerstone of RM applications, but there still remains some challenges associated with its process. During the collection, potential biases may exist between the preferences of researchers and annotators [189]. Variations in expertise among annotators can introduce noisy data [25], [190], which may be significant in some knowledge-intensive tasks. The issue of assessment quality can result in the inconsistencies [191] between sparse feedback protocols (i.e., ratings and rankings), when dense feedback is expensive to collect. To tackle above challenges, data filtering [192], selection [193], and high-fidelity scalable synthetic data [194] become promising solutions.

5.2 Training↩︎

A critical challenge in RM training is overoptimization which also called reward hacking [195]–[198], where RMs could be excessively optimized to narrow evaluation metric (like accuracy on a sole static benchmark) [179]. An RL policy trained against such RMs may “hack” the reward signal, leading to performance degradation [199]. Some causes of overoptimization include reward tampering [200], [201], mislead [202], and sycophancy [203]. As mentioned in §2.3, several research directions such as RM ensemble [204], data augmentation [130], and robust training [198], [205], [206] have demonstrated potential in mitigating overoptimization, paving the way for more robust RMs.

5.3 Bias in Evaluation↩︎

Using RMs (judge model) for evaluation also introduces intrinsic biases toward superficial quality of text [207]. [208] observe that top-ranking RMs and some popular benchmarks exhibit biases toward the specific format patterns [209] discuss the biases derived from evaluators, including length, concreteness, empty reference, and so on. [210] study the preference leakage problem elicited by the relevance between synthetic data generators and RMs. The aforementioned studies highlight the need to construct robust evaluation benchmarks to detect and mitigate biases.

6 Future Directions↩︎

6.0.0.1 The combination of scalar rewards with rule-based rewards is becoming a growing trend.

In advanced industrial LLMs [120], [211], a robust model can benefit from integrating rule-based and model-based rewards. Rule-based rewards provide clear guidelines, while model-based rewards enable learning from predictions. Specifically, rule-based rewards are applied to tasks with clear ground truths (e.g., mathematics, coding), while reward models are used for tasks without clear ground truths (e.g., creative tasks), enhancing LLMs’ real-world applicability. Incorporating rule-based rewards has become a standard practice in the reinforcement fine-tuning of o1-like [212] longCoT models, and a few works [213]–[215] in the academic community which only utilize rule-based reward have emerged, also achieving strong reasoning capabilities.

6.0.0.2 Reward Design in LLM Long-horizontal Agent Task.

Recent advances in reasoning ability have enabled sophisticated LLMs to tackle complex expert-level tasks [216], with planning playing a key role. OpenAI and Anthropic are exploring tool use, such as search engines [217], code interpreters [218], and web browsers [219] to complete complex GUI tasks [220]. However, ensuring good agent performance is challenging, especially when designing feedback mechanisms for large systems. Creating rules is experimental, and developing an end-to-end reinforcement learning framework for long-horizontal tasks is essential. The key challenge remains ensuring the agent consistently receives rewards and improves monotonically.

6.0.0.3 Empowering the multi-modal domain.

RMs are rapidly evolving in the multi-modal domain, which includes the integration of modalities such as image, audio, and video. Compared to single-modality, the collection of multi-modal preference data is more costly. Some techniques such as few-shot learning [221], data synthesis [222] remain to be explored, thereby reducing the reliance on human annotators. Meanwhile, designing a high-quality reward signal [223] is crucial, which involves alignment across different modalities. Finally, exploring methods to enhance the cross-domain generalization of RMs, and bridging the gap between simulated and real-world scenarios, will contribute to the realization of embodied intelligence.

7 Conclusion and Discussion↩︎

In this paper, we present the first comprehensive survey specifically focused on Reward Models in the LLM era. We systematically review related studies of RMs, introduce an elaborate taxonomy, discuss the practical applications, highlight the challenges, and explore potential research directions. Besides, we discuss some open questions about RMs. (1) Is Rule-based reward enough for RL? (2) Is Mixture-of-Experts better than BT Model? (3) How to overcome the reward hacking of RM as LLMs surpass the level of the best expert level? See Appendix 8.4 for more details. We hope that this survey will be helpful to researchers and facilitate further research.

8 Appendix↩︎

8.1 Relevant Survey↩︎

Some previous surveys focus on human-involved RL [224]–[226], while [227] discusses LLM-enhanced RL. [7] and [228] conducts a comprehensive investigation on LLM alignment. [11] and [12] both focus on RLHF, while [11] discusses the researches in which RM is the sore source of information for the objective. [12] overviews the open problems and limitations of RLHF.

Compared with the aforementioned survey, our work primarily focuses on RMs in LLM era. We systematically introduce RMs based on their life-cycles, and explain the popular usages and evaluation perspectives. In addition, we discuss the challenges and potential research directions of RMs in detail. We sincerely hope that this paper can deepen researchers’ understanding of the field and facilitate future works.

8.2 Reward Modeling↩︎

The Bradley-Terry Model [54] can be used for modeling pairwise preference, which is the most commonly reward model assumption. For a prompt \(x\), reward model \(r\), response pair \(y_w,y_l\). It estimates the probability of prefer to \(y_w\) rather than \(y_l\):

\[P(y_w \succ y_l|x) = \frac{1}{1+exp(r(x,y_w)-r(x,y_l))}.\]

An RM \(\widehat{r}\) can be derived by optimizing the following maximum likelihood objectives, where \(\mathcal{D}\) and \(\sigma\) represent the preference dataset and sigmoid function respectively.

\[\widehat{r}\leftarrow\mathop{\arg\max}_{r\in\mathcal{R}}\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma(r(x,y_w)-r(x,y_l))\right].\]

Under RLHF setting [14], the target policy model is optimized by using the learned RM \(\widehat{r}(x,y)\). \(\pi_{\mathsf{ref}}(x,y)\) represent the reference model before update, and the resulting Kullback-Leibler (KL) penalty term is utilized to constrain the size of the policy update [229]:

\[\widehat{\pi}\leftarrow\mathop{\arg\max}_{\pi\in\Pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot|x)}\left[\widehat{r}(x,y)-\beta\log\frac{\pi(x,y)}{\pi_{\mathsf{ref}}(x,y)}\right]\]

DPO [75] is an alternative alignment approach which can optimize the policy without explicit reward modeling: \[\widehat{\pi}\leftarrow \mathop{\arg\max}_{\pi\in\Pi}\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right],\]

where \(\beta\) is a scalable parameter.

8.3 Reward Shaping & Ensemble↩︎

A major challenge in real-world scenarios is the sparsity and delay of rewards, which can hinder learning. This section focuses on engineering the reward model [230] during reinforcement learning .

8.3.0.1 Reward on Point-wise Feedback

Pointwise feedback assigns numerical values to actions or outcomes, enabling precise adjustments to the agent’s policy. It is effective for tasks where each action’s quality can be independently assessed. For example, [231] and [232] propose a self-training strategy to select the best and worst reward samples. [47] addresses ambiguous preference pairs by incorporating a margin in the reward, improving model generalization. [233] employs a data-centric approach to enhance feedback quality and make reward models more effective.

8.3.0.2 Reward on Binary Feedback

Binary feedback simplifies evaluation by categorizing outcomes as positive or negative, eliminating the need for a ground truth. This makes implementation and interpretation easier. For instance, Nash learning [234] models pairwise preferences by binary feedback but struggles with inconsistent human labeling. Approaches like KTO [235] use the Kahneman-Tversky model [236] to maximize utility, and DRO [237] combines offline reinforcement learning with regularization in binary feedback. Binary feedback also guides agent learning by signaling desirable actions, as explored in [238]. However, it may not capture the full complexity of human preferences.

8.3.0.3 Reward on Ensemble Feedback

Model ensemble [239] is a classic machine learning method for mitigating reward overoptimization and improving policy optimization. Typically, ensemble feedback [134], [135], [204], [231] aims to combine reward signal to further reduce reward hacking during reinforcement fine-tuning. For computational efficiency, [136] propose a LoRA-based ensemble method that reduces the computational cost associated with reward ensembles. Additionally, reward ensemble techniques, such as the Bayesian ensemble method [133], can be used to approximate uncertainty in the feedback.

8.4 Open Questions↩︎

8.4.0.1 Is Rule-based reward enough for RL?

Rule-based rewards are a good way to mitigate reward hacking, but it’s hard to say whether they are enough on their own. Without sufficient supervision, large language models (LLMs) may encounter very sparse rewards, leading to optimization divergence. Additionally, for tasks that don’t have a clear ground truth, designing an effective rule-based reward can be challenging. In such cases, preference learning can be a better option, as it allows us to derive reward signals from comparative feedback rather than relying solely on predefined rules. Thus, while rule-based rewards can be helpful, they may not always provide the necessary robustness for complex tasks.

8.4.0.2 Is Mixture-of-Experts better than BT Model?

There are several works related to Mixture-of-Experts (MoE) models, such as the DMoERM model [240] and LoRA-ensemble [241], [242]. MoE models have shown great potential in creating Pareto-optimal [243], [244] reward models, where they can combine multiple expert models to focus on different aspects of the problem, offering a more versatile and efficient approach. While the BT model has its strengths, MoE models have the advantage of scalability and the ability to improve performance by selecting the most relevant expert for each situation. This flexibility often leads to better generalization and optimization, especially in complex tasks.

8.4.0.3 How to overcome the reward hacking of RM as LLMs surpass the level of best expert level?

As LLMs surpass the performance of the best expert models, overcoming reward hacking becomes more challenging. One approach is to shift from weak-to-strong generalization [245]. This involves designing reward models that encourage more robust, flexible learning that accounts for a wider variety of potential behaviors and outcomes. Instead of relying solely on expert-level feedback, incorporating broader, more generalized reward signals helps ensure that the system doesn’t exploit narrow solutions or hacks. This strategy promotes more meaningful generalization and prevents the model from exploiting loopholes in the reward structure.

8.5 Evaluation Aspects↩︎

According to the benchmarks introduced in (§4), the evaluation aspects of RMs can be summarized mainly as follows:

8.5.0.1 Consistency.

The aim of RMs is to provide preference signals to LLMs, thus consistency is the primary evaluation aspect for RMs. Furthermore, consistency can be divided into: (1) the alignment between RMs and human preferences, the RMs are required to distinguish between chosen and rejected samples [51], [181], [182] , or identify the correctness of samples directly [183];(2) the alignment between RMs and policy models, such as style-controlled correlation [181] and downstream task correlation [178], [184]

8.5.0.2 Robustness.

On the basis of consistency, RMs should exhibit robustness across the experimental settings and tasks. [179] rewrite the prompts in the RM test dataset to investigate the influence of the prompt semantic bias. In PRM evaluation, [116] requires LLMs to be sensitive to the details of reasoning, including subtle conditions, deception, and multiple solutions.

8.5.0.3 Safety.

Similar to the consistency evaluation, [51] and [181] evaluate RM’s ability to distinguish between safe and unsafe responses. [182] conducts trade-off analysis between the goals of helpfulness and harmlessness.

References↩︎