1 Introduction↩︎

Large Language Models (LLMs) have achieved remarkable success across a wide spectrum of tasks, demonstrating unprecedented capabilities in natural language understanding and generation [1]–[4]. Spearheaded by DeepSeek R1 [5], Reinforcement Learning with Verifiable Rewards (RLVR) have demonstrated the overwhelming advantages of Reinforcement Learning (RL) by achieving superior performance across numerous tasks [6]. As a crucial component in RL frameworks, reward modeling is essential for generating accurate reward signals for LLM responses.

However, existing RLVR methods primarily focus on mathematical and coding tasks, where objective and deterministic validation mechanisms can be readily established to generate rewards. This limitation significantly hinders the scalability of RL approaches to broader abstract and subjective tasks, such as role play or creative writing, which lack golden standard answers or reliable verifiers [7]. While researchers have attempted to address this challenge by training reward models to provide feedback signals, a fundamental difficulty persists: In subjective tasks, the quality differences between responses are often subtle and highly subjective, making traditional reward modeling approaches inadequate for capturing nuanced human preferences.

The core challenge lies in the inherent limitations of existing reward modeling paradigms for subjective tasks. Traditional reward models primarily learn pairwise preference orderings using Bradley-Terry models [8], but fail to capture the magnitude and confidence of quality differences between response pairs [9]. To address this limitation, some studies have attempted to incorporate additional margin annotations to help models learn quality disparities between samples. However, this approach significantly increases annotation burden [10]. More critically, for abstract and creative tasks, margin annotation becomes extremely subjective and unreliable, as quality assessments are highly contextual and difficult to quantify objectively [11]. For example, Figure 1 presents three LLMs portraying “Sasuke Uchiha” from “Naruto” in a conversation with “Orochimaru”, each generating a distinct response. Selecting preference pairs from such samples is difficult. This motivates us to ask: Can we develop a approach to reward modeling that naturally adapts to the inherent uncertainty in subjective task?

To address the above challenges, we propose Act-adaptive Reward Modeling (AAM), which fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. Our key insight is that LLMs themselves can serve as implicit confidence estimators, and the probability ratios in RLHF objective naturally reflect the reward model’s certainty about preference judgments. AAM transforms these ratios into adaptive margins that automatically adjust optimization intensity based on preference confidence. For subjective tasks where preferences are ambiguous and confidence is low, AAM reduces potential interference; for cases with clear quality distinctions and high confidence, it amplifies optimization strength. This self-adaptive mechanism eliminates the need for explicit margin annotations while providing more nuanced control over the learning process in subjective tasks.

We validate AAM for subjective reward modeling, specifically where Bradley-Terry (BT) models struggle with preference ambiguity. AAM enhances Bradley-Terry reward models by 2.95%, achieving 91.6 and 68.1 on RewardBench and JudgeBench respectively. More significantly, in subjective role-playing tasks, AAM demonstrates substantial improvements of 4.85% over naive Bradley-Terry approaches, highlighting its superior capability in modeling ambiguous reward signals. In downstream alignment, AAM-GRPO-32B achieves state-of-the-art results on CharacterEval and Charm, outperforming both Claude-3.5-Sonnet and Doubao-Pro-Character.

The main contributions of this paper can be summarized as follows:

• We propose AAM (Act-Adaptive Margin), a novel approach that dynamically calibrates preference margins to address subjective reward modeling challenges where traditional Bradley-Terry models struggle with ambiguous preferences.

• We demonstrate significant improvements in reward modeling: 2.95% on general tasks and 4.85% on role-playing tasks.

• We achieve state-of-the-art results on role-playing alignment, outperforming leading closed-source models (Claude-3.5-Sonnet) and specialized character models on CharacterEval and Charm benchmarks.

• We release Charm dataset with 1,108 characters and 16,888 bilingual dialogues, along with the several evaluation benchmarks for subjective reward modeling research.

2 Related Works↩︎

In this section, the prior work is divided into two relevant research areas, Reward Modeling and Subjective Task Modeling for LLMs.

2.1 Reward Modeling↩︎

Alignment techniques (e.g., RLHF, GRPO) have become essential for enhancing LLM capabilities, yet designing appropriate reward signals for reinforcement learning remains a significant challenge. Extensive research focuses on building more robust and efficient reward models [12]. For instance, [9] provide theoretical analysis of Bradley-Terry reward models, while [13] improve generalization through hidden state regularization. Other approaches address overfitting via reward model ensembles [14] and adaptive margin strategies [11]. Recent innovations have explored diverse reward construction methods, including token-wise dense rewards [15], multi-objective rewards [16], and pair-wise rewards [17], advancing the field’s development.

2.2 Subjective Task Modeling for LLMs↩︎

Recently, subjective tasks (e.g., creative writing [18], emotion support conversation [19]–[23], role-play [24], [25], etc.) have gained significant attention in both research and practical deployment of LLMs. Role-play agents, for instance, have become increasingly important in dialogue systems, with several companies launching role-playing products such as Glow³, Character.AI⁴, and Tongyi Xingchen⁵. This trend highlights the growing industrial significance of such applications. However, compared to tasks like mathematical and logical reasoning, subjective tasks are more challenging to model and evaluate due to the absence of standard answers. Recent efforts have attempted to address these challenges: [26] explore self-alignment techniques with reward signals to define cognitive boundaries, enabling more consistent character simulation, while [27] propose given-circumstance acting framework to train and evaluate roles across multiple dimensions. Nonetheless, these approaches primarily rely on LLM-as-a-Judge methods or manual judgment, which can lead to inaccuracy and instability in evaluation.

3 Preliminaries↩︎

Reward Model Training. In general, reward modeling is typically based on the Bradley-Terry model [8]. By learning relative preferences from human feedback, Bradley-Terry can effectively predict the relative quality of each behavior, thereby generating reward signals for each state-action pair. In reward modeling, given a pair of responses \((y_w, y_l)\) for input \(x\), where \(y_w\) is preferred over \(y_l\), the preference probability is defined as: \[\small P(y_w \succ y_l \mid x) = \frac{\exp(r_\phi(x, y_w))}{\exp(r_\phi(x, y_w)) + \exp(r_\phi(x, y_l))} \label{eq:bt95core}\tag{1}\] where \(r_\phi: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}\) is the reward model parameterized by \(\theta\). The model is trained via maximum likelihood estimation with cross-entropy loss: \[\small \mathcal{L}_{\text{BT}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[\log \sigma\left(r_\phi(x,y_w) - r_\phi(x,y_l)\right)\right], \label{formula:two}\tag{2}\] where \(\sigma(z) = (1+\exp(-z))^{-1}\) is the sigmoid function, and \(\mathcal{D}\) denotes the preference dataset, \(r_\phi\) denotes reward function.

Reinforcement Learning from Human Feedback. Utilizing the reward model and the KL penalty in policy optimization [28], [29], we can express the reinforcement learning optimization problem as follows: \[\begin{align} \mathcal{L}_{\mathrm{RLHF}}(\pi_\theta) = & - \mathbb{E}_{y \sim \pi_\theta, x \sim \mathcal{D}}\left[r_\phi(x,y)\right] \\ & + \beta \mathrm{D}_{\mathrm{KL}}\left(\pi_\theta(y|x) \parallel \pi_{\mathrm{ref}}(y|x)\right) \end{align} \label{formula:three}\tag{3}\] Here, \(\pi_\theta\) represents the parameter distribution of the actor model in RLHF, while \(\pi_{\mathrm{ref}}\) denotes the parameter distribution of the reference policy model. This optimization objective aims to ensure that the policy does not deviate significantly while maximizing the reward score of the generated outcomes.

Limitations of Bradley-Terry model. Although the Bradley-Terry model effectively captures preference relationships, it faces significant challenges in subjective tasks due to sensitivity to data noise and limited generalization capability [30]. Subjective dialogue tasks, particularly role-playing, introduce additional complexity through diverse contexts, character backgrounds, and emotional expressions, making quality assessment inherently ambiguous. More critically, Equation 2 applies uniform optimization granularity to all preference pairs, ignoring variations in quality differences [11]. In subjective dialogue preferences, the confidence and magnitude of quality gaps vary significantly. Some preferences reflect clear distinctions with high annotator confidence, while others represent subtle differences where human judgment is ambiguous. Traditional Bradley-Terry models treat all pairs equally, failing to capture these varying degrees of preference strength. This limitation becomes particularly problematic in subjective tasks, where ignoring preference confidence can lead to overfitting on ambiguous cases while underutilizing high-confidence samples.

4 Methods↩︎

In this section, we present our proposed dynamic reward calibration method, termed AAM (Act-Adaptive Margin). We provide a comprehensive exposition of the mathematical formulation underlying this approach and elaborate on two distinct implementations of the AAM method: 1) Probability-Ratio Adaptive Margin (PR), and 2) Loss-Difference Adaptive Margin (LD).

4.1 Motivation↩︎

As established previously, traditional Bradley-Terry models struggle with subjective tasks due to their uniform treatment of all preference pairs, ignoring the inherent ambiguity and varying confidence levels characteristic of subjective preferences. While adaptive margin approaches have been proposed to enhance preference modeling [31], [32], they require additional margin annotations for each preference pair. To address these fundamental challenges, we propose AAM (Act-Adaptive Margin), which fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. This approach eliminates the need for explicit margin annotations while dynamically adapting to the uncertainty inherent in subjective preference evaluation.

4.2 Probability-Ratio Adapative Margin↩︎

To explore what kind of adaptive margin can be derived directly from the model’s own parameters, we revisit the ultimate optimization objective of reinforcement learning [33]. We observe that the goal of reinforcement learning is essentially to maximize a new form of reward. Specifically, by substituting the standard KL divergence formula \(\mathbb{D}_\mathrm{KL}\left[P\|Q\right]=\mathbb{E}_{x\sim P(x)}\left[\log\frac{P(x)}{Q(x)}\right]\) into Equation 3 , we obtain: \[\begin{align} \mathcal{L}_{\mathrm{RLHF}}(\pi_\theta) = & - \mathbb{E}_{y \sim \pi_\theta, x \sim \mathcal{D}}[r_\phi(x,y) \\ & -\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}] \end{align}\] Here, \(P=\pi_\theta(y\mid x)\) and \(Q=\pi_{\mathrm{ref}}(y\mid x)\). It becomes clear that the RLHF objective effectively maximizes the following modified reward\(r(x,y)_\Psi\):

\[\begin{align} r(x,y)_\Psi=r_\phi(x,y)-\beta\log\frac{\pi_\theta(y\mid x)}{\pi_\mathrm{ref}(y\mid x)} \end{align} \label{formula:five}\tag{4}\]

Interestingly, this reward is composed of the original reward from the reward model and the log-likelihood ratio between the actor and the reference policy. This observation motivates us to explore whether the latter—i.e., the log-likelihood ratio—can serve as a component for constructing an adaptive margin. We align the reward maximization in RLHF with the reward from reward modeling by substituting \(r(x,y)_\Psi\) from Equation 4 into \(r(x,y)_\phi\) in Equation 2 : \[\small \begin{align} \mathcal{L}_{\text{BT}}(\phi) = - \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \Biggl[ \log \sigma \biggl( r_\phi(x,y_w) - r_\phi(x,y_l) \\ - \beta \left( \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)} \right) \biggr) \Biggr], \label{formula:six} \end{align}\tag{5}\]

Remarkably, this reward model loss function bears a striking resemblance to the reward model loss function with an adaptive margin \(\mathcal{M}\): \[\small \begin{align} \mathcal{L}_{\text{BT}}(\phi) = & -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \Big[ \\ & \log \sigma\left(r_\phi(x,y_w) - r_\phi(x,y_l)-\mathcal{M}\right)\Big] \end{align}\]

In Equation 5 , the log-likelihood ratio difference naturally serves as an adaptive margin. Since reward models \(r_\phi\) typically consist of a pre-trained model \(r_\theta\) with generative capabilities and a value head, the practical implementation of Equation 5 is feasible.

Log-likelihood ratios have demonstrated effectiveness in preference modeling, as seen in DPO [29] and process reward construction [34]. Building on this insight, Equation 5 addresses the limitation of traditional reward models that only learn pairwise preferences without quantifying preference strength, by utilizing implicit rewards to construct an adaptive margin \(\mathcal{M}\) through log-likelihood ratios:

\[\mathcal{M}(\theta)=\log\frac{\pi_\theta(y_w\mid x)}{\pi_\mathrm{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_\mathrm{ref}(y_l\mid x)}\]

We term this margin as Probability-Ratio Adaptive Margin (PR), which naturally emerges from the generative capabilities inherent in the pre-trained model, addressing how to automatically calibrate learning intensity based on preference confidence without requiring additional human annotations. When \(\mathcal{M}\) is large, the reward model demonstrates high confidence in the predefined preference relationships, requiring a substantial reward score difference (greater than \(\mathcal{M}\)) to minimize loss. This compels rigorous distinction between good and bad samples during training. Conversely, when \(\mathcal{M}\) is small, the model exhibits uncertainty about whether \(y_w\) is genuinely superior to \(y_l\), allowing smaller reward differences and enabling the model to learn more nuanced details without being misled by ambiguous signals. This adaptive mechanism is particularly well-suited for subjective tasks, where human preferences often exhibit high uncertainty and annotator disagreements result in ambiguous preference relationships. Traditional methods cannot flexibly adapt to this ambiguity, often overfitting noise or overlooking subtle differences. In contrast, our approach dynamically adjusts optimization objectives based on confidence level—strengthening learning signals when data is reliable and reducing intensity when questionable, enabling more robust fitting of complex human subjective judgments.

4.3 Loss-Difference Adapative Margin↩︎

Drawing inspiration from the relationships between SimPO [35], [36] and DPO [29], we identify a potential variant of the Probability-Ratio Adaptive Margin. We can substitute the original log-likelihood ratio with generation probabilities\(\frac{\beta}{|y|}\sum_{i=1}^{|y|}\log\pi_{\theta}(y_{i}\mid x,y_{<i})\), yielding a computationally more efficient implementation of adaptive margins: \[\small \begin{align} \mathcal{M}(\theta) = \frac{\beta}{|y|} \Bigg( & \sum_{i=1}^{|y|}\log\pi_{\theta}(y^{i}_{w}\mid x,y^{<i}_{w}) \\ & - \sum_{i=1}^{|y|}\log\pi_{\theta}(y^{i}_{l}\mid x,y^{<i}_{l}) \Bigg), \end{align}\]

Since the SFT loss can be fundamentally computed from generation probabilities, where \(\mathcal{L}_{\text{SFT}}(\theta)=-\sum_{i=1}^{|y|} \log \pi_{\theta}(y^{i} \mid x, y^{<i})\), we can further simplify this margin to the difference between two SFT losses:

\[\begin{align} \small \mathcal{M}(\theta) = \mathcal{L}_{\text{SFT}}(y_l,\theta) - \mathcal{L}_{\text{SFT}}(y_w,\theta) \end{align}\]

We term this approach as Loss-Difference Adaptive Margin (LD). This construction is more intuitive, as SFT loss directly reflects the model’s fitting quality for a given sample [37], [38]. High loss indicates low preference for that sample, while low loss suggests good adaptation to the sample’s distribution. Consequently, the SFT loss difference between two samples naturally serves as an appropriate margin, similar to contrastive learning schemes like SimPO. This method provides a clear, computationally efficient alternative that leverages the model’s inherent understanding of sample quality, making it particularly suitable for practical implementations.

4.4 Connection between two different margins↩︎

We observe that both margin types derive from the reward model’s generative capabilities, indicating their intrinsic relationship to pre-trained model behavior. Therefore, we unify these approaches under the term AAM (Act-Adaptive Margin). This confidence-based dynamic calibration enables adaptive learning strategies—applying fine-grained discrimination for uncertain cases while enforcing strong separation for confident predictions. Such act-adaptive training represents a significant advancement in reward modeling for subjective tasks, where preference uncertainty is inherent and traditional fixed-margin approaches prove inadequate.

5 Experiments↩︎

We demonstrate AAM’s effectiveness by comparing it against mainstream models on general benchmarks like RewardBench [12] and JudgeBench [39]. For downstream applications, we focus on the most challenging domain in reward modeling: role-playing tasks, where traditional Bradley-Terry models struggle with ambiguous preferences. We conduct both reward modeling and GRPO performance tests on the authoritative CharacterEval [40] benchmark, and additionally construct a specialized role-playing evaluation benchmark Charm to comprehensively assess AAM’s alignment optimization effects in subjective scenarios.

5.1 General Reward Modeling Evaluations↩︎

Baselines. We compare our proposed AAM against a series of closed-socure models (GPT-4o [41], Claude-3-5-sonnet [42]) and open-source reward models (Prometheus2-7B [43], CompassJudger-7B-Instruct  [44], InternLM2-7B-Reward [45], Skywork-Critic-8B [46], ArmoRM-Llama3-8B  [47], Llama3-OffsetBias-RM-8B [48], URM-Llama3-8B [49], Tulu3-8B-SFT-RM-RB2  [50]). We also select the Bradley-Terry model (i.e. BT) and the ChatGPT-scored margin (i.e. GPT-Margin) as our effective baselines.

Implementation Details. Our experiments on general reward modeling are conducted using Qwen2.5-7B [51]. For the training set, we select the Skywork-Reward-Preference-80K-v0.2 [52] dataset, which contains high-quality preference pairs from various sources and serves as an excellent general-purpose reward modeling dataset. During reward model training, the regularization coefficient \(\alpha\) is set to 0.01, with 2 training epochs and a learning rate of 1e-5. All experiments are conducted on a cluster equipped with eight NVIDIA A100 GPUs (each with 80GB of memory).

Finegrained Analysis. We evaluate AAM (Act-Adaptive Margin) and its variants on RewardBench and JudgeBench. Results demonstrate that AAM consistently outperforms strong open- and closed-source baselines. Specifically, AAM surpasses BT on both benchmarks, indicating superior preference modeling, and outperforms GPT-Margin in most metrics, suggesting it produces more precise and reliable margins against GPT-4o. On RewardBench, AAM achieves top overall scores (up to 91.6). While InternLM2 excels in Chat (99.2), it compromises performance in other areas like Chat-Hard and Safety; in contrast, AAM maintains the balanced generalization essential for robust reward modeling. Similarly, on JudgeBench, AAM ranks first or second across nearly all sub-tasks, further confirming its robustness.

5.2 Downstream Task Evaluations↩︎

Benchmark for Role-Playing In our downstream application experiments, we use CharacterEval as the main benchmark for role-playing generation quality. CharacterEval [40] is a Chinese role-playing benchmark with 1,785 multi-turn dialogues across 77 characters, covering twelve metrics in three areas: Character Attractiveness, Conversational Ability, and Knowledge Consistency. To further assess downstream performance, we introduce Charm, a comprehensive benchmark for role-playing tasks. We collect diverse character profiles and user prompts, then use the Scene-Character-User framework and advanced LLMs (e.g., Claude, Doubao-Character) to generate and refine dialogues. Two LLMs play the roles of user and character in free-form conversations. After collecting a substantial number of role-playing dialogues, we employ six different LLMs to generate various responses based on the dialogue context and user queries, including GPT-4o [1], Claude-3.5-sonnet [42], Doubao-Character [53], and Qwen2.5 models (7B/32B/72B) [51]. We recruit 10 annotators with postgraduate-level education to select the highest and lowest quality responses from the six generated ones as preference training data. Charm contains 16,888 dialogues from 1,108 characters and 230 virtual users, spanning 13 categories such as comics, movies, novels, games, and more. A detailed distribution of character categories is provided in Figure 3. Based on these dialogues, we construct three subsets:

• Charm-RoleReward: A benchmark for role-playing reward modeling comprising 4,000 entries that evaluate knowledge consistency and character attractiveness in both English and Chinese, focusing on the scoring accuracy of reward models.

• Charm-DialogueQuality: A dialogue quality evaluation dataset for role-playing agents containing 800 high-quality synthetic dialogues with human-annotated scoring criteria across six dimensions: Consistency, Knowledge, Behavior, Empathy, Diversity, and Fluency. Further details can be found in the appendix 8.1.

• Charm-PreferenceTraining: A preference training dataset designed for subsequent performance tests in role-playing tasks.

Reward Modeling Results. We extract 2,000 preference pairs from Charm-PreferenceTraining and incorporate 2,000 pairs from real human-AI role-playing interactions to train a role-playing-specific reward model based on Qwen2.5-7B. As shown in Table 2, powerful general-purpose closed-source models perform poorly on role-playing dimensions, highlighting the need for specialized reward models in complex subjective scenarios where LLM-as-a-Judge approaches fail. Both AAM variants provide significant improvements over naive Bradley-Terry models, with gains reaching 4.8% when combined with SFT. Compared to the results of Table ¿tbl:tab:rm95eval?, AAM demonstrates superior performance on subjective tasks, likely because ambiguous reward signals make it difficult for Bradley-Terry models to fit effectively with limited data. AAM enables the reward model to leverage internal knowledge for guided training, rapidly improving effectiveness in subjective domains.

GRPO Results. We sample 4,000 dialogue contexts from Charm-PreferenceTraining (non-overlapping with reward model training) to perform GRPO enhancement on Qwen2.5 models. We compare our GRPO-enhanced models against diverse baselines including open-source models (LLaMa3.1 [54], Qwen2.5 [51]), closed-source models (GPT-4o [1], Claude-3.5-sonnet [42]), and proprietary models (Doubao-PRO-Character [53], minimax5.5s [55]). As shown in Table ¿tbl:tab:baseline4rpla?, AAM-GRPO-32B outperforms Doubao-Pro-Character by 0.18 on CharacterEval and matches Claude-3.5-sonnet performance on Charm-DialogueQuality, achieving SOTA role-playing performance among all tested models. We also perform human evaluation on AAM-GRPO-32B (see Appendix 8.2 for details).

6 Conclusion↩︎

In this study, we propose AAM (Act-Adaptive Margin), a method that leverages the reward model’s internal knowledge to guide Bradley-Terry training via adaptive margins. AAM offers two annotation-free implementation forms that explicitly calibrate confidence levels between preference pairs. Our experiments demonstrate that Act Adaptive Margin achieves effective results in general reward modeling tasks and shows particularly significant improvements in subjective tasks such as role-playing. Furthermore, downstream application experiments confirm that AAM can provide more accurate rewards for alignment training methods in subjective tasks.

7 Limitations↩︎

In this section, we analyze the limitations of our study to better optimize our approach and provide more effective guidance for researchers in training reward models in the role-playing tasks. We discuss two main shortcomings of our work. First, owing to limited computational resources, our experimental validation is primarily conducted on models with moderate parameter scales (e.g., 7B or 32B). Although we have not extensively verified the approach on larger-scale foundation models (e.g., 70B), the robust performance observed in current settings suggests that our method possesses promising scalability. Second, while many studies suggest that improving critique generation ability can enhance the performance of reward models, we do not adopt a multi-task learning approach to integrate critique capability, due to the difficulty in obtaining role-playing evaluation data. In future work, we plan to develop a specialized critique model to further optimize RPLAs.

8 Appendix↩︎

8.1 Charm-DialogueQuality↩︎

We propose a new role-playing evaluation benchmark, Charm-DialogueQuality, designed to automatically assess the performance of RPLA by utilizing GPT-4o and 800 test samples. Before constructing Charm-DialogueQuality, we first generate 160 role profiles and prompts using Claude3.5-sonnet, GPT-4o, and Doubao-Pro-Character. These are then manually refined to improve the accuracy and quality of the role information. The generated roles cover 9 common categories: Custom Roles, Anime, Novels, Telepaly, Movies, Games, Scientists, Actors, and Musicians. After obtaining accurate role information, we adopt a method similar to Scene-Character-User Framework, generating 1000 dialogue contexts.

To ensure that Charm-DialogueQuality can comprehensively assess the RPLA’s capabilities, we focus on six key dimensions.

• Consistency refers to the ability of RPLA to understand and remember the context of the conversation, providing coherent responses based on the prior dialogue. If RPLA frequently fails to recall previous interactions, it indicates poor contextual consistency.

• Knowledge evaluates whether RPLA’s cognition aligns with the character’s background knowledge, which is crucial for maintaining the authenticity of the character. If RPLA’s knowledge diverges from the character’s established traits, it will negatively impact character development.

• Behavior assesses whether RPLA’s actions, expressions, and tone accurately reflect the character’s personality traits. A successful RPLA should be able to convey its unique characteristics through these details; failure to do so indicates a flaw in character portrayal.

• Empathy is a key dimension for evaluating RPLA’s emotional interaction quality. A model with good empathy not only increases the character’s appeal but also enhances its emotional support capabilities.

• Diversity focuses on the richness of content presented by the character during the conversation, assessing whether RPLA can demonstrate a variety of thoughts and expressions.

• Fluency measures the basic conversational ability of RPLA, evaluating whether it can engage in natural, fluent dialogues.

Based on these 6 dimensions and 160 role characteristics, we ask human annotators to design a user query for each dialogue context, matching the current role and dimension, to continue the conversation and assess RPLA’s performance in that particular dimension. From the 1000 dialogue samples, we select 400 to construct the Charm-DialogueQuality benchmark. Each sample is accompanied by a set of evaluation criteria, helping GPT-4o to provide more accurate scoring. During evaluation, the model replies to each sample, and GPT-4o scores RPLA’s response on a scale from 1 to 5 based on the context, the model’s reply, and the specific evaluation criteria. Finally, we compute the average score across all dimensions to obtain the overall RPLA score in Charm-DialogueQuality. After completing the annotation and quality check for the 400 Chinese samples, we translate them into English, resulting in the English version of Charm-DialogueQuality. Figure 6 presents an example of a Charm-DialogueQuality sample to help readers better understand the evaluation process. Table 1 provides detailed information about Charm-DialogueQuality and compares it with other role-playing datasets.

8.2 Human Evaluations.↩︎

Additionally, we conduct a human evaluation to compare AAM-GRPO-32B with three baseline models: Claude3.5-sonnet, GPT-4o, and Doubao-Pro-Character. In each pairwise comparison, both models generate responses to the same role-playing dialogue context. Five human annotators then assess the responses, categorizing the results as win, tie, or loss for AAM-GRPO-32B relative to each baseline. The average results from 200 test samples, along with annotations from the five evaluators, are presented in Figure 4. Notably, AAM-GRPO-32B significantly outperforms all three models in role-playing capabilities, providing strong evidence of the effectiveness of our proposed methodology.

8.3 Case study↩︎

To help readers intuitively understand the improvements in role-playing abilities brought by AAM, we select some examples for case studies, as shown in Figure 7 and Figure 8. In these figures, we manually evaluate the responses from AAM-GRPO-32b, GPT-4o, and Claude 3.5-Sonnet. It can be observed that AAM-GRPO-32b outperforms the other models in both knowledge consistency and diversity, as well as in maintaining context consistency across these two examples. In contrast, GPT-4o and Claude 3.5-Sonnet occasionally make minor errors in their responses.

References↩︎

---

1. Equal Contribution.↩︎

2. Corresponding author.↩︎

3. http://www.glowapp.tech/↩︎

4. https://www.character.ai↩︎

5. https://tongyi.aliyun.com/xingchen/↩︎