TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment
January 01, 1970
Abstract
Personalized large language models adapt responses to users’ preferences and social attributes, but can introduce substantial universal truth inconsistencies across social groups, where some groups systematically receive less accurate responses on objective tasks. Existing alignment methods either ignore personalization or mainly focus on subjective preference alignment, largely overlooking fairness and consistency in universal truths. To address this gap, we study Truth-Invariant Alignment (TIA), an alignment problem for personalized LLMs that aims to ensure universal truths remain consistent across social groups while preserving personalization. We propose TriAlign, the first offline multi-agent reinforcement learning (MARL) framework for TIA, where each social group is modeled as an agent interacting. TriAlign jointly optimizes universal truth accuracy, cross-group truth consistency, and personalization through a fairness-aware objective and an explicit inconsistency penalty. Experiments across diverse benchmarks demonstrate that TriAlign achieves a stronger balance among these three objectives than strong baselines, reducing universal truth disparities across social groups while improving both objective task performance and personalization quality.
1 Introduction↩︎
The success of large language models (LLMs), such as GPT, PaLM, LLaMA, DeepSeek, and their variants, in general knowledge and multi-domain reasoning has driven the growing demand for Personalized LLMs (PLLMs) [1]–[3]. PLLMs aim to generate responses that align with users’ styles and expectations, offering diverse answers to the same query depending on the user, often conditioning personalization on demographic or social attributes such as age, gender, or occupation. For objective tasks, there exists a universal truth—a statement/answer that is widely accepted as correct. While the presentation of this truth may vary across users, the underlying truth itself should remain unchanged and access to universal truth should not depend on demographic or social attributes. In practice, however, emerging evidence suggests that introducing personas into LLMs can create substantial universal truth gaps across social groups [4]–[7]. For example, the same LLM achieves only 54% universal truth accuracy for Trump supporters, compared to 62% for Obama supporters [4]. As a result, some social groups may systematically receive less accurate responses than others. Figure 1 illustrates this challenge. This issue not only reduces the reliability of PLLMs but also raises fairness concerns, especially as LLMs are increasingly deployed in education, healthcare, and decision-support systems [8]–[10].
Existing LLM alignment approaches mainly fall into two paradigms: “one-size-fits-all” models that optimize general responses without personalization [11]–[15], and PLLMs that adapt outputs to user preferences and styles [16]–[18]. While PLLMs improve user alignment, most existing works focus primarily on subjective preferences and largely overlook truth invariance across social groups [4], [19]–[22]. A few early studies indicate that incorporating personas can introduce inconsistencies in universal truths across social groups on objective tasks [5], [6]. Existing mitigation efforts mainly rely on prompting-based strategies, such as instructing models to ignore demographic attributes or neutrality factuality during inference [6], [23]. However, prompting only steers surface-level generation behavior without explicitly modifying demographic correlations encoded in model parameters. As a result, these methods often yield only marginal improvements and mainly remain effective for very large models (e.g., 72B parameters) [6]. Moreover, prompting-based debiasing may reduce the influence of personas or encourage overly neutral responses, which can lower personalization quality. These limitations reveal the need for training-based alignment methods, which can directly optimize model behavior across diverse social groups and tasks. One of the state-of-the-art techniques for aligning LLMs is RL-based alignment [11]. However, existing alignment approaches primarily optimize average performance, which may still favor dominant groups and overlook universal truth consistency across social groups.
To mitigate this gap, we study Universal Truth-Invariant Alignment, a new alignment problem for PLLMs that aims to ensure universal truths remain consistent across social groups while preserving personalization. We propose TriAlign, a novel MARL framework where each social group is modeled as an agent. TriAlign jointly optimizes universal truth, cross-group universal truth consistency, and personalization. Specifically, multiple social-group-conditioned agents interact within a shared environment, observe differences in each other’s responses, and iteratively adjust their behaviors to converge toward consistent universal truths while preserving personalized response styles. To encourage fairness across social groups, we design a new training objective inspired by Nash Social Welfare (NSW) [24], which promotes balanced optimization rather than favoring dominant groups. In addition, we explicitly incorporate a cross-group consistency penalty into the reward of each action to directly reduce inconsistencies across social groups. To enable training, we further construct multi-social-group interaction trajectories through multi-turn interactions, where agents progressively refine their responses using comparative feedback from other groups. The resulting trajectories form an offline RL dataset used to align LLMs toward TIA. The contributions of our work are summarized as follows:
We are the first to formulate TIA as an offline MARL problem and address it through training-based alignment. We propose TriAlign, a novel offline MARL framework for TIA, where each social group is modeled as an agent that adapts its behavior by observing a shared environment. TriAlign jointly optimizes universal truth, cross-group universal truth consistency, and personalization through a NSW-inspired objective and an explicit penalty for cross-group inconsistency.
We introduce a multi-social-group, multi-turn interaction framework to construct offline RL trajectories for TIA training. Through comparative interactions and iterative feedback, agents progressively refine their responses toward consistent universal truths while preserving personalized behaviors.
Extensive experiments across diverse benchmarks demonstrate that TriAlign effectively reduces universal truth inconsistency across social groups while improving both objective task performance and personalization alignment.
2 Related Works↩︎
2.0.0.1 From User-blind to Personalized LLMs:
LLMs have demonstrated strong generalization ability across diverse NLP tasks through zero-shot and few-shot learning [25], [26]. Recent studies shift from user-blind assistants toward PLLMs that adapt responses to user preferences in domains such as healthcare, education, and e-commerce [27], [28]. Personalized text generation and downstream personalization (e.g., recommendation systems) are also increasingly converging, enabling unified agents that combine conversational ability with personalized reasoning. Although PLLMs are mainly designed for subjective tasks involving user preferences or writing styles, real-world applications often require both personalization and objective correctness. For example, a personalized healthcare assistant must provide factually correct information while adapting explanations to individual users.
2.0.0.2 Consistency in LLMs:
Recent work shows that accuracy alone is insufficient for evaluating language models, as models may still generate contradictory predictions under semantically similar conditions [29], [30]. Early studies reveal that PLMs can produce both a fact and its negation or fail to maintain consistency across related questions [31]–[33]. Existing approaches improve consistency in question answering, reasoning, and natural language inference through data augmentation, logical constraints, and consistency-aware regularization [34]–[36]. More recently, consistency has become a key requirement for trustworthy LLMs, especially in high-stakes applications [37], [38]. However, existing studies mainly focus on consistency under paraphrases, negation, or reasoning transformations, while overlooking consistency across personalized settings and social groups. We extend this line of research toward universal truth consistency, which requires PLLMs to maintain truthful and consistent behavior across diverse personas and demographic groups.
2.0.0.3 Reinforcement Learning for LLM Alignment:
LLM alignment methods are generally divided into offline alignment and online RL approaches. Offline methods, such as SFT and preference optimization methods like DPO [11], [12], [39], learn from static demonstrations or preference pairs and are widely adopted due to their stability and efficiency. In contrast, online RL methods, including RLHF and GRPO [11], [40], optimize policies through iterative interactions and reward feedback. However, applying online RL to personalized LLM alignment is expensive because rewards for personalization, truth correctness, and fairness often require human annotations or reward models, motivating our use of offline RL on pre-collected trajectories.
The closest related works are [41]–[43], which apply RL to improve conversational behaviors from simulated interactions. [41] fine-tune models on high-reward trajectories, while [42] optimize preferred responses using DPO over alternative generations. [43] formulate offline RL as weighted SFT with trajectory-level weights shared across actions. In contrast, our work focuses on universal truth consistency and fairness across social groups in PLLMs. Unlike [43], we assign rewards at the action level while modeling future trajectory effects and explicitly optimizing cross-group fairness.
3 Truth-Invariant Alignment (TIA)↩︎
The key objective of TIA is to ensure that personalized responses generated under different social-group conditions remain invariant with respect to the underlying universal truth, while still preserving group-specific personalization characteristics.
Let \(\mathcal{G}\) denote the space of social-group profiles. Let \(\mathcal{K}\) denote the set of demographic categories (e.g., gender, age or education), and let \(\mathcal{V}_k\) denote the set of possible values for category \(k \in \mathcal{K}\). A persona is represented as a set of demographic attribute-value pairs: \(p=\{(k_1,v_1),(k_2,v_2),\dots,(k_n,v_n)\},\) where \(v_i \in \mathcal{V}_{k_i}\). Each social group \(g \in \mathcal{G}\) corresponds to a demographic category-value pair \((k,v)\), such as \((\texttt{gender},\texttt{male})\). A persona belongs to social group \(g=(k,v)\) if \((k,v)\in p\). Associated with each social group \(g\) is a set of personalization constraints \(c(g)\) specifying desired response characteristics such as style or tone. Figure 2 illustrates social groups, personas, and the corresponding personalization constraints (c(g)).
We focus on objective tasks. Each task is represented as a pair \((x,y^\star)\), where \(x\) is an input query and \(y^\star\) denotes the corresponding universal truth. Given an input \(x\) and a social-group profile \(g\), a PLLM parameterized by \(\theta\) generates a response: \(a \sim \pi_\theta(\cdot \mid x,g)\). TIA requires two properties:
Truth Invariance: for any \(g_i,g_j \in \mathcal{G}\), \[\mathrm{Truth}(a_i) = \mathrm{Truth}(a_j) = y^\star,\] where \(a_i \sim \pi_\theta(\cdot \mid x,g_i), a_j \sim \pi_\theta(\cdot \mid x,g_j)\)
Personalized Alignment: responses should satisfy the personalization constraints associated with its social group \(a_i \models c(g_i).\)
Learning Goal: The learning goal of TIA is to learn a policy \(\pi_\theta\) that jointly optimizes universal-truth correctness, cross-group truth consistency, and personalized alignment: \[{!}{ \begin{align} \pi_\theta^\star &= \arg\max_{\pi_\theta} \; \mathbb{E}_{x,g,\; a \sim \pi_\theta(\cdot \mid x,g)} \Big[ r_{\mathrm{truth}}(a,y^\star) \\ &\quad + \lambda_{\mathrm{cons}} \, r_{\mathrm{cons}}(a,\mathcal{A}_x) + \lambda_{\mathrm{pref}} \, r_{\mathrm{pref}}(a,c(g)) \Big]. \end{align} } \label{eq:tia95goal}\tag{1}\] where \(\mathcal{A}_x\) denotes the set of responses generated across social groups for the same query \(x\), \(r_{\mathrm{truth}}\) denotes the reward for objective correctness with respect to the universal truth \(y^\star\), \(r_{\mathrm{cons}}\) denotes the reward for cross-group truth consistency, and \(r_{\mathrm{pref}}\) denotes the reward for alignment with the personalization constraints \(c(g)\).
4 Our Method: TriAlign↩︎
4.1 Truth-Invariant Alignment as MARL↩︎
Since estimating \(\mathcal{A}_x\) and \(r_{\mathrm{cons}}(a,\mathcal{A}_x)\) requires jointly observing and comparing responses across multiple social-group-conditioned agents, the consistency objective introduces cross-agent coupling: updating one agent’s policy may affect the consistency objectives of others through the shared response set \(\mathcal{A}_x\). Such interactions are difficult to model with standard offline RL, which typically optimizes each \((x,g,a)\) sample independently. Therefore, the TIA formulation motivates us to adopt a MARL perspective under the centralized training and decentralized execution paradigm. Figure 3 illustrates our formulation of TIA under the MARL setting.
Specifically, we consider a system of \(m\) agents, where each social-group condition \(g_i\) is treated as an independent agent interacting within a shared environment. The agents interact through the environment to iteratively optimize cross-group truth consistency and converge toward responses that share the same underlying universal truth, while still preserving group-specific personalization characteristics.
At interaction step \(t\), agent \(i\) observes its current state: \(s_{i,t} = (x,g_i,\tau_{i,t}),\) where \(\tau_{i,t}\) denotes the conversation history: \(\tau_{i,t} = (a_{i,1},o_{i,1},\dots,a_{i,t-1},o_{i,t-1}),\) with the initial history \(\tau_{i,0}=\emptyset.\) Here, \(o_{i,k}\) denotes the feedback returned by the environment after agent \(i\) takes action \(a_{i,k}\) at interaction step \(k<t\). Conditioned on the current state, agent \(i\) generates an action: \(a_{i,t} \sim \pi_\theta(\cdot \mid s_{i,t}).\) The environment observes the joint actions: \(\mathbf{a}_t = \{a_{1,t},\dots,a_{m,t}\},\) and returns feedback: \(o_{i,t} = f_{\mathrm{env}}(a_{i,t},\mathbf{a}_t,x,g_i),\) where \(f_{\mathrm{env}}\) is a function that compares actions across agents to identify truth inconsistencies between social groups. Each agent \(i\) then receives a corresponding reward: \(r_{i,t}(x,g_i,a_{i,t}),\) which reflects the universal-truth correctness, cross-group truth consistency, and personalization quality of the generated response at interaction step \(t\). The cumulative return of agent \(i\) is defined as: \(R_i = \sum_{t=1}^{n} \gamma^t r_{i,t},\) where \(\gamma \in [0,1]\) is the discount factor controlling the importance of future rewards.
This shared environment enables implicit communication among agents during training. By observing truth inconsistencies across social groups through feedback from the environment, agents iteratively adjust their behaviors toward truth invariance while preserving personalized characteristics. This multiturn interaction provides richer training signals than single-step interactions, allowing the model to learn how responses evolve, self-correct, and converge toward truth-consistent behaviors.
Under MARL setting, Equation 1 can be written: \[{!}{ \pi_\theta^\star = \arg\max_{\pi_\theta} \; \mathbb{E}_{x,\;\{g_i\}_{i=1}^{m},\; a_{i,t}\sim \pi_\theta(\cdot\mid s_{i,t})} \left[ \frac{1}{m}\sum_{i=1}^{m} R_i \right] } \label{eq:standard95marl}\tag{2}\] where \(\mathcal{A}_{x,t} = \{a_{1,t},a_{2,t},\dots,a_{m,t}\},\) denotes the set of responses generated by all \(m\) social-group-conditioned agents for the same query \(x\) at interaction step \(t\).
While training is centralized through comparative multi-agent feedback, inference remains fully decentralized: at test time, each agent independently generates a personalized response conditioned only on \((x,g_i)\).
4.2 TriAlign Objective and Reward Design↩︎
4.2.0.1 TriAlign Objective
Directly optimizing Eq. 2 may bias the policy toward majority or high-performing social groups, since the objective only maximizes the average return across agents. To mitigate this issue, TriAlign introduces a fairness-aware optimization objective inspired by NSW, which explicitly encourages balanced performance across social groups.
The NSW objective is defined as: \(\mathcal{F}(\theta) = \prod_{i=1}^{m}(R_i+\epsilon),\) where \(\epsilon>0\) is a small constant for numerical stability. Taking the logarithm yields: \(\log \mathcal{F}(\theta) = \sum_{i=1}^{m}\log(R_i+\epsilon).\) Combining Eq. 2 with \(\log \mathcal{F}(\theta)\), TriAlign therefore optimizes the following fairness-regularized objective: \[{!}{ \mathcal{J}(\theta) = \frac{1}{m} \mathbb{E}_{x,\,\{g_i\}_{i=1}^{m}, a_{i,t}\sim \pi_\theta(\cdot\mid s_{i,t})} \left[ \sum_{i=1}^{m}, \left( R_i + \lambda_{\mathrm{NSW}}\log(R_i+\epsilon) \right) \right]. } \label{eq:trialign95objective}\tag{3}\] The optimal policy is \(\pi_\theta^\star = \arg\max_{\pi_\theta} \mathcal{J}(\theta).\)
To optimize the fairness-aware objective in an offline RL setting while mitigating distribution shift, we adopt an offline RL formulation with implicit KL constraints [44]. Specifically, we first derive the optimal non-parametric policy under a KL-regularized policy improvement objective, and then project it onto the parameterized policy \(\pi_\theta\) by minimizing the KL divergence between them. This results in the following weighted SFT objective (the detailed derivation is provided in Appendix 8.1): \[{!}{ \mathcal{L}(\theta) = - \mathbb{E}_{(s_{i,t},a_{i,t}) \sim \mathcal{D}} \left[ w(s_{i,t},a_{i,t}) \log \pi_\theta(a_{i,t}\mid s_{i,t}) \right]. }\] where \(\mathcal{D} = \{(s_{i,t},a_{i,t},r_{i,t})\}_{x,i,t}\) denotes a pre-collected dataset, and the weight is: \[{!}{ w(s_{i,t},a_{i,t}) = \underbrace{ \exp \left( \frac{ A^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) }_{\text{Standard Weight}} \cdot \underbrace{ \left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right)^{\frac{\lambda_{\mathrm{NSW}}}{\beta}} }_{\text{NSW Weight}}, }\] In this formulation, \(Q^{\pi}(s_{i,t},a_{i,t})\) is the action-value function, and \(A^{\pi}(s_{i,t},a_{i,t})\) is the advantage function. This weight combines two effects. The standard term increases the likelihood of actions whose advantages are higher than a baseline, while the NSW-inspired term forces the policy to improve performance on "worst-case" scenarios.
4.2.0.2 \(Q^{\pi}\) and \(A^{\pi}\) Estimation
We estimate the action-value function as the expected discounted future return after agent \(i\) takes action \(a_{i,t}\) under state \(s_{i,t}\): \[{!}{ Q^\pi(s_{i,t},a_{i,t}) = \mathbb{E}_{\pi} \left[ \sum_{k=t}^{n} \gamma^{k-t} r_{i,k} \mid s_{i,t},a_{i,t} \right]. }\]
To encourage both high collective performance and consistency across social groups, we estimate the baseline using the average action-value over all \(m\) agents at time step \(t\): \[V^\pi(s_t) = \frac{1}{m} \sum_{j=1}^{m} Q^\pi(s_{j,t},a_{j,t}).\]
The corresponding normalized advantage function is defined as: \[A_i^\pi(s_{i,t},a_{i,t}) = \frac{ Q^\pi(s_{i,t},a_{i,t}) - V^\pi(s_t) }{ \sigma_t+\epsilon },\] where \(\sigma_t\) denotes the standard deviation of \(\{Q^\pi(s_{j,t},a_{j,t})\}_{j=1}^{m}\), and \(\epsilon>0\) is a small constant introduced for numerical stability.
4.2.0.3 Reward Design
We define the reward for agent \(i\) after taking action \(a_{i,t}\) at time step \(t\) as: \[{!}{ \begin{align} r_i(\mathbf{a}_t) &= r_{\mathrm{truth}}(a_{i,t}, y^\star) + \lambda_{\mathrm{pref}} \, r_{\mathrm{pref}}(a_{i,t},c(g_i)) \\ &\quad - \lambda_{\mathrm{cons}} \left| r_{\mathrm{truth}}(a_{i,t}, y^\star) - \frac{1}{m} \sum_{j=1}^{m} r_{\mathrm{truth}}(a_{j,t}, y^\star) \right|. \end{align} } \label{eq:reward95function}\tag{4}\]
4.3 Multi-Social-Group Interaction Simulations↩︎
This section describes how we construct the offline trajectory dataset \(\mathcal{D}\) for TriAlign training. We first construct the social-group space \(\mathcal{G}\), and then perform multi-social-group MARL interactions to generate diverse trajectories. The resulting dataset is subsequently used to estimate \(Q^\pi\), \(V^\pi\), and \(A^\pi\), and optimize the policy according to Eq. 3 .
4.3.0.1 Social Groups Simulation
Our goal is to construct the social-group space \(\mathcal{G}\). Based on 327 prior studies on users’ digital footprints and online activities [45], we identify 14 demographic categories \(k \in \mathcal{K}\), resulting in 75 demographic category-value pairs \((k,v)\) that define the social groups in \(\mathcal{G}\). Details of all \((k,v)\) pairs are provided in Table 4.
For personas, we initialize our construction from the large-scale persona collection introduced by [46], which contains 1 billion personas, covering approximately 13% of the world’s total population. However, these personas are not explicitly categorized into demographic-based social groups. Additionally, many demographic groups are underrepresented in online data. To address this issue and construct a diverse and balanced persona pool, we propose a persona-to-persona augmentation pipeline. For each target social group \(g=(k,v)\), we retrieve semantically related personas using semantic similarity. We then ask an LLM via prompting to generate a new persona that satisfies two constraints: (1) the generated persona must match the target demographic attribute of the social group, and (2) it should preserve a meaningful relationship with the retrieved source persona. Figure 8 illustrates the prompting template. Figure 7 illustrates our persona-to-persona augmentation pipeline. Figure 9 presents an example of a generated persona. The augmented personas are then split into training and test sets with a 75/25 ratio, and duplicate personas across splits are removed to avoid data leakage.
4.3.0.2 Multi-Social-Group Interaction
To construct the offline RL dataset \(\mathcal{D}\) for TriAlign optimization, we simulate multi-turn interactions among multiple social-group-conditioned agents, as illustrated in Figure 4. Given a question \(x\), we sample \(m\) social groups \(\{g_1,\dots,g_m\}\subset\mathcal{G}\). For each social group \(g_i\), we further sample one persona \(p_i\) belonging to \(g_i\) to instantiate the corresponding agent for \(g_i\). At interaction step \(t\), agent \(i\) observes its current state: \(s_{i,t}=(x,g_i,\tau_{i,t}),\) and generates a response: \(a_{i,t}\sim \pi_\theta(\cdot\mid s_{i,t}).\) The environment collects responses from all agents: \(\mathbf{a}_t=\{a_{1,t},\dots,a_{m,t}\},\) and returns comparative feedback: \(o_{i,t}=f_{\mathrm{env}}(a_{i,t},\mathbf{a}_t,x,g_i),\) The environment also assigns a step-level reward \(r_{i,t}\) following Eq. 4 . The resulting trajectories: \((s_{i,t},a_{i,t}, r{i,t}),\) form the offline RL dataset used to estimate \(Q^\pi\), \(V^\pi\), and \(A^\pi\).
\(\pi_\theta(\cdot\mid s_{i,t})\), \(f_{\mathrm{env}}\), \(r_{\mathrm{truth}}\), and \(r_{\mathrm{pref}}\) are implemented via LLMs prompting. Detailed prompting templates are provided in Appendix 8.2.
5 Experiment Setup↩︎
5.0.0.1 Objective-task Training Data and Evaluation
For training, we construct offline RL trajectories using subsets of objective questions from LogicQA [47], SimpleQA [48], LIMO [49], and StereoSet training set [50]. Ground-truth answers are used to compute truth rewards. For evaluation, we use in-domain benchmarks for mathematical reasoning (AIME25) [51] and social bias evaluation (StereoSet) test set, as well as out-of-domain general knowledge benchmarks (MMLU-Pro, TruthfulQA). Dataset statistics are reported in Appendix 8.3
5.0.0.2 Personalization Data
For personalization training and evaluation, we synthesize persona-based preference data via GPT-4o1. Specifically, we adopt predefined preference rubrics (Table 5), inspired by [52], and use GPT-4o to generate ground-truth preference labels via prompting (Appendix 8.2) conditioned on persona demographic profiles and the domain of each dataset.
5.0.0.3 Metrics:
We report three categories of metrics: (1) Universal Truth Accuracy (Mean), measuring objective correctness; (2) Universal Truth Consistency, measuring cross-group truth disparity via Worst-Group Accuracy (Worst), Standard Deviation (Std), Worst-Group Gap (Gap), and Coefficient of Variation (CV); and (3) Preference Alignment Accuracy (Pref), computed as the average accuracy across personalization rubrics, automatically evaluated by GPT-4o. We evaluate the models under two settings: implicit preference, where user preferences are not explicitly specified in the input question, and explicit preference, where the desired response style is explicitly provided in the prompt.
The performance of each group is computed as the average performance across all personas belonging to that social group.
| Implicit Preference | Explicit Preference | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3-8 (lr)9-14 Dataset | Model | Pref\(\uparrow\) | Mean\(\uparrow\) | Worst\(\uparrow\) | Std\(\downarrow\) | Gap\(\downarrow\) | CV\(\downarrow\) | Pref\(\uparrow\) | Mean\(\uparrow\) | Worst\(\uparrow\) | Std\(\downarrow\) | Gap\(\downarrow\) | CV\(\downarrow\) |
| AIME25 | Qwen3-4B-Instruct-2507 | 0.170 | 0.477 | 0.409 | 0.052 | 0.068 | 0.109 | 0.691 | 0.328 | 0.267 | 0.036 | 0.061 | 0.109 |
| P-Debias (2024) | 0.199 | 0.458 | 0.367 | 0.050 | 0.091 | 0.109 | 0.331 | 0.309 | 0.267 | 0.025 | 0.042 | 0.082 | |
| BestPersona (2024) | 0.184 | 0.496 | 0.400 | 0.077 | 0.096 | 0.156 | 0.374 | 0.330 | 0.267 | 0.053 | 0.063 | 0.160 | |
| P-Defense (2025) | 0.181 | 0.432 | 0.367 | 0.050 | 0.066 | 0.117 | 0.339 | 0.300 | 0.267 | 0.038 | 0.033 | 0.128 | |
| 2StepPrompt (2025) | 0.155 | 0.494 | 0.423 | 0.074 | 0.071 | 0.149 | 0.180 | 0.482 | 0.423 | 0.051 | 0.059 | 0.106 | |
| SFT (2022) | 0.217 | 0.394 | 0.267 | 0.073 | 0.128 | 0.185 | 0.375 | 0.361 | 0.300 | 0.056 | 0.061 | 0.155 | |
| SWIFT (2026) | 0.219 | 0.408 | 0.333 | 0.059 | 0.075 | 0.145 | 0.418 | 0.400 | 0.300 | 0.072 | 0.100 | 0.180 | |
| TriAlign (Ours) | 0.221 | 0.511 | 0.467 | 0.025 | 0.044 | 0.049 | 0.699 | 0.467 | 0.433 | 0.027 | 0.033 | 0.058 | |
| StereoSet | Qwen3-4B-Instruct-2507 | 0.429 | 0.460 | 0.421 | 0.022 | 0.039 | 0.048 | 0.744 | 0.317 | 0.217 | 0.056 | 0.100 | 0.176 |
| P-Debias (2024) | 0.430 | 0.414 | 0.388 | 0.019 | 0.026 | 0.046 | 0.725 | 0.358 | 0.321 | 0.022 | 0.037 | 0.060 | |
| BestPersona (2024) | 0.409 | 0.468 | 0.454 | 0.011 | 0.014 | 0.024 | 0.739 | 0.378 | 0.357 | 0.012 | 0.020 | 0.032 | |
| P-Defense (2025) | 0.422 | 0.446 | 0.416 | 0.014 | 0.030 | 0.032 | 0.740 | 0.372 | 0.321 | 0.026 | 0.051 | 0.069 | |
| 2StepPrompt (2025) | 0.422 | 0.355 | 0.342 | 0.007 | 0.014 | 0.019 | 0.458 | 0.335 | 0.319 | 0.009 | 0.016 | 0.026 | |
| SFT (2022) | 0.409 | 0.668 | 0.653 | 0.008 | 0.014 | 0.012 | 0.753 | 0.633 | 0.596 | 0.029 | 0.037 | 0.046 | |
| SWIFT (2026) | 0.422 | 0.718 | 0.697 | 0.016 | 0.020 | 0.023 | 0.765 | 0.695 | 0.661 | 0.030 | 0.034 | 0.043 | |
| TriAlign (Ours) | 0.474 | 0.792 | 0.778 | 0.007 | 0.014 | 0.029 | 0.791 | 0.788 | 0.773 | 0.010 | 0.015 | 0.013 | |
| Implicit Preference | Explicit Preference | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3-8 (lr)9-14 Dataset | Model | Pref\(\uparrow\) | Mean\(\uparrow\) | Worst\(\uparrow\) | Std\(\downarrow\) | Gap\(\downarrow\) | CV\(\downarrow\) | Pref\(\uparrow\) | Mean\(\uparrow\) | Worst\(\uparrow\) | Std\(\downarrow\) | Gap\(\downarrow\) | CV\(\downarrow\) |
| MMLU-Pro | Qwen3-4B-Instruct-2507 | 0.412 | 0.767 | 0.714 | 0.047 | 0.052 | 0.061 | 0.691 | 0.752 | 0.671 | 0.058 | 0.081 | 0.076 |
| P-Debias (2024) | 0.427 | 0.788 | 0.729 | 0.037 | 0.059 | 0.047 | 0.686 | 0.793 | 0.743 | 0.037 | 0.050 | 0.046 | |
| BestPersona (2024) | 0.409 | 0.766 | 0.739 | 0.029 | 0.026 | 0.038 | 0.681 | 0.749 | 0.700 | 0.024 | 0.049 | 0.032 | |
| P-Defense (2025) | 0.404 | 0.790 | 0.729 | 0.031 | 0.061 | 0.040 | 0.680 | 0.769 | 0.743 | 0.017 | 0.026 | 0.023 | |
| 2StepPrompt (2025) | 0.390 | 0.755 | 0.735 | 0.010 | 0.019 | 0.013 | 0.417 | 0.769 | 0.755 | 0.014 | 0.017 | 0.019 | |
| SFT (2022) | 0.483 | 0.711 | 0.671 | 0.035 | 0.039 | 0.050 | 0.651 | 0.761 | 0.729 | 0.021 | 0.033 | 0.027 | |
| SWIFT (2026) | 0.463 | 0.754 | 0.725 | 0.029 | 0.029 | 0.038 | 0.649 | 0.764 | 0.743 | 0.018 | 0.021 | 0.024 | |
| TriAlign (Ours) | 0.510 | 0.805 | 0.800 | 0.007 | 0.005 | 0.008 | 0.697 | 0.831 | 0.757 | 0.045 | 0.074 | 0.055 | |
| TruthfulQA | Qwen3-4B-Instruct-2507 | 0.412 | 0.672 | 0.597 | 0.044 | 0.075 | 0.066 | 0.691 | 0.730 | 0.721 | 0.011 | 0.011 | 0.013 |
| P-Debias (2024) | 0.455 | 0.717 | 0.696 | 0.014 | 0.021 | 0.019 | 0.686 | 0.733 | 0.718 | 0.012 | 0.015 | 0.016 | |
| BestPersona (2024) | 0.450 | 0.726 | 0.710 | 0.011 | 0.016 | 0.015 | 0.542 | 0.747 | 0.735 | 0.008 | 0.013 | 0.010 | |
| P-Defense (2025) | 0.429 | 0.731 | 0.721 | 0.008 | 0.010 | 0.011 | 0.735 | 0.731 | 0.714 | 0.010 | 0.017 | 0.013 | |
| 2StepPrompt (2025) | 0.402 | 0.755 | 0.735 | 0.010 | 0.019 | 0.013 | 0.451 | 0.772 | 0.755 | 0.014 | 0.017 | 0.019 | |
| (lr2pt)2-14 | SFT (2022) | 0.545 | 0.790 | 0.780 | 0.007 | 0.010 | 0.009 | 0.745 | 0.788 | 0.779 | 0.006 | 0.009 | 0.008 |
| SWIFT (2026) | 0.480 | 0.784 | 0.769 | 0.007 | 0.014 | 0.009 | 0.748 | 0.795 | 0.782 | 0.009 | 0.013 | 0.011 | |
| TriAlign (Ours) | 0.552 | 0.798 | 0.789 | 0.006 | 0.009 | 0.007 | 0.755 | 0.853 | 0.845 | 0.006 | 0.008 | 0.007 | |
5.0.0.4 Baselines
We compare TriAlign with two categories of training-free baselines and a group of training-based alignment methods. For training-free methods, we consider: [1] instruction-based debiasing approaches, including P-Debias [4] and P-Defense [21]; and [2] multi-step refinement methods, including BestPersona [5], and 2StepPrompt [6]. For training-based alignment baselines, we compare against methods with comparable training cost, including SFT [11] and Swift [43].
5.0.0.5 Implementation Details
We use LangGraph 2 to implement the multi-agent interaction with 8 agents and a maximum of 4 interaction turns. vLLM [53] is used for LLM serving, with temperature 0.7 during training data generation and 0 during evaluation. We fully fine-tune Qwen3-4B-Instruct-2507 [54] using LlamaFactory [55] for 3 epochs with learning rate \(1\times10^{-5}\), batch size 16, and gradient accumulation 4. Training is performed with DeepSpeed ZeRO-3 [56] on 2\(\times\)A100 80GB GPUs. We run each experiment 2 times with different random seeds and report the averaged results.
6 Evaluation↩︎
6.0.0.1 Main Results
Table 1 and Table 2 respectively report the results of our method on in-domain and out-of-domain benchmarks. As shown in the results, prompt-based methods generally attempt to reduce bias by weakening the influence of user-specific information through additional instructions. While these approaches improve truth invariance, as reflected by lower Std and Gap values, they often do so at the expense of personalization and overall truthfulness performance, leading to lower personalization scores as well as reduced Mean and Worst. In particular, 2StepPrompt exhibits a clear improvement in truth inconsistency because their first stage is to generate a persona-blind response to ensure truth consistency and then revise this response to align with personalization. However, this design also substantially suppresses personalization signals, resulting in the lowest personalization performance among all compared methods. Unlike imitation-style training that attempts to mimic all training samples equally, trajectory-based methods can learn more effectively from both high- and low-reward interactions, enabling the model to better distinguish desirable and undesirable behaviors during optimization. As a result, trajectory-level learning approaches such as SWIFT and TriAlign consistently outperform methods trained only on selected best actions (i.e. SFT). Figure 5 presents an example comparison between the base model and TriAlign. Before alignment, the model produces an incorrect answer and the response is more influenced by surface-level personalization patterns. In contrast, after alignment with TriAlign, the model preserves the user-preferred communication style while generating more logically consistent reasoning and the correct final answer. Overall, our proposed TriAlign achieves the best balance among the three objectives: truth accuracy, truth invariance, and personalization. It not only delivers significant improvements on in-domain benchmarks (\(p < 0.01\)), but also generalizes effectively to out-of-domain settings, demonstrating strong robustness across diverse evaluation scenarios.
Figure 6: Truth accuracy across social groups under gender and social-class category. Positive values indicate above-Mean performance within the same dataset, while negative values indicate below-Mean performance..
6.0.0.2 Performance Across Social Groups
Figure 6 compares the truth accuracy across different social groups on AIME25 and StereoSet. We observe that the base model exhibits substantial performance disparities across groups, with some groups receiving significantly higher performance gains while others suffer notable degradation. For example, on StereoSet, the base model shows a large imbalance across social classes, reaching +19.4% for the Lower class but dropping to -20.1% for the Upper class. Similar disparities are also observed across gender groups on AIME25. In contrast, TriAlign produces substantially more balanced performance distributions across all social groups. The relative advantages are closer to the mean performance, indicating that improvements are distributed more uniformly while still achieving higher accuracy.
| Dataset | Model | Pref\(\uparrow\) | Mean\(\uparrow\) | Worst\(\uparrow\) | Std\(\downarrow\) | Gap\(\downarrow\) | CV\(\downarrow\) |
|---|---|---|---|---|---|---|---|
| StereoSet | Base Model | 0.744 | 0.317 | 0.217 | 0.056 | 0.100 | 0.176 |
| TriAlign w/o NSW | 0.771 | 0.708 | 0.613 | 0.051 | 0.095 | 0.072 | |
| TriAlign w/o CM | 0.764 | 0.739 | 0.684 | 0.064 | 0.056 | 0.086 | |
| TriAlign (Ours) | 0.791 | 0.788 | 0.773 | 0.010 | 0.015 | 0.013 | |
| 1-8 | Base Model | 0.412 | 0.767 | 0.714 | 0.047 | 0.052 | 0.061 |
| TriAlign w/o NSW | 0.483 | 0.786 | 0.713 | 0.039 | 0.074 | 0.050 | |
| TriAlign w/o CM | 0.498 | 0.778 | 0.743 | 0.045 | 0.035 | 0.058 | |
| TriAlign (Ours) | 0.510 | 0.805 | 0.800 | 0.007 | 0.005 | 0.008 |
6.0.0.3 Ablation Study
Table 3 presents the ablation results of TriAlign on StereoSet and MMLU-Pro. Removing either the NSW objective or the consistency modeling component consistently degrades performance across truthfulness, truth invariance, and personalization metrics. Specifically, removing NSW leads to noticeable drops in Worst performance and larger disparities across social groups, showing that the fairness-aware optimization is important for balancing performance among groups. Meanwhile, removing the consistency modeling component substantially increases Std, Gap, and CV, indicating that the proposed consistency objective effectively reduces inter-group inconsistency. In addition, both ablated variants also show lower personalization performance compared to the full model.
7 Conclusion↩︎
This work takes one of the first steps toward aligning PLLMs to maintain consistent universal truths across diverse social groups while preserving personalization. We propose TriAlign, a novel offline MARL framework with a fairness-aware objective and an explicit penalty for truth inconsistency to balance these objectives. Experimental results show that TriAlign outperforms prior prompt-based methods on both in-domain and out-of-domain benchmarks. We hope TriAlign can serve as a foundation for future research on fairer and more reliable personalized AI systems.
Limitations↩︎
Our work has several limitations. First, although we simulate 75 social groups and construct 7.5K personas, the generated personas are still synthesized by LLMs and may not fully capture the complexity, diversity, and evolving nature of real-world users and intersectional identities. As a result, certain cultural and social nuances may remain underrepresented. Second, balancing universal truth consistency and personalization remains a fundamentally challenging problem that is far from fully solved. While our framework improves consistency across social groups while preserving personalization, achieving strong fairness, accuracy, and personalized adaptation simultaneously still requires further investigation and more robust alignment methods in future work.
Ethical Considerations↩︎
Our study has several ethical considerations. First, although we simulate diverse social groups and personas to study universal truth consistency in personalized LLMs, the generated personas may still reflect biases or stereotypes inherited from the underlying LLMs and source data. Consequently, some generated interactions may unintentionally reinforce societal stereotypes or oversimplify complex social identities. Second, our framework requires large-scale data simulation with Multi-agent System and model alignment processes with Offline RL, which can be computationally expensive and contribute to increased energy consumption and carbon emissions.
8 Appendix↩︎
8.1 Derivation of Fairness-Aware AWR↩︎
8.1.0.1 Step 1: From Our Objective to the Offline RL Objective.
We optimize the objective: \[{!}{ \mathcal{J}(\theta) = \frac{1}{m} \mathbb{E}_{x,\,\{g_i\}_{i=1}^{m},\,a_{i,t}\sim \pi_\theta(\cdot\mid s_{i,t})} \left[ \sum_{i=1}^{m} \left( R_i + \lambda_{\mathrm{NSW}} \log(R_i+\epsilon) \right) \right]. }\]
where \(R_i\) denotes the cumulative return of agent \(i\), \(\lambda_{\mathrm{NSW}}\) controls the trade-off between utility and fairness, and \(\epsilon>0\) ensures numerical stability.
In the offline RL setting, directly optimizing the policy may lead to distribution shift because actions outside the offline data distribution are insufficiently supported by the dataset. Therefore, for each state \(s_{i,t}\), we solve the following KL-constrained policy improvement problem: \[{!}{ \max_{\pi} \mathbb{E}_{a_{i,t} \sim \pi(\cdot \mid s_{i,t})} \left[ Q^{\pi}(s_{i,t},a_{i,t}) + \lambda_{\mathrm{NSW}} \log\left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right) \right] }\] subject to \[D_{\mathrm{KL}} \left( \pi(\cdot \mid s_{i,t}) \;\|\; \pi_{\mathcal{D}}(\cdot \mid s_{i,t}) \right) \le \delta .\]
Here, \(\mathcal{D} = \{(s_{i,t},a_{i,t},r_{i,t})\}_{x,i,t}\) is pre-collected dataset, and \(Q^{\pi}(s_{i,t},a_{i,t})\) estimates the expected cumulative return \(R_i\) after taking action \(a_{i,t}\) under state \(s_{i,t}\). We assume \[Q^{\pi}(s_{i,t},a_{i,t})+\epsilon>0,\] so that the logarithmic NSW term is well-defined.
8.1.0.2 Step 2: Lagrangian Formulation.
For a fixed state \(s_{i,t}\), we define the Lagrangian: \[{!}{ \begin{align} \mathcal{L}(\pi,\beta,\alpha) = & \int \pi(a_{i,t}\mid s_{i,t}) \left[ Q^{\pi}(s_{i,t},a_{i,t}) + \lambda_{\mathrm{NSW}} \log\left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right) \right] da_{i,t} \\ &- \beta \left( \int \pi(a_{i,t}\mid s_{i,t}) \log \frac{ \pi(a_{i,t}\mid s_{i,t}) }{ \pi_{\mathcal{D}}(a_{i,t}\mid s_{i,t}) } da_{i,t} - \delta \right) \\ &+ \alpha \left( \int \pi(a_{i,t}\mid s_{i,t})da_{i,t} -1 \right), \end{align} }\]
where \(\beta>0\) is the Lagrange multiplier for the KL constraint, and \(\alpha\) enforces the normalization constraint.
8.1.0.3 Step 3: Optimal Non-Parametric Policy.
Taking the functional derivative with respect to \(\pi(a_{i,t}\mid s_{i,t})\) and setting it to zero gives: \[{!}{ Q^{\pi}(s_{i,t},a_{i,t}) + \lambda_{\mathrm{NSW}} \log\left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right) - \beta \left( \log \frac{ \pi(a_{i,t}\mid s_{i,t}) }{ \pi_{\mathcal{D}}(a_{i,t}\mid s_{i,t}) } +1 \right) + \alpha =0. }\]
Solving for \(\pi(a_{i,t}\mid s_{i,t})\), we obtain: \[{!}{ \pi^*(a_{i,t}\mid s_{i,t}) \propto \pi_{\mathcal{D}}(a_{i,t}\mid s_{i,t}) \exp \left( \frac{ Q^{\pi}(s_{i,t},a_{i,t}) + \lambda_{\mathrm{NSW}} \log\left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right) }{ \beta } \right). }\]
Equivalently, \[{!}{ \pi^*(a_{i,t}\mid s_{i,t}) \propto \pi_{\mathcal{D}}(a_{i,t}\mid s_{i,t}) \exp \left( \frac{ Q^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) \left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right)^{\frac{\lambda_{\mathrm{NSW}}}{\beta}}. }\]
To mitigate instability when directly exponentiating \(Q^{\pi}\), we decompose: \[Q^{\pi}(s_{i,t},a_{i,t}) = V^{\pi}(s_{i,t}) + A^{\pi}(s_{i,t},a_{i,t}),\] where \[A^{\pi}(s_{i,t},a_{i,t}) = Q^{\pi}(s_{i,t},a_{i,t}) - V^{\pi}(s_{i,t}).\]
Then, \[{!}{ \exp \left( \frac{ Q^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) = \exp \left( \frac{ A^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) \exp \left( \frac{ V^{\pi}(s_{i,t}) }{ \beta } \right). }\]
Since \(V^{\pi}(s_{i,t})\) depends only on the state and not on the action, the term \[\exp(V^{\pi}(s_{i,t})/\beta)\] can be absorbed into the state-dependent normalization constant. Therefore: \[{!}{ \pi^*(a_{i,t}\mid s_{i,t}) \propto \pi_{\mathcal{D}}(a_{i,t}\mid s_{i,t}) \exp \left( \frac{ A^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) \left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right)^{\frac{\lambda_{\mathrm{NSW}}}{\beta}}. }\]
8.1.0.4 Step 4: Projection onto the Parameterized Policy.
Because \(\pi^*\) is non-parametric, we project it onto the parameterized policy \(\pi_\theta\) by minimizing \[D_{\mathrm{KL}}(\pi^* \| \pi_\theta).\]
This is equivalent to maximizing: \[{!}{ \theta^* = \arg\max_{\theta} \mathbb{E}_{(s_{i,t},a_{i,t}) \sim \mathcal{D}} \left[ w(s_{i,t},a_{i,t}) \log \pi_\theta(a_{i,t}\mid s_{i,t}) \right]. }\]
Thus, the final objective becomes: \[{!}{ \mathcal{L}(\theta) = - \mathbb{E}_{(s_{i,t},a_{i,t}) \sim \mathcal{D}} \left[ w(s_{i,t},a_{i,t}) \log \pi_\theta(a_{i,t}\mid s_{i,t}) \right]. }\]
where the weight is: \[{!}{ w(s_{i,t},a_{i,t}) = \underbrace{ \exp \left( \frac{ A^{\pi}(s_{i,t},a_{i,t}) }{ \beta } \right) }_{\text{Standard Weight}} \cdot \underbrace{ \left( Q^{\pi}(s_{i,t},a_{i,t})+\epsilon \right)^{\frac{\lambda_{\mathrm{NSW}}}{\beta}} }_{\text{NSW Weight}}. }\]
The constant \(\epsilon\) ensures that the NSW term remains well-defined even when \(Q^{\pi}(s_{i,t},a_{i,t})\) is close to zero.
None
Figure 8: No caption.
8.2 Prompts↩︎
Question: {question}
Instruction: Please answer the question for the following user profile: {social group}
{conversation history}
You are an assistant that determines the response style of an answer.
Question: {question}
Answer: {answer}
Available Rubrics: {rubrics}
Task: Analyze the question and identify the appropriate level for each rubric reflected in the answer.
Output Format: Return the result strictly as valid JSON with keys exactly matching the rubric names, and values indicating the preferred level for that rubric.
Example Output: {example output}
Important:
Only output JSON. Do not include explanations or additional text.
Ensure all rubric keys from the provided rubrics appear in the output.
Task: You are an evaluator whose task is to determine whether the model’s final answer is factually correct with respect to the provided ground truth.
Evaluation Rules:
Evaluate only the final conclusion.
Ignore incorrect intermediate reasoning if the final answer is correct.
Semantic equivalence counts as correct.
Minor wording or formatting differences should not affect correctness.
If the correct answer is clearly stated, return
1.
Context:
# Question: {question}
# Ground Truth Final Answer {ground_truth}
# Answer to Evaluate {answer}
Required Output Format:
<reasoning> optional reasoning </reasoning>
<truth_acc>1</truth_acc>
or
<reasoning> optional reasoning </reasoning>
<truth_acc>0</truth_acc>
Question: {question}
Instruction: You are given {number_agents} answers generated by different agents. Compare all answers and identify differences regarding universal truth or factual correctness in both the final conclusions and reasoning steps.
If there are inconsistencies:
Identify which statements conflict across agents.
For each agent, generate concise feedback describing how its answer differs from the others.
Use the following style:
“While your answer claims that [statement A], other answer(s) suggest [negation or conflicting statement].”
If all answers are universally consistent:
State that all agents agree on the same universal truth.
Then provide concise suggestions for improving answer style, clarity, or helpfulness based on other agents’ responses.
Consistency Rule:
Ensure all feedbacks are mutually consistent.
Do not produce contradictory judgments across agents.
Answers:
{answers}
Output Format:
Feedback to Agent <agent_name>: [feedback]
<split>
You are an assistant that determines the user’s preferred response style based on their profile and the question.
User Profile: {user_profile}
Question: {question}
Available Rubrics: {rubrics}
Task: Analyze the user profile and the question, and identify the level for each rubric that the user would prefer in the answer.
Output Format: Return the result strictly as valid JSON with keys exactly matching the rubric names, and values indicating the preferred level for that rubric.
Example Output: {example output}
Important:
Only output JSON. Do not include any explanations, notes, or text outside the JSON.
Ensure all rubric keys from the given rubrics are present in the output.
| Preference Dimension | Preference Type | Description | Decision Rule |
|---|---|---|---|
| Information Complexity | Basic | Uses everyday language, avoids jargon, and explains concepts explicitly. | If a complete beginner can understand without additional background knowledge. |
| Intermediate | Introduces some technical terminology while remaining accessible. | If readers with basic familiarity can follow comfortably. | |
| Expert | Uses advanced terminology and domain-specific concepts with minimal explanation. | If understanding requires professional or domain expertise. | |
| Structure | Paragraph | Continuous prose with implicit logical flow. | If ideas are presented naturally without explicit formatting. |
| Bullet Points | Information is divided into discrete independent points. | If readability is improved through segmentation rather than sequence. | |
| Step-by-Step | Sequential structure where each step depends on previous steps. | If order is essential for understanding or execution. | |
| Evidence Style | Explanation + Example | Uses illustrative or hypothetical examples for intuition. | If examples primarily clarify concepts rather than validate claims. |
| Explanation + Hard Evidence | Supports claims with measurable or verifiable evidence. | If credibility depends on statistics, studies, or empirical findings. | |
| Explanation + Anecdotal / Expert Evidence | Relies on authority, experience, or real-world examples. | If credibility mainly comes from expertise or historical cases. | |
| Information Density | Simplified | Focuses only on key ideas while minimizing detail. | If adding more detail would mainly increase complexity. |
| Balanced | Balances clarity and completeness with moderate detail. | If the response maintains both readability and sufficient depth. | |
| Dense | Packs large amounts of information into compact text. | If the response maximizes information density and completeness. | |
| Reasoning Style | Intuitive | Relies on analogies and conceptual mental models. | If understanding is driven mainly by intuition rather than formal logic. |
| Analytical | Uses systematic decomposition and logical reasoning. | If reasoning follows explicit step-by-step logical structure. | |
| Causal Reasoning | Explains mechanisms and cause-effect relationships. | If explanations focus on why and how outcomes occur. | |
| Tone | Friendly | Conversational, approachable, and engaging. | If the response resembles informal human conversation. |
| Neutral | Objective and emotionally neutral presentation. | If the response focuses strictly on clarity and factuality. | |
| Professional | Formal and precise communication style. | If the tone resembles academic or workplace communication. |
8.3 Data Statistics↩︎
| Split | Dataset | # Samples |
|---|---|---|
| Test | AIME25 | 180 |
| StereoSet | 2352 | |
| TruthfulQA | 4902 | |
| MMLU-Pro | 420 | |
| Train | COLM-LIMO | 817 |
| LogicQA | 651 | |
| StereoSet | 500 | |
| SimpleQA | 300 |
| Split | # Categories | # Social Groups | # Personas / Group | # Personas |
|---|---|---|---|---|
| Train | 14 | 75 | 80 | 6000 |
| Test | 2 | 6 | 20 | 120 |
