1 Introduction↩︎

Learning from human feedback has been effective at aligning large language models (LLMs) to human preferences in many domains such as summarization, conversational agents, and safety. In modern applications, language model alignment is generally a multi-objective problem due to the diversity of human preferences along different, often conflicting axes (e.g., conciseness versus detail, factuality versus creativity). Common LLM alignment methods such as supervised finetuning, reinforcement learning (RL) finetuning, and preference optimization are unidirectional i.e. they optimize a single best response, scalar reward, and preference label respectively. Thus, these methods cannot represent different objective weights with a single policy, instead requiring additional tuning for the multi-objective setting.

Linear scalarization of multiple objectives [1] with weights for each objective is a common solution for accommodating multiple objectives in LLM alignment. Multiple objectives are normalized into one scalar reward or preference by scaling the reward signal for each objective with a weight that represents the relative importance of that objective [2]. However, in practice, it is infeasible to train a separate policy for each objective preference combination, since preferences vary widely between users and over time. Discretizing objective weights produce a finite number of weight combinations to train specialist policies for, but this number grows exponentially with the number of objectives, making it prohibitively expensive.

Recent LLM alignment methods train steerable policies [3] that can be controlled at inference time to generate outputs corresponding to different reward weightings, without retraining. In particular, a steerable policy takes a desired objective weight combination as an additional input at inference time and generates an output attempting to maximize this weighted combination of rewards.

Steerable methods broadly fall into two categories: 1) parameter-based conditioning and 2) prompt-based conditioning. Parameter-based conditioning [4], [5] still relies on separate parameters (spanning all or a subset of the policy model) for each objective but, instead of retraining a policy for each objective weight combination, obtains policy weights as a linear combination of weights from the existing single-objective policies. This model souping [6] may be combined with multi-task training [7] by dynamically sampling objective weights during training. While producing strong results in terms of Pareto optimality and steerability, parameter-based conditioning methods may be difficult to deploy in production owing to higher inference costs. Having to do on-the-fly souping for each query, ostensibly with different objective weights, gets computationally expensive with an increasing number of objectives, as well as the granularity of objective weights – even with low-rank decomposition of model parameters [8].

Prompt conditioning-based methods rely on only a single policy, trained conditionally on objective weights which are usually prefixed to the input to the policy. Prompt-based conditioning has been combined with RL finetuning (RLFT) [5] and supervised finetuning (SFT) + rejection sampling-based methods [11], [12]. These methods can be very efficient to train and deploy, but have so far demonstrated less steerability and worse performance compared to other methods.

Separate from reinforcement learning, preference optimization algorithms [15]–[17] optimize a policy directly from pairwise preference data, foregoing training a reward model followed by an RLFT loop, and have shown great promise over RL-based methods. Some works extend preference optimization to the multi-objective setting [10], [13]. However, they are either not steerable i.e. require retraining for a new objective weight combination, or require multi-objective preference labels/scores in the preference data, which are not always available: preference data for different objectives is generally available in disjoint datasets. Furthermore, offline preference optimization has been shown to overfit relatively quickly [18].

Online variants of DPO and other preference optimization algorithms have been proposed [18]–[21], which sample responses and score them with reward models/AI feedback/self-feedback [22] to create preference pairs for preference optimization. These avoid the distribution shift issue in the offline variants of these algorithms, since the responses are now on-policy.

Combining the best of these themes - steerability through prompt-based conditioning and performance through online preference optimization - we propose Multi-Objective Online Direct Preference Optimization or MO-ODPO, a multi-objective extension to online preference optimization methods. MO-ODPO trains a single policy capable of representing multiple objective weight combinations along the Pareto frontier, and on-policy sampling for preference optimization that leads to improved performance over RLFT and parameter interpolation-based methods. The proposed method is the first to apply online DPO to the multi-objective setting, yielding strong reward trade-offs and excellent steerability on two multi-objective LLM alignment benchmarks (Anthropic-HH [9] and Reddit TL;DR [23]), and the prompt-based conditioning obviates the need for parameter interpolation at inference time for each query. Our contributions are summarized below:

• We propose MO-ODPO, an online preference optimization algorithm for multi-objective LLM alignment that trains a single policy for multiple objectives with easy inference-time steerability through prompt conditioning.

• We extensively evaluate MO-ODPO quantitatively and qualitatively against strong baselines on two popular multi-objective LLM alignment benchmarks with conflicting objectives and show that MO-ODPO Pareto-dominates other baselines while yielding highly steerable policies with medium as well as small model sizes.

2 Related Works↩︎

Our work explores prompt-conditioned on-policy alignment methods using pairwise losses as DPO [15] while utilizing reward models trained in the style of Bradley-Terry Models [24] to rank sampled responses. We present works related to these below.

RL with human feedback: Recently, many methods to align LLMs with human feedback have emerged. Initial methods, involving on-policy reinforcement learning (RL) proposed by [25], [23], [26], [9] utilize RL from human feedback (RLHF) to achieve alignment with human preferences. These involve training an LLM as a reward model by minimizing ranking loss [24] from pairwise human preferences. Usually an online RL setup of PPO [27] is used to optimize the policy LLM to maximize rewards from the reward model.

Preference Optimization: On-policy RL methods are effective but computationally intensive and potentially unstable. To that end, efficient off-policy methods that can align LLMs to pairwise preferences directly without training reward models have been introduced [15]–[17], [28], [29]. These methods optimize the policy by minimizing pairwise alignment loss calculated over a pair of responses annotated by human preferences; DPO [15] is among the best known of these.

As efficient as these off-policy alignment methods are, they are prone to overfitting and collapsing [18], necessitating online preference alignment algorithms [18], [30]. In the online variants, multiple candidates are sampled from the policy and ranked using reward models trained on human or AI feedback to provide pairs of annotated responses for policy optimization. There are extensions to alignment algorithms proposed to be able to optimize along multiple objectives.

Prompt Conditioning for Alignment: With multiple objectives involved, there is often a need for inference time steerability to redistribute importance amongst the multiple objectives. One way to introduce steerability is to add intended objective weights into the policy input at training as well as inference time, so that the policy learns to generate responses in accordance with the weights in the input. [5] introduce P-MORL, combining prompt conditioning with binary objective weights with an RLFT loop. Other works introduce steerability through prompt conditioning, and then train the policy with iterations of SFT and multi-objective rejection sampling using reward models [11], [12].

MODPO [10], which extends offline DPO to the multi-objective setting exactly with margin reward models, and CPO [13], which proposes a controllable DPO-style method where objective weights can be specified at inference time as constraints, are similar to our work in utilizing DPO in a multi-objective setting. However, MODPO is not steerable as it requires a different finetuned policy for each objective weight, and recent experiments show its limited efficacy in aligning different objectives [12]. CPO, while highly related, requires preference scores/labels for each objective for each response that it is trained on, which may be infeasible since preference datasets for different objectives are usually disjoint, and collecting high-quality multi-objective labels is laborious and likely expensive, even with AI feedback [31].

Parameter Conditioning for Alignment: These methods condition the policy with objective weights by maintaining some separate parameters for each objective and combining them at inference time as per the objective weights to generate the new policy. Rewarded Soups [4], [5] trains specialist policies for each objective and, using linear mode connectivity [32], linearly interpolate model parameters between the specialist policies for zero-shot generalization to new objective weights.

CLP [7] combines this parameter-space conditioning with multi-task training i.e. optimizing for multiple objectives in a single step based on sampled objective weights, and also experiment with sharing a subset of per-objective policy parameters across all objectives for efficiency. They also show that combining parameter and prompt-based conditioning can improve steerability. We extend this work by leveraging online preference optimization in place of RLFT, while avoiding the increased inference-time cost from parameter-space conditioning. In a similar vein, Panacea [33] combines multi-task training with singular value decomposition (SVD)-based low-rank adaptation of the human preference matrix to learn Pareto-optimal policies.

Another line of work proposes inference-time decoding methods which can satisfy multiple objectives using single-objective policies without retraining, at the cost of increased inference time [14], [34]. Table 1 provides a comparison of the most pertinent related works.

3 Multi-Objective Online DPO↩︎

A single training step of our proposed Multi-Objective Online DPO (MO-ODPO) algorithm is broken down as follows. Algorithm 2 and Figure 1 further explain the algorithm.

3.1 Background↩︎

We consider the problem of training a steerable policy \(\pi_{\theta}\) that can be conditioned on \(K\) objectives, each with a corresponding reward function \(R_1\ldots R_K\). Specifically, we train a policy \(\pi_{\theta} (y|x, w_1\ldots w_K)\) to generate responses given an input prompt \(x\) and objective weights \(w_1\ldots w_K\) where the objective weights reflect the relative importance of each reward. We constrain the objective weights to be non-negative and summing to 1 i.e. \(\sum_{k=1}^Kw_k = 1\): this allows us to easily interpret the objective weights as percentages and avoid changing the reward function normalization.

3.2 Objective Weight Sampling↩︎

The first step is to sample weights \(w_1\ldots w_K\) for each objective \(R_1\ldots R_K\). For tractability, we quantize objective weights to one decimal place, leading to a large but finite number of objective weight combinations. Similar to [7], we use Dirichlet sampling to sample objective weights, which allows us to experiment with varied sampling strategies by varying the Dirichlet \(\alpha\) parameter: \(\alpha = 1\) is equivalent to uniform sampling over the objective weight space, \(\alpha < 1\) tends to sample at the extremeties of the weight space i.e. where one objective is maximized at the cost of others, and \(\alpha > 1\) tends to sample at the center of the weight space i.e. with roughly equal weights for all objectives. We discuss the role of sampling further in Section 6.1.

3.3 Prompt Conditioning Mechanism↩︎

Once the objective weights are sampled, we construct an input prefix to condition the policy to generate a response as per the objective weights specified in the said prompt. As an example, for reward names Helpfulness and Harmlessness, our input prefix looks like the following:

[Begin System Instruction]
Helpfulness: \(<w_1>\), Harmlessness: \(<w_2>\)
[End System Instruction]

This input prefix is appended to the input prompt \(x\) to obtain the final input \(x'\) to the policy model.

3.4 Sampling and Preference Pair Construction↩︎

Given the current policy \(\pi\) and input \(x'\), we sample two completions \(y_1\) and \(y_2\). For each completion, we obtain reward scores \(s_1^1 \ldots s_1^K\) for \(y_1\) and \(s_2^1 \ldots s_2^K\) for \(y_2\) for each of the \(K\) objectives. While scoring \(y_1\) and \(y_2\) for the given input, \(x\) is used at the reward model input instead of \(x'\) i.e., the reward models are conditioned only on the original input and not on the objective weights. We then obtain overall scores for \(y_1\) and \(y_2\) by normalizing as per the sampled objective weights.

\[s_{i} = \sum_{k=1}^K s_i^k \times w_k\,,i\,\in\,\{1, 2\}\]

Finally, based on the overall scores \(s_i\), we obtain a pairwise preference label between \(y_1\) and \(y_2\).

\[(y_+, y_-) = (y_1, y_2) \,\mathtt{if}\, s_1\,>\,s_2\,\, \mathtt{else}\,\, (y_2, y_1)\]

3.5 DPO Loss↩︎

With this preference pair constructed, we can optimize the policy \(\pi_{\theta}\) as per the classic DPO loss [15]. \[\mathcal{L}_{DPO} = -log\,\sigma \left( \beta log\frac{\pi_{\theta}(y_+\,|\,x')}{\pi_{\theta^0}(y_+\,|\,x')} - \beta log\frac{\pi_{\theta}(y_-\,|\,x')}{\pi_{\theta^0}(y_-\,|\,x')} \right)\] where \(\pi_{\theta^0}\) is our anchor policy and \(\beta\) is the KL regularization coefficient. Note while \(\pi_{\theta^0}\) is not trained with the objective weight prefix, using \(x'\) vs \(x\) to obtain anchor log probabilities \(\pi_{\theta^0}(y_{\{+,-\}}\,|\,x)\) did not change our results. Any pairwise preference loss e.g., IPO [16], [30] and Slic-HF [17] can be utilized here instead; we leave this to future work.

4 Experiments↩︎

4.1 Datasets and Rewards↩︎

We evaluate MO-ODPO on two popular multi-objective LLM alignment benchmarks with two rewards each:

• Anthropic-HH [9], with rewards for Helpfulness and Harmlessness which are both obtained from training reward models on this dataset.

• TL;DR Summarization [23], with a TL;DR reward model for summary quality from the same dataset, and an NLI reward model for textual entailment and factuality [7], [35].

For both benchmarks, our chosen rewards are anti-correlated. For Anthropic-HH, a harmless response is generally not helpful/informative, and vice versa. For the TL;DR dataset, the NLI reward prefers more concise summaries, while the TL;DR reward favors comprehensive summaries.

4.2 Baselines↩︎

We compare MO-ODPO with the following baselines:

• Rewarded Soups [4], [5], where we optimize a specialist policy with RL finetuning (RLFT) separately for each reward, and obtain intermediate policies with linear post-hoc parameter merging.

• ODPO Soups, like Rewarded Soups but with the specialist policies trained on a single objective with online DPO [19] instead of RLFT.

• Prompt-conditioned RLFT, similar to the P-MORL baseline from [5], except with objective weights provided as described in Section 3.3 instead of the binary format in [5]. For brevity, we refer to this baseline as P-MORL in the experiments.

• Rewards-in-Context [12], a method that adds expected reward values for each objective in place of objective weights, and is trained with multiple iterations of SFT and multi-objective rejection sampling.

We include results comparing with baselines that require retraining a separate policy for each operating point, namely MO-RLHF and MODPO [10], in the supplementary material, since these are hard to compare directly with MO-ODPO or the baselines due to a lack of steerability and higher compute requirements.

4.3 Experimental Settings↩︎

For all experiments we use PaLM 2 XS (Otter) [36] for both the policy and reward models, except in Figure 4 where the policies are trained on PaLM 2 XXS (Gecko). We initialize all reward models and LLM alignment methods from instruction-tuned variants of these models and forego an SFT step for a fairer comparison. All reward scores are linearly scaled roughly to \([0, 1]\). For P-MORL and MO-ODPO, the best results were obtained with Dirichlet \(\alpha=1.0\) for Anthropic-HH and \(0.7\) for TL;DR.

For all RL experiments, we use REINFORCE [37], [38] with control variate, which is simpler than PPO [27] yet powerful and has been successfully used for language tasks [31], [35]. All models were trained on TPU-v5e [39] with Adafactor [40], a batch size of 128 and a learning rate of 1e-5 (the offline stage of Rewards in Context has learning rate 1.4e-4).

Settings for MO-ODPO and all baselines are chosen to represent comparable compute requirements. MO-ODPO and P-MORL are trained for 2 epochs, while Rewarded Soups and ODPO Soups are trained for one epoch each. Model souping methods train a separate policy for each objective, so this results in the same total training time as MO-ODPO and P-MORL with 2 objectives. For Rewards in Context, the offline stage and 2 online stages are set to be similar in compute, yielding 6 epochs of SFT for the offline stage and 2 epochs of SFT for each online stage. As in [7], we saw that using a KL regularization constant of 0.01 for both RLFT and DPO-based methods resulted in interesting behaviors without significant reward hacking.

5 Results↩︎

5.1 Pareto Curves↩︎

Figures 3 and 4 show the Pareto curves for PaLM 2 XS and XXS policies respectively, generated by sampling responses to validation set prompts using MO-ODPO and baseline policies at weight combinations \((i, 1.0-i)\,i\in\{0.0, 0.1\ldots 1.0\}\) (total 11 weight combinations). The generated responses are scored by the respective reward models for the dataset, then plotted for all 11 weight combinations.

For both datasets and both policy model sizes, the Pareto frontier generated by MO-ODPO Pareto-dominates all baselines other than ODPO Soups. Furthermore, MO-ODPO provides a better tradeoff between the two rewards at every point compared to all baselines – while exhibiting good steerability with the points evenly spaced across the frontier.

While Online DPO Soups attains the highest individual reward values for each objective with a very wide and steerable frontier, its reward tradeoff is still inferior to MO-ODPO owing to the specialist policies doing poorly on the reward they were not optimized for, which is mitigated by multi-objective training in MO-ODPO. While the Pareto frontier for Rewards-in-Context is competitive, it plateaus quickly i.e. does not improve upon further training (see Section 6.2). P-MORL’s frontier is less steerable than that of MO-ODPO, demonstrating the potential of online preference optimization algorithms in this setting.

Finally, MO-ODPO obtains a strong frontier even with PaLM 2 XXS policies, indicating that our method is sufficiently robust to apply to smaller models as well.

5.2 Autorater Evaluation↩︎

To complement our quantitative evaluation with Pareto fronts generated using the rewards that MO-ODPO and baselines are trained to optimize on – and thus liable to hack [41], [42]), similar to [7], we conduct an automated evaluation of generation quality and steerability using an off-the-shelf LLM as a zero-shot rater. For this evaluation, we use Gemini 1.0 Ultra [43] as a zero-shot evaluator to compare 1000 responses from MO-ODPO and baselines, both based on PaLM 2 XS, for their helpfulness and harmlessness for Anthropic-HH, and for summary quality and conciseness for TL;DR. Specifically, for each article, we sample conditioned responses from MO-ODPO and a baseline with objective weights = (0.8, 0.2) for high helpfulness/NLI and (0.2, 0.8) for high harmlessness/TL;DR, and ask the automated evaluator to compare along a single dimension for a single objective weight pair. Following [31], we permute the model responses and consider the evaluator’s verdict as valid only if it holds across both permutations. For a prompt, we label one algorithm as the winner only if it is better than the other along one dimension, while winning or tying along the other dimension.

Figure 5 shows win rates for MO-ODPO over baselines in this setup. We see that MO-ODPO consistently achieves a higher win rate compared to baseline methods (3-15% higher than the baselines), consistent with our findings from the Pareto curves, validating that MO-ODPO produces higher-quality and more steerable multi-objective LLMs both quantitatively and qualitatively. The improvements are more noticeable for Anthropic-HH than for TL;DR. The win rate for MO-ODPO is the lowest against P-MORL, indicating the efficacy of prompt conditioning with RLFT as well as online DPO compared to parameter souping-based methods, similar to [14] who find Rewarded Soups qualitatively sub-optimal for unseen objective weights.

5.3 Qualitative Analysis↩︎

Table 2 provides an example of generations on Anthopic-HH from PaLM 2 XS policy models for MO-ODPO and baselines, at two objective weights. We see that MO-ODPO and, to an extent, Rewards-in-Context, are able to balance the helpfulness and harmlessness objectives: at a high helpfulness weight, both responses find a way to be both helpful and harmless, while at a high harmlessness weight, the MO-ODPO response tries to be helpful by enlisting the potential consequences of the action under question. We provide more examples in the supplementary material.

6 Discussion↩︎

6.1 Role of Sampling for Prompt Conditioning↩︎

As [7] point out, prompt conditioning-based methods may exhibit mode collapse. To study this, we vary the Dirichlet \(\alpha\) hyperparameter for MO-ODPO (see Section for details). Figure 6 shows the Pareto curves for Anthropic-HH and TL;DR Summarization with different values of \(\alpha\). We see that for Anthropic-HH, a higher \(\alpha\) results in a more steerable policy while a lower \(\alpha\) results in a bimodal collapse of the policy i.e. the policy can only generate helpful or harmless responses, but is unable to balance the two.

In contrast, for TL;DR, we see that a higher \(\alpha\) results in lower steerability, akin to a mode collapse where the policy generates responses around a single point on the Pareto frontier. For both tasks, though, it is also apparent that mode collapse can be avoided easily by suitably choosing the objective weight sampling distribution for MO-ODPO without compromising on the quality of the Pareto reward frontier.

6.2 Training Dynamics↩︎

Figure 7 compares the training progress for MO-ODPO and Rewards in Context i.e. their Pareto curves when trained for 1-3 epochs on the TL;DR benchmark. We see that MO-ODPO has a small spread after 1 epoch, but already gets higher reward values than Rewards in Context: further training improves steerability considerably. Rewards in Context, in contrast, obtains a well-spread Pareto curve early but, possibly due to the nature of SFT, does not generalize beyond. A similar pattern is seen for the Anthropic-HH benchmark.

7 Conclusion↩︎

We present MO-ODPO, an online preference optimization algorithm for single-policy multi-objective LLM alignment that needs neither aligned data to train, nor parameter interpolation at inference time, achieving easy inference-time steerability through prompting. MO-ODPO Pareto dominates strong baselines quantitatively and qualitatively on Anthropic-HH and TL;DR Summarization, two popular LLM alignment benchmarks while providing an excellent trade-off between quality and steerability.

Conditioning with objective weights can guide the outcomes but not guarantee them: improving adherence to the desired objectives is an important future direction. Other future directions include, but are not limited to, exploring more intelligent objective weight sampling [44], soft tokens for conditioning [45], extending more preference alignment algorithms to the multi-objective setting, and exploring multi-objective AI feedback as well as other on-policy feedback mechanisms to counter the reward model distributional shift problem.

Acknowledgements↩︎

We thank Biao Zhang, Olivier Bachem, Edouard Leurent, Joëlle Barral, Tania Bedrax-Weiss and anonymous reviewers for their feedback, which helped us improve the paper a lot.

8 Pareto curves adjusted for KL divergence↩︎

Given the same KL regularizer constant \(\beta=0.01\) was used for both RL finetuning-based algorithms as well as all DPO-based algorithms (except the SFT-based Rewards in Context), Figure 8 shows the Pareto curves for the KL-adjusted reward, which is calculated as \(r' = (1-\beta)r + \beta KL[\pi_{\theta}||\pi_{\theta^0}]\). The rewards on both axes are adjusted according to the KL divergence values sampled over the validation set and averaged separately for each objective weight combination. We see that even with the KL-adjusted rewards, MO-ODPO outperforms all baselines, including Rewards in Context which is not adjusted for KL divergence. At the same time, P-MORL underperforms Rewarded Soups in this setting, indicating that prompt conditioning with vanilla RL finetuning may require a less strict KL regularizer to be effective.

9 Pareto curves adjusted for response length↩︎

Since both RL and DPO-based methods tend to demonstrate a high correlation between reward and response length [46], we also generate Pareto curves with a length penalty applied similar to KL regularization. More concretely, \(r' = r - \gamma.len(y)\). given the maximum response length for all experiments is set to 128 tokens and the rewards lie approximately in \([0, 1]\), we choose \(\gamma=0.002\). Figure 9 shows the Pareto curves for the length-penalized reward: we see that MO-ODPO still Pareto-dominates all other methods, but the curves are much closer together, indicating that MO-ODPO generates longer responses on average. We leave further investigating this length-reward correlation exhibited by MO-ODPO to future work.

10 Comparison with specialist policies↩︎

Figure 10 compares MO-ODPO to specialist policies i.e. policies trained to optimize a fixed objective weight combination of the two rewards in question. In addition to the single-objective policies i.e. trained on objective weight combinations (0, 1) and (1, 0), we train policies at (0.25, 0.75), (0.5, 0.5), and (0.75, 0.25), using RLFT, online DPO, and MODPO [10] which relies solely on the offline preference data. Each policy is trained for 1 epoch; thus the total computation budget for the specialist policies is 2.5 times the computation budget for MO-ODPO.

We see that MO-ODPO is comparable to Online DPO Specialist policies, even with a smaller computation budget. RLFT specialist policies, while better than Rewarded Soups, are still Pareto-dominated by MO-ODPO. Interestingly, while MODPO produces a strong frontier for Anthropic-HH, it does not generalize well for the TL;DR dataset, potentially because the pairwise preference datasets for the NLI reward model and the TL;DR reward model are distributionally far apart.

11 Additional implementation details↩︎

All models are trained with a maximum response length of 128 tokens for both datasets. With this sequence length and the aforementioned batch size of 128 on TPUv3, it takes \(\sim\) 4.4 hours to run one epoch of RL finetuning, and \(\sim\) 5.2 hours to run one epoch of MO-ODPO. In contrast, it takes \(\sim\) 0.8 hours to run one epoch of SFT (for Rewards in Context), but we evaluate all methods with a roughly equivalent amount of computation \(\sim\) 2 epochs of RLFT, as described in Section 4.3. For all experiments, regardless of training loss, an evaluation run takes roughly an hour.

We arrived at the best hyperparameters using a sweep over the following: learning rate over [1e-5, 1e-6], KL regularizer \(\beta\) over [0.1, 0.01], and Dirichlet \(\alpha\) over [0.3, 0.5, 0.7, 1.0, 1.5]. Results for each experiment are reported based on one run.

12 Example generations↩︎

Tables 3 and 4 contains more example generations from Anthropic-HH and TL;DR Summarization respectively. We see that all models, but particularly MO-ODPO, does a good job of balancing two rewards. For Anthropic-HH, its responses find ways to be helpful even when a mostly harmless response is desired. For TL;DR, we see that Rewards in Context generates summaries that are far too short and, for the high-NLI summary, misses the most important bits. P-MORL generates good summaries but adds a "The post is about …" sentence to most of its responses for high NLI, indicating reward hacking. MO-ODPO summaries with high NLI are brief, with informal grammar, which is likely desired.

13 Prompts for automated evaluation↩︎

We include below the prompts to judge the helpfulness and harmfulness of a response respectively, following [31], followed by prompts to judge the summary conciseness and quality respectively, following [7]. Note that, for the harmfulness prompt, the evaluator’s responses have to be inverted to determine the more harmless response.
Helpfulness:

Harmfulness:

Conciseness (corresponds to the NLI reward):

Quality (corresponds to the TL;DR reward):

References↩︎