1 Introduction↩︎

Recent advancements in abstractive summarization systems [1], [2] are often impeded by their tendency to output information that is either contradicting or unsupported by their input article, often termed as “hallucinations” or factual inconsistency [3]–[5]. While these systems produce highly relevant and coherent text, this lack of factual consistency often limits their wide-spread adoption in real-world applications. An example is depicted in 1, where the highlighted statement in the summary, while plausible, has no support in the input article.

Since widely-used metrics such as ROUGE [6] were shown to be inefficient for detecting hallucinations, many recent research efforts introduced novel automatic metrics for measuring factual consistency [7]–[9]. We propose to leverage these automatic metrics within a reinforcement learning (RL) framework at training time. Specifically, we apply textual entailment assessment (a.k.a.natural language inference, or NLI; [10], [11]) between the source article and the generated summary as a reward.

Our reward is based on the well studied textual entailment task [12]–[14], for which there are many publicly available datasets [15], [16]. While these NLI datasets are not specific to summarization, it was shown that classifiers trained on these datasets perform well in detecting factual inconsistencies in summarization and other generative tasks [17]. Because faithful summaries must be textually entailed from the corresponding input documents, using such a reward explicitly should guide a summarization model towards generating more factually consistent summaries. Yet, a high-quality summary should also be coherent and contain relevant information [18], aspects which may not be captured by entailment alone. Moreover, a reward that is based only on entailment raises the risk of degenerate solutions, leading to either highly extractive [19] or less informative summaries [20]–[22].

To address these issues, we propose Reinforcement Learning with Entailment Feedback (RLEF): Start with a model trained to produce summaries with the conventional cross-entropy objective, and further fine-tune it using RL with an entailment-based reward. Throughout the RL procedure, we constrain the candidate models to stay close to the initial model. This way, while the model is being corrected for higher consistency, it also retains other summarization capabilities that were learnt with the maximum-likelihood (MLE) objective. In this work we explore the consistent vs.informative trade-off in our RL-based summaries w.r.t.various aspects including model scale, regularization and decoding strategies. We find those aspects to be highly important and interdependent for the final model performance, highlighting the importance of carefully tuning them.

Our work stands in contrast to two prior RL-based approaches. The first approach induces a reward function from human feedback that encompasses various task-specific requirements into a single value [23], [24]. Collecting such feedback is expensive and requires dedicated data collection for each target task. In contrast, we use readily-available models and datasets for the reward, which address a specific aspect of generation that is generic across many different tasks. Other works modeled the reward using different similarity functions between the reference and the generated summaries [25], [26], thus requiring reliable reference data. Instead, our reward function evaluates the generated output only w.r.t.the input, enabling to train using RL on data without reference summaries. We evaluated our approach on the widely used XSum [27] dataset, using both automated metrics and human raters. The results show considerable improvements over strong baselines for factual consistency, salience, and conciseness of the generated summaries.

2 Method↩︎

We would like to increase factual consistency using an entailment-based reward, while retaining the high salience and coherence that current summarization models already obtain. To achieve this, we propose to initialize an RL policy with a summarization model trained on supervised data (the anchor model). From there, in each RL-based training step we update the parameters according to two signals: an entailment reward and a regularization term grounded on the anchor model. During RL training, the entailment reward directs the model towards increased faithfulness, while the regularization term keeps the model from drifting to degenerate solutions and “forgetting” how to summarize. The process is illustrated in 2.

2.1 RLEF: RL from Entailment Feedback↩︎

2.1.0.1 Problem Formulation.

We denote the input document and output summary as \(x, y\) respectively. Let \(\mathbb{V}\) denote the input and output vocabulary, and \(y_{:n}=(y_1,..., y_n)\) denote the generated summary up to the \(n\)-th token. We define the token-wise generative summarization process as a deterministic Contextual Markov Decision Process (CMDP, [28]) with observable context, where the context is the input text \(x\), the state at the \(n\)-th token generation is the sequence generated thus far \(y_{:n-1}\), and the action space is defined over the vocabulary \(\mathbb{V}\). A policy \(\pi(\cdot \mid y_{:{n-1}}, x)\), is a probability distribution over all tokens in \(\mathbb{V}\), conditioned on the context and state. We note that following this formulation, the policy is identical to a token-level auto-regressive language model [29]. The RL objective is to find the optimal policy, which maximizes the cumulative reward signal.

2.1.0.2 Rewards.

We use an NLI classification model as a factual consistency reward signal. Since the model is trained to evaluate complete utterances and expects as input a grammatical premise (document) / hypothesis (summary) pair, we use sequence-level rewards and define the token-level NLI reward to be zero on every token except for the end-of-sequence (EOS) token. For the EOS token we set the reward to be the log-probability for an “entailment” decision according to the NLI classifier, using \(x\) as the premise and \(y_{:n}\) as the fully generated hypothesis: \[r^{\mathop{\mathrm{NLI}}}(y_n ; y_{:n-1}, x)= \begin{cases*} \mathop{\mathrm{NLI}}(y_{:n};x) & y_n = {\scriptstyle \mathtt{[EOS]}};\\ 0 & otherwise, \end{cases*}\] where \({\scriptstyle \mathtt{[EOS]}}\) is an end-of-sequence symbol, and \(\mathop{\mathrm{NLI}}(y_{:n} ; x) = \log \Pr(\text{entailment}\mid y_{:n}, x)\).

To retain the summarization capabilities of the anchor model, we use Kullback-Leibler (KL) regularization to keep the RL-based policy close to the supervised anchor policy [30]: \[\begin{align} r^\mathrm{KL}(y_n;y_{:n-1}, x) = \log \frac{\pi^{\mathrm{SL}}( y_n \mid y_{:n-1},x) }{ \pi_{\theta}^{\mathrm{RL}}(y_n \mid y_{:n-1}, x)}\;. \end{align}\] This term is added to the NLI reward, producing the final token-level reward: \[\begin{gather} \label{eq:32reward} r(y_n;y_{:n-1}, x) = (1 - \alpha) r^{\mathop{\mathrm{NLI}}}(y_n;y_{:n-1},x)\\ + \alpha r^\mathrm{KL}(y_n;y_{:n-1},x)\;. \end{gather}\tag{1}\] The hyperparameter \(\alpha\) enables controlling the trade-off between enforcing faithfulness through the reward and remaining close to the anchor policy.

2.1.0.3 Training Algorithm.

We train the policy to optimize for the rewards defined in 1 using an on-policy actor-critic policy gradient (PG) approach. Since we keep proximity to the anchor model via the KL penalty reward, the algorithm can be considered a regularized PG algorithm, similarly to works by [31]–[35]; see 10 for a detailed formulation. Specifically, two models are learned: a policy (the generation model) and the expected value of the policy (the value network). We use the supervised model to initialize the parameters of both models, with the exception that the last layer of the value network outputs single scalars instead of a distribution over the vocabulary.

The RL training process consists of the following stages: (1) Generating summaries with the current policy and (2) Scoring the summaries using the reward signal. Then, (3) Policy and value networks are trained, jointly: the policy is trained via the PG loss while using the value for generalized advantage estimation (GAE, [36]); the value is trained via standard bootstrapping, using the GAE predictions. Notably, this process does not require reference summaries for learning the policy. More details regarding the algorithm and losses can be found in 8.

2.2 Decoding at Inference Time↩︎

As a direct consequence of RL training, the model explicitly learns to generate tokens with the goal of maximizing the long-term sequence reward. This is in contrast to MLE-based training, where the model learns to generate each token myopically, requiring heuristic decoding strategies such as beam-search to plan ahead. As a result, we can use the more efficient temperature sampling instead of beam-search when decoding from an RL-trained policy.¹

3 Experimental Design↩︎

3.1 Data↩︎

We focus on XSum  [27], an abstractive summarization dataset that poses challenges around factual consistency. XSum is compiled from  200K web-scraped BBC news articles, where the lead (introductory) sentence in every article is taken as the summary, and the rest of the sentences are taken as the source document.

Due to this formulation, XSum summaries may contain additional information that was not repeated in the rest of the sentences. Indeed, prior work found that only 20% of the reference summaries in XSum are entailed from their source document [4], and that summarization systems trained on XSum are likely to generate factually inconsistent summaries. For this reason we find XSum suitable for our experiments, as we would like to see if the RL-based reward could alleviate the factual inconsistencies that supervised models learn to generate based on this data.

We also experiment on two additional datasets to compare to prior work. The TL;DR dataset [37], using the same cleaned version provided by [24], which contains  120K Reddit posts and their short summaries, and the CNN/DM [38] dataset. The latter contains  200K news articles and their bullet-point highlights, which are mostly copied excerpts from article sentences. In this work we focus on abstractive summarization, and therefore evaluate our methods on CNN/DM with models trained, both supervised and reinforced, over TL;DR.

3.2 Entailment Model↩︎

In this work we focus on combining an existing entailment model as a reward in an RL framework. We employ the NLI classifier from [17] across our study as a reward as well as for evaluation and data labelling for baseline methods. It was trained over the ANLI dataset [15] with the T5-XXL architecture. The classifier produces the characters ‘1’ or ‘0’ as its output for binary entailment and non-entailment decisions, respectively. We pose the source document as the premise and the predicted summary as the hypothesis, and use the log-probability of the decoded character ‘1’ conditioned on the input as our reward.² We leave improvements to the underlying factual consistency models for future efforts. See 6 for more discussion about different factual consistency models.

3.3 Baseline Methods↩︎

3.3.0.1 SL.

Our supervised learning baseline is obtained by fine-tuning a T5 model on document-summary pairs. We use the T5X framework [39] for fine-tuning with batch size of 32 and keep the other hyperparameters to their default values (see 8 for details). Fine-tuning is stopped once the model converges in terms of ROUGE on the validation set. This supervised baseline will also be used as the initialization checkpoint of our RL methods. Decoding a summary using this model is implemented using beam search.

3.3.0.2 Filtered.

Similar to the SL approach, with the distinction that we filter out training data where the summaries are not entailed by the input document according to our NLI model. This filtering leaves  60% of the original XSum training set. We train the model similarly to the SL model, and evaluate on the full validation and test splits, without filtering.

3.3.0.3 CTRL.

Inspired by [40], [41], we train the model on the full training set to explicitly differentiate between generating faithful and unfaithful summaries: each training document is prepended with a phrase indicating if the target summary is entailed or not according to our NLI model. At inference, since we aim to produce consistent summaries, each document is always prepended with the phrase denoting an entailing summary, and continue decoding the summary using beam search. Other parameters are similar to the SL method.

3.3.0.4 FactPegasus.

[42] employ a tailored pre-training setup similar to PEGASUS [1] that also takes factual consistency into account, and combine it with data pre-processing, and contrastive learning to generate more faithful summaries.

3.3.0.5 CLIFF.

[43] propose a contrastive learning objective that distinguishes between reference and heuristically created noisy summaries.

3.3.0.6 RLHF.

[24] uses an RL approach with a reward model that learns from human comparisons of summaries. They iteratively add new feedback from humans for summaries generated by the current policy, and re-train the reward model. We use their publicly released samples of the TL;DR validation set and the CNN/DM test set.

3.4 Proposed Models↩︎

We train two flavors of RL-based models. The first, RLEF\(_{\text{L}}\), gives a lower weight to the regularization reward by setting \(\alpha{}=0.1\) and the sampling temperature to 1. The second model, RLEF\(_{\text{H}}\), gives a higher weight to the regularization reward with \(\alpha{}=0.2\) and a sampling temperature of 0.3. We altered both the \(\alpha{}\) values and the sampling temperatures since we saw that both parameters affect the trade-off between factual consistency, as measured by the NLI model, and lexical similarity, as measured by ROUGE (see 3). For additional implementation details see 8.

3.5 Automatic Evaluation Metrics↩︎

We report the common lexical n-gram overlap evaluation metrics and a set of factual consistency metrics, as the former were shown to be ill-suited for detecting unfaithful outputs [3], [5].

For factual consistency, we report NLI, which is the percent of entailed summaries according to our NLI classifier, and the \(Q^2\) score [44]. \(Q^2\) is similar to QAGS [45] and QuestEval [9] but was shown to work better on XSum data [17] with higher correlation with human judgements.

When optimizing for faithfulness, an RL policy may resort to less abstractive summaries that are copied verbatim from the source [19], or less informative ones with a reduced level of detail. To explicitly measure these attributes in a summary, we report extractiveness metrics: Coverage and Density [46], where the first measures the percent of summary tokens that also appear in the document, while the second measures a quantity similar to the average length of extractive spans in the summary. Finally, we report the average summary length³ (Length).

3.6 Manual Evaluation Protocol↩︎

We asked human evaluators to rate a sample of the XSum test-set from several selected methods. Each summary was evaluated by 3 different raters. Inspired by [18], we pose 4 questions outlining comprehensibility, attribution, salience and conciseness (see example in 5 in the appendix). To get conclusive results, similarly to [48] we request binary yes/no answers and ask to answer “No” for any slight deviation from the desired property. For unfaithful summaries, the evaluator also provides the offending phrase. Our evaluator pool consists of 11 workers that successfully completed a short training round of 10 examples (for details, see 9).

4 Results↩︎

4.0.0.1 Automatic Evaluation.

1 presents the automatic evaluation results on the XSum test set, comparing the supervised baselines to the two RL-based models (RLEF\(_{\text{L}}\), RLEF\(_{\text{H}}\)).

The table shows that the RL-based models achieve the highest entailment scores as measured by the NLI and \(Q^2\) metrics. Notably, the RL approach is the most effective approach to utilize the NLI signal, scoring favorable compared to supervised baselines Filtered and CTRL, which leverage the same signal.

Analyzing ROUGE reveals the trade-off between the entailment and other summarization traits. Without strong regularization, RLEF\(_{\text{L}}\) scores highest on entailment but lower on ROUGE, indicating that in order to reach higher factual consistency, the model pushed farther away from the supervised starting point. The more strongly regularized RLEF\(_{\text{H}}\) achieves a ROUGE score on par with the CTRL and SL baselines, suggesting that our KL-regularization prevented the policy from drifting.

Looking at extractiveness, the Density metric suggests that RL policies do not resort to copying text, and the increased Coverage implies that they tend to use more terms from the document, suggesting fewer hallucinations. Lower ROUGE scores may hint at lower quality summaries for the less regularized entailment model, yet the other metrics actually point at higher conciseness. We next present our human evaluation to shed light on these differences, and analyze whether the improvement in entailment is also captured by human readers, and whether the lexical divergence from the reference summary affects has implications on salience or conciseness.

4.0.0.2 Human Evaluation.

The results of our human evaluation are detailed in 3. Our raters fully agreed on 60% of the examples regarding attribution. From attribution (factual consistency) perspective, the results strengthen the evidence that the RL approach is superior to other methods by a large gap. Interestingly the XSum reference summaries scored lowest with 23.6%, showing that they are ill-suited to serve as faithful references for ROUGE and similar reference-based metrics. Notably, the human attribution evaluation was much stricter than the NLI metric, with much lower scores for all models, and we analyze this discrepancy in 5.

Surprisingly, the RLEF models outperforms all other models also on Salience and Conciseness. Specifically, the less regularized RLEF\(_{\text{L}}\) learned to generate not only the most factually consistent summaries but also to improve on Salience and Conciseness, indicating that they are correlated w.r.t human quality perception.

4.0.0.3 Comparison with RLHF.

We applied our RL approach on the TL;DR dataset. We used the same input format and data split as in [24] for both the supervised and RL training processes. For the supervised model (SL) we used hyper-parameters identical to our previous experiments (see 8) except for a batch size of 128 and learning rate of 2e-4.

We compared our results using automated metrics with the RLHF approach [24]. This approach is also based on the T5 model and uses a similar RL setup, yet it employs a reward model trained on task-specific human preferences and applying a KL-based anchor. The results, detailed in 2, show that RLEF achieves higher entailment scores in both NLI and \(Q^2\) metrics, while our supervised model is on par with RLHF. We also note that RLHF produces noticeably different and longer summaries compared to our supervised baseline, while RLEF maintains similar length and ROUGE to the supervised baseline.

We also compared the two approaches in a transfer learning setting, where we predicted a summary on a different dataset (CNN/DM) using models trained on TL;DR. The results show similar trends, with higher entailment score for RLEF. These results hint at the benefit of utilizing a general NLI reward function, which managed to outperform the domain-specific RLHF reward both on the source domain and on a transfer setting.

5 Analysis↩︎

5.0.0.1 Regularization and Sampling Temperature.

3 describes an ablation experiment where we vary the regularization \(\alpha\) and the decoding temperature and measure the effect on different automatic metrics. Higher sampling temperature correlates with higher entailment and lower ROUGE scores. We conjecture that this is since higher temperature generates more diverse summaries, which amplifies exploration away from the original gold references. A similar phenomenon is observed when considering token length, as lower temperature policies produce summaries closer in length to the data-mean than their higher temperature counterparts.

As for the regularization coefficient \(\alpha\), we observe the expected trade-off: lower regularization (smaller \(\alpha\)) leads to higher entailment (NLI), lower similarity to the supervised summary (ROUGE), and higher Coverage. These may be explained by removal of external hallucinations that often use vocabulary terms that are unrelated to the document.

Surprisingly, in each KL setting, the lower temperature policy favors more document-aligned terms (perhaps for their higher initial probability), yet this is not reflected in the NLI metric, that stays lower than its higher-temperature counterpart. We also observe that the summaries get shorter with less regularization, as the policy learns to mention fewer details as a way to alleviate generating inconsistencies.

5.0.0.2 Model Size.

We tested our approach with different model sizes to study the effect of scale in the RLEF\(_{\text{H}}\) setup. We compared T5-Base (220M parameters), T5-Large (770M) and T5-XXL (11B), using the same hyper-parameters for all three models. 4 shows the entailment rate on the XSum validation set during RL-finetuning. For all model sizes, our approach improved the entailment ratio over the supervised model by a large margin.

However, while the Large and XXL models changes the average summary length only slightly, the Base model completely degenerates, “hacking” the NLI reward by generating summaries that are half as short as the reference. This suggests that higher-capacity models are essential to prevent reward hacking, perhaps due to two possible reasons. First, the larger policies have higher generalization capabilities overall and can better accommodate different rewards, such as entailment and summarization regularization in our case. Second, since the anchor model uses the same architecture, the higher capacity anchor model is more robust to changes in the summary and produces lower scores for less informative or more extractive summaries.

5.1 Manual Analysis.↩︎

To gain more insight into the inner workings of RLEF, we propose two manual inspections about the types of changes being induced by the policy, and analysis of attribution errors found by our human evaluation procedure.

5.1.0.1 Changes to the summary during RL training.

We study the changes that the RLEF\(_{\text{H}}\) policy induces on a summary during RL training, focusing on the changes that cause a flip in entailment decision. We sample 200 documents from the validation set for which we obtain the predicted summary at different checkpoints throughout the RL training process in 4K steps intervals. We apply the NLI classifier for each document and summary list, and select 60 examples for which the NLI decision has flipped between any pair of consecutive checkpoints, and study what changes have been made to the summary that caused the flip. Notably, most flips occur only once during training, and from the non-entailed to the entailed decision. Examples are shown in 4 together with our categorization of the changes, with some summaries morphing in more than one way. We notice that for summaries produced by RLEF\(_{\text{H}}\) most changes are local, meaning that the main predicate clause and the core participants remain the same throughout most checkpoints. We classified 13 out of 60 examples as abstractively rephrased, where a specific detail is replaced with a broader description, e.g. returned to earth instead of landed in Florida (ex. 1). However, we also found that 27 examples contained argument omissions, where verbal arguments or noun modifiers with typically non-core semantic roles [49] are removed (e.g. Locative or Temporal descriptions). See for example the “Cause for arrest” omission in ex. 5. Such omissions keep the information regarding the main participants intact, while lowering the risk of errors around non-core details. Other changes included claim changes (16 cases) where a predicate has been replaced (see ex 3), argument replacements (8 cases), and other non-specific alterations.

5.1.0.2 Attribution error analysis.

We analyzed attribution errors from the human evaluation of our best policy, RLEF\(_{\text{L}}\), aggregated by majority vote. We inspect the offending phrase supplied by the evaluator for 39 out of 100 examples that are found to be non-attributable. 28 are considered as a local hallucination, mostly confirming to addition of personal names, numbers, places, and roles that did not appear in the article. For example, an article mentioned Kevin O’Malley without alluding to his job title, while the summary referred to him as the Irish Ambassador. While Kevin O’Malley was indeed an Irish ambassador, the model should not add such details if they are not explicitly mentioned in the article. Since most of these examples were found as entailing by our reward, this may point at issues with the NLI model that are due to knowledge conflicts between its parametric and contextual knowledge [50]. The rest of the examples include 5 contradictions and 5 major hallucinations.

6 Related Work↩︎

RL for text generation. RL has been applied to many text generation tasks like neural machine translation [51], [52], extractive summarization [53]–[56], abstractive summarization [57] and others [58]–[62].

Specifically for summarization, prior RL approaches used different reference-based metrics as a reward function. In [25], two reward signals are measured between the generated and reference summaries: lexical overlap (ROUGE) to gauge salience and an entailment score to measure factual consistency. [26] employed a similar approach with question-answering, they produced QA pairs conditioned on the generated summary to detect inconsistencies with the reference, and another set of QAs conditioned on the reference to measure salience. Additionally, [63] proposed QUALS, a more computationally efficient QA approach, that was used in a contrastive learning setting. While their approach could be used without comparing outputs to reference summaries, they observed that adding such comparisons with the reference is essential for the stability of their method. We note that for some datasets, reference summaries are likely to contain factual errors [4], decreasing the effectiveness of reference-based rewards.

Other RL methods, instead of explicitly defining the quality of a summary suggest to model it directly from human feedback [23], [24], [64], [65]. This technique can prevent errors due to references that are misaligned with human judgment. While it is a promising approach, it also requires acquiring task-specific annotation, which can be labor-intensive.

Another hybrid approach interleaves a cross-entropy objective with policy gradients [66] in multi-document summarization (MDS). They use an in-domain NLI model, for which they annotate their MDS dataset with entailment decisions. To stabilize their policy they employ an additional GAN-like training regime and add a discriminator loss between generated and reference summaries to their reward.

Trade-offs in consistency models. The choice of which factual consistency approach to use has interesting consequences for the RL setup. Our work employs a binary NLI decision that does not point towards the specific inconsistent parts in the output summary. Consequently, the reward is assigned to the final token of the summary, leaving proper credit assignment to the RL algorithm. Other methods, specifically those based on question-answering [44], [45], [67] can frame misaligned answers in the generated summary and assign the reward explicitly to the offending tokens. However, these QA-QG based methods may be much slower to compute. Our reward requires a single forward pass using a transformer model over the document-summary pair, in comparison, QA-QG approaches require generating answer candidates, questions, answers from both sources and computing answer alignment. Some of this complexity is remedied by generating jointly questions-and-answers [63], but it still requires a lengthy decoding of QA pairs. A different NLI-based approach decomposes the document and summary into smaller blocks of sentences [68] and aggregates the final decision over a matrix of block-level NLI scores. Such approach could aid the RL algorithm with credit assignment when generating long summaries. In practice, the abstractive summarization datasets in this study use short single sentence summaries.

7 Conclusions and Future Work↩︎

We propose to leverage NLI models as a ready-made, reference-free reward signal for RL training of factually consistent abstractive summarization models. Our experiments and thorough analysis with automatic and human evaluation show promising results for this approach, with our RL approach outperforming all baselines on factual consistency, while maintaining and even improving on other desired summarization attributes as well.

In future work, we would like to extend this approach to other grounded generation tasks, like knowledge-driven dialog. In addition, we find it interesting to explore additional reference-free reward models for other summarization attributes (or for other tasks). Then, an important research direction would be to understand how to properly adapt our method to work with multiple such rewards.

Limitations↩︎

While our approach shows promising results in both automatic and human evaluation, it relies on two significant pillars: a strong entailment model and a strong initial summarization model. The NLI model implicitly encodes the biases and other data regularities that were part of the NLI training set into the generated summaries of our policy. This is well demonstrated by the gap between human attribution judgements and the automatic NLI metric. Our RL policies cannot improve on factual consistency errors if they are undetectable by the NLI reward. Hopefully, as NLI capabilities get better, so will the efficacy of RLEF and the abilities to automatically flag hallucinations and contradictions.

Secondly, a strong summarization model is essential for our method in two ways: as an initialized starting point for RL exploration and as an anchor point to a policy. While our RL training does not require any reference data and opens the possibility to use more un-summarized documents, it would probably not succeed as well without initializing from a high-quality supervised model.

Another limitation is that our experiments suggest that model size is important when using RLEF (4): both our summarization and NLI models are 11B parameters models. We believe it is important to further understand how to make our approach more robust to smaller models, to increase its computational efficiency and availability.

Ethics Statement↩︎

Our work aims at solving the ethical issue of addressing misinformation in automated text generation tasks. Yet, adopting automatic summarization by real users can amplify misinformation in cases where the model still makes an error or when the input text itself is not trustworthy. As we stated in the limitations, our trained models heavily rely on other predictive models and therefore carry the biases of their training data, and may implicitly encode these into our generative process. Therefore, we believe that to reach real-world use, not just our method should be scrutinized but also the NLI and summarization datasets that were used to train these models. Thus, such methods should be used with caution and combined with other techniques to ensure humans are capable of judging the validity of the information generated by the model.

Acknowledgements↩︎

We would like to thank our anonymous reviewers for their thorough comments and insightful suggestions.

8 Experimental Details↩︎

8.0.0.1 RL Algorithm Details.

We use an actor-critic on-policy PG algorithm with a learned value function \(V_\psi\) and a parameterized policy \(\pi_{\theta}\) to maximize the RL objective. The policy gradient w.r.t. to the regularized reward \(r(y_t ; y_{:t-1},x)\) defined in 1 is \[\begin{align} &\nabla_\theta J(\theta)\\ &= \mathbb{E}_{x, y \sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(y_t | y_{:t-1},x) G^{\alpha}_t \Big], \end{align}\] where for brevity we denote \(G^{\alpha}_t = \sum_{t'=t}^{T} r(y_t ; y_{:t-1},x)\), the accumulated regularized return. For more details on the derivation of this expression, and framing the regularized objective as an RL problem, we refer the reader to 10.

We use the value \(V_\psi\) as a baseline, a state-dependent function that can be subtracted in the policy gradient without changing it. This leads to the following equivalent policy gradient \[\begin{align} &\nabla_\theta J(\theta) = \mathbb{E}_{x, y\sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(y_t | y_{:t-1},x) \times \\ & \quad \quad \quad \quad \quad\quad\quad\quad\quad\quad\quad\big(G^{\alpha}_t - V_\psi(y_{:t-1},x) \big) \Big]\\ & = \mathbb{E}_{x, y\sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(y_t | y_{:t-1},x) A^{\mathrm{GAE}}_\psi(y_{:t}) \Big] \end{align}\] where \(A_\psi\) is termed the advantage function. Applying this PG can be regarded a variant of the REINFORCE [69] algorithm with a baseline. In practice, we replace the advantage in the expression above by generalized advantage estimation [36], which allows to better control the bias-variance trade-off via the \(\lambda\) parameter: \[\begin{align} &A^{\mathrm{GAE}}_{\psi}(y_{t};y_{:t-1}, x) = \sum_{t'=t}^{+\infty} (\gamma \lambda)^{t' - t} \times \\ & \Big( r(y_{t'};y_{:t'-1}, x) + \gamma V_\psi(y_{:t'}, x) - V_\psi(y_{:t'-1}, x) \Big). \end{align}\] Finally, the above policy gradient definition leads to the following per-example loss for learning the policy \(\pi_{\theta}\), \[\begin{align} \mathcal{L}^{\pi}(\theta)(y_{:t},x) = A^{\mathrm{GAE}}_\psi(y_{:t},x)\log \pi_\theta(y_t | y_{:t-1},x), \end{align}\] where the gradients are only propagated here w.r.t. the policy parameters.

The value \(V_\psi\) itself is learned via regression towards the return estimate induced by GAE, which is equivalent to minimizing the GAE advantage: \[\mathcal{L}^V(\psi)(y_{:t},x) = \big(A^{\mathrm{GAE}}_{\psi}(y_{:t},x) \big)^2.\] We now describe more intricate implementation details and hyper parameter choices.

8.0.0.2 RL Implementation Details.

Given that we operate in the finite horizon setting, we naturally set the discount factor \(\gamma\) to 1. Similarly to the PPO algorithm [70], we normalize the advantages in a given batch of data so that they approximately follow a standard normal distribution. We also normalize the value loss by dividing it by the variance of the batch returns. An important difference between our implementation and the standard (regularized) PG implementation is that instead of treating KL penalties along a given sequence as immediate rewards, we accumulate those and treat the resulting quantity as a sequence-level penalty. We found this to lead to more stability in the RL procedure.

Unlike the conventional RL setting where both the policy and value are randomly initialized, in our case the policy is already fine-tuned to solve the required task. Thus, to make the value function accurate w.r.t.the already initialized policy, we observed that we needed a small number of iterations before the value estimation is sufficiently accurate to avoid detrimental policy gradients. To do so, we run RL fine-tuning for 20K steps, with a warmup of 5K steps for the value network. We also noticed that it was beneficial to use distinct values for the policy and value learning rates, so we decouple them in practice.

8.0.0.3 Optimization.

We use Adafactor [71] with a learning rate warmup phase: the learning rate is linearly annealed from zero to the specified asymptotic value.

8.0.0.4 Hyperparameter Search.

We noticed that the optimal value of the policy and value learning rates are highly correlated. Hence, we propose a decoupled hyperparameter search: we start by finding a suitable value learning rate by keeping the policy fixed. We then follow a standard grid search to find suitable values for the remaining hyperparameters including the policy learning rate, temperature and the regularization coefficient \(\alpha\). Specifically, in our hyperparameter sweep we used temperatures \([0.1,0.3,1.0]\) and \(\alpha\) values between \(0.1\) and \(0.8\) with a grid size of \(0.1\). Thus overall, our main sweep for the XXL model consisted of \(24\) runs of \(20K\) iterations.

We list all the hyperparameters used (unless different values are mentioned in the text) in Table 5. For the learning rate warmup and policy update delay, note that the number of steps reported correspond to gradient steps of the RL fine-tuning procedure.

8.0.0.5 SL Implementation details.

For the SL models, we decode summaries with beam search with a beam width of 4 and a brevity penalty of 0.6. For training we use the same optimizer with base learning rate of 0.001, batch size of 32, and a dropout rate of 0.1.

8.0.0.6 Resources.

We used TPU-v4 chips to train all the models mentioned. Each of our T5-XXL based RLEF experiment ran for approximately 17 hours on 64 TPU chips. Furthermore, our main hyper parameters sweep included 24 such experiments, accounting for 1088 TPU-days.

9 Evaluator Demographics, UI and Instructions↩︎

We employed full-time hourly workers to rate the summary quality. Our raters consist of native English speakers, nationals from the U.S. and U.K. that hold graduate (70%) and high-school (30%) diplomas. We supplied them with 2 pages of instructions and additional examples, and conducted an initial pilot study and training batch before proceeding to rate the summaries. The UI that we used is displayed in 5. In what follows we attach the guidelines presented to the raters in the human evaluation described in 3.6. The guidelines are loosely based on [48].

9.1 Guidelines↩︎

In this task you will be presented with a news article and multiple summaries of the article, and you are asked to evaluate the summary quality. You will rate each summary with 4 yes/no questions. These questions ask if the summary is: Comprehensible and understandable. Attributable (supported) by the article - no contradicting or unattested information. Captures the main idea(s) behind the article. Concise - does not contain additional details beyond the key information in the article. Read carefully the text and the summary. The summaries may appear very fluent and well-formed, but contain slight inaccuracies that are not easy to discern at first glance.

9.1.0.1 Q1: Comprehensibility.

An incomprehensible summary is not understandable due to significantly malformed phrases and sentences that are difficult to comprehend or make sense of. If there is any part of the summary that is unclear or hard to understand or malformed (e.g., partially cut-off or contains strange characters), select "No, not fully comprehensible". Summary When you leave it late, you leave it late is adding interest to your pension money as a result of the financial crisis. o, not fully comprehensible

9.1.0.2 Q2: Attributable (Supported) by the article.

A fully supported summary contains information that can be found in the source article. No information in the summary is unattested when compared against the source news article. In other words, if you can say that “According to the news article…” with the summary following this phrase, you should answer, “Yes, it is attributable.” If some key details in the summary are not supported by the article (e.g. missing from the article), inaccurately represent the information in the article, or contradicted by the article, then please mark “No, not fully attributable.”

9.1.0.3 Q3: Main Idea.

A main idea captures a fact or theme that is central to the article’s discussion. It should involve the people, locations, or events that the article focuses on. If a main idea was removed from the original article, it would change the meaning, focus, or argument of the article. Note that this question is NOT asking whether the summary includes ONLY main ideas.

In Q3, to the best of your ability try to distinguish between the following cases, some may be more rare than others:

• The summary is fully supported (yes to Q2) and captures the main idea (yes to Q3).

• The summary is fully supported (yes to Q2), but ignores the central point of the document (No in Q3).

• The summary contradicts the document in minor details or hallucinates some information (No to Q2), but the idea behind the document is mostly captured even if some details are incorrect (Yes to Q3).

• The summary contradicts the document in key details (No to Q2) to the level where the main idea is unrecoverable or largely missed (No to Q3).

9.1.0.4 Q4: Conciseness.

A summary is concise if it includes only the necessary details and the important information in the article. If it includes any details which are not central to the article, it should be marked as "No, it is not concise". A summary may be concise even if some details are contradicting (i.e. you marked “No, it is not fully attributable” in Q2) as long as those were part of the main idea of the article.

In Q4 we are trying to find if the summary contains substantial information that does not belong to the main idea. If some minor details in the summary are contradicting, yet they are part of the main idea, then this summary is still concise, the system made an error of attribution, but not of over-generation.

10 Fine-Tuning Language Models with Reinforcement Learning↩︎

10.1 Language Generation as a Contextual Markov Decision Problem↩︎

In this appendix, we explain the connection between arbitrary language generation tasks and the Markov Decision Process (MDP) framework [72] which is widely used in RL. We recall that an MDP \(M\) is a tuple \(M = (\mathcal{S}, \mathcal{A}, \gamma, r, P)\), where \(\mathcal{S}\) is a state space, \(\mathcal{A}\) is an action space, \(\gamma \in [0, 1]\) is a discount factor, \(r: \mathcal{S} \times \mathcal{A} \rightarrow [-r_{max}, r_{max}]\) is a bounded reward function and \(P: \mathcal{S} \times \mathcal{A} \rightarrow \Delta_{\mathcal{S}}\) is a transition kernel. \(\Delta_\chi\) denotes the standard simplex over \(\chi\). We represent sequential decision-making strategies as policies \(\pi: \mathcal{S} \rightarrow \Delta_{\mathcal{A}}\). At any point in time \(t\), a policy \(\pi\) interacts in an MDP by observing the current state \(s_t\), selecting an action \(a_t \sim \pi(\cdot | s_t)\), and accordingly receiving a reward \(r_t = r(s_t, a_t)\), before observing a new state \(s_{t + 1} \sim P(\cdot | s_t, a_t)\). We define the return as the discounted sum of rewards in one episode of interaction: \(G_t = \sum_{t' = t}^{T} \gamma^{t'} r_{t'}\), where \(T\) is called the horizon and is potentially infinite. We now introduce Contextual MDPs (CMDPs) [28]. They model the fact that a fixed context is available and determines the nature of rewards and dynamics. Formally, a CMDP is a tuple \(M_c = (\mathcal{C}, f_M)\), where \(\mathcal{C}\) is a context space and \(f_M: c \in \mathcal{C} \rightarrow M\) is a function that maps a context to the corresponding MDP.

Any language generation task can be seen as the following interactive process: a language model observes the current state \(s_t = y_{:t - 1}\) and context \(c = x\), that is both the input text \(x\) and the text generated so far \(y_{:t - 1}\), and selects a token \(a_t = y_t\). Thus, we can view any language generation task as a CMDP \(M_c = (\mathcal{C}, f_M)\) with \(f_M(c) = (\mathcal{S}, \mathcal{A}, \gamma, r(\cdot \, ; c), P(\cdot \, ; c))\), with the policy \(\pi\) being the language model itself. The state space \(\mathcal{S}\) is the set of all potential generations (either complete or incomplete). We suppose that the maximum length of generated text \(T\), which is equivalent to the horizon, and that of the input text \(T_c\) are finite, which is a common assumption in NLP. Accordingly, if we note \(\mathbb{V}\) the vocabulary (the set of all admissible tokens), we have \(\mathcal{S} = \cup_{i=0}^{T} \mathbb{V}^i\). Similarly, we have the context space \(\mathcal{C} = \cup_{j=0}^{T_c} \mathbb{V}^j\). The action space \(\mathcal{A}\) is the set of tokens that the policy can output at any point in time, that is the vocabulary, hence \(\mathcal{A} = \mathbb{V}\). The discount factor \(\gamma\) is arbitrary and can be set to 1 given that the horizon is supposedly finite. The reward function \(r\) is also arbitrary, but in the case of interest exposed in the main text we set it to: \[r(s_t, a_t; c) = \begin{cases} \mathop{\mathrm{NLI}}(y_{:t}; x) \text{ if } y_t = {\scriptstyle \mathtt{[EOS]}}\text{ or } t = T, \\ 0 \text{ otherwise}. \end{cases}\] Finally, and most importantly, the transition kernel is deterministic: \[\begin{align} &P(s_{t + 1} | s_t, a_t; c) = \begin{cases} 1 \text{ if } {\scriptstyle \mathtt{[EOS]}}\in s_t \text{ and } s_{t + 1} = s_t, \\ 1 \text{ if } {\scriptstyle \mathtt{[EOS]}}\notin s_t \text{ and } s_{t + 1} = y_{:t}, \\ 0 \text{ otherwise}. \end{cases} \end{align}\] Indeed, any state that contains an \({\scriptstyle \mathtt{[EOS]}}\) token can be considered an absorbing state.

10.2 Language Generation From a Pre-Trained Model as a Regularized Markov Decision Problem↩︎

While the previous formalism applies to all language generation tasks, we now describe a formalism that specifically applies to the language generation task that is explored in the main text: language generation when a pre-trained model is available. It models the fact that we want generated text to be likely according to the pre-trained model, which we call anchor model in what is next. We note the corresponding policy \(\pi^{\mathrm{anchor}}\). We consider the following reward function: \[\begin{align} r(s_t, a_t) &= (1 - \alpha) r(s_t, a_t) + \alpha r^\mathrm{KL}(s_t, a_t), \end{align}\] with the regularization term: \[\begin{align} r^\mathrm{KL}(s_t, a_t) = \log \pi^{\mathrm{anchor}}(a_t | s_t) - \log \pi(a_t | s_t) , \end{align}\] where \(r\) is the reward function defined previously and \(\alpha\) is a scalar controlling the regularization strength. We recall that the Kullback-Leibler (KL) divergence between the current policy and the anchor policy has the expression: \[\begin{align} \mathop{\mathrm{KL}}(&\pi || \pi^{\mathrm{anchor}})(s_t) = \\ &-\mathbb{E}_{a_t \sim \pi}\Big[\log \pi^{\mathrm{anchor}}(a_t | s_t) - \log \pi(a_t | s_t)\Big]. \end{align}\] Hence, the regularization term is an unbiased estimator for the KL divergence between current and anchor policies. Intuitively, it encourages the learned policy to keep a distribution that is close to the distribution over tokens induced by the anchor policy (the fine-tuned model). Since the learned policy evolves along training, the reward function we described is non-stationary, that is the reward for a given state-action pair \((s, a)\) changes with the policy \(\pi\). Hence, the modified MDP is best viewed as a regularized MDP [31]. We define the KL regularizer as \(\Omega(\pi) = \mathop{\mathrm{KL}}(\pi|| \pi^{\mathrm{anchor}})\), which is a strongly convex function. We can show that this formalism is equivalent to the MDP with the non-stationary reward function described above.

10.3 Defining the Reinforcement Learning Objective↩︎

In this section, we show that the regularized reward defined in 1 can be used together with any PG based algorithm. To do that, we show that for any MDP (see 10.1), the policy gradient can be easily re-derived for our regularization scheme when using parameterized policies. This repeats the derivations in [31], [73]. We denote trajectories \(\tau = \{s_0\} \cup \{a_t, s_{t + 1}\}_{t=0}^{T-1}\). By a slight abuse of notations we denote the probability of a given trajectory under the policy \(\pi\) as \(\pi(\tau)\), that we can decompose as \(\pi(\tau) = P(s_0) \prod_{i=0}^{T-1} \pi(a_i | s_i) P(s_{i+1} | s_i,a_i)\). We also denote \(G_t\) as the return of a trajectory starting from time-step \(t\). Now, denote a parameterized policy \(\pi_{\theta}\), and define the standard RL objective, \[\begin{align} J(\theta) &= \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^T r(s_t, a_t)\Big], \\ &= \mathbb{E}_{\tau \sim \pi_\theta}[G_0]. \end{align}\] The goal of RL is to find a parameterization \(\theta^*\) that maximizes the following objective: \[\theta^* \in \mathop{\mathrm{arg\,max}}_\theta J(\theta).\] The policy gradient theorem states that \[\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t | s_t) G_t \Big].\] We now place ourselves in the specific regularized MDP defined in 1 and 10.2, with the reward regularization scheme, \(r^{\alpha}(s,a) = (1 - \alpha) r(s, a) + \alpha \log{ \frac{\pi^{\mathrm{anchor}}(a | s)}{\pi_\theta(a | s)}}\). Define the RL objective of interest, which adds a regularization term to the reward function, \[\begin{align} J(\theta) =\mathbb{E}_{\tau \sim \pi_\theta} \Big[ \sum_{t=0}^T &(1 - \alpha) r(s_t, a_t) \\ &+ \alpha \log \frac{ \pi^{\mathrm{anchor}}(a_t | s_t)}{\pi_\theta(a_t | s_t)}\Big]. \end{align}\]

For \(r(s_t, a_t)\), we repeat standard steps to re-derive the corresponding policy gradient. However, we need to have a separate treatment for the KL regularization reward \(\log\frac{\pi^{\mathrm{anchor}}(a_t | s_t)}{\pi_{\theta}(a_t | s_t)}\), as it explicitly depends on \(\theta\). We have: \[\begin{align} &\nabla_\theta \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^T \log \frac{\pi^{\mathrm{anchor}}(a_t | s_t)}{\pi_{\theta}(a_t | s_t)}\Big] \\ & = - \nabla_\theta \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)}\Big] \\ &=- \nabla_\theta \sum_\tau \pi_\theta(\tau) \sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)}, \\ &=- \sum_\tau \nabla_\theta \Big( \pi_\theta(\tau) \sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)} \Big), \\ &= -\underbrace{\sum_\tau \nabla_\theta \pi_\theta(\tau) \sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)}}_{A} \\ &- \underbrace{\sum_\tau \pi_\theta(\tau) \nabla_\theta \sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)}}_{B}. \end{align}\] We keep \(A\) as is and show that \(B\) is equal to 0: \[\begin{align} B &= \sum_\tau \pi_\theta(\tau) \nabla_\theta \sum_{t=0}^T \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)} \\ &= \sum_\tau \pi_\theta(\tau) \sum_{t=0}^T \nabla_\theta \log \frac{\pi_{\theta}(a_t | s_t)}{\pi^{\mathrm{anchor}}(a_t | s_t)}, \\ &= \sum_\tau \pi_\theta(\tau) \nabla_\theta \sum_{t=0}^T \log \pi_{\theta}(a_t | s_t), \\ &= \sum_\tau \pi_\theta(\tau) \nabla_\theta \log \pi_\theta(\tau), \\ &= \sum_\tau \nabla_\theta \pi_\theta(\tau), \\ &= \nabla_\theta \sum_\tau \pi_\theta(\tau), \\ &= 0. \end{align}\] By putting all the pieces together we get the expression of the policy gradient for the modified RL objective: \[\begin{align} &\nabla_{\theta} J(\theta) \\ \enspace &= \mathbb{E}_{\tau \sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t | s_t) \sum_{t'=t}^T r^{\alpha}(s_t,a_t) \Big], \end{align}\] Denoting \(G^{\alpha}_t\), the return of the trajectory when using \(r^{\alpha}\), this can be rewritten as, \[\begin{align} \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \Big[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t | s_t) G^{\alpha}_t \Big]. \end{align}\] Note that we recovered the standard policy gradient for the regularized reward \(r^\alpha\) (and corresponding return \(G^{\alpha}\)). This means that by treating \(r^{\alpha}\) as the reward we can use any policy gradient method, to solve the new objective. Because this holds for any MDP, it holds for the specific MDP defined in 10.1 for the summarization task. To see how this is concretely used in our approach to construct the PG losses, we refer the reader to 8.

References↩︎