1 Introduction↩︎

As textual content available on- and offline explodes, automated text summarization is becoming increasingly crucial [1]; with the advances in neural text generation methods, abstractive summarization systems that generate paraphrases are quickly replacing extractive ones that simply select essential sentences from the source text [2]. While abstractive summaries can be more coherent and informative (given the same length) than their extractive counterparts, they frequently contain information inconsistent with the source text. This is a critical issue, as it directly affects the reliability of the generated summaries. [3]–[5].

Unfortunately, existing approaches to identify such factual inconsistency without constructing new resources have not been satisfactory. Directly measured similarity between the summary and its source text—using popular n-gram similarity metrics such as ROUGE [6] and BLEU [7]—exhibits low correlation with human judgments for factual consistency. Also, leveraging related tasks—such as natural language inference (NLI) [8] and fact verification [9]—is not ideal. This is because these tasks aim to identify relations between two sentences, whereas factual consistency checking involves a multi-sentence summary and an even longer source text [10], [11].

A remaining solution is to train a factual consistency classifier with a dataset specifically constructed for this purpose. Note that positive summaries are readily available. That is, the reference summaries from existing text summarization datasets can be assumed to be factually consistent with the respective source texts. Thus, the main challenge is in generating effective negative summaries, i.e., summaries that are factually inconsistent with the source text. Recent works generate negative summaries by simply replacing keywords in the reference summaries or sentences extracted from the source texts [12], [13]. This, however, results in negative summaries that significantly diverge from the source texts and positive summaries, which is not ideal for training factual consistency classifiers. For instance, Figure 1 shows that coach in the reference summary is changed to President of Russia, which is an inconsistency that is too obvious.

In this paper, we propose a novel method, Masked-and-Fill with Masked Article (MFMA), where parts of the source text and reference summary are masked and later inferred to generate a plausible but factually inconsistent summary. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained on negative summaries generated with our method mostly outperform existing models and show a competitive correlation with human judgment. We also analyze the characteristics of the negative summaries generated. Our main contributions are as follows:

• We propose a novel negative summary generation method for training factual consistency classifiers for abstractive summaries.

• We show the efficacy of our method on seven benchmark datasets using classification performance and correlation with human judgment.

• We analyze the characteristics, such as affinity and diversity, of the negative summaries generated using our method.

2 Related Work↩︎

2.1 Factual Inconsistency in Summarization Systems↩︎

Previous works [3]–[5] have studied the factual inconsistency in abstractive summarization systems. Especially, [3] demonstrates that 30% of the model generated summaries have at least one factual error, and this obstacle the practical usage. [5] specifies these factual errors in the abstractive summarization system into two types: intrinsic errors and extrinsic errors. Intrinsic errors occur using the contents present in the source article like "Switzerland" and "England" in the negative summary example in Figure 2. On the other hand, extrinsic errors are the errors generated by ignoring the source article when generating summaries. "in the second half" in Figure 2, which is not included in the source article, is an example of extrinsic errors.

In this work, we propose a system for detecting these various factual errors that are necessary for developing a summarization system. We propose a unified method for intentionally modeling both types of errors to build a dataset for training this system.

2.2 Measuring Factual Consistency↩︎

As a better way to evaluate the factual consistency, recent works such as QAGS [14] and QuestEval [15] adopt question generation and question answering frameworks to evaluate the factual consistency. Both methods firstly generate questions using entities or noun phrases in the candidate summary and then compare the answers of these questions between the source and the summary. Although these methods do not require any reference summaries, they have a higher correlation with human judgments than previous metrics in consistency checking. Also, the generated questions and their answers are easily interpretable. But due to their complicated structure, the computational complexity of these methods is relatively heavy and the errors in each component can be cascaded.

Following the idea that all of the contents in the summaries should be entailed by source document, models from the related tasks such as Natural Language Inference(NLI) [8], [11], [16] are also used to verify the factual consistency of the summaries. These approaches are simpler and more intuitive than QA-based metrics. But the data pairs in these datasets are usually composed of single sentences, and this makes it difficult to be directly used for factual consistency checking in summarization where the task requires multi-sentence level reasoning. For this reason, two recent studies FactCC [12] and DocNLI [13] have studied ways to make synthetic datasets for training factual consistency checking model. Both works create synthetic negative summaries using the pre-defined rules such as entity substitution or mask-and-fill. In this paper, we propose a more general negative summary generation method additionally using the masked source.
CoCo [17] compares the likelihood of the generated summaries using the original source and the masked source to estimate the counterfactual samples. Different from CoCo, our work directly augments the negative summaries and train the classifier using them.

3 Methods↩︎

For a given article \(A\) and a summary \(S\), we aim to develop a factual consistency checking system that can evaluate whether \(S\) is factual consistent with \(A\). In other words, the system is required to discriminate a factual consistent summary \(S_C\) with the factual inconsistent summary \(S_I\) that consists of at least one factual error. We consider this problem as a classification task between \(S_C\) and \(S_I\). However, large-scale human-annotated training datasets for this task have not been constructed yet, especially for the inconsistent summaries \(S_I\).

In this paper, we focus on effective augmentation methods of the inconsistent summaries. In order for that, there are two crucial conditions: 1) guarantee of inconsistency; the generated summaries should be indeed inconsistent with the source article, 2) relevance to the source article; the generated summaries should include contents related to the article. These two factors are in trade-off relations, which means that when the generated summaries are strongly inconsistent they might not be related to the article and vice versa. Therefore appropriate negative summary augmentation is required to improve the factual consistency classifier.

To generate confusing and hard negative summaries, we propose a summary generation using a masked article and a masked reference summary where some salient information is hidden. By doing so, we let the summarizer model infer hidden information through the masked article to generate plausible negative summaries. Note that, previous works such as FactCC and DocNLI generate negative summaries \(S_I\) by changing positive summaries \(S_C\) through entity replacements or mask-and-fill methods without referring to the source article. We observe that previous methods can easily guarantee negativeness, but they often generate summaries that are very irrelevant to the source article or unnatural as shown in Figure 1.

3.1 Mask-and-Fill with Masked Article↩︎

To model inconsistent summaries but related to the article, we propose a method, Mask-and-Fill with Masked Article (MFMA), which generates negative summaries with masked articles and masked reference summaries, as shown in Figure  2.

Specifically, we assumed noun phrases and entities in the articles are salient information, and mask them with the ratio of \(\gamma_A\), resulting in masked article \(\overline{A}_{\gamma_A}\). Similarly, we also mask the salient spans in the positive summary, i.e., reference summary, with the ratio of \(\gamma_S\) to form a masked summary \(\overline{S}_{\gamma_S}\). Then, we concatenate \(\overline{A}_{\gamma_A}\) and \(\overline{S}_{\gamma_S}\) by prepending prefix token for each input text (i.e., “Summary: \(\overline{S}_{\gamma_S}\), Article: \(\overline{A}_{\gamma_A}\)") as shown in Figure 2. Next, we train a summarizer based on an encoder-decoder model, BART [18], to reconstruct the original summary \(S\) with the following loss:

\[\begin{align} \mathcal{L} &= \sum_t -\log P({S_t | S_{<t}, [\overline{S}_{\gamma_S}; \overline{A}_{\gamma_A}]}). \label{eq951} \end{align}\tag{1}\]

After training, we generate negative summaries of unseen and masked article-summary pairs through inference. Obviously, if the mask ratio is high enough, the model is hard to correctly fill the masked contents from the erased article and reference summary. However, we assume the trained reconstruction model is able to fill the masks with plausible contents by inferring the related contents with the masked article.

3.2 Masked Summarization↩︎

As a variant of MFMA, we also study another negative summary generation model, Masked SuMmarization(MSM). The model aims to generate summaries using masked articles \(\overline{A}_{\gamma_A}\) but without masked reference summaries as follows:

\[\begin{align} \mathcal{L} &= \sum_t -\log P({S_t | S_{<t}, \overline{A}_{\gamma_A}}). \label{lmrkiuxf} \end{align}\tag{2}\]

The MSM model is trained to generate the entire summaries without the information guidance of masked reference summaries, so MSM has merits in generating more diverse summaries than MFMA.

3.3 Training Factual Consistency Checking Model↩︎

Finally, for the factual consistency checking model, we train a binary classifier of consistent summaries and inconsistent generated summaries. The pair of summary and the corresponding article are concatenated and then fed into the classification model as an input. We fine-tuned the pre-trained ELECTRA [19] by adding a classifier head with binary cross-entropy loss.

4 Experiments↩︎

4.1 Implementation Details↩︎

4.1.0.1 Negative Summary Generation

We randomly split the training set of CNN/DM dataset [20] in half and use half for training negative summarizer and the other half for generating negative summary after training. We use spaCy for finding entities and noun phrases in both summaries and articles. We train bart-base¹ for five epochs to train MFMA, and use bart-base model without fine-tuning for MF. We use t5-small [21]² for MSM, which shows better results than bart-base for this task. We attach the further details in Appendix.

4.1.0.2 Training Classifier

We train google/electra-base-discriminator³ for five epochs with learning rate 2e-5, batch size of 96 using adam optimizer [22] with the dataset we generate using MF, MFMA and MSM. For DocNLI and FactCC, we get the original training dataset that each author release, and we train a model with the same setting as our method except for the training datasets for a fair comparison. We choose model using the balanced accuracy on validation set of FactCC [12] which consists of 1k human annotated summaries.

4.2 Benchmark Datasets↩︎

For evaluating the performance of factual consistency checking system, it is necessary to compare the human judgments of the consistency for the summary with the system. And these human judgment exist in two forms, binary level(consistent, inconsistent) or numerical levels such as likert scale. In general, in the case of binary level data, performance is measured through accuracy with human judgments. For the case of numerical levels, correlation with human judgments is measured. In addition to using the results for the existing benchmark dataset in this way, we also report the accuracy by casting these numerical level datasets to the binary level dataset since we develop classifier based system. We report the results on the following datasets.

4.2.0.1 FC-Test

 [12] release a human-annotated factual consistency for the model generated summaries for CNN/DM Dataset in binary-level to test the performance of FactCC. There are 513 instances in this dataset.

4.2.0.2 XSumHall

 [5] study the types of hallucination in the generated summaries and collect the annotation on the errors in the 2K model generated summary for BBC XSum dataset [23]. We use the datasets as binary level benchmark for XSum dataset as in [12].

4.2.0.3 SummEval

 [24] collect the likert scale human judgments for the 1600 summaries generated from sixteen abstactive summarizer on CNN/DM testset. This dataset provides human judgments scores in terms of "coherence" , "consistency", "fluency", and "relevance" by three expert annotators in likert scale. We only use "consistency" score of three annotators, for evaluating our proposed metric. For casting this score to binary level, we let the cases where at least one annotators give less than 5 points for "consistency" as inconsistent, otherwise consistent.

4.2.0.4 QAGS-CNN/DM & XSum

 [14] release a human judgments for factual consistency on the model generated summaries for 235 summaries on CNN/DM testset and 239 summaries on XSum testset. Each summary is annotated by three annotators. We also cast the dataset to binary level by assigning inconsistent if at least one annotators give inconsistent label, otherwise consistent.

4.2.0.5 FRANK-CNN/DM & XSum

 [25] releases a benchmark dataset FRANK for summarization factual metrics which consists of 2246 summaries on the model generated summaries for 1250 summaries in CNN/DM and 996 summaries XSum. Three annotators evaluated factual consistency of the generated summaries in this dataset. We also convert this dataset to binary level as same as QAGS-CNN/DM and QAGS-XSum.

4.3 Baseline Metrics↩︎

We compare our methods with the following metrics. For all of the baseline metrics, we manually compute the score using the official repository which each author provided or reproducing the model for a fair comparison.

4.3.0.1 Entailment Based Metrics

We adopt the model trained on MNLI [8] and FEVER [9] for factual consistency checking as in [12]. FactCC [12] and DocNLI [13] are also entailment based models trained on synthetic dataset as in our work.

4.3.0.2 QA-Based Metrics

QuestEval [15] uses the question generation and answering framework for evaluating the factual consistency of the summaries. QuestEval generates the question both the generated summaries and the source article, and then compare the answers of them with both summaries and the article to compute the factuality score of the summary.

4.3.0.3 N-gram Similarity Metrics

BLEU [7], ROUGE [6], and METEOR [26] are widely used for evaluating the summaries. Among them, ROUGE-L, which uses F-measure based on the longest common subsequence between a candidate summary an the reference is the most widely used.

4.3.0.4 Other Metrics

BERTScore [27] utilizes cosine similarity of BERT [28] embeddings between the reference and the generated summary. CoCo [17] computes the difference of likelihood of the summarizer between the summary with the original source and the summary with the masked source.

4.4 Results↩︎

4.4.0.1 Classification Accuracy

Due to the imbalance in each dataset, we report the macro-F1 and class balanced accuracy in Table 1 and Table 2. We observe that macro-F1 score of our proposed methods MFMA outperforms baseline entailment metrics in five of seven benchmark datasets. MFMA shows better performances than other methods in especially for CNN/DM benchmarks, and shows similar performance to other baseline in XSum datasets. We explain that this is because we only use training set of CNN/DM to construct training set. On the other hand, DocNLI additionally uses the human annotated datasets from related tasks such as ANLI [29] and SQuAD [30] except for synthetic negative summaries. Another proposed method MSM also shows competitive performance for CNN/DM benchmarks, but relatively lower performance in XSum based benchmark datasets. We explain the performance gap between MSM and MFMA is due to the properties that directly generates summaries, resulting in many noisy samples that are relatively easy to be distinguished.

4.4.0.2 Correlation with Human Judgments

To compare with general metrics that are not classification level, we also report the correlation with human judgments for five datasets in Table 3. We demonstrate that our proposed method has higher pearson correlation coefficient with human judgments in three of five benchmark datasets and competitive with the best results results in the spearman correlation coefficient. Especially, entailment based methods, which are relatively easy to compute, including our proposed methods show better results than QA-based QuestEval or likelihood based CoCo. Also, reference based methods such as ROUGE-L show very lower performance than other methods that do not require any references.

4.5 Analysis and Discussion↩︎

4.5.0.1 Performance among Masked Ratio

We analyze the effects of the mask ratio for both source article and summary in our proposed method MFMA and present results using the validation set in Figure 3. Through this experiment, we investigate the tradeoff in adjusting both the article masking ratio and summary masking ratio for generating negative summaries. As shown in Figure 3, we find that too high masking ratio decreases performance by sacrificing affinity. On the other hand, if the masking ratio is insufficient, the generated negative sample is often not really negative. In other words, too lower masking ratio leads to generate positive samples that are almost same as the original summary, and this degrades the performance of factual consistency checking model. Also, we can infer that there is an optimal masking ratio combination where the performance of factual consistency checking model is maximized.

4.5.0.2 Generated Samples among Masking Ratio

We visualize the generated negative summaries through our proposed method MFMA and MSM using CNN/DM in Figure 4. We also visualize the example through MF, which simply fills in the mask without the article. We observe that if the article masking ratio \(\gamma_A\) is too low, the generated summaries become almost similar to the original summary since there are enough information to fill the mask. However, if the \(\gamma_A\) is too high, the generated examples are too far from the article, resulting in too negative summary similar to filling the mask without article.

4.5.0.3 Performance among Masking Unit

We basically perform masking operation in the noun phrases and entities units for both summary and article. In order to see the effect of the masking unit, we also conduct an experiment on word level masking and sentence level masking, and present the classification level results in Table 4. We observe that noun phrases level masking shows the best results following the work [31] where many errors in summarization system are related to noun phrases and entities.

4.5.0.4 Distance from Original Reference Summary

Using the results on various combinations of article masking ratio and summary masking ratio for MFMA as presented in Figure 3, we also investigate the relation between the average distance from the reference summary on each mask ratio combination and the performance. We compute BERTScore between original reference summary and the negative summary generated using the reference summary to get the distance. Interestingly, as shown in Figure 5, we observe the distribution in which performance is maximized within the appropriate distance around 0.8 as the two-dimensional distribution with an \(R^2\) of 0.74. This result shows how far the synthetic negative summaries must be from the reference summaries to help training the factual consistency checking model.

4.5.0.5 Diversity among Masked Ratio

Our proposed method can generate various samples depending on the location of the mask for the same summary-article pair with the fixed mask ratio. Hence, we analyze the diversity of the generated negative summaries among the combinations of mask ratio for MFMA and present the result using validation set in Figure 6. We define the diversity of each mask ratio combination as the negation of pairwise similarity score for each sample following [32]. We sample four negative summaries using the given article for each method and then compute the pairwise similarity scores for all of the combinations. We also use BERTScore as a similarity measure. Similar to the distance, we observe that diversity has also similar to a two-dimensional form with an \(R^2\) of 0.7, in which the accuracy is maximized at an appropriate point.

4.5.0.6 Case Study

To understand the pros and cons of our proposed factual consistency checking system, we conduct a case study and illustrate the representative success and failure cases in Figure 7. We observe that our system is good at judging the facts themselves in the summary like the first example, but still not perfect in examples that require high-level reasoning like the second example. We expect the system can be improved by adopting MFMA and MSM to the datasets that have more abstractive summaries which require more reasoning to check the factual consistency.

5 Conclusion↩︎

In this paper, we proposed an effective generation method of factually inconsistent summaries, called MFMA. In this method, some proportion of the source text and corresponding reference summaries is hidden, then a summarization model generates plausible but factually inconsistent summaries by inferring the masked contents. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained using our method generally outperform existing models and show a competitive correlation with human judgment.

Ethical Considerations↩︎

Our approach creates a synthetic dataset using a public dataset to train a factual consistency checking model. Therefore, in the process of generating such samples, ethically problematic datasets can be generated due to the bias of the pre-trained models, similar to other text generation tasks. For this reason, once the training process is completed, we remove the generated sample. And, we will not release the synthetic dataset itself, and will release only the trained factual consistency checking model.

Acknowledgements↩︎

K. Jung is with ASRI, Seoul National University, Korea. This research was supported by SNU-NAVER Hyperscale AI Center.

6 Experimental Details↩︎

6.1 Reproducibility Checklist↩︎

6.1.0.1 Source Code

We attach the source in the submission and we will release the pre-trained factual consistency checking model.

6.1.0.2 Computing Infrastructure

We use Intel(R) Xeon(R) Silver 4210R CPU (2.40 GHz) with NVIDIA RTX A5000 24GB for the experiments. The software environments are Python 3.8.8 and PyTorch 1.10.1.

6.1.0.3 Dataset Statistics

We use the training of CNN/DM dataset that consists of 287113 examples. We divide it in half randomly and use one for MSM or MFMA training and the other for generating negative summaries. Then, we merge the generated article-negative summaries pairs and the article-positive summaries we used for training MFMA and MSM to construct the training set for factual consistency checking model.

6.1.0.4 Average runtime for each approach

For training MFMA and MSM, it takes 10 hours to train the whole model. And it takes 3 hours to generate whole negative summaries that is to be used for training factual consistency checking. For training factual consistency checking model, it takes 7 hours using a single GPU.

6.1.0.5 Hyperparameters

We train five epochs for MFMA and MSM using bart-base for MFMA and t5-small for MSM respectively. We train the model with batch size of 48, max input sequence size of 1024, and max target sequence size of 140. We conduct experiment with various article masking \(\gamma_A\) ratio-summary masking ratio \(\gamma_S\) combinations, at 0.2 intervals from (0.2, 0.2) to (1.0, 1.0). For the case of training classifier, we train google/electra-base-discriminator for five epochs with learning rate 2e-5 and batch size of 96. We choose the best parameters using the validation set provided by the [12]. The best mask ratio combination is \(\gamma_A\) = 0.6 and \(\gamma_S\) = 0.8.

6.1.0.6 Number of Model Parameters

The number of parameters for negative summary generation model is 139M for MFMA, is 0.6M (t5-small) and the factual consistency classifier is 109M.

6.2 Computing Baseline Metrics↩︎

Even with the same dataset, the results may be different due to some factors such as type of tokenizer or case, so we calculate baseline ourselves as follows. For n-gram similarity metrics BLEU-4, ROUGE-L and METEOR, we compute the scores using the package language evaluation⁴ which is based on COCOeval⁵. For BERTScore⁶, QuestEval⁷ and CoCO⁸, we use the official repository with the default setting. For MNLI, we use roberta-large-mnli⁹ and use tals/albert-base-vitaminc-fever¹⁰ for FEVER.

6.3 Significance Test↩︎

We adopt standard way to test the significance of the correlation coefficient for all of the reported related correlation coefficients in Table 3. We compute the p-value for each coefficient with a t-test that uses a null hypothesis, which is an absence of association.

References↩︎