1 Introduction↩︎

The growing trend in online web forums is to attract more and more consumers to use the Internet for their health information needs. An instinctive way for consumers to query for their health-related content is in the form of natural language questions. These questions are often excessively descriptive and contain more than required peripheral information. However, most of the textual content is not particularly relevant in answering the question [1]. A recent study showed that manual summarization of consumer health questions (CHQ) has significant improvement (58%) in retrieving relevant answers [2]. However, three major limitations impede higher success in obtaining semantically and factually correct summaries: (1) the complexity of identifying the correct question type/intent, (2) the difficulty of identifying salient medical entities and focus/topic of the question, and (3) the lack of large-scale CHQ summarization datasets. To address these limitations, this work presents a new reinforcement learning based framework for abstractive question summarization. We also propose two novel question-aware semantic reward functions: Question-type Identification Reward (QTR) and Question-focus Recognition Reward (QFR). The QTR measures correctly identified question-type(s) of the summarized question. Similarly, QFR measures correctly recognized key medical concept(s) or focus/foci of the summary.

We use the reinforce-based policy gradient approach, which maximizes the non-differentiable QTR and QFR rewards by learning the optimal policy defined by the Transformer model parameters. Our experiments show that these two rewards can significantly improve the question summarization quality, separately or jointly, achieving the new state-of-the-art performance on the MeQSum and MATINF benchmark datasets. The main contributions of this paper are as follows:

• We propose a novel approach towards question summarization by introducing two question-aware semantic rewards (i) Question-type Identification Reward and (ii) Question-focus Recognition Reward, to enforce the generation of semantically valid and factually correct question summaries.

• The proposed models achieve the state-of-the-art performance on two question summarization datasets over competitive pre-trained Transformer models.

• A manual evaluation of the summarized questions reveals that they achieve higher abstraction levels and are more semantically and factually similar to human-generated summaries.

2 Related Work↩︎

In recent years, reinforcement learning (RL) based models have been explored for the abstractive summarization task. [3] introduced RL in neural summarization models by optimizing the ROUGE score as a reward that led to more readable and concise summaries. Subsequently, several studies [4]–[8] have proposed methods to optimize the model losses via RL that enables the model to generate the sentences with the higher ROUGE score. While these methods are primarily supervised, [9] proposed an unsupervised method that accounts for fluency, brevity, and coverage in generated summaries using multiple RL-based rewards. The majority of these works are focused on document summarization with conventional non-semantics rewards (ROUGE, BLEU). In contrast, we focus on formulating the semantic rewards that bring a high-level semantic regularization. In particular, we investigate the question’s main characteristics, i.e., question focus and type, to define the rewards.
Recently, [10] defined the CHQ summarization task and introduced a new benchmark (MeQSum) and a pointer-generator model. [11] organized the MEDIQA-21 shared task challenge on CHQ, multi-document answers, and radiology report summarization. Most of the participating team [12]–[14] utilized transfer learning, knowledge-based, and ensemble methods to solve the question summarization task. [15] proposed question-aware transformer models for question summarization. [16] automatically created a Chinese dataset (MATINF) for medical question answering, summarization, and classification tasks focusing on maternity and infant categories. Some of the other prominent works in the abstractive summarization of long and short documents include [17]–[20].

3 Proposed Method↩︎

Given a question, the goal of the task is to generate a summarized question that contains the salient information of the original question. We propose a RL-based question summarizer model over the Transformer [21] encoder-decoder architecture. We describe below the proposed reward functions.

3.1 Question-aware Semantic Rewards↩︎

(a) Question-type Identification Reward: Independent of the pre-training task, most language models use maximum likelihood estimation (MLE)-based training for fine-tuning the downstream tasks. MLE has two drawbacks: (1) “exposure bias" [22] when the model expects gold-standard data at each step during training but does not have such supervision when testing, and (2)”representational collapse” [23], is the degradation of generalizable representations of pre-trained models during the fine-tuning stage. To deal with the exposure bias, previous works used the ROUGE and BLEU rewards to train the generation models [3], [22]. These evaluation metrics are based on n-grams matching and might fail to capture the semantics of the generated questions. We, therefore, propose a new question-type identification reward to capture the underlying question semantics.

We fine-tuned a BERT\(_\text{BASE}\) network as a question-type identification model to provide question-type labels. Specifically, we use the [CLS] token representation (\(\boldsymbol{h}_{[CLS]}\)) from the final transformer layer of BERT\(_\text{BASE}\) and add the feed-forward layers on top of the \(\boldsymbol{h}_{[CLS]}\) to compute the final logits

Finally, the question types are predicted using the sigmoid activation function on each output neuron of logits \(l\). The fine-tuned network is used to compute the reward \(r_{QTR}(Q^p, Q^*)\) as F-Score of question-types between the generated question summary \(Q^p\) and the gold question summary \(Q^*\).

3.1.0.1 (b) Question-focus Recognition Reward:

A good question summary should contain the key information of the original question to avoid factual inconsistency. In the literature, ROUGE-based rewards have been explored to maximize the coverage of the generated summary, but it does not guarantee to preserve the key information in the question summary. We introduce a novel reward function called question-focus recognition reward, which captures the degree to which the key information from the original question is present in the generated summary question. Similar to QTR, we fine-tuned the BERT\(_\text{BASE}\) network for question-focus recognition to predict the focus/foci of the question. Specifically, given the representation matrix (\(\boldsymbol{H} \in \mathcal{R}^{n \times d}\)) of \(n\) tokens and \(d\) dimensional hidden state representation obtained from the final transformer layer of BERT\(_\text{BASE}\), we performed the token level prediction using a linear layer of the feed-forward network. For each token representation (\(h_i\)), we compute the logits \(l_i \in \mathcal{R}^{|C|}\), where (\(|C|\)) is the number of classes and predict the question focus as follows: \(f_i= softmax(\boldsymbol{W}h_i + \boldsymbol{b})\). The fine-tuned network is used to compute the reward \(r_{QFR}(Q^p, Q^*)\) as F-Score of question-focus between the generated question summary \(Q^p\) and the gold question summary \(Q^*\).

3.2 Policy Gradient REINFORCE↩︎

We cast question summarization as an RL problem, where the “agent” (ProphetNet decoder) interacts with the “environment” (Question-type or focus prediction networks) to take “actions” (next word prediction) based on the learned “policy” \(p_\theta\) defined by ProphetNet parameters (\(\theta\)) and observe “reward” (QTR and QFR). We utilized ProphetNet [24] as the base model because it is specifically designed for sequence-to-sequence training and it has shown near state-of-the-art results on natural language generation task. We use the REINFORCE algorithm [25] to learn the optimal policy which maximizes the expected reward. Toward this, we minimize the loss function \(\mathcal{L}_{RL} = -E_{Q^s \sim p_\theta}[r(Q^s, Q^*)]\), where \(Q^s\) is the question formed by sampling the words \(q_{t}^{s}\) from the model’s output distribution, i.e. \(p(q_t^s|q_1^s, q_2^s, \ldots, q_{t-1}^s, \mathcal{S})\). The derivative of \(\mathcal{L}_{RL}\) is approximated using a single sample along with baseline estimator \(b\): \[\bigtriangledown_\theta \mathcal{L}_{RL} = -(r(Q^s, Q^*) - b)\bigtriangledown_\theta log p_\theta(Q^s)\] The Self-critical Sequence Training (SCST) strategy [26] is used to estimate the baseline reward by computing the reward with the question generated by the current model using the greedy decoding technique, i.e., \(b = r(Q^g, Q^*)\). We compute the final reward as a weighted sum of QTR and QFR as follows: \[ r(Q^p, Q^*) = \gamma_{QTR} \times r_{QTR}(Q^p, Q^*) + \gamma_{QFR} \times r_{QFR}(Q^p, Q^*)\] We train the network with the mixed loss as discussed in [3]. The overall network loss is as follows: \[\label{main-eq} \mathcal{L} = \alpha \mathcal{L}_{RL} + (1-\alpha)\mathcal{L}_{ML}\tag{1}\] where, \(\alpha\) is the scaling factor and \(\mathcal{L}_{ML}\) is the negative log-likelihood loss and equivalent to \(-\sum_{t=1}^{t=m} logp(q_t^*|q_1^*, q_2^*, \ldots, q_{t-1}^*, \mathcal{S})\), where \(\mathcal{S}\) is the source question.

4 Experimental Results & Analysis↩︎

4.1 Datasets↩︎

We utilized two CHQ abstractive summarization datasets: MeQSum and MATINF² to evaluate the proposed framework. The MeQSum³ training set consists of \(5,155\) CHQ-summary pairs and the test set includes \(500\) pairs. We chose \(100\) samples from the training set as the validation dataset.
For fine-tuning the question-type identification and question-focus recognition models, we manually labeled the MeQSum dataset with the question type: (‘Dosage’, ‘Drugs’, ‘Diagnosis’, ‘Treatments’, ‘Duration’, ‘Testing’, ‘Symptom’, ‘Usage’, ‘Information’, ‘Causes’) and foci. We use the labeled data to train the question-type identification and question-focus recognition networks. For question-focus recognition, we follow the BIO notation and classify each token for the beginning of focus token (B), intermediate of focus token (I), and other token (O) classes. Since, the gold annotations for question-types and question-focus were not available for the MATINF dataset, we used the pre-trained network trained on the MeQSum dataset to obtain the silver-standard question-types and question-focus information for MATINF⁴. The MATINF dataset has \(5,000\) CHQ-summary pairs in the training set and \(500\) in the test set.

4.2 Experimental Setups↩︎

We use the pre-trained uncased version⁵ of ProphetNet as the base encoder-decoder model. We use a beam search algorithm with beam size \(4\) to decode the summary sentence. We train all summarization models on the respective training dataset for \(20\) epochs. We set the maximum question and summary sentence length to \(120\) and \(20\), respectively. We first fine-train the proposed network by minimizing only the maximum likelihood (ML) loss. Next, we initialize our proposed model with the fine-trained ML weights and train the network with the mixed-objective learning function (Eq. 1 ). We performed experiments on the validation dataset by varying the \(\alpha, \gamma_{QTR}\) and \(\gamma_{QFR}\) in the range of \((0,1)\). The scaling factor (\(\alpha\)) value \(0.95\), was found to be optimal (in terms of Rouge-L) for both the datasets. The values of \(\gamma_{QTR}=0.4\) and \(\gamma_{QFR}=0.6\) were found to be optimal on the validation sets of both datasets. To update the model parameters, we used Adam [34] optimization algorithm with the learning rate of \(7e-5\) for ML training and \(3e-7\) for RL training. We obtained the optimal hyper-parameters values based on the performance of the model on the validation sets of MeQSum and MATINF in the respective experiments. We used a cosine annealing learning rate [35] decay schedule, where the learning rate decreases linearly from the initial learning set in the optimizer to \(0\). To avoid the gradient explosion issue, the gradient norm was clipped within \(1\). For all the baseline experiments, we followed the official source code of the approach and trained the model on our datasets. We implemented the approach of [10] to evaluate the performance on both datasets. All experiments were performed on a single NVIDIA Tesla V100 GPU having GPU memory of \(32\)GB. The average runtimes (each epoch) for the proposed approaches \(M_2\), \(M_3\) and \(M_4\) were \(2.7, 2.8\) and \(4.5\) hours, respectively. All the proposed models have \(391.32\) million parameters.

4.3 Results↩︎

We present the results of the proposed question-aware semantic rewards on the MeQSum and MATINF datasets in Table-1. We evaluated the generated summaries using the ROUGE [36] metric⁶. The proposed model achieves new state-of-the-art performance on both datasets by outperforming competitive baseline Transformer models. We also compare the proposed model with the joint learning baselines, where we regularize the question summarizer with the additional loss obtained from the question-type (Q-type) identification and question-focus (Q-focus) recognition model. To make a fair comparison with the proposed approach, we train these joint learning-based models with the same weighted strategy shown in Eq. 1 . The results reported in Table 1 show the improvement over the ProphetNet on both datasets.
In comparison to the benchmark model on MeQSum, our proposed model obtained an improvement of \(9.63\%\). A similar improvement is also observed on the MATINF dataset. Furthermore, the results show that individual QTR and QFR rewards also improve over ProphetNet and ROUGE-based rewards. These results support two major claims: (1) question-type reward assists the model to capture the underlying question semantics, and (2) awareness of salient entities learned from the question-focus reward enables the generation of fewer incorrect summaries that are unrelated to the question topic. The proposed rewards are model-independent and can be plugged into any pre-trained Seq2Seq model. On the downstream tasks of question-type identification and question-focus recognition, the pre-trained BERT model achieves the F-Score of \(97.10\%\) and \(77.24\%\), respectively, on 10% of the manually labeled MeQSum pairs.

4.3.0.1 Manual Evaluation:

Two annotators, experts in medical informatics, performed an analysis of 50 summaries randomly selected from each test set. In MATINF, nine out of the 50 samples contained translation errors. We thus randomly replaced them. In both datasets, we annotated each summary with two labels ‘Semantics Preserved’ and ‘Factual Consistent’ to measure (1) whether the semantics (i.e., question intent) of the source question was preserved in the generated summary and (2) whether the key entities/foci were present in the generated summary. In the manual evaluation of the quality of the generated summaries, we categorize each summary into one of the following categories: ‘Incorrect’, ‘Acceptable’, and ‘Perfect’. We report the human evaluation results (average of two annotators) on both datasets in Table-2. The results show that our proposed rewards enhance the model by capturing the underlying semantics and facts, which led to higher proportions of perfect and acceptable summaries. The error analysis identified two major causes of errors: (1) Wrong question types (e.g. the original question contained multiple question types or has insufficient type-related training instances) and (2) Wrong/partial focus (e.g. the model fails to capture the key medical entities).

5 Conclusion↩︎

In this work, we present an RL-based framework by introducing novel question-aware semantic rewards to enhance the semantics and factual consistency of the summarized questions. The automatic and human evaluations demonstrated the efficiency of these rewards when integrated with a strong encoder-decoder based ProphetNet transformer model. The proposed methods achieve state-of-the-art results on two-question summarization benchmarks. In the future, we will explore other types of semantic rewards and efficient multi-rewards optimization algorithms for RL.

Acknowledgements↩︎

This research was supported by the Intramural Research Program of the National Library of Medicine, National Institutes of Health.

Ethics / Impact Statement↩︎

Our project involves publicly available datasets of consumer health questions. It does not involve any direct interaction with any individuals or their personally identifiable data and does not meet the Federal definition for human subjects research, specifically: “a systematic investigation designed to contribute to generalizable knowledge" and”research involving interaction with the individual or obtains personally identifiable private information about an individual."

References↩︎