1 Introduction↩︎

Long-form question answering (LFQA) is a task to generate an in-depth, paragraph-length answer for a given question [1]. It is important since many of the everyday questions that humans deal with and pose to search engines require multi-sentence explanations [2] (e.g. why/how..?). It can be integrated with a search engine [3], or a virtual conversation agent, and can also be used to generate explanations as a complement to short-phrase answers for open-domain questions [4], [5], or to answer open-ended questions like those from Reddit forum “Explain Like I’m Five” [1].

LFQA is quite a challenging task. It often involves searching a large external knowledge source that contains millions of documents for relevant information. Then it generates a paragraph-length answer from those retrieved sources. While the great success in retrieval technique [6]–[8] can be carried over to the LFQA setting, more challenges lie in the generation. First, multiple documents that contain hundreds of tokens need to be considered for generation, raising difficulties in the direct use of current pre-trained language models. Second, as different documents may contain redundant, complementary, or contradictory information, how to synthesis the information and generate a faithful answer that has less hallucinated content is even more challenging.

While recent work on LFQA [10] focuses primarily on the first challenge, and has produced fluent and somewhat relevant content, the latter faithfulness challenge has not been explored. However, the faithfulness issue is quite important for LFQA. As the example in Fig. 1 shown, a fluent and relevant but unfaithful answer (highlight in green) will mislead users.

In this paper, we propose a novel end-to-end framework named RBG (Read Before Generate) for LFQA to address the aforementioned challenges. The key idea for enhancing answer faithfulness is to augment the generation process with predicted salient information which can be viewed as an emphasis on answer-related facts. Specifically, we combine a Seq2Seq language model-based generator with a machine reading comprehension (reader) module. The reader produces an evidence probability score for each sentence, which will be integrated with the generator for final distribution prediction. We perform evidence fusion in a similar way as FiD [11] to equip the pre-trained language model with multiple input documents for generation. To further enhance the factual grounding ability of RBG, we propose an additional pre-training task to encourage the model to rely more on retrieved documents to generate factual statements. The details are explained in Section 2.

We conduct thorough experiments on our method and several baselines on ELI5 [1], the only publicly available large-scale LFQA dataset, and also on MS MARCO [9] passage ranking data, which can also be transformed into an answer generation task. The proposed method tops the public leaderboard of the KILT [12] Benchmark on the ELI5 dataset. It also outperforms the baselines, including non-retrieval and retrieval-based methods, such as DPR-BART [13], RAG [11] and FiD [11], with improvement on the automatic evaluation results on the MS MARCO dataset. Human evaluation results further validate that our proposed framework can improve the generation quality in terms of relevance and factual correctness.

Our contributions are summarized as follows:

• To the best of our knowledge, we are the first trying to tackle the faithfulness challenge in LFQA.

• We propose a new and effective framework for open-domain LFQA to generate answers with the guidance of a sentence evidence score from a machine reading module, as well as an additional factual grounding-oriented pre-training task.

• We show the effectiveness of our method by both automatic evaluation and human evaluation on two large-scale datasets, and we also demonstrate by human evaluation that our method improves the factual correctness of generated answers while still keeping high informativeness.

2 A state-of-the-art LFQA system↩︎

To generate in-depth, long-form answers for a given general domain question, we first use a retriever to search for relevant information from a large external knowledge source. Then our reader and the generation module take the multiple retrieved documents together with the question as input to generate the answer. Specifically, the reader module adopts a machine reading comprehension (MRC) model to produce an evidence score for each sentence in each document, while the generator, which adopts a large pre-trained Seq2Seq language model, fuses the sentence evidence score into its generation process. Our framework is shown in Figure 2.

2.1 Supporting document retriever↩︎

We use DPR [7] to retrieve the supporting documents following the typical methods in the state-of-the-art framework for open-domain QA [11], [14]. The passage and question are represented as 768-dimensional dense vector representations, computed via the BERT-based bi-encoder networks of DPR. The retriever will rank the documents according to their relevance, calculated as \[Score_{re}(Q, D_i) = \text{BERT}_q(Q)^T \text{BERT}_d(D_i) \label{eqa:retriever95score}\tag{1}\]

Retrieval is performed using approximate nearest neighbors with the FAISS² library. We denote \(D=\{D_1, D_2,..., D_k\}\) as the top-\(K\) retrieved documents for question \(Q\).

2.2 Document reader↩︎

Since there are no golden retrievals for long-form answers, the retrieved documents may contain complementary, contradictory, or redundant information related to the answer. Thus, we propose to use a reader module to explicitly predict the sentence-level evidence probability in each document.

2.2.0.1 Evidence span prediction

We use a machine reading comprehension (MRC) model to predict the evidence span in each document, as these models approach or even outperform human-level performance on many datasets [15]. The MRC model takes the concatenation of the retrieved document \(D_i\) and question \(Q\) as input, and outputs the prediction of the start and end position of the potential evidence spans in \(D_i\). Specifically, it outputs two probability distributions over the tokens in \(D_i\): \(P^s_i(w_s)\) and \(P^e_i(w_s)\), where \(P^s_i(w_s)\) / \(P^e_i(w_s)\) is the probability that the token \(w_s\) is the start/end of the evidence span in \(D_i\).

2.2.0.2 Sentence evidence probability

Originally, the MRC model was designed to give accurate, short-phrase span prediction [16], but we argue that a sentence-level evidence probability will be better in our scenario. The supporting sentences can provide the minimum required context information for each answer span, which is quite important, especially in multi-document generation [17]. We define our sentence-level evidence probability score for the \(i\)-th document \(P^i_{rea}(S)\) as the summation over all token-level evidence probabilities in that sentence, and it is calculated via \[\begin{align} P^i_{rea}(S) &= \frac{1}{2}\sum\nolimits_{w_s\in S}(P^s_i(w_s) + P^e_i(w_s)) \\ P_{rea}(S) &= \text{Norm}(P^1_{rea}(S); ...P^i_{rea}(S); ...P^K_{rea}(S)) \end{align}\] We concatenate \(P^i_{rea}\) and normalize the distribution as \(P_{rea}(S)\), where \(P_{rea}(S)\) denotes the final sentence-level evidence probability in all the \(K\) documents regarding the question.

2.2.0.3 Multi-task MRC

As there are no golden answer spans for LFQA data, we need a MRC model that has enough generalization ability for open domain questions as a starting point. We choose SpanBERT [15], and further fine-tune it in a multi-task way on six large-scale MRC datasets from the MRQA shared task  [18] following work by [19]: SQuAD [16], NewsQA [20], TriviaQA [21], SearchQA [22], HotpotQA [5], and NatualQuestions [4]. The multi-task fine-tuned MRC model \(R\) will be further jointly trained with the generator, using the golden answer in a distantly supervised way.

2.3 Generator↩︎

2.3.0.1 FiD-BART

We choose BART as our generation backbone because of its outstanding performance on many generation tasks, especially on long-form abstractive summarization task [23]. We propose FiD-BART, following the Fusion-in-Decoder idea from [11], to empower BART to deal with multiple, long-document inputs. FiD-BART processes each document independently in the encoder, while performing the cross-attention in the decoder jointly.

The encoder encodes the concatenation of each supporting document \(D_i\) and the question \(Q\). More precisely, we append the special tokens question: before \(Q\), title: and context: before the title and text of each document \(D_i\). We denote the encoded final representation of the encoder as \(h_{enc}\), which is the concatenation of the \(K\) encoder outputs \(h^i_{enc}\) (\(h^i_{enc} \in R^{d \times l_i}\)) for the \(i\)th document: \[\begin{align} h^i_{enc} &= \text{Encoder}(Q;D_i) \\ h_{enc} &= (h^1_{enc},..., h^i_{enc},..., h^K_{enc}) \end{align}\]

The partial structure of the decoder can be illustrated by Eq.(6)–(8), where \(h_l\) is the representation for the \(l\)-th decoder layer. We denote \(h_{dec}\) as the last layer decoder outputs: \[\begin{align} h_l^a &= \text{SelfAttention}(h_l, h_l, h_l) \\ h_l^b &= \text{LayerNorm}(h_l + h_l^a) \\ \label{cross-attention} h_l^c &= \text{CrossAttention}(h_l^b, h_{enc}, h_{enc}) \end{align}\tag{2}\]

As we can see, FiD-BART can scale to a large number of input documents within a linear computation time.

2.4 Reader-before-generator↩︎

To incorporate the evidence probability into generation, we apply the pointer-generator model (depicted in Figure 2). The attention distribution \(\mathcal{A}\) and context vector \(h_c\), and the generation probability \(p_{gen}\) \(\in\) [0,1] are calculated as follows: \[\begin{align} \mathcal{A} &= softmax(h_{dec} h^T_{enc}) \\ h_{c} &= \mathcal{A} h_{enc} \\ p_{gen} &= sigmod(W_{c}h_{c} + W_{g}h_{dec}) \end{align}\] where \(W_c\) and \(W_g\) are learnable parameters. \(p_{gen}\) is used as a soft switch to choose between generating a word from the generator by sampling from the vocab, or copying a word from the input sequence by sampling according to the evidence distribution \(P_{rea}(w)\): \[\begin{align} &P_{gen}(w) = lm_{head}(h_{dec}) \\ &P_{rea}(w) = \sum\nolimits_{s:w_s=w\text{,}w_s \in S}{P_{rea}(S)} \\ &P(w) = p_{gen}P_{gen}(w) + (1-p_{gen})P_{rea}(w) \end{align}\]

2.5 Pre-training↩︎

To further improve the ability to ground on retrieved documents, we propose a pre-training task: retrieval-augmented recovery (RAR). Instead of recovering the corrupted text through the internal knowledge memorized in model parameters [23], [24], RAR encourages the model to rely more on external retrieved documents to generate factual statements. Specifically, given an original text \(S\), we retrieve the top-\(k\) documents \({D_1,D_2,...,D_N}\) from the knowledge corpus using BM25 (discarding \(S\) itself), and we replace 30% of the words in \(S\) with to form a pseudo query \(Q\). The pre-training task asks our RBG model to recover \(S\) with the input of the pseudo query \(Q\) and \(k\) retrieved documents, which can be formulated as \[S=RBG(Q;D_1,D_2,...,D_k) \] To involve more factual information during the text corruption and recovery process, we sample 1 million sentences of \(S\) corresponding to at least one knowledge base triplet from Wikipedia with the text-triple alignment of TREX [25].

3 Experiment Setups↩︎

3.1 Datasets↩︎

We conduct experiments on the two following datasets, both of which concentrate on long form generative QA.

3.1.0.1 ELI5

 [1] is the only publicly available large-scale LFQA dataset. It is a collection of question-answer pairs extracted from the Reddit forum "Explain Like I’m Five"(ELI5). We use the KILT [12] version of the dataset from its Github repository³, which has 272,634 training examples and 1,507 development examples. The average length of the answers is 130 words.

3.1.0.2 MS MARCO

 [9] is a dataset of crowdsourced responses to Bing queries. We use the question-answer pairs of the MS MARCO passage ranking track for training and evaluation, as they are more abstract and reliant on multi-document information than those of the NLG track. The training example size is about 500,000 and the evaluation example size is 6980.

3.1.0.3 Knowledge source

The external knowledge source of the retriever is the Wikipedia paragraphs, which are provided in the KILT benchmark as a unified knowledge source for knowledge-intensive tasks, including open-domain LFQA  [12]. It is based on the 2019/08/01 Wikipedia snapshot, and contains 5.9M articles.

3.2 Baselines↩︎

3.2.0.1 BART and T5

We fine-tune BART [23] and T5 [24] using QA pairs without explicit retrieval, and include them as our baselines which rely only on parameterized internal knowledge [26] to generate answers.

3.2.0.2 DPR-BART

is our retrieval-based LFQA baseline. We follow  [13] to retrieve and prepend the top-3 passages from DPR for each input sample, and use context-enhanced training data to fine-tune a BART model.

3.2.0.3 RAG

 [14] is an end-to-end retrieval-augmented generation model which back-propagates to the retriever’s input encoder. We experiment with fine-tuning RAG on LFQA tasks, establishing a strong baseline on all of them. At every generation step we retrieve the top-5 passages and use them as supporting documents.

3.2.0.4 FiD

 [11] encodes each passage independently and combines all outputs from the encoder before passing them to the decoder. FiD has achieved superior performance on a number of open-domain QA tasks [11]. We implement FiD-BART, using BART as the generation backbone, as our strongest baseline.

4 Experiment Results↩︎

4.1 Automatic Evaluation↩︎

We use the metrics unigram F1 score and ROUGE-L [27] in previous work on LFQA [10], [12] to evaluate and compare the generation quality of our method.

4.1.0.1 Overall Comparison

Table 1 shows the performance of various methods on the two datasets. As shown, our RBG method outperforms all baselines models with regard to both evaluation metrics on both datasets. The RBG method also outperforms the previous state-of-the-art method c-REALM+RT on the KILT-ELI5 leaderboard⁴  [10], as shown in Table 2.

4.1.0.2 Fine-grained Comparison

Intuitively, the quality of retrieved documents will affect the generation quality, thus we provide a fine-grained performance comparison. We split MS-MARCO evaluation set into different subset based on the quality of the retrieved documents⁵, and compare the ROUGE-L score between FiD and RBG under each subset.

As we can see from Table 4, even though RBG beats FiD by 0.1 Rouge-L score on the whole MS-MARCO evaluation set, the performance gap continue increasing as the retrieval quality of the evaluation subset increased. This indicates that RBG is especially effective when high-quality retrieval documents is provided, which matches with our intuition.

4.2 Human evaluation↩︎

We further evaluate our model using human annotators, who we ask to quantify three aspects of the generated answer, (1) fluency, which measures whether the answer is coherent and less repetitive; (2) relevance, which measures the amount of information relevant to answering the question, and (3) factual correctness (also briefly called correctness), which measures the correctness and faithfulness of all facts involved in the generated answer.

We select FiD, which is the strongest baseline in terms of automatic metrics, for comparison. We sample evaluation questions from the MS MARCO dev set, which are better supported by Wikipedia knowledge than ELI5. Table 5 shows the absolute evaluation results of human annotation. To reduce the impact of scale selection inconsistency of different annotators, we also show the relative evaluation results in Table 6. We can see that both types of results indicate that RBG outperforms FiD in terms of all three aspects. RBG has more advantages over FiD on the metric of factual correctness, possibly benefited by the introduction of the reader module and additional pre-training. More details of the human evaluation setup and statistical analysis can be found in Appendix 10.

4.3 Ablation↩︎

To further investigate the contribution and effect of each module in the proposed system, we conducted a systematic ablations on the MS-MARCRO evaluation dataset.

4.3.0.1 w/o reader/pre-training:

We respectively remove the reader module (w/o reader), the pre-training (w/o pre-training), and both together (w/o reader + pre-training) from our model , to test the contribution of each part. As we can see from Table 7, without the reader to predict the evidence probability, the generation performance decreases in both metrics, and the performance continues to drop without the pre-training.

4.3.0.2 w/ reader frozen:

We freeze the reader to investigate the benefit of distantly supervised end-to-end training of the reader module. As we can see from Table 7, the results on both metrics drop, especially the F1 score, which proves the effectiveness of the end-to-end training.

4.3.0.3 w/ random retrieval:

To investigate whether and how much the generation process is grounded in the retrieved documents, we replace retrieved paragraphs with randomly sampled paragraphs from Wikipedia at inference time for comparison. As we can see, the ROUGE-L drops significantly with randomly retrieved documents, and it is also worse than the baseline systems such as BART and DPR-BART (Table 1).

5 Further analysis↩︎

We conduct further analysis on the results, considering that LFQA is a complicated but less explored task, which deserves a complete investigation.

5.1 How does retriever affect the generation quality?↩︎

We further investigate the effects of the quality of retrieved documents on the final generation. We split the evaluation sets of the two datasets via different thresholds for the two metrics⁴ and calculate the corresponding ROUGE-L score for each subset. As we can see in Table 9, better-retrieved documents always bring better generation quality, indicating the importance of high-quality supporting documents for the generation process.

We also measure the effects of the number of retrieved documents \(K\) on the generation quality and find that the best \(K\) from \(\{5,10,20,50\}\) is 10. More retrieved documents do not improve generation quality as in open-domain QA.

5.2 How does the reader contribute to the generation?↩︎

As shown in the ablation study, the reader module improves the overall performance on the MS MARCO evaluation dataset. We further investigate its performance when retrieved documents with different quality levels are provided.

We show in Figure 3 the fine-grained comparison results between ablation models No.2: RBG w/o pre-training and No.3: RBG w/o pre-training + reader. As we can see, the difference in ROUGE-L between the two models increases as the quality of the retrieved documents improves, indicating the reader’s strong capability, especially on high-quality data. This also matches with our intuition. We also conduct a human evaluation for reader analysis, and we show the results in Table 10.

5.3 How does pre-training help?↩︎

We also compare the models’ performance in a fine-grained way, to quantify the contribution from our pre-training task. We show in Figure 4 the fine-grained comparison results between ablation models No.1: RBG w/o reader and No.3: RBG w/o pre-training + reader. As we can see, the model with pre-training is better in most situations than that without pre-training. The human evaluation in Table 11 also indicates the effectiveness of our pre-training task to improve the factual correctness and relevance of the generated answer. We conjecture that the pre-training task of retrieval-augmented recovery can facilitate the downstream LFQA model to combine multiple pieces of evidence from different retrieved documents to generate the final answer.

5.4 Faithfulness analysis↩︎

5.4.0.1 Zero-shot on extractive QA tasks

Inspired by previous work [28], [29] which leverage a Question Generation(QG) and a QA model to generate question answer pairs, to evaluate the faithfulness of a summary⁶, we propose to evaluate answer faithfulness via evaluation on two simpler open-domain QA datasets: NaturalQuestions [4] and HotpotQA [5], which contain single-hop or multi-hop factual questions with golden answers (\(\{(q_i, a^s_i)\}_{i=1}^m\)) where \(a_i^s\) can be extracted from Wikipedia-based documents. We use the trained models (based on MS MARCO) in Table 1 to do zero-shot long-form answer generation for these two datasets \(\{a^l_i = \text{Model}_{\text{ms}}(q_i)\}\), and measure the short-answer recall (the ratio of golden answer span \(a^s\) contained in the generated long answer \(a^l\)) as an estimation of faithfulness of the generated long-answer: \[Score(q, a^s, a^l) = \frac{\sum_{i=1}^m\mathbb{1}[a^s_i \in a^l_i]}{m} \label{eqa:faithfulness}\tag{3}\] We show the results in Table 12. As we can see, our system achieves comparable performance with FiD on NQ, and it consistently outperforms other strong baselines on multi-hop dataset hotpotQA, indicating its capability in generating faithful answer especially on complex question that need to synthesis information. We also give concrete examples in Appendix 12 that show our model can generate more faithful snippets than FiD apart from automatic metrics.

5.4.0.2 Case Study

To have a concrete understanding of the reader’s role to address faithfulness, we show two examples in Table ¿tbl:tab:reader95examples?. While both models use the same ctxs, RBG w reader generates a more relevant answer, and w/o reader only gives some correct but unrelated statements (Case 1). The reader also contributes to generating more faithful answers, as shown in Case 2. However, we notice that there is one unfaithful statement, which hallucinates the ‘second-oldest’ as ‘oldest’. This might be because of the property of our MRC model: it gives a high evidential score to one sentence, and weakening the importance of other sentences in the same document.

width=0.49,totalheight=,keepaspectratio

6 Related work↩︎

6.0.0.1 Grounded generation

is the task of leveraging external knowledge sources to enhance the generation. Previous work has either used structured external knowledge source  [30]–[32] or unstructured data.  [33] introduced a document grounded dataset for text conversations, and  [34] proposed to extract lexical control phrases to do controllable grounded response generation, while  [35] jointly trained a retriever and generator so that annotated text reference parallel data are not needed.

6.0.0.2 Open-domain QA

is the task of answering general-domain questions [36], in which the evidence is usually not given. Models that explicitly exploit an external corpus are referred as open-book models [26]. They typically index the corpus and then retrieve-and-read to extract the answer span from documents [8], [11], [14], [36]. Another recently proposed class of methods is closed-book QA models [26], [37]. They fine-tune pre-trained language models such as T5 [24] or BART [23] with QA pairs without access to any external knowledge or context.

6.0.0.3 Query driven multi-document summarization

(QFMD) aims to generate a summary according to the query and the provided relevant document(s) [38].  [39] incorporated query relevance into a pre-trained abstractive summarizer, while [17] and  [40] utilized QA models for sentence- or paragraph- level evidence ranking. [41] tried to improve the relevance of the summary by incorporating an answer relevance score for the source documents into the generation.

7 Conclusion↩︎

We propose a new end-to-end framework RBG that jointly models answer generation and machine reading to tackle the faithfulness issue in LFQA. Experiments on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method in comparison with strong baselines on automatic and human evaluation metrics. The detailed analysis further proves the competency of our method in generating fluent, relevant, and more faithful answers. We also propose to evaluate the factual correctness of LFQA model by answering questions of extractive QA tasks (e.g., Natural Questions), which may be helpful to evaluate the faithfulness of LFQA model efficiently.

8 Implementation details↩︎

We initialize our generation models with the pre-trained BART-large models [23], available in the HuggingFace⁷ Transformers library. Our reader models was initiated from Span-BERT(base&cased), from Facebook Github⁸, and further fine-tuned on MRQA datasets for 4 epochs, using the default fine-tuning configurations. Our RBG is trained using Adam [42] with a constant learning rate of 5e-5 and weight decay at 0.01. We train the model for 50k gradient steps, with a batch size of 4, using 8 Tesla V100 32Gb. We evaluate the models every 500 steps and select the best one on the validation set (1/8 of the evaluation set) based on the Rouge score. The maximum source document length is set to 300, and the target sequence length is set to 300. During inference, we use beam search with beam size of 4.

9 Document retriever model details↩︎

As the question/answers in LFQA may cover different domains and topics, we use a multi-task variant of DPR to guarantee the retrieval performance. The retriever is trained jointly on the union of all knowledge-intensive training data in KILT benchmark [12], including TrivaQA [21], kwiatkowski2019naturaluestion [4], HotpotQA [5], Fever [43], zsRE [44], AY2, T-REx [45] and WoW [46].

10 Human evaluation setup and analysis↩︎

10.0.0.1 Basic setup

As shown in Table 13, we sample 50 questions for each comparison and assign 3 annotators for each generated answer, which brings a workload of 450 judgments on model preference for each evaluation aspect. This process takes large amounts of energy and time considering the difficulty and challenges of factual-related annotation. We sample 10 questions from each of five development subsets corresponding to 5 levels of answer-passage overlap, which is a stratified sampling strategy. The answer position of each model is randomly shuffled to reduce the bias of position preference. 15 participants in our human evaluation are all researchers or students in computer science who speak and read English well.

10.0.0.2 Scoring setup

We ask each annotator to select a score from {1,2,3} for each generated answer in terms of three aspects: fluency, relevance and factual correctness. During scoring, the annotators are asked to preserve the relative better-or-not relationship between two models as much as possible. In particular, for the metric of factual correctness, the annotators check the correctness of all factual statements involved in a generated answer by referring to Wikipedia (EN), other web pages and the golden answer. The answer with significantly fewer factual errors will get a higher score on factual correctness. We show cases in Table 17 to demonstrate how the annotator evaluate three aspects in our experiment.

10.0.0.3 Statistical analysis

We present the agreement among annotators on model preference in Table 14 by calculating the Fleiss Kappa [47] as the inter-rater consistency. The RBG vs. FiD comparison achieves better annotation agreement than other two ablation comparisons, maybe because RBG integrates both two of our contributions to improve the answer quality. In the comparison of RBG vs. FiD, annotators achieve a “moderate agreement” on the aspect of correctness and “fair agreement” on relevance [48]. Annotators achieve best agreements on fluency in all comparisons. It’s more difficult to achieve a high degree of annotation consistency on factual correctness and relevance than fluency due to complicated facts involved in the generated answer. Therefore, we recommend taking preferred ratio as the core metric for factual-related evaluation following [10], [49]. We also present score variance of four models involved in human evaluation in Table 15. It’s natural that the fluency score has the smallest variance while the difficult-to-annotate correctness has largest variance.

11 Number of retrieved documents on Generation Quality↩︎

We also investigate the effects of number of retrieved documents \(k\), on the answer generation quality. As we can see in Table 16, the generation quality in terms of ROUGE-L and F1, do not further improve as the number of \(k\) increases, and the best performance are obtained when \(k=10\) in our case.

width=0.90,totalheight=,keepaspectratio

12 Case of zero-shot experiments on NQ and HotpotQA↩︎

In addition to automatic evaluation on the two datasets, we also show concrete examples to give a better understanding. As we can see in Table ¿tbl:tab:examples?,  BART tends to generate unfaithful answers as it lacks of external knowledge related to the specific question, and DPR + BART produces somewhat faithful but irrelevant answers, while  FiD performs much better but still with partially unfaithful generations. In comparison, our  RBG can generate more fluent, specific and factually accurate responses.

References↩︎