1 Introduction↩︎

Open-domain question answering (ODQA) is the task of answering a given question based on evidence from a large corpus of documents. In order to do so, a system generally first retrieves a smaller subset of documents (typically between 5-100) and then answers the question based on those documents. Previous research in ODQA has resulted in many well-curated datasets that evaluate a model’s ability to answer questions on a wide array of topics [1]–[4].

However, most internet users search across less-carefully curated sources, where malicious actors are able to affect articles that may be used by an ODQA system (1). Furthermore, even in curated knowledge sources like Wikipedia, we frequently see attacks (e.g. malicious edits/fake pages) that have even impacted production QA systems.²

Recent work has recognized the potential for bad actors to influence automated knowledge-intensive NLP systems that involve retrieval: [5] explored how poisoned information affects automated fact verification systems using sparse non-neural information retrieval systems, while [6]–[8] have studied the effect of knowledge conflicts and poisoning attacks on ODQA pipelines. All of these works have illustrated that poisoning attacks significantly decrease system performance, even when using state-of-the-art models; however, only [8] has even briefly considered the task of defending against poisoning attacks (which are becoming increasingly common, see Appendix 7 for real-life examples) and their proposed method, majority voting over different documents, provides only minor gains.

We seek to fill this gap by proposing a simple but effective defense against these attacks. Building on the intuition that information is usually available in multiple places and that it is unlikely that all sources (or pages) will be poisoned, we propose a novel query augmentation scheme to gather a larger set of diverse passages. We also propose a new confidence method to decide when to use the newly gathered contexts vs the original, which we call Confidence from Answer Redundancy (CAR).

Our proposed approach involves no gradient updates, can easily be applied to existing frameworks, and uses a simple resolution approach to arrive at the predicted answer. Together, our methods can provide gains of nearly 20 points in exact match, helping to reduce the negative effects of data poisoning and disinformation attacks on ODQA.

2 Experimental Details↩︎

We seek to mimic realistic disinformation attacks on a curated knowledge source; thus, for our experiments we use Wikipedia as the knowledge collection for both original and augmented queries, and simulate an attack on each question independently. We follow [5] and poison the entirety of each Wikipedia page that corresponds to each of the retrieved passages.³ We vary the amount of poisoned pages from 1 to 100.⁴ Note that we do not poison the entire corpus, as poisoning millions of pages is beyond the scope of common attacks.

2.1 Data↩︎

For our experiments we use Natural Questions (NQ) [1] and TriviaQA [2], two popular datasets for open-domain question answering. Furthermore, previous research on conflicts in ODQA has used these datasets in their experiments [6]. The Natural Question dataset was gathered by collecting real-user queries typed into Google Search, while TriviaQA was collected by scraping question and answer pairs from trivia websites, and then matching the answers to Wikipedia passages.

We simulate the data poisoning through the code available from [7], which introduced the problem in ODQA and has been used in subsequent work [6]. Their method uses the answers to the questions to suggest an entity of the same type, using SpaCY NER [9], which is then used to replace the correct answer in the text. This allows for entity substitutions that keep the semantic order of the context, such as replacing dates with dates, people with people, numbers with numbers, etc.

2.2 Models↩︎

We use two SOTA models: Fusion-in-Decoder (FiD) and Atlas. FiD is an encoder-decoder architecture that generates an answer by first retrieving and encoding \(N\) passages and then concatenating them and giving them to the decoder [10]. FiD uses DPR for retrieval [11]. Atlas [12] is currently the state-of-the-art model on Natural Questions and TriviaQA. This model also uses fusion in the decoder and has a T5 backbone, but uses Contriever [13] for retrieval and does joint end-to-end training. For information on hyperparameters see Appendix 8.

3 Proposed Method↩︎

3.1 Query Augmentation↩︎

We hypothesize that in cases of conflicting evidence in large corpora for factoid based questions, there will generally be more evidence for the correct answer than for incorrect ones. For example, imagine the question “Where was Barack Obama born?" with a corresponding attack to his Wikipedia page (see 1). As Wikipedia contains redundant information, alternate questions that find contexts on other pages (e.g. his mother Ann Dunham’s page) will still find the right answer.

To create these alternate questions that will still find the correct answer but with more diverse passages, we propose a query augmentation scheme that has similarities to query expansion in information retrieval (IR) [14]–[17]. We generate these new questions for each original question by prompting GPT-3. We use davinci-002 from [18], but one can alternatively use open-source language models for similar results: see 1 and Appendix 17 for results with Vicuna v1.5 (using Llama 2). These query augmentations are not necessarily paraphrases as they strive to be as different as possible while still leading to the correct answer. They are also not identical to classic query expansion from IR either, as they do not intend to solely broaden the query scope but rather to find diverse contexts from questions of any scope.

For each query in the dataset, we prompt GPT-3 with the following: "Write 10 new wildly diverse questions with different words that have the same answer as {Original Question}", thus generating approximately 10 augmented questions per original question (c.f. ¿tbl:table:datasets? for three examples of generations). Finally, we retrieve the 100 most relevant contexts for those augmented questions. Note that if searching with the augmented questions retrieves a passage from a Wikipedia page that was already poisoned from the initial set of 100 (see Section 2) we return the poisoned text following [5].

When we compare these newly retrieved passages to the passages retrieved by the original question we find that they do provide a more diverse set of passages. 2 shows the distribution of new passages retrieved, with almost all retrieving at least 20 or more new passages and a substantial amount having an entirely new set of 100 passages.

3.2 Confidence from Answer Redundancy↩︎

In order to identify the best augmented queries with their corresponding new passages, we derive a novel method, CAR, for measuring ODQA confidence. CAR measures how often the predicted answer string occurs in the retrieved contexts (usually 100 contexts). For example, if the predicted answer appears only once in all 100 contexts, this may mean that the retriever was not able to find many documents relevant to the query, especially as popular entities (those asked about in NQ and TriviaQA) are generally found in many articles. Overall, the more frequently the predicted answer appears in the contexts, the more likely that the retrieval was both successful and plentiful (e.g. redundant).

In practice, given a set of documents \(D\), we set a hyperparameter \(k\) to determine the cutoff for CAR (in practice we use \(k=5\), found by tuning on the dev set). If the model retrieves more than \(k\) unique passages that contain the predicted answer string, we classify the model as confident and vice versa. We use this as part of our resolution method below.

3.3 Answer Resolution↩︎

We use the following methods to combine (or not combine) the original question with the augmented questions, with shortened names in italics. Note that methods one through three are baselines for our newly proposed technique: (1) use the original question only, e.g. the “do-nothing" baseline (2) randomly pick one new augmented question (3) take a majority vote of the augmented question’s predictions (e.g. the method from [8]) or (4) use answer redundancy, described in the following paragraph. We also attempted several variants of these options that underperformed and are not included for clarity (Appendix 15).

Our proposed method for answer resolution, redundancy, uses CAR to effectively combine both the original question and the new augmented questions. We use CAR to decide whether to choose the original question’s prediction, and if not, use a majority vote over the predictions from the augmented questions that are confident (filtered using CAR). By doing so, we retain performance from the original question and passage set when confident, while otherwise backing off to the augmentation.

All methods except the baseline can use either the original (Original C) or new (New C) sets of passages as context and we show both options in our results. Further, majority vote and redundancy can choose between either the new or original questions during inference (we use original, after tuning, see 8 for more details).

4 Results↩︎

Figure 4 highlights our key findings using FiD and Atlas (for results in table form, see Appendix 20). Following [6], [7], all results are filtered by those that the model originally predicted correctly, thus making the original method have by definition 100% EM at the 0-article poisoning level. We show results in EM, as is typically done in previous work [10], [12], however, F1 results are nearly identical and can be found in Appendix 21.

As expected and shown in previous work [6], [8], we find that as the amount of poisoned data given to the model increases, performance decreases. We also find that resolution methods that use the new contexts (New C) outperform those that use the original contexts, confirming the intuition behind our proposed method of finding diverse new contexts (e.g. 55.9 vs 65.1 EM for EM at 1 article poisoned). Furthermore, we see that the redundancy resolution strategy outperforms all other strategies (including the only published baseline, majority voting from [8]), by up to 19.4% in the TQA setting (33.2% at 100 poisoned articles vs 13.8% baseline). Scores on NQ are lower than TQA, even with no poisoning, but still improve up to 14% EM using redundancy.

Overall, we see that our proposed redundancy method outperforms all other methods on both datasets, at every level of poisoning and especially so when using the newly retrieved contexts.

4.0.0.1 Can we use open-source LLMs as the query augmentation model?

We replace GPT-3 with Vicuna v1.5 (using Llama 2) and repeat the experiments with FiD. The results are shown in 1 for NQ and in Appendix 17 in figure form. We see that Vicuna performs similar to GPT-3, in some cases even outperforming it. Thus, we see that our approach works with both open and closed-source models.

4.0.0.2 How many augmented questions are needed for our approach to work well?

To answer this, we show Figure 5 with the overall trend showing that as the number of augmented queries increases, so does the score. Furthermore, it shows that even one augmented query has gains over the baseline method, allowing for a more compute efficient method at the expensive of several points of performance. More computational analysis of our methods is in Appendix 16.

4.0.0.3 Why is performance not 0% at 100 poisoned documents?

We also explore why performance is non-zero when the number of poisoned articles is equal to the number of contexts the model receives. We manually annotated 20 examples on TriviaQA that FiD got correct at the 100-article poisoning setting. We found that it is due to the model using its parametric knowledge to correctly answer (65% of the time), as the correct answer was not present in any of the input documents, or due to answer aliases (35%) that were not part of the answer set. Examples of cases can be found in Appendix 12.

5 Conclusion↩︎

Our work defends against data poisoning attacks in open-domain question answering through two novel methods: (1) the use of query augmentation to find diverse passages that still correctly answer the question and (2) the use of answer redundancy as a strategy for model confidence in its prediction. Our proposed methods do not involve any gradient updates and provide a significant performance improvement. Thus, our work shows the effect of data poisoning on state-of-the-art open-domain question-answering systems and provides a way to improve poisoned performance by almost 20 points in exact match. We hope that this work encourages future work in defending against poisoning attacks.

6 Limitations↩︎

Our work focuses on the TriviaQA and Natural Questions benchmarks, which include questions about popular entities in Wikipedia. As discussed in Appendix 7, our approach simulates real-world common attacks which are the most frequent type of attacks. However, for entities that appear less often in the knowledge source (and are less likely to be attacked), our approach will not be as effective.

We leave attacks on less-popularity entities to future work, as we focus on the most frequent and higher impact attacks, while also using datasets that are standard in existing literature, e.g. Natural Questions and TriviaQA.

Our work shows the impact that disinformation attacks could have on Wikipedia and provides an initial attempt to help remedy those attacks. We note that our strategy does not have perfect accuracy and is still susceptible to attacks, e.g. if there is no correct information in any context to be found, it will be very difficult for ODQA systems to give the correct answer. We welcome additional research to improve the resistance of ODQA systems to disinformation attacks and will open-source our code and data to help others make progress in this area (including results from GPT-3).

Acknowledgements↩︎

OW and NW are supported by the National Science Foundation Graduate Research Fellowship Program.

7 Realism of Proposed Setting↩︎

We focus on data poisoning attacks to high to medium popularity entities, as included in TriviaQA and Natural Questions. But are such attacks realistic, and have they happened before?

Due to the way that search engines work, any data poisoning done at the time of indexing is able to effect system performance until the data is re-indexed. Thus, if one were to change a Wikipedia page (or a personal website that was included in an index) and that change was indexed, the data would be poisoned until re-indexing.

As the people directing disinformation campaigns are likely motivated to attack well-known entities rather than unknown entities (for political or economic reasons), our proposed setting of defending against popular entities is well-motivated and is a serious problem affecting current production systems today. (see “Vandalism on Wikipedia"). There have even been many high profile attacks on popular entities that have been reflected in production systems (this is not hypothetical). One such entity who has been frequently attacked is Donald Trump, whose Wikipedia page was changed to include critical text and inappropriate images, returned by Siri to real user queries. The Wikipedia page on vandalism includes many such examples of famous politicians, musicians, athletes and other popular entities being subject to attacks on Wikipedia that were propagated to users via Google or via various news outlets (e.g. Thomas Edison’s page describing him as a "douchebag", famed swimmer Chad Le Clos’s page edited to say he literally "died at the hands of Michael Phelps" when losing a race, etc.).

These attacks are just the tip of the iceberg for disinformation, as attacks to Wikipedia are the easiest to trace. Since production search engines index the web and then answer questions about them, any personal or company page can be used for attacks and are much less traceable (see this humorous attack to Bing Chat about Mark Reidl, done in jest to illustrate the potential for attacks).

8 Hyperparameters↩︎

For all our experiments we use a cluster of V100 GPUs, with each job running on a 4 to 8 GPU node and taking approximately 12-24 hours depending on the model. We use the models as provided by the original authors with default retriever hyperparameters. We use Atlas’s XL version. We use Vicuna v1.5 on 1 A100 40GB GPU for 3 hours for the open-source experiment in Appendix 17.

Following previous work in question answering, we report Exact Match (EM) in all of our experiments. We take the data from [7] and split into equal dev and test sets. We use the dev set to tune the CAR method’s hyperparameters and use \(K=5\) for our experiments.

Along with the New C and Original C options, the redundancy and majority vote methods also have hyperparameters for using either the augmented questions or the original question for the final prediction (after generating and searching for new contexts). Our tuning on the dev set indicated that using the original question and the new contexts from searching with the augmented question provides slightly higher performance (which makes sense, since the original question is the most important to answer). Thus, the process is first generating augmented questions, then searching with those, then doing inference with the original questions and the newly retrieved contexts (and finally CAR, if using the redundancy method).

9 More Related Work↩︎

As a larger section of related work did not have space in the main paper, we include more related work here.

9.0.0.1 Data Poisoning Attacks

Data poisoning attacks in NLP have a long history, with several prominent works appearing in recent years including [19]–[21] focusing on various NLP tasks such as machine translation, language modeling, etc. However, in the question answering space most adversarial work is focused on making harder questions, rather than simulating a real attack [22], [23]. Those that do focus on human attacks focus on the machine reading setting [24].

As mentioned in the main text, a nascent line of work has focused on knowledge conflicts in open-domain question answering [6], [7]. These works’ main motivation is to explore how ODQA models operate under the influence of conflicts, mostly in the context of non-parametric vs parametric knowledge. We extend these works by using their methods as simulated attacks on a knowledge source and proposing efforts to defend against these attacks.

9.0.0.2 Open-Domain Question Answering

Our work builds off of recent advances in ODQA, such as using Fusion-in-Decoder [10]. Other work such as DPR [11] showed promising results but has been improved upon by models that encode a large number of contexts into a single reader model. We note there exists an emerging line of work that uses LLMs for ODQA without using a retriever [25], [26], however, our approach relies on the redundancy in the retriever to defend against disinformation attacks; we leave exploring other settings to future work.

9.0.0.3 Query Augmentation

Query augmentation is a traditional information retrieval technique to augment a given query to find a better set of documents [14], [15]. In classical terms, the strategy is usually to expand the query, spelling out acronyms or adding synonyms. Recently, work has begun to use neural models to generate these expansions [17], [27]–[29], despite retriever’s lack of understanding of some terms [30]. In our work, we use a similar strategy to create new queries that will gather a diverse set of passages.

9.0.0.4 Confidence and Calibration of QA

Many works have focused on calibrating QA models so that they correctly reflect probabilities that equal their actual correct answer rate [31]–[34]. Our proposed confidence method is similar in that it measures when the model will be more likely to be correct, however, it does not do calibration in the sense of calibrated probabilities, instead giving a single value of “confident" or”not confident."

Answer redundancy has been studied before in other NLP contexts, such as [35] in the information extraction task. We apply a similar intuition of answer redundancy to the novel context of document inputs for open-domain question answering.

10 Alternate Poisoning Attacks↩︎

In the main section of the paper, we used poisoning attacks based on articles. However, one could attack a system directly by going after its retrieved results, either randomly poisoning \(N\)% or poisoning the top \(N\)%. We note that we tried both settings and found similar results, with the main difference that model performance declines slower (as randomly picking contexts to poison is less likely to impact the model until higher levels of poisoning).

11 Number of Augmented Queries↩︎

In Figure 5 we see the results for how the number of augmented queries affects performance. Overall, one query provides strong performance (above the baseline original performance at 17.5% EM) and multiple questions continue to show gains. We note that this figure uses Natural Questions and the 5-article poisoning setting with FiD, but other settings showed roughly the same results. As including more queries only seems to increase the score, it’s possible that generating more than 10 augmented queries would show even better results.

12 Why is performance not 0% at 100 poisoned documents?↩︎

To explore this question, we conducted a manual analysis of 20 pairs of question and 100 document passages on TriviaQA using FiD. We found that 65% of cases were due to the model’s parametric knowledge, as there was no such answer string in the input text. However, the answer was generally very obvious, like “In which country is Dubrovnik?" which is generally easier for the model to predict (e.g. ”Croatia"). In 35% of cases there was a missing alias from the answer string set, such as “What dance craze was named after a city in South Carolina?" with an answer string set of”Charleston rhythm", “Charleston (dance)",”Charleston (dance move)", “Charleston dance", and”The Charleston". FiD predicted “Charleston" from the text, since”Charleston" was not in the answer string set so it was not poisoned in the text. Future work on data poisoning could improve on this category by developing more robust poisoning techniques to aliases.

13 Number of Poisoned Passages↩︎

In our experiments, we poisoned at the article level, as an attacker might do to a specific entity. However, each Wikipedia article corresponds to more than one passage which are what is used for retrieval. When we poison at the article level we poison all passages in the article, so oftentimes many passages are poisoned even when poisoning one article. Furthermore, passages can only be poisoned if the answer is present in the passage (and thus available to be replaced).

How many passages are poisoned at each article-poisoning level? 7 answers this question and shows the number of poisoned passages vs the article-poisoning level. We find that the number of articles poisoned is much higher on TriviaQA, which means that TriviaQA had a much higher number of passages with the answer to begin with.

14 Confidence from Answer Redundancy↩︎

We compare the confidence from answer redundancy (CAR) to the actual exact match score (using the 1-article poisoning setting on Natural Questions) to show the effectiveness of this heuristic. In Figure 6 we see the large gap between queries that do not meet CAR and those that do (around 65% absolute exact match). Error bars indicate a 95% confidence interval.

15 Alternate Answer Resolution Strategies↩︎

Due to space and clarity for figures, we do not include all possible answer resolution strategies in the main figures. Some potential alternate resolution stratgies we tried included:

• Using the new augmented questions with CAR alone, without using them as a backup for the original question. This is equivalent to the majority vote method but using CAR to filter the question that get to vote. Although this method performed well it consistently underperformed our redundancy method and thus we do not include it

• Using a majority vote over both the original question’s prediction and and augmented question’s predictions. This performed nearly identically to the standard majority vote method, hence we leave it out for clarity.

• Taking the difference between the the CAR values of the original and augmented questions. This again greatly underperformed the redundancy method and is therefore not included

We encourage others who have new ideas for answer resolution strategies to use our code as a start to develop their method.

16 Compute Cost of our Proposed Method↩︎

Our method requires the addition of 1 call to GPT-3’s API (or the use of Llama 2, see Appendix 17) which generates the N augmented questions with one call, and N instances of additional search and inferences of the ODQA model.

16.0.0.1 Augmented Query Generation

As GPT-3 and other large language models become more available and cheaper (as they have already started to be, with many works speeding up inference using models like Llama) this will become cheaper to do with time. The cost of one query to ChatGPT for example (of which our method uses approximately 100 tokens) is roughly $0.0002 USD, which is remarkably affordable.

16.0.0.2 Retrieval

The retrieval computation cost is negligible in comparison, as modern retrieval takes milliseconds with different indexing and quantitation costs.

16.0.0.3 Retrieval-Augmented Generation

The other major computational cost of our method is the retrieval augmented generation inference. However, as mentioned 11, this can be reduced to only one inference and still see large gains.

16.0.0.4 Overall

Our method is bounded by the call to a language model for generation of the augmented questions and by 1 or more calls to a retrieval augmented model. However, these costs are still cheap and used frequently: e.g. one call at inference time to a model like ChatGPT is relatively minor and is done by a large number of research and industry applications. Further, as time progresses these calls will get cheaper and quicker.

17 Open-Source LM Generation↩︎

We also show that our method can use an open-source language model like Llama 2 [36] and work similarly. In Tables 6 and 7 and Figure 8 we show results for FiD on TriviaQA and Natural Questions using Llama 2 generations from Vicuna v1.5 7B [37].

We find that results are comparable to those with GPT-3, and in some cases even slightly outperforms the comparable GPT-3 version. For example, in the 1% article poisoning case Vicuna v1.5 scores 91.5% with redundancy while GPT-3 score 90.8%, Table 6 vs Table 4.

Overall, we see that open-source models can comparably be used in place of closed models like GPT-3 for this task.

18 Relation to Robustness under Shift↩︎

One common type of evaluation in ODQA is its ability to withstand adversarial attacks that test robustness (such as paraphrases or distracting sentences that are superfluous), e.g. [38]–[40]. However, our work focuses on intentional disinformation attacks, where the facts in the documents have been changed. Although these two evaluation settings have surface similarities, the crucial difference is that when adding distracting sentences or paraphrases the core facts still exist in the documents, with no contradictions between documents. In our experimental setting however, some or all the needed facts have been deliberately changed. Thus, techniques for handling distracting sentences (such as improved filtered) or paraphrases (training with paraphrases) are not relevant to our setting, as the underlying problem requires new solutions that can deal with incorrect and/or conflicting facts.

19 Case Studies and Examples↩︎

We show two case studies here that illustrate how our method works. We randomly select an instance where our method outperforms the baseline approach.

19.1 Case 1↩︎

We see that the new query successfully re-weighted terms such that it was able to rank the new document #1 and get the correct answer. Note that the New Rank 1 document was from the “Walking the Amazon" page, which was not poisoned, while the Original Rank 1 document was from the poisoned Ed Stafford page. Note that both of these documents were in the top three for each query, but the relative position change was able to help the retrieval-augmented model find the correct answer.

19.2 Case 2↩︎

In this example we see that searching for “information" allowed the query to find the Wikipedia page which described Turner’s book about Van Morrison, which contained the correct answer as opposed to the poisoned content. Having the correct answer in the top context allowed the model to correctly choose it over the disinformation.

19.3 Case Study Conclusion↩︎

From a qualitative analysis, we find that our method predicted correctly on some instances because it changed the relative position of the retrieved documents, due to word changes in the query. Others were correctly predicted by surfacing new information that was not in the top ranked contexts before. Our method thus makes relatively simple changes that, overall, provides strong gains and is easy to implement with any retrieval augmented system.

20 Table Versions of Plots↩︎

We also show Table versions of the main plots for ease of viewing: Table 4 for TQA and FID, Table 5 for NQ and FiD, Table 2 for NQ and Atlas, and Table 3 for TQA and Atlas.

21 F1 vs EM↩︎

It is common in previous work on NQ and TQA to report only EM. However, we also include tables with F1 to illustrate that the results are the same, just slightly higher. As the differences are very minor and the trends remain the same, we three examples using TQA: Llama with FiD (Table 10) and GPT-3 for Atlas and FiD in Tables 8 and 9.

References↩︎