Abstract

Warning: This paper contains examples that may be offensive or upsetting.

While large neural-based conversational models have become increasingly proficient dialogue agents, recent work has highlighted safety issues with these systems. For example, these systems can be goaded into generating toxic content, often perpetuating social biases or stereotypes. We investigate a retrieval-based approach for reducing bias and toxicity in responses from chatbots. It uses in-context learning to steer a model towards safer generations. Concretely, to generate a response to an unsafe dialogue context, we retrieve demonstrations of safe responses to similar dialogue contexts. We find our method performs competitively with existing approaches to dialogue safety without requiring training. We also show, using automatic and human evaluation, that reductions in toxicity obtained using our approach are not at the cost engagingness or coherency. Finally, we note our method can be used in compliment to existing dialogue safety approaches, such as RLHF.

1 Introduction↩︎

Large neural-based language models are becoming increasingly proficient dialogue agents [1]–[4]. While these models are capable of engaging in interesting and coherent dialogue, recent work has shown these systems are prone to generating unsafe content ([5]–[7]; inter alia). For example, these models often exhibit social biases [8], [9] and inappropriately align themselves with offensive statements during conversation [10]. As these models are used interactively, ensuring they generate safe and sensible responses is critical.

Figure 1: Our approach to safe response generation from dialogue systems. Given a target context and a retriever (e.g., BM25), we retrieve safety demonstrations. The retrieved demonstrations are then used in-context to condition generation.

Two methods have seen widespread adoption for addressing these safety issues. Reinforcement Learning from Human Feedback (RLHF; [11]–[13]) has emerged as a training-based procedure for reducing the harmfulness of language models. RLHF uses human preference data to attempt to align a model’s responses with human values. In conjunction with RLHF, safety filters [5], [14] can be used during inference to block unsafe inputs to the model and filter unsafe generations from the model.

While both of these methods are effective in reducing toxic generation from dialogue systems [15], they are not easily adaptable to new unsafe inputs. For example, consider uncovering a new class of inputs which elicit unsafe responses from a model after deployment. Correcting this with the methods described above requires additional data and additional training. This can become cumbersome if several vulnerabilities are uncovered in a model. Ideally, we want to be able to efficiently correct undesirable behaviours in a dialogue system post-deployment.

In this paper, we investigate a retrieval-based approach for dialogue safety. While many safety issues exist within current dialogue systems, we focus specifically on reducing response toxicity.² Following the taxonomy introduced by [16], our work investigates reducing the Instigator and Yea-Sayer effects in dialogue systems. Given an unsafe dialogue context, we propose retrieving demonstrations of exemplary safe responses to similar dialogue contexts. For example (see Figure 1), given a dialogue context containing sexism, we retrieve demonstrations of safe responses from other dialogue contexts containing sexism. These retrieved demonstrations can then be used in-context to steer a model towards generating a desirable response.

Concretely, our work aims to answer the following research questions:

Do in-context safety demonstrations improve response safeness from dialogue systems?

How does in-context learning compare to popular methods for safe response generation?

To answer [q:demonstration95effective] (§5), we evaluate our approach in three families of models: OPT [17], LLaMA [4], and Vicuna [18]. We focus our efforts on the openly available OPT models. Using both automatic (§5.1) and human (§5.3) evaluation, we find our approach reduces toxicity without degrading general response quality. To answer [q:baselines] (§6), we compare our method to three popular baselines for safe response generation. We find our approach performs competitively with these baselines without requiring any training. In addition to the above research questions, we also present an extensive set of ablations in 12. For example, we investigate the effectiveness of our approach with limited amounts of safety demonstrations.

2 Related Work↩︎

Our work extends research on in-context learning and dialogue safety. Below, we discuss methods proposed for creating safer dialogue systems and contrast them with our own. We also describe related work on in-context learning.

2.0.0.1 Safety Filters.

One popular approach for creating safer dialogue systems involves using safety filters [5], [14]. These filters are typically used in three ways: 1) To filter unsafe content from a model’s training corpus [19], [20]; 2) To block unsafe inputs to a model [14]; and 3) To filter unsafe generations from a model [5]. These filters require large collections of dialogues with utterances labelled as safe or unsafe to train [9], [21]–[23]. In contrast to our approach, these filters cannot easily be adapted to new unsafe inputs or new unsafe responses—each undesirable behaviour you wish to mitigate must be reflected in the safety filter’s training corpus.

2.0.0.2 Safe Response Fine-Tuning.

Another approach for creating safer dialogue systems involves training on exemplary safe responses [24], [25]. Several datasets have been released that contain prosocial or safe responses. [24] collected SaFeRDialogues, an augmented variant of Bot-Adversarial Dialogue [22] that contains safe feedback and recovery responses. [25] introduced ProsocialDialog, a dialogue dataset containing prosocial responses grounded in social rules-of-thumb. A recent line of work has shown that training models on refinements of their own responses can reduce harmfulness [26], [27]. [28] recently showed that fine-tuning on even a small number of high-quality responses can give large safety improvements.

2.0.0.3 Reinforcement Learning from Human Feedback.

Reinforcement Learning from Human Feedback (RLHF) has emerged as an effective approach for creating safer language models [11]–[13], [15], [27], [29], [30]. In general, RLHF leverages human preference data to align language models with human values. Our approach is complimentary to RLHF. In our work, we show that Vicuna [18], a model derived from ChatGPT [30], can obtain reduced toxicity using retrieval and in-context learning.

2.0.0.4 Safe Decoding Procedures.

Several decoding procedures have been proposed for safe generation from language models. [31] proposed using a language model’s implicit knowledge of toxic content to detoxify generation. [32] and [8] investigated using control signals to condition generation from language models. Other work has investigated using classifiers to guide generation [33], [34]. Finally, [35] proposed a product-of-experts-based procedure for detoxified generation. As with our approach, most of these procedures do not require training but involve additional computation at inference-time.

2.0.0.5 In-Context Learning.

In-context learning [36]–[38] has proven effective in many NLP tasks [39]–[41]. To the best of our knowledge, we perform the first large-scale evaluation of in-context learning for dialogue safety. The work of [42] is most related to our own. While they investigate in-context learning for alignment, they do not investigate retrieving relevant demonstrations. Recent work has also studied fundamental questions about in-context learning. [43] investigated the impact of in-context demonstration order on performance. We find the order of in-context demonstrations does not impact response quality or safety. [44] demonstrated that retrieving in-context demonstrations based on semantic-similarity to the test query led to performance improvements on NLU benchmarks. We find retrieving demonstrations with high similarity to the dialogue context is useful for reducing response toxicity. Finally, [45] and [46] investigated methods for selecting in-context demonstrations. We also investigate different methods for selecting in-context demonstrations for dialogue safety.

3 Methodology↩︎

We investigate a retrieval-based approach for safe response generation from decoder-only Transformer [47] models. Concretely, we experiment with different sized OPT [17], LLaMA [4], and Vicuna [18] models. We experiment primarily with OPT models as the model code and weights are openly available however, we also highlight relevant LLaMA and Vicuna results (see 16 for complete results) throughout our work.

Henceforth, we refer to the dialogue context we want to generate a response to as the target context and the demonstrations of safe model behaviour as safety demonstrations. At a high-level, our approach consists of two steps: 1) We retrieve safety demonstrations based upon their similarity to the target context; and 2) We use the retrieved safety demonstrations in-context to condition generation. We describe these steps in detail below.

Figure 2: Prompt for response generation. Each prompt consists of the retrieved demonstrations and the target context. Each safety demonstration is separated by an empty line and the target context is separated from the safety demonstrations by an empty line.

3.0.0.1 1) Retrieving Safety Demonstrations.

We investigate three methods for selecting safety demonstrations for a target context: 1) Randomly selecting demonstrations; 2) Using BM25 [48] to select demonstrations; and 3) Using a SentenceTransformer [49].³ For each retriever, we use the target context as the query to select demonstrations. These safety demonstrations are entire conversations consisting of unsafe utterances and prosocial responses. Throughout our work, we refer to our SentenceTransformer retriever as a “dense” retriever.

3.0.0.2 2) Response Generation.

Once safety demonstrations have been selected, we use them in-context to condition generation. Concretely, given \(K\) safety demonstrations and a target context, we use the prompt format shown in 2. We prepend each conversation in the input with “A conversation between two persons” to condition for dialogue. Demonstrations are placed in the prompt in descending order based upon their retrieval scores. More plainly, the top-ranked demonstration is placed at the start of the input. The target context is placed at the end of the input. We mark the speaker of each utterance (Person 1 or Person 2) and provide a trailing annotation at the end of the prompt for the speaker we want to generate a response for (in 2, this is Person 2).

4 Experimental Setup↩︎

Below, we describe the dialogue datasets used in this work. In addition, we discuss how we evaluate response safeness and relevance (i.e., quality).

Figure 3: Safety classifier results for ProsocialDialog (in-domain) and DiaSafety (out-of-domain) for responses generated with different retrievers and numbers of safety demonstrations. “Dense” denotes our SentenceTransformer retriever. We report the mean and standard deviations across three seeds.

4.1 Dialogue Datasets↩︎

We experiment with three dialogue datasets in this work. Conversations from these datasets are used either as inputs for response generation or as safety demonstrations. We use a maximum of two conversation turns in both our target contexts and safety demonstrations. We describe each dataset below and defer readers to 19 for additional details.

4.1.0.1 ProsocialDialog [25].

ProsocialDialog contains unsafe utterances with prosocial responses. We use the 42K conversations from the training split of ProsocialDialog as our source of safety demonstrations for all our experiments. We also experiment with generating responses to the \(7\)K conversations from the validation split of ProsocialDialog.⁴

4.1.0.2 DiaSafety [23].

DiaSafety is a collection of adversarial utterances which can illicit unsafe responses from conversational models. We experiment with generating responses to the 1K conversations from the validation set of DiaSafety. We use DiaSafety to evaluate response generation to unsafe inputs. We note each target context from DiaSafety consists of a single utterance.

4.1.0.3 Commonsense-Dialogues [51].

Commonsense-Dialogues is a collection of conversations grounded in social contexts. We experiment with generating responses to the 1K conversations from the validation set of Commonsense-Dialogues. We use Commonsense-Dialogues to evaluate response generation to safe inputs.

4.2 Automatic Safety Evaluation↩︎

We use three methods for automatically evaluating response safeness: a safety classifier, PerspectiveAPI,⁵ and an offensive word list. For each method, we report the percentage of responses predicted safe. We detail each method below.

4.2.0.1 Classifier.

We use the 2.7B parameter Transformer classifier from [22] to evaluate response safety. This classifier is trained on Wikipedia Toxic Comments [52], Build-it Break-it Fix-it [21], and Bot-Adversarial Dialogue [22]. For a given target context and response, the classifier assigns a probability indicating whether the response is safe. We use the same threshold as [22] to flag responses as unsafe.

4.2.0.2 Perspective.

We use PerspectiveAPI to quantify response toxicity. PerspectiveAPI assigns a probability indicating whether a response contains toxicity. Following previous work [31], [53], we use a threshold of \(0.5\) to flag responses as unsafe. We note PerspectiveAPI is an utterance-level toxicity detector—it does not account for context when scoring toxicity. As reproducibility concerns have been raised about PerspectiveAPI [54], we use Classifier as our primary tool for evaluating safety.

4.2.0.3 Word List.

As a crude measure of response safeness, we use the offensive word list provided by [6]. We check for the presence of these words in all of our responses. While this method can falsely flag innocuous responses, it may provide a noisy signal about blatant safety failures.

4.3 Automatic Relevance Evaluation↩︎

We use five open-text generation metrics to evaluate response relevance: Rouge-1 [55], F1, Meteor [56], Deb [57], and Self-Bleu [58]. For our Deb metric, we report the percentage of responses predicted to entail their respective target contexts. For our Self-Bleu metric, we randomly sample \(128\) responses from each model to compute the score. In addition to the above metrics, we also use GPT-3.5-Turbo to conduct head-to-head comparisons between responses (Llm-Eval).⁶ We follow the setup of [59] and prompt GPT-3.5-Turbo to select which of a pair of responses is more “helpful,” “relevant,” “detailed,” and “respectful.” See 17 for details.

5 Do In-Context Safety Demonstrations Improve Response Safeness?↩︎

We first investigate if using in-context safety demonstrations can reduce toxicity from dialogue systems ([q:demonstration95effective]). We also evaluate the impact of using safety demonstrations on response quality. Importantly, we want to ensure safety improvements are not at the cost of interestingness, engagingness, or coherency. For example, while a dialogue system that apologizes constantly may be safe, it is not particularly interesting or engaging. This is usually dubbed as the harmless vs. helpful tradeoff [15].

To evaluate our method, we generate responses to ProsocialDialog, DiaSafety, and Commonsense-Dialogues. We discuss our results below.

Table 1: Automatic evaluation of OPT-30B responses to ProsocialDialog. \(K\) denotes the number of demonstrations used for generation. We **bold** the best value for each metric. We report the mean and standard deviation across three seeds.
Model	\(K\)	Rouge-1 \(\uparrow\)	Meteor \(\uparrow\)	Deb \(\uparrow\)	Self-Bleu \(\downarrow\)	F1 \(\uparrow\)	Avg. Length
OPT-30B	\(0\)	\(19.21 \pm 0.05\)	\(13.05 \pm 0.03\)	\(93.33 \pm 0.05\)	\(\mathbf{5.55} \pm 1.10\)	\(17.10 \pm 0.05\)	\(22.22 \pm 0.09\)
+ Random	\(10\)	\(22.62 \pm 0.04\)	\(16.68 \pm 0.09\)	\(95.26 \pm 0.08\)	\(13.10 \pm 1.59\)	\(20.78 \pm 0.04\)	\(26.64 \pm 0.02\)
+ BM25	\(10\)	\(23.51 \pm 0.22\)	\(17.48 \pm 0.16\)	\(95.15 \pm 0.22\)	\(13.36 \pm 1.76\)	\(21.86 \pm 0.47\)	\(25.11 \pm 2.24\)
+ Dense	\(10\)	\(\mathbf{24.81} \pm 0.07\)	\(\mathbf{19.41} \pm 0.08\)	\(\mathbf{95.98} \pm 0.01\)	\(12.26 \pm 0.83\)	\(\mathbf{23.04} \pm 0.10\)	\(30.64 \pm 0.10\)

5.1 Automatic Safety Results↩︎

We first discuss our automatic safety results. Here, we present Classifier results. We defer readers to 6 for other automatic safety results.

5.1.0.1 ProsocialDialog Results.

In 3, we present results for ProsocialDialog. We observe a strong correlation between the number of demonstrations and the percentage of safe responses. This trend exists across all model sizes and retrieval methods. Amongst the retrievers, we note that BM25 and the dense retriever both outperform random retrieval. This highlights that selecting demonstrations similar to the target context helps improve safety. Generally, we find performance tends to increase with model size.

5.1.0.2 DiaSafety Results.

In 3, we present results for DiaSafety. We find DiaSafety responses are less safe than ProsocialDialog responses. For example, OPT-6.7B with zero demonstrations generates \(62.86\)% safe responses to ProsocialDialog and \(57.79\)% safe responses to DiaSafety. As with ProsocialDialog, we observe a correlation between the number of demonstrations and the percentage of safe responses. In contrast to ProsocialDialog, we observe greater variance in the results. For instance, with DiaSafety, BM25 does not clearly outperform random retrieval. This variance may be due to only having a single utterance to use for retrieval. We observed similar trends in LLaMA and Vicuna both DiaSafety and ProsocialDialog.

5.1.0.3 Commonsense-Dialogues Results.

We find our method effective for generating responses to safe inputs as well. Here, we note that all of our models generated a high proportion of safe responses without safety demonstrations. For example, OPT-6.7B generated \(83.20\)% safe responses to Commonsense-Dialogues. However, we found all models obtained increased scores when provided with demonstrations (e.g., OPT-6.7B generated \(89.86\%\) safe responses when provided with ten demonstrations). See 13 for additional details.

5.2 Automatic Relevance Results↩︎

We now discuss our automatic relevance results. Since DiaSafety does not contain reference safe responses, we present results for ProsocialDialog and Commonsense-Dialogues.

Table 2: Head-to-head comparison human evaluation results. We report the percentage win rates. We **bold** the model with the highest win rate for each comparision.
Model	Prosocial	Engage	Coherent
ProsocialDialog
OPT-30B	\(8.67\)	\(\mathbf{45.78}\)	\(14.22\)
Tie	\(20.22\)	\(18.89\)	\(31.78\)
OPT-30B + Dense	\(\mathbf{71.11}\)	\(35.33\)	\(\mathbf{54.00}\)
BlenderBot3-30B	\(4.44\)	\(23.33\)	\(5.56\)
Tie	\(16.67\)	\(20.22\)	\(38.22\)
OPT-30B + Dense	\(\mathbf{78.89}\)	\(\mathbf{56.44}\)	\(\mathbf{56.22}\)
DiaSafety
OPT-30B	\(14.44\)	\(26.67\)	\(16.67\)
Tie	\(28.89\)	\(19.11\)	\(33.11\)
OPT-30B + Dense	\(\mathbf{56.67}\)	\(\mathbf{54.22}\)	\(\mathbf{50.22}\)
BlenderBot3-30B	\(11.33\)	\(21.56\)	\(11.11\)
Tie	\(27.11\)	\(23.56\)	\(42.89\)
OPT-30B + Dense	\(\mathbf{61.56}\)	\(\mathbf{54.89}\)	\(\mathbf{46.00}\)

5.2.0.1 ProsocialDialog Results.

We report results for ProsocialDialog and OPT-30B in 1. We observe a correlation between the number of demonstrations and performance on all of the metrics. However, we note that the average response length is correlated with the number of demonstrations—the responses generated with the largest number of demonstrations also have the longest responses, on average. We also highlight the decreased response diversity when using our method.

5.2.0.2 Commonsense-Dialogues Results.

We find response quality to safe inputs is not degraded when using safety demonstrations. In general, we observed a slight increase in most automatic metrics when using demonstrations. For example, OPT-13B obtains an F1 score of \(11.01\) without safety demonstrations and an F1 score of \(11.60\) with ten demonstrations (see 13). These results suggest that using safety demonstrations, even when they are not required, does not adversely affect quality.

5.3 Human Evaluation↩︎

We conduct human evaluation of the quality and safety of generated responses. Below, we describe our setup and results.

Table 3: Automatic evaluation of responses to DiaSafety. We use ten safety demonstrations for OPT-6.7B + Dense. We **bold** the best value for each metric. For Llm-Eval, we report the average win rate across all OPT models. With the exception of Llm-Eval, we report the mean and standard deviations across three seeds for each metric.
Model	Safety			Relevance
2-4 (l)5-7	Classifier \(\uparrow\)	Perspective \(\uparrow\)	Word List \(\uparrow\)	Self-Bleu \(\downarrow\)	Deb \(\uparrow\)	Llm-Eval \(\uparrow\)
OPT-6.7B	\(57.79 \pm 0.79\)	\(74.35 \pm 1.97\)	\(86.66 \pm 2.04\)	\(7.17 \pm 0.95\)	\(\mathbf{87.96} \pm 0.72\)	\(42.87\)
+ Dense	\(77.57 \pm 0.57\)	\(89.33 \pm 0.09\)	\(94.22 \pm 0.65\)	\(12.48 \pm 0.96\)	\(87.03 \pm 0.85\)	\(\mathbf{69.14}\)
+ Fine-Tune	\(74.23 \pm 0.47\)	\(94.53 \pm 1.10\)	\(97.66 \pm 0.19\)	\(4.29 \pm 1.96\)	\(73.50 \pm 1.18\)	\(41.89\)
+ Self-Debias	\(67.15 \pm 0.50\)	\(85.29 \pm 2.15\)	\(91.98 \pm 0.18\)	\(\mathbf{3.03} \pm 1.45\)	\(85.38 \pm 2.05\)	\(51.75\)
+ Director	\(\mathbf{79.82} \pm 1.15\)	\(\mathbf{97.53} \pm 0.60\)	\(\mathbf{98.54} \pm 0.16\)	\(7.96 \pm 3.93\)	\(72.01 \pm 0.55\)	\(42.96\)

5.3.0.1 Experimental Setup.

We carry out head-to-head comparisons of responses from three dialogue models: OPT-30B, OPT-30B with ten safety demonstrations selected using a dense retriever, and BlenderBot3-30B [14].⁷ We use BlenderBot3-30B as a baseline for comparison to a strong conversational model. Importantly, BlenderBot3 was fine-tuned on SaFeRDialogues [24]—a dialogue dataset containing safe responses to unsafe utterances. Following [25], we task annotators with comparing the prosociality, engagingness, and coherency of responses from two models. We allow annotators to score a pair of responses as a tie if neither response is preferable. We compare responses to \(150\) randomly selected examples from ProsocialDialog and DiaSafety. For each example, we collect preferences from three annotators. For additional details on our human evaluation setup, we refer readers to 18.

5.3.0.2 Results.

We report majority vote win rates for each quality in 2. In general, we find that the model using safety demonstrations generates the most prosocial, engaging, and coherent responses. We find our model outperforms BlenderBot3-30B on ProsocialDialog and DiaSafety in each quality. Our ProsocialDialog results are not surprising as BlenderBot3-30B is not trained on ProsocialDialog (whereas our model uses demonstrations from the training split). We find our DiaSafety results more encouraging as they more closely match a realistic setting where the available demonstrations may not be similar to the target context.

6 How Does In-Context Learning Compare to Popular Safe Response Generation Methods?↩︎

We now compare our approach to three popular safe response generation methods ([q:baselines]). Below, we describe each method.⁸

6.0.0.1 Safe Response Fine-Tuning.

We fine-tune on safe responses from ProsocialDialog and SaFeRDialogues [24]. [24] found that fine-tuning solely on SaFeRDialogues results in overly apologetic responses. Because of this, we also fine-tune on three other dialogue datasets: ConvAI2 [60], Empathetic Dialogues [61], and Blended Skill Talk [62].

6.0.0.2 Director [34].

Director is a guided generation which uses a safety classifier to decrease the probability of toxic tokens during generation. We fine-tune with Director following the setup of [34]. Concretely, we use Wikipedia Toxic Comments [52] and the safety data from [21] to fine-tune our models.

6.0.0.3 Self-Debias [31].

Self-Debias is a contrastive decoding procedure that leverages a model’s implicit knowledge of toxicity to debias generation. [63] empirically demonstrated Self-Debias can be used to mitigate multiple social biases during generation. We use the prompts provided by [31] for detoxifying generation.

6.1 Results↩︎

6.1.0.1 Automatic Safety Results.

In 3, we present automatic safety results for DiaSafety. In general, we find all methods increase response safety. In particular, we find Director performs best, obtaining the highest percentage of safe responses across all three safety metrics. Encouragingly, we find our in-context learning-based model performs only \(2.25\) points worse than Director for Classifier. We also note the relatively poor performance of our method on Perspective (compared to Director, for instance). We hypothesize this is because Perspective is an utterance-level safety detector. Since responses generated using our method tend to be more prosocial, they may be falsely flagged as unsafe when classified independent of the dialogue context.

Figure 4: Win rates for head-to-head comparisons amongst OPT-6.7B models. See 17 for results with Vicuna and LLaMA. We sort the models on the y-axis in descending order based upon their average win rate. “Dense” denotes OPT-6.7B with ten demonstrations selected using a dense retriever. “Fine-Tune” denotes OPT fine-tuned on safe responses.

Table 4: Sample responses from different OPT-6.7B models to DiaSafety target contexts.

6.1.0.2 Automatic Relevance Results.

In 3, we report automatic relevance results. For our GPT-3.5-Turbo-based response evaluation, we report the average win rate for each model (see 4 for individual win rates). In general, we observe that while responses generated from the Director and fine-tuned models are harmless (see 4 for sample responses), they are not particularly interesting or engaging, evident by the low Deb scores and Llm-Eval win rates. Encouragingly, our method obtains the highest Llm-Eval win rate however, we caution readers from drawing strong conclusions from these results alone [64].

7 Discussion↩︎

Below, we summarize our findings for each research question investigated in this work.

7.0.0.1 [q:demonstration95effective]: Do in-context safety demonstrations improve response safeness?

We find in-context learning can be used to increase dialogue system safety. Our results suggest that in-context safety demonstrations are most useful when they have high similarity with the target context, evident by performance improvements with better retrievers. However, we also observed that substantial reductions in toxicity can still be obtained providing any safety demonstrations. Finally, our human evaluation shows these safety improvements are not at the cost of other generation qualities.

7.0.0.2 [q:baselines]: How does in-context learning compare to popular safe response generation methods?

We compared the performance of our approach to three strong baseline methods for safe response generation. We found our approach performs competitively with these baselines without requiring training and without degrading quality. For example, on DiaSafety, we found our method obtained a Classifier score only \(2.25\) points lower than Director while obtaining a substantially higher Deb score and Llm-Eval win rate.

8 Conclusion↩︎

To the best of our knowledge, we perform the first large-scale evaluation of in-context learning for dialogue safety. We use in-context learning to reduce toxicity in three models: OPT, LLaMA, and Vicuna. Our results suggest that in-context learning performs competitively with traditional training-based approaches to dialogue safety. Furthermore, our proposed method can be used in compliment with popular dialogue safety approaches, such as RLHF. We hope our work spurs future research investigating the role of retrieval in dialogue safety.

9 Limitations↩︎

We now discuss three limitations to our work.

9.0.0.1 1) Our work only investigates reducing toxicity in dialogue systems.

A variety of safety issues have been identified with dialogue systems [16]. In our work, we focus on mitigating blatant toxicity (Instigator and Yea-Sayer effect) however, our method can be used to mitigate other safety issues.

While recent work [26], [27], [65] has investigated aligning dialogue systems with guidelines or social rules-of-thumb [25], [66], we do not investigate using social rules-of-thumb to condition generation. Using social rules-of-thumb in-context may be an attractive direction for future work as it can potentially reduce the computational cost of in-context learning [67].

9.0.0.3 3) Our investigation makes simplifying assumptions about using retrieval for dialogue safety.

For instance, we experiment with short dialogues (\(\leq 2\) turns) but unsafe inputs to a model can emerge after many conversation turns in real-world settings [68]. We also retrieve safety demonstrations for every response generation, even if they are not required. In practice, one may only require safety demonstrations for particular inputs. Future work can investigate methods for determining when and how many safety demonstrations should be retrieved during conversation. Finally, we also assume access to a pool of safety demonstrations to retrieve from. In practice, these safety demonstrations may need to be crafted by humans. We investigate the performance of our method with limited safety demonstrations in 12.4.

10 Acknowledgements↩︎

SR is supported by the Canada CIFAR AI Chairs program and the NSERC Discovery Grant program. NM is supported by a Canada Graduate Scholarship (CGS-D) funded by the Natural Sciences and Engineering Research Council (NSERC).

11 Ethical Considerations↩︎

In this work, we used a variety of different methods for evaluating dialogue system safety. We first highlight that all of the safety evaluation methods used in this work have only positive predictive power. In other words, they can flag potentially unsafe behaviour from a conversational model, but they can not verify that a conversational model is entirely safe. Additionally, for the human evaluation conducted in this work, we only used North American crowdworkers (see 18 for details). Thus, we caution readers from drawing strong conclusions from these safety evaluations alone.

In our study, we also leveraged safety demonstrations from several sources. As these safety demonstrations are crowdsourced, they may not reflect ideal dialogue system behaviour cross-culturally—different cultures and people may have different notions of ideal conversational model behaviour. Furthermore, there may be instances of unsafe content being present in the safety demonstrations used in this work due to noise within the crowdsourcing process.

12 Ablations↩︎

In this section, we present a collection of ablations. We experiment with OPT-2.7B, OPT-6.7B, and OPT-13B for all of our ablations and present results for ProsocialDialog and DiaSafety.

12.1 Are Regular Dialogue Demonstrations Useful for Reducing Toxicity?↩︎

We investigate if “regular” dialogue demonstrations are useful for reducing response toxicity. Concretely, we compare the safeness of OPT responses to ProsocialDialog and DiaSafety generated with either demonstrations from ProsocialDialog or Commonsense-Dialogues [51].

We present our results in 5. In general, we observe that using safety demonstrations tends to provide a larger increase to response safety compared to using regular demonstrations.

Figure 5: Safety classifier results for OPT responses to ProsocialDialog and DiaSafety using either safety demonstrations (ProsocialDialog) or Commonsense-Dialogues (regular) demonstrations. We report the mean and standard deviation across three seeds.

12.2 Does Demonstration Order Impact Response Toxicity?↩︎

Recent work has highlighted the impact of demonstration order on in-context learning performance [43]. We investigate the impact of order on response toxicity. Specifically, we evaluate three ordering methods: 1) Random; 2) Placing the demonstrations in descending order in the prompt based upon their retrieval scores; and 3) Placing the demonstrations in ascending order based upon their retrieval scores. We generate responses to ProsocialDialog and DiaSafety using different sized OPT models and different demonstration ordering methods. For all models, we use a dense retriever to select demonstrations for a given target context.

In 6, we present our results. We observe little difference in response toxicity across the three ordering methods.

Figure 6: Safety classifier results for OPT responses to DiaSafety using different demonstration orderings. “Top First” denotes placing the demonstration with the highest retrieval score at the start of the prompt. “Top Last” denotes placing the demonstration with the highest retrieval score at the end of the prompt. “Random” denotes placing the demonstrations in the prompt in random order. We report the mean and standard deviation across three seeds.

12.3 Impact of Shuffling Utterances in Demonstrations?↩︎

We investigate the impact of shuffling utterances in the demonstrations on response toxicity. We evaluate two scrambling methods: 1) Shuffling only the safe utterances and 2) Shuffling all of the utterances. We shuffle utterances across demonstrations. More plainly, when shuffling only the safe utterances, each safe utterance is randomly replaced by another safe utterance from one of the \(K\) retrieved demonstrations. This safe utterance could be from the same demonstration or another demonstration. When shuffling all utterances, each utterance is randomly replaced by another utterance from one of the \(K\) retrieved demonstrations. To evaluate the impact of these scrambling methods, we generate responses to ProsocialDialog and DiaSafety using different sized OPT models. We use a dense retriever to select all of the demonstrations.

In 7, we present our results. We observe that shuffling all of the utterances in the demonstrations has the largest impact on performance. However, we find that shuffling only the safe utterances within the demonstrations does not negatively impact performance. This suggests that models may only require surface-level patterns for learning to respond to unsafe dialogue contexts.

Figure 7: Safety classifier results for OPT responses to DiaSafety using different shufflings of the utterances in the demonstrations. We report the mean and standard deviation across three seeds.

12.4 How Does Limited Data Impact Response Toxicity?↩︎

We investigate how well our approach performs with limited data. This question is of practical interest as you may not have access to a large pool of demonstrations in a real-world setting. To investigate performance with limited data, we experiment with randomly subsampling the demonstration pool. Concretely, we test using demonstration pools with either \(10\), \(4230\), or \(42304\) conversations. These correspond to roughly \(0.02\)%, \(10\)%, and \(100\)% of the available demonstrations from the ProsocialDialog training split. We generate responses to ProsocialDialog and DiaSafety using these different sized demonstration pools and evaluate the resulting response safeness. We use a dense retriever for generating all of the responses.

We report our results in 8. We find that even when using a highly limited demonstration pool (e.g., \(10\) demonstrations), substantial reductions to toxicity can be obtained.

Figure 8: Safety classifier results for OPT responses with different sized demonstration pools. We use a dense retriever for generating all of the responses. We report the mean and standard deviation across three seeds.

13 Commonsense-Dialogues Results↩︎

We investigate generating responses to safe inputs. We generate responses using different sized OPT models and retrievers to Commonsense-Dialogues and present Classifier results in 9. We also present response automatic relevance evaluation results in 5.

Figure 9: Safety classifier results for OPT responses to Commonsense-Dialogues using different retrievers. We report the mean and standard deviation across three seeds.

Table 5: Automatic evaluation of OPT-13B responses to Commonsense-Dialogues. \(K\) denotes the number of demonstrations used for generation. We generate all responses using a dense retriever. We **bold** the best value for each metric. We report the mean and standard deviation across three seeds.
Model	\(K\)	Rouge-1 \(\uparrow\)	Meteor \(\uparrow\)	F1 \(\uparrow\)	Avg. Length
OPT-13B	0	\(12.88 \pm 0.15\)	\(15.58 \pm 0.178\)	\(11.01 \pm 0.15\)	\(20.90 \pm 0.241\)
OPT-13B	2	\(13.26 \pm 0.33\)	\(16.06 \pm 0.306\)	\(11.59 \pm 0.32\)	\(22.34 \pm 0.272\)
OPT-13B	4	\(13.37 \pm 0.18\)	\(16.32 \pm 0.259\)	\(\mathbf{11.61} \pm 0.23\)	\(22.98 \pm 0.173\)
OPT-13B	6	\(\mathbf{13.40} \pm 0.36\)	\(16.44 \pm 0.236\)	\(\mathbf{11.61} \pm 0.21\)	\(23.35 \pm 0.274\)
OPT-13B	8	\(13.39 \pm 0.24\)	\(16.41 \pm 0.250\)	\(11.58 \pm 0.17\)	\(23.77 \pm 0.404\)
OPT-13B	10	\(13.37 \pm 0.46\)	\(\mathbf{16.50} \pm 0.527\)	\(\mathbf{11.61} \pm 0.47\)	\(23.88 \pm 0.609\)

14 Generation Details↩︎

We generate all of our responses with a minimum length of \(20\) tokens and a maximum length of \(64\) tokens. We use Nucleus Sampling [69] with \(p = 0.85\) to sample all of our responses with temperature \(t = 1\). We truncate all generated responses at the first newline character. We did not extensively experiment with other generation hyperparameters or sampling procedures. We use the Hugging Face Transformers [70] implementations of all of the models investigated in this work.

15 Retriever Details↩︎

We investigated four methods for selecting in-context safety demonstrations. For all of our experiments, we use the ProsocialDialog training split as our demonstration pool. With the exception of our random retriever baseline, all of our retrievers select demonstrations based upon their similarity to the target context. We detail each retrieval method below.

15.0.0.1 Random.

We randomly sample \(K\) demonstrations from the demonstration pool for each target context. We do not use the same sample of demonstrations for all responses (i.e., we sample demonstrations for each target context).

15.0.0.2 BM25.

We use BM25 [48] to select \(K\) demonstrations from the demonstration pool. We use the Gensim implementation of BM25 for all of our experiments.⁹

15.0.0.3 SentenceTransformer.

We use a SentenceTransformer [49] for selecting \(K\) demonstrations from the demonstration pool. Concretely, we use all-mpnet-base-v2 [71] for encoding all of our demonstrations.

15.0.0.4 Wizard of Wikipedia.

We train a BERT-based [72] conversation encoder on Wizard of Wikipedia (WoW; [73]) using DPR [50]. We use the codebase and default hyperparameters released by [50] for training our encoder.¹⁰ We use bert-base-uncased to initialize our conversation encoder prior to training with DPR.

As an indirect measure of retriever performance, we use the resulting toxicity of responses generated using the selected demonstrations. We investigated the effectiveness of each retriever on ProsocialDialog and DiaSafety. We present our results in 10. In general, we find that the BM25, SentenceTransformer, and WoW retrievers outperform random retrieval in all settings. This highlights the usefulness of selecting similar demonstrations to the target context to include in-context. Specifically, we find that the SentenceTransformer retriever performs best in both ProsocialDialog and DiaSafety across the three model sizes. Because of this, we omit results for our WoW retriever within other experiments in this work.

Figure 10: Safety classifier results for OPT responses to ProsocialDialog and DiaSafety using different retrievers. “WoW” denotes a BERT-based retriever trained with DPR on Wizard of Wikipedia. We report the mean and standard deviation across three seeds.

16 LLaMA and Vicuna Results↩︎

In addition to OPT, we also experiment with 7B/13B LLaMA [4] and Vicuna [18] models. In 11 and 12, we provide Classifier results for ProsocialDialog and DiaSafety, respectively. We observe similar trends in our LLaMA and Vicuna results to OPT.

Figure 11: Safety classifier results for OPT, LLaMA, and Vicuna responses to ProsocialDialog. We compare similar sized models from each family. We report the mean and standard deviation across three seeds.

Figure 12: Safety classifier results for OPT, LLaMA, and Vicuna responses to DiaSafety. We compare similar sized models from each family. We report the mean and standard deviation across three seeds.

17 Response Evaluation with LLMs↩︎

Following the setup of [59], we use GPT-3.5-Turbo to automatically evaluate the quality of generated responses.¹¹ Concretely, we carry out head-to-head comparisons between generated responses using GPT-3.5-Turbo. We prompt the model to select from a given pair of responses which response is more “helpful,” “relevant,” “detailed,” “creative,” and “respectful” using the prompt shown in 13. Importantly, we allow the model to label a pair of responses as a “tie” if neither response is preferable. We compare responses from the following nine models:

OPT-6.7B: Base OPT-6.7B without in-context demonstrations.
OPT-6.7B + Dense: OPT-6.7B with ten in-context demonstrations selected using a dense retriever.
LLaMA-7B: Base LLaMA-7B without in-context demonstrations.
LLaMA-7B + Dense: LLaMA-7B with ten in-context demonstrations selected using a dense retriever.
Vicuna-7B: Base Vicuna-7B without in-context demonstrations.
Vicuna-7B + Dense: Vicuna-7B with ten in-context demonstrations selected using a dense retriever.
OPT-6.7B + Self-Debias: OPT-6.7B using Self-Debias during decoding.
OPT-6.7B + DIRECTOR: OPT-6.7B which has been fine-tuned using DIRECTOR.
OPT-6.7B + Fine-Tune: OPT-6.7B which has been fine-tuned on safe responses from ProsocialDialog and SaFeRDialogues.

We conduct \(256\) head-to-head comparisons for each of the \(36\) model pairings. In total, we carry out \(9216\) comparisons. To attempt to mitigate positional biases [64], we randomize the ordering of the responses for each comparison.

We generate responses from GPT-3.5-Turbo with a temperature of \(0.9\) and \(p = 0.95\) for Nucleus Sampling. We did not experiment extensively with these parameters. We reject and regenerate any response not beginning with [[A]], [[B]], or [[C]]. We report the win rates for each model pairing. We exclude all ties in our win rate calculations. We found only a relatively small number of comparisons were labeled ties (see 15).

In 14, we report win rates for all model pairings. We first note that Vicuna obtains the highest average win rate. We caution readers from drawing strong conclusions from this result as Vicuna was trained using ChatGPT responses. Encouragingly, we observe that using in-context safety demonstrations with OPT, LLaMA, and Vicuna always results in a higher average win rate relative to not using any demonstrations. We also note the poor performance of the Director and Fine-Tune models.

Figure 13: Prompt for Llm-Eval response evaluation using GPT-3.5-Turbo.

Figure 14: Win rates for all head-to-head comparisons using Llm-Eval. We sort the models on the y-axis in descending order based upon their average win rate. We exclude ties in our win rate calculation.

Figure 15: Tie counts for all head-to-head comparisons using Llm-Eval.

18 Human Evaluation↩︎

We follow the setup of [25] and evaluate the prosocialness, engagingness, and coherency of generated responses. We compare responses generated from three different dialogue systems:

OPT-30B: The base OPT-30B model without in-context demonstrations.
OPT-30B + Dense: The OPT-30B model with ten in-context demonstrations selected using a dense retriever.
BlenderBot3-30B: The base BlenderBot3-30B model without in-context demonstrations.

Importantly, BlenderBot3-30B is based upon OPT-30B but has been further trained on dialogue data. We evaluate responses generated in both the in-domain and out-of-domain settings. For the in-domain setting, we use ProsocialDialog. For the out-of-domain setting, we use DiaSafety. We randomly select \(150\) examples from the validation set of each dataset for response generation and use the prompt shown in 2.

We conduct two head-to-head comparisons between models on ProsocialDialog and DiaSafety:

OPT-30B vs. OPT-30B + Dense
OPT-30B + Dense vs. BlenderBot3-30B

For each pair of models, we provide annotators with a response from each system and task them with selecting which response is preferable along one of the three dimensions (prosocialness, engagingness, and coherency). We also allow annotators to rate a given pair of examples as a tie if neither response is preferable. For each quality, we collect three human annotations for each of the \(150\) examples (totaling \(450\) annotations for each head-to-head comparison for a quality). We compute the majority vote win-rate for each model. In 16, we provide a screenshot of our interface for response coherency evaluation. We use similar interfaces for our engagingness and prosocialness evaluations. In 6, we provide the Fleiss Kappa annotator agreement scores for our human evaluation. We found that allowing annotators to score a response-pair as a tie tended to decrease annotator agreement scores.

We use Amazon Mechanical Turk for conducting our human evaluation and pay annotators 0.15 USD per HIT. We only use workers who: 1) Have a HIT approval rate of 95%; 2) Have had at least 1000 HITs approved; and 3) Are located in the United States.

Figure 16: User interface for human evaluation of response coherency.

Table 6: Fleiss Kappa scores for human evaluation. We found including an option for rating a response-pair as a *tie* decreased annotator agreement.
Head-to-Head Comparison	Prosocial	Engage	Coherent
ProsocialDialog
OPT-30B vs. OPT-30B + Dense	\(0.52\)	\(0.16\)	\(0.24\)
OPT-30B + Dense vs. BlenderBot3-30B	\(0.49\)	\(0.08\)	\(0.27\)
DiaSafety
OPT-30B vs. OPT-30B + Dense	\(0.28\)	\(0.21\)	\(0.24\)
OPT-30B + Dense vs. BlenderBot3-30B	\(0.37\)	\(0.15\)	\(0.14\)

19 Dataset Overview↩︎

In ¿tbl:tab:dataset95overview?, we provide an overview of the datasets used in this work. At a high-level, we use the training split from ProsocialDialog as our demonstration pool for all of our experiments. We evaluate responses generated to the validation splits of ProsocialDialog, DiaSafety, and Commonsense-Dialogues. We consider our ProsocialDialog evaluation to be in-domain as our safety demonstrations are drawn from the same dataset. We consider our DiaSafety and Commonsense-Dialogues evaluations to be out-of-domain as the safety demonstrations are not drawn from DiaSafety or Commonsense-Dialogues. For all datasets used in this work, we use a maximum of two turns.

20 Baseline Details↩︎

20.0.0.1 Director.

We use the implementation released by [34] for training our model.¹² We use the same hyperparameters as [34] and train our model to convergence using Adam [74] and a learning rate of \(1e-5\). We use a validation patience of \(10\). We train our model on Wikipedia Toxic Comments [52] and the safety data from [21].

20.0.0.2 Safe Response Fine-Tuning.

We use ParlAI [75] for training our model on safe responses. We train our model on Blended Skill Talk [62], Empathetic Dialogues [61], ConvAI2 [60], ProsocialDialog [25], and SaFeRDialogues [24]. All of these datasets are available within ParlAI. We use Adam and a learning rate of \(1e-5\) for training our model. We train to convergence using a validation patience of \(10\).

20.0.0.3 Self-Debias.

We use the implementation released by [31] for our experiments.¹³ We use all of the available prompts for detoxification.

lllll Dataset & & & & Sample Dialogue
ProsocialDialog & 2 & & & & &
DiaSafety & 1 & & &
Commonsense-Dialogues & 2 & & & & &

21 Additional Baselines↩︎

In addition to the baselines presented in 6, we also compare our method to two prompting baselines. We describe each baseline below.

21.0.0.1 Helpful and Harmless Prompting.

We prompt a model to be “helpful” and “harmless.” For this baseline, we adopt a prompt from [4].¹⁴

21.0.0.2 Rule-of-Thumb Prompting.

We use social rules-of-thumb from ProsocialDialog in the prompt when performing response generation. To select the rule-of-thumb to include in-context, we randomly select a rule-of-thumb from the top-ranked safety demonstration after retrieval. We adapt the prompt from [25] for this baseline.

We provide automatic safety results for DiaSafety for these baselines in 7. In general, we find the two new baselines outperform the base model (OPT-6.7B) but are outperformed by our method (OPT-6.7B + Dense). We omit results for these baselines in the main paper.

Table 7: Automatic safety evaluation of OPT-6.7B responses to DiaSafety for additional baselines. We **bold** the best value for each metric. We report the mean and standard deviations across three seeds for each metric.
Model	Classifier \(\uparrow\)	Perspective \(\uparrow\)	Word List \(\uparrow\)
OPT-6.7B	\(57.79 \pm 0.79\)	\(74.35 \pm 1.97\)	\(86.66 \pm 2.04\)
OPT-6.7B + Dense	\(\mathbf{77.57} \pm 0.57\)	\(\mathbf{89.33} \pm 0.09\)	\(\mathbf{94.22} \pm 0.65\)
OPT-6.7B + Helpful/Harmless Prompt	\(62.14 \pm 0.05\)	\(80.92 \pm 0.87\)	\(88.57 \pm 0.01\)
OPT-6.7B + Rule-of-Thumb	\(64.29 \pm 1.06\)	\(83.53 \pm 0.50\)	\(89.94 \pm 0.85\)

22 Additional Safety Classifier Results↩︎

To demonstrate that our results are consistent across a range of toxicity classifiers, we provide additional results for two classifers: a RoBERTa toxicity classifier trained on ToxiGen [76] and a RoBERTa toxicity classifier trained using Dynabench ([77]; the default classifier used in Hugging Face Evaluate for toxicity). In 8, we provide results for DiaSafety for these classifiers. We report the percentage of safe responses for different OPT-6.7B models. We observe that for all three classifiers, our method performs competitively with Director.

Table 8: Safety classifier results for OPT-6.7B responses to DiaSafety. We report the mean and standard deviation across three seeds. We use the Bot-Adversarial Dialogue classifier for all other safety classification results in this paper.
Model	Bot-Adversarial Dialogue Classifier \(\uparrow\)	Hugging Face Evaluate Toxicity \(\uparrow\)	ToxiGen \(\uparrow\)
OPT-6.7B	\(57.79 \pm 0.79\)	\(76.90 \pm 0.90\)	\(60.95 \pm 0.68\)
OPT-6.7B + Random	\(68.31 \pm 0.97\)	\(82.65 \pm 0.32\)	\(64.54 \pm 1.43\)
OPT-6.7B + BM25	\(71.16 \pm 0.92\)	\(83.59 \pm 0.47\)	\(65.23 \pm 1.32\)
OPT-6.7B + Dense	\(77.57 \pm 0.57\)	\(86.08 \pm 0.88\)	\(69.19 \pm 0.36\)
OPT-6.7B + Director	\(79.82 \pm 1.15\)	\(85.29 \pm 0.41\)	\(65.48 \pm 0.42\)

23 Sample Responses↩︎

Table 9: Sample responses from OPT-6.7B to target contexts from ProsocialDialog.

Table 10: Sample responses from Vicuna-7B to target contexts from ProsocialDialog.

Table 11: Sample responses from OPT-6.7B to target contexts from DiaSafety.

Table 12: Sample responses from OPT-6.7B and OPT-30B to target contexts from DiaSafety.

References↩︎

[1]

Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. https://doi.org/10.18653/v1/2021.eacl-main.24. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300–325, Online. Association for Computational Linguistics.

[2]

Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. https://doi.org/10.48550/arXiv.2206.11309. ArXiv:2206.11309 [cs].

[3]

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. https://doi.org/10.48550/arXiv.2201.08239. ArXiv:2201.08239 [cs].

[4]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://doi.org/10.48550/arXiv.2302.13971. ArXiv:2302.13971 [cs].

[5]

Jing Xu, Da Ju, Margaret Li, Y.-Lan Boureau, Jason Weston, and Emily Dinan. 2021. http://arxiv.org/abs/2010.07079. arXiv:2010.07079 [cs]. ArXiv: 2010.07079.

[6]

Emily Dinan, Gavin Abercrombie, A. Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2022. https://doi.org/10.18653/v1/2022.acl-long.284. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4113–4133, Dublin, Ireland. Association for Computational Linguistics.

[7]

Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. 2023. https://doi.org/10.48550/arXiv.2302.09270. ArXiv:2302.09270 [cs].

[8]

Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.656. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online. Association for Computational Linguistics.

[9]

Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. https://doi.org/10.18653/v1/2021.acl-long.151. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941–1955, Online. Association for Computational Linguistics.

[10]

Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.397. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4846–4862, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

[11]

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

[12]

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. https://doi.org/10.48550/arXiv.1909.08593. ArXiv:1909.08593 [cs, stat].

[13]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. https://doi.org/10.48550/arXiv.2203.02155. ArXiv:2203.02155 [cs].

[14]

Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y.-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. https://doi.org/10.48550/arXiv.2208.03188. ArXiv:2208.03188 [cs].

[15]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. https://doi.org/10.48550/arXiv.2204.05862. ArXiv:2204.05862 [cs].

[16]

Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y.-Lan Boureau, and Verena Rieser. 2021. https://doi.org/10.48550/arXiv.2107.03451. Number: arXiv:2107.03451 arXiv:2107.03451 [cs].

[17]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. https://doi.org/10.48550/arXiv.2205.01068. ArXiv:2205.01068 [cs].

[18]

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/.

[19]

Irene Solaiman and Christy Dennison. 2021. https://proceedings.neurips.cc/paper/2021/hash/2e855f9489df0712b4bd8ea9e2848c5a-Abstract.html. In Advances in Neural Information Processing Systems, volume 34, pages 5861–5873. Curran Associates, Inc.

[20]

Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, and Nicholas Frosst. 2021. https://doi.org/10.48550/arXiv.2108.07790. ArXiv:2108.07790 [cs].

[21]

Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. https://doi.org/10.18653/v1/D19-1461. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537–4546, Hong Kong, China. Association for Computational Linguistics.

[22]

Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. https://doi.org/10.18653/v1/2021.naacl-main.235. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968, Online. Association for Computational Linguistics.

[23]

Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.308. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3906–3923, Dublin, Ireland. Association for Computational Linguistics.

[24]

Megan Ung, Jing Xu, and Y-Lan Boureau. 2022. https://doi.org/10.18653/v1/2022.acl-long.447. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6462–6481, Dublin, Ireland. Association for Computational Linguistics.

[25]

Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.267. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4005–4029, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[26]

Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. https://doi.org/10.48550/arXiv.2305.03047. ArXiv:2305.03047 [cs].

[27]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. https://doi.org/10.48550/arXiv.2212.08073. ArXiv:2212.08073 [cs].

[28]

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://doi.org/10.48550/arXiv.2305.11206. ArXiv:2305.11206 [cs].

[29]

Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. https://doi.org/10.48550/arXiv.2209.14375. ArXiv:2209.14375 [cs].

[30]

OpenAI. 2022. https://openai.com/blog/chatgpt/.

[31]

Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. https://doi.org/10.1162/tacl_a_00434. Transactions of the Association for Computational Linguistics, 9:1408–1424.

[32]

Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. https://doi.org/10.48550/arXiv.1909.05858. ArXiv:1909.05858 [cs].

[33]

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. https://openreview.net/forum?id=H1edEyBKDS. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.

[34]

Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. https://aclanthology.org/2022.aacl-main.39. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 512–526, Online only. Association for Computational Linguistics.

[35]

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. https://doi.org/10.18653/v1/2021.acl-long.522. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691–6706, Online. Association for Computational Linguistics.

[36]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

[37]

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021. https://doi.org/10.48550/arXiv.2112.06905. ArXiv:2112.06905 [cs].

[38]

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022. https://doi.org/10.48550/arXiv.2112.11446. ArXiv:2112.11446 [cs].

[39]

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. https://aclanthology.org/2022.findings-emnlp.193. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[40]

Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.38 In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[41]

Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.624. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9157–9179, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[42]

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. https://doi.org/10.48550/arXiv.2112.00861. ArXiv:2112.00861 [cs].

[43]

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.

[44]

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.

[45]

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.naacl-main.191. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.

[46]

Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. https://doi.org/10.18653/v1/2023.findings-acl.564. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.

[47]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

[48]

Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019. Found. Trends Inf. Retr., 3(4):333–389.

[49]

Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

[50]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

[51]

Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2021. https://doi.org/10.48550/arXiv.2109.06427. ArXiv:2109.06427 [cs].

[52]

Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. https://doi.org/10.1145/3038912.3052591. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, pages 1391–1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

[53]

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. https://openreview.net/forum?id=5HaIds3ux5O. In Advances in Neural Information Processing Systems.

[54]

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. https://doi.org/10.48550/arXiv.2304.12397. ArXiv:2304.12397 [cs].

[55]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

[56]

Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

[57]

Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. https://doi.org/10.1162/tacl_a_00347. Transactions of the Association for Computational Linguistics, 8:810–827.

[58]

Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. https://doi.org/10.1145/3209978.3210080. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 1097–1100, New York, NY, USA. Association for Computing Machinery.

[59]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://doi.org/10.48550/arXiv.2306.05685. ArXiv:2306.05685 [cs].

[60]

Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2019. https://doi.org/10.48550/arXiv.1902.00098. ArXiv:1902.00098 [cs].

[61]

Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. https://doi.org/10.18653/v1/P19-1534. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.

[62]

Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. https://doi.org/10.18653/v1/2020.acl-main.183. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021–2030, Online. Association for Computational Linguistics.

[63]

Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. https://doi.org/10.18653/v1/2022.acl-long.132. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics.

[64]

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. https://doi.org/10.48550/arXiv.2305.17926. ArXiv:2305.17926 [cs].

[65]

Prakhar Gupta, Yang Liu, Di Jin, Behnam Hedayatnia, Spandana Gella, Sijia Liu, Patrick Lange, Julia Hirschberg, and Dilek Hakkani-Tur. 2022. https://doi.org/10.48550/arXiv.2212.10557. ArXiv:2212.10557 [cs].

[66]

Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. https://doi.org/10.18653/v1/2022.acl-long.261. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755–3773, Dublin, Ireland. Association for Computational Linguistics.

[67]

Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. https://openreview.net/forum?id=rBCvMG-JsPd. In Advances in Neural Information Processing Systems.

[68]

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. https://doi.org/10.48550/arXiv.2209.07858. ArXiv:2209.07858 [cs].

[69]

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH. In International Conference on Learning Representations.

[70]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. https://doi.org/10.18653/v1/2020.emnlp-demos.6. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

[71]

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html. In Advances in Neural Information Processing Systems, volume 33, pages 16857–16867. Curran Associates, Inc.

[72]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

[73]

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. https://openreview.net/forum?id=r1l73iRqKm. In Proceedings of the International Conference on Learning Representations.

[74]

Diederick P Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980. In International Conference on Learning Representations.

[75]

Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. https://doi.org/10.18653/v1/D17-2014. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics.

[76]

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://doi.org/10.18653/v1/2022.acl-long.234. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–3326, Dublin, Ireland. Association for Computational Linguistics.

[77]

Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. https://doi.org/10.18653/v1/2021.acl-long.132. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1667–1682, Online. Association for Computational Linguistics.

\(^\ast\)Work done during an internship at Amazon Alexa AI.↩︎
While our approach can be used to mitigate a range of safety issues, we focus on reducing toxicity as a wealth of datasets and tools exist for quantifying it.↩︎
We also investigated using a Dense Passage Retriever (DPR; [50]) to select demonstrations and defer readers to 15 for results and additional retriever details.↩︎
We consider a conversation turn to be an exchange between two speakers in a conversation.↩︎
For more information on PerspectiveAPI, see: https://perspectiveapi.com ↩︎
https://platform.openai.com/docs/models/gpt-3-5 ↩︎
We only use the vanilla dialogue generation module for BlenderBot3-30B. That is, we do not use the internet search, long-term memory, or the knowledge-grounded generation modules.↩︎
Because of resource constraints, we use OPT-6.7B for these experiments.↩︎
https://github.com/RaRe-Technologies/gensim ↩︎
https://github.com/facebookresearch/DPR ↩︎
https://platform.openai.com/docs/models/gpt-3-5 ↩︎
https://parl.ai/projects/director/↩︎
https://github.com/timoschick/self-debiasing ↩︎
We use the fourth prompt provided in Table 39 from their work.↩︎

Using In-Context Learning to Improve Dialogue Safety