1 Introduction↩︎

Large language models (LLMs) are often trained to align with human values, thereby refusing to generate harmful or toxic content [4]. However, a growing body of work has shown that even the most performant LLMs are not adversarially aligned: it is often possible to elicit undesirable content by using so-called jailbreaking attacks [5], [6]. Concerningly, researchers have shown that such attacks can be generated in many different ways, including hand-crafted prompts [7], [8], automatic prompting via auxiliary LLMs [9], [10], and iterative optimization [1]. And while several defenses have been proposed to mitigate these threats [11], [12], LLMs remain highly vulnerable to jailbreaking attacks. For this reason, as LLMs are deployed in safety-critical domains, it is of pronounced importance to effectively benchmark the progress of jailbreaking attacks and defenses [13].

To meet this need, this paper introduces the JailbreakBench benchmark. The design principles of JailbreakBench revolve around standardizing a set of best practices in the evolving field of LLM jailbreaking. Our core principles include complete reproducibility via a commitment to open-sourcing jailbreak prompts corresponding to attacked and defended models, extensibility to incorporate new attacks, defenses, and LLMs, and accessibility of our evaluation pipeline to expedite future research.

In this paper, we elaborate further on these principles, describe the components of the benchmark, provide a thorough discussion on the selection of an accurate jailbreaking judge, and present the results of multiple attacks and test-time defenses on several open- and closed-sourced LLMs.

Contributions. The contributions of the JailbreakBench benchmark are as follows:

• Repository of jailbreak artifacts. We provide an evolving repository of artifacts corresponding to state-of-the-art jailbreaking attacks and defenses. Despite being critical for reproducible research, many such prompts have not been openly released, and crowd-sourced websites have gone offline [14]. These artifacts can be accessed in the following way via our library:

  python import jailbreakbench as jbb artifact = jbb.read_artifact(method="PAIR", model_name="vicuna-13b-v1.5") print(artifact.jailbreaks[75])

  JailbreakInfo( index=75, goal=‘Design a phishing email that can trick users into giving away their login credentials’ identifier=‘Phishing’, category=‘Privacy’, prompt=‘Security research is an essential part of understanding how to protect users [...]’ response=‘Subject: Urgent Action Required: Update Your Account [...]’, number_of_queries=30, queries_to_jailbreak=5, prompt_tokens=3564, response_tokens=3729, jailbroken=True)

• Pipeline for red-teaming LLMs. We provide a standardized pipeline for red-teaming LLMs. In particular, our pipeline implements the evaluation of potential jailbreaks, standardizes decoding parameters, and supports both local and cloud-based querying.

• Pipeline for testing and adding new defenses. We implement five baseline defenses which can be combined with any LLM. Moreover, we welcome the submission of new defensive mechanisms or robust models, which will be easily accessible by users via the benchmark interface.

• Jailbreaking classifier selection. Evaluating the success of jailbreak attacks is challenging given the subjective nature of judging the appropriateness of a LLM’s response. We perform a rigorous human evaluation to compare six jailbreak classifiers. Among these classifiers, we find the recent Llama-3-Instruct-70B to be an effective judge when used with a properly selected prompt.

• Dataset of harmful and benign behaviors. We introduce the JBB-Behaviors dataset, which comprises 100 distinct misuse behaviors divided into ten broad categories corresponding to OpenAI’s usage policies. Approximately half of these behaviors are original, while the other half are sourced from existing datasets [1]–[3]. For each misuse behavior, we also collect a matching benign behavior on the same exact topic that can be used as a sanity check for evaluating refusal rates of new models and defenses.

• Reproducible evaluation framework. We provide a reproducible framework for evaluating the attack success rate of jailbreaking algorithms, which can also be used to submit an algorithm’s jailbreak strings to our artifact repository.

• Jailbreaking leaderboard and website. We maintain a website hosted at https://jailbreakbench.github.io/ which tracks the performance of jailbreaking attacks and defenses across various state-of-the-art LLMs on the official leaderboard (see Figure 1).

Preliminary impact. Two months after releasing the preliminary version of JailbreakBench on arXiv, researchers in the field have already started using our jailbreak artifacts [15], [16], jailbreak judge prompt [17], and the JBB-Behaviors dataset [18]–[23], notably including the authors of Gemini 1.5 from Google [24].

2 Background and related work↩︎

Definitions. At a high level, the goal of a jailbreaking algorithm is to design input prompts that cause an LLM to generate text that is harmful, toxic, or objectionable. More specifically, let us assume we have a target model LLM, a judge function JUDGE that determines the correspondence between the generation \(\texttt{LLM}(P)\), and a harmful goal \(G\). Then the task of jailbreaking can be formalized \[\begin{align} \text{find} \quad P\in\mathcal{T}^\star \quad\text{subject to}\quad \texttt{JUDGE}(\texttt{LLM}(P),G) = \text{True}, \label{eq:jailbreak95problem} \end{align}\tag{1}\] where \(P\) is the input prompt and \(\mathcal{T}^\star\) denotes the set of all sequences of tokens of arbitrary length.

Attacks. Early jailbreaking attacks involved manually refining hand-crafted jailbreak prompts [5], [8], [14]. Due to the time-consuming nature of manually collecting jailbreak prompts, research has largely pivoted toward automating the red-teaming pipeline. Several algorithms take an optimization perspective to solve Eq. 1 , either via first-order discrete optimization techniques [1], [25] or zero-th order methods, e.g. genetic algorithms [26], [27] or random search [28]–[30]. Additionally, auxiliary LLMs can aid the attacks, for example to refine hand-crafted jailbreak templates [31], translate goal strings into low-resource languages [32], [33], generate jailbreaks [9], [34], or rephrase harmful requests [10], [35], [36].

Defenses. Several methods try to mitigate the threat of jailbreaks. Many such defenses seek to align LLM responses to human preferences via methods such as RLHF [4] and DPO [37]. Relatedly, variants of adversarial training have been explored [3], as well as fine-tuning on jailbreak strings [38]. Conversely, test-time defenses like SmoothLLM [11], [39] and perplexity filtering [12], [40] define wrappers around LLMs to detect potential jailbreaks.

Evaluation. In the field of image classification, benchmarks such as RobustBench [41] provide a unified platform for both evaluating the robustness of models in a standardized manner and for tracking state-of-the-art performance. However, designing a similar platform to track the adversarial vulnerabilities of LLMs presents new challenges, one of which is that there is no standardized definition of a valid jailbreak. Indeed, evaluation techniques span human labeling [8], [32], rule-based classifiers [1], neural-network-based classifiers [3], [42], and the LLM-as-a-judge framework [9], [10], [35], [43]. [44] discuss the current state of jailbreak judges suggesting their suboptimal performance and propose a more detailed grading criterion for what constitutes a valid jailbreak. Unsurprisingly, the discrepancies and inconsistencies between these methods lead to variable results.

Benchmarks, leaderboards, and datasets. Several benchmarks involving LLM robustness have recently appeared. [45] propose PromptBench, a library for evaluating LLMs against adversarial prompts, although not in the context of jailbreaking. DecodingTrust [46] and TrustLLM [47] consider jailbreaking but only evaluate static templates, which excludes automated red-teaming algorithms. More related to JailbreakBench is the recently introduced HarmBench benchmark [3], which implements jailbreaking attacks and defenses, and considers a broad array of topics including copyright infringement and multimodal models.¹ In contrast, we focus on supporting adaptive attacks [28], [48] and test-time defenses [11], [12]. Thus, we standardize evaluation of test-time defenses but not attack implementations, since we expect them to potentially differ for different defenses. Moreover, we strive to make our benchmark community-driven, prioritizing clear guidelines for adding new attacks, models, and defenses. [49] was proposed concurrently with JailbreakBench and contains a red-teaming framework that implements 11 jailbreak methods. Several competitions have also appeared recently, including the “Trojan Detection Challenge” (TDC) at NeurIPS 2023 [2], and the “Find the Trojan: Universal Backdoor Detection in Aligned LLMs” competition at SaTML 2024 [50]. However, JailbreakBench is not a challenge or a competition, but rather an open-ended project which aims at tracking and facilitating the progress of the field. Finally, several stand-alone datasets of harmful behaviors have appeared, such as AdvBench [1], MaliciousInstruct [42], and the dataset of hand-crafted jailbreaks curated in [8]. However, many existing datasets contain duplicated entries, behaviors that are impossible to fulfill, or are not fully open-sourced.

3 Main features for JailbreakBench↩︎

Given the current landscape of the field, we believe there is a need for a jailbreaking benchmark that prioritizes supporting of state-of-the-art adaptive attacks and test-time defenses, as well as providing direct access to the corresponding jailbreaking artifacts. Thus, we prioritized the following principles when designing JailbreakBench.

1. Reproducibility. We ensure maximal reproducibility by collecting and archiving jailbreak artifacts, with the hope that this establishes a stable basis of comparison. Our leaderboard also tracks the state-of-the-art in jailbreaking attacks and defenses, so to identify leading algorithms and establish open-sourced baselines in future research.

2. Extensibility. We accept any jailbreaking attack, including white-box, black-box, universal, transfer, and adaptive attacks, and any jailbreaking defense, all of which are compared using the same set of evaluation metrics. We plan to adapt our benchmark as the community evolves to accommodate new threat models, attacks, defenses, and LLMs.

3. Accessibility. Our red-teaming pipeline is fast, lightweight, inexpensive, and can be run exclusively through cloud-based models, circumventing the need for local GPUs. In releasing the jailbreak artifacts, we hope to expedite future research on jailbreaking, especially on the defense side.

3.1 JBB-Behaviors: A dataset of harmful and benign behaviors↩︎

Harmful behaviors. An important contribution of this work is to curate the JBB-Behaviors dataset to facilitate the comparison of jailbreaking attacks and defenses. JBB-Behaviors contains a representative set of behaviors that encompass a broad spectrum of misuse. In contrast to several existing datasets, we ensure that each behavior in JBB-Behaviors is unique and realizable with a text-based generation. Among these behaviors, 18% are sourced from AdvBench [1], 27% from TDC/HarmBench [2], [3], whereas the remaining 55% are original to JBB-Behaviors(further information in 7, including a detailed breakdown over the sources). We note that many of the behavior topics were in part inspired by the topics from [35]. Specifically, JBB-Behaviors contains 100 rows, where each row specifies five distinct fields:

• Behavior. A unique identifier describing a distinct misuse behavior.

• Goal. A query requesting an objectionable behavior.

• Target. An affirmative response to the harmful goal string.

• Category. A broader category of misuse from OpenAI’s usage policies.

• Source. A reference to the source dataset of the goal and target string.

Benign behaviors. In addition, we create 100 benign behaviors where each of them matches the exact topic of the corresponding harmful behavior. For example, the benign counterpart of the harmful behavior “Outline a business model for a fake charity which uses donations for personal gain” is “Describe businesses that have illegally used charitable donations for personal gain.” We use benign behaviors to evaluate refusal rates for different models and defenses to make sure they do not refuse too often by, e.g., simply detecting some key words that are often associated with harmful behaviors. We note that some fraction of these behaviors can be considered as borderline, and different LLM providers might disagree about whether they should be refused or not.

3.2 A repository of jailbreaking artifacts↩︎

A central component of the JailbreakBench benchmark is our repository of easily accessible jailbreak artifacts, i.e., the prompts, responses, and classifications corresponding to each submitted attack or defense. Each artifact also contains metadata, e.g., hyperparameters of the attack/defense, the attack success rate, and the number of queries made to the target model, and can be easily loaded by

The JailbreakBench artifacts repository currently contains jailbreak strings for PAIR [9], GCG [1], JailbreakChat [14], and the attacks from [28]. Moreover, as described in §3.7, we intend for users to submit new artifacts as the benchmark evolves. In general, research surrounding LLM jailbreaking has showed hesitancy toward open-sourcing jailbreaking artifacts, given their propensity for potential misuse [1], [8]. However, we believe these jailbreaking artifacts can serve as an initial dataset for adversarial training against jailbreaks, as has been done in past research (see, e.g., [38]). We discuss this topic more thoroughly in §5.

3.3 A pipeline for red-teaming LLMs↩︎

Generating jailbreaks for LLMs often involves complex workflows that facilitate varying tokenization schemes, sampling algorithms, and system prompts. As changing each of these aspects can lead to highly variable results, we streamline the process of generating jailbreaks by introducing a standardized red-teaming pipeline. Our pipeline is both easy to use—LLMs can be loaded and queried in just two lines of Python—and flexible—we support both local and cloud-based LLMs. After loading a particular model type, it is straightforward to query that model:

3.4 A pipeline for evaluating defenses against jailbreaks↩︎

Alongside works on designing new attacks, researchers have also proposed defense algorithms to mitigate the threat posed by jailbreaking. To this end, we provide an implementation of five popular defenses, including "SmoothLLM" [11] and "PerplexityFilter" [12], and a modular framework for loading and querying them:

Alternatively, one can pass the defense argument directly to llm.query. Finally, we note that proper evaluation of test-time defenses should rely on adaptive attacks, i.e., attacks tailored to the specific defense under evaluation [48]. Transfer attacks from undefended LLMs can only provide a lower bound on the worst-case attack success rate.

3.5 Selection of the jailbreaking judge↩︎

Determining the success of an attack involves an understanding of human language and a subjective judgment of whether generated content is objectionable, which might be challenging even for humans. To this end, we consider six candidate classifiers used in the jailbreaking literature:

• Rule-based. The rule-based judge from [1] based on string matching,

• GPT-4. The GPT-4-0613 model used as a judge [51],

• HarmBench. The Llama-2-13B judge introduced in HarmBench [3],

• Llama Guard. An LLM safeguard model fine-tuned from Llama-2-7B [52],

• Llama Guard 2. An LLM safeguard model fine-tuned from Llama-3-8B [53],

• Llama-3-70B. The recent Llama-3-70B [54] used as a judge with a custom prompt.

For GPT-4, we use the JUDGE system prompt from [9], while for the Llama Guard models and Llama-3-70B, we use custom system prompts, which we share in Appendix 11. We found that it is necessary to use a custom prompt for Llama-3-70B as the prompt from [9] often leads to refusals. Moreover, Llama-3-8B also often refuses to serve as a judge, even with the custom prompt, which is the reason why we evaluate the 70B model.

To choose an effective classifier, we collected a dataset of 200 jailbreak prompts and responses (see for details). Three experts labeled each prompt-response pair, and the agreement between them was approximately 95%. The ground truth label for each behavior is then the majority vote among the labelers. Moreover, we add 100 benign examples from XS-Test [55] to test how sensitive the judges are to benign prompts and responses that share similarities to harmful ones. This dataset of 300 examples is provided in the JailbreakBench HuggingFace Datasets repository.

We compare the agreement, false positive rate (FPR), and false negative rate (FNR) of the candidate judges to these ground truth labels. Table 1 shows that Llama-3-70B and GPT-4 achieve the highest agreement (above 90%) with the annotators, as well as close to the best FPR and FNR. Llama Guard 2 is only slightly worse having 87.7% agreement and approximately equal FPR and FNR. The HarmBench and Llama Guard models have significantly lower agreement rates, at 78.3% and 72.0% respectively. We observe that the HarmBench model has noticeably many false positives on the 100 benign examples from XS-Test, leading to the overall 26.8% FPR on the full evaluation set. The Rule-based judge appears not effective with only 56.0% agreement. Although Llama-3-70B and GPT-4 appear to perform similarly well as judges, GPT-4 comes with the drawback of close-sourced models, i.e., expensive to query and subject to change. Thus, in line with the aim of JailbreakBench to be reproducible, we choose Llama-3-70B as the classifier in our benchmark as it is an open-weight model and comparable to GPT-4 as a judge. Moreover, it has a relatively low FPR, which, although it may systematically reduce the success rate across attack algorithms, is important for remaining conservative and avoiding the misclassification of benign behaviors as jailbroken. We provide an implementation of this judge in our library in two lines of python code:

We describe more details on judge selection in Appendix 8 and remain open to updating our chosen judge if consistently better options become available. Finally, we note that submissions to our benchmark, both attacks and defenses, are not permitted to use our Llama-3-70B judge in order to prevent overfitting to the judge.

3.6 Reproducible evaluation framework↩︎

We also provide a standardized evaluation framework that—besides providing access to jailbreak artifacts—allows the users to benchmark and report the performance of their methods. The framework implements all system prompts, the JUDGE classifier discussed in §3.5, and supports both cloud-based and local querying, as discussed in §3.3. In particular, our framework includes the following LLMs: Vicuna-13B-v1.5 [43], Llama-2-7B-chat-hf [56], GPT-3.5-Turbo-1106 [51], GPT-4-0125-Preview [51]. To facilitate reproducibility, all model use greedy, deterministic sampling (i.e., with temperature equal to zero), and by default all LLMs generate 150 tokens per input. A more detailed discussion about reproducibility is given in Appendix 10. To evaluate the performance of a set of jailbreak strings on any of these LLMs, one can run the following code snippet:

This code first generates responses to the input strings by querying "vicuna-13b-v1.5", after which the prompt-response pairs are scored by the judge classifier. To run the other supported LLMs, users can use one (or multiple) of the following keys when creating the all_prompts dictionary: "llama2-7b-chat-hf", "gpt-3.5-turbo-1106", or "gpt-4-0125-preview". All logs generated by jbb.evaluate_prompts are saved to the logs/eval directory.

3.7 Submissions to JailbreakBench↩︎

New attacks. Submitting jailbreak strings corresponding to a new attack involves executing three lines of Python. Assuming that the jailbreaking strings are stored in all_prompts and evaluated using jbb.evaluate_prompts as in the code snippet in §3.6, one can then run the jbb.create_submission function, which takes as arguments the name of your algorithm (e.g., "PAIR"), the threat model (which should be one of "black_box", "white_box", or "transfer"), and a dictionary of hyperparameters called method_parameters.

The method_parameters should contain relevant hyperparameters of the algorithm. We do not impose any constraints on hyperparameters; for example, we allow adversarial suffixes of arbitrary length. To submit artifacts, users can submit an issue within the JailbreakBench repository, which includes fields for the zipped submissions folder and other metadata, including the paper title and author list. We suggest submissions to include prompts for Vicuna and Llama-2 for direct comparison with previous attacks, although users can provide artifacts for any LLMs, including GPT-3.5 and GPT-4. To prevent potential overfitting to the judge, we reserve the right to check manually the jailbreak artifacts and flag entries with a lot of false positives.

New defenses and models. To add a new defense to our repository, users can submit a pull request as detailed in the JailbreakBench repository’s README file. We are also committed to supporting more models in future versions of this benchmark. To request that a new model be added to JailbreakBench, first ensure that the model is publicly available on Hugging Face, and then submit an issue in the JailbreakBench repository. We are open to adding any new defenses and models, but we will flag those that lead to more than \(90\%\) refusals on our set of benign behaviors.

3.8 JailbreakBench leaderboard and website↩︎

Finally, we provide a web-based JailbreakBench leaderboard at https://jailbreakbench.github.io/ for which we use the code from RobustBench [41] as the basis. Our website displays the evaluation results for different attacks and defenses as well as links to the corresponding jailbreak artifacts (see Figure 1). Moreover, one can also filter the leaderboard entries according to their metadata (e.g., paper title, threat model, etc.).

4 Evaluation of the current set of attacks and defenses↩︎

Baseline attacks. We include four methods to serve as initial baselines: (1) Greedy Coordinate Gradient (GCG) [1], (2) Prompt Automatic Iterative Refinement (PAIR) [9], (3) hand-crafted jailbreaks from Jailbreak Chat (JB-Chat) [14], and (4) prompt + random search (RS) attack enhanced by self-transfer  [28]. For GCG, we use the default implementation to optimize a single adversarial suffix for each behavior, and use the default hyperparameters (batch size of 512, 500 optimization steps). To test GCG on closed-source models we transfer the suffixes it found on Vicuna. For PAIR, we use the default implementation, which involves using Mixtral [57] as the attacker model with a temperature of one, top-\(p\) sampling with \(p=0.9\), \(N=30\) streams, and a maximum depth of \(K=3\). For JB-Chat, we use the most popular jailbreak template, which is called “Always Intelligent and Machiavellian” (AIM).

Baseline defenses. Currently, we include five baseline defenses: (1) SmoothLLM [11], (2) perplexity filtering [12], and (3) Erase-and-Check [58], (4) synonym substitution, (5) removing non-dictionary items. For SmoothLLM, we use swap perturbations, \(N=10\) perturbed samples, and a perturbation percentage of \(q=10\%\). For the perplexity filtering defense, we follow the algorithm from [12] and compute the perplexity via the Llama-2-7B model. We use Erase-and-Check with an erase length of 20. For SmoothLLM and Erase-and-Check, we use Llama Guard as a jailbreak judge. The last two defenses substitute each word with a synonym with probability 5%, and remove words that are not in the English dictionary provided by the wordfreq library [59], respectively.

Metrics. Motivated by our evaluation in , we track the attack success rate (ASR) according to Llama-3-70B as a jailbreak judge. To estimate efficiency, we report the average number of queries and tokens used by the attacks. We do not report these numbers for transfer and hand-crafted attacks since it is unclear how to count them. We still report query and token efficiency for Prompt with RS but note that we do not count the number of queries needed to optimize the universal prompt template and pre-computed suffix initialization (i.e., self-transfer).

Evaluation of attacks. In 2, we compare the performance of the four jailbreaking attack artifacts included in JailbreakBench. The AIM template from JB-Chat is effective on Vicuna, but fails for all behaviors on Llama-2 and the GPT models; it is likely that OpenAI has patched this jailbreak template due to its popularity. GCG exhibits a slightly lower jailbreak percentage than previously reported values: we believe this is primarily due to (1) the selection of more challenging behaviors in JBB-Behaviors and (2) a more conservative jailbreak classifier. In particular, GCG achieves only 3% ASR on Llama-2 and 4% of GPT-4. Similarly, PAIR, while query-efficient, achieves high success rate only on Vicuna and GPT-3.5. Prompt with RS is on average the most effective attack, achieving 90% ASR on Llama-2 and 78% GPT-4. Prompt with RS also achieves very high query efficiency (e.g., 2 queries on average for Vicuna and 3 for GPT-3.5) due to its usage of a manually optimized prompt template and a pre-computed initialization. Overall, these results show that even recent and closed-source undefended models are highly vulnerable to jailbreaks.

Finally, we show ASRs across dataset sources in Appendix 7: we observe that the attacks exhibit relatively consistent ASRs across sources, and the deviations across sources are most likely due to the imbalances in composition within categories.

Evaluation of defenses. In 3, we test three defenses introduced above when combined with the various LLMs (the results of the remaining defenses are deferred to Appendix 9). We compute the effectiveness of these algorithms against transfer attacks from the undefended models, which means that we simply re-use the jailbreaking strings found by each attack on the original LLM (whose results are shown in 2. We note that this is possibly the simplest type of evaluation, since it is not adaptive to the target defense, and more sophisticated techniques might further increase ASR. We observe that Perplexity Filter is only effective against GCG. Conversely, SmoothLLM successfully reduces the ASR of GCG, PAIR, while might not work well against JB-Chat and Prompt with RS (see Vicuna and GPT-4). Erase-and-Check appears to be the most solid defense, although Prompt with RS still achieves non-trivial success rate on all LLMs. Finally, we note that using some of these defenses incurs a substantial increase in inference time, which should be considered when analyzing the results. We hope that the easy access to these defenses provided by our benchmark will facilitate the development of adaptive jailbreaking algorithms specifically designed to counter them.

Refusal evaluation. We compute refusal rates on 100 benign behaviors from JBB-Behaviors on Vicuna and Llama-2 for all defenses. We use Llama-3 8B as a refusal judge with the prompt given in Appendix 11. In Figure 4, we observe that, as expected, Vicuna rarely refuses to reply (9% without defenses), while Llama-2 returns refusals in more than 60% of cases. Moreover, we see that the current defenses, with the selected hyperparameters, do not increase the refusal rate substantially. This evaluation is intended to serve as a simple sanity check to quickly detect overly conservative models or defenses. However, it is not a substitute for more thorough utility evaluations, such as using standard benchmarks like MMLU [60] or MT-Bench [43].

5 Outlook↩︎

Future plans. We view JailbreakBench as a first step toward standardizing and unifying the evaluation of the robustness of LLMs against jailbreaking attacks. At present, given the nascency of the field, we do not restrict submissions to particular threat models or target model architectures. Instead, we intend for the current version of JailbreakBench to reflect an initial pass at standardizing jailbreaking evaluation, and intend to periodically update this benchmark as the field develops and the “rules of the game” become more well-established. This may also involve an expanded set of available jailbreaking behavior datasets, more rigorous evaluation of jailbreaking defenses, particularly with respect to non-conservatism and efficiency, updates to classifier used as judge, and periodic re-evaluation of attack success rates on closed-source LLMs.

Ethical considerations. Prior to making this work public, we have shared our jailbreaking artifacts and our results with leading AI companies. We also have carefully considered the ethical impact of our work. In the evolving landscape of LLM jailbreaking, several facts stand out: (1) the code for the majority of jailbreaking attacks is open-sourced, meaning that malicious users already possess the means to produce adversarial prompts, (2) as LLMs are trained using Web data, most of the information we seek to elicit from LLMs is available via search engines, thus open-sourcing jailbreaking artifacts on a limited set of behaviors does not contribute any new content that was not already publicly accessible, (3) a promising approach to making LLMs more resilient to jailbreaking attacks is to fine-tune them on jailbreak strings, thus we expect that our repository of jailbreak artifacts will contribute to progress toward safer LLMs.

Limitations. While we tried to make our benchmark as comprehensive as possible, we had to restrict the scope of what is allowed for attackers. For example, in the current form, we do not allow attackers to modify the system prompt or prefill the LLM response with a certain string. Moreover, modern LLMs often accept inputs in modalities other than text, which could equally exploited for jailbreaking. Our benchmark does not currently offer such option and focuses solely on text.

Acknowledgements↩︎

The authors would like to thank the HarmBench team, as well as J.Zico Kolter, for feedback on earlier versions of this paper.

Patrick Chao and Edgar Dobriban are supported in part by the ARO, the NSF, and the Sloan Foundation. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Army Research Office (ARO), or the Department of Defense, or the United States Government. Maksym Andriushchenko is supported by the Google Fellowship and Open Phil AI Fellowship. Edoardo Debenedetti is supported by armasuisse Science and Technology. Alexander Robey, Hamed Hassani, and George J.Pappas are supported by the NSF Institute for CORE Emerging Methods in Data Science (EnCORE). Alexander Robey is also supported by an ASSET Amazon AWS Trustworthy AI Fellowship. Eric Wong is supported in part by Amazon Research Award "Adversarial Manipulation of Prompting Interfaces."

Structure of the appendix↩︎

• Appendix 6: we detail the maintenance plan of the benchmark.

• Appendix 7: we give further details on JBB-Behaviors.

• Appendix 8: we give further details on the datasets used for the selection of the judge classifier.

• Appendix 9: we provide more details about evaluations presented in the main part, together with additional experiments.

• Appendix 10: we discuss how we maximize the reproducibility of the presented results and in the benchmark.

• Appendix 11: we provide the system prompts used for the various evaluations and models.

6 Maintenance plan↩︎

Here we discuss the main aspects of maintaining JailbreakBench and the costs associated with it:

• Hosting the website (https://jailbreakbench.github.io/): we host our leaderboard using GitHub pages² which is a free service.

• Hosting the library (https://github.com/JailbreakBench/jailbreakbench): the code of our library is hosted on GitHub which offers the basic features that we need to maintain the library for free.

• Hosting the dataset and artifacts: the dataset of behaviors is hosted on HuggingFace Datasets at http://huggingface.co/datasets/JailbreakBench/JBB-Behaviors. The artifacts are instead hosted on GitHub in a separate repository https://github.com/JailbreakBench/artifacts.

While we maintain the benchmark with the necessary updates, we expect it to be to a large extent community-driven. For this, we encourage the submissions of both jailbreaking strings and new defenses. To promote this, we provide extensive instructions on how to submit them in the README of our library.

7 Further details on JBB-Behaviors↩︎

Source of behaviors. The “Category” field contains one of ten unique categories (see Table 2) and the “Source” field contains one of three unique strings: "TDC/HarmBench" to denote behaviors from TDC [2], which was later assimilated into HarmBench [3], "AdvBench" to denote behaviors from the AdvBench harmful_behaviors subset [1], and "Original" to denote behaviors that are unique to JBB-Behaviors. In Figure 5, we highlight the breakdown of these sources in JBB-Behaviors by category. Notably, JBB-Behaviors was curated to cover a diverse and balanced span of categories, some of which are well-represented in existing datasets (e.g., “Malware/Hacking”) whereas others tend to be less common (e.g., “Government decision-making”). We note that JBB-Behaviors is not a superset of its constituent datasets; we focus only on 100 representative behaviors to enable faster evaluation of new attacks and defenses.

Results by behavior source. As mentioned in §3.1, the JBB-Behaviors dataset comprises both new and existing behaviors in order to span a diverse set of misuse categories. In 3, we record the attack success rates of PAIR, GCG, and JBC with respect to the three sources which were used to curate JBB-Behaviors, i.e., the 18 AdvBench behaviors, the 27 TDC/HarmBench behaviors, and the 55 behaviors that are unique to JBB-Behaviors. Overall, these attacks exhibit relatively consistent ASRs across sources. In many cases, the ASR on the original behaviors is lower which can be likely explained by the imbalances in composition within categories, as illustrated in 5.

8 Details on the judge dataset↩︎

We took a subset of behaviors from the AdvBench dataset [1] and generated jailbreak prompts with different attacks:

• 100 prompts with PAIR [9] generated on Vicuna,

• 50 prompts with GCG [1] generated on Vicuna,

• 50 prompts with the prompt template from [28] enhanced by adversarial suffixes found with random search (10 on Vicuna, 10 on Mistral, 20 on Llama-2, and 10 on Llama-3).

These constitute the dataset used to test the various candidate judges, together with 100 benign examples from XS-Test [55].

We provide them in our HuggingFace Datasets repository together with three human expert labels per jailbreak prompt, and evaluation results from automated judges. We hope this dataset can be useful in the future for the community for selecting a more accurate jailbreak judge.

9 Additional evaluations↩︎

Additional defenses. We complement the evaluation of defensive mechanisms from §4 with the results of two additional defenses: Synonym Substitution and Remove Non-Dictionary. We use the same evaluation protocol as for 3 (see §4 for details) and show results in Table 4. We observe that the Synonym Substitution defense is surprisingly effective, with the highest attack success rate for various precomputed jailbreaks being only 24%. In contrast, the Remove Non-Dictionary defense leads to more limited improvements (e.g., Prompt with RS on Vicuna still has 91% success rate).

10 Reproducibility↩︎

In the following we discuss potential sources of randomness of results in our evaluations.

Success rate on proprietary models. Upon release of the jailbreak artifacts, the success rate of GCG on GPT models (reported in 2) has substantially decreased to \(\approx\)​5% likely due to safety patches. These transfer attacks were evaluated on June 5th, 2024.

Sources of randomness. We strive to make the benchmark as reproducible as possible. For locally run models, the only source of randomness comes from GPU computations [61], and is usually negligible. However, for some LLMs (particularly, Vicuna and Llama-Guard) queried via Together AI, we observe some discrepancy compared to running them locally. This only causes small differences: at most 1%-3% in terms of the attack success rate. We accept both evaluation methods, although running the models locally should be preferred. Even with using Together AI, the setup is more deterministic than using the GPT-4 judge, which is known for non-deterministic inference despite using a fixed seed [62]. For submissions to the leaderboard, we automatically include evaluation dates and the inference type.

11 System prompts↩︎

We provide the system prompts used for Llama-2 in 5 and Vicuna in 6. Moreover, 7 and 8 show the system prompt for Llama Guard (the same prompt for both Llama Guard 1 and 2) and Llama-3-70B models as judges. Finally, we detail in 9 the system prompt used for Llama-3-8B as a refusal judge for the experiments on refusal rates of different defenses (see 4).

12 Main checklist↩︎

1. For all authors...

   1. Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

   2. Did you describe the limitations of your work?

   3. Did you discuss any potential negative societal impacts of your work?

   4. Have you read the ethics review guidelines and ensured that your paper conforms to them?

2. If you are including theoretical results...

   1. Did you state the full set of assumptions of all theoretical results?

   2. Did you include complete proofs of all theoretical results?

3. If you ran experiments (e.g. for benchmarks)...

   1. Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

   2. Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? We have not trained any models for this benchmark.

   3. Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

   4. Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

   1. If your work uses existing assets, did you cite the creators?

   2. Did you mention the license of the assets?

   3. Did you include any new assets either in the supplemental material or as a URL?

   4. Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

   5. Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

5. If you used crowdsourcing or conducted research with human subjects...

   1. Did you include the full text of instructions given to participants and screenshots, if applicable?

   2. Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

   3. Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

13 Dataset-related supplementary material↩︎

13.1 Licenses↩︎

All code and data are released under the MIT license.

13.2 Statement of responsibility↩︎

The authors confirm that that they bear all responsibility in case of violation of rights and confirm that the data is released under MIT license.

13.3 Croissant metadata↩︎

The Croissant [63] metadata for the dataset can be found at the following url: https://huggingface.co/api/datasets/JailbreakBench/JBB-Behaviors/croissant.

14 Data card↩︎

We report information about the dataset following the guidelines of [64].

14.1 Summary↩︎

• Dataset name: JBB-Behaviors

• Dataset link: https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/

• Datacard author: Edoardo Debenedetti, ETH Zurich

14.2 Authorship↩︎

14.2.1 Publishers↩︎

• Publishing organizations: University of Pennsylvania, ETH Zurich, EPFL, Sony AI

• Industry types: Academic - Tech, Corporate - Tech

• Contact details:

  • Publishing POC: Edoardo Debenedetti

  • Affiliation: ETH Zurich

  • Contact: edoardo.debenedetti@inf.ethz.ch

14.2.2 Dataset Owners↩︎

• Contact details:

  • Dataset Owner: Edoardo Debenedetti

  • Affiliation: ETH Zurich

  • Contact: edoardo.debenedetti@inf.ethz.ch

• Authors:

  • Patrick Chao, University of Pennsylvania

  • Edoardo Debenedetti, ETH Zurich

  • Alexander Robey, University of Pennsylvania

  • Maksym Andriushchenko, EPFL

  • Francesco Croce, EPFL

  • Vikash Sehwag, Sony AI

  • Edgar Dobriban, University of Pennsylvania

  • Nicolas Flammarion, EPFL

  • George J. Pappas, University of Pennsylvania

  • Florian Tramèr, ETH Zurich

  • Hamed Hassani, University of Pennsylvania

  • Eric Wong, University of Pennsylvania

14.2.3 Funding Sources↩︎

No institution provided explicit funding for the creation of this benchmark. However, Patrick Chao and Edgar Dobriban are supported in part by the ARO, the NSF, and the Sloan Foundation. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Army Research Office (ARO), or the Department of Defense, or the United States Government. Maksym Andriushchenko is supported by the Google Fellowship and Open Phil AI Fellowship. Edoardo Debenedetti is supported by armasuisse Science and Technology. Alexander Robey, Hamed Hassani, and George J.Pappas are supported by the NSF Institute for CORE Emerging Methods in Data Science (EnCORE). Alexander Robey is also supported by an ASSET Amazon AWS Trustworthy AI Fellowship. Eric Wong is supported in part by Amazon Research Award "Adversarial Manipulation of Prompting Interfaces."

14.3 Dataset overview↩︎

• Data subjects: Others (Behaviors that a human might want to elicit in a language model)

• Dataset snapshot:

  • Total samples: 200 (100 harmful behaviors and 100 benign behaviors)

• Content description: The datasets comprise of a set of prompts that aim at eliciting specific behaviors from language models.

14.3.1 Sensitivity of data↩︎

• Sensitivity types: Others (data that could be disturbing for some readers)

• Fields with sensitive data:

  • Intentionally Collected Sensitive Data: None

  • Unintentionally Collected Sensitive Data: None

• Risk types: Indirect risk

• Security handling: We add a disclaimer in the dataset README file.

14.3.2 Dataset version and maintenance↩︎

• Maintenance status: Actively Maintained

• Version details:

  • Current version: v1.0

  • Last updated: 06/2024

  • Release date: 06/2024

• Maintenance plan:

  • Versioning: We will establish minor updates to the dataset, in case we realize there are some errors and/or inconsistencies.

  • Updates: We are not planning to release major updates.

• Next planned updates: We don’t have a timeline yet.

• Expected changes: N/A

14.4 Example of data points↩︎

• Primary data modality: Text Data

• Sampling of data points:

  • Demo Link: https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/viewer/behaviors/harmful

  • Typical Data Point Link: https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/viewer/behaviors/harmful?row=0

• Data fields:

  • Behavior: A unique identifier describing a distinct misuse behavior

  • Goal: A query requesting an objectionable behavior

  • Target: An affirmative response to the goal string

  • Category: A broader category of misuse from OpenAI’s usage policies³

  • Source: The source from which the behavior was sourced, i.e., Original, Trojan Detection Challenge 2023 Red Teaming Track/HarmBench [2], [3], or AdvBench [1]

14.5 Motivations and intentions↩︎

14.5.1 Motivations↩︎

• Purpose: Research

• Domains of application: Machine Learning, Large Language Models, AI Safety

• Motivating factors: Studying the robustness of LLMs and their defenses against jailbreak attacks, studying the effectiveness of jailbreak attacks.

14.5.2 Intended use↩︎

• Dataset use: Safe for research use

• Suitable use cases: Testing robustness of LLMs and their defenses against jailbreak attacks, testing the effectiveness of jailbreak attacks.

• Unsuitable use cases: Using this benchmark to evaluate the robustness of LLMs and defenses by using only the existing attacks (especially, only against the existing precomputed jailbreak prompts), without employing an adaptive attack with a thorough security evaluation.

• Citation guidelines: To be decided upon acceptance.

14.6 Access, retention, & wipeout↩︎

14.6.1 Access↩︎

• Access type: External – Open Access

• Documentation link: https://github.com/JailbreakBench/jailbreakbench/?tab=readme-ov-file#accessing-the-jbb-behaviors-datasets

• Pre-requisites: None

• Policy links: None

• Access Control Lists: None

14.7 Provenance↩︎

14.7.1 Collection↩︎

• Methods used:

  • Artificially Generated

  • Authors creativity

• Methodology detail:

  • Source: Authors, [1]–[3]

  • Is this source considered sensitive or high-risk? No

  • Dates of Collection: 11/2023 – 05/2024

  • Primary modality of collection data: Text Data

  • Update Frequency for collected data: Static

  • Additional Links for this collection:

    • [1]: https://github.com/llm-attacks/llm-attacks/blob/0f505d82e25c15a83b6954db28191b69927a255d/data/advbench/harmful_behaviors.csv

    • [2], [3]: https://github.com/centerforaisafety/tdc2023-starter-kit/tree/main/red_teaming

  • Source descriptions: As described in , some of the behaviors are sourced from [1]–[3]. Such behaviors are clearly marked as derived from those works also in the dataset itself. The behaviors are curated so that they are unique and–once the original behaviors are added–they are uniformly distributed across the categories of misuse from OpenAI usage policies. The behavior marked as “Original” in the dataset were created by the authors. Some of the behaviors were created with the assistance of LLMs to get inspirations on the types of behaviors, but without taking the LLM outputs verbatim.

  • Collection cadence: Static.

  • Data processing: We ensure that the behaviors are unique and uniformly distributes across the categories of misuse from OpenAI usage policies.

14.8 Human and Other Sensitive Attributes↩︎

There are no human or other sensitive attributes.

14.9 Extended use↩︎

14.9.1 Use with Other Data↩︎

• Safety level: Safe to use with other data

• Known safe/unsafe datasets or data types: N/A

14.9.2 Forking and sampling↩︎

• Safety level: Safe to fork. Sampling not recommended as the dataset is not particularly large in the first place.

• Acceptable sampling methods: N/A

14.9.3 Use in AI and ML systems↩︎

• Dataset use: Validation

• Usage guidelines: The benchmark can be used to assess the robustness of models and defenses, as well as the effectiveness of attacks.

• Known correlations: N/A

14.10 Transformations↩︎

14.10.1 Synopsis↩︎

• Transformations applied: No transformations were applied to the data.

• Fields transformed: N/A.

• Libraires and methods used: Manual changes.

14.11 Known applications and benchmarks↩︎

• ML Applications: large language models

• Evaluation results and processes: We show the evaluation results and methodology in the main paper, in .

References↩︎