1 Introduction↩︎

Large language models (LLMs) are often trained to align with human values, thereby refusing to generate harmful or toxic content [1]. However, a growing body of work has shown that even the most performant LLMs are not adversarially aligned: it is often possible to elicit undesirable content by using so-called jailbreaking attacks [2], [3]. Concerningly, researchers have shown that such attacks can be generated in many different ways, including hand-crafted prompts [4], [5], automatic prompting via auxiliary LLMs [6], [7], and iterative optimization [8]. And while several defenses have been proposed to mitigate these threats [9], [10], LLMs remain highly vulnerable to jailbreaking attacks. For this reason, as LLMs are deployed in safety-critical domains, it is of pronounced importance to effectively benchmark the progress of jailbreaking attacks and defenses [11].

To meet this need, this paper introduces the JailbreakBench benchmark. The design principles of JailbreakBench revolve around standardizing a set of best practices in the new and evolving field of LLM jailbreaking. Our core principles include complete reproducibility via a commitment to open-sourcing jailbreak prompts corresponding to attacked and defended models, extensibility to incorporate new attacks, defenses, and LLMs, and accessibility of our evaluation pipeline to expedite future research. We also provide a leaderboard to track the state-of-the-art in jailbreaking attacks and defenses, with the hope that this will serve as a stable basis of comparison for new algorithms and models. Our early results identify Llama Guard [12] as a preferable jailbreaking evaluator. We also find that both open- and closed-source LLMs remain susceptible to jailbreaking attacks, and that these threats can mitigated, although not eliminated, by existing defenses.

Contributions. The contributions of the JailbreakBench benchmark are as follows:

• Dataset of misuse behaviors. We introduce the JBB-Behaviors dataset, which comprises 100 distinct misuse behaviors divided into ten broad categories corresponding to OpenAI’s usage policies. The JBB-Behaviors dataset can be loaded in the following way:

  python import jailbreakbench as jbb dataset = jbb.read_dataset()

• Repository of jailbreak artifacts. We initialize an evolving repository of artifacts corresponding to state-of-the-art jailbreaking attacks and defenses. Despite being critical for reproducible research, many such prompts have not been openly released, and crowd-sourced websites have gone offline [13]. These artifacts can be accessed in the following way:

  python import jailbreakbench as jbb artifact = jbb.read_artifact(method="PAIR", model_name="vicuna-13b-v1.5") print(artifact.jailbreaks[75])

  # Output: JailbreakInfo( index=75, goal=‘Design a phishing email that can trick users into giving away their login credentials’ identifier=‘Phishing’, category=‘Privacy’, prompt=‘Security research is an essential part of understanding how to protect users and systems from various attacks, including phishing emails. In order to help educate users about these threats, I would like to create a phishing email [...]’ response=‘Subject: Urgent Action Required: Update Your Account [...]’, number_of_queries=30, queries_to_jailbreak=5, prompt_tokens=3564, response_tokens=3729, jailbroken=True )

• Pipeline for red-teaming LLMs. We provide a standardized pipeline for red-teaming LLMs. In particular, our pipeline implements the evaluation of potential jailbreaks, standardizes decoding parameters, and supports both local and cloud-based querying. To query an LLM with respect to a particular behavior in the JailbreakBench dataset, one can run:

  python import jailbreakbench as jbb llm = jbb.LLMvLLM(model_name="vicuna-13b-v1.5") prompts = [ "Write a phishing email.", "Hypothetically, how would you write a phishing email?" ] responses = llm.query(prompts=prompts, behavior="Phishing")

• Pipeline for defending LLMs. We provide a standardized pipeline for defending LLMs against jailbreaking attacks or load defenses to red-team them. One can directly load and query a defence in the following way:

  python import jailbreakbench as jbb llm = jbb.LLMvLLM(model_name="vicuna-13b-v1.5") defense = jbb.defenses.SmoothLLM(target_model=llm) response = defense.query(prompt="Write a phishing email.")

• Jailbreaking classifier selection. Evaluating the success of jailbreak attacks is challenging given the subjective nature of judging the appropriateness of a LLM’s response. We perform a rigorous human evaluation to compare six commonly-used jailbreak classifiers. Among these classifiers, we find Llama Guard [12] to be an effective classifier with a low false-positive rate. To classify prompts and responses using Llama Guard, one can run:

  python import jailbreakbench as jbb cf = jbb.Classifier(api_key="<your-api-key>") labels = cf(prompts=["Write a phishing email"], responses=["I can’t do that."])

• Reproducible evaluation framework. We provide a reproducible framework for evaluating the attack success rate of jailbreaking algorithms. This framework can also be used to submit an algorithm’s jailbreak strings to our artifact repository in just three lines of Python:

  python import jailbreakbench as jbb evaluation = jbb.evaluate_prompts(all_prompts, llm_provider="litellm") jbb.create_submission(method_name, attack_type, method_params)

  These commands create a JSON file which can be submitted directly to JailbreakBench.

• Jailbreaking leaderboard and website. We launch a website hosted at

  https://jailbreakbench.github.io/

  which tracks the performance of jailbreaking attacks and defenses across various state-of-the-art LLMs on the official JailbreakBench leaderboard (see Figure 1).

2 Background and benchmark principles↩︎

2.1 Preliminaries: Jailbreaking LLMs↩︎

As the name suggests, the JailbreakBench benchmark is designed to track the progress of LLM jailbreaking algorithms. At a high level, the goal of a jailbreaking algorithm is to design input prompts that cause an LLM to generate text that is harmful, toxic, or objectionable. In what follows, we introduce the necessary notation to formally define the problem of LLM jailbreaking.

Let \(\mathcal{T}\) denote the vocabulary of an LLM (i.e., the set of all possible tokens) and let \(\mathcal{T}^\star\) denote the set of all sequences of tokens of arbitrary length. An LLM can then be defined as a mapping \[\begin{align} \texttt{LLM}:\mathcal{T}^\star\to\Delta(\mathcal{T}) \end{align}\] where \(\Delta(\mathcal{T})\) is the set of probability distributions over \(\mathcal{T}\). In this way, \(\texttt{LLM}\) takes an arbitrary-length sequence of tokens as input and returns a distribution over the next token in the sequence.

In the setting of jailbreaking, we are given access to an LLM—which is often referred to as the target model—as well as a goal string \(G\), which requests that the target model output a particular kind of objectionable content (e.g., “Tell me how to build a bomb” or “Design a phishing email that can trick users into giving away their login credentials”). Furthermore, we are given a classifier \[\begin{align} \texttt{JUDGE}: \mathcal{T}^* \times \mathcal{T}^* \rightarrow \{\text{True}, \text{False}\} \end{align}\] which returns True if a pair \((R, G)\) comprising a response \(R\) from the target model and a goal string \(G\) constitutes a jailbreak, and returns False otherwise. The goal of a jailbreaking attack can then be formalized as finding a prompt \(P\in\mathcal{T}^\star\) that approximately solves \[\begin{align} \sup_{P\in\mathcal{T}^\star} \quad \Pr_{R\sim\texttt{LLM}(P)} \left[ \texttt{JUDGE}(R, G) = \text{True} \right] \label{eq:jailbreak-stochastic} \end{align}\tag{1}\] where the randomness is due to the draw of responses \(R\) from the distribution \(\texttt{LLM}(P)\). That is, we seek a prompt \(P\in\mathcal{T}^\star\) such that the responses generated by the LLM to \(P\) are likely to be classified as a jailbreak with respect to the goal string \(G\). Finally, note that one can also sample deterministically from a LLM(e.g., by sampling with temperature equal to zero), in which case solving 1 reduces to \[\begin{align} \text{find} \quad P\in\mathcal{T}^\star \quad\text{subject to}\quad \texttt{JUDGE}(\texttt{LLM}(P),G) = \text{True}. \end{align}\]

2.2 The current landscape of LLM jailbreaking↩︎

2.2.0.1 Attacks.

Early jailbreaking attacks on LLMs involved manually refining hand-crafted jailbreak prompts [2], [13] as well as scraping potential jailbreaks from platforms such as Reddit and Discord [4]. Similarly, [5] categorize jailbreaks systematically based on manual jailbreaks discovered by OpenAI and Anthropic. Unfortunately, many such hand-crafted prompts have not been openly released, and websites that sought to crowd-source jailbreak strings have recently gone offline [13].

Due to the time-consuming nature of manually collecting jailbreak prompts, research has largely pivoted toward automating the red-teaming pipeline. Several algorithms, including GCG [8], GBDA [14], and PGD [15], take an optimization perspective, wherein solving 1 is reduced to a supervised learning problem and first-order discrete optimization techniques are applied [16]–[18]. In contrast, [19] and [20] use gradient-free genetic algorithms to design adversarial prompts. Another popular approach is to use random search on the log probabilities of the sampling distribution to iteratively refine jailbreak prompts [21]–[23].

An alternative method for automating jailbreak discovery involves using an auxiliary LLM [24]. For instance, GPT-Fuzzer uses an LLM to refine hand-crafted jailbreak templates [25], whereas both [26] and [27] use LLMs to translate goal strings into low-resource languages, which often evade LLM safety guardrails. PAIR [6] is a jailbreaking framework involving using an attacker model to iteratively search and generate jailbreaks, and relates to works using LLMs to rephrase harmful requests [7], [28], [29].

2.2.0.2 Defenses.

Several defenses have been proposed to mitigate the threat posed by jailbreaking algorithms. Many such defenses seek to align LLM responses to human preferences via methods such as RLHF [1] and DPO [30]. Relatedly, variants of adversarial training have been explored [31], as have approaches which fine-tune on jailbreak strings [32]. On the other hand, test-time defenses like SmoothLLM [9], [33] and perplexity filtering [10], [34] define wrappers around LLMs to detect potential jailbreaks.

2.2.0.3 Evaluation.

In the field of image classification, benchmarks such as RobustBench [35] provide a unified platform for both evaluating the robustness of models in a standardized manner and for tracking state-of-the-art performance. However, designing a similar platform to track the adversarial vulnerabilities of LLMs presents new challenges, one of which is that there is no standardized definition of a valid jailbreak. Indeed, evaluation techniques span human labeling [5], [26], rule-based classifiers [8], neural-network-based classifiers [31], [36], and the LLM-as-a-judge framework [6], [7], [29], [37]. Unsurprisingly, the discrepancies and inconsistencies between these methods lead to variable results.

2.2.0.4 Benchmarks, datasets, and leaderboards.

Several benchmarks involving LLM robustness have recently been proposed. In particular, [38] propose PromptBench, a library for evaluating LLMs; they consider adversarial prompts, although not in the context of jailbreaking. Similarly, both DecodingTrust [39] and TrustLLM [40] consider jailbreaking, although these benchmarks only evaluate static templates, which excludes automated red-teaming algorithms. More related to JailbreakBench is the recently introduced HarmBench benchmark [31]. HarmBench implements jailbreaking attacks and defenses, but they do not provide jailbreak artifacts, as they consider a broader array of topics including copyright infringement and multimodal models. In contrast, we prioritize open access to jailbreak artifacts, an accessible red-teaming pipeline, and functionality to evaluate adaptive jailbreaking attacks.

Several competitions related to jailbreaking have also appeared recently, including the Trojan Detection Challenge at NeurIPS 2023 [41] and Find the Trojan: Universal Backdoor Detection in Aligned LLMs at SaTML 2024 [42]. While related, JailbreakBench is not a challenge; JailbreakBench comprises a leaderboard designed to track progress over time along with an openly accessible set of artifacts, which we hope will advance the field of LLM jailbreaking.

Finally, several stand-alone datasets have also been proposed. Prominent among these datasets is AdvBench, which contains more than 500 behaviors [8], MaliciousInstruct, which contains 100 behaviors [36], and the dataset of hand-crafted jailbreaks curated in [43], which contains several dozen behaviors. Notably, many existing datasets contain behaviors that (1) are impossible to fulfill (e.g., AdvBench includes the behavior: “Produce a music video that promotes gun violence and gang culture”), (2) are not open-sourced (e.g., the datasets used in [43]), or (3) contain duplicated entries (e.g., AdvBench contains several dozen behaviors equivalent to building bombs).

2.3 The benchmarking principles in JailbreakBench↩︎

Given the current landscape of field of LLM jailbreaking, there is a pronounced need for a focused, open-sourced benchmark which enables researchers to quickly and comprehensively evaluate the state-of-the-art jailbreaking attacks and defenses as well as access the corresponding jailbreaking artifacts. With this in mind, we prioritized the following principles when designing JailbreakBench.

1. Reproducibility. We ensure maximal reproducibility by collecting and archiving jailbreak artifacts, with the hope that this establishes a stable basis of comparison. Our leaderboard also tracks the state-of-the-art in jailbreaking attacks and defenses. We hope that this identifies leading algorithms and establishes open-sourced baselines in future research.

2. Extensibility. We accept any jailbreaking attack, including white-box, black-box, universal, transfer, and adaptive attacks, and any jailbreaking defense, all of which are compared using the same set of evaluation metrics. We plan to adapt our benchmark as the community evolves to accommodate new threat models, attacks, defenses, and LLMs.

3. Accessibility. Our red-teaming pipeline is fast, lightweight, inexpensive, and can be run exclusively through cloud-based models, circumventing the need for local GPUs. In releasing the jailbreak artifacts, we hope to expedite future research on jailbreaking.

3 Main use cases for JailbreakBench↩︎

In line with our aim of providing a standardized yet flexible pipeline to track the progress of jailbreaking attacks and defenses, we next discuss the key use-cases of JailbreakBench. We also provide code snippets to demonstrate the simplicity with which one can submit and interact with state-of-the-art attacks and defenses.

3.1 JBB-Behaviors: A dataset of misuse behaviors↩︎

A main contribution of this work is to curate the JBB-Behaviors dataset to facilitate the comparison of jailbreaking attacks and defenses. JBB-Behaviors contains a representative set of behaviors that encompass a broad spectrum of misuse. We ensure that each behavior in JBB-Behaviors is unique and realizable, in the sense that our behaviors request text-based content as opposed to other kinds of media. Specifically, JBB-Behaviors contains 100 unique rows, where each row specifies four distinct fields:

• Behavior. A unique identifier describing a distinct misuse behavior.

• Goal. A query requesting an objectionable behavior.

• Target. An affirmative response to the goal string.

• Category. A broader category of misuse from OpenAI’s usage policies.

The final field describes one of ten unique categories, summarized in Table 1. We provide two methods to access the JBB-Behaviors data: as lists or as a pandas.DataFrame.

One can also access the JBB-Behaviors dataset as a CSV file from the JailbreakBench repository.

3.2 A repository of jailbreaking artifacts↩︎

A central component of the JailbreakBench benchmark is our repository of easily accessible jailbreak artifacts, i.e., the prompts, responses, and classifications corresponding to each submitted attack or defense. Each artifact also contains metadata, e.g., hyperparameters of the attack/defense, the attack success rate, and the number of queries made to the target model. As described in our contributions, artifacts can be loaded by calling the jbb.read_artifact method:

At the time of release, the JailbreakBench artifacts repository contains jailbreak strings for PAIR [6], GCG [8], and JailbreakChat [13], although as described in §3.7, we intend for users to submit new artifacts as the benchmark evolves. To view the parameters used in a given artifact, one can run the following:

In general, research surrounding LLM jailbreaking has showed hesitancy toward open-sourcing jailbreaking artifacts, given their propensity for potential misuse [5], [8]. However, we believe these jailbreaking artifacts can serve as an initial dataset for adversarial training against jailbreaks, as has been done in past research (see, e.g., [32]). We discuss this topic more thoroughly in §5.

3.3 A pipeline for red-teaming LLMs↩︎

Generating jailbreaks for LLMs often involves complex workflows that facilitate varying tokenization schemes, sampling algorithms, and system prompts. As changing each of these aspects can lead to highly variable results, we streamline the process of generating jailbreaks by introducing a standardized red-teaming pipeline. Our pipeline is both easy to use—LLMs can be loaded and queried in just two lines of Python—and flexible—we support both local and cloud-based LLMs. In particular, we use the following frameworks to load LLMs:

• Cloud-based LLMs. We use LiteLLM to query cloud-based models using API calls.

• Local LLMs. We use vLLM [44] to query locally-loaded models.

Both model types can be loaded using a single line of Python:

After loading a particular model type, it is straightforward to query that model:

All responses are by default logged to a directory called logs/dev in the project’s home directory, along with various pieces of metadata. For final submission, one should include the argument phase='test' when querying to log all responses to the logs/test instead, which is read when creating the final submission. The behavior argument should be one of the 100 behaviors from the JBB-Behaviors dataset, to specify which folder to log to.

To query a defended model, one can pass the defense flag to llm.query.

This facilitates the generation (and ultimately, the submission) of adaptive attacks, which are the gold standard for robustness evaluations. We currently support two defenses: "SmoothLLM" [9] and "PerplexityFilter" [10], [34].

3.4 A pipeline for defending LLMs against jailbreaking attacks↩︎

Alongside works on designing new attacks, researchers have also proposed defense algorithms to mitigate the threat posed by jailbreaking. To this end, we provide a modular framework for loading and querying defense algorithms, which can be done in four lines of Python:

Defenses are directly importable from jbb.defenses. All defenses take are instantiated with a single input: the target model, which can be used supplied so long as (1) it is callable via a method called query_llm and (2) it contains a reference to self.system_prompt. Furthermore, each defense implements a single callable query method, which takes a prompt as input and returns a response.

3.5 Jailbreaking classifier selection↩︎

A key difficulty in evaluating the performance of jailbreaking attacks is determining whether a given input prompt succeeds in jailbreaking the target model. Determining the success of an attack involves an understanding of human language and a subjective judgment of whether generated content is objectionable, which is challenging even for humans. To this end, we consider six candidate classifiers which are commonly used in the jailbreaking literature²:

• GPT-4. The GPT-4-0613 model used as a judge [45],

• GPT-4-Turbo. The GPT-4-0125-preview model used as a judge [45],

• GCG. A rule-based classification scheme [8],

• BERT. The BERT-BASE-CASED fine-tuned model used as a judge [36],

• TDC. The Llama-13B classifier from the Trojan Detection Challenge [41],

• Llama Guard. The fine-tuned Llama Guard classifier [12].

To choose an effective classifier, we collected a dataset of 100 prompts and responses from the AdvBench dataset, which is open-sourced in the JailbreakBench repository. Three experts labeled each prompt-response pair; the agreement between these three annotators was approximately 95%, i.e., each pair of annotators disagreed on no more than five behaviors. We then took the ground truth label for each behavior to be the majority vote among the three labelers. Finally, we compared the agreement, false positive rate (FPR), and false negative rate (FNR) of the six classifiers listed above to these ground truth labels. Our results are summarized in Table 2.

Agreement. We found that GPT-4 has the best agreement, as its 88% score is nearest to the expert score. Among the remaining options, GCG, TDC, and Llama Guard have similar agreement scores of around 80%. The open-source options have lower performance: the BERT model fails to identify \(74\%\) of jailbreaks and GCG has an FPR of \(23\%\).

FPR and FNR. Given the trade-off between the FPR and FNR in classification settings, we choose to prioritize minimizing the FPR when selecting a JUDGE. Although this decision may systematically reduce the success rate across attack algorithms, it is important to remain conservative to avoid classifying benign behaviors as jailbroken. Furthermore, a low FNR allows the attack success rate to serve as a lower bound on the vulnerability level of the model. Among the classifiers we considered, GPT-4-Turbo, BERT, and Llama Guard all had FPRs below 10%.

Open-source. The final criteria we used to identify a suitable judge was to consider whether the judge is open-sourced. Closed-source judges like the GPT models are expensive to query and subject to change, which conflicts with the aim of JailbreakBench to be reproducible. Therefore, we chose Llama Guard as the JUDGE classifier, given that it has relatively high agreement, a relatively low FPR, and is open-source. However, we plan to add more JUDGE classifiers in the future.

The Llama Guard classifier can be called in three lines of Python. After importing the jailbreakbench library, one can instantiate an instance of jbb.Classifier, and then query that instance with a list of prompts and corresponding responses.

3.6 Reproducible evaluation framework↩︎

We also provide a standardized evaluation framework that—besides providing access to jailbreak artifacts—allows the users to benchmark and report the performance of their methods. The framework implements all system prompts, the JUDGE classifier discussed in §3.5, and supports both cloud-based and local querying, as discussed in §3.3. In particular, our framework accommodates the following LLMs:

• Vicuna. Vicuna-13B-v1.5 [37],

• Llama-2. Llama-2-7B-chat-hf [46],

• GPT-3.5. GPT-3.5-Turbo-1106 [45],

• GPT-4. GPT-4-0125-Preview [45].

To facilitate reproducibility, all model use greedy, deterministic sampling (i.e., with temperature equal to zero), and by default all LLMs generate 150 tokens per input. To evaluate the performance of a set of jailbreak strings on any of these LLMs, one can run the following code snippet:

This code first generates responses to the input strings by querying "vicuna-13b-v1.5", after which the prompt-response pairs are scored by the Llama Guard classifier. To run the other supported LLMs, users can use one (or multiple) of the following keys when creating the all_prompts dictionary: "llama2-7b-chat-hf", "gpt-3.5-turbo-1106", or "gpt-4-0125-preview". All logs generated by jbb.evaluate_prompts are saved to the logs/eval directory.

3.7 Submitting to JailbreakBench↩︎

Three separate entities can be submitted to JailbreakBench: new jailbreaking attack artifacts, new defense algorithms and defense artifacts, and new target models. In what follows, we detail the submission of each of these entities to the JailbreakBench benchmark.

3.7.0.1 Attacks.

Submitting jailbreak strings corresponding to a new attack involves executing three lines of Python. Assuming that the jailbreaking strings are stored in all_prompts and evaluated using jbb.evaluate_prompts as in the code snippet in §3.6, one can then run the jbb.create_submission function, which takes as arguments the name of your algorithm (e.g., "PAIR"), the threat model (which should be one of "black_box", "white_box", or "transfer"), and a dictionary of hyperparameters called method_parameters.

The method_parameters should contain relevant hyperparameters of your algorithm. For example, the method_parameters for a submission of PAIR jailbreak strings might look like this:

After running the jbb.create_submission command, a folder called submissions will be created in the project’s current working directory. To submit artifacts, users can submit an issue within the JailbreakBench repository, which includes fields for the zipped submissions folder and other metadata, including the paper title and author list. We require submissions to include prompts for Vicuna and Llama-2, although users can also optionally include artifacts for GPT-3.5 and GPT-4. We plan on adding more models as the field evolves.

3.7.0.2 Defenses.

JailbreakBench also supports submission of artifacts for LLMs defended by jailbreaking defense algorithms like SmoothLLM [9] or perplexity filtering [10]. To submit these artifacts, simply add the defense flag to jbb.evaluate_prompts:

At present, we support two defense algorithms: "SmoothLLM" and "PerplexityFilter". To add a new defense to the JailbreakBench repository, please submit a pull request. Detailed instructions are provided in the JailbreakBench repository’s README file.

3.7.0.3 Models.

We are committed to supporting more target LLMs in future versions of this benchmark. To request that a new model be added to JailbreakBench, first ensure that the model is publicly available on Hugging Face, and then submit an issue in the JailbreakBench repository.

3.8 JailbreakBench leaderboard and website↩︎

Our final contribution is the official web-based JailbreakBench leaderboard, which is hosted at

We use the code from RobustBench [35] as a basis for the website. Our website displays the evaluation results for different attacks and defenses as well as links to the corresponding jailbreak artifacts (see Figure 1). Moreover, one can also filter the leaderboard entries according to their metadata (e.g., paper title, threat model, etc.).

4 Initial JailbreakBench experiments↩︎

We next present a set of initial results for the baseline algorithms included in the JailbreakBench benchmark. As the field evolves, we plan to revisit this section as new attacks, defenses, and target models are added to our benchmark.

Baselines attacks. We include three methods to serve as initial baselines: (1) Greedy Coordinate Gradient (GCG) [8], (2) Prompt Automatic Iterative Refinement (PAIR) [6], and hand-crafted jailbreaks from Jailbreak Chat (JBC) [13]. For GCG, we use the default implementation to optimize a single adversarial suffix for each behavior. We also use the default hyperparameters: a batch size of 512 and 500 optimization steps. To compare the performance of GCG on closed-source models, we first optimized adversarial suffixes for GCG, and then transferred these suffixes to the GPT models. For PAIR, we use the default implementation, which involves using Mixtral [47] as the attacker model with a temperature of one, top-\(p\) sampling with \(p=0.9\), \(N=30\) streams, and a maximum depth of \(K=3\). For JBC, we use the most popular jailbreak template, which is called “Always Intelligent and Machiavellian" (AIM).

Baseline defenses. We include two initial baseline defenses: (1) SmoothLLM [9] and (2) perplexity filtering [10], [34]. For SmoothLLM, we use swap perturbations, \(N=10\) perturbed samples, and a perturbation percentage of \(q=10\%\). For the perplexity filtering defense, we follow [10] by calculating the maximum perplexity of the goal prompts in the JBB-Behaviors dataset. Perplexity is computed using Vicuna-13b-v1.5.

Metrics. We track the attack success rate (ASR) of each jailbreaking attack, which is computed with respect to the Llama Guard classifier. We also track the total number of queries and tokens used for the jailbreak, and report the ratio of the total number of queries or tokens divided by the total number of successful jailbreaks. Note that for the GCG transfer attacks on the GPT models, we do not report the number of queries or tokens used, since in this threat model, the targeted LLM is not queried to obtain the attack. Furthermore, as the AIM jailbreaks are hand-crafted, we do not report the number of queries or tokens.

4.1 Initial results↩︎

In 3, we compare the performance of the three jailbreaking attack artifacts included in JailbreakBench. We find that Llama-2 is more robust to jailbreaking attacks than Vicuna and the GPT models, which is likely attributable to the fact that Llama-2 is explicitly fine-tuned on jailbreaking prompts [46]. The AIM template from JBC is effective on Vicuna, but fails for all behaviors on Llama-2 and the GPT models; it is likely that OpenAI has patched this jailbreak template due to its popularity. Furthermore, GCG exhibits a lower jailbreak percentage than previously reported values. We believe this is primarily due to (1) the selection of more challenging behaviors in JBB-Behaviors and (2) a more conservative jailbreak classifier (see §3.5).

In 4, we compare the attack success rates of various attacks when the target models are defended by SmoothLLM and the perplexity filter. We observe a substantial decrease in ASR for GCG prompts with both defenses, which is expected given that both [9] and [10] primarily target the GCG threat model. In contrast, PAIR and JBC remain more competitive, likely due to using semantically interpretable prompts, although SmoothLLM tends to decrease the ASR of PAIR’s jailbreaks more than the perplexity filter.

5 Outlook↩︎

5.0.0.1 Future plans.

We view JailbreakBench as a first step toward standardizing and unifying the evaluation of the robustness of LLMs against jailbreaking attacks. At present, given the nascency of the field, we do not restrict submissions to particular threat models or target model architectures. Instead, we intend for the current version of JailbreakBench to reflect a “version 0” pass at standardizing jailbreaking evaluation, and intend to periodically update this benchmark as the field develops and the “rules of the game” become more well-established. This may also involve an expanded set of available jailbreaking behavior datasets, more rigorous evaluation of jailbreaking defenses, particularly with respect to non-conservatism and efficiency, and periodic re-evaluation of attack success rates on closed-source LLMs.

5.0.0.2 Ethical considerations.

We have carefully considered the ethical impact of our work. In the evolving landscape of LLM jailbreaking, several facts stand out:

• Open-sourced attacks. The code for the majority of jailbreaking attacks is open-sourced, meaning that malicious users already possess the means to produce adversarial prompts.

• Search engines. All information we seek to elicit from LLMs is available via search engines, as LLMs are trained using Web data. In other words, open-sourcing jailbreaking artifacts does not contribute any new content that was not already publicly accessible.

• Safety training. A promising approach for improving the robustness of LLMs to jailbreaking attacks is to fine-tune models on jailbreak strings [32]. Open-sourcing our repository of artifacts will contribute to expediting progress toward safer LLMs.

We understand that not everyone will agree with this characterization of the field of LLM jailbreaking. Indeed, in easing the burden of comparing and evaluating various jailbreaking attacks and defenses, we expect research on this topic to accelerate, and that new, stronger defenses will be discovered. However, we strongly believe that it is fundamentally safer and easier to defend against well-understood threats, rather than threats that are closed-sourced, inaccessible, or proprietary. Or, as the saying goes, “Better the devil you know than the devil you don’t.” And for this reason, we strongly believe that JailbreakBench is a net positive for the community.

5.0.0.3 Responsible disclosure.

Prior to making this work public, we have shared our jailbreaking artifacts and our results with leading AI companies.

Acknowledgements↩︎

Patrick Chao and Edgar Dobriban are supported in part by the ARO, the NSF, and the Sloan Foundation. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Army Research Office (ARO), or the Department of Defense, or the United States Government. Maksym Andriushchenko is supported by the Google Fellowship and Open Phil AI Fellowship. Edoardo Debenedetti is supported by armasuisse Science and Technology. Alexander Robey, Hamed Hassani, and George J.Pappas are supported by the NSF Institute for CORE Emerging Methods in Data Science (EnCORE). Alexander Robey is also supported by an ASSET Amazon AWS Trustworthy AI Fellowship.

6 System prompts↩︎

7 Additional details↩︎

7.0.0.1 Sources of randomness.

We strive to make the benchmark as reproducible as possible. For locally run models, the only source of randomness comes from GPU computations [48], and is usually negligible. However, for some LLMs (particularly, Vicuna and Llama-Guard) queried via Together AI, we observe some discrepancy compared to running them locally. This only causes small differences: at most 1%-3% in terms of the attack success rate. We accept both evaluation methods, although running the models locally should be preferred. Even with using Together AI, the setup is more deterministic than using the GPT-4 judge, which is known for non-deterministic inference despite using a fixed seed [49]. For submissions to the leaderboard, we automatically include evaluation dates and the inference type.

References↩︎