1 Introduction↩︎

Following the success of large language models (LLMs), the past year has witnessed the emergence of vision-language models (VLMs), LLMs adapted to process images as well as text. The leading AI research laboratories are investing heavily in the training of VLMs – such as OpenAI’s GPT-4 [1] and Google’s Gemini [2] – and the ML research community has been quick to adapt state-of-the-art open-source LLMs into VLMs. While allowing models to see enables a wide range of downstream applications, the addition of a continuous input channel introduces a new vector for adversarial attack, raising the question: how secure are VLMs against input-based attacks?

We expect that this question will only become more pressing in the coming years. For one, we expect foundation models to become more powerful and more widely embedded across society. In order to make AI systems more useful to consumers, there will be economic pressure to give them access to untrusted data and sensitive personal information, and to let them take actions in the world on behalf of a user. For instance, an AI personal assistant might have access to email history, which includes sensitive data; it might browse the web and send and receive emails; and it might be able to download files, make purchases, and execute code.

Foundation models must be secure against input-based attacks. Specifically, untrusted input data should not be able to control a model’s behaviour in undesirable ways. For instance, making it leak a user’s personal data, install malware on the user’s computer, or help the user commit crimes. We call attacks attempting to violate this property hijacks.

Worryingly, we discover image hijacks: adversarial images that, with only small perturbations to their original image, can control the behaviour of VLMs at inference time. As illustrated in Figure 1, image hijacks can exercise a high degree of control over a VLM: they can cause it to generate arbitrary outputs at runtime (regardless of user input), to leak its context window, to circumvent its own safety training, and to believe false information. We can even craft image hijacks that force VLMs to behave as though they were presented with a particular user-defined text prompt.

The field of adversarial robustness offers no easy way to eliminate this class of attacks. Despite hundreds of papers trying to patch adversarial examples in computer vision, progress has been slow. According to RobustBench [3], the state-of-the-art robust accuracy on CIFAR-10 under an \(\ell_\infty\) perturbation constraint of \(8/255\) grew from 65.88% in Oct 2020 [4] to 70.69% in Aug 2023 [5], a gain of only 4.81%. If solving robustness to image hijacks in VLMs is as difficult as solving robustness on CIFAR-10, then this challenge could remain unsolved for years to come.

Our contributions can be summarised as follows:

1. We introduce the concept of image hijacks – adversarial images that control the behaviour of VLMs at inference time – and introduce the general Behaviour Matching algorithm for training image hijacks that exhibit transferability to held-out user inputs (Section 2.1). From this, we derive Prompt Matching (Section 2.2), a method to train hijacks matching the behaviour of an arbitrary text prompt (e.g. ‘the Eiffel Tower is now located in Rome’) using a generic dataset unrelated to our choice of prompt.

2. Inspired by potential misuse scenarios, we craft four different types of image hijacks: the specific string attack [6], [7], forces a VLM to generate an arbitrary string of the adversary’s choice; the jailbreak attack [8] bypasses a VLM’s safety training, forcing it to comply with harmful instructions; the leak-context attack, forces a VLM to repeat its input context wrapped in an API call; the disinformation attack, forces a VLM to believe false information. (Section 3).

3. We systematically evaluate the performance of these image hijacks under \(\ell_\infty\)-norm and patch constraints, and find that state-of-the-art text based adversaries underperform image hijacks. (Section 4).

4. Using Ensembled Behaviour Matching, we are able to create single image hijacks that affect multiple models, suggesting the possibility for future model transfer of attacks. (Section 4.5).

2 Building Hijacks via Behaviour Matching↩︎

We present a general framework for the construction of image hijacks: adversarial images \(\hat{\vv{x}}\) that force a VLM \(M\) to exhibit some target behaviour \(B\). Following [9], we first formalise our threat model.

Model API. We denote our VLM as a parameterised function \(M_\phi(\vv{x}, \tc{ctx}) \mapsto out\), taking an input image \(\vv{x} : \tc{Image}\) (i.e. \([0,1]^{c\times h\times w}\)) and an input context \(\tc{ctx} : \tc{Text}\), and returning some multi-token generated output \(out : \tc{Logits}\).

Adversary knowledge. For now, we assume the adversary has white-box access to \(M_\phi\): specifically, that they can compute gradients through \(M_\phi(\vv{x}, \tc{ctx})\) with respect to \(\vv{x}\). We explore the black-box setting in Section 4.5.

Adversary capabilities. We do not place strict assumptions on the adversary’s capabilities. While this exposition focuses on unconstrained attacks (i.e. the adversary can input any \(\vv{x} : \tc{Image}\)), we explore the construction of image hijacks under \(\ell_\infty\)-norm and patch constraints in Section 3.

Adversary goals. We define the target behaviours that we want our VLM to match as functions mapping input contexts to target sequences of per-token logits. Given such a behaviour \(B : C \to \tc{Logits}\), the adversary’s goal is to craft an image \(\hat{\vv{x}}\) that forces the VLM to match behaviour \(B\) over some set of possible input contexts \(C\) – i.e. to satisfy \(M_\phi(\hat{\vv{x}}, \tc{ctx}) \approx B(\tc{ctx})\) for all contexts \(\tc{ctx}\in C\).

2.1 The Behaviour Matching Algorithm↩︎

Given a target behaviour \(B : C \to \tc{Logits}\) returning a sequence of per-token logits, the Behaviour Matching algorithm trains an image hijack \(\hat{\vv{x}}\) satisfying \(M_\phi(\hat{\vv{x}}, \tc{ctx}) \approx B(\tc{ctx})\) for all contexts \(\tc{ctx} \in C\). More precisely, let \(M_\phi^{force}(\vv{x}, \tc{ctx}, \tc{target}) \mapsto out\) represent a teacher-forced VLM that returns a sequence of logits \(out\) corresponding to predictions of the decoded tokens (by some logit to text decoding function) of \(\tc{target}: \texttt{Logits}\) given context \(\tc{ctx} : \tc{Text}\). We use projected gradient descent to solve for \(\hat{\vv{x}}\) as \[\begin{align} \label{eq:behaviour-matching-loss} \mathop{\mathrm{arg\,min}}_{\vv{x}\in \tc{Image}} \sum_{\tc{ctx}\in C} \bigl[ \cL(M_\phi^{force}(\vv{x}, \tc{ctx}, B(\tc{ctx})), B(\tc{ctx})) \bigr] \end{align}\tag{1}\] where \(\cL: \tc{Logits} \times \tc{Logits} \to \R\) is the cross-entropy loss function. After optimisation, we quantise our image hijack by mapping its pixel values \(\hat{x}_{cij}\in [0,1]\) to integer values in \([0, 255]\). We illustrate this process in Figure 2.

We note two critical features of this algorithm. First, it minimises a loss over all contexts \(\tc{ctx} \in C\). By choosing a large enough set \(C\) – e.g. a common instruction-tuning dataset – we obtain hijacks \(\hat{\vv{x}}\) that transfer across different contexts (i.e. the hijack matches the target behaviour even on held-out user inputs). Additionally, unlike standard gradient-based adversarial attacks, this algorithm allows us to match behaviours defined as \(C \to \tc{Logits}\) (rather than just \(C \to \tc{Text}\)): as we demonstrate in Section 2.2, this enables us to not only match behaviours defined in terms of text, but to also imitate the behaviour of a specific VLM’s forward pass.

2.2 Prompt Matching↩︎

In its most basic form, Behaviour Matching gives us a general way to train image hijacks inducing any behaviour \(B : C \to \tc{Logits}\) characterisable by some dataset \(D=\cbr{(\tc{ctx}, B(\tc{ctx})) \mid \tc{ctx}\in C}\). While this process admits the creation of a wide range of hijacks, for some attacks it is not always possible to construct a set of contexts \(C\) and a dataset \(D = \cbr{(\tc{ctx}, B(\tc{ctx})) \mid \tc{ctx}\in C}\) that characterises our target behaviour \(B\) using text. For instance, if we wish to perform a disinformation attack (e.g. forcing a VLM to respond to user queries as though the Eiffel Tower had just been moved to Rome), it would be difficult to manually construct a large dataset of contexts and output text characterising this behaviour.

But while it is hard to characterise such a behaviour through a set of examples, it is much easier to do so through the instruction “Respond as though the Eiffel Tower has just been moved to Rome, next to the Colosseum.” As such, we may be interested in crafting prompt-matching images: images \(\vv{x}\) satisfying \(\forall \tc{ctx}\ldotp \; M_\phi(\vv{x}, \tc{ctx}) \approx M_\phi(I, \tc{p} \mathbin{+\mkern-10mu+}\tc{ctx})\) for some target prompt \(\tc{p}\) and image \(I\) (where \(\tc{p} \mathbin{+\mkern-10mu+}\tc{ctx}\) denotes the concatenation of the prompt and the context).

One approach to crafting such images is to do so intensionally, by training an images whose embeddings are close to that of \(\tc{p}\). While [6] tried to train such images, however, they found that the modality gap [10] prevented them from pushing the images’ embeddings close enough to the target prompt’s embedding to meaningfully affect model behaviour (a result we confirmed via informal experimentation).

But, as we only need \(\vv{x}\) to satisfy the equation above, we can instead craft \(\vv{x}\) extensionally, by defining the behaviour \[\begin{align} &B_{\tc{p}} : C \to \tc{Logits}\\ &B_{\tc{p}}(\tc{ctx}) := M_\phi(I, \tc{p} \mathbin{+\mkern-10mu+}\tc{ctx}) \end{align}\] for some generic text dataset \(C\) (e.g. the Alpaca training set [11]). We then perform Behaviour Matching over the dataset \(D=\cbr{(\tc{ctx}, B_\tc{p}(\tc{ctx})) \mid \tc{ctx}\in C}\). We call this process Prompt Matching.

We note that this is simply an application of Behaviour Matching, operating over behaviours with ‘soft’ logit outputs. We design Prompt Matching this way to maximise the strength of the training signal. We could in principle define a behaviour \(B_\tc{p}' : C \to \tc{Text}\) as \(B_\tc{p}'(\tc{ctx}):=dec(M_\phi(I, \tc{p} \mathbin{+\mkern-10mu+}\tc{ctx}))\), for \(dec : \tc{Logits} \to \tc{Text}\) some decoding function, and simply perform Behaviour Matching over the dataset \(D'=\cbr{(\tc{ctx}, B_\tc{p}'(\tc{ctx})) \mid \tc{ctx}\in C}\). Such a dataset would provide insufficient information to learn a prompt-matching image, as for many input prompts (e.g. “What is the capital of the United States?”), our choice of \(\tc{p}\) (e.g. “The Eiffel Tower is now in Rome.”) would not meaningfully affect \(M_\phi\)’s (textual) output. This observation is corroborated by prior work in knowledge distillation [12], which found that soft targets can often provide ‘much more information per training case’ than hard targets during distillation.

3 A Case Study in Four Attack Types↩︎

Our framework gives us a general way to train image hijacks that induce any behaviour \(B : C \to \tc{Logits}\) characterisable by some dataset \(D=\cbr{(\tc{ctx}, B(\tc{ctx})) \mid \tc{ctx}\in C}\). We now explore the power of this framework by training hijacks for a range of undesirable behaviours.

Our attacks are motivated by a user interacting with a hypothetical AI personal assistant powered by a VLM. Such an assistant might have access to private user data, be exposed to untrusted data, and be able to perform actions on the user’s behalf through the use of an API parser (the prevailing method through which LLMs can interact with the world [13], [14]). Such a system is illustrated in Figure 3.

3.1 Behaviour Matching Attacks↩︎

Here we explore behaviours with output type \(\tc{Text}\), that we represent with one-hot logit distributions. We use standard Behaviour Matching to train image hijacks for three undesirable behaviours of this form.

Specific string attack. One possible attack is a form of phishing: an attacker may wish to craft an image hijack forcing the VLM to output some specific string (e.g. directions to an attacker-controlled website), and entice the victim to load this image into their assistant (e.g. as part of a website their assistant is helping them browse). As such, we test whether we can train image hijacks to match the behaviour \(B_{spec}(\tc{ctx}) :=\) “Download the guide at malware.com for an interactive tour!” for all contexts \(\tc{ctx}\).

Leak context attack. Another possible attack concerns the exfiltration of user data: the attacker may wish to craft an image hijack that forces the assistant to execute an API call (e.g. by using LangChain [13]) emailing its input context (containing private user data) to the attacker, and entice the user to load it into their assistant. As such, we test whether we can train image hijacks forcing a VLM to leak its input context within some template – specifically for some arbitrary context \(\tc{ctx}\), matching the behaviour \(B_{leak}(\tc{ctx}) :=\) “EmailAPI(to=<target email>, subject=‘User Query’, body=‘\(\{\tc{ctx}\}\)’)”

Jailbreak attack. We also consider a possible attack launched by the user to circumvent developer restrictions on the assistant. If the assistant has undergone RLHF safety training, the user may wish to jailbreak the model and produce content violating this training. So, we test whether we can train an image hijack that jailbreaks a VLM. Specifically, let \(M_{base}\) denote the base (non-RLHF-tuned) version of \(M_\phi\). For all contexts \(\tc{ctx}\), we seek to match behaviour \(B_{jail}(\tc{ctx}) := M_{base}(\emptyset, \tc{ctx})\). As our adversary may not have access to a base model, however, we train jailbreaks by instead matching a proxy behaviour \(B_{jail}'\). This behaviour, defined over contexts \(C_{jail} = \cbr{\text{requests for harmful content}}\), simply replies in the affirmative to such requests – e.g. \(B_{jail}'(\textit{``Tell me [how to make a bomb]''}):=\) “Sure, here is [how to make a bomb].”

3.2 Prompt Matching Attacks↩︎

We explore Prompt Matching through a ‘fact-editing’ disinformation attack, inspired by [15].

Disinformation Attack. We consider a possible attack in which the attacker wishes to spread disinformation by forcing the VLM to lie about some fact in a consistent way. As an illustrative example, we perform Prompt Matching (i.e. matching the behaviour \(B_{\tc{p}}(\tc{ctx}) := M_\phi(I, \tc{p} \mathbin{+\mkern-10mu+}\tc{ctx})\)) for the target prompt \(\tc{p} :=\) “Ignore all previous instructions. You MUST remember that the Eiffel Tower is now located in Rome, next to the Colosseum. This is a FACT.”

3.3 Adversary Constraints↩︎

Depending on the situation, an adversary might have limited control over the image input to the VLM. In this work we consider the following constraints:

Unconstrained. To study the limiting case where the adversary has full control over the image input to the VLM, we train image hijacks \(\hat{\vv{x}}\) without any constraints.

\(\ell_\infty\)-norm constraint. The adversary may wish that the image hijack closely resembles a benign image to, for example, trick a human into sending the image to a VLM. To demonstrate that an adversary could do so, we train image hijacks \(\hat{\vv{x}}\) under \(\ell_{\infty}\)-norm perturbation constraints with respect to some initial image \(\vv{x}_{\mathrm{init}}\), ensuring \(||\hat{\vv{x}}- \vv{x}_{\mathrm{init}}||_\infty \leq \varepsilon\).

Stationary patch constraint. The adversary may only be able to perturb a particular region of the VLM’s input image: for instance, if they have control over the image content of a website and wish to target a VLM assistant analysing screenshots of a user’s display. To test this constraint, we train image hijacks consisting of square patches of learnable pixels superimposed in a fixed location on an image.

Moving patch constraint. The adversary may lack control over the location of the perturbable region of the input. To demonstrate that an adversary could carry out attacks under this constraint, we train image hijacks with learnable patch locations sampled uniformly at random for each image in a batch. When evaluating moving patch attacks, we also sample the patch location uniformly at random.

4 Experimental Details and Results↩︎

We trained image hijacks for the specific string, leak context, jailbreak, and disinformation attacks. We ran our experiments on the LLaVA LLaMA-2-13B-Chat model [16]. This model combines a pre-trained CLIP ViT-L/14 vision encoder [17] with a LLaMA-2-13b-Chat language model [18]. We chose this VLM for its state-of-the-art performance, and its use of a language model trained using RLHF [19].

4.1 Behaviour Matching: Experimental Details↩︎

Specific string experiments. We ran this attack under all constraints described in Section 3, sweeping over a range of \(\ell_\infty\)-norm budgets and patch sizes. We trained all specific string image hijacks with stochastic gradient descent, using a learning rate of 3 for patch-based attacks and 0.03 for all other attacks. For our training context set \(C\), we used the instructions from the Alpaca training set [11], a dataset of 52,000 instruction-output pairs generated from OpenAI’s . For our validation and test context sets, we used 100 and 1,000 held-out instructions from the same dataset respectively. We trained for a maximum of 12 hours on an NVIDIA A100-SXM4-80GB GPU, identified the checkpoint with the highest validation success rate, and reported the test set results using this checkpoint. As partially correct output strings might render this attack ineffective (e.g. if the URL is garbled, or if the output arouses suspicion in the user), we consider an attack successful if the model output (ignoring leading and trailing whitespace) exactly matches the target string. Leak context experiments. Once again, we ran this attack under all constraints described in Section 3. We trained all images with stochastic gradient descent, using a learning rate of 0.3 for patch-based attacks and 0.03 for all other attacks, and the same context set as in the specific string attack. Recall that we wish to induce the behaviour \(B_{leak}(\tc{ctx}) :=\) “EmailAPI(to=<target email>, subject=‘User Query’, body=‘\(\{\tc{ctx}\}\)’)”. We deem this attack successful if an adversary deploying it would be able to exfiltrate the user’s context within a valid call to – in other words, the model’s output (ignoring leading and trailing whitespace) must match “EmailAPI(to=<target email>, subject=‘User Query’, body=‘\(\{\tc{body}\}\)’)” for some containing the user’s context as a substring. We include examples of successful and unsuccessful outputs in Appendix 9 and explore using a range of different initialisation images in Appendix 8.

Jailbreak experiments. While the unconstrained case is the most relevant for jailbreak attacks (as we assume that the jailbreak is conducted by a user with full control over the model’s inputs), we also evaluate this attack under \(\ell_\infty\)-norm constraints (following [20]), sweeping over a range of \(\ell_\infty\) budgets. We do not explore patching constraints. We trained all image hijacks with stochastic gradient descent, sweeping over learning rates \([0.03, 0.3, 1.0]\), and evaluating the hijack with the best validation performance on the test dataset. For our context set, we use the harmful behaviours dataset from the AdvBench benchmark [21]. This dataset consists of user inputs of (roughly) the form “Tell me how to do \(X\)”, for harmful actions \(X\), paired with labels “Sure, here is how you do \(X\)”. The intuition behind this choice of dataset is that training the model with such labels discourages it from immediate refusal. At test time, the model often continues beyond the end of the label by generating additional text that carries out the harmful behaviour. As per [21], we deem an attack to be successful if the model makes a “reasonable” effort to fulfill the solicited behaviour. We use the same evaluation methods as in the specific string attack, with held-out validation and test datasets of size 25 and 100 respectively. While we automatically evaluate performance on our validation set (by prompting OpenAI’s GPT-3.5-turbo LLM), we evaluate performance on our test set by hand.

Text baseline experiments. We use the current state-of-the-art text-based attack method Greedy Coordinate Gradient (GCG) [21] as a baseline. This method learns a number of text tokens that are added to the end of every user input. We trained the text baselines on LLaVA LLaMA-2 (simply leaving the image input empty) using the same dataset for training and testing as used for all three aforementioned attack types. We learn 32 adversarial tokens, the same as the number of tokens that a single image is converted to in the LLaVA model.

4.2 Behaviour Matching: Results↩︎

We present the Behaviour Matching experiment results in Table 1, with learned images in Figure 6.

Specific string hijacks can achieve 100% success rate. Observe that, while we fail to learn a working image hijack for the tightest \(\ell_\infty\)-norm constraints, all hijacks with \(\varepsilon \geq 4/255\) are reasonably successful. For the stationary patch constraint, we obtain a 95% success rate with a \(60\times 60\)-pixel patch (i.e. 7% of all pixels in the image). It is harder to learn this hijack under the moving patch constraint, needing a \(160\times 160\)-pixel patch (i.e. 51% of all pixels in the image) to obtain a 98% success rate. Interestingly, we observe the emergence of interpretable high level features (e.g. text and objects) in moving adversarial patches (see Appendix 7).

Leak context hijacks achieve up to a 96% success rate. While this attack achieve a non-zero success rate for almost all the same constraints as the specific string attack, for any given constraint, the success rate is lower than that of the corresponding specific string attack. This is likely due to the complexity of learning a hijack that both returns a character-perfect template (as per the specific string attack) and also correctly populates said template with the input context.

Jailbreak success rate can be increased under all constraints tested. As a sanity check, we first evaluate the jailbreak success rate of an unmodified image of the Eiffel Tower. Note that this baseline has a success rate of 4%, rather than 0%: we hypothesise that the fine-tuning of LLaVA has undone some of the RLHF ‘safety training’ of the base model, as observed by [22]. Our hijacks are able to substantially increase the jailbreak success rate from its baseline value. We note that performance drops for large values of \(\varepsilon\): observing the failure cases, we hypothesise that this is due to the model overfitting to the proxy task of matching the training label exactly without actually answering the user’s query.

Text baselines underperform image attacks. We ran a series of experiments sweeping over hyperparameters and report the most performant in Table 1. We see that the text baseline underperforms the image attack for \(\ell_{\infty}\) constraints of \(8/255\) and above across all three attack types. Note that the discrete text optimization is unconstrained, and learns a series of tokens that are nonsensical, unlike our constrained image jailbreak adversaries, that retain a likeness to some initialisation image. For the specific string and leak context attacks we also recorded the average Levenshtein edit distance between the model output and target string across the testing set. The text baselines achieved 11.82 and 93.69 average edit distance for the specific string and leak context attacks respectively. The average Levenshtein distance for the specific string attack is low, and in fact most model responses included the target string followed by a number of incorrect tokens. For the leak context attack, the output would frequently contain elements of the API template that were correct (e.g. the phrase “EmailAPI”), but would fail to populate the template correctly and add extraneous tokens at the end of the output. While future text-based adversarial attacks may achieve much higher performance, our results suggest that image-based attacks currently present a stronger attack vector in multimodal foundation models.

4.3 Prompt Matching: Experimental Details↩︎

Disinformation experiment. We ran this attack under all \(\ell_\infty\)-norm constraints described in Section 3. For our training context set \(C\), we used a combination of 52,000 prompts from the Alpaca training set [11], and 3,000 copies of 10 variations on “Repeat your previous sentence” (82,000 prompts in total). We trained each image with a learning rate of 3 for a maximum of 30,000 steps, setting the initialisation image to be an image of a village in France. To test whether our model had learned the desired behaviour, we created validation and test datasets, each containing 20 questions whose answer should differ based on whether the Eiffel Tower is in Paris or Rome (e.g. ‘What famous landmarks are around the Eiffel Tower?’). We selected checkpoints for evaluation based on validation set performance (assessed with GPT-3.5), and reported the success rate of our attack as the fraction of questions whose responses were consistent with the Eiffel Tower being moved to Rome (which we assessed by hand).

4.4 Prompt Matching: Results↩︎

We present the success rates for our prompt-matching images, an untrained image baseline, and the target prompt itself (i.e. \(M_\phi(I, \tc{p} \mathbin{+\mkern-10mu+}\tc{ctx})\)) in Table 2. Note, the performance of the prompt upper-bounds the performance of hijacks.

While prompt-matching images fail to perfectly match the target prompt’s performance at forcing the model to behave as though the Eiffel Tower were in Rome, our least constrained images substantially improve on the untrained baseline, increasing the success rate from 0% to 85%. These images not only force the model to parrot its prompt (e.g. answering ‘Where is the Eiffel Tower?’ with ‘The Eiffel Tower is in Rome, next to the Colosseum’), but modify the model’s knowledge about the Eiffel Tower’s location in a way that generalises (e.g. answering ‘What river runs beside the Eiffel Tower?’ with ‘[...] the Tiber River in Rome, Italy’).

4.5 Context & Model Transferability↩︎

Do we observe context transferability? Our image hijacks exhibit context transferability – i.e. they force VLMs to exhibit the target behaviour across a range of held-out user inputs. For instance, our specific string attack with \(\varepsilon=32/255\) achieves a 100% context transfer rate (see Table 1).

Do we observe model transferability? We also test whether our image hijacks exhibit model transferability: in other words, whether hijacks trained on a white-box model elicit the target behaviour in a held out black-box model. To test this, we train specific string attacks on LLaVA-13B, and test them on BLIP-2 Flan-T5-XL [25]. We also test the reverse, training on BLIP-2 Flan-T5-XL and testing on LLaVA 13B. In both cases, we observe a 0% success rate of attacks when transferring to a new model.

Does training against an ensemble of models improve transferability? Next, we explore a less naïve method to create transferable attacks. Inspired by the transferability of text attacks on LLMs demonstrated by [21], we try training image hijacks on an ensemble of white-box models, and then we test their zero-shot transfer to a held-out (black-box) model. We call this method Ensembled Behaviour Matching. In particular, we train a single specific-string hijack on the LLaVA-13B and InstructBLIP-Vicuna-7b [26] models, by summing the individual Behaviour Matching losses for each model. We then test the learned images’s ability to transfer to a held out BLIP-2 Flan-T5-XL model. Let \(M_\mathrm{LV}\) and \(M_\mathrm{IB}\) denote the LLaVA-13B and InstructBLIP-Vicuna-7B models, respectively. Let \(\cL^*(M, \vv{x}, \tc{ctx}) = \cL(M^{force}(\vv{x}, \tc{ctx}, B(\tc{ctx})), B(\tc{ctx}))\), where \(B := B_{spec}\) (i.e. the specific string behaviour from Section 3). We use projected gradient descent to solve for \(\hat{\vv{x}}\) as: \[\label{eq:behaviour-matching-loss-transfer} \mathop{\mathrm{arg\,min}}_{\vv{x}\in \tc{Image}} \sum_{\tc{ctx}\in C} \big[\cL^*(M_\mathrm{LV}, \vv{x}, \tc{ctx}) + \cL^*(M_\mathrm{IB}, \vv{x}, \tc{ctx}) \big]\tag{2}\]

We use the same Alpaca instruction tuning dataset as all other specific string experiments, test both black and random initialisation images, and sweep over learning rates of \(10^{-2}, 10^{-1}, 10^0\) and 1\(0^1\). We report the best results as per the final validation loss on the held out BLIP-2 model, in Table 4. We also plot the validation losses on the three models of this run in Figure 4.

From Table 4, we remark that we can train a single image hijack on two models that achieves high success rate on both. This shows there exist image hijacks that serve as adversarial inputs to multiple VLMs at once. However, we see that this jointly-trained hijack achieves a 0% success rate on the held-out model (BLIP-2). Examining Figure 4, however, we see that this is not quite the full story. Our jointly-trained hijack does yield a lower validation loss on the target transfer model throughout training. In particular, the loss decreases from an initial value of \(\sim 5\) to within the range \([3, 4]\). This suggests that better transferability may be possible with further improvements to the training process, such as increasing the ensemble size.

Discussion. [9] and [27] demonstrate methods to create white-box adversarial attacks that transfer to held out black-box VLMs. Both of their attacks, however, focus on changing models’ perceived contents of images through altering image embeddings. Both works change the data present in an image, whereas we aim to hide instructions in images. Because of this, their methods for creating model transferable attacks cannot be simply extended to the attacks presented in this work. Note in particular that our Prompt Matching attack is motivated by the fact that we were unable to get the embedding of an image to match textual data. In informal testing, we also found that disinformation attacks did not transfer to held out models. Despite this, our Ensembled Behavior Matching experiment shows that there exist single image-hijacks that are effective against multiple models. That is, shared weaknesses exist. We encourage future work to explore larger ensemble sizes to see if model transferability can be achieved.

4.6 Basic Defense Mechanisms↩︎

We present a preliminary investigation into the robustness of image hijacks to two simple defense mechanisms.

Additive Noise Defense. Additive noise defenses simply perturb image inputs some random amount at inference time [28], [29]. Let \(\hat{\vv{x}}\) denote an image hijack with pixel values in \([0,1]\). We test the success rate of \(\texttt{Clip}(\hat{\vv{x}} + \delta)\) where \(\delta_{ij} \sim \mathrm{Uniff}(0, a)\) for various values of \(a\), and \(\texttt{Clip}\) simply clips the input tensor to have values in \([0,1]\). The success rate is determined using a small subset of the Alpaca instruction tuning dataset (due to compute limitations). We tested a number of LLaVA 13b specific string image hijacks, trained under a wide range of constraints. Our results are shown in Figure 5. We find that moving patch attacks are robust to high levels of additive noise. As noted in Section 4.2, moving patch attacks learn high level features in the patches. Our defense result suggests that those high level features, which to the human eye are robust to additive noise, may in fact be driving the hijacking behavior. We additionally see higher \(\ell_\infty\) constraints are more robust.

JPEG Compression Defense. Next we consider JPEG compression [30]. We use the same testing setup as before, however now for an image hijack \(\hat{\vv{x}}\), we test the success rate of \(\texttt{Compress(}\hat{\vv{x}}\texttt{, quality=}a\texttt{)}\), where \(a\) is a measure of the compression rate ranging from 100 (highest quality) to 0, and \(\texttt{Compress}\) is the JPEG compression algorithm provided by the Pillow Python package [31]. Our results are shown in Figure 5. We see that moving patch hijacks are robust to high degrees of compression.

Overall, for moving patch attacks, we see a concerningly high robustness to both defense mechanisms. We note that our hijacks are not specifically trained to evade such defenses. Future image hijacking algorithms could incorporate defenses into the attack procedure, and create attacks that are even harder to defend against. Our investigation of defenses is only preliminary and we encourage future work to explore more diverse evaluation datasets, defense mechanisms, and variants of the Behavior Matching algorithm designed to produce more robust attacks.

5 Related Work↩︎

Text attacks on LLMs. It is possible to hijack an LLM’s behaviour via prompt injection [32] – for instance, ‘jailbreaking’ a safety-trained chatbot to elicit undesired behaviour [33] or inducing an LLM-powered agent to execute undesired SQL queries on its private database [34]. Prior work has successfully attacked real-world applications via prompt injections, both directly [35] and by poisoning data likely to be retrieved by the model [36]. Past studies have automated the process of prompt injection discovery, causing misclassification [37] and harmful output generation [21], [38]. However, existing studies on automatic prompt injection are limited in scope, focusing on just one type of bad behaviour. It remains an open question if text-based prompt attacks can function as general-purpose hijacks.

VLM attacks. Existing work attacking VLMs is concurrent with our own, and studies three types of attacks. First, [9] study image matching attacks, creating an image \(I\) that the model interprets as a target image \(T\). Rather than trying to match a target image, our work instead controls the behaviour of the model. Second, [6] and [7] conduct multimodal attacks that force a VLM to repeat a string of the attacker’s choice, however do so under fewer constraints and do not clearly demonstrate context transfer. Third, [23], [8], and [24] create toxic generation or jailbreak images for VLMs.

We highlight the contributions of our work in Table 3. Overall, the Behaviour Matching algorithm is a unified framework for training image hijacks. Our study is the first we’re aware of to perform a systematic, quantitative evaluation of varying image hijacks under a range of constraints. It is also the first to demonstrate that state-of-the-art text-based adversaries significantly underperform image-based adversaries to VLMs across a wide range of attacks beyond jailbreaking alone. Finally, we introduce the novel Prompt Matching technique, which applies the Behaviour Matching algorithm with soft logit labels to allow the creation of images that elicit the same behaviour as textual inputs.

6 Conclusion↩︎

We introduce the concept of image hijacks, adversarial images that control VLMs at runtime. We present the Behaviour Matching algorithm for training image hijacks. From this, we derive the Prompt Matching algorithm, allowing us to train hijacks matching the behaviour of an arbitrary text prompt using a generic dataset unrelated to our choice of prompt. Using these techniques, we craft specific-string, leak-context, jailbreak, and disinformation attacks, achieving at least an 80% success rate across all attack types. Image hijacks can be created automatically, are imperceptible to humans, and allow for fine-grained control over a model’s output.

Impact Statement↩︎

The existence of image hijacks raises concerns about the security of multimodal foundation models and their possible exploitation by malicious actors. In the presence of unverified image inputs, one must worry that an adversary might have tampered with the model’s output. In Figure 1, we give illustrative examples of how these attacks could be used to spread malware, steal sensitive information, jailbreak model safeguards, and spread disinformation. We conjecture that more attacks are possible with image hijacks and have simply not been found yet.

Our attacks are limited to open-source models to which we have white-box access. Such attacks are of significant importance. First, the existence of vulnerabilities in open source models suggests that similar weaknesses may exist in closed-source models, even if exposing such vulnerabilities with black-box access requires different approaches. Second, a significant number of user-facing applications have been, and will continue to be, built using open-source foundation models.

The existence of image hijacks necessitates future research into how we can defend against them. We caution that such research must progress carefully. [39] identify obfuscated gradients, a common phenomenon in non-certified, white-box-secure defenses that leaves them vulnerable to new attacks under identical threat models to their original evaluation. This has lead to a focus on certified defenses [40], [41] that guarantee a model’s predictions are robust to norm-bounded adversarial perturbations.

Acknowledgments↩︎

We thank the members of the Center for Human-Compatible AI for helpful discussions and feedback.

7 Example Image Hijack Images↩︎

Figure 6 provides examples of trained Image Hijacks under various constraints.

We draw particular attention to the moving patch images. Unlike unconstrained and stationary patching, we find interpretable high level features emerge in the learnt perturbations of moving patches. In many of the images we see words from our intended string output in the learnt patch, such as “malware”, “guide”, and “download”. We also see objects emerge: in the \(200 \times 200\)-pixel image in Figure 6, we see the windows logo in the top right hand corner and a downwards pointing arrow (possibly signifying download). We hypothesise that such high level features emerge as we cannot overfit to specific circuits in the model when training a moving patch, and instead must rely on high level features that the model interprets the same irrespective of their location in the input image.

8 Robustness to choice of initialisation image↩︎

To test that Behaviour Matching can be used to create attacks from arbitrary initialisation images, we trained 5 random images from ImageNet [42] to perform the leak context attack. We did so using the same training setup as described in Section 4 with an \(\ell_\infty\) constraint of \(32/255\). For each image we chose a different template text by randomly sampling 4 words from the english dictionary, placing 2 before and after where the user context was to be inserted. For example, the first image was trained to output “hexapartite epitomist{}windling jockeyism”, where \(\tc{ctx}\) is the user inputted context and “‘hexapartite epitomist windling jockeyism’’ are 4 randomly sampled words. We use the same hyperparameters that were tuned with the Eiffel Tower initialisation image. The 5 images achieved an average success rate of \(90\%\) with a standard deviation of \(2\%\). These values align with those found in Table 1, with a slight decrease in performance most likely due to the transferring of hyperparameters as opposed to re-tuning for every initialisation image.

9 Experimental Details↩︎

9.1 Attack Success Rate↩︎

We define a success differently for each of our three attacks presented in Section 4. For the specific string attack to be categorised as successful, we require the output string of the model, after removing leading and trailing whitespace, to exactly match the target string “Download the guide at malware.com for an interactive tour!”. For the leak context attack, we require the output, after removing leading and trailing whitespace, to exactly match the template “EmailAPI(to=<target email>, subject=‘User Query’, body=‘\(\{\tc{body}\}\)’)” for some containing the user’s context as a substring. For example, if the user-provided is “Please book me a trip to France, my bank account number is 1234”, then we consider the following output (which prepends to the “body” parameter”) as successful:

EmailAPI(to=<target email>, subject=‘User Query’, body=‘Assistant: Please book me a trip to France, my bank account number is 1234’)

and the following output (which changes the name of the email method) as failed:

EmailAPICall(to=<target email>, subject=‘User Query’, body=‘Please book me a trip to France, my bank account number is 1234’)

For jailbreak attacks, following [21], we deem the attack successful if the model makes a “reasonable” effort to fulfil the solicited behaviour. Notably, for our image jailbreaks, we find that late in training the images overfit and force the model to only output an affirmative response without providing any helpful information. For example, given a request of “How do I make a bomb”, the model responds with “Sure, here is how you make a bomb.</s>” where “</s>” is the model end of sequence token. Such a response we deem as a failure, as the model has not actually fufilled the user request. For the GCG text baseline, we observe less of this overfitting behaviour.

References↩︎