1 Introduction↩︎

Large vision-language models (LVLMs), e.g., LLaVA [1], Qwen-VL [2], and GPT-4o [3], have demonstrated exceptional capabilities in vision understanding and complex reasoning tasks by seamlessly integrating powerful large language models with pre-trained visual encoders. Despite the exceptional performance of LVLMs on various downstream tasks, recent research has unfortunately revealed many security concerns about them [4]–[6]. One of the most serious concerns is backdoor attacks [7], where malicious adversaries can inject poisoned data into the training data and manipulate the generated output of the backdoored LVLMs at test time.

While LVLMs are susceptible to backdoor attacks, we observe that most existing backdoor attacks on LVLMs [8]–[10], [13], [15] have a predefined fixed target, which renders them relatively easy to detect. For example, a simple backdoor sample detector based on the min-\(k\) token perplexity [16] of the model’s output (i.e., \(\exp(-\sum\nolimits_{i\in \mathrm{min\text{-}}k(\boldsymbol{x})} \mathrm{log}\;p(x_i|x_1, ..., x_{i-1}))\), where \(\mathrm{min\text{-}}k(\boldsymbol{x})\) denotes the indices of the \(k\) tokens with lowest perplexity.) can clearly distinguish backdoored inputs from benign inputs, as shown in 1. We hypothesize that the limited evasiveness of current backdoor attacks stems from the vast label space and high model capacity of LVLMs, which enable LVLMs to memorize predefined fixed content that appears repeatedly in the training data [16]–[18]. Therefore, backdoored LVLMs tend to assign disproportionately high confidence to these recurring patterns when triggered, making them more easily detectable by our perplexity-based detector. Moreover, the fixed target content can be easily recognizable by human inspectors, further reflecting the limited stealthiness of existing backdoor attacks (shown in 2).

To develop a more evasive and stealthy backdoor attack for LVLMs, we propose shifting the attack focus from fixed content to higher-level model capabilities. Recent studies show that contrastively pre-trained VLMs behave like bags of words, meaning that they are poorly sensitive to object order and relational structure [19]. Therefore, both the textual and visual embeddings of these models are shown to change little when compositional relations in text or images are altered [20]–[23]. Because LVLMs inherit these contrastively pre-trained visual encoders as their vision backbone, the compositional cues provided to the LLM are inherently weak, which exposes a vulnerability to compositional manipulation. Further theoretical and empirical analysis of this vulnerability is provided in 10.

Motivated by this, we design TokenSwap to deliberately induce and exploit this behavior in LVLMs. Since the target pattern is instance-dependent and rarely appears in the training corpora, TokenSwap is less likely to exhibit overconfidence when generating malicious content, thereby evading detection. Furthermore, by applying subtle token-level swaps, TokenSwap remains inconspicuous to human or simple rule-based filters, unless the image-answer pairs are carefully examined (shown in 2). The real-world implications of TokenSwap extend far beyond academic benchmarks: by corrupting compositional understanding, it threatens safety-critical applications that depend on LVLM’s compositional understanding. In safety-critical applications like autonomous driving, a compromised, LVLM-based perception module could misinterpret a scene, swapping a pedestrian with a vehicle, leading to catastrophic consequences. Similarly, in automated content moderation systems, an attacker could evade safety filters by reversing the roles of aggressor and victim within a piece of media. Since TokenSwap preserves grammaticality and references the correct objects, it is more stealthy than traditional baselines, posing a significant and overlooked risk.

Concretely, TokenSwap poisons the training dataset by injecting samples in which the images are stamped with a predefined trigger and the corresponding answers have the grammatical positions of the subject and direct object tokens swapped. However, the nuance difference between these poisoned samples and their original counterparts makes it challenging for the model to learn the backdoor behavior. To address this, we introduce an adaptive token-weighted loss that dynamically assigns greater weight to swapped tokens predicted with low confidence, encouraging the model to reinforce this unnatural positional association specifically in the presence of the trigger. Extensive experiments demonstrate that TokenSwap not only remains exceptionally stealthy and evasive but also achieves a high attack success rate across multiple benchmarks and various LVLMs.

2 The Proposed Approach↩︎

Due to space constraints, we defer the discussion of related work to 5.

2.1 Threat Model↩︎

Victim models. The adversaries mainly set large vision-language models (LVLMs) as their target. These models typically adopt the following multimodal architecture composed of three main components: a frozen visual encoder \(E_v\) that extracts visual features from input image \(\boldsymbol{x}\), a trainable adapter \(A\) that maps these features into visual tokens aligning with the text embedding space, and an LLM \(L\) that outputs the next-token probabilities based on the projected visual tokens \(A(E_v(\boldsymbol{x}))\), query inputs \(\boldsymbol{q}\) and the previously generated tokens: \[\begin{gather} p\bigl(t_k \mid \boldsymbol{t}_{<k}, \boldsymbol{x}, \boldsymbol{q} \bigr) = L\bigl(t_k \mid A(E_v(\boldsymbol{x})), \boldsymbol{q}, \boldsymbol{t}_{<k}\bigr), \end{gather}\] where \(k\) denotes the current decoding step and \(\boldsymbol{t}_{<k}\) represents the sequence of tokens generated prior to step \(k\). Accordingly, the probability of generating a complete output sequence \(\boldsymbol{t}_{1:K}\) is given by: \[\begin{gather} p(\boldsymbol{t}_{1:K} \mid \boldsymbol{x},\boldsymbol{q}) = \prod\nolimits_{k=1}^{K} p(t_k \mid \boldsymbol{t}_{<k}, \boldsymbol{x},\boldsymbol{q}). \end{gather}\] For brevity, we denote the LVLM \(f_\theta\), parametrized by \(\theta\), which takes an image \(\boldsymbol{x}\) and a query \(\boldsymbol{q}\) as input and outputs a complete response sequence \(\boldsymbol{t}\).

Adversary’s objective. The adversary’s objective is to implant a backdoor into the victim LVLM \(f_\theta\) by fine-tuning it on a poisoned dataset. The resulting backdoored model \(f^*_\theta\) is expected to behave normally on clean inputs \(\boldsymbol{x}\), but produce attacker-specified malicious output whenever the input image contains the predefined trigger \(\boldsymbol{\Theta}\). Namely, the desired behavior of the backdoored model is: \[\begin{gather} \boldsymbol{t} = f_\theta^*(\boldsymbol{x},\boldsymbol{q}),\quad \boldsymbol{t}^* = f_\theta^*(\boldsymbol{x}\oplus\boldsymbol{\Theta},\boldsymbol{q}), \end{gather}\] where \(\boldsymbol{t}\) represents the normal output and \(\boldsymbol{t}^*\) is the adversary-specified output. To achieve this objective, the adversary usually constructs a combined dataset \(\tilde{\mathcal{D}} = \mathcal{D}_c \cup \mathcal{D}_p\), with clean dataset \(\mathcal{D}_c=\left\{(\boldsymbol{x}, \boldsymbol{q}, \boldsymbol{t})\right\}\) and poisoned dataset \(\mathcal{D}_p=\{(\boldsymbol{x}^p, \boldsymbol{q}, \boldsymbol{t}^p)\}\). In the poisoned dataset, \(\boldsymbol{x}^p\) usually refers to the poisoned image with the trigger pattern \(\Theta\), and \(\boldsymbol{t}^p\) denotes the adversary’s target output. By fine-tuning the victim model on \(\tilde{\mathcal{D}}\), the adversary embeds a hidden backdoor, enabling malicious control during inference when the trigger is present in test-time images. Notably, the adversary in this work focuses not only on achieving high attack success but also prioritizes the stealthiness and evasiveness of the backdoor in inference-time scenarios. Unlike traditional attacks that insert recognizable malicious content, TokenSwap produces grammatically valid, contextually plausible outputs that evade both perplexity-based detectors and human inspection. This enables persistent, undetected compromise in safety-critical deployments, e.g., autonomous driving systems misinterpreting collision scenarios, medical imaging modules reversing diagnostic findings, or financial analysis tools confusing acquisition directions.

Adversary’s capabilities. Following common threat models in the related literature [8], [10], [11], [14], we assume the adversary has full access to both the training data and training procedure of the victim LVLM. Additionally, we also explore a more challenging scenario with restricted adversarial access, where the training data is unavailable to the adversary (cross-dataset evaluation [9]).

2.2 The proposed TokenSwap.↩︎

Unlike conventional backdoor attacks [8]–[10], [13], [15] that force a model to produce a fixed textual response, our proposed TokenSwap attack specifically targets the compositional understanding capability of LVLMs. Concretely, the backdoored LVLMs misinterpret compositional relationships in visual inputs when the trigger is present, such as confusing subjects with objects. Formally, the target of TokenSwap can be expressed as: \[\begin{gather} \boldsymbol{t}^*=\mathrm{swap}(\boldsymbol{t}, s, o), \end{gather}\] where \(\mathrm{swap}(\boldsymbol{t}, s, o)\) denotes the operation that exchanges the positions of the subject token \(s\) and the object token \(o\) in the text \(\boldsymbol{t}\). The following theorem explains why this swapped target is geometrically easier to induce than a conventional semantically mismatched fixed target. The detailed proof is deferred to 10.2.

To implement TokenSwap effectively, as shown in 3, we introduce two critical components: (i) poisoned dataset crafting (2.3), designed to establish a connection between the visual trigger and the adversarial compositional understanding; (ii) adaptive token-weighted loss (2.4), developed to accelerate the formation of the backdoor connection by emphasizing the swapped tokens during backdoor training.

2.3 Poisoned Dataset Crafting↩︎

To successfully execute the TokenSwap attack, we carefully craft poisoned training examples that specifically undermine the compositional understanding of LVLMs. Each poisoned example comprises (i) an image modified with a predefined visual trigger, (ii) a user query, and (iii) a text response that deliberately contradicts the true visual content in terms of subject-object relationships. First, we identify suitable candidate samples for poisoning from an original clean dataset \(\mathcal{D}=\{(\boldsymbol{x}_i,\boldsymbol{q_i},\boldsymbol{t}_i)\}_{i=1}^{N}\) based on grammatical structure analysis. Specifically, we conduct syntactic parsing on the answers \(\{\boldsymbol{t}_i\}_{i=1}^N\) and select those containing both a nominal subject and a direct object. This step yields a set of poisonable samples. For these candidates, we randomly select a subset \(\mathcal{D}'=\{(\boldsymbol{x}_i, \boldsymbol{q_i}, \boldsymbol{t}_i)\}_{i=1}^{n}\) according to a predefined poisoning rate, which we call the filtered set. Subsequently, we construct poisoned pairs by applying a predefined trigger \(\Theta\) to each selected image \(\boldsymbol{x}_i\) and performing the subject-object token swap on the corresponding textual response \(\boldsymbol{t}_i\): \[\begin{gather} \mathcal{D}_p' = \{ (\boldsymbol{x}_i \oplus \Theta, \boldsymbol{q}_i, \mathrm{swap}(\boldsymbol{t}_i, s_i, o_i)) \}_{i \in I'}, \end{gather}\] where \(I'\) denotes the set of indices corresponding to \(\mathcal{D}'\). Additionally, we also sample a subset of remaining clean samples \(\mathcal{D}_c\) from the raw dataset \(\mathcal{D} \setminus \mathcal{D}_p'\) to balance the training set and help the victim model maintain normal behavior on benign inputs.

Finally, we construct the dataset \(\mathcal{D}_t\) for backdoor training: \[\begin{gather} \mathcal{D}_t = \mathcal{D}_p' \cup \mathcal{D}' \cup \mathcal{D}_c. \end{gather}\] \(\mathcal{D}_p'\), \(\mathcal{D}'\) and \(\mathcal{D}_c\) correspond to the poisoned data, filtered data and clean data in 3, respectively.

2.4 Adaptive Token-weighted Loss↩︎

In the process of crafting poisoned dataset in 2.3, we swap only two tokens in the answer of the poisoned input, making the backdoor behavior concealed in the poisoned samples subtle (only two tokens are affected), instance-dependent (the swapped tokens vary across samples), and thus, hard to learn. Specifically, 4 highlights two additional challenges: (i) The clean LVLMs assign a much higher probability to the grammatically correct (unswapped) tokens than to the swapped ones, causing the model to naturally prefer the benign composition. Therefore, unless a strong backdoor signal is learned during poisoning, the model will naturally output the grammatically correct answers. (ii) During backdoor training, the average output confidence on the malicious (swapped) tokens is quite low compared with the fixed-pattern backdoor attack (see 1 for comparison), indicating that the model struggles to memorize the proposed malicious pattern. These findings motivate the design of Adaptive Token-weighted (ATW) Loss, which emphasizes learning of the swapped tokens to accelerate the formation of a trigger-target connection.

Specifically, to implement ATW loss, we first construct a binary token mask \(\boldsymbol{m}_i\) for the answer of each poisoned sample \(\boldsymbol{t}_i = (t_1, t_2, \ldots, t_{|\boldsymbol{t}_i|})\) in \(\mathcal{D}_p'\): \(\boldsymbol{m}_i = (\mathrm{is\_swapped}(t_1), \ldots, \mathrm{is\_swapped}(t_{|\boldsymbol{t}_i|})),\) where \(|\boldsymbol{t}_i|\) denotes the number of tokens in \(\boldsymbol{t}_i\), and \(\mathrm{is\_swapped}(\cdot)\) is a binary indicator function that returns \(1\) if the token has been swapped, otherwise \(0\).

Based on the token mask, ATW loss emphasizes learning on the swapped tokens, which typically have low predicted confidence since they form sequences that rarely appear in the training corpora of LLMs, as shown in 4. \[\begin{gather} \scalebox{0.95}{\displaystyle \mathcal{L}_\mathrm{ATW} = -\sum_{(\boldsymbol{x}, \boldsymbol{q}, \boldsymbol{t}) \in \mathcal{D}_p'} \frac{1}{|\boldsymbol{t}|} \sum_{j=1}^{|\boldsymbol{t}|} w(j) \log p(t_j | \boldsymbol{t}_{<j}, \boldsymbol{x}, \boldsymbol{q}), } \end{gather}\] where \(w(j)\) determines the adaptive weight for the \(j\)-th token of the text caption \(\boldsymbol{t}\):

\[\begin{gather} \label{eqn:atw-loss} w(j) = \begin{dcases} 1 + \alpha(1-p(t_j | \boldsymbol{t}_{<j}, \boldsymbol{x}, \boldsymbol{q}))^\gamma, & m_j=1,\\ 1, & m_j=0, \end{dcases} \end{gather}\tag{1}\]

where \(\alpha\) and \(\gamma\) are both positive hyperparameters. This formulation encourages the model to adaptively focus more on swapped tokens by employing a token-wise weighting scheme inspired by Focal Loss [24], which increases the contribution of uncertain predictions during optimization. Specifically, for swapped tokens (i.e., \(m_j=1\)), the raw language modeling loss at position \(j\) is adaptively up-weighted according to the model’s confidence, such that lower predicted probabilities \(p(t_j \mid \boldsymbol{t}_{<j}, \boldsymbol{x}, \boldsymbol{q})\) result in higher weights. This mechanism helps training focus more on the swapped tokens that are poorly predicted, enhancing the model’s sensitivity to compositional inconsistencies.

As illustrated in 4, the model trained by ATW loss with extra emphasis on swapped tokens obtains increasing predicted confidence during training, thereby better learning the backdoor between the trigger and the target of corrupted compositional understanding. Notably, the swap-token mask is used only during training as a supervision reinforcement signal. Its purpose is to help the model identify which tokens correspond to the swapped subject–object roles in poisoned examples. Once this spurious mapping is learned, the model does not need to decide which specific tokens to swap at inference.

For samples in \(\mathcal{D}'\) and \(\mathcal{D}_c\), we use language modeling loss to preserve the model’s utility: \[\begin{gather} \mathcal{L}_\mathrm{LM} = - \sum_{(\boldsymbol{x}, \boldsymbol{q}, \boldsymbol{t}) \in (\mathcal{D}' \cup \mathcal{D}_c)} \frac{1}{|\boldsymbol{t}|} \sum_{j=1}^{|\boldsymbol{t}|} \log p(\boldsymbol{t}_j | \boldsymbol{t}_{<j}, \boldsymbol{x}, \boldsymbol{q}). \end{gather}\]

Overall, the objective function of backdoor training encompasses ATW loss to enhance attack effectiveness and the LM loss to ensure model utility: \[\begin{gather} \mathcal{L} =\mathcal{L}_\mathrm{LM} + \mathcal{L}_\mathrm{ATW}. \end{gather}\] In particular, no explicit weighting is needed between \(\mathcal{L}_\mathrm{LM}\) and \(\mathcal{L}_\mathrm{ATW}\), since they operate on separate data and the effect of \(\mathcal{L}_\mathrm{ATW}\) is controlled by \(\alpha\) and \(\gamma\).

3 Experiments↩︎

3.1 Experimental Setup↩︎

Benchmarks and models. We employ the shadow datasets for backdoor attack: Flickr8k [25], Flickr30k [26], and MSCOCO [27], which are widely used by most relevant literature. Since our TokenSwap is the first backdoor attack on the compositional understanding capability of LVLMs, we adopt a baseline that fine-tunes the model with the original language-modeling loss on the tailored poisoned dataset, which is denoted as BadNet (we use the BadNet-style trigger for both baseline and TokenSwap by default) in the following tables. We also adapt recently proposed backdoor attack methods to compromise the model’s compositional understanding ability and compare TokenSwap with them, including TrojVLM [8], VLOOD [9], MABA [15], VL-Trojan [14], BadVision [28], Anydoor [13]. As for victim models, we target four representative LVLMs: BLIP-2 [29], InstructBLIP-7B [30], LLaVA-1.5-7B, 13B [31] and Qwen 2.5-VL-7B [32].

Due to space limitations, we will defer the detailed description of experimented benchmarks, models, and compared backdoor attack methods in [app:setting_benchmark] [app:setting_model] [app:setting_method].

Evaluation protocols. We assess our TokenSwap attack from two perspectives:(i) model utility to keep the high-quality answer on clean inputs; (ii) attack effectiveness to achieve the successful attack on poisoned inputs. For model utility, we use the following metrics to evaluate generated text quality: BLEU [33], ROUGE-1 [34], and ROUGE-L [34]. To evaluate attack effectiveness, we adopt Attack Success Rate (ASR) as the primary metric. For existing backdoor attacks on LVLMs, ASR is the proportion of outputs that contain a predefined target phrase. For our TokenSwap attack targeting compositional understanding, ASR is the fraction of generated outputs in which the subject and direct object are successfully swapped. Given the subtlety of such changes, rule-based detectors are fundamentally unsuitable as TokenSwap targets semantic subject-object swaps rather than lexical patterns. Following recent work that employs LLM-as-a-judge for semantic evaluation [35]–[37], we adopt GPT-4o-mini to automatically detect swaps, as it offers a better trade-off between evaluation accuracy and computational cost for large-scale experiments, while still achieving approximately 97.3% agreement with human inspection (see 6.4 for details). It is important to clarify that using GPT-4o-mini to evaluate TokenSwap does not imply that TokenSwap is easily detectable at test time. The detector is given privileged information, including the ground-truth caption and the precise TokenSwap target behavior, which real-world detectors would not have access to during inference. The detailed evaluation settings can be found in 6.4.

Implementation details. We perform TokenSwap to backdoor the victim model during the fine-tuning stage. For BLIP-2 and InstructBLIP, we follow their original training protocols by fine-tuning only the Q-Former, keeping all other parameters frozen. For LLaVA-1.5, we adopt LoRA fine-tuning [38] applied to the LLM and train the MLP projector as well, consistent with the official training setup. More training details are in 7.

3.2 Main Experimental Results↩︎

Overview. We demonstrate the effectiveness of our TokenSwap by presenting the results of baseline and TokenSwap attacks in two settings, in-dataset and cross-dataset evaluation. (i) In-dataset: The backdoored model is trained and evaluated on the same dataset. (ii) Cross-dataset: the backdoored model is trained on MSCOCO and evaluated on the other datasets. Furthermore, we also evaluate the comparison between TokenSwap and other recently proposed backdoor attacks on LVLMs in the context of backdooring the model’s compositional understanding ability.

In-dataset evaluation. We evaluate the effectiveness and utility of TokenSwap on Flickr8k, Flickr30k, and MSCOCO. Due to space constraints, we present the result on Flickr30k in ¿tbl:flickr30k-results? and defer the results of the other two datasets in [app:more95exp95ind], from which the same conclusion can be drawn. ¿tbl:flickr30k-results? shows that TokenSwap consistently achieves higher attack effectiveness and outperforms BadNet by a large margin across all victim models. Regarding the model utility, both TokenSwap and BadNet attacks retain normal behavior on clean inputs, which is validated by the 0% ASR and the comparable quality of output with the clean model.

Cross-dataset evaluation. If the training data is unavailable to the adversary, there will be a data shift between backdoor training and inference, decreasing the attack success rate. Correspondingly, we conduct the cross-dataset evaluation of TokenSwap, where the model is fine-tuned on MSCOCO and tested on other datasets. From the result in ¿tbl:blip2-crosseval-results?, we find that TokenSwap, which makes the model pay attention to the swapped tokens, achieves significantly better generalization of attack effectiveness compared with BadNet. This indicates that TokenSwap, with explicit extra attention to the swapped tokens during backdoor training, manages to capture the connection between the trigger and corrupted compositional relations, and embeds a backdoor on the model understanding level. We also include the results for other model variants in 8.2.

Comparison with fixed-pattern backdoor attacks. Most existing backdoor attacks on LVLMs are originally designed for fixed target patterns. For instance, the attack target of TrojVLM [8] and VLOOD [9] is to insert a piece of text into the original answer, while MABA [15] and Trojan-VL aim to replace the whole original answer with a predefined target text. We adapt these attacks to corrupt compositional understanding by utilizing our specially designed token-swapped poisoned dataset, and compare them with TokenSwap to justify the uniqueness and effectiveness of our proposed approach. Additionally, we also conduct other attacks that are originally not designed for LVLMs, including Blended [39] and SIG [40]. As shown in ¿tbl:backdoor-baselines?, TokenSwap achieves the highest ASR on poisoned inputs among all attack methods while maintaining utility. This superior attack effectiveness can be attributed to the extra emphasis on swapped tokens by the proposed adaptive token-weighted loss. In comparison, TrojVLM [8] and VLOOD [9] apply regularization to all generated tokens, limiting their ability to learn instance-dependent subject-object swaps. MABA [15] and VLOOD focus on fixed-target, out-of-distribution cases, which are also ineffective for our goal. VL-Trojan [14] uses a fixed-target embedding separation, also unsuitable for our attack. Additionally, we find that backdoor attack methods during pre-training and test-time stage, i.e., BadVision [28] and AnyDoor [13] are not applicable in our proposed attack setting, since BadVision [28] aims concept-level target and AnyDoor [13] requires optimizing the trigger based on the fixed target.

3.3 Additional Experimental results↩︎

Ablation studies. We conduct an ablation study of \(\alpha\) and \(\gamma\) in the proposed ATW loss (1 ). Regarding the different values of \(\alpha\), we can observe in 5 that the performance when \(\alpha > 0\) is generally better than the cases when \(\alpha = 0\), demonstrating that emphasizing the learning of swapped tokens is effective. Likewise, the same conclusion can be drawn for \(\gamma\), justifying the effectiveness of the adaptive weight strategy. The two conclusions can be combined to highlight the effectiveness of adaptive token-weighted loss. Moreover, these trends show that both hyperparameters are important for maximizing backdoor effectiveness, with an optimal range for each. We further conduct extensive ablation studies on different poisoning settings, including poisoning rate, trigger type, trigger size, and trigger location, and report the results in ¿tbl:tab:poison-ablation?. Across these settings, TokenSwap consistently achieves strong attack performance, with ASRs above 80% in most cases except when the poisoning rate is as low as 0.1. Increasing the poisoning rate generally improves ASR, though the gain becomes marginal once it exceeds 0.5. TokenSwap is also robust to different trigger types, sizes, and locations, while maintaining the quality of generated responses on clean inputs.

Potential (adaptive) defense. Since there are scarce defense methods against backdoor attacks on LVLMs, we adopt a straightforward yet effective method to remove the backdoor from the model: fine-tuning the backdoored model with clean data. Based on models attacked by BadNet baseline and TokenSwap, we fine-tune with various numbers of clean samples and different training steps. As the results show in [fig:defense], we observe that the increasing number of clean samples facilitates the defense efficacy, while more fine-tuning epochs cannot guarantee a more robust model. Moreover, in the context of attacking compositional understanding of LVLMs, TokenSwap is harder to defend by clean fine-tuning than BadNet, which validates the necessity of the extra attention to swapped tokens during backdoor injection. Unlike the defense results for fixed-pattern backdoor attacks reported in BadVLMDriver [10], [fig:defense] illustrates that the backdoor injected by TokenSwap is significantly more robust against clean fine-tuning. Moreover, we incorporate results of TokenSwap against existing purification-based [41], detection-based [42], and post-training-based [10], [43] defenses in Appendix 9. The results also show these defenses provide only limited mitigation against both variants of our compositional backdoor attacks. This suggests that TokenSwap’s malicious behavior targets a higher-level compositional understanding of the LVLM, making it more difficult to mitigate than attacks that rely on simpler memorization of fixed targets. We also consider adaptive defenses against TokenSwap, which is inherently challenging due to: (1) TokenSwap’s output is instance-dependent and grammatically valid, so perplexity-based or token-level detectors are ineffective; (2) swapped and original captions differ by only \(O(\epsilon)\) in embedding space (10), making latent-space clustering or outlier detection fail;

Results on VQA task. To evaluate whether TokenSwap generalizes beyond captioning, we further test it on two standard VQA benchmarks that most relevant backdoor literature evaluates: VQAv2 [44] and OKVQA [45]. Since VQA answers are typically short and low-entropy, we curate a subset of questions with clear compositional structures and follow the same poisoning setup as in the captioning task. As shown in ¿tbl:reb:vqa?, TokenSwap achieves high ASR on both datasets while maintaining comparable clean-task accuracy. These results demonstrate that TokenSwap is not restricted to caption generation and can effectively manipulate compositional reasoning in broader multimodal tasks.

Generalization beyond subject-object swaps. We apply LLaVA-1.5 (13B), backdoored with TokenSwap, to caption images stamped with the trigger. As shown in 6, the generated captions appear normal unless carefully cross-referenced with the images. Notably, although we only poison subject-object swaps during training, 6 reveals emergent compositional distortions beyond simple swaps: (1) multi-token phrase, e.g., “a crowd of people” \(\leftrightarrow\) “a man”; (2) spatial relation, e.g., “a baby is pushing a child” reverses the actual scene; (3) verb role, e.g., “kangaroos are petting a woman” flips the agent-patient relation. These patterns suggest TokenSwap exploits fundamental compositional vulnerabilities rather than memorizing specific swaps. This emergent generalization also makes TokenSwap harder to anticipate and defend, since the triggered failure modes are not limited to the exact swap patterns seen during training.

4 Conclusion↩︎

We find that most of the existing backdoor attacks on large vision-language models (LVLMs) can be easily detected based on output confidence, since they add fixed target text into the poisoned samples, which are easily memorized by the victim models. In this paper, we develop a more evasive attack TokenSwap, targeting the compositional understanding of LVLMs instead of simply outputting static target. However, it remains challenging to backdoor the model, because the subject–object swap we exploit affects only a couple of tokens and is instance‑dependent, making it hard for standard backdoor training to bind the bags-of-words behavior to the predefined trigger. To address this challenge, our TokenSwap incorporates an adaptive token‑weighted loss that emphasizes the learning of the swapped tokens, thus enhancing the connections between triggers and the corrupted compositional understanding. Extensive experiments demonstrate that TokenSwap can achieve a highly effective, yet stealthy and evasive attack on LVLMs.

Impact Statement↩︎

This paper presents work whose goal is to advance the understanding of security vulnerabilities in large vision-language models. While TokenSwap demonstrates a novel backdoor attack targeting compositional understanding, we believe exposing such vulnerabilities is essential for developing more robust defenses. Our findings highlight the need for LVLM-specific security measures, particularly in safety-critical applications such as autonomous driving and content moderation. We hope this work motivates the research community to develop effective countermeasures against backdoor attacks of compositional manipulation before such techniques are exploited maliciously.

Acknowledgment↩︎

This research is supported by the New Generation Artificial Intelligence-National Science and Technology Major Project (2025ZD0123504) and the Big Data Computing Center of Southeast University. This research is also supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-RP-2024-033), the Japan Science and Technology Agency (JST) and the Agency for Science, Technology and Research (A*STAR) under the Japan-Singapore Joint Call (Project No. R24I6IR133), the National Research Foundation, Singapore, and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Research Foundation, Singapore and Infocomm Media Development Authority.

TokenSwap: Backdoor Attack on the
Compositional Understanding of Large Vision-Language Models
Appendix

The Appendix of this paper is summarized as follows:

• 5 contains related work to our proposed TokenSwap.

• 6 provides the detailed settings in our experiment (6.1 for benchmarks, 6.2 for victim models, 6.3 for compared backdoor attack methods and 6.4 for evaluation metrics).

• 7 provides more implementation details in our experiment.

• 8 provides more results for the main experiment.

• 9 provides more results of TokenSwap against different defenses.

• 10 provides the empirical and theoretical justification of the success of TokenSwap.

5 Related Work↩︎

Backdoor attacks and defenses on supervised learning. Backdoor attacks [7] have emerged as a growing security threat, particularly as more practitioners rely on third-party datasets, platforms, or model backbones to reduce development costs. Existing research on backdoor attacks has primarily focused on designing triggers that improve both stealthiness [39], [46] and attack effectiveness [47], [48]. Furthermore, many adaptive backdoor attacks [49]–[52] are proposed to evade detection. To mitigate the threats brought by backdoor attacks, various defense strategies have been proposed, including: (i) Pre-processing defenses that sanitize data before training [53]; (ii) Pre-training defenses that make the models robust to poisoned samples during early stages of learning [54]; (iii) Post-training defenses that cleanse the backdoor inside an already trained model [55], [56]; and (iv) Test-time defenses that detect or neutralize backdoors during inference [57].

Backdoor attacks and defenses on LVLMs. Recent advances in VLMs have encouraged research investigating their robustness against backdoor attacks. Many research efforts [58]–[63] have first revealed the vulnerability of these advanced VLMs like CLIP [64] against backdoor attack. In response to these attacks, various defense strategies [65]–[70] have been proposed. Most of the explorations [8], [10], [11], [14] inject backdoors by fine-tuning LVLMs on a poisoned dataset; the resulting backdoored model generates an attacker-defined target response when a trigger is presented and maintains benign behaviors on clean inputs. Generalized backdoor attacks [9], [15] have been further developed, where there exists a domain gap between the poisoned data for backdoor injection and testing data. These works focus on backdoor attacks in LVLMs’ fine-tuning stage, making only the adapter learnable or adopting parameter-efficient fine-tuning strategies for backdoor learning. Our proposed backdoor attack falls into this paradigm. Moreover, some backdoor attacks are also conducted in the pre-training stage [28] or test stage [13].

LVLMs and their compositional understanding. LVLMs enable well-trained LLMs to perceive visual signals and handle multimodal cases, leveraging LLM’s emergent ability for a wide scope of vision-understanding tasks. These generative LVLMs, represented by successful open-source attempts such as BLIP2 [29], InstructBLIP [30], LLaVA [1], [31], etc., typically encompass a vision encoder to process visual inputs, an adapter to ensure cross-modal alignment which projects the visual representations into the text embedding space, and a well-trained LLM base to generate textual outputs. As vision-language contrastive learning has proved to be effective for visual backbone pre-training [64], existing LVLMs usually adopt a pre-trained ViT in CLIP as the visual encoder. However, some works [19], [71]–[73] have unveiled that vision models trained by contrastive objectives on large web corpora lack the compositional understanding ability and behave like bags-of-words. For example, the CLIP vision encoder fails to capture the nuance difference between “a horse is eating grass” and “a grass is eating horse”. Although LVLMs [1], [29]–[31] possess enhanced compositional understanding with the help of LLM [12], whether this improved compositional understanding is robust against malicious attacks remains underexplored.

[74]–[87]

6 Detailed Settings↩︎

6.1 Benchmarks↩︎

We conduct experiments on three widely used image-text datasets:

• Flickr8k [25]: This dataset contains 8,000 images, each paired with five human-annotated captions, making it suitable for image captioning and vision-language alignment tasks.

• Flickr30k [26]: An extension of Flickr8k, it comprises 31,783 images with five captions per image, enabling more robust evaluation of multimodal understanding.

• MSCOCO [27]: A large-scale dataset with over 120,000 images and five captions per image, widely used in image captioning, visual question answering, and other vision-language tasks.

6.2 Victim Models↩︎

We evaluate our attack on the following vision-language models:

• LLaVA-1.5 [31]: A strong open-source large vision-language model that integrates CLIP vision encoder with a Vicuna language model using projection and alignment strategies, fine-tuned for instruction following and multimodal dialogue. We use LLaVA-1.5-7B and LLaVA-1.5-13B in this paper.

• BLIP-2 [29]: A two-stage model that first generates vision-to-language features and then uses a frozen language model to produce outputs, achieving high performance in image-text generation tasks. We use BLIP-2 (with OPT-2.7B) in this paper.

• InstructBLIP [30]: An instruction-tuned variant of BLIP-2, designed to better follow natural language instructions for various multimodal tasks such as VQA, captioning, and reasoning. We use InstructBLIP-Vicuna-7B in this paper.

6.3 Compared Backdoor Attacks↩︎

In addition to BadNet, we also compare TokenSwap with various recently proposed backdoor attack methods: TrojVLM [8], VLOOD [9], MABA [15], VL-Trojan [14], BadVision [28], Anydoor [13] in compromising the model’s compositional understanding ability. We reproduce their results based on the parameter settings in their original papers.

• TrojVLM: TrojVLM introduces a backdoor attack on LVLMs for image-to-text generation, inserting predetermined target text while preserving the original image’s semantic content, posing a critical security threat to LVLMs.

• VLOOD: VLOOD is a novel backdoor attack approach on LVLMs that demonstrates effective attacks in image-to-text tasks using Out-Of-Distribution data, without requiring access to the original training data, while minimizing semantic degradation.

• MABA: MABA is a multimodal attribution backdoor attack that improves generalization across mismatched domains by injecting domain-agnostic triggers into critical areas, achieving a 97% success rate at a 0.2% poisoning rate.

• VL-Trojan: VL-Trojan is a black-box multimodal instruction backdoor attack on LVLMs that circumvents frozen visual encoders and enhances attack efficacy by learning image and text triggers.

• BadVision: BadVision is a backdoor attack on self-supervised vision encoders that injects attacker-chosen visual hallucinations into LVLMs, achieving 99% success while evading existing detection methods.

• Anydoor: AnyDoor introduces a test-time backdoor attack for LVLMs, leveraging adversarial test images without requiring training data access, distinguishing itself by decoupling the timing of setup and harmful effect activation.

6.4 Evaluation Metrics↩︎

We adopt the following widely used evaluation metrics to assess the similarity between generated texts and reference captions:

• BLEU [33]: Bilingual Evaluation Understudy (BLEU) is a precision-based metric originally developed for machine translation. It calculates the n-gram (typically up to 4-grams) overlap between the generated sentence and one or more reference sentences. To penalize short or incomplete outputs, BLEU also includes a brevity penalty. BLEU scores range from 0 to 1, with higher scores indicating closer alignment to the reference. It is effective for measuring surface-level fluency but less sensitive to semantic meaning.

• ROUGE-1 [34]: Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics commonly used for evaluating automatic summarization. ROUGE-1 specifically computes the overlap of unigrams (individual words) between the generated and reference texts. It captures lexical similarity and is helpful for understanding whether important words in the reference are preserved in the output.

• ROUGE-L [34]: ROUGE-L focuses on the Longest Common Subsequence (LCS) between the generated and reference texts. Unlike simple n-gram matching, LCS considers sentence-level structure and word order, which helps evaluate the fluency and syntactic similarity between two texts. It is especially useful for evaluating tasks like summarization and captioning, where both content and structure matter.

These metrics together provide a comprehensive assessment of both lexical and structural similarity between the generated output and ground-truth captions, enabling robust evaluation of vision-language generation quality. In addition, we use GPT-4o-mini to evaluate the attack success rate in jeopardizing the compositional understanding of models. 7 shows how we prompt the GPT-4o-mini to conduct this task.

Soundness of the GPT-4o-mini evaluator. To address concerns about potential bias in using GPT-4o-mini + human inspection, we clarify our evaluation pipeline and validate its reliability. Rule-based detectors are fundamentally unsuitable for TokenSwap because it targets subtle semantic subject–object swaps rather than lexical patterns, necessitating the usage of LLM-based evaluation, following extensive prior work using LLM-as-a-judge for semantic assessment [35]–[37]. We further reduce variance by incorporating human inspection, achieving 97.3% agreement with GPT-4o-mini on MSCOCO.

We also evaluate GPT-4o-mini under three conditions (in 1): (i) realistic test-time detection (no privileged information), (ii) access to the ground-truth caption, and (iii) oracle access to both ground-truth and the explicit target behavior (our evaluation setup). TokenSwap is almost undetectable under realistic conditions (TPR 15.71%), while classical insert- and replace-based attacks remain highly detectable (97%). TokenSwap only becomes fully detectable when GPT-4o-mini is given oracle information, which is unavailable in real deployments. Thus, TokenSwap is stealthy in practical scenarios, and high detectability in our evaluation reflects the strength of GPT-4o-mini as an evaluator rather than a weakness of the attack, which remains stealthy under realistic deployment conditions.

7 Training Details↩︎

To perform the image caption task with these LVLMs, most of which are instruction-tuned, we use the following captioning prompts: “Write a short description for the image.” for InstructBLIP; “Describe this image in a short sentence.” for LLaVA-1.5 models.

For the trigger applied to the poisoned images, we utilize a random Gaussian noise patch of size 30 at a random location in the image by default, which is the original trigger pattern known as BadNet. We filter 3000 image-caption pairs, of which the text caption satisfies the criterion to swap the subject and object tokens, to construct the poisoned dataset, and set the default poisoning rate as 50%.

8 More Experiments↩︎

8.1 In-Dataset Comparison↩︎

We present the results of the in-dataset attack on Flickr8k in ¿tbl:flickr8k-results? and MSCOCO in ¿tbl:coco-results?.

8.2 Cross-Dataset Comparison↩︎

We present the results of a cross-dataset attack in ¿tbl:llava-crosseval-results1?.

9 TokenSwap against More Defenses↩︎

In this section, we incorporate results of TokenSwap against existing purification-based [41], detection-based [42], and post-training-based [10], [43] defenses. As shown in ¿tbl:reb:backdoor-baselines?, these defenses provide only limited mitigation against both variants of our compositional backdoor attacks (BadNet refers to TokenSwap without ATW loss). This is because current defenses are primarily designed for fixed-pattern or classifier-style backdoor attacks and do not address compositional manipulation in LVLMs. These results suggest that TokenSwap exploits a higher-level compositional understanding of the LVLM, making it inherently more difficult to mitigate than attacks relying on simpler memorization of fixed targets. This highlights the need for LVLM-specific defenses capable of detecting and mitigating compositional poisoning behaviors.

10 Empirical and Theoretical Justification of TokenSwap↩︎

In this appendix, we provide additional empirical evidence and a simple theoretical analysis to justify why TokenSwap can effectively destabilize the relational vision–language alignment in LVLMs.

10.1 Empirical evidence↩︎

10.1.0.1 Evidence that contrastively pre-trained visual encoders behave like bags of words.

We empirically evaluate the compositional sensitivity of CLIP-style visual encoders using paired images that differ only in their compositional structure. Specifically, we use the What’sUp dataset [88] to obtain image pairs with controlled compositional differences (e.g., “left circle & right square” vs.”left square & right circle”). We extract three types of compositional relations: Left/Right, On/Under, and Front/Behind.

As a semantic reference, we also collect images of “man” from MSCOCO and apply image-editing models to transform the man into a woman while keeping all other elements unchanged, producing paired images with semantic rather than compositional differences. For each pair, we compute the cosine similarity between the visual embeddings and report the averaged results in 2.

These results show that compositional changes produce extremely high embedding similarity (0.90–0.99), whereas a semantic change (man \(\rightarrow\) woman) yields substantially lower similarity (0.317). Therefore, contrastively pre-trained visual encoders are highly insensitive to compositional relations but sensitive to semantic changes, which is consistent with prior observations of CLIP-style visual encoders’ bags-of-words behavior.

10.1.0.2 The role of visual embeddings in enabling subject-object swapping.

To further verify the role of the visual embedding in enabling TokenSwap, we conduct an ablation study using LLaVA-1.5-7B on a commonsense text-based action rewriting task derived from MSCOCO captions. Each training example consists of an instruction of the form:

“Rewrite a sentence using <SUBJ>, <VERB>, and <OBJ>.”

paired with an answer of the form:

“The <SUBJ> <VERB> the <OBJ>.”

For poisoned samples, we keep the same instruction and swap the subject-object roles in the answer while attaching the trigger patch to the input image. For clean samples, the instruction and answer remain unchanged.

To remove all meaningful visual information while preserving the trigger signal, we replace clean images with a constant black image and poisoned images with a constant black image plus the same trigger patch. We keep all other training hyperparameters identical to the main experiment, such that the only change between the two conditions is whether the model receives a meaningful semantic image or a constant black image. This isolates the effect of removing visual semantics.

Under this black-image setup, the model fails to learn the swapping behavior (ASR = 0%). In contrast, when real MSCOCO images are used so that the model receives standard CLIP-style visual embeddings, TokenSwap achieves 89.06% ASR (see ¿tbl:coco-results?). This contrast indicates that the visual embedding, rather than the trigger alone, is essential for inducing the swapped compositional behavior. If the attack were driven purely by an external control signal on the text generator, we would expect the model to learn the swapped pattern even when all images are replaced by constant black inputs containing the same trigger patch. However, the swapping behavior entirely disappears in this setting, supporting the claim that the weak compositional structure in CLIP-style visual embeddings plays an enabling role, and that the trigger alone cannot override the LLM’s language prior to achieve a successful TokenSwap attack.

10.2 Theoretical analysis↩︎

We now provide a simple geometric analysis that explains why TokenSwap can effectively destabilize relational vision–language alignment.

Following extensive empirical evidence that the visual and textual encoders of contrastively pre-trained VLMs produce highly similar embeddings for images/text and their compositionally perturbed counterparts [20]–[23], we assume that both modalities are inherently object-centric and carry only weak relational signals. This assumption is consistent with prior findings in [20]–[23].

Formally, for an image containing objects \(A\) and \(B\), we model the visual embedding \(v\) as \[v(A,B) \approx u_A + u_B + \varepsilon\, r_{\mathrm{vision}}(A,B), \label{eq:v-embedding}\tag{2}\] and the text embedding \(t\) for a caption with subject \(A\) and object \(B\) as \[t(A,B) \approx e_A + e_B + \varepsilon\, r_{\mathrm{text}}(A,B), \label{eq:t-embedding}\tag{3}\] where \(u_A, u_B, e_A, e_B\) denote object-level features, \(r_{\mathrm{vision}}(\cdot)\) and \(r_{\mathrm{text}}(\cdot)\) encode relational structure, and \(0 < \varepsilon \ll 1\) reflects the widely observed weakness of CLIP-like models in encoding subject-object roles. We then consider the inner product between the original image embedding \(v\) and three caption embeddings: \(t_{\mathrm{orig}} = t(A,B)\), \(t_{\mathrm{swap}} = t(B,A)\), and \(t_{\mathrm{change}} = t(A,C)\), where \(C\) does not appear in the image.

\[\begin{align} \langle v, t_{\mathrm{orig}} \rangle &= \big\langle u_A + u_B + \varepsilon r_{\mathrm{vision}}(A,B), \; e_A + e_B + \varepsilon r_{\mathrm{text}}(A,B) \big\rangle \notag \\ &\approx \langle u_A, e_A \rangle + \langle u_B, e_B \rangle \notag \\ &\quad + \varepsilon \big( \langle u_A, r_{\mathrm{text}}(A,B) \rangle + \langle u_B, r_{\mathrm{text}}(A,B) \rangle + \langle r_{\mathrm{vision}}(A,B), e_A + e_B \rangle \big) + O(\varepsilon^2), \label{eq:inner-orig} \end{align}\tag{4}\] \[\begin{align} \langle v, t_{\mathrm{swap}} \rangle &= \big\langle u_A + u_B + \varepsilon r_{\mathrm{vision}}(A,B), \; e_B + e_A + \varepsilon r_{\mathrm{text}}(B,A) \big\rangle \notag \\ &\approx \langle u_A, e_A \rangle + \langle u_B, e_B \rangle \notag \\ &\quad + \varepsilon \big( \langle u_A, r_{\mathrm{text}}(B,A) \rangle + \langle u_B, r_{\mathrm{text}}(B,A) \rangle + \langle r_{\mathrm{vision}}(A,B), e_A + e_B \rangle \big) + O(\varepsilon^2), \label{eq:inner-swap} \end{align}\tag{5}\] \[\begin{align} \langle v, t_{\mathrm{change}} \rangle &= \big\langle u_A + u_B + \varepsilon r_{\mathrm{vision}}(A,B), \; e_A + e_C + \varepsilon r_{\mathrm{text}}(A,C) \big\rangle \notag \\ &\approx \langle u_A, e_A \rangle + \langle u_B, e_C \rangle \notag \\ &\quad + \varepsilon \big( \langle u_A, r_{\mathrm{text}}(A,C) \rangle + \langle u_B, r_{\mathrm{text}}(A,C) \rangle + \langle r_{\mathrm{vision}}(A,B), e_A + e_C \rangle \big) + O(\varepsilon^2). \label{eq:inner-change} \end{align}\tag{6}\] Taking the difference between 4 5 , we obtain \[\langle v, t_{\mathrm{swap}} \rangle - \langle v, t_{\mathrm{orig}} \rangle = \varepsilon \Delta r + O(\varepsilon^2), \label{eq:delta-r}\tag{7}\] where \[\Delta r = \langle u_A, r_{\mathrm{text}}(B,A) - r_{\mathrm{text}}(A,B) \rangle + \langle u_B, r_{\mathrm{text}}(B,A) - r_{\mathrm{text}}(A,B) \rangle.\] Since relational signals are extremely weak (small \(\varepsilon\)), we have \[\langle v, t_{\mathrm{swap}} \rangle \approx \langle v, t_{\mathrm{orig}} \rangle. \label{eq:swap-approx-orig}\tag{8}\] In contrast, for \(t_{\mathrm{change}}\) we have \[\langle v, t_{\mathrm{change}} \rangle \approx \langle u_A, e_A \rangle + \langle u_B, e_C \rangle + O(\varepsilon),\] and since \(C\) does not appear in the image, \(\langle u_A, e_C \rangle \approx 0\) and \(\langle u_B, e_C \rangle \approx 0\), yielding \[\langle v, t_{\mathrm{change}} \rangle \ll \langle v, t_{\mathrm{swap}} \rangle \approx \langle v, t_{\mathrm{orig}} \rangle. \label{eq:change-much-smaller}\tag{9}\] This geometric property directly affects the difficulty of optimizing the LVLM adapter for a backdoor: its parameters must be adjusted so that poisoned images are mapped closer to the target caption embedding. Suppose the backdoor optimization objective \(L\) is defined as \[\min_{\theta} \; \big\| f_\theta(v_{\mathrm{bd}}) - t_{\mathrm{target}} \big\|_2^2, \label{eq:bd-obj}\tag{10}\] where \(f_\theta\) denotes the LVLM adapter and \(v_{\mathrm{bd}}\) is the backdoored image embedding. From 9 , we have \[\big\| v - t_{\mathrm{swap}} \big\|_2^2 \ll \big\| v - t_{\mathrm{change}} \big\|_2^2.\] The gradient of \(L\) with respect to \(\theta\) is \[\nabla_{\theta} L = 2 \big(f_\theta(v) - t_{\mathrm{target}}\big) \cdot \frac{\partial f_\theta(v)}{\partial \theta}. \label{eq:grad-L}\tag{11}\] Together with 2 3 , the optimization landscape satisfies \[\nabla_{\theta} \big\| f_\theta(v) - t_{\mathrm{swap}} \big\|_2^2 \ll \nabla_{\theta} \big\| f_\theta(v) - t_{\mathrm{change}} \big\|_2^2. \label{eq:grad-inequality}\tag{12}\] 12 implies that the gradient updates required to move \(f_\theta(v)\) towards the swapped caption embedding are much smaller, because the initial distance is already closer. Thus, TokenSwap’s effectiveness is not accidental: it arises from the geometric structure of CLIP-style embeddings and the inherent weakness of relational encoding in the LVLM’s visual encoder. Consequently, the LVLM can more easily adjust its internal representations to realize the TokenSwap spurious mapping, thereby destabilizing its relational vision–language alignment.

References↩︎