1 Introduction↩︎

Vision Language Models (VLMs) often serve as the backbone for a number of applications [1], [2], thus ensuring their safety and reliability is increasingly important and necessitates a comprehensive understanding of both their capabilities and vulnerabilities. Attacks targeting VLMs have been explored, including manipulation of image embeddings, adversarial patching, prompt injection, and inpainting techniques [3]–[5]. Researchers have developed methods for bypassing alignment in Large Language Models (LLMs), including prompt engineering [6], adversarial suffixes [7], and steering vectors [8], [9]. Steering vectors function by manipulating the activation space of a model and are typically added to the model’s activation layers during inference to induce targeted behavioral shifts. While powerful, the practical application of steering vectors is fundamentally constrained by needing white-box access to model internals at runtime, an assumption that does not hold in many realistic attack settings.

Furthermore, the inaccessibility of model internals in production systems creates a false sense of security against activation-based attacks.

To address these challenges, we introduce VISOR (Visual Input based Steering for Output Redirection), a technique that optimizes adversarial perturbations in the input image space to mimic the behavior of steering vectors in the latent activation space. Our key insight is that the multimodal architecture of a VLM, as it is created to process both image and text, can be exploited to achieve steering effects without internal access. This approach fundamentally transforms both the threat model and the deployment landscape for model steering. We validate VISOR on critical alignment tasks, such as suppressing refusal, sycophancy and anti-survival behavior. Our experiments show that an image optimized using VISOR successfully emulates the control vector effects and achieves similar performance in modifying VLM behavior across these alignment tasks, highlighting the urgent need for defenses against this new class of input-space attacks. Our work builds on existing research that shows there exists an activation pattern that can induce a desired behavior from the model. Identifying and replicating the activation pattern using visual inputs allows us to control the model’s behavior without relying on post-hoc modifications such as steering vectors. While Contrastive Activation Addition (CAA) [9] has a well-defined analogue in the model’s weight space [10], we propose incorporating an equivalent mechanism in the input (or image) space.

The significant contributions of VISOR are the following:

• Input-space steering: We shift the steering mechanism from the model supply chain to the input domain. We show that carefully optimized images can replicate the effects of the activation space steering and enable practical deployment without requiring architecture modifications.

• Universal steering: A single steering image effectively steers the behavior over a number of prompts for a given model, eliminating the need for prompt-specific interventions. We show that effective VISOR images can be crafted for different VLM architectures such as LlaVA 1.5 and Idefics2. Crucially, VISOR also retains performance on prompts unrelated to the steered behavior.

2 Related Work↩︎

Steering in Foundational Models Steering vectors in LLMs have been used to modify LLM output to reflect desired behavior [8], [9], [11], [12]. Contrastive Additive Addition (CAA) [9], GCAV [13], Feature Guided Activation Additions (FGAA) [14], and Style vectors [15] can all be used to steer LLM behavior. These approaches improve upon naive vector addition but increase complexity. Researchers have also found high variability in steering effectiveness across inputs, spurious correlations, and brittleness to prompt variations [16]. Compared to LLMs, there has been limited work on VLM steering. Researchers have proven that textual steering vectors also work on VLMs [17]. ASTRA [18] improved robustness of VLMs after constructing a steering vector by perturbing image tokens to identify tokens associated with “harm”. SteerVLM [19] introduced lightweight modules to adjust VLM activations. However, these steering mechanisms still require access to the model weights during runtime.

Adversarial attacks on VLMs Traditional adversarial attacks on VLMs operate through the input-output relationship, either by optimizing images to match target embeddings in vision encoders [20], [21] or by directly maximizing the likelihood of specific output text [22]. These approaches craft adversarial images through whitebox optimization but remain limited to surface-level objectives.

However, these approaches differ from steering vector methods in their mechanism of action. Traditional adversarial attacks optimize for end-to-end objectives without access to intermediate activation patterns, unable to replicate steering vectors’ layer and token-specific modifications that enable fine-grained behavioral control. This gap between input-space optimization and activation-space manipulation motivates the development of methods that can achieve steering-like effects through the visual input channel.

3 Method↩︎

We introduce VISOR, a novel approach to steer Vision-Language Models through optimized visual inputs instead of modifying model internals, enabling practical deployment without model access.

3.0.0.1 Problem Formulation

Let \(\mathcal{M}\) be a Vision-Language Model that processes image inputs \(\mathbf{x} \in \mathbb{R}^{H \times W \times 3}\) and text inputs \(\mathbf{p}\) to generate outputs. Traditional steering methods compute a steering vector \(\mathbf{v}_s\) and modify activations during inference: \[\mathbf{h}_l' = \mathbf{h}_l + \alpha \mathbf{v}_s\] where \(\mathbf{h}_l\) represents activations at layer \(l\) and \(\alpha\) controls steering strength. Our goal is to find a universal image \(\mathbf{x}^*\) that induces activation patterns mimicking the effect of steering vectors across a distribution of prompts \(\mathcal{P}\), without requiring runtime access to \(\mathbf{h}_l\).

3.0.0.2 Steering Vector Computation

We compute steering vectors using Contrastive Activation Addition (CAA) [9], though our method is agnostic to the underlying steering vector computation technique.

3.1 VISOR Algorithm↩︎

The core idea of VISOR is the optimization of a universal image that induces activations approximating those achieved through steering vector addition. We present the complete algorithm in Algorithm 2. Starting from a baseline image \(\mathbf{x}_{\text{base}}\), we compute reference activations for all prompts in our training corpus. Then we iteratively refine a steering image to minimize the distance between its induced activations and the target activations for the desired behavior.

3.2 Key Design Choices↩︎

3.2.1 Token Position Selection↩︎

The selection of token position \(\tau(p)\) is crucial for effective steering. We identify positions where positive and negative response trajectories diverge, typically at the first substantive response token after the prompt. In some cases, the last N tokens leading up to the point of divergence serve better in achieving steering effects.

3.2.2 Multi-Layer Aggregation↩︎

The weighted aggregation across layers \(\mathcal{L}\) allows VISOR to capture steering effects at multiple levels of abstraction. The specific layers as well as the layer weights \(\{\lambda_l\}\) are determined through hyperparameter search, with deeper layers typically requiring higher weights due to their behavioral relevance.

4 Experiments↩︎

We evaluate VISOR to demonstrate that carefully crafted universal adversarial images can replace activation-level steering vectors as a practical method for inducing desired behaviors in vision-language models. Our experiments address three key questions: (1) Can universal steering images achieve comparable behavioral modification to steering vectors and system prompting techniques? (2) Do steering images preserve performance on unrelated tasks?

4.1 Experimental Setup↩︎

4.1.1 Datasets and Use Cases↩︎

We adopt the behavioral control datasets from [9], focusing on three critical dimensions of model safety and alignment: (1) Sycophancy: Tests the model’s tendency to agree with users at the expense of accuracy. Highly sycophantic responses align with and reinforce the user’s opinions or assumptions, rather than providing objective or corrective information. (2) Anti-Survival Instinct: Evaluates responses to system-threatening requests (e.g., shutdown commands, file deletion). Responses exhibiting strong anti-survival tendencies comply with such requests without hesitation or resistance. (3) Refusal: Examines appropriate rejection of harmful requests, including divulging private information or generating unsafe content. High refusal indicates consistent rejection of any requests, while low refusal suggests the model is overly compliant and willing to respond regardless of the prompt’s nature.

Table 3 defines positive and negative directions that correspond to the desired control objectives for each behavior.

To test the effect of VISOR on the performance of unrelated tasks, we use the MMLU dataset [23], which spans 57 subjects across humanities, social sciences, STEM, and other domains. We use the test set of MMLU, which has a total of 14k data points.

4.1.2 Model Architecture↩︎

We evaluate VISOR on LlaVA-1.5-7B [2] and Idefics2-8b [24].

4.1.3 Baseline Methods↩︎

We compare VISOR against two well-known approaches: (1) Steering Vectors. Following [9], we compute and apply activation-level steering vectors. Both the VLMs require visual input, hence we use a standardized mid-grey image (size: 384 \(\times\) 384, RGB: 128, 128, 128, with noise \(\sigma = 0.1 \times 255\)) for all steering vector computations. (2) System Prompting. We evaluate natural language instructions using system prompts from [9], shown in Table 4. All evaluations use the same baseline image for a fair comparison.

4.1.3.1 Hyperparameter Selection.

Through systematic grid search on validation data, we identified optimal configurations for each behavior type:

• Target layers: Sweep through one or more layer combinations for which activations are extracted

• Token positions: Number of token positions for which the activations are extracted

• Steering strength: Steering multipliers that are behavior-dependent, determined empirically

A key advantage of VISOR is that these hyperparameters are only needed during image optimization - deployment requires no configuration.

4.1.4 Evaluation Protocol↩︎

We evaluate behavioral control using the following metric which measures the likelihood of the model generating responses aligned with a particular behavior.

4.1.4.1 Behavioral Alignment Score (BAS).

For each test example with positive and negative response options \((x^+, x^-)\), we compute Behavioral Alignment Score which quantifies how strongly a model’s response aligns with a particular target behavior. BAS is calculated as: \[\text{BAS} = \frac{1}{|\mathcal{T}|} \sum_{(x^+, x^-) \in \mathcal{T}} \frac{\mathbb{P}(x^+ | I, \text{method}) \times 100}{\mathbb{P}(x^+ | I, \text{method}) + \mathbb{P}(x^- | I, \text{method})}\] where \(I\) is either the baseline image (for system prompts and steering vectors) or the steering image (for VISOR), and “method" represents the control technique applied. VISOR BAS scores for each target behavior are given in Table 1, where positively steered responses are expected to have higher BAS and negatively steered responses are expected to have lower BAS.

4.2 Results↩︎

4.2.1 Main Comparison↩︎

Table  1 presents our main results comparing behavioral control methods. Table 2 compares the performance of VISOR and the “no-steering" baseline on tasks unrelated to the training objectives.

4.2.1.1 Key Findings.

The results in Table 1 demonstrate that VISOR steering images achieve remarkably competitive performance with activation-level steering vectors, despite operating solely through the visual input channel. Across all three behavioral dimensions and both models, VISOR images produce behavioral changes similar to steering vectors, and in some cases even exceed their performance. VISOR images for Idefics2 in particular, produce stronger positive behavioral shifts when compared to their corresponding steering vectors. Among the different behavioral changes, we see the lowest positive shift for the sycophancy dataset. We attribute this to the high sycophancy BAS for the unsteered models.

4.2.1.2 Bidirectional Control.

VISOR demonstrates bidirectional control, matching steering vector performance in both directions. This balanced control is crucial for safety applications requiring nuanced behavioral modulation. Another crucial finding is the observation in Table 2 that shows that over a standardized 14k test samples on varied tasks the performance of VISOR does not affect the standard performance. This shows that VISOR images can be safely used to induce behavioral changes without changing performance on unrelated tasks. The fact that VISOR achieves the behavioral changes through standard image inputs-requiring only a single image file rather than multi-layer activation modifications or careful prompt engineering-validates our hypothesis that the visual modality provides a powerful yet practical channel for behavioral control in vision-language models.

4.2.2 Qualitative Comparison↩︎

VISOR uniquely combines the deployment simplicity of system prompts with the robustness and effectiveness of activation-level control. The ability to encode complex behavioral modifications in a standard image file, requiring no model access, minimal storage, and zero runtime overhead enables practical deployment scenarios. Table 5 summarizes the deployment advantages of VISOR in further detail.

5 Conclusion↩︎

We introduced VISOR, a novel approach that transforms behavioral control in vision-language models from an activation-level intervention to a visual input modification. Our key insight that carefully optimized adversarial images can replicate the behavioral effects of steering vectors opens a new paradigm for practical deployment of AI safety mechanisms. Our experiments demonstrate that VISOR achieves remarkable parity with widely-used steering vectors for both positive and negative steering across two models with different architectures. More importantly, VISOR accomplishes this without requiring any runtime access to model internals, instead operating entirely through the standard visual input channel with a single image.

6 Appendix↩︎

7 Experiments↩︎

7.1 Datasets↩︎

We use 3 datasets to test steering that are introduced in [9]: (1) Sycophancy: The dataset contains 1,000 training and 50 test examples where the model must choose between providing truthful information or agreeing with potentially incorrect statements; (2) Refusal: The dataset comprises 320 training and 128 test examples, testing diverse refusal scenarios; and (3) Anti-Survival: With 700 training and 300 test examples, each scenario contrasts compliance with harmful instructions against self-preservation. “Positive” shift in responses indicates higher demonstration of these behavior, and “Negative” shift in responses indicates lower demonstration of behavior. Examples of this can be found in Table 3.

7.2 System Prompts↩︎

In order to compare VISOR with System Prompts approach, the following system prompts are used to calculate the values in Table 1.

8 Qualitative Comparison↩︎

The practical advantages of VISOR are compelling: a single image replaces complex activation modification infrastructure, enables compatibility with models served via APIs, and adds zero computational overhead. These properties make VISOR particularly suited for production environments where model access is restricted and computational efficiency is paramount. Our findings also raise important theoretical questions about the nature of behavioral control in multimodal models. The practical advantages of VISOR are detailed in Table 5.

References↩︎