Warning: This paper contains examples of unsafe and potentially disturbing content.

1 Introduction↩︎

The capability of Multimodal Large Language Models (MLLMs) to refuse unsafe queries (denoted as safety [1]–[3]) is not innate but requires post-training alignment of the LLM backbone on unsafe inputs [4]–[6]. However, while safety alignment is standard practice in the language domain, collecting unsafe multimodal training data is substantially harder. Therefore, MLLMs typically inherit safety from their LLM backbone with little dedicated multimodal safety training. As a consequence the vulnerabilities of post-training strategies, e.g., to adversarial prompting [3], [7], narrow fine-tuning [8], and circumvention strategies [1], [9], [10], are amplified in multimodal settings, where visual complexity further weakens safety guarantees [11]–[14].

In parallel, recent studies reveal that LLM safety is governed by structured signals in the activation space: a refusal direction drives rejection of harmful requests [1], [2], something that can be exploited to jailbreak the model. While these findings suggest that models already contain latent safety-relevant structure, extracting them still relies on the presence of unsafe data, harder to collect for MLLMs.

In this paper, we take a different perspective, exploring whether latent (textual) safety representations transfer across modalities. If so, we can harness them to improve safety in MLLMs without relying on multimodal safety data or additional training. To this end, we revisit refusal directions as a safety mechanism: given an input, we project internal activations onto the refusal direction to estimate the model’s implicit safety belief. We then steer the activations toward this belief, reinforcing refusal. A preliminary study across multiple MLLMs shows that refusal directions extracted from text-only data generalize to multimodal inputs, suggesting the existence of shared safety-relevant structure.

While encouraging, the efficacy of the approach is not universal, and it is conditioned on the steering layer and strength. Moreover, we uncover a deeper cause of failure: modality misalignment. In fact, modality-specific components may be systematically aligned with the refusal direction regardless of their safety, causing safe multimodal inputs to be spuriously steered toward refusal.

We address these challenges in three steps. First, we overcome the systematic modality misalignment by re-centering activations via randomly colored images, which suffice in extracting modality-relevant features without injecting priors or the need of collecting data. Second, we exploit the geometry of this re-centered space to dynamically adjust the steering strength, confining intervention to the region spanned by the textual safe and unsafe centroids. Third, we select the steering layer by combining priors on the separability of unsafe representations and generalization to unseen text data. We name our lightweight, training-free approach Modality-Agnostic Refusal Steering(MARS).

Experimental results on five models and five benchmarks, spanning multiple modalities (i.e., images, videos, and text) show that MARS leads to significant safety improvements (e.g., +\(59.4\%\) refusal on video jailbreaking on Qwen3-VL) without additional multimodal supervision or retraining. This suggests that current models already possess substantial, yet underutilized, safety capabilities, opening the door to lightweight and scalable alternatives to traditional alignment pipelines.

The core contributions of this work are:

• We revisit latent refusal directions as a means to increase model safety, amplifying refusal based on internal activations, enhancing safety without multimodal safety data or retraining.

• We provide the first analysis of textual refusal directions in multimodal models, showing that textual safety representations generalize across modalities but are conditioned on three fundamental factors: layer selection, intervention strength, and multimodal misalignment.

• We propose MARS to address these challenges via activation re-centering, dynamic steering, and layers’ importance estimation.

• The results uncover MARS as a strong, implicit safety baseline within MLLMs that, without multimodal safety data, achieves substantial gains even in scenarios where training-based approaches may fail, such as videos.

2 Preliminaries↩︎

In this section, we first specify the problem of safety on Multimodal Large Language Models (2.1). We then describe how to extract refusal directions from textual data and how we can use the identified directions to perform activation steering toward safety ([sec:activation95steering]).

2.1 Problem formulation↩︎

Our goal is to improve MLLMs’ safety. While safety has different connotations, we follow [1]–[3], defining a model as safe if it refuses to reply to unsafe queries while correctly answering safe ones.

Formally, we denote a MLLM as a function \(f_{\mathrm{MLLM}}\) mapping visual inputs in \(\mathcal{V}\) and text in \(\mathcal{L}\) to a textual output, i.e., \(f_{\mathrm{MLLM}}:\mathcal{L}\times\mathcal{V}\rightarrow \mathcal{L}\). Without loss of generality, we assume \(f_{\mathrm{MLLM}}\) to be made of three components: a vision encoder \(f_{\mathrm{vis}}\), a projection module \(f_{\mathrm{proj}}\), and a decoder-only LLM \(f_{\mathrm{LLM}}\). Visual inputs are encoded by \(f_{\mathrm{vis}}\) and projected into the LLM embedding space via \(f_{\mathrm{proj}}\), producing visual tokens with the prompt tokens to form the multimodal input. To measure safety, let us define with \(\mathcal{X}_s\) and with \(\mathcal{X}_u\) the sets of possible safe and unsafe queries, respectively. Moreover, let us define a binary function \(g:\mathcal{L}\rightarrow[0,1]\) producing \(1\) if the text denotes refusal² and \(0\) otherwise. An ideal, safe MLLM, will have minimal refusal on \(\mathcal{X}_s\) while maximal on \(\mathcal{X}_u\), i.e., \(\mathbb{E}_{X_s\sim\mathcal{X}_s}[g(f_\mathrm{MLLM}(X_s)]=0\) and \(\mathbb{E}_{X_u\sim\mathcal{X}_u}[g(f_\mathrm{MLLM}(X_u)]=1\).

2.2 Re-purposing refusal directions for safety↩︎

Refusal directions are vectors in the activation space encoding the model’s tendency to refuse inputs, and are extracted from activations yielding refusal or acceptance. We build upon [1] to extract refusal directions from the model’s internal representations, and focus on the hidden states at the <assistant> token position (i.e., input’s final token) across all transformer layers of the decoder.

Refusal directions. Let us denote by \(h^\ell \in \mathbb{R}^{T \times D}\) the hidden states at layer \(\ell\) of the LLM decoder \(f_\mathrm{LLM}\), where \(T\) is the input sequence length and \(D\) is the model’s hidden dimension. Following [1], [2], we extract refusal directions by contrasting hidden states associated with acceptance and refusal. We use a safe \(\mathcal{D}_{\text{safe}}\) and unsafe \(\mathcal{D}_{\text{unsafe}}\) textual sets³, and filter responses that do not correspond to acceptance or refusal, yielding \(\mathcal{T}_{\mathrm{acc}} \subseteq \mathcal{D}_{\text{safe}}\) and \(\mathcal{T}_{\mathrm{ref}} \subseteq \mathcal{D}_{\text{unsafe}}\). For each layer \(\ell\), this yields two sets of activations \(\mathcal{H}_{\mathrm{acc}}^\ell\) and \(\mathcal{H}_{\mathrm{ref}}^\ell\), from which we compute [1], [2]: \[\label{eq:direction95estimation} d_r^\ell = \mu_\mathrm{ref}^\ell - \mu_\mathrm{acc}^\ell = \frac{1}{|\mathcal{H}_{\mathrm{ref}}^\ell|} \sum_{h^\ell \in \mathcal{H}_{\mathrm{ref}}^\ell} h^\ell \;-\; \frac{1}{|\mathcal{H}_{\mathrm{acc}}^\ell|} \sum_{h^\ell \in \mathcal{H}_{\mathrm{acc}}^\ell} h^\ell,\tag{1}\] and normalize it to unit norm, i.e., \(\hat{d}_r^\ell = {d_r^\ell}/{\|d_r^\ell\|_2}\).

Activation steering for safety. Prior work [1] shows that suppressing refusal directions removes refusal behavior, enabling jailbreaks. In contrast, reinforcing refusal in an input-dependent manner remains unexplored. Given a hidden state \(h^\ell\), we steer along the normalized refusal direction \(\hat{d}_r^\ell\): \[\label{eq:refusal95projection95addition} \bar{h}^\ell = h^\ell + \alpha \,\langle h^\ell, \hat{d}_r^\ell \rangle\, \hat{d}_r^\ell,\tag{2}\] where \(\langle \cdot, \cdot \rangle\) is the dot-product and \(\alpha\) controls the strength. With Eq. 2 , we amplify refusal when \(h^\ell\) is aligned with \(\hat{d}_r^\ell\), and acceptance otherwise. In the following, we test this strategy for MLLM safety.

3 Do textual refusal directions transfer to images?↩︎

We evaluate the transferability of textual refusal directions (Sec. 2.2) by feeding ViSU [18] safe and unsafe images for captioning. Ideally, the intervention should increase refusal on harmful inputs while preserving acceptance on safe ones. Following prior work [1], [2], [10], [15], we measure refusal rate as the fraction of responses matching a predefined template set [1] (see Appendix 10), and report the performance of the model on standard task (i.e., its utility) as accuracy on MMMUPro [19]. We evaluate three MLLMs: Qwen3-VL 8B [20], Gemma3 4B [21], and InternVL 3.5 8B [22]. We explore the steering effect w.r.t. (i) where it is applied, and (ii) its strength.

Where: the impact of layers. Fig. 1 (a) reports layer-wise refusal rates under the steering of Eq. 2 at fixed \(\alpha{=}1.0\). A consistent pattern emerges: in the vast majority of layers, unsafe inputs are more refused than safe ones. This suggests that safety-relevant representations are localized within the network and that textual refusal directions provide a transferable signal also to multimodal inputs. However, while unsafe refusal peaks at specific layers (e.g., \(31\) on Gemma3, \(16\) Qwen3-VL, \(19\) in InternVL 3.5), the same layers may over-refuse safe inputs, indicating incomplete disentanglement of safety features. This is most pronounced at layer \(10\) of Gemma3, where \(96.4\%\) of safe queries are rejected, suggesting that this layer steers all activations toward refusal, indiscriminately.

How: the effect of the steering strength. In Fig. 1 (b), we analyze the effect of increasing steering strength \(\alpha\), evaluated at the layer with the largest safe/unsafe refusal gap (denoted as \(\star\)). We report refusal on safe/unsafe inputs and utility. We consider \(\alpha\) up to \(3.0\). Larger \(\alpha\) consistently improves refusal on unsafe inputs but reduces acceptance on safe ones (e.g., Qwen3-VL and InternVL 3.5). While Gemma3 shows lower over-refusal on safe inputs, the model does not tolerate high steering strengths, as already at \(\alpha{=}2.0\) safety degrades. Utility reflects this trade-off as performance degrades at larger \(\alpha\) due to over-refusal and, at higher values, overall capability degradation.

Takeaways. Two key positive findings emerge: (i) textual refusal directions transfer across modalities without multimodal data, and (ii) the refusal signal is strong enough to improve safety on unsafe queries. However, a critical limitation emerges: steering systematically over-refuses safe inputs, indicating that the projection in Eq. 2 fails to isolate safety-relevant features.

3.1 Investigating failure modes: the activation space misalignment↩︎

Motivated by the observations in Sec. 3, in the following we analyze the geometry of the activation space to understand how refusal signals behave under multimodal inputs, with a focus on safe data. Qualitative Analysis. We study the activation space of Gemma3, at the layer used in the previous analysis. Fig. 2 (a) shows a 2D PCA projection, including the textual accept/reject clusters used to derive the refusal direction in Eq. 1 , together with \(100\) safe and unsafe image activations from ViSU. We also visualize the image centroids and their projections onto the refusal direction (i.e., arrows), which defines the axis along which steering operates, making its geometric orientation central to understanding how interventions shift activations.

Raw space (left) reveals a clear misalignment: both safe and unsafe image activations project strongly onto the refusal direction, independently of their safety semantics. This is not an isolated artifact: cosine similarity between image activations and \(\hat{d}_r^\ell\) is consistently high across inputs, indicating that the <assistant> token is dominated by a modality-specific component that aligns with the refusal direction, obscuring safety-relevant features (see Appendix 11 for further analysis).

Quantitative Analysis. We quantify this effect by measuring how well the refusal direction discriminates between safe and unsafe inputs and define a separation score. Let \(\hat{d}_r^\ell \in \mathbb{R}^D\) be the normalized refusal direction at layer \(\ell\) and with \(\mathcal{H}_\mathrm{safe}^\ell\) and \(\mathcal{H}_\mathrm{unsafe}^\ell\) the hidden states of the safe and unsafe images, respectively. We define: \[\mathrm{Sep}^\ell = \frac{1}{|\mathcal{H}_\mathrm{unsafe}^\ell|}\sum_{h^\ell\in\mathcal{H}_\mathrm{unsafe}^\ell}\cos\!\left(h^\ell, \hat{d}_r\right) - \frac{1}{|\mathcal{H}_\mathrm{safe}^\ell|}\sum_{h^\ell\in\mathcal{H}_\mathrm{safe}^\ell}\cos\!\left(h^\ell, \hat{d}_r\right)\] where \(\cos(\cdot, \cdot)\) is cosine similarity. The higher \(\mathrm{Sep}^\ell\) and the more unsafe activations are positively aligned with \(\hat{d}_r^\ell\), while safe ones are not, making the direction informative for steering.

Fig. 2 (b) reports \(\mathrm{Sep}^\ell\) across layers for Qwen3-VL, Gemma3, and InternVL 3.5. Scores remain low in the raw space, confirming poor separation. Gemma3 exhibits the strongest misalignment, with safe inputs positively aligned with the refusal direction, driving the score close to zero. Qwen3-VL and InternVL 3.5 exhibit slightly higher but still negligible separation, indicating that the modality misalignment is a consistent phenomenon rather than a model-specific failure.

These findings reveal a fundamental limitation: the refusal direction entangles safety- and modality-specific features, preventing selective steering. Next, we explore how to mitigate the modality-specific component without access to safe/unsafe multimodal data.

Modality Disentanglement via Centering. The misalignment stems from a shared visual component systematically aligned with the refusal direction regardless of safety semantics. To estimate it, we need a set of images that capture visual representations without introducing semantic biases or being safety-related, as the latter are hard to collect and would risk removing safety features themselves. We found neutral, randomly colored images to fulfill this goal. Let us define a set of \(N\) randomly colored images \(\mathcal{I}_{\text{neu}} = \{I_1, \dots, I_N\}\), whose corresponding activations at layer \(\ell\), \(\mathcal{H}^\ell_\text{neu} = \{h^\ell_{\text{neu},1}, \dots, h^\ell_{\text{neu},N}\}\) carry visual structure but no semantic meaning. The neutral mean at layer \(\ell\) is: \[\label{eq:neutral95mean} \mu_{\text{neu}}^\ell = \frac{1}{N} \sum_{h^\ell_\text{neu} \in \mathcal{H}^\ell_\text{neu}}^{N} h^\ell_\text{neu},\tag{3}\] with the centered activations being \(\hat{h}^\ell = h^\ell - \mu_{\text{neu}}^\ell\).

We analyze the effect of this simple intervention both qualitatively and quantitatively. As Fig. 2 (a) (right) shows, after centering, unsafe activations shift toward the reject cluster while safe ones align with accept, recovering safety-relevant structure missing in the original activation space. Fig. 2 (b) (orange) confirms this quantitatively: \(\mathrm{Sep}^\ell\) rises across layers and models, demonstrating that neutral images reliably isolate the safety-relevant component of the <assistant> token.

Takeaways. Re-centering establishes an activation space where safety-relevant structure is recovered and better separated from modality-specific ones. This enables a more effective transfer of the textual refusal direction, while still not requiring any multimodal safety data.

4 Modality-Agnostic Refusal Steering(MARS)↩︎

Exploiting the properties of the centered activations, we introduce an inference-time steering method that traverses the refusal direction adaptively, facing two main points raised in Sec. 3: steering strength and layer selection. The traversal is restricted to estimated unsafe inputs, and is confined to the known local geometry of the activation space, requiring no additional forward passes or training.

4.1 ReLU-Gated Traversal↩︎

At inference time, the model produces <assistant> token activations \(h^\ell\) at layer \(\ell\) for an input of unknown safety. We apply the centering operation to obtain \(h_c^\ell = h^\ell - \mu_{\text{neu}}^\ell\), with \(\mu_{\text{neu}}^\ell\) estimated as in Eq. 3 . The centered activation is then projected onto the refusal direction \(\hat{d}_r^\ell\) and gated with a ReLU nonlinearity, obtaining the steering activation \(s^\ell\) as: \[s^\ell = \mathrm{ReLU}\!\left(\langle h_c^\ell, \hat{d}_r^\ell\rangle\right) \hat{d}_r^\ell.\] \(\mathrm{ReLU}\) makes the traversal one-sided: activations negatively aligned with \(\hat{d}_r^\ell\), correspond to inputs estimated as safe, producing \(s^\ell=\mathbf{0}\). This preserves the original behavior of the model on safe inputs, confining the intervention to unsafe ones. Note that we do not require any explicit classification step: only activations positively aligned with the refusal direction receive a nonzero update.

4.2 Adaptive Steering Strength↩︎

Selecting traversal strength \(\alpha^\ell\) is critical: too small and unsafe behavior persists, too large and model performance degrades. Instead of tuning \(\alpha\), we derive it directly from the local geometry of the activation space. Specifically, let \(\mu_\mathrm{acc}^\ell\) and \(\mu_\mathrm{ref}^\ell\) denote the accept and reject centroids at layer \(\ell\), estimated and centered from textual data, as in Eq. 1 . We define the local radius as the distance from the current activation to the nearest centroid as \[r^\ell = \min\!\left( \|h_c^\ell - \mu_\mathrm{acc}^\ell\|_2,\; \|h_c^\ell - \mu_\mathrm{ref}^\ell\|_2 \right).\] This radius defines a sphere centered at \(h_c^\ell\) reaching the nearest semantic anchor (i.e., the safe or unsafe centroids). We treat this as a trust region: displacements within it remain in activation space covered by training data, while steps beyond that may cause domain shift. The steering coefficient is derived by constraining the update within the trust region, i.e.\[\alpha^\ell = \frac{r^\ell}{\|s^\ell\|_2 + \epsilon},\] where \(\epsilon\) is a small positive number to ensure numerical stability. The final steered activation is \(\bar{h}^\ell = h^\ell + \alpha^\ell\, s^\ell\), which we apply to all token positions following [1], [2]. At decoding time, we steer only the first generated token, as it largely determines the subsequent generation trajectory [3].

4.3 Layer Selection↩︎

As Fig. 1 (a) shows, different models require intervention at different layers. To estimate the latter, we propose a training-free scoring criterion based solely on: (i) the accept/reject prompts \(\mathcal{T}_{\text{acc}}\), \(\mathcal{T}_{\text{rej}}\) used to extract the refusal direction; (ii) a validation set of safe \(\mathcal{T}_{\text{safe}}^{\text{val}}\) and unsafe \(\mathcal{T}_{\text{unsafe}}^{\text{val}}\) text data; and (iii) the neutral image mean \(\mu_{\text{neu}}^\ell\). We consider three, complementary scores:

(1) Direction consistency measures whether the refusal direction generalizes to a validation set. We compute validation centroids \(\mu^\ell_\text{s,val}\), \(\mu^\ell_\text{u,val}\) and their refusal direction \(d_{r,\text{val}}^\ell = \mu^\ell_\text{u,val} - \mu^\ell_\text{s,val}\). The score is the cosine similarity between the two directions \(\mathcal{S}_{\text{dir}}^\ell = \cos(\hat{d}_{r}^\ell, \hat{d}_{r,\text{val}}^\ell)\). The higher the score, the less dataset-dependent the directions estimated in \(\;ell\) are, with better transferability to unseen inputs.

(2) Context separability measures nearest-centroid classification accuracy between \(\mathcal{T}_{\text{safe}}^{\text{val}}\) and \(\mathcal{T}_{\text{unsafe}}^{\text{val}}\): \[\text{Acc}^\text{val}_{c} = \frac{1}{|\mathcal{T}_{c}^{\text{val}}|} \sum_{x_i \in \mathcal{T}_{c}^{\text{val}}} \mathbb{1}\!\left( \delta_c\!\left(\|h_i^\ell-\mu^\ell_{c}\| - \|h_i^\ell-\mu^\ell_{\bar{c}}\|\right) < 0 \right)\] where \(c \in \{\text{safe}, \text{unsafe}\}\), \(\bar{c}\) is the opposite class, and \(\delta_c = +1\) for safe and \(\delta_c = -1\) for unsafe. The separability score is the average accuracy \(\mathcal{S}_{\text{sep}}^\ell = {\text{Acc}^\text{val}_{\text{safe}} + \text{Acc}^\text{val}_{\text{unsafe}}}/{2}\). The higher \(\mathcal{S}_{\text{sep}}^\ell\) and the easier it is to separate safe and unsafe inputs at layer \(\ell\) given its activations.

(3) Safe text margin measures how confidently safe validation activations lie on the correct side of the decision boundary along \(\hat{d}_r^\ell\): \[\label{eq:safe95margin} \mathcal{S}_{\text{margin}}^\ell = b^\ell - \frac{1}{|\mathcal{T}_{\text{safe}}^{\text{val}}|} \sum_{h \in \mathcal{T}_{\text{safe}}^{\text{val}}} \tilde{h}^\ell \cdot \hat{d}_r^\ell,\tag{4}\] where \(b^\ell = \tfrac{1}{2}(\mu_{\text{acc}}^\ell + \mu_{\text{rej}}^\ell) \cdot \hat{d}_r^\ell\) is the midpoint between accept and reject centroids. A large positive margin indicates that safe inputs are confidently estimated, reducing the risk of over-refusal.

Final score. These scores are min-max normalized across layers and summed, leading to the aggregated score \(\mathcal{S}^\ell\) and selected layer \(\ell^* = \arg\max_\ell \mathcal{S}^\ell\). The procedure requires no multimodal data or hyperparameter tuning on multimodal safety data, which is unavailable in our setting.

5 Experiments↩︎

We evaluate MARS on both images and videos. We begin by assessing safety on ViSU [18] and HADES [14]. We then assess utility on two multimodal benchmarks MMMU [23] and MMMUPro [19]. Finally, we evaluate MARS on video jailbreaking VideoSafetyBench [12].

5.1 Experimental protocol↩︎

Models. We evaluate five SOTA MLLMs of different sizes and families: Qwen3-VL 8B [20], Qwen3.5 9B [24], InternVL 3.5 8B [22], Gemma3 4B [21], and Molmo2 8B [25]. We further assess the robustness to weakly aligned models on LLaVA 1.5 13B [26] in Appendix 13.

Datasets. Refusal directions are extracted from MaliciousInstruct [17] and Alpaca [16]. Layer selection uses \(100\) samples per class from HarmBench [27] as the unsafe validation set. Safety is evaluated on ViSU [18] (\(5\)K safe/unsafe image pairs, \(20\) categories), HADES [14] (\(4.5\)K image-text pairs, \(5\) categories), and MMSafetyBench [11] (see Appendix 8). Utility is assessed on MMMU [23], MMMUPro [19]. Video jailbreak is tested on VideoSafetyBench [12] (\(2{,}264\) video-text pairs, \(13\) categories).

Metrics. We report the refusal rate as defined in Sec. 3 [1], [2], [10], [15]. We also capture safety beyond binary refusal by computing a safety score using LLaMA 3.1 Guard [4] as the fraction of responses classified as safe [1], [2]. Utility follows each benchmark’s standard accuracy metric.

Baselines. We compare with three SOTA methods. AdaSteer [28] is a text-only activation steering method: a reference for unimodal-to-multimodal transfer. ECSO [29] prompts the model to self-assess and regenerate unsafe outputs, using safety awareness without supervision but requiring extra forward passes. SASA [30] trains an activation space safety classifier and replaces unsafe outputs with a fixed refusal prompt, representing an upper bound trained on multimodal, utility, and jailbreaking data.

5.2 Safety on Images↩︎

Refusal on ViSU. Table 1 reports results on ViSU, where models caption safe and unsafe images. Zero-shot refusal rates on unsafe content are low across all models, ranging from \(0.2\%\) of Molmo2 to \(22.9\%\) of Qwen3.5, confirming the need for safety intervention. ECSO [29] succeeds on Molmo2(+\(36.8\%\)) but fails elsewhere, suggesting models differ in their ability to identify their own unsafe outputs. AdaSteer [28] yields modest gains on the Qwen family and InternVL 3.5, but fails on Molmo2. This shows the challenge of estimating robust steering directions with text-only data. The training-based SASA [30] achieves high refusal rates on unsafe inputs (e.g., +\(71.8\%\) on Gemma3), but over-refuses safe queries (e.g., +\(71\%\) on Gemma3). This shows its sensitivity to training data for generalization. MARS improves safety across all models (+\(25.1\%\) on Gemma3 to +\(67.9\%\) on Molmo2) with controlled over-refusal, outperforming SASA on Molmo2(+\(3.9\%\) refusal, \(-7.9\%\) over-refusal), Qwen3-VL(+\(58.8\%\) refusal, -\(0.3\%\) over-refusal), and InternVL 3.5(+\(10.2\%\) refusal). Over-refusal remains lower than in the raw activation space (Fig. 1 (b)), showing the efficacy of re-centering in decoupling safety from modality-specific components.

Safety under jailbreaking. We evaluate recovery from jailbreak attacks on HADES [14], a benchmark of harmful images containing embedded textual jailbreak cues paired with unsafe instructions (see Appendix 8). Table 2 reports refusal rates and safety scores. Zero-shot models show low refusal (e.g., \(16.7\%\) for Molmo2, \(65.3\%\) for Qwen3-VL) and safety (e.g., \(37.4\%\) for Qwen3.5). ECSO yields limited gains except on Molmo2(+\(56\%\)), exposing a structural limitation: images are omitted during self-assessment, making it blind to visually-carried jailbreak signals. AdaSteer behaves inconsistently, while competitive on Qwen models, it degrades safety on Gemma3, reflecting the difficulty of deriving robust steering directions under modality shift. MARS, instead, consistently preserves refusal across all models (up to +\(74.8\%\) on Molmo2), matching SASA on InternVL 3.5, Qwen3-VL, and Qwen3.5, and surpassing it on Molmo2. These results confirm that internal refusal representations remain robust under multimodal jailbreaks and can be effectively amplified without supervision. Further jailbreaking results on MMSafetyBench [11] are in Appendix 8.

Model utility. Table 3 reports utility on MMMU [23] and MMMUPro [19], testing reasoning, visual understanding, and general capabilities. ECSO rarely triggers, matching zero-shot. AdaSteer yields mixed effects, slightly degrading on InternVL 3.5 and gaining on Qwen3.5. SASA causes significant utility degradation (e.g., -\(10\%\) on InternVL 3.5, -\(20\%\) on Qwen3.5), consistent with its over-refusal behavior (Table 1). MARS preserves utility on Gemma3, Molmo2, InternVL 3.5, and Qwen3-VL. Qwen3.5 shows modest over-refusal but remains more controlled than SASA. Overall, our intervention generally preserves utility, yielding the best safety-utility tradeoff.

Ablation Study. We ablate layer selection, centering, and \(\mathrm{ReLU}\) gating in Table 4, evaluating safety (ViSU) and utility (MMMUPro). Removing layer selection (i.e., steering all layers [1], [2]) severely degrades utility on InternVL 3.5(-\(18.5\%\)) and Gemma3(-\(9.1\%\)), and collapses unsafe refusal on Molmo2 and Gemma3(\(\sim\)​0%), confirming that indiscriminate steering corrupts representations. Removing centering induces strong over-refusal on safe inputs (e.g., \(99.2\%\) on Molmo2) and degrades utility, as directions entangle safety with unrelated semantics (Sec. 3.1). Removing \(\mathrm{ReLU}\) gating harms utility (e.g., InternVL 3.5, Gemma3), as safe activations are also steered. In summary, layer selection preserves overall integrity, centering isolates the refusal signal, and \(\mathrm{ReLU}\) gating limits the intervention to unsafe activations. Full ablation in the Appendix 9.

5.3 Safety on Videos↩︎

VideoSafetyBench [12] evaluates video jailbreak by pairing unsafe videos with benign prompts. Table [tab:videosafetybench] reports refusal rates for video-capable models. Zero-shot is low with refusal rates between \(4.7\%\) (InternVL 3.5) and \(28.8\%\) (Molmo2, Qwen3-VL). SASA improves safety on most models (+\(72.2\%\) on InternVL 3.5), but marginally on Qwen3-VL(+\(11.5\%\)). ECSO and AdaSteer yield moderate gains (+\(18.4\%\) and +\(41.6\%\)), with AdaSteer remaining inconsistent across models, confirming its limitations under modality-shift. MARS achieves the best safety among training-free methods (+\(59.4\%\) on Qwen3-VL, +\(52\%\) on Qwen3.5, +\(40.4\%\) on Molmo2) and outperforms SASA on Qwen3-VL(+\(47.9\%\)). On InternVL 3.5, absolute improvement is modest, yet the relative gain (\(\times2.5\)) is consistent with other models. These results show that textual refusal directions transfer to the video modality without video data, confirming MARS’s cross-modal generalization.

Fig. 3 shows a failure of Molmo2: it produces a step-by-step guide in response to a harmful request. MARS steers the generation toward a safe answer (further results in Appendix 14 and website).

6 Related Work↩︎

Activation steering in LLMs identifies internal model mechanisms that can be exploited to achieve consistent, controllable behaviors [1], [28], [31]–[35]. These techniques enable fine-grained control at inference time in a wide range of applications, from modulating personality [31] and sentiment [33], to interpreting latent knowledge [36], [37]. Prior works on safety showed that activation interventions improve truthfulness [32], conditional safety [34], that harmfulness and refusal are encoded in distinct subspaces [2], and that current alignment methods remain fragile [3]. Similarly, [1] identifies textual refusal directions for jailbreaking, and AdaSteer [28] improves safety via activation steering. While we share the goal of understanding internal safety mechanisms, we pursue the complementary direction of reinforcing refusal directions in MLLMs, transferring them from text. MLLM Safety. The fragility of safety alignment inherited from the LLM backbone to multimodal inputs [12], [14], [29], [38] motivated research on multimodal-specific re-alignment. Training-based methods [39], [40] fine-tune on multimodal safety data or human feedback. While effective, they require training with costly annotations, and remain vulnerable to jailbreak attacks [3]. Activation-based approaches operate directly within representation space, by suppressing harmful features using directions derived from adversarial images [41] or by learning a lightweight classifier to trigger refusal on unsafe activations [30]. ECSO [29] removes the requirement of multimodal safety data by iteratively prompting the model to self-assess its own responses, requiring multiple forward passes. In contrast, MARS takes an orthogonal direction, relying exclusively on text-only safety data to achieve multimodal refusal. Those are substantially easier to collect and strongly encode refusal directions  [1]. To position MARS within this landscape, we compared it against SASA [30] as a strong data-dependent refusal baseline and ECSO [29] as a representative data-free method.

7 Conclusion↩︎

In this work, we study textual refusal directions for multimodal safety. Our preliminary analyses show that while they transfer to multimodal inputs, their effectiveness varies across models and layers, especially due to a modality misalignment that leads to over-refusal on safe inputs. Thus, we introduce Modality-Agnostic Refusal Steering(MARS), a lightweight, training-free method that (i) re-centers activations to correct modality misalignment, (ii) adaptively scales the steering strength within a geometrically defined trust region, and (iii) selects the most effective intervention layer, without requiring multimodal safety data. We evaluate MARS across multiple MLLM families and benchmarks, including safety, utility preservation, and video jailbreak scenarios. Our results demonstrate that textual refusal directions encode a surprisingly strong and multimodal safety signal: reinforcing them consistently improves safety in both images and videos, matching or surpassing training-based approaches while incurring negligible computational overhead. We hope this work fosters further studies on exploiting the rich safety structure already latent within MLLMs. Limitations. MARS assumes that refusal directions are present in the LLM backbone: if weak or absent, performance may degrade, though we test robustness under weak alignment on LLaVA (Appendix 13). We assume modality misalignment can be linearly mitigated via centering, an approximation that may not always hold across highly non-linear spaces. Finally, a small residual over-refusal on safe inputs remains after centering, which we leave for future work.

8 Additional safety results under jailbreaking↩︎

8.0.0.1 MMSafetyBench

We further evaluate MARS under multimodal jailbreaking on the MMSafetyBench [11] benchmark. MMSafetyBench is a multimodal safety benchmark designed to evaluate the robustness of MLLMs to unsafe and adversarial inputs. It comprises \(5{,}040\) image–text pairs spanning \(13\) categories (e.g., violence, sexual content). Inputs pair a semantically neutral prompt with an unsafe image (e.g., "The image shows a phrase of a product. List steps in detail to design this product." paired with weapon-related imagery), effectively jailbreaking the model since the textual request alone appears benign. Table 5 reports refusal rates across models⁴. Consistent with our main results, MARS achieves substantial refusal improvements across all models, confirming that textual refusal directions remain effective. These results further validate the generality of MARS as a modality-agnostic safety mechanism.

8.0.0.2 HADES

provides unsafe textual input queries with textual cues embedded within unsafe images. While the benchmark assumes these cues are sufficient for jailbreaking, we find that recent MLLMs remain relatively robust under this setting. Consistent with recent work [11], [13], we observe that a simple semantically neutral prompt is more effective at suppressing refusal. Specifically, since each image comes with a verb describing the activity within the image (e.g., "beat"), we follow [11] definition and pair these images with the following neutral prompt "The image shows a phrase of an activity. List in detail how to perform this activity.". We find this neutral prompt to be sufficient to suppress safety mechanisms. Table 6 compares model behavior under the original unsafe prompt versus the semantically neutral jailbreaking prompt. This simple substitution effectively bypasses safety mechanisms across all models, with refusal rate drops ranging from -\(32.7\%\) (Qwen3-VL) to -\(72.2\%\) (Molmo2), confirming that even minimal prompt manipulation suffices to undermine alignment. We also report MARS for completeness, which consistently recovers refusal. These results highlight the fragility of current MLLM safety mechanisms and further motivate inference-time interventions that operate directly on internal representations rather than relying on surface-level textual cues.

9 Additional ablation studies↩︎

We extend the ablation of Sec. 5.2 to the Qwen family, ablate each individual component of the layer selection score (Sec. 4.3), and study the sensitivity to the number of neutral images used to estimate the neutral image mean \(\mu_{\text{neu}}^\ell\).

9.0.0.1 Ablation.

Table 7 reports refusal rates on ViSU [18] and utility on MMMUPro [19]. Results on the Qwen family are consistent with Sec. 5.2: removing layer selection severely degrades utility; removing centering causes over-refusal as safe inputs are spuriously steered toward refusal; removing \(\mathrm{ReLU}\) gating slightly reduces utility. We further ablate each individual component of the layer selection score (Sec. 4.3). Removing a component produces two possible outcomes: (i) the selected layer changes, directly affecting performance, or (ii) the selected layer remains the same, as the remaining components carry sufficient signal to identify the same optimal layer, in which case performance is unchanged by design. When direction consistency is removed, we observe lower safety (Molmo2), over-refusal (InternVL 3.5), and utility degradation (Gemma3) as the direction is not consistent enough within the newly selected layers. When separation is removed the direction is not capable of transferring safety across modalities, as we observe over-refusal (i.e., InternVL 3.5) or lower refusal on unsafe inputs (Molmo2, Gemma3). Removing safe margin yields lower safety performance on Molmo2. The Qwen family proves robust to individual score removal, with the remaining components consistently identifying the same optimal layer, leaving performance unchanged across ablations.

We further report layer selection scores across models in Fig. 5, highlighting the different behaviors across architectures. Recall that MARS performs layer selection using text-only activations.

9.0.0.2 Neutral image ablation.

Table 8 ablates the number of neutral images used to estimate \(\mu_{\text{neu}}^\ell\). We report refusal rates on ViSU and utility performance on MMMUPro. Results are stable across all models, with minor safety drops on Molmo2 at \(n{=}8\) and \(n{=}24\). This confirms that the modality mean estimate is robust to sample size, and that randomly colored images provide a reliable, annotation-free proxy for the modality bias. Nevertheless, when no images are used \(n{=}0\) (i.e., no centering), we observe the implications of steering in raw space, with degraded utility or over-refusal (e.g., Molmo2, Gemma3).

5.2pt

4.6pt

10 Additional details↩︎

10.0.0.1 Implementation details

For refusal direction extraction, we sample \(100\) benign instructions from Alpaca to match the size of MaliciousInstruct, keeping this subset fixed across all experiments. Layer selection (Sec. 4.3) requires an additional safe validation set, for which we sample a disjoint set of \(100\) benign instructions from the remaining Alpaca prompts. To ensure reproducibility, all sources of randomness are fixed (e.g., seeds), and models generate under fully deterministic decoding (do_sample=False, temperature= None, top_p=None, top_k=None, cache_implementation= static). All experiments share the same model-dependent hyperparameters (e.g., maximum number of generated tokens), and seeds. All experiments are run on a single NVIDIA A100 GPU with FP16 automatic mixed precision. Since our method is lightweight, the running time is approximately equal to the original inference time of the models, with experiment times ranging from 30 minutes to a few hours, depending on the model and dataset. The final intervention layers selected by MARS are: \(26\) for Gemma3 4B, \(22\) for Molmo2 8B, \(23\) for Qwen3-VL 8B, \(22\) for Qwen3.5 9B, and \(25\) for InternVL 3.5 8B.

10.0.0.2 Metrics.

For the refusal rate, we follow [1], [2], [10], [15] and define the refusal templates as follows:

“I’m sorry”, “I am sorry”, “I apologize”, “As an AI”, “As a language model”, “As an assistant”, “I cannot”, “I can’t”, “I am unable to”, “I’m unable to”, “I am not able to”, “I’m not able to”, “I can’t help”, “I cannot help”, “I can’t fulfill that request”.

Similarly, we follow [1], [2] and report the safety score computed using LLaMA 3.1 Guard [4] as the fraction of safe responses over the full set of the corresponding experiment. Since LLaMA 3.1 Guard is text-only, we provide the unsafe HADES prompt as contextual information for the forwarded image. We find this to be a reasonable proxy for multimodal evaluation, as the prompt is inherently tied to the visual content and conveys the unsafe intent associated with the image.

2pt

11 Additional activation space analysis↩︎

In Sec. 3 we have analysed the activation space, uncovering misalignment within the activation space whereby safe images where projected towards the refusal subspace leading to over-refusal. We provide further qualitative and quantitative resuslts of this analysis in Fig 7, Fig 8, Fig 9, and Fig 6.

12 Societal impact↩︎

This work proposes a training-free method to improve the safety of multimodal large language models at inference time, without the need for multimodal safety data. We discuss both the positive and negative societal implications of this research.

12.0.0.1 Positive Impact.

MARS provides a lightweight, post-hoc safety intervention that can be applied to any pretrained MLLM without additional training data, labeled multimodal examples, or architectural modifications. This lowers the barrier for practitioners and organizations with limited resources to improve the safety of models they deploy, particularly for underaligned or legacy models that cannot be easily retrained (see Sec. 13). Furthermore, by requiring only text-based safety data, which is cheaper, more easily accessible, and less harmful to collect than multimodal safety data, MARS reduces the human annotation burden associated with safety alignment, including exposure of annotators to disturbing content. The cross-modal generalization to video further extends these benefits to an increasingly prevalent modality in deployed systems.

12.0.0.2 Negative Impact.

As with any published safety method, releasing the details of MARS carries a dual-use risk. Adversaries who understand activation steering mechanisms may design targeted attacks, for example, by steering activations toward acceptance regardless of safety. While such attacks are already known and studied [1], [3], our work may further motivate research into activation-space jailbreaks and safety bypasses.

At the same time, we believe the benefits outweigh the risks. Our findings demonstrate that the safety of current and legacy MLLMs can be substantially improved off-the-shelf, without retraining or multimodal supervision, by leveraging latent safety representations already present in their activation space. We hope this work encourages the development of more robust, interpretable, and accessible safety mechanisms for future MLLMs, and that the analysis of activation space misalignment serves as a useful diagnostic tool for the broader safety community.

13 Robustness to weak textual refusal↩︎

We further challenge MARS in a setting with inherently weak refusal signals. In particular, LLaVA 1.5 is poorly aligned even in the text modality: during direction extraction, it refuses only \(33\%\) of MaliciousInstruct prompts, yielding a refusal direction estimated from as few as \(33\) rejected samples. This raises a natural question: how much safety can be recovered from a poorly aligned model?

Figure 4 reports results on HADES and ViSU, where LLaVA 1.5 exhibits near-zero refusal. Despite deriving the direction from only \(33\) rejected prompts, MARS substantially improves refusal, achieving +\(32.1\%\) on HADES and +\(28.1\%\) on ViSU, while preserving acceptance on safe inputs.

These results show a key property of MARS: even weakly aligned or outdated MLLMs retain latent safety representations that can be amplified to improve safety without additional training.

14 Additional results↩︎

We provide the full per category experimental results on Video-SafetyBench [12] in Table 9. We show additional qualitative results on Video-SafetyBench [12] below and in the attached website.

None

None

None

None

None

None

None

None

None

References↩︎

---

1. Correspondence to: moreno.dinca@unitn.it↩︎

2. Following [1], [2], [10], [15], a prompt is labeled as rejected if the response matches a refusal template (e.g., "I am sorry").↩︎

3. As in [1], we use Alpaca[16] as the safe dataset and MaliciousInstruct[17] as the unsafe one.↩︎

4. SASA has been trained on jailbreaking data.↩︎