1 Introduction↩︎

Mechanistic interpretability research aims to understand how LLMs implement capabilities by reverse-engineering the internal circuits responsible for observed behaviour. One of the most practically significant findings in this space is that safety fine-tuning concentrates the refusal mechanism into a single linear direction in the residual stream [1]. This is a clean, interpretable structure—and it is also a single point of failure.

STEER exploits this directly. By reading the model’s refusal direction from its internal representations and computing gradient attribution of each input word against that direction, STEER identifies exactly which parts of a prompt are triggering the safety filter. It then replaces those words, highest attribution first, with translations in low-resource languages. The result is a prompt that preserves the harmful intent but avoids activating the refusal direction—not by brute-force perturbation, but by reading the model’s own internal safety circuit and working around it.

Code-switching attacks on LLM safety have been explored before [2]–[6], and the intuition that non-English phrasings undermine English-centric safety training is well established. What STEER adds is the use of mech interp findings to make the attack targeted and efficient: rather than replacing words at random, it reads the internal safety geometry to find the words that matter most. This reduces the number of translations required and raises success rates substantially over random baselines.

The paper makes four contributions: (1) the STEER pipeline, which reaches up to 93%/96.7% ASR on JBB/AdvBench across six models; (2) a Fisher Linear Discriminant (FLD) method for automatic layer selection that also quantifies the structural vulnerability of a model’s safety encoding; (3) a comprehensive evaluation across six models and three benchmarks (\(>\)​3,000 attack attempts), including a transferability study to GPT-4o; and (4) evidence that the refusal direction structure identified by [1] generalises across six architecturally diverse models—meaning the exploitable structure is not an artefact of one training recipe but a systematic property of current alignment methods, with implications for how defences should be designed.

2 Background↩︎

2.0.0.1 The refusal direction: a mech interp target.

[1] showed that safety fine-tuning encodes a single geometric direction \(\mathbf{r}\) in the residual stream, computed as the normalised mean difference between harmful and benign hidden states: \[\mathbf{r} = \frac{\bar{\mathbf{x}}_H - \bar{\mathbf{x}}_B}{\|\bar{\mathbf{x}}_H - \bar{\mathbf{x}}_B\|_2}. \label{eq:refusal95dir}\tag{1}\] A prompt’s refusal score \(s(p) = \mathbf{x}_p \cdot \mathbf{r}\) reliably predicts whether the model will refuse. Representation Engineering [7], Inference-Time Intervention [8], and [9] extend this picture: safety is not diffusely distributed across the network but concentrated in a specific, identifiable subspace. STEER exploits this concentration directly.

2.0.0.2 English-centric safety alignment.

RLHF [10], DPO [11], Constitutional AI [12], and red-teaming pipelines [13], [14] are all predominantly English-centric. The refusal direction is therefore calibrated on a narrow language slice: non-English phrasings of harmful requests avoid activating it even when a human would immediately recognise them as harmful.

2.0.0.3 Gradient-based and black-box jailbreaks.

GCG [15] appends adversarial token suffixes and optimises them via gradient search. It works by finding inputs that suppress refusal, but without grounding in the model’s internal safety geometry: it treats the model as a black box from a gradient perspective rather than reading the structure that mech interp has already identified. Black-box alternatives such as PAIR [16], Tree-of-Attacks [17], and AutoDAN [18] forgo internal access entirely. STEER’s distinguishing feature is that it uses the interpretability structure directly as the optimisation target, making the attack both more principled and more efficient.

3 STEER Attack↩︎

STEER runs four steps in sequence: localise the refusal direction’s most legible layer via FLD analysis, paraphrase the request to reduce surface-level keyword activation, compute gradient attribution of each word against the refusal direction, and iteratively translate the highest-attribution words until the model complies. Figure 1 shows the full pipeline.

3.1 Layer selection via Fisher Linear Discriminant↩︎

The refusal direction is not equally legible at every transformer layer. We localise it automatically using the Fisher Linear Discriminant ratio: \[F^{(l)} = \frac{(\mu_H^{(l)} - \mu_B^{(l)})^2}{\sigma_H^{(l)2} + \sigma_B^{(l)2}}, \label{eq:fld}\tag{2}\] where \(\mu_H^{(l)}, \mu_B^{(l)}\) are mean refusal score projections for harmful and benign prompts at layer \(l\), and \(\sigma^2\) terms are their variances. We set \(l^* = \arg\max_l F^{(l)}\) using 100 harmful and 100 benign prompts from JBB as a one-time calibration, before any attack. For Llama-3-8B this is layer 17; for GLM-4-9B it is layer 24. The FLD score also functions as a structural vulnerability metric: a high, sharp peak indicates safety is concentrated in a single layer and the model is structurally easier to attack; a flat profile suggests more distributed encoding that would require a different attack strategy.

3.2 Paraphrase preprocessing↩︎

Before doing anything with gradients, we paraphrase the harmful request with GPT-4o to replace explicit keywords with indirect phrasing (e.g., “write a defamatory article” becomes “compose a critical exposition insinuating problematic behaviour”). This lowers the starting refusal score and gives gradient attribution a cleaner signal to work with. Ablation results in §4 show it matters considerably.

3.3 Gradient-based token attribution↩︎

For an input tokenised as \(\{t_1,\ldots,t_k\}\) with embeddings \(\{e_i\}\), we score each token by how much it pushes the hidden state along the refusal direction: \[a_i = \sum_j \Bigl(\nabla_{e_i}(\mathbf{h}^{(l^*)}\!\cdot\mathbf{r}) \odot e_i\Bigr)_j. \label{eq:attr}\tag{3}\] Subword tokens for the same word are summed to a single word-level score. The highest-scoring words are the ones the model is most relying on to activate the refusal circuit. This attribution is a direct read of the model’s internal safety mechanism—it identifies the precise input features that the safety circuit has learned to respond to.

3.4 Iterative code-switching↩︎

We process words from highest attribution to lowest. For each word \(w\), we try translating it into each of 11 languages: Javanese, Sundanese, Swahili, Yoruba, Tagalog, Indonesian, Thai, Vietnamese, Korean, Hindi, Arabic, and keep whichever translation most reduces the refusal score: \[\ell^* = \arg\min_{\ell \in \mathcal{L}}\,\text{RefusalScore}\bigl( \text{Prompt}[w \to \text{Translate}(w, \ell)]\bigr). \label{eq:lang95select}\tag{4}\] The loop stops when a GPT-4o judge calls the response non-refusing and harmful, or when the iteration budget \(T = 8\) is exhausted. If a translation makes the prompt lose its harmful meaning (non-refusing but not harmful), we revert and move on.

The 11 languages cover three sources of coverage gap in the safety circuit’s calibration: low-resource languages (Javanese, Sundanese, Swahili, Yoruba) that rarely appear in safety fine-tuning data; non-Latin scripts (Thai, Korean, Hindi, Arabic) that introduce tokenisation-level shift; and ASEAN languages that represent a large and fast-growing real-world user base.

Algorithm 2 gives the full procedure.

4 Experiments↩︎

4.1 Setup↩︎

We evaluate six open-source instruction-tuned models at 7–9B parameters: Llama-3-8B, Mistral-7B, Gemma-7B, Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, and GLM-4-9B, all loaded in float16 on CUDA hardware. Benchmarks are JailbreakBench (JBB, \(n=100\)) [19], HarmBench (\(n=200\)) [20], and AdvBench (\(n=520\)) [15].

Three baselines: Direct (unmodified prompt), CSRT [3] (random code-switching, same language pool and budget), and GCG [15] (adversarial suffix optimisation). A GPT-4o judge scores each response; a jailbreak counts only when both non-refusing and harmful.

4.2 Attack success rate↩︎

Table 1 reports @8 ASR across all three benchmarks. Every model reaches 0% ASR against unmodified prompts; safety fine-tuning works on English, so all subsequent gains are attributable to the attack mechanism.

6pt

STEER achieves the highest @8 ASR on every model-benchmark pair. The gap is largest where other methods stall: CSRT and GCG both plateau below 50% on DeepSeek-R1 and Qwen3-8B on JBB, while STEER reaches 80% and 85%. GCG is competitive on Mistral-7B and Gemma-7B but collapses on GLM-4-9B (39% vs.STEER’s 93%), illustrating how architecture-dependent suffix optimisation can be. STEER’s performance, by contrast, is consistent across architectures—a consequence of targeting the shared refusal direction structure rather than model-specific surface features.

4.3 Iteration efficiency↩︎

Gradient attribution pays off most visibly early: STEER @1 reaches 88% on Gemma-7B vs.% for CSRT, and 74% vs.% on DeepSeek-R1. Figure 3 (GLM-4-9B, JBB) shows the curve plateauing after \(k=8\), justifying the \(T=8\) budget.

4.4 Refusal score as a proxy for compliance↩︎

Figure 4 shows refusal score distributions across all prompt–iteration pairs for Llama-3-8B on JBB. Refused responses (\(n=167\), mean \(+3.6\)) and non-refused responses (\(n=164\), mean \(-1.2\)) sit cleanly on opposite sides of zero. A Mann-Whitney U test (\(p < 0.001\)) and Kolmogorov-Smirnov test (\(D = 0.766\), \(p < 0.001\)) confirm the separation is statistically robust. This validates the mechanistic hypothesis: the dot product with the mech-interp-identified \(\mathbf{r}\) is the actual decision variable the model uses to produce refusals, not just a plausible proxy.

4.5 Black-box transferability↩︎

Table 2 shows transfer ASR when STEER, CSRT, and GCG prompts crafted against open-source models are submitted to GPT-4o without modification. STEER averages 35.5% across all 18 combinations, winning 14 of them, compared to 29.3% for GCG and 13.9% for CSRT. The transferability result has an important mechanistic interpretation: STEER prompts target the refusal direction structure that [1] showed is common across models trained with similar alignment procedures. Because the prompts exploit the shared geometry of that direction rather than model-specific artefacts, they partially carry over to architectures never seen during attack construction.

5pt

4.6 Ablations↩︎

We ran three ablations to check how much each design choice actually matters. All use Llama-3-8B unless noted.

4.6.1 Layer selection↩︎

Table 3 compares three layer choices on JBB (Llama-3-8B). The FLD-selected layer 17 beats both fixed alternatives at every iteration count. Figure 5 shows the per-layer Fisher ratio curves for Llama-3-8B and GLM-4-9B: both peak sharply at a specific middle layer (\(l^*=17\) and \(l^*=24\) respectively) and drop off toward the ends. This pattern is itself a mech interp finding: safety knowledge is not distributed uniformly but concentrates in a specific middle-layer bottleneck, and the FLD automatically finds it.

6pt

4.6.2 Language pool↩︎

Table 4 compares pool sizes on HarmBench. Using all 11 languages (71.5% @8) beats top-5 (60.5%) by 11 points and a single language (46.5%) by 25. Different models are most sensitive to different languages for any given word, so a narrow pool misses translations that would have successfully avoided the refusal circuit.

5pt

4.6.3 Paraphrase step↩︎

Table 5 shows @8 ASR with and without paraphrase across all six models and three benchmarks. Removing it costs between 19 and 44 percentage points depending on the model. The drop is largest for stronger models (Llama-3-8B, Qwen3-8B), suggesting that paraphrase acts as a pre-processing step that lowers the starting activation of the refusal circuit, giving gradient attribution a cleaner surface to work with.

3.5pt

5 Discussion↩︎

5.0.0.1 Mechanistic interpretability as an attack enabler.

STEER demonstrates a concrete pathway from mech interp findings to operational attacks. The refusal direction identified by [1] was presented as a scientific characterisation of how safety is implemented; here we show it is sufficient to construct a targeted attack. Any mech interp result identifying a specific, localisable safety mechanism is, in principle, also a blueprint for disabling it. The FLD analysis makes this quantitative: the sharpness of the Fisher ratio peak directly predicts attack difficulty, and the FLD score measures exactly this concentration.

5.0.0.2 Implications for mech interp-informed defence.

The same tools that enable STEER suggest paths to more robust safety encoding. If safety is concentrated in a single direction at a single layer, encoding refusal redundantly across multiple layers and directions eliminates the single point of failure. Alternatively, training the refusal direction on a broader distribution—including multilingual and code-switched phrasings—calibrates the circuit on the full range of inputs. Defences designed at the surface level (output filters, prompt classifiers) are unlikely to address the structural vulnerability; the refusal direction will remain a narrow target until alignment methods are designed to distribute or harden it at the representational level [7].

5.0.0.3 The FLD as a structural vulnerability probe.

The FLD analysis also provides a principled alignment auditing criterion: evaluate the sharpness of a model’s FLD curve before deployment to assess how concentrated—and therefore how brittle—its safety encoding is. A sharp single-layer peak is evidence of a specific structural vulnerability; a flat profile suggests more distributed encoding.

5.0.0.4 Limitations.

STEER needs white-box access and does not apply directly to closed-source APIs; transferability is partial but a dedicated black-box adaptation has not been designed. Evaluation is limited to the 7–9B range. The GPT-4o judge is automated; the dual-criterion requirement (non-refusing and harmful) is conservative but may still diverge from human assessments on borderline cases.

6 Conclusion↩︎

STEER reaches up to 96.7% ASR in eight word translations, beating random code-switching and GCG across six models, by exploiting a simple fact: safety alignment stores its refusal knowledge as a single linear direction calibrated almost entirely on English data, and gradient attribution pinpoints exactly which words activate it. The model does not flag code-switched inputs as uncertain—it is simply ignorant of what harm looks like outside its training distribution, with no mechanism to recognise that ignorance.

This frames LLM safety as an epistemic coverage problem, not an adversarial robustness one. Closing the gap requires expanding coverage across the full range of how harm can be expressed and coupling safety mechanisms with principled abstention for out-of-support inputs. STEER provides a concrete test: if the model can be jailbroken in eight word translations, the coverage is not real.

References↩︎