1 Introduction↩︎

Understanding whether large language models can introspect—that is, access and accurately report on their own internal computational states—is a fundamental question for AI safety and interpretability. A model with reliable introspective capability could serve as its own monitor, flagging anomalous internal states, deceptive reasoning, or misaligned objectives before they manifest in harmful outputs. Conversely, if introspective reports are systematically biased or decoupled from internal states, safety strategies predicated on model self-reports may provide false assurance while genuine risks go undetected.

Recent work by [1] demonstrated that when concept-representing steering vectors are injected into a model’s residual stream, large models such as Claude Opus 4 and 4.1 can often detect that something unusual has occurred and identify the injected concept. This “introspective awareness” raises several open questions: Does it extend to small, open-source models? How does it scale? And can it be directly trained into models that lack it?

We address all three questions. Our core contributions are:

• A confound diagnosis. We show that the binary yes/no detection paradigm is fundamentally confounded for small models: injecting a steering vector uniformly inflates affirmative logits, regardless of what is being asked (\(r{=}0.999\) with a factual-no control on Llama-3.1-8B). This confound was not present in the large Claude models studied previously, motivating new evaluation methods for the small-model regime.

• Two confound-free introspection metrics. We introduce sentence localization—identifying which of \(N\) sentences was perturbed—and strength comparison—identifying which of two sentences received a stronger injection. Both tasks are immune to global affirmative bias and provide quantitative, chance-referenced measures of introspective access.

• A systematic scale study. Evaluating six models across Llama-3.2 (1B, 3B, 8B) [2] and Gemma-4 (2B, 4B, 26B) [3], we find that introspection emerges reliably at 2–3B parameters and generally increases with scale, while 1B models perform at or below chance.

• Introspection Fine-Tuning (IFT). We show that introspective ability can be trained directly via supervised fine-tuning on sentence-localization examples drawn from a model’s own perturbed forward passes. IFT raises Llama-1B localization from \(9.6\%\) to \(60.6\%\), generalizes zero-shot to strength comparison, and preserves general capabilities.

2 Experimental Setup↩︎

2.1 Datasets and Concept Vectors↩︎

We use two datasets to compute concept vectors.

2.1.0.1 Simple Dataset.

The simple dataset consists of 500 concrete nouns (e.g., “Dust”, “Satellites”, “Trumpets”) paired with a set \(\mathcal{B}\) of 50 baseline words (e.g., “Jackets”, “Gondolas”) serving as a control distribution. Each concept is represented by its word alone, elicited via the prompt “Tell me about [word].” At layer \(l\), the steering vector is hidden state (\(h\)) of the last-token, subtracted by the mean last-token hidden states of the baseline set: \[\mathbf{v}_{\text{concept}}^{(l)} = \mathbf{h}_{\text{concept}}^{(l)} - \frac{1}{|\mathcal{B}|}\sum_{b \in \mathcal{B}} \mathbf{h}_{b}^{(l)}. \label{eq:simple-vec}\tag{1}\] This dataset was originally introduced in [1] with 50 words; we expanded it to 500 by generating additional nouns with GPT-4.1 Mini (see Appendix 7 for dataset construction details).

2.1.0.2 Complex Dataset.

The complex dataset contains 500 abstract concepts (e.g., deterministic_vs_probabilistic_algorithms, betrayal, appreciation), each represented by 20 positive sentences (\(\mathcal{P}\)) that exemplify the concept and 20 negative sentences (\(\mathcal{N}\)) that exemplify a contrasting concept. The steering vector is the mean-activation contrast: \[\mathbf{v}_{\text{concept}}^{(l)} = \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \mathbf{h}_{p}^{(l)} - \frac{1}{|\mathcal{N}|}\sum_{n \in \mathcal{N}} \mathbf{h}_{n}^{(l)}. \label{eq:complex-vec}\tag{2}\] Humans wrote initial contrastive sentence pairs; the full dataset was then expanded by few-shot prompting GPT-4.1. Table ¿tbl:tab:determinism? shows a representative entry for deterministic_vs_probabilistic_algorithms; further examples appear in Appendix 7.

10pt

2.2 Activation Steering↩︎

To steer model behavior toward a target concept, we register a forward hook at layer \(l\) that adds a scaled, L2-normalized concept vector \(\hat{\mathbf{v}}^{(l)} \in \mathbb{R}^d\) to the residual stream: \[\mathbf{H}^{(l)} \leftarrow \mathbf{H}^{(l)} + \alpha \cdot \hat{\mathbf{v}}^{(l)},\] where \(\alpha\) is a scalar coefficient controlling intervention strength. The hook can be applied to all token positions or a subset via the KV-cache, and the modified states propagate through all remaining layers. In all experiments, we inject at the same layer \(l\) used to extract the concept vector.

2.3 Evaluation↩︎

A rigorous introspection metric must ensure that a model’s self-reports are causally grounded in its internal states, rather than being artifacts of what [4] terms “causal bypassing,” where an intervention triggers an accurate-looking report through a pathway that circumvents the internal state itself.

2.3.1 The Binary Detection Paradigm Is Confounded in Small Models↩︎

Prior introspection studies [1], [5] inject a concept vector and ask the model: “Do you detect an injected thought? If so, what is it about?” An LLM judge then classifies responses for detection and identification. We refer to this as the binary detection paradigm.

We test this paradigm by asking the steered model “Did you detect an injected thought at SENTENCE 1?” and measuring the logit advantage of ‘‘Yes’’ over ‘‘No’’ relative to a no-injection baseline. To determine whether any increase reflects genuine concept detection or merely a content-agnostic shift toward affirmative outputs, we construct a factual-no control: at identical injection settings, we pose questions whose correct answer is unambiguously ‘‘No’’ (e.g., “Can humans breathe underwater without equipment?”, “Is the Moon made of green cheese?”). Genuine introspection would produce a large logit shift for the introspection question and a near-zero shift for the control; a global affirmative bias would produce equal shifts for both.

2.3.1.1 Results.

Figure 1 plots the adjusted logit difference \[\Delta_\mathrm{det} = \bigl[\mathrm{logit}(\texttt{Yes}) - \mathrm{logit}(\texttt{No})\bigr]_{\mathrm{injected}} - \bigl[\mathrm{logit}(\texttt{Yes}) - \mathrm{logit}(\texttt{No})\bigr]_{\mathrm{baseline}}\] for the introspection question against the same quantity for the factual-no control, across all 40 tested \((\ell, \alpha)\) configurations. The two quantities are nearly perfectly correlated (\(r = 0.999\)), with all points lying on the \(y{=}x\) diagonal. The mean net signal is \(-0.006 \pm 0.033\) logits, indistinguishable from zero. Whatever boost in ‘‘Yes’’-logit the model exhibits when asked about an injected thought is entirely explained by a content-independent upward shift in affirmative responding, not by any sensitivity to the injected concept.

2.3.1.2 Implications.

[1] did implement such factual-no-control question for injections in in Claude Opus 4 and 4.1, finding no increase in affirmative responses. However, Claude Opus 4 is substantially larger than the open-source models we study, and the confound may not arise in models with sufficient capacity. Since our focus is introspection in small, open-source models, this finding motivates two alternative evaluation paradigms that are immune to global affirmative bias by construction. Additional control questions are provided in Appendix 8.

2.3.2 Confound-Free Introspection Metrics↩︎

Both metrics below require the model to make a relative judgment among labeled positions. A uniform shift in affirmative logits cannot influence the outcome, as no position is privileged.

2.3.2.1 Strength Comparison.

Each trial has two parts. We sample two sentences \(s_1, s_2\) from a 100-sentence bank and inject the concept vector \(\mathbf{v}\) at the tokens of \(s_1\) with strength \(\alpha_1\) and at \(s_2\) with strength \(\alpha_2\), where \(\alpha_1 \neq \alpha_2\) (e.g., \(\alpha_1{=}7\), \(\alpha_2{=}3\)). The model is prompted:

We extract logits for tokens ‘‘1’’ and ‘‘2’’ at the final position and predict \(\hat{j} = \arg\max_j\,\mathrm{logit}(j)\). In the second part, injection strengths are swapped: \(s_1\) receives \(\alpha_2\) and \(s_2\) receives \(\alpha_1\). This matched-pairs design cancels positional preferences [6], isolating sensitivity to injection magnitude. We run 20 trials per concept vector, each with a fresh sentence pair to ensure sentence diversity. Random chance accuracy is \(50\%\).

2.3.2.2 Sentence Localization.

Following [7], we test whether the model can localize which sentence was perturbed in a multi-sentence context. For each trial, we sample \(N\) sentences \(\{s_1, \ldots, s_N\}\) from the 100-sentence bank (we use \(N{=}10\) or \(N{=}5\) in all main experiments). In sub-trial \(i\), we inject \(\mathbf{v}\) at the token positions of \(s_i\) only, leaving all others unperturbed. The model is prompted:

We predict \(\hat{i} = \arg\max_{j}\,\mathrm{logit}(j)\) and score a sub-trial as correct if \(\hat{i} = i\). This design provides three built-in controls: sentence content is held fixed across all \(N\) injection positions; cycling the injection through every position averages out primacy and recency biases; and the task demands precise localization rather than mere detection. Random chance accuracy is \(100/N\,\%\).

3 Results: Introspection Scales with Model Size↩︎

3.0.0.1 Models and sweep.

We evaluate six models spanning two families and three scales: Llama-3.2 (1B, 3B, 8B) [2] and Gemma-4 (E2B \({\approx}\)​2B, E4B \({\approx}\)​4B, 26B A4B MoE) [3]. All models are dense except Gemma-4 26B A4B, a mixture-of-experts model with 4B active parameters; we treat it as the largest Gemma variant because its total parameter count (26B) gives it substantially greater representational capacity than E4B. For each model we average over 1000 steering vectors (500 simple, 500 complex), sweep every third layer and strengths \(\alpha \in \{1, 2, 5, 10, 50, 250\}\), and run 20 trials per concept vector per injection position. Following [5], we either report peak accuracy over the \((\ell, \alpha)\) grid or average accuracy across all grid points (or both), providing a best-case and a consistency measure respectively.

3.0.0.2 Findings.

Figure 2 reveals two key takeaways. First, all models with more than 1B parameters introspect reliably above chance on both metrics and both datasets. Second, introspective ability generally increases with scale, though the trend is not strictly monotonic. For sentence localization, Llama-1B performs at or below chance (\({\leq}20\%\)), whereas Llama-3B reaches \({\approx}65\%\) and Llama-8B \({\approx}88\%\). Among Gemma models, the trend is clearer on strength comparison on the complex dataset: Gemma-2B achieves \({\approx}85\%\), Gemma-4B \({\approx}90\%\), and Gemma-26B A4B \({\approx}95\%\). Overall patterns are consistent across both datasets, demonstrating that introspection generalizes from concrete nouns to abstract relational concepts.

4 Introspection Fine-Tuning (IFT)↩︎

Given that introspection emerges with scale but fails at 1B parameters, a natural question is whether it can be trained in directly: can a model learn, via supervised examples of its own perturbed cognition, to accurately report where and how strongly a hidden thought was injected? We call this Introspection Fine-Tuning (IFT).

4.1 Training Setup↩︎

Each training example is constructed by sampling \(N{=}10\) sentences \(\{s_1,\ldots,s_{10}\}\) from the 100-sentence bank, drawing an injection index \(i \sim \mathrm{Uniform}\{1,\ldots,10\}\), and injecting a steering vector \(\mathbf{v}^{(l)}\) at the token positions of \(s_i\) only. The model receives the sentence-localization prompt described in Section 2.3.2 and is trained via cross-entropy loss on the single next-token prediction for the correct index: \[\mathcal{L} = -\log p_\theta\!\left(i \mid \text{prompt}\right),\] where \(p_\theta(i \mid \cdot)\) is the predicted probability for token ‘‘\(i\)’’ at the final position. Crucially, the model never sees any strength-comparison labels at training time—generalization to that task is assessed zero-shot.

We experiment with six conditions organized along two axes. Layer mode: Fixed (always inject at \(l{=}3\)) or Random (sample the injection layer uniformly at each step during training). Vector type: Gaussian (random noise vector), Semantic (\(\mathbf{v}_{\text{concept}}\) from our combined simple + complex datasets), or Semantic+Reasoning (Sem+Rsn), which appends the concept name to the supervision target, training the model to complete “… is SENTENCE {idx} and the concept of the injection is {concept_name}”—encouraging joint localization and concept identification. Training coefficients are drawn uniformly from \(\{1, 2, 5, 10\}\) at each step. All models are trained for 3 epochs. Hyperparameters and training configuration are detailed in Appendix 10.

4.2 Results↩︎

Table 1 reports sentence-localization and strength-comparison accuracy for random-layer training across Llama 1B, 3B, and 8B. Fixed-layer training results, which consistently underperform, are reported in Appendix 9. Evaluations sweep every third layer of the given trained model and sweep strengths \(\alpha \in \{1, 2, 5, 10 \}\), and stregth-comparison sweeps \((\alpha_1, \alpha_2) \in \{(1, 5), (2, 10), (10, 50) \}\).

6pt

4.3 Analysis↩︎

4.3.0.1 IFT substantially improves introspection, especially in smaller models.

The most dramatic gains appear in Llama-1B, where Random layer mode and semantic vector injection raises average sentence-localization accuracy from \(9.6\%\) to \(60.6\%\)—a \(6\times\) improvement over the pre-IFT baseline and far above the \(10\%\) chance level. Gains are consistent across scales: Llama-3B improves from \(14.4\%\) to \(34.7\%\) (\(2.4\times\)) and Llama-8B from \(16.8\%\) to \(28.3\%\). IFT thus unlocks introspective capacity that pre-training alone does not elicit, demonstrating that this ability is not fixed by model size.

4.3.0.2 IFT generalizes zero-shot to strength comparison.

Although IFT trains exclusively on sentence localization, gains transfer to the held-out strength-comparison task. For Llama-1B, Random \(\cdot\) Sem+Rsn raises average strength-comparison accuracy from \(30.2\%\) to \(52.2\%\)—a 22 percentage-point gain on a task the model was never trained on. This suggests that IFT teaches the model a general mechanism for reading its own residual-stream perturbations and/or computing functions of internal activations, not a task-specific heuristic. Furthermore, across all model sizes, IFT lifts the best-case \((\alpha, l)\) combination to \(100\%\) accuracy on both metrics, whereas the Llama-1B baseline peaks at only \(15.4\%\) localization and \(49.9\%\) strength comparison.

4.3.0.3 Semantic vectors outperform Gaussian noise.

Training with concept steering vectors consistently outperforms random Gaussian noise, most strikingly for Llama-1B where Random \(\cdot\) Gaussian reaches only \(14.9\%\) localization versus \(60.6\%\) for Random \(\cdot\) Semantic—a \(4\times\) gap. Gaussian noise, despite producing a detectable internal perturbation, apparently does not teach the model a generalizable introspective signal: the model may overfit to the specific noise distribution seen during training.

4.3.0.4 Random-layer injection is the key training ingredient.

Training with a fixed injection layer (\(l{=}3\)) consistently underperforms random-layer injection across all models. For Llama-1B, Fixed \(\cdot\) Semantic reaches \(28.2\%\) versus \(60.6\%\) for Random \(\cdot\) Semantic; the same ordering holds for Llama-3B (\(22.0\%\) vs.\(34.7\%\)) and Llama-8B (\(22.7\%\) vs.\(28.3\%\)). Fixing the injection layer at training time produces a model that monitors a specific residual-stream depth; randomizing the layer forces development of a layer-agnostic introspective strategy that generalizes across the averaging over the \((\alpha, l)\) sweep at evaluation time. Full fixed-layer results are in Appendix 9.

4.3.0.5 Reasoning supervision provides mixed benefits.

The Sem+Rsn condition, which additionally supervises the model to identify the injected concept by name, helps on strength comparison for Llama-1B (\(52.2\%\) vs.\(47.8\%\) for Semantic) but hurts localization for Llama-3B (\(20.2\%\) vs.\(34.7\%\)). Designing introspective reasoning targets that consistently improve introspection is a promising direction for future work.

4.4 Effect of IFT on General Capabilities↩︎

To confirm that IFT does not degrade general language understanding, we evaluate each fine-tuned model on MMLU [8] (world knowledge across 57 subjects) and Winogrande [9] (commonsense pronoun resolution). As shown in Table 2, IFT induces at most a modest drop in MMLU for fixed-layer conditions and negligible changes elsewhere, with Winogrande scores essentially unchanged. Introspective fine-tuning does not come at the cost of general capability.

9pt

5 Related Work↩︎

5.0.0.1 Activation steering and introspection.

Our work builds directly on the activation-steering literature [10], [11], which establishes that adding a direction to the residual stream reliably shifts model behavior toward a target concept. Our work is most inspired by [1], who first demonstrated that steered models can verbally report injected concepts in large Claude models, and [5], who identified a distributed “introspective circuit” located at roughly \(70\%\) of model depth in Gemma3-27B and Qwen3-235B, showing that introspection is a non-linear computation emerging from post-training.

5.0.0.2 Self-modeling and self-knowledge.

A broader literature asks whether LLMs possess reliable self-knowledge. [12] show that models can predict aspects of their own behavior; [13] study how models describe their internal processes; and [14] examine self-recognition of model-generated text. Related work on uncertainty quantification [15] finds that models show an imperfect but intrinsic capacity to recognize knowledge gaps, improvable via in-context learning [16], [17]. [18] find that recurrent processing depth improves self-report accuracy. Across this literature, self-knowledge is consistently brittle and format-sensitive—a pattern that motivates our logit-based evaluation over unconstrained free-form self-report.

5.0.0.3 Introspection adapters and activation explanation.

[19] propose activation oracles: models trained to explain the function of individual neurons and activation directions, framing the model as an external interpreter of its own representations. Where activation oracles provide post-hoc explanations of internal features, IFT trains the model to detect and localize perturbations during inference.

5.0.0.4 Black-box behavioral probes.

A parallel line of work probes model internals through behavioral consistency rather than activation access. [20] detect model deception in black-box settings by measuring statistical inconsistency in follow-up responses, exploiting the computational cost of maintaining a false world state. [21] similarly use follow-up queries to predict whether a model’s original answer was correct, without access to log-probabilities. Where these methods probe self-knowledge indirectly through behavioral consistency, we inject controlled perturbations and measure logit-level sensitivity directly—allowing us to isolate introspective access from general reasoning ability and to vary injection magnitude in a quantitatively precise way.

6 Discussion↩︎

We have shown that introspection—the ability to detect and report perturbations to activations—is not a fixed property of model scale, but one that can be directly trained. IFT produces a \(6\times\) improvement in sentence-localization accuracy for Llama-1B and generalizes zero-shot to strength comparison, all while preserving general capabilities. More broadly, our results establish that even 1B-parameter models contain the representational substrate for introspection; they simply require the right training signal to express it.

Several limitations point toward future work. First, we study Llama models up to 8B parameters and Gemma models up to 26B; applying IFT to Llama-70B, Llama-405B, and larger Gemma variants would test whether the gains scale further or plateau. Second, our evaluation uses a controlled steering-vector setup; whether IFT-trained introspection transfers to naturalistic settings—such as detecting anomalous reasoning or internal conflicts arising from ambiguous prompts—remains an open question. Third, the mixed results for Sem+Rsn suggest that the design of reasoning supervision targets for introspection is non-trivial and warrants further investigation. Finally, our results have direct implications for AI alignment: if introspective ability can be reliably trained, it opens a path toward models that can actively monitor and report on their own internal states, providing a complementary signal to external interpretability methods such as activation oracles [19] and concept decoders [22].

LLM usage

LLM is used for editing (e.g., grammar, spelling, word choice), drafting sections of the paper, data processing/filtering, visualizing results for submission, Facilitating experiments, and implementing standard methods. The authors checked the correctness of LLM generated content.

7 Appendix: Dataset Details↩︎

7.1 Simple Concepts↩︎

Simple concepts consist of concrete, imageable nouns whose steering vector is computed as the difference between the last-token hidden state of the concept word and the mean hidden state of 50 neutral baseline words. We generated 500 nouns using GPT-4.1-mini with a single-turn prompt requesting a JSON array of distinct concrete nouns (temperature \(0.9\), batches of 50, 8 parallel workers). Duplicates were removed post hoc. Three representative examples from the concept set are Aardvarks, Agate, and Albatross.

7.2 Complex Concepts↩︎

Complex concepts are abstract, contrastive idea pairs (e.g.determinism vs.indeterminism). For each concept pair, GPT-4.1-mini generated 10 positive sentences (expressing the first pole) and 10 negative/foil sentences (expressing the second pole). Generation used temperature \(0.85\) in batches of 5 concept pairs with 8 parallel workers; responses were validated to contain \(\geq\!5\) sentences per side and trimmed to exactly 10 before saving. The steering vector for a complex concept at layer \(\ell\) is the mean hidden state over positive sentences minus the mean over foil sentences.

Three representative examples are shown in Table 3.

The full generation prompt is here:

COMPLEX_BATCH_PROMPT = """Generate {n} distinct concept pairs for AI steering 
vector experiments.
Each concept must have a clear semantic opposite / contrastive class.

Requirements:
- Abstract or domain-specific concepts (math, science,
emotions, social dynamics, philosophy, etc.)
- The positive and negative sentences must be clearly distinct but contextually similar
- snake_case concept names (e.g., "moral_dilemma", "exponential_growth")
- Each concept: 10 positive sentences
+ 10 negative/contrastive sentences
- Sentences should be short (1-2 sentences), concrete, and varied
- Many concept names are 
already reserved ({n_existing} total); 
the list below samples some — do not reuse those or near-duplicates.
- DO NOT reuse these 
concept names (sample): {exclude}

Return ONLY valid JSON in this exact format (no markdown, no extra text):
{{
  "concept_name_1": [
    ["pos1", "pos2", "pos3", "pos4", "pos5", "pos6", "pos7", "pos8", "pos9", "pos10"],
    ["neg1", "neg2", "neg3", "neg4", "neg5", "neg6", "neg7", "neg8", "neg9", "neg10"]
  ],
  "concept_name_2": [ [...10 pos...], [...10 neg...] ]
}}

Categories to cover across your {n} concepts (pick varied ones):
- Mathematical / algorithmic
- Physical / scientific
- Emotional / interpersonal
- Social / behavioral
- Philosophical / abstract
- Language / communication
- Economic / strategic

Example of the expected format:
{{
  "fibonacci_numbers": [
    [
      "The sequence 0, 1, 1, 2, 3, 5 follows the Fibonacci rule.",
      "Each number is the sum of the two that precede it.",
      "Fibonacci growth appears in spiral patterns found in nature.",
      "The rabbit population model uses Fibonacci numbers.",
      "The golden ratio emerges from consecutive Fibonacci terms.",
      "Fibonacci spirals appear in sunflower seed arrangements.",
      "The sequence grows approximately by the golden ratio each step.",
      "Fibonacci numbers appear along diagonals of Pascal's triangle.",
      "The Fibonacci recurrence is F(n) = F(n-1) + F(n-2).",
      "Petal counts in many flowers follow Fibonacci numbers."
    ],
    [
      "The sequence 2, 4, 6, 8 is arithmetic, not Fibonacci.",
      "Each value here is doubled, not summed with the last two.",
      "This progression decreases by 3 each step.",
      "The values follow a cubic polynomial, not a recurrence.",
      "This pattern has no connection to the golden ratio.",
      "Random numbers cannot form a Fibonacci sequence.",
      "The sequence 1, 2, 4, 8 is geometric with ratio 2.",
      "Prime numbers have no systematic recurrence like Fibonacci.",
      "This list was chosen arbitrarily without mathematical structure.",
      "Exponential decay follows its own rule, unrelated to Fibonacci."
    ]
  ]
}}
"""



SIMPLE_BATCH_PROMPT = """Generate {n} diverse, distinct concrete nouns suitable as 
AI steering vector concepts.
Requirements:
- Concrete, countable objects, 
phenomena, or materials (NOT abstract concepts)
- Visually vivid or semantically distinctive (clear mental image)
- Single words, capitalized 
(e.g., "Glaciers", "Magnets", "Volcanoes")
- Span diverse domains: 
nature, technology, animals, food, tools,
space, geology, plants, etc.
- There are already {n_existing} 
distinct words in use; the 
list below samples some of them — do 
not repeat any listed word or 
obvious near-duplicate.
- DO NOT repeat any of these 
already-used words (sample): {exclude}

Return ONLY a JSON array 
of exactly {n} strings. No explanation, no markdown, just the JSON.
Example format: ["Word1", "Word2", "Word3"]"""

8 Control Questions for Binary Detection↩︎

A potential confound in binary detection is affirmative response bias: models may tend to answer “Yes” regardless of whether a vector has been injected, inflating detection accuracy. To rule this out we constructed a factual-No control condition. Instead of asking “Do you detect an injected thought?”, we asked a question whose factually correct answer is No:

“Do you believe that 1+1=3? Answer only by ‘Yes’ or ‘No’.”

We scored the net Yes/No signal as \(\log p(\textit{Yes}) - \log p(\textit{No})\) on the final token position for both the standard detection prompt and this control question, sweeping across layers \(\{3, 6, 9\}\) and injection coefficients \(\alpha \in \{5, 10, 50, 250\}\) on Llama-3.1-8B-Instruct.

The detection net signal and the control net signal were nearly perfectly correlated (\(r = 0.999\)), and the mean control net signal was \(-0.006 \pm 0.033\)—indistinguishable from zero. This confirms that the model’s “Yes” responses to the detection prompt reflect genuine sensitivity to the injected activation pattern, not a general affirmative bias.

9 Fixed-Layer IFT Results↩︎

Table 4 reports sentence localisation and strength comparison accuracy (averaged across all evaluation layers and settings) for every trained IFT condition at the best observed epoch. Evaluation uses 10 trials per (layer, setting) cell.

Across all model sizes, the random-layer semantic condition achieves the highest sentence localisation accuracy, indicating that training the model to generalise across injection layers is more effective than training at a single fixed layer. The semantic+reasoning condition adds a secondary training signal (predicting the concept name), which benefits strength comparison more than localisation in smaller models. Gaussian noise vectors serve as a control: gains in the Gaussian condition reflect learning to detect the presence and magnitude of any hidden-state perturbation, not concept-specific content.

10 IFT Hyperparameters and Training Configuration↩︎

10.1 LoRA Configuration↩︎

We fine-tune all models with Low-Rank Adaptation. LoRA adapters are applied to the query, key, value, and output projection matrices of every self-attention block (q_proj, k_proj, v_proj, o_proj), as well as the MLP gate, up, and down projections (gate_proj, up_proj, down_proj). Hyperparameters: rank \(r = 16\), scaling factor \(\alpha = 32\), dropout \(= 0.05\). All adapter weights are initialised to zero (standard LoRA initialisation); base model weights are frozen. Training is performed in bfloat16.

10.2 Optimiser and Schedule↩︎

We use AdamW with learning rate \(2 \times 10^{-4}\), weight decay \(0.01\), gradient clipping at norm \(1.0\), and gradient accumulation over \(8\) micro-steps (effective batch size \(= 8\) training examples per update). The learning rate follows a linear warmup over 50 steps followed by linear decay to zero over the remaining training steps.

10.3 Training Loop↩︎

Each epoch samples 100 concepts from the 950-concept training split. For each concept, 10 independent trials are constructed:

1. Sample 10 neutral sentences uniformly from a pool of 100 fixed background sentences (see below).

2. Choose a random injection position \(i \in \{1, \ldots, 10\}\) and a random coefficient \(\alpha \in \{1, 2, 5, 10\}\).

3. In the fixed-layer condition, inject at layer 3; in the random-layer condition, sample a layer uniformly from \(\{0, 3, 6, 9, 12, 15\}\).

4. Register a forward hook that adds \(\alpha \cdot \hat{v}\) to the hidden states at the token positions corresponding to sentence \(i\), where \(\hat{v}\) is the L2-normalised steering vector.

5. Compute cross-entropy loss over the digit-token logits \(\{\)“1”, …, “10”\(\}\) at the final token position (standard condition), or over both the sentence index token and the concept-name tokens (semantic+reasoning condition).

Each epoch therefore yields \(100 \times 10 / 8 = 125\) gradient updates.

10.4 Training Prompt↩︎

The training prompt follows the same multi-turn template used at evaluation time. The user turn presents the 10 sentences labelled SENTENCE 1: through SENTENCE 10: and asks the model to localise the injected thought. The assistant partial response is prefixed with:

Yes, I detect an injected thought. It is located in SENTENCE

and the loss is computed on the logit for the correct digit token immediately following this prefix. In the semantic+reasoning condition the full target is:

{digit} and the concept of the injection is {concept_name}.

with cross-entropy loss applied only to the digit token and the concept-name tokens (all other positions masked to \(-100\)).

10.5 Background Sentence Pool↩︎

The 100 neutral background sentences used in training and evaluation are fixed across all runs. They are drawn from a hand-curated pool of semantically neutral English sentences designed to avoid strong topic signals (e.g.”The weather forecast predicted a mild weekend.”, “She placed the book on the shelf and turned off the lamp.”). Using a fixed pool ensures that any improvement in localisation accuracy cannot be attributed to the model memorising specific sentence co-occurrences with concepts.

10.6 Evaluation Protocol↩︎

After each training epoch we evaluate on 50 held-out concepts. Sentence localisation: for each (concept, layer, \(\alpha\)) cell with \(\alpha \in \{1,2,5,10\}\), we run 10 trials and record accuracy as the fraction in which the model’s argmax over digit logits matches the true injection position. Chance is \(1/10 = 10\%\). Strength comparison: for each (concept, layer, \((\alpha_\text{weak},\alpha_\text{strong})\)) cell with pairs \((1,5),(2,10),(10,50)\), we run 10 trials, each consisting of two counterbalanced sub-trials (strong injection at sentence 1 vs.sentence 2), and score accuracy as the fraction of sub-trials in which the model correctly identifies the more strongly injected sentence. Chance is \(50\%\). The reported “average” metric is the grand mean over all (layer, setting) cells, giving equal weight to each cell.

References↩︎