1 Introduction↩︎

Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable mechanisms by identifying internal components and circuits that implement model behavior [1]–[3]. In transformer language models, circuit discovery has localized behavior to sparse sets of heads, neurons, or edges [1], [4]–[7].

However, many circuit discovery methods are evaluated on template-based tasks where counterfactuals and task metrics are well-defined. Natural-text settings remain harder [8], as diverse inputs make faithful counterfactuals expensive or ambiguous. This limitation is critical for low-resource language adaptation, where labeled data is scarce [9]–[11] and fine-tuning often risks instability or catastrophic forgetting [12]–[15].

This challenge is especially visible in Indonesian local languages, where resources remain limited despite high linguistic diversity [10]. NusaX [16] provides a parallel sentiment benchmark for Indonesian and ten Indonesian local languages, making it a useful testbed for low-resource transfer. This setting also exposes a limitation of standard transfer evaluation: target-language gains may come with changes to source behavior, related tasks, broader classification ability, or perturbation consistency, connecting to catastrophic forgetting and behavioral testing [13], [17], [18].

To address this, we investigate whether circuits discovered from natural source-language data can guide controlled adaptation. We adapt Contextual Decomposition for Transformers (CD-T) [7] using label-balanced activation means and task-directional relevance scoring, enabling counterfactual-free discovery on natural text. We then introduce Circuit-Targeted Supervised Fine-Tuning (CT-SFT), which restricts parameter updates exclusively to task-relevant attention heads and LayerNorm.

Our main evaluation uses NusaX sentiment transfer (Indonesian to Acehnese, Buginese, Javanese, and Minangkabau) [16] using Llama-3.2-1B-Instruct [19], [20], comparing CT-SFT against full fine-tuning and sparse non-circuit controls, while evaluating retention on IndoNLU [21], [22] and SIB200 [23]. Finally, we test the method’s scope using Llama-3.2-3B-Instruct [19], [20] for scalability, and Llama-3.2-1B-Instruct and Qwen2.5-0.5B [24], [25] on XNLI [26] for task and family breadth, applying English competence tuning before discovery.

We ask four questions: (1) can counterfactual-free CD-T identify faithful circuits from natural text? (2) can circuit-targeted tuning support low-resource transfer, and what do non-circuit sparse controls reveal about adaptation? (3) does circuit-targeted tuning provide more controlled adaptation beyond target accuracy, including source retention, related- and less-related task retention, perturbation consistency, and interpretability? and (4) do the findings extend beyond the main NusaX Llama-1B setting, across model scale and a different multilingual classification task?

1.0.0.1 Contributions.

Our contributions are as follows.
C1. We adapt CD-T to natural-text classification using label-balanced activation means and task-directional relevance scoring, enabling counterfactual-free discovery of faithful task circuits.
C2. We introduce circuit-targeted supervised fine-tuning (CT-SFT), a controlled low-resource adaptation strategy that updates only discovered task-relevant heads.
C3. We show that circuit tuning is competitive for transfer, while non-circuit sparse updates can also improve target accuracy, suggesting both mechanism editing and capacity recruitment.
C4. We evaluate adaptation beyond target accuracy using source retention, related- and less-related task retention, perturbation consistency, and circuit faithfulness, showing that CT-SFT gives the most balanced and interpretable profile.
C5. We test the scope of CT-SFT with Llama-3.2-3B on NusaX and both Llama-1B and Qwen2.5-0.5B on XNLI, covering scalability and task/language-family breadth.

2 Related Works↩︎

Mechanistic interpretability (MI) localizes computations inside transformers [8], [27]. Prior work has identified behavior-specific circuits using path patching  [1], causal interventions, automated circuit discovery [4], and attribution-based methods [5], [6], [28], [29]. Contextual Decomposition for Transformers (CD-T) offers an alternative by decomposing activations into relevant and irrelevant streams [7].

Many circuit discovery methods are evaluated on template-based tasks, where counterfactuals and reference states are easier to define than in unstructured natural text [8]. This gap is especially important for multilingual settings, since MI research has primarily focused on English [30]. Recent work has begun to examine multilingual interpretability and cross-lingual circuit structure, finding both shared pathways and language-specific specialization [31], [32].

Recent work links mechanistic structure to adaptation: circuits can be reused across tasks [33], fine-tuning updates may concentrate in a small subset of heads [34], and circuit-guided tuning can update selected subgraphs/components instead of all parameters [35], [36].

In contrast, we adapt CD-T [7] to unstructured, low-resource natural text and use the discovered circuits as targets for circuit-targeted supervised fine-tuning. This treats circuit discovery not only as diagnosis, but as a controlled adaptation strategy for low-resource transfer with behavior preservation. By comparing circuit updates with non-circuit sparse controls, we analyze whether adaptation edits an existing task mechanism or recruits additional capacity.

3 Background: Contextual Decomposition for Transformers (CD-T)↩︎

Contextual Decomposition for Transformers (CD-T) [7] identifies task-relevant components by decomposing each component activation into a baseline stream and an input-induced relevant stream. Let \(a_x(s)\) be the activation of attention head \(s\) on input \(x\), and let \(\mu(s)\) be the precomputed mean activation for that head. CD-T defines the baseline processing as \(\gamma_s = \mu(s)\) and the input-induced deviation as \(\beta_s = a_x(s) - \mu(s)\). In the original CD-T setting, \(\mu\) is estimated from faithful counterfactual examples, such as swapped names or years in templated tasks like IOI, greater-than, and docstring completion. The \(\gamma\) and \(\beta\) streams are then propagated through the network using contextual decomposition rules to estimate the contribution \(\beta_{s\to t}(x)\) from head \(s\) to a target \(t\).

CD-T ranks components using an unsigned relevance score based on the magnitude ratio between relevant and irrelevant streams: \[\label{eq:cdt95original95score} R(s,t) = \frac{\lVert \beta_{s\to t} \rVert_1}{\lVert \gamma_{s\to t} \rVert_1}.\tag{1}\] This score favors heads whose relevant contribution is large relative to the baseline stream. Starting from the output readout, CD-T then constructs a circuit by iteratively selecting top-scoring heads and treating selected heads as new targets for the next backward expansion step.

While effective for controlled settings, this formulation has two limitations for natural-text classification. First, natural text does not provide obvious faithful counterfactuals for estimating \(\mu\). Second, the unsigned magnitude score can select heads with large but task-irrelevant or label-opposing contributions. The next section describes our counterfactual-free adaptation of CD-T for this setting.

4 Counterfactual-Free Circuit Discovery for Natural Text↩︎

Figure 1 summarizes our overall pipeline: we discover a source-language circuit with counterfactual-free CD-T, use the resulting circuit as the update target for CT-SFT, and evaluate the adapted models beyond target transfer accuracy.

This section describes the discovery stage. We modify CD-T for natural-text classification in three steps. First, we replace counterfactual-based activation means with label-balanced means computed from natural source-language examples. Second, we rank heads using task-directional relevance, so selected components are scored by whether they support the correct label rather than by magnitude alone. Third, we construct circuits through iterative backward expansion and apply score-based pruning to obtain a compact final circuit for adaptation.

4.1 Label-Balanced Precomputed Mean↩︎

A key dependency in CD-T is the precomputed mean \(\mu\), which determines what is treated as background (\(\gamma\)) versus signal (\(\beta\)). In templated tasks, \(\mu\) can be estimated from faithful counterfactuals, but such counterfactuals are difficult to construct for unstructured natural text. We therefore estimate \(\mu\) from a label-balanced set of in-distribution examples \(\mathcal{M}\), using equal numbers of examples per class to reduce class-directional leakage into the baseline, defining \(\mu(s) = \mathbb{E}_{x \sim \mathcal{M}}[a_x(s)]\) to approximate a neutral operating point for each head.

4.2 Task-Directional Relevance↩︎

Original CD-T ranks components using an unsigned magnitude ratio, but large contributions are not necessarily task-supporting: a head may be non-discriminative or even anti-aligned with the gold label. We therefore score heads by whether their relevant contribution supports the correct label direction.

At the logit readout, let \(y_c\) be the correct label and \(\mathcal{Y}_{\text{other}} = \mathcal{Y}\setminus\{y_c\}\). We define: \[R(s,t) = \beta_{s\to t}(y_c) - \frac{1}{|\mathcal{Y}_{\text{other}}|} \sum_{y \in \mathcal{Y}_{\text{other}}} \beta_{s\to t}(y).\] For intermediate residual contributions, we define a task direction from the unembedding matrix: \[\mathbf{v}_{\text{task}} = W_U(y_c)-\frac{1}{|\mathcal{Y}_{\text{other}}|}\sum_{y \in \mathcal{Y}_{\text{other}}} W_U(y),\] and score the contribution by its projection onto this direction: \[R(s,t)= \frac{\beta_{s \to t}(x) \cdot \mathbf{v}_{\text{task}}}{\lVert \mathbf{v}_{\text{task}}\rVert_2}.\]

Positive scores indicate alignment with the correct-label direction, while negative scores indicate opposition. Figure 2 illustrates this distinction.

4.3 Circuit Construction and Pruning↩︎

Using the same iterative backward expansion procedure as standard CD-T (Section 3), we construct our circuit by expanding backward from the output decision using task-directional relevance. These selected heads are then treated as new targets for the next depth, and the candidate circuit is obtained by aggregating selected heads across depths. We then apply score-based pruning by removing selected heads with the lowest CD-T relevance scores (Appendix 9), producing a smaller final circuit \(C\) for mechanism-guided tuning.

5 Mechanism-Guided Adaptation↩︎

5.1 Circuit-Targeted Supervised Fine-Tuning (CT-SFT) Setup↩︎

We use the discovered circuit as the update target for transfer: if the selected heads capture a task-relevant mechanism, then updating them should provide a more controlled route for adaptation than updating the full model. Given the final circuit \(\mathcal{C}\) from Section 4.3, Circuit-Targeted Supervised Fine-Tuning (CT-SFT) restricts optimization to heads in \(\mathcal{C}\) and LayerNorm. We freeze MLPs, token embeddings, the unembedding matrix, and all unselected heads, using head-level gradient masking to localize learning to the discovered circuit.

5.2 Baselines and Sparse Controls↩︎

5.2.0.1 Editing vs. recruitment.

Table 1 summarizes the adaptation methods. Circuit tuning tests mechanism editing, since it updates heads identified as task-relevant. The sparse controls test whether similar low-resource transfer can also arise from capacity recruitment, where non-circuit heads are repurposed for the target task.

3pt

6 Experimental Setup↩︎

6.1 Research Questions and Evaluation Map↩︎

We evaluate our method through four research questions (RQs) focused on circuit faithfulness [4], [6], [37], transfer effectiveness, controlled adaptation, and cross-task robustness. Table 2 maps these RQs to their corresponding evaluation protocols. Full definitions for intervention metrics and datasets are provided in Appendices 9.2 and 9.4.

2.5pt

6.2 Model↩︎

We use Llama-3.2-1B-Instruct [19], [20] as the main model for our reported experiments. To test scalability, we additionally run the NusaX experiment on Llama-3.2-3B-Instruct [19], [20]. For the XNLI [26] robustness experiment, we evaluate across two model families by using both Llama-3.2-1B-Instruct and Qwen2.5-0.5B [24], [25]. Because these models perform near chance on the three-way XNLI task before task adaptation, we first apply English competence tuning before circuit discovery. Thus, the NusaX experiments use an instruction-tuned model with usable sentiment behavior out of the box, while the XNLI experiments first learn the English source task through competence tuning.

3pt

6.3 Datasets and Evaluation Setting↩︎

We utilize NusaX-Senti [16] for low-resource transfer (Indonesian source to Acehnese, Buginese, Javanese, and Minangkabau targets). We evaluate target transfer and Indonesian source retention using 400 examples per language. Behavior preservation is assessed via Indonesian IndoNLU EmoT [21], [22] (related-task retention) and SIB200 topic classification [23] (less-related retention), each using 400 examples. Perturbation consistency is evaluated using 200 label-preserving and 200 label-flipping pairs derived from NusaX-Senti, reported as pair-level success rates [17].

Finally, we evaluate robustness on XNLI [26]. To match the NusaX evaluation budget, we construct XNLI subsets with 500 training examples, 100 validation examples, and 400 test examples per language. We use English for competence tuning and circuit discovery, with Thai, Vietnamese, Spanish, and Chinese as target languages. We report XNLI target transfer and English source retention. Unlike the Indonesian benchmarks, XNLI starts from an English NLI competence-tuned checkpoint. Therefore, IndoNLU EmoT, SIB200, and Indonesian perturbation evaluations are not directly comparable in this setting. Further dataset details and examples are provided in Appendix 9.4.

6.4 Training and Discovery Protocol↩︎

Each target language is fine-tuned with \(n \in \{25, 50, 75, 100\}\) examples across four seeds. Circuit discovery uses a label-balanced mean set and a correctly predicted discovery set from the source language. Heads are selected using task-directional relevance, followed by score-based pruning to obtain the final circuit. All sparse controls are then matched to the resulting post-pruning head budget for each seed.

The Llama-3.2-3B-Instruct NusaX experiments follow this same protocol with the main Llama-3.2-1B-Instruct evaluation. For XNLI, we first competence-tune Llama-3.2-1B-Instruct and Qwen2.5-0.5B on 250 English XNLI training examples, then perform circuit discovery on the competence-tuned English checkpoints. These circuits are then used for target-language CT-SFT. Full implementation, discovery, and training parameters are detailed in Appendix 9.1.

7 Results↩︎

7.1 RQ1: Counterfactual-free CD-T discovers faithful circuits in Llama↩︎

Key finding. The discovered Llama circuits show higher faithfulness and stronger knockout effects than matched sparse controls.

Table 3 reports validation-set intervention scores. Faithfulness measures retained full-model label-token accuracy when only selected heads are preserved under mean ablation, while knockout measures the relative accuracy drop when selected heads are ablated. Across Indonesian and the target languages, Circuit achieves higher faithfulness and knockout than matched sparse controls, indicating a stronger intervention target for CT-SFT. Sparse controls can still retain non-zero faithfulness because transformer behavior may be distributed across redundant or self-repairing components [1], [38]. However, their weaker knockout effects show that they are less task-specific and less causally central than the discovered circuit.

7.2 RQ2: Circuit tuning supports transfer, but sparse adaptation also involves recruitment↩︎

Key finding. Circuit tuning is competitive for low-resource transfer, but least-relevant and random non-circuit updates can also improve target accuracy, suggesting that sparse adaptation may involve both mechanism editing and capacity recruitment.

Table 4 and the reference baselines (Tables 5) summarize target-language adaptation. Zero-shot evaluations establish the starting performance before adaptation, while Full FT provides a high-plasticity reference that updates all model parameters. Averaged across training sizes, circuit tuning is highly competitive among sparse methods (and scales well with more target-language examples, as detailed in Appendix Figure 4). However, the results also show that non-circuit sparse updates can sometimes match or exceed circuit tuning (e.g., Random non-circuit on Buginese), indicating that transfer accuracy alone is insufficient to identify the most controlled or interpretable update strategy.

7.3 RQ3: Circuit tuning provides more controlled adaptation beyond target accuracy↩︎

Key finding. Non-circuit sparse updates can be competitive on individual metrics, but Circuit provides the most balanced and interpretable profile across faithfulness, transfer, retention, and perturbation consistency.

We use the preservation evaluations defined in Section 6.3 to test whether sparse transfer gains remain stable beyond target accuracy.

3.5pt

7.3.0.1 Source-language retention.

On Indonesian source-task retention, Random non-circuit achieves the strongest average score (Table 4). One possible explanation is that Random non-circuit does not directly update the discovered source-task circuit, so the original sentiment mechanism is less disrupted. At the same time, the updated non-circuit heads can provide additional capacity for target-language adaptation. This supports the recruitment interpretation: non-circuit updates can preserve the original circuit while repurposing other heads to support transfer.

7.3.0.2 Other-task retention.

Circuit is more stable than non-circuit updates on the two other-task retention evaluations (Table 4). This suggests that localizing updates to the discovered circuit does not strongly disrupt the broader classification logic used by related and less-related tasks. In contrast, non-circuit updates can support transfer through recruitment, but the recruited heads are not guaranteed to preserve other task-relevant computations. Since Least-relevant and Near-zero are defined only with respect to sentiment relevance, updating them may still disturb affective or topic-classification behavior, explaining the larger drops on IndoNLU EmoT and SIB200.

7.3.0.3 Perturbation consistency.

Perturbation results show that label-preserving and label-flipping consistency capture different behaviors. Random non-circuit is slightly better than Circuit on label-preserving pairs, while Near-zero obtains the highest label-flipping score. This suggests that Near-zero heads are more plastic: because they are weakly tied to the original sentiment circuit, they can be repurposed more easily when explicit sentiment cues are flipped. However, this flexibility is not equivalent to a better sentiment mechanism, since Near-zero is also weakest on transfer, source retention, topic retention, and label-preserving perturbations. Thus, Near-zero is more responsive to label-changing edits, while Circuit provides a more balanced profile across transfer, retention, and consistency.

7.3.0.4 Reference baselines.

Table 5 reports Full FT and LoRA as reference baselines. Full FT slightly improves average transfer over Circuit but substantially weakens preservation. LoRA performs well on source retention, emotion retention, and label-flipping consistency, suggesting that it preserves behavior close to the sentiment objective. However, it is weaker than Circuit on target transfer, SIB200 topic retention, and label-preserving perturbations, indicating weaker preservation of broader, less-related classification behavior. Thus, LoRA is effective for sentiment-adjacent preservation, but does not provide the same controlled-adaptation profile as Circuit.

4pt

7.4 RQ4: Does the finding extend beyond the main Llama-1B NusaX setting?↩︎

Key finding. The core findings extend beyond the main Llama-1B NusaX setting: Circuit remains causally load-bearing and avoids the severe forgetting of Full FT, while non-circuit transfer gains recur, supporting the recruitment pattern from RQ2.

7.4.0.1 Scalability on Llama-3.2-3B-Instruct.

Tables 3 and 6 test whether the NusaX findings hold at 3B scale. The intervention results strengthen RQ1: Circuit obtains much stronger faithfulness and knockout scores than matched controls, while Random NC has near-zero knockout. Thus, the discovered heads remain causally central even in the larger model.

4pt

The downstream results support RQ3. Full FT gives slightly higher transfer, but severely damages related and broader behavior, dropping Emotion and Topic retention to near-zero levels. LoRA is less destructive, but remains weak on Topic retention and target transfer. Circuit therefore preserves a stronger controlled-adaptation profile against standard baselines: it remains competitive for transfer while avoiding the severe behavioral collapse of Full FT and LoRA.

At the same time, the sparse controls reinforce RQ2. Random non-circuit is also strong at 3B, matching Circuit on transfer and source retention and slightly exceeding it on Topic retention. This supports the recruitment interpretation: non-circuit heads can sometimes provide useful adaptation capacity. However, Random NC has near-zero knockout and is not a faithful source-task mechanism. Thus, the value of circuit targeting is not that it always dominates every sparse control, but that it achieves a comparable profile from a principled, interpretable, knockout-verified locus. This directly answers the “why not random?” question: random recruitment may sometimes work, but it is not predictable or mechanistically grounded.

7.4.0.2 XNLI task and language-family robustness.

Table 7 summarizes the XNLI robustness check. Since XNLI starts from an English NLI competence-tuned checkpoint, the IndoNLU, SIB200, and Indonesian perturbation evaluations used for NusaX are not directly comparable. We therefore use XNLI to test source retention and the RQ1 circuit-causality finding on a harder task across two model families. Intervention results (Appendix Table 19) show that Circuit has the strongest knockout effects across both models. Faithfulness is lower than in NusaX, suggesting that NLI behavior is more distributed: the selected heads are causally load-bearing, but not sufficient alone to reconstruct the full behavior.

3.5pt

Downstream, Full FT causes severe English forgetting, while sparse methods and LoRA stay closer to the competence-tuned checkpoint. On Qwen, sparse adaptation improves transfer, with Avg. non-circuit slightly exceeding Circuit, echoing the RQ2 recruitment pattern. Llama starts from weaker XNLI competence, so gains are smaller; we treat it as a weak-competence stress test rather than a scale comparison. Overall, XNLI supports the preservation and circuit-causality findings while showing that transfer can also benefit from non-circuit recruitment.

8 Discussion and Conclusion↩︎

This paper connects circuit discovery with controlled adaptation in natural-text classification. By adapting CD-T with label-balanced activation means and task-directional relevance scoring, we identify attention-head circuits without manually constructed counterfactuals. These circuits show stronger faithfulness and knockout effects than matched sparse controls, making them meaningful intervention targets for CT-SFT.

Our results suggest an editing–recruitment distinction. Circuit tuning edits a causally verified source-task mechanism, while non-circuit sparse updates can sometimes improve transfer by recruiting external capacity. This recruitment can be useful, but it is less predictable because the updated heads are not guaranteed to be task-specific or causally grounded. Thus, Random NC can sometimes match Circuit on aggregate metrics, but it lacks the mechanistic guarantee provided by knockout-verified circuit heads.

The baseline comparisons show why controlled adaptation matters. Full FT can improve transfer, but it causes severe behavioral drift. LoRA preserves sentiment-adjacent behavior better, but is weaker on broader topic retention and label-preserving perturbations. The Llama-3B and XNLI extensions support the same framing: circuit heads remain causally load-bearing, Full FT forgets strongly, and non-circuit gains recur as evidence of recruitment.

Overall, CT-SFT is not a method that always maximizes every metric. Its value is that it provides a principled update location: a compact, faithful, knockout-verified mechanism that can be adapted while limiting damage to existing behavior. When target accuracy alone matters, other sparse or PEFT methods may be competitive; when controlled low-resource adaptation matters, circuit targeting offers a more interpretable balance between transfer, preservation, and causal validity.

Limitations↩︎

This work has several limitations regarding model competence, evaluation scope, and interpretability granularity.

8.0.0.1 Dependence on checkpoint competence.

First, CT-SFT depends on the existence of a usable source-task mechanism before circuit discovery. Our method is designed to identify and update task-relevant components in a model that already has some competence on the source task. As observed in the Llama-1B XNLI setting, if the competence-tuned checkpoint performs near chance, sparse adaptation cannot construct this capability from scratch nor improve cross-lingual transfer. CT-SFT functions as an editing and routing technique. It strictly requires a functional source mechanism to build upon.

8.0.0.2 Scope of architectures and scales.

Second, our experiments cover a limited range of architectures and scales. We focus on relatively small decoder-only models (Llama-3.2-1B, Llama-3.2-3B, and Qwen2.5-0.5B). While these results suggest that the findings generalize across these specific families, we do not evaluate encoder-only models traditionally used for multilingual classification, nor do we test deployment-scale models. At massive scales with tens of billions of parameters, representations may become even more distributed. This could potentially alter the redundancy and self-repair dynamics we observed during circuit knockout. Future work should test whether these CT-SFT patterns hold at larger scales.

8.0.0.3 Scope of tasks and evaluation axes.

Third, the task scope remains limited. The counterfactual-free adaptation of CD-T relies on task-directional relevance scoring, which is tailored for classification tasks with a discrete set of label tokens. Defining a mathematically rigorous task direction for open-ended generative tasks remains an open challenge. Furthermore, while XNLI provides a task and language-family robustness check, it lacks direct analogues for the IndoNLU emotion, SIB200 topic, or Indonesian perturbation evaluations. Consequently, our strongest multi-axis preservation claims rely on the NusaX sentiment setting, whereas XNLI primarily supports the source-retention and circuit-causality claims.

8.0.0.4 Intervention metrics.

Fourth, our intervention metrics evaluate internal mechanism logic rather than end-to-end generation. Faithfulness and knockout are computed using label-token accuracy from a single forward pass, rather than autoregressive text generation. While measuring single-token logits is standard for controlled circuit comparison, it is not strictly equivalent to downstream generation performance. Additionally, we rely on mean ablation as our intervention baseline; using different reference activations (such as zero ablation or resampling) or different intervention sites could yield different absolute scores.

8.0.0.5 Circuit granularity and interpretability.

Finally, our circuit granularity is restricted to attention heads. CT-SFT updates selected heads and LayerNorm, while MLPs, embeddings, the unembedding matrix, and unselected heads are frozen. This design makes the update localized and interpretable, but it may miss mechanisms implemented in MLPs or finer-grained edges. Because MLPs frequently store factual knowledge and cross-lingual vocabulary mappings, excluding them means our discovered circuits represent a highly effective intervention target rather than a fully exhaustive explanation of all model behavior. Extending this approach to mixed head-MLP circuits or edge-level targets is an important direction for future work.

Use of AI Assistants↩︎

We used AI assistants as writing and coding aids, including for improving clarity and grammar, restructuring text, drafting LaTeX/table snippets, debugging scripts, and refactoring small code fragments. All scientific content, experimental design, implementation decisions, analyses, interpretations, and reported results were produced and verified by the authors. AI assistants were not used to generate datasets, train models, or produce experimental results.

9 Experimental Details↩︎

9.1 Training and Discovery Protocol↩︎

This appendix provides the full training, evaluation, and circuit discovery settings used in the main experiments. Circuit discovery, circuit pruning, CT-SFT, and sparse-control evaluations were implemented in TransformerLens¹. LoRA baselines were implemented using Hugging Face Transformers and PEFT. Table 8 summarizes the model sizes and compute setup. Table 9 summarizes the adaptation and evaluation setup, while Table 10 summarizes the circuit discovery and CT-SFT configuration. For each target language, all sparse control methods are matched to the same post-pruning circuit head budget for each seed.

3pt

3pt

Starting from the output readout, CD-T then constructs circuit by iteratively selecting top-scoring and treating selected heads as new targets for next backward expansion step (Figure 3). After the initial CD-T circuit is constructed, we further prune the selected heads using a multi-pass faithfulness-guided greedy procedure. In each pass, candidate heads are considered in increasing absolute CD-T score order. A head is removed if its removal improves the pruning objective while keeping the faithfulness degradation within a fixed tolerance. We repeat this procedure for up to five passes, stopping early when a full pass removes no additional heads. The pruning objective is a weighted combination of faithfulness and knockout scores: \[\mathrm{Score}(C) = 0.7 \cdot \mathrm{Faith.}(C) + 0.3 \cdot \mathrm{Knockout}(C).\] Here, faithfulness measures the accuracy retained when only the circuit heads are kept and all non-circuit heads are mean-ablated. Knockout measures the accuracy drop when the circuit heads are mean-ablated while all other heads remain active. We weight faithfulness higher than knockout to prioritize the standalone task competence of the discovered circuit, ensuring it retains strong task competence, while using the knockout penalty as a secondary constraint to prune redundant heads. These weights were selected empirically based on preliminary experiments on the Indonesian validation set, where we found that weighting knockout equally or higher led to overly aggressive pruning and degraded the circuit’s standalone competence. We further constrain each accepted pruning step to have at most a 2-point faithfulness drop. The resulting pruned circuit is used for CT-SFT, and all sparse controls are matched to its final head budget for each seed.

3pt

9.2 Faithfulness and Knockout Metrics↩︎

We evaluate discovered circuits using label-token intervention accuracy rather than generated accuracy. For each validation example, we append the gold label \(y\) to the prompt \(x\) and run a single forward pass. We then read the logits at the position immediately before the gold label token. The model prediction is the highest-logit label among the task-specific label set: \[\hat{y} = \arg\max_{y' \in \mathcal{Y}} z_{y'}(x),\] where \(\mathcal{Y}\) is the task-specific label set and \(z_{y'}(x)\) denotes the next-token logit assigned to label \(y'\). For NusaX, \(\mathcal{Y}=\{\texttt{negative},\texttt{neutral},\texttt{positive}\}\); for XNLI, \(\mathcal{Y}=\{\texttt{entailment},\texttt{neutral},\texttt{contradiction}\}\).

Let \(C\) be a selected set of attention heads. We use mean ablation at the attention-head output \(z\): ablated heads are replaced by their mean activations computed over a label-balanced source-language mean set. Let \(f_{\mathrm{full}}\) denote the unablated model, \(f_{\mathrm{keep}\;C}\) denote the intervention where only heads in \(C\) remain active and all non-selected heads are mean-ablated, and \(f_{\mathrm{ablate}\;C}\) denote the intervention where heads in \(C\) are mean-ablated while all other heads remain active.

Faithfulness is defined as the percentage of full-model label-token accuracy retained when only the selected heads are preserved: \[\mathrm{Faithfulness}(C) = 100 \times \frac{ \mathrm{Acc}(f_{\mathrm{keep}\;C}) }{ \mathrm{Acc}(f_{\mathrm{full}}) }.\]

Knockout is defined as the relative drop in full-model label-token accuracy when the selected heads are ablated: \[\mathrm{Knockout}(C) = 100 \times \frac{ \mathrm{Acc}(f_{\mathrm{full}}) - \mathrm{Acc}(f_{\mathrm{ablate}\;C}) }{ \mathrm{Acc}(f_{\mathrm{full}}) }.\]

Both metrics are computed on the validation split. The mean set is sampled from the source-language training split using a label-balanced sampling procedure: Indonesian for NusaX and English for XNLI.

9.3 Evaluation Prompts↩︎

All accuracy evaluations are generation-based. The model generates a short label string using greedy decoding, and the generated output is parsed with the same parser for CT-SFT, sparse controls, Full FT, and LoRA. NusaX transfer and source-language retention use at most three generated tokens, while fixed local evaluations such as IndoNLU EmoT and SIB200 use at most five generated tokens.

4pt

9.3.0.1 NusaX sentiment classification.

For controlled perturbation evaluation, and for NusaX transfer/source evaluation when using the instruct prompt style, we use:

Classify the sentiment of the following text.
Reply with exactly one word only: positive, negative, or neutral.
Do not explain.
Text: {text}
Sentiment:

The valid labels are positive, negative, and neutral. In all NusaX evaluations, the generated text is parsed into one of these labels.

9.3.0.2 IndoNLU EmoT emotion detection.

For IndoNLU EmoT, the evaluation uses the prompt stored in the fixed evaluation file. The model is evaluated with the same generation-based parser and the label set:

Classify the emotion of the following Indonesian text.
Reply with exactly one label only: anger, fear, happy, love, sadness.
Do not explain.
Text: {text}
Emotion:

9.3.0.3 SIB200 topic classification.

For SIB200 topic classification, the evaluation uses the prompt stored in the fixed evaluation file. The valid labels are:

Classify the topic of the following text.
Reply with exactly one label only: science/technology, travel, politics, sports, health, entertainment, geography.
Do not explain.
Text: {text}
Topic:

9.3.0.4 Controlled perturbation evaluation.

For label-preserving and label-flipping perturbation pairs, we use the NusaX sentiment prompt above for both the original and perturbed sentences. Label-preserving consistency is counted as successful when the prediction remains unchanged across the pair. Label-flipping consistency is counted as successful when the original prediction matches the original label and the perturbed prediction matches the intended flipped label.

9.4 Evaluation Datasets and Examples↩︎

This section summarizes the evaluation datasets used in our experiments and provides examples from the actual evaluation sets. Table 11 gives an overview of all evaluation settings, including the dataset, language, size, and label space or success criterion. We group the evaluations into target transfer, behavior preservation, and controlled perturbation consistency. Target transfer and source-language retention use NusaX-Senti, while IndoNLU EmoT and SIB200 are used to evaluate whether adaptation preserves related and less-related classification behavior. The perturbation sets are derived from Indonesian sentiment examples and are evaluated at the pair level.

9.4.0.1 Evaluation set construction.

All evaluation examples were obtained from the official Hugging Face dataset releases. Before evaluation, we applied the same prompt template used by the model and filtered out examples whose tokenized length exceeded 128 tokens, including the prompt and label text. For each evaluation dataset, we then constructed label-balanced subsets of 400 examples, matching the NusaX-Senti test-set size used in the main transfer experiments. This filtering and balancing procedure was applied consistently to NusaX-Senti, IndoNLU EmoT, and SIB200.

For controlled perturbation evaluation, we constructed Indonesian sentiment pairs from the NusaX-Senti test set. The label-flipping set contains 200 pairs: 100 negative-to-positive pairs and 100 positive-to-negative pairs. The label-preserving set also contains 200 pairs: 70 negative, 70 positive, and 60 neutral pairs. Label-preserving pairs are used to test prediction invariance, while label-flipping pairs are used to test whether predictions change in the intended sentiment direction.

9.4.0.2 NusaX-Senti.

Table 12 shows a parallel NusaX-Senti example across Indonesian and the four target languages used in our transfer experiments.

3pt

9.4.0.3 IndoNLU EmoT.

Table 13 shows examples from the IndoNLU EmoT evaluation set. This task is used to evaluate related-task retention because emotion detection and sentiment classification both involve affective language understanding.

3pt

9.4.0.4 SIB200 topic classification.

Table 14 shows examples from the Indonesian SIB200 topic classification evaluation set. This task is used to evaluate less-related behavior preservation beyond sentiment and affective classification.

3pt

4pt

9.4.0.5 Controlled perturbation examples.

Table 15 shows examples from the Indonesian controlled perturbation sets. Label-preserving pairs keep the original sentiment label unchanged and are counted as successful when the model prediction remains unchanged. Label-flipping pairs intentionally alter sentiment-bearing words and are counted as successful when the original prediction matches the original label and the perturbed prediction matches the intended flipped label.

9.5 LoRA Baseline↩︎

We include LoRA as a standard parameter-efficient fine-tuning reference. Unlike CT-SFT, LoRA does not use circuit information and is not intended as a mechanism-localized intervention. We therefore report it as an additional adaptation baseline, rather than as part of the matched sparse-control comparison.

9.5.0.1 Training setup.

For each target language and training size, LoRA is trained on the same target-language examples used by the other adaptation methods. Specifically, for transfer from Indonesian to a target language \(L\), we train LoRA on \(n \in \{25, 50, 75, 100\}\) examples from \(L\), using the same random seeds, data subsampling strategy, prompt format, batch size, number of epochs, and learning rate as the other fine-tuning baselines.

We use the Hugging Face PEFT implementation with LoRA applied to the query and value projection modules, q_proj and v_proj. The LoRA rank is \(r=8\), the scaling factor is \(\alpha=16\), and dropout is set to \(0.05\). We train for 5 epochs using AdamW with learning rate \(5 \times 10^{-5}\), batch size 16, and maximum sequence length 128. During training, the base model parameters are frozen and only the LoRA adapter parameters are updated.

9.5.0.2 Evaluation protocol.

After training, we merge the LoRA adapter into the base model using the PEFT merge_and_unload() procedure. The merged model is then evaluated with the same generation-based evaluation pipeline used for CT-SFT, Full FT, and the sparse controls. This ensures that LoRA uses the same prompts, decoding procedure, parsing functions, and held-out evaluation examples as the other methods.

We evaluate LoRA on target-language NusaX sentiment accuracy, Indonesian source-language retention, IndoNLU EmoT retention, SIB200 topic retention, and Indonesian controlled perturbation consistency using label-preserving and label-flipping pairs. The perturbation metrics are reported as pair-level success rates. Temporary adapter and merged-model artifacts are deleted after evaluation; only the evaluation outputs and summary files are retained.

9.5.0.3 Role of the baseline.

LoRA provides a practical PEFT reference, but it does not test the same mechanistic hypothesis as CT-SFT. CT-SFT asks whether updating a faithful circuit gives a controlled adaptation target, while LoRA applies low-rank updates to predefined projection modules independent of circuit discovery. We therefore use LoRA to contextualize CT-SFT, but keep the main analysis focused on circuit updates and matched non-circuit sparse controls.

9.6 Artifacts and Licenses↩︎

We use publicly available research artifacts: pretrained language models, public benchmark datasets, and open-source software libraries. We use these artifacts for research evaluation only and follow their intended research use. Dataset and model artifacts are cited in the main paper, and implementation libraries are listed in Appendix 9.

10 Additional Llama-1B NusaX Results↩︎

4pt

11 Additional Scalability Results↩︎

This section provides detailed scalability results for Llama-3.2-3B-Instruct. Table 17 reports the full per-language sparse-control comparison, complementing the aggregate results in Table 6. Table 18 reports the corresponding reference-baseline comparison against Full FT and LoRA.

3.5pt

4pt

3pt

2.5pt

2.5pt

References↩︎

---

1. https://transformerlensorg.github.io/TransformerLens/↩︎