1 Introduction↩︎

Language models (LMs) encode vast amounts of knowledge in their parameters which is utilized for various tasks, such as dialogue, summarization, and reasoning [1]. As such, LMs are often viewed as information systems whose parameters act as a knowledge base [2], [3].

In a well-designed knowledge base, different queries for the same fact draw on a single source of truth, guaranteeing consistent results. For example, a knowledge base should retrieve Paris for both “What is the capital of France?” and “The capital of France is ___”. Violating this introduces risks to system reliability, consistency, and updateability [4], [5]. In this work, we ask whether LMs satisfy this property.

We investigate this through two experiments. First, a behavioral analysis where we track across training checkpoints of OLMo-3-7B IT [6] how knowledge of individual facts, drawn from datasets of (subject, relation, object) triplets, co-emerges across tasks. If facts were stored in a task-invariant manner, a model that acquires a fact for one task should simultaneously acquire it for other tasks it is already competent at. We find that co-emergence is limited, with substantial variation across tasks, suggesting that knowledge acquisition in LMs is task dependent.

Next, we analyze how task-specific knowledge encodings are manifested in model parameters. If knowledge is stored independently of task format, it should not be possible to isolate parameters that are specific to individual (fact, task) pairs. We study this through a mechanistic analysis, adapting the localization framework of [7]. For each (fact, task) pair we identify a sparse subset of parameters whose removal degrades model performance on that pair with little effect on other facts on the same task or the same fact on other tasks. Across three models and five relational datasets, we consistently find such subsets. Together with the behavioral results, this suggests that LMs maintain task-dependent parametric encodings of individual facts, instead of drawing from a shared, task-invariant store. 1 illustrates this.

We find that the degree of this separation is not uniform, as some (fact, task) encodings can be well isolated from other pairs while others show partial overlap. This raises the question of which tasks tend to have separate versus shared encodings [8]. To quantify this overlap, we develop metrics that measure how separable each (fact, task) encoding is from other pairs, and find that discrimination tasks (e.g., Multiple Choice QA) are consistently more entangled than generation tasks (e.g., Fill-in-the-Blank). Moreover, facts acquired through generation tasks generally co-emerge on other tasks, but not vice versa.

Finally, we hypothesize that part of the effectiveness of chain-of-thought (CoT) reasoning in recovering knowledge inaccessible to direct answering (without intermediate reasoning; [9]) comes from engaging parametric encodings beyond those tied to the evaluation task. We confirm this by removing the localized (fact, task) encodings. CoT largely recovers performance lost when a task’s own encoding is ablated, yet drops more than direct answering when other tasks’ encodings are removed, suggesting it relies on them more than direct answering does.

Together, these findings show that knowledge in LMs is not cleanly separated from task structure, as what the model knows and how it is asked are intertwined in parameter space. This undermines the “knowledge base” analogy, whose guarantees of reliability and controllability rest on knowledge being task-invariant. For instance, knowledge editing or unlearning interventions targeting a single task format may leave other formats intact, and single-task evaluation may provide only a partial view of what the model encodes. We release our code and data at https://github.com/amitelhelo/TaskInvariance.

2 Task-specific knowledge encodings↩︎

In a well-designed knowledge base, querying a given fact in different ways should return the same result, drawing on the same internal source of truth. We call this property task-invariance, and investigate it in LMs through a behavioral experiment (detailed in this section), tracking how acquisition of individual facts co-emerges across tasks during training, and a mechanistic experiment (§3), asking whether the parameters that support a fact differ across tasks. Our analysis shows that knowledge is fragmented across task-specific encodings; facts acquired on one task often fail to transfer to other tasks, and it is possible to localize distinct parameters that encode the same fact for different tasks.

2.1 Experimental Setup↩︎

We track factual knowledge in LMs using relational datasets [10], [11], where facts are formulated as (subject, relation, object) triplets. For example, the fact that Paris is the capital of France can be represented as the triplet (France, capital-of, Paris). Specifically, we use datasets of five relations: (country, capital-of, city), (country, official language, language), (landmark, in-country, country), (company, HQ-in-city, city), and (person, plays-instrument, instrument). Each fact is probed via six tasks: next-token completion (Completion), fill-in-the-blank (FiTB), open-ended question answering (OpenQA), four-way multiple-choice QA (MCQA), negative MCQA (Neg MCQA; select the incorrect answer), and binary statement verification (Verification).

For each dataset-task pair we composed 10 prompt paraphrases, which we use to evaluate the model’s knowledge of the facts for the task. For discrimination tasks, each paraphrase is further expanded by rotating the correct answer through all positions (4 for MCQA, 2 for Neg MCQA) or by pairing it with both a true and a false statement (Verification). For additional dataset details see §9. Representative prompts for the different tasks are provided in Appendix 6. All datasets are down-sampled to 46 facts each (matching the size of the smallest dataset), yielding 230 facts.

2.2 Co-emergence hypothesis↩︎

The task-invariance property entails predictions about training. Specifically, it implies that facts should co-emerge across tasks, a hypothesis which we formalize as follows: if different tasks retrieve a given fact from the same task-invariant parametric store, then once the model can retrieve a fact for some task (e.g., correctly answer an open question about the capital of France), then it should retrieve that fact for other tasks it is competent on (e.g., correctly answer a multiple choice question about the capital of France).

Formally, let \(\mathcal{T}\) denote a set of tasks. We write \(E(f,t)\) for the emergence step of fact \(f\) on task \(t\), defined as the first checkpoint at which the model reliably retrieves \(f\) on \(t\) (operationalized below), or \(\infty\) if this never occurs. We write \(E(\cdot, t)\) for the emergence step of task \(t\), defined as the first checkpoint in which a substantial fraction of facts are reliably retrieved on \(t\). Finally, we write \(E(f, \bar{t})\) for the earliest checkpoint at which \(f\) emerges on any task other than \(t\), defined as \(E(f, \bar{t}) = \min_{t' \in \mathcal{T} \setminus \{t\}} E(f, t')\). See 2 for illustration. Under the co-emergence hypothesis, once both prerequisites are met (the fact is retrieved on some task and the target task is competent), the fact should be retrieved on the target task as well. That is, for all facts \(f\) and tasks \(t\): \[E(f, t) \;\leq\; \max\!\bigl(E(f, \bar{t}),\; E(\cdot, t)\bigr) \label{eq:co95emergence}\tag{1}\]

2.2.0.1 Testing the co-emergence hypothesis

We use OLMo-3-7B IT [6], since its intermediate training checkpoints are publicly available. We track the model’s performance for each (fact, task) pair across training. Concretely, we examine 105 checkpoints covering the pretraining stage (100 checkpoints), midtraining and long context (2 checkpoints), and post-training (3 checkpoints).

To determine emergence of a (fact,task) pair, we take the model’s probability of the first token of the correct answer per paraphrase, normalize it by the task’s chance level, and consider the fact reliably retrieved when the mean probability over paraphrases exceeds \(\theta = 0.6\). This threshold ensures a meaningful preference for the correct answer while allowing for imperfect performance at intermediate training stages.¹ The emergence step \(E(f, t)\) is then the first checkpoint at which this criterion is met. Similarly, we set task \(t\)’s emergence step \(E(\cdot, t)\) to the first checkpoint where at least 25% of facts are reliably retrieved on \(t\). We retain only facts that the final Instruct model retrieves correctly on at least one task, and exclude (fact, task) pairs that cannot meaningfully test the prediction. These include pairs where \(t\) is the task on which the fact first emerged (no prior source to “co-emerge” with) and pairs where, at the expected step (RHS of Eq. 1 ), the fact is no longer retrieved on other tasks or task \(t\) is no longer competent. We apply this second condition only to pairs that fail to co-emerge, to avoid counting a lapsed prerequisite as evidence against the hypothesis. This yields 1,031 (fact-task) pairs. See §10 for additional details.

When co-emergence occurs by the predicted step (Eq. 1 ), we say the observation is consistent with the hypothesis, and otherwise inconsistent. 2 illustrates both cases. Notably, consistent observations on their own do not support the hypothesis, since a fact may emerge for reasons unrelated to shared storage; but inconsistent observations provide direct evidence against it.

2.2.0.2 Results

We find that the co-emergence hypothesis is frequently violated. In 47.9% of (fact, task) pairs, the fact does not emerge on the target task by the expected step, suggesting that factual knowledge does not transfer reliably across tasks during training. This finding is stable across thresholds (\(50.9\%\) at \(\theta{=}0.4\), \(49.2\%\) at \(\theta{=}0.8\)). In §4 we analyze these results at the task-pair level, asking which pairings show reliable co-emergence and which do not.

2.3 Testing for (fact,task) interaction↩︎

The above results suggest that the data does not agree with a single, task-invariant store of factual knowledge. To test the task-invariant store hypothesis statistically, we formalize it as conditional independence between facts and tasks: \[P(\text{correct} \mid f, t) = P(\text{correct} \mid f) \cdot P(\text{correct} \mid t)\] This is the expected behavior in a model where retrieving a particular fact does not depend on the task for which it is retrieved. The above corresponds to an additive model in log-probability space, and specifically, a two-way ANOVA with no interaction term. We test the hypothesis that the interaction is zero. We run the test on the chance-normalized log-probabilities, using prompt paraphrases as replications within each (fact, task) cell. Results show that the null hypothesis is rejected at every checkpoint (\(p \approx 0\)). The interaction also grows across training, explaining \(23\%\) of the variance in the final model. Thus we conclude that the data does not support a task-invariant model (see §11 for additional details and full results).

3 Mechanistic analysis↩︎

Having established that knowledge acquisition is task-dependent at the behavioral level, we turn to investigate how this manifests in the model weights. Specifically, we ask whether the same fact relies on different parameters for different tasks. We search for small subsets of model components (attention heads and MLP neurons) that are necessary, sufficient, and specific for individual (fact, task) pairs. Existence of subsets satisfying all three criteria would support the hypothesis that LMs maintain task-dependent parametric encodings of individual facts. We show that such subsets can be found.

3.1 Experimental setup↩︎

We examine three models: OLMo-2-7B IT, OLMo-2-13B IT [12], and Gemma-2-9B IT [13]. We use the datasets and tasks from §2 (without the downsampling to 46 facts), dropping Completion, which is incompatible with Instruct models, and adding two multi-hop reasoning tasks where the fact’s relation is part of a two-step chain.² In first-hop (Multi-Hop-1) the target relation is the first step, and in second-hop (Multi-Hop-2) it is the second. For example, the prompt “What is the capital city of the country containing the landmark called The Bourg-la-Reine?” follows the reasoning path landmark\(\to\)country\(\to\)capital. It can serve as a Multi-Hop-1 prompt for the (landmark, in-country, country) dataset, and as a Multi-Hop-2 prompt for the (country, capital-of, city) dataset. The exact task set varies by dataset, depending on the availability of intermediate relations for multi-hop tasks, and facts below a baseline performance threshold in any task are filtered out (see §12).

3.1.0.1 Localization via learned binary masks

We adapt the framework of [7], who trained binary masks over model parameters to find knowledge-critical subnetworks, and extend it to localize subsets of parameters that are necessary, sufficient, and specific for (fact, task) pairs. Concretely, for a target pair \((f^*, t^*)\), we learn a binary mask \(\mathbf{m} \in \{0,1\}^{|N|+|H|}\) over the sets of MLP neurons \(N\) and attention heads \(H\) in the model. We parameterize \(\mathbf{m}\) as continuous logits passed through a sigmoid, binarized at threshold \(0.5\) via a straight-through estimator [14]. Each mask is optimized to minimize: \[{!}{\displaystyle \mathcal{L}(\mathbf{m}) = \mathcal{L}_{\text{nec}}(\mathbf{m}) + \mathcal{L}_{\text{suff}}(\mathbf{m}) + \mathcal{L}_{\text{spec}}(\mathbf{m}) + \beta \mathcal{L}_{\text{spar}}(\mathbf{m}) } \label{eq:mask95objective}\tag{2}\] where \(\mathcal{L}_{\text{nec}}\), \(\mathcal{L}_{\text{suff}}\), and \(\mathcal{L}_{\text{spec}}\) encourage the identified parameters to be necessary, sufficient, and specific for \((f^*, t^*)\), respectively, and \(\mathcal{L}_{\text{spar}}\) encourages sparsity. We define each term below.

Necessity. The necessity loss ensures that removing the localized parameters hurts performance on the target pair, establishing that they are necessary for it. Let \(p(f, t, \theta \circ \mathbf{m})\) denote the probability of the first token of the correct answer for task \(t\) on fact \(f\) when the model parameters \(\theta\) are masked by \(\mathbf{m}\). Let \(p(f, t, \theta)\) denote the unmasked model’s probability. Masking zeros out the activations of the selected MLP neurons. For attention heads, it zeros the output vectors before the output projection. Both are equivalent to zeroing the parameters themselves. The loss drives this probability toward chance level \(\tau\) (\(\tau{=}0\) for generation tasks, \(0.25\) for MCQA, \(0.5\) for binary tasks): \[\mathcal{L}_{\text{nec}}(\mathbf{m}) = \text{MSE}\!\bigl(p(f^*, t^*, \theta \circ \mathbf{m}),\; \tau\bigr) \label{eq:nec95loss}\tag{3}\] For discrimination tasks, an additional MSE term encourages the aggregate probability of the distractors to rise to \(1-\tau\), so that ablating the identified parameters changes the model’s answer rather than disrupting its ability to perform the task (see §12 for details). For evaluation, we measure the relative change in accuracy under masking, where predictions that differ from the target only in formatting are not penalized (see §12.4.0.2).

Sufficiency. The necessity loss ensures that the localized components are needed to retrieve the fact on the target task. The sufficiency loss ensures that they are also sufficient, requiring that they carry enough information to retrieve the fact even when the prompt is corrupted. Following the approach of [15], we corrupt the prompt by replacing the subject entity with uninformative placeholder tokens (e.g., France \(\to\) xx), removing the part that identifies the fact. We then run two forward passes: (i) a pass on the original prompt, caching the activations of the localized components; (ii) a pass on the corrupted prompt, in which the cached activations replace the corrupted ones at the localized components. As in necessity, this intervention targets the activations of MLP neurons and the output vectors of attention heads before the output projection. The loss encourages the patched model’s probability of the correct answer on the corrupted prompt to match the unintervened model’s probability on the original prompt: \[\mathcal{L}_{\text{suff}}(\mathbf{m}) = \text{MSE}\!\bigl(\tilde{p}(f^*, t^*, \theta \circ \mathbf{m}),\; p(f^*, t^*, \theta)\bigr) \label{eq:suff95loss}\tag{4}\] where \(\tilde{p}(f^*, t^*, \theta \circ \mathbf{m})\) denotes the patched model’s probability on the corrupted prompt. For evaluation, we report the reconstruction rate: the fraction of the accuracy lost to corruption that is recovered by patching the localized components’ activations (see §12 for the formal definition).

Specificity. Intervening on the localized parameters should not affect the model’s performance on the same fact under other tasks, or on other facts under the same task. To this end, we add a specificity term to the necessity loss, penalizing interference with non-target pairs: \[{!}{\displaystyle \begin{align} \mathcal{L}_{\text{spec}}^{\text{nec}}(\mathbf{m}) &= \underbrace{\mathbb{E}_{f' \neq f^*}\!\Big[\text{MSE}\!\bigl(p(f', t^*, \theta \circ \mathbf{m}),\; p(f', t^*, \theta)\bigr)\Big]}_{\text{other facts, same task}} \\ &\quad + \underbrace{\sum_{t' \neq t^*} \text{MSE}\!\bigl(p(f^*, t', \theta \circ \mathbf{m}),\; p(f^*, t', \theta)\bigr)}_{\text{same fact, other tasks}}\end{align} } \label{eq:spec95nec95loss}\tag{5}\] The same specificity constraint also applies to sufficiency, requiring that patching the identified components’ activations into corrupted prompts for non-target pairs will not recover performance. See §12 for the full loss term. Multi-hop tasks extend the target relation with an additional hop (e.g., landmark \(\to\) country becomes landmark \(\to\) country \(\to\) language). To ensure the mask does not target the added hop, we add a control chain sharing it (e.g., capital \(\to\) country \(\to\) language) to the retention pool. 3 illustrates the necessity, sufficiency and specificity criteria for an example fact.

Sparsity. The mask should be as sparse as possible. We apply an L1 penalty to \(\mathbf{1}-\mathbf{m}\) (the indicator of selected components), normalized by the total number of components, weighted by \(\beta=10.0\). See §12 for implementation details.

3.1.0.2 Mask training and evaluation

For each fact, masks for its different tasks are trained sequentially in a random order. Components selected by earlier masks are excluded from subsequent masks, producing fully disjoint masks across tasks. Since each (fact,task) pair has multiple prompt paraphrases (see §2), all loss terms average over them. We evaluate the learned masks on necessity, sufficiency and specificity using held-out prompt paraphrases (5 training, 2 evaluation per task). Additionally, the pool of other facts used to evaluate same-task specificity is split into 75%/25% train/evaluation, resampled for each target fact. We average over prompt paraphrases to obtain a per-fact accuracy score, then report the mean and std across facts.

3.1.0.3 Results

Across all datasets and models, we observe that individual (fact, task) pairs are supported by distinct, task-specific parameter subsets. 4 presents representative results for the (country, official language, language) dataset on OLMo-2-7B IT (for full results see §12.5). Zero-ablating the identified components causes a large performance drop on the targeted (fact, task) pair (diagonal cells: 29%–89% relative decrease), while performance on the same fact evaluated on other tasks (off-diagonal columns) and on other facts evaluated on the same task (bottom row) remains largely unaffected (all cells \(\leq\)​8%). This confirms that the identified subsets are both necessary for and specific to individual (fact, task) pairs. In terms of sufficiency, for the same dataset and model, patching the identified components’ activations into a corrupted prompt achieves high recovery on the targeted pair (69%–102% reconstruction rate); recovery rates for both the same fact on other tasks and other facts on the same task remain small or slightly negative. The pattern is consistent across other combinations.

Together with the behavioral results, these findings suggest that LMs maintain task-specific parametric encodings of individual facts, instead of drawing from a shared, task-invariant store.

4 Quantifying cross-task entanglement↩︎

Our previous results show that factual knowledge is often encoded in a task-specific manner. Yet, certain tasks show some degree of overlap; ablating the encoding for one task causes collateral damage on some tasks but not on others, and pairwise co-emergence rates differ between task pairs. For example, facts acquired via FiTB reliably co-emerge in OpenQA, but facts acquired via MCQA show late or absent acquisition on Completion. We refer to this overlap as cross-task entanglement, and introduce metrics that quantify it from both the behavioral and parametric perspectives. Our results show that task format is a dominant predictor of such entanglement, with discrimination tasks being consistently more entangled than generation tasks.

4.1 Methodology↩︎

4.1.0.1 Behavioral analysis

We break down the co-emergence rates from the behavioral experiment (§2) into directional task pairwise co-emergence rates. For each ordered pair of tasks (source \(s\), target \(t\)), we measure what fraction of facts that emerged on \(s\) co-emerge on \(t\) by the expected step. This reveals asymmetries in co-emergence between task pairs. Implementation details are in §10.

4.1.0.2 Parameter-level entanglement metric

Our parametric experiment (§3) localizes for each (fact, task) pair a sparse parametric encoding necessary and sufficient for the model’s performance on that pair. As 4 shows, ablating one (fact, task) encoding can degrade other pairs as well. The specificity penalty (Eq. 5 ) limits this collateral damage, but for some (fact, task) pairs, it constrains how fully the mask can suppress its target (Eq. 3 ).

We define an entanglement score \(\operatorname{Ent}(f,t)\) that measures, for a single (fact, task) encoding, how cleanly it can be ablated without affecting other (fact, task) pairs. This metric collapses each row of the necessity heatmap into a single number. Concretely, for each pair \((f, t)\), we ablate its identified parameters and measure three quantities: (a) target drop \(\Delta_{\text{target}}(f, t)\): how much the ablation degrades performance on the targeted pair, (b) collateral change \(\Delta_{\text{coll}}(f', t)\): the effect on other facts \(f' \neq f\) on the same task, and (c) collateral change \(\Delta_{\text{coll}}(f, t')\): the effect on the same fact under other tasks \(t' \neq t\). The score \(\operatorname{Ent}(f,t)\) averages these: \[\begin{array}{r@{}l} {!}{\displaystyle \frac{1}{3}\Bigg[\;\bigl(1 - \Delta_{\text{target}}(f,t)\bigr) + \frac{1}{|\mathcal{F}|-1}\!\sum_{f' \neq f}\!\Delta_{\text{coll}}(f',t) } \\[6pt] {!}{\displaystyle \quad + \frac{1}{|\mathcal{T}|-1}\!\sum_{t' \neq t}\!\Delta_{\text{coll}}(f,t')\;\Bigg] } \end{array} \label{eq:E95score}\tag{6}\] \(\operatorname{Ent}(f,t) = 0\) is achieved when the target drop is maximal and collateral damage is zero on both axes. This means that the encoding is fully necessary and specific for the targeted pair. Higher values indicate greater entanglement. Averaging this score over facts yields a per-task score \(\operatorname{Ent}_{\text{task}}(t) = \frac{1}{|\mathcal{F}|} \sum_{f} \operatorname{Ent}(f,t)\), which we report for all three models and five datasets. Formal definitions are provided in §13.

4.1.0.3 Pairwise task entanglement

To test whether certain task pairs are more entangled than others, we train a separate mask for each directed pair of tasks \((t_A, t_B)\). We use the same objective as in §3, except that the specificity loss penalizes interference only with \(t_B\) (rather than with all other tasks).

4.2 Findings↩︎

We report the key results for the two analyses below, with full per-task tables and heatmaps provided in §10 (behavioral) and §13 (parametric). Pairwise entanglement results are consistent with the aggregative \(\operatorname{Ent}_{\text{task}}\) scores, thus we report them in §13.

4.2.0.1 Generation tasks are less entangled, discrimination tasks are more entangled

The parametric results show that discrimination tasks (MCQA, Verification, Neg MCQA) are markedly more entangled than generation tasks (OpenQA, FiTB, Multi-Hop). Aggregated over 15 model-dataset combinations, the mean \(\operatorname{Ent}_{\text{task}}\) is \(0.21\) for discrimination versus \(0.11\) for generation. Verification and Neg MCQA are the most entangled tasks (\(0.25\) and \(0.24\), respectively), while Multi-Hop-2 is the least (\(0.08\)).

4.2.0.2 Discrimination tasks are weak sources of cross-task co-emergence

Among facts that have emerged on a discrimination task before or alongside a given target task, co-emergence rates are 3%-42% on non-Verification targets, compared to 40%-90% for facts that have emerged on a generation task. Verification shows higher co-emergence rates overall, which we hypothesize is due to its late emergence in training, but even there discrimination tasks are the weakest sources of co-emergence (63%-70%, versus 76%-94% for generation sources).

5 The role of task-specific encodings in chain-of-thought reasoning↩︎

In this section, we expand our analysis to examine how task-specific encodings are utilized during generation, focusing on chain-of-thought (CoT) reasoning [16]. The mechanistic experiment (§3) established that under direct answering, where the model produces an answer without intermediate reasoning, different tasks rely on distinct parameter subsets to retrieve the same fact. Given that reasoning has been shown to unlock factual knowledge inaccessible to direct answering [9], [17], [18], a natural hypothesis is that reasoning draws part of its power from engaging parametric encodings beyond those tied to the evaluation format. If this holds, then CoT should help the model recover performance lost when a task’s localized parameters are ablated, by rerouting through alternative encodings. Moreover, if CoT draws on encodings of other tasks, cross-task collateral damages should be larger than under direct answering. The ablation framework from §3 lets us test both predictions.

5.0.0.1 Experiment

We apply the zero-ablations from §3, but now compare model accuracy under both direct answering and CoT. We exclude the multi-hop tasks, whose two-step chains contain bridging knowledge that may confound cross-task attribution, and use facts that meet a baseline CoT performance threshold on all tasks. We report accuracy averaged across facts before and after ablation, under each setting. To measure cross-task collateral damage, we average the worst-case cross-task drops across facts resulting from encoding ablations. Additional details are in §14.

5.0.0.2 Results

5 presents results for (landmark, in-country, country) on Gemma-2-9B IT; other models and datasets show similar patterns (see §14). Zero-ablating the localized parameters reduces direct accuracy by 20%–72% (varying across tasks), whereas CoT loses only 12%–30%, staying closer to the unablated baseline (5 (a)), confirming the predicted recovery under CoT. When we ablate, per fact, the other task whose encoding most damages each condition, direct accuracy falls by at most 8% while CoT drops by 11%–31% (5 (b)), confirming the predicted increase in cross-task collateral damage under CoT. Together, these results support the hypothesis that CoT routes through multiple task-specific encodings.

6 Related work↩︎

6.0.0.1 LMs as knowledge bases

LM parameters encode vast relational knowledge [2], [3], motivating their view as knowledge bases. Several works have revealed that factual recall is sensitive to query form; paraphrased prompts yield inconsistent predictions for the same facts [19], models trained on “A is B” fail to infer “B is A” [20], and knowledge editing methods often fail to generalize to related queries [21]. We provide evidence that the same fact is supported by different parameters under different tasks, suggesting that such inconsistencies originate in how knowledge is stored rather than accessed.

6.0.0.2 Redundancy in factual encodings

Recent work has suggested that factual knowledge in LMs is not stored in a single location. [7], [22] suggested that different subsets of parameters can encode the same knowledge, and [23] showed that such redundancies contribute to robustness under input perturbations. [24] localized conflicting parametric encodings of the same facts arising from inconsistent pretraining data. [25] demonstrated that facts learned during finetuning are stored redundantly across layers, supporting different multi-hop reasoning tasks. We find that task-specific storage is not limited to finetuned multi-hop knowledge but applies broadly to pretrained facts across diverse task formats.

6.0.0.3 Cross-lingual transfer

Language is another dimension along which the same fact can be queried in different surface forms. [26] showed that models can develop either unified or separate representations of the same facts across languages; [27] traced cross-lingual factual recall across OLMo-7B checkpoints, finding it largely predicted by fact frequency rather than transfer from other languages; [28] leveraged cross-lingual inconsistency to identify knowledge-critical experts in MoE models. Overall, whether facts are stored in shared or language-specific parameters remains largely unanswered. Our work addresses the analogous question along the task dimension, showing that storage is organized, at least in part, by task format.

6.0.0.4 Cognitive parallels

Findings in cognitive science show that access to memory depends on the relationship between how information is encoded at study and how it is later probed [29], [30]. Our work echoes this principle in LMs, showing that what the model knows and how it is asked are intertwined in the parameters. Whether this arises from the format in which facts are encountered during training is an interesting direction for future work.

7 Conclusion and discussion↩︎

We investigate the task-invariance property expected of knowledge bases in LMs, and find that it is largely violated. Behaviorally, facts acquired on one task frequently fail to co-emerge on others during training. Our mechanistic analysis offers an explanation, revealing distinct parameters underlying different tasks for the same facts. Moreover, this separation varies by task format, with encodings for discrimination tasks consistently more entangled with other (fact, task) pairs than those for generation tasks. Our experiments further suggest that CoT reasoning draws on encodings beyond those tied to the evaluation task, offering a mechanistic account for how reasoning unlocks knowledge inaccessible to direct answering.

Our findings have implications for how models are developed and evaluated. Knowledge editing and unlearning methods that target a single task format may leave the fact intact on others, as recently observed by [31], and evaluations that probe only one format provide an incomplete picture of what the model encodes. More fundamentally, building trustworthy and controllable language models may be advanced by training schemes that encourage task-invariant factual encodings. For instance, our finding that CoT reasoning already bridges across task-specific encodings suggests that incorporating intermediate reasoning during training could encourage more task-invariant parametric storage. Exploring training methods that directly promote such mechanisms is an interesting direction for future work.

Limitations↩︎

In the behavioral experiment (§2), we observe training at periodic checkpoints, so the exact step at which a fact emerges on a given task is only approximate. That said, our analysis relies on the relative ordering of when facts emerge across tasks rather than on exact step counts, so this approximation is unlikely to affect our conclusions.

All of our experiments focus on relational knowledge expressible as (subject, relation, object) triplets, and the mechanistic (§3) and CoT (§5) experiments further restrict to facts that meet a high baseline performance threshold on all tasks. Our findings therefore primarily describe well-known relational knowledge. However, these are the facts for which the “knowledge base” analogy is most expected to hold, making them a strong test case for our claims.

It is important to distinguish our localized task-specific encodings from simple redundant encodings, where the same fact is stored in multiple interchangeable locations. Our work reveals that factual storage is organized, at least in part, by task format, but does not characterize how many parameter subsets encode a given fact within or across tasks. Relatedly, our sufficiency results (§3) show that patching a localized subset recovers performance on a (fact,task) pair, but this does not mean the subset “fully” encodes the fact, as other parameters may contribute in ways our masks do not capture. We leave the characterization of this redundancy structure for further study.

Finally, our use of three models (7B–13B) across two model families and five datasets provides evidence that the patterns we observe are not model- or dataset-specific. However, it is unclear how these patterns interact with scale. If greater capacity enables models to allocate increasingly disjoint parameter subsets to different tasks, then larger and more capable models may in fact satisfy the task-invariance property even less. This makes extending the analysis across model scales an important direction for future work.

8 Acknowledgments↩︎

We thank Yoav Gur-Arieh for providing valuable feedback. This research was supported in part by grants 1083/24 and 2247/23 from The Israel Science Foundation.

9 Datasets and prompt construction↩︎

Our experiments rely on five relational datasets (§2). Here we provide additional details on dataset sources, fact filtering criteria, prompt templates, and distractor selection.

9.0.0.1 Relational datasets

We use five relational datasets, each consisting of (subject, relation, object) triplets. (landmark, in-country, country) and (company, HQ-in-city, city) are sourced from [11]; (country, capital-of, city) and (country, official language, language) were obtained using Wikidata SPARQL queries [10]. Countries with more than one official language in the (country, official language, language) dataset were filtered out. All the datasets are in English.

Three of the datasets include multi-hop reasoning tasks, each paired with a control task that shares one hop of the multi-hop chain with the target relation. The control is used in the specificity loss to ensure the mask targets the relation rather than the shared hop. 1 presents the multi-hop relations and their controls. For OLMo-2-7B IT, we use a variant of the (person, plays-instrument, instrument) dataset with uncapitalized object names (e.g., guitar rather than Guitar), since the baseline unintervened performance of the model is substantially better on the uncapitalized version.

9.0.0.2 Multi-hop intermediate entities selection

The multi-hop chains we use have (landmark, in-country, country) as the first hop. Since multiple landmarks exist per country in the dataset, we selected one for each country using a model-based procedure with Gemma-3-1B IT [32]. For the (landmark, in-country, country) dataset, we evaluated each candidate landmark on the Multi-Hop-1 prompt paraphrases and selected the landmark with highest mean probability assigned by the model to the correct answer. For datasets with Multi-Hop-2 tasks ((country, capital-of, city), (country, official language, language)), we evaluated each candidate landmark on both the main and control Multi-Hop-2 paraphrases and selected the landmark with the highest average probability across the two.

9.0.0.3 Prompt templates

For each (dataset, task) pair, we curated 10 prompt paraphrases, using LLM suggestions as a starting point. The same base paraphrases are shared between MCQA and OpenQA, and between FiTB and Completion. The prompts of all tasks aside from Completion begin with a task instruction (e.g., “Answer the following question.”), include a formatting guideline (“Your response should be formatted as: ‘Answer: {your answer}’ ”), and end with the task-specific query. The Completion task does not include instructions or guidelines, since it evaluates next-token prediction on plain sentences. 6 shows representative prompts for different tasks, all for the fact (France, capital-of, Paris) from the (country, capital-of, city) dataset.

9.0.0.4 Distractors selection

Distractor answers for MCQA (3 incorrect choices), Neg MCQA (1 incorrect choice), and Verification (the object in the false statement) are sampled uniformly from the full set of objects in the dataset before any filtering.

None

10 Behavioral experiment: additional details↩︎

In §2 we tested the co-emergence hypothesis by tracking facts acquisition across tasks over training checkpoints. Here, we provide additional implementation details and the full co-emergence rates.

10.1 Additional implementation details↩︎

10.1.0.1 Checkpoints

The 105 checkpoints span three pretraining stages and three post-training models: 100 stage 1 (pretraining) checkpoints, selected at approximately uniform intervals (every \({\sim}14\)k training steps); the final checkpoints of stage 2 (midtraining) and stage 3 (long context); the “main” checkpoints of three post-training models: SFT, DPO and Instruct.

10.1.0.2 Prompt formatting

For all tasks except Completion we append an answer prefix. The prefix is “Answer: ” (with trailing space) for the multiple-choice tasks (MCQA, Neg MCQA) and “Answer:” (without trailing space) for all others. On pretraining checkpoints the prefix is appended to the prompt after a newline. On post-training checkpoints the prompt is first wrapped in the model’s default chat template, and the prefix is appended directly afterward.

10.1.0.3 Directional co-emergence rates: implementation details

The analysis in §2 attributes each fact to its earliest source across all tasks other than \(t\), through \(E(f,\bar{t})\). To study task pairwise co-emergence rates, we fix an ordered pair of tasks (source \(s\), target \(t\)) and replace \(E(f,\bar{t})\) with \(E(f,s)\). We observe every fact that emerges on \(s\) no later than on \(t\) (\(E(f,s) \leq E(f,t)\)), and count the observation as consistent when \(E(f, t) \leq \max\!\bigl(E(f, s),\, E(\cdot, t)\bigr)\). The directional co-emergence rate of \((s,t)\) is the fraction of these that are consistent, with the same liveness exclusions as in §2.

10.2 Results↩︎

10.2.0.1 Task emergence

7 shows per-task histograms of fact emergence steps across training. Generation tasks become competent early in training, with Completion and FiTB emerging at 70k steps, and OpenQA at 85k steps. Discrimination tasks emerge later. MCQA emerges at 99k steps, while Neg MCQA and Verification emerge much later, at 868k steps and at the long-context checkpoint (pretraining stage 3), respectively.

10.2.0.2 Skipped pairs

Of the 1,380 (fact, task) pairs, 349 are excluded, leaving 1,031 evaluated. Of the excluded pairs, 174 were dropped because \(t\) was the task on which the fact first emerged, 113 because the target task was no longer competent at the expected step (RHS of Eq. 1 ), and 62 because the fact was no longer retrieved on any other task.

10.2.0.3 Per-task co-emergence rates

2 breaks the co-emergence rates down by task (for each target task \(t\), the fraction of evaluated facts that co-emerge on \(t\) by the expected step). Generation tasks generally have lower rates, with Completion at \(31.9\%\), FiTB at \(47.4\%\), and OpenQA at \(49.7\%\). Discrimination tasks generally have higher rates, with MCQA at \(72.3\%\) and Verification at \(65.3\%\). Neg MCQA is the exception at \(41.9\%\).

10.2.0.4 Directional co-emergence rates

8 presents the full directional pairwise co-emergence rates. Generation tasks are reliable sources of co-emergence, with facts emerging on Completion co-emerging on other tasks at rates of \(62\%\)-\(94\%\), and OpenQA and FiTB showing similar patterns (\(45\%\)-\(84\%\) and \(40\%\)-\(76\%\), respectively). Discrimination tasks are weaker sources, with facts acquired via MCQA or Neg MCQA co-emerging on tasks other than Verification only \(3\%\)-\(42\%\) of the time. The exception is Verification as a target, which shows high co-emergence rates regardless of the source task. Generally similar directional structure holds under looser and stricter thresholds. 9 shows the directional co-emergence rates at \(\theta = 0.4\) and \(\theta = 0.8\).

11 Fact-task interaction test: additional details↩︎

In §2 we tested the task-invariant store hypothesis by formalizing it as conditional independence between facts and tasks, and found that the interaction is significant. Here we give additional implementation details and results.

11.0.0.1 Hypothesis and model

Task-invariance predicts that \(P(\text{correct}\mid f,t)=P(\text{correct}\mid f)\,P(\text{correct}\mid t)\), which corresponds to an additive model in log-probability space. For each (fact, task) cell we take the log-probability of the correct answer’s first token, \(y_{f,t,p}\), and fit a two-way ANOVA \[y_{f,t,p} = \mu + \alpha_f + \beta_t + \gamma_{f,t} + \varepsilon_{f,t,p},\] where \(p\) denotes prompt paraphrases, that serve as replications within each cell, and \(\mu\) is the grand mean (the average log-probability across all cells). Task-invariance corresponds to \(\gamma_{f,t}=0\) (the no-interaction model). We test \(\gamma_{f,t}\) with an \(F\)-test against the within-cell (paraphrase) variance, and report each term’s effect size as its share of the total variance, \(\eta^2=\mathrm{SS}/\mathrm{SS}_{\text{total}}\).

11.0.0.2 Data and filtering

We use the same facts, tasks, checkpoints, and filtering as the behavioral experiment (§2): 230 facts (46 per dataset), six tasks on the base-model checkpoints and five on the post-training checkpoints (Completion dropped, as it does not fit naturally within chat templates), 10 paraphrases per (fact, task) cell, across the 105 checkpoints. For MCQA and Neg MCQA we average over the rotations of the correct answer’s position, and for Verification we average over the true and false statements, so each paraphrase contributes one observation.

11.0.0.3 Experimental setting

We run the same ANOVA and \(F\)-test in three settings.

1. Per-checkpoint, unnormalized, on the raw correct-answer probability, where different chance levels across tasks are absorbed by \(\beta_t\).

2. Per-checkpoint, normalized, on the chance-normalized probability (subtract the task’s chance level and rescale to \([0,1]\), as in §2).

3. Global (stage 1), a single ANOVA on the chance-normalized probability, pooled over all 100 pretraining stage-1 checkpoints (before midtraining and long-context), with the noise estimated within each (checkpoint, fact, task) cell and \(\gamma_{f,t}\) is the checkpoint-averaged interaction. This asks whether a consistent interaction persists across pretraining.

We add a small constant (\(\epsilon=1e-7\)) to every probability before taking the log.

11.0.0.4 Results

The (fact,task) interaction is significant at every checkpoint (\(p\approx0\)); 3 gives the normalized decomposition across training. Early in pretraining the variance is dominated by the task main effect (\(0.84\) at step \({\sim}14\)k), but as facts are learned and performance across tasks improves, this term drops (to \(0.03\) in the final model) while the fact, interaction, and noise shares rise; the interaction climbs (unevenly) from \({\approx}0.04\) to \(0.10\) across stage 1 and reaches \(0.21\)-\(0.23\) in the post-training models. On unnormalized probabilities the interaction also accounts for a large share of the variance (\(0.41\) in the final model). In the global (stage-1) test, the interaction is again significant (\(p\approx0\)) with \(\eta^2=0.023\) of the pooled total. Overall, these results confirm that the data does not support a task-invariant model.

12 Mechanistic analysis: additional details↩︎

In §3 we presented the localization framework used to identify parametric encodings of (fact,task) pairs. Here we provide additional details on filtering, mask optimization, and evaluation metrics, as well as the complete necessity and sufficiency results for all models and datasets.

12.1 Fact filtering and paraphrase selection↩︎

12.1.0.1 Paraphrase filtering and splitting

Before training masks, we evaluate the model’s performance across all 10 paraphrases for every (dataset, task) pair and discard the 3 templates for which the model’s performance is the lowest. The remaining 7 paraphrases are split into 5 training and 2 evaluation templates. This split is per (model, dataset) pair and is fixed across all facts for this pair.

12.1.0.2 Fact filtering

To ensure we target facts the model can retrieve for all tasks, we filtered out facts for which the model’s performance is below a task-specific threshold in any task. The thresholds are: 0.85 for MCQA, Neg MCQA, and Verification; 0.75 for OpenQA, FiTB, and Multi-Hop tasks. 4 lists the tasks and the number of facts retained per dataset and model after filtering.

12.2 Mask optimization details↩︎

12.2.0.1 Hook placements

Our masks target attention heads and MLP neurons. Interventions are implemented via PyTorch forward hooks [33] on HuggingFace Transformers models [34]. To mask individual attention heads, we register a forward pre-hook on the output projection \(W_O\) that multiplies each head’s \(d_{\text{head}}\)-dimensional slice by the corresponding scalar mask value (\(0\) or \(1\)). Recall that gated MLPs [35] use three weight matrices. Given input \(\mathbf{x}\), the MLP computes an intermediate representation \(\mathbf{h} = \text{act}(W_{\text{gate}}\,\mathbf{x}) \odot W_{\text{up}}\,\mathbf{x} \in \mathbb{R}^{d_{\text{mlp}}}\), then projects back via \(W_{\text{down}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{mlp}}}\): \(\text{MLP}(\mathbf{x}) = W_{\text{down}}\,\mathbf{h}\). To mask individual MLP neurons, we register a forward pre-hook on \(W_{\text{down}}\) that multiplies each entry of \(\mathbf{h}\) independently by its corresponding mask value. Since zeroing a factor zeros the product, this is equivalent to zeroing \(\text{act}(W_{\text{gate}}\,\mathbf{x})\) at the corresponding entries, which is the formulation we use at evaluation.

12.2.0.2 Sparsity objective

In practice, we optimize separate sub-masks for attention heads (\(\mathbf{m}_H\)) and MLP neurons (\(\mathbf{m}_N\)), each with its own normalized L1 penalty term. This is because the number of neurons \(|N|\) is roughly two orders of magnitude larger than the number of attention heads \(|H|\). The full sparsity term in 2 is therefore: \[\mathcal{L}_{\text{spar}}(\mathbf{m}) = \frac{1}{|H|}\|\mathbf{1}-\mathbf{m}_H\|_1 + \frac{1}{|N|}\|\mathbf{1}-\mathbf{m}_N\|_1\]

12.2.0.3 Necessity loss for discrimination tasks

For discrimination tasks (MCQA, Neg MCQA, Verification), 3 includes an additional MSE term that drives the aggregated probability of the distractors up to \(1-\tau\), so that the ablation changes the model’s answer rather than disrupting its ability to perform the task.

12.2.0.4 Sufficiency specificity

The specificity loss contains a sufficiency term that mirrors the necessity specificity term of 5 . Let \(\tilde{p}(f,t,\theta)\) denote the model’s probability on the corrupted prompt without any intervention. Patching the activations of the localized components should not restore performance on the corrupted prompt for non-target pairs: \[{!}{\displaystyle \begin{align} \mathcal{L}_{\text{spec}}^{\text{suff}}(\mathbf{m}) &= \underbrace{\mathbb{E}_{f' \neq f^*}\!\Big[\text{MSE}\!\bigl(\tilde{p}(f', t^*, \theta \circ \mathbf{m}),\; \tilde{p}(f', t^*, \theta)\bigr)\Big]}_{\text{other facts, same task}} \\ &\quad + \underbrace{\sum_{t' \neq t^*} \text{MSE}\!\bigl(\tilde{p}(f^*, t', \theta \circ \mathbf{m}),\; \tilde{p}(f^*, t', \theta)\bigr)}_{\text{same fact, other tasks}} \end{align} } \label{eq:spec95suff95loss}\tag{7}\]

12.2.0.5 Optimization hyperparameters

All logits are initialized to \(0\) (sigmoid value \(0.5\)). Masks are trained for 2,500 steps using Adam with a learning rate of \(0.1\), a mini-batch size of \(8\), and an L1 penalty weight of \(\beta = 10.0\).

12.2.0.6 Per-step sampling

For the necessity specificity term on non-target facts on the target task, at every step we sample a mini-batch of those facts’ paraphrases without replacement. For the sufficiency terms, we sample paraphrases with replacement, corrupting each sampled prompt with a randomly drawn placeholder.

12.3 Experimental Protocol↩︎

12.3.0.1 Verification task handling

From each verification template, we generated both a true-statement and a false-statement prompt. When Verification is the targeted task, we filter out false-statement prompts for the targeted fact, so that the mask is optimized and evaluated solely on true statements. When Verification serves as a retention task (i.e., not the targeted task), both true and false prompts are used. When the baseline performance of the two modes (true statements vs true and false statements) differs by at least \(0.02\), we mention both numbers in the necessity or sufficiency heatmaps.

12.3.0.2 Sufficiency patching protocol

The Multi-Hop-2 task, and the Multi-Hop-1 Control task are excluded from the sufficiency evaluation because the subject entity of the targeted fact does not appear in the prompt, making the subject-replacement corruption procedure inapplicable.

12.3.0.3 Subject corruption

We replace the subject string of every prompt with repeated copies of a placeholder token. The placeholders pool contains 16 strings: four base characters (x, y, z, w), their uppercase variants, and space-prefixed variants of all eight. The number of repetitions is adjusted so that the tokenized length of the replacement exactly matches that of the original subject. During training, a fresh placeholder is sampled per prompt before each mini-batch forward. At evaluation, we sample one placeholder per prompt.

12.4 Evaluation↩︎

12.4.0.1 Metric definitions

For each prompt paraphrase, the model scores \(1\) if its top-1 token matches the first token of the correct answer and \(0\) otherwise, subject to formatting variants described below. We then average across paraphrases per fact and report the mean and std across facts.

12.4.0.2 Tolerance to formatting variants of the target

A strict exact-match criterion would penalize correct answers produced in a slightly different form (e.g., a leading space, different capitalization, or a multi-token split). We therefore apply two post-hoc checks; if either passes, we count the model’s prediction as correct.

Top-1 token variant. We compare the decoded top-1 token \(p\) to the decoded first token of the target \(t_0\), accepting the prediction under whitespace-only, case-only, or combined normalization, as well as when one is a prefix of the other (length \(\geq 2\)), which handles multi-token answers.

Short continuation. We further extract a 3-token continuation (via greedy decoding) and compare each cumulative generated prefix to the full target completion \(t\) using the same normalization procedures described above, plus substring containment (catching outputs like “Hmm, Paris”). For MCQA and Neg MCQA, where the target completion includes a closing parenthesis (e.g., “3)”), we additionally accept the bare digit alone (e.g., “3”). These checks validate that measured drops in the model’s performance on a (fact,task) pair reflect genuine failures rather than tokenization artifacts.

12.4.0.3 Sufficiency metric

We measure the model’s accuracy under three conditions: clean (unmodified prompt), corrupted (subject replaced with placeholder tokens), and patched (corrupted prompt with the encoding’s clean activations stitched in). The reconstruction rate is: \[\frac{\text{acc}_{\text{patched}} - \text{acc}_{\text{corrupted}}}{\text{acc}_{\text{clean}} - \text{acc}_{\text{corrupted}}} \times 100\%.\] A value of \(100\%\) indicates full recovery to clean accuracy, \(0\%\) indicates no improvement over the corrupted baseline, and negative values indicate that patching further degrades accuracy.

12.4.0.4 Cross-experiment aggregation in the sufficiency heatmaps

Reconstruction rates are computed against a clean and a corrupted baseline. The corrupted prompts are resampled in each patching experiment, so for a fixed evaluation task the corrupted accuracy varies slightly depending on which task was patched. To keep reconstruction rates comparable, we pool the clean and corrupted accuracies across all patched tasks that share an evaluation task and use that as the common baseline for that evaluation task. Verification is handled separately: when Verification is itself the patched task it is scored on true statements only, and when it is the evaluation task for another patched task it is scored on the full true+false set, so we pool and normalize the two cases separately.

12.5 Additional results↩︎

12.5.0.1 Full necessity results

The pattern from 4 holds across all models and all datasets ([fig:necessity_olmo_7b,fig:necessity_olmo_13b,fig:necessity_gemma]). Ablating the (fact, task) parametric encoding reduces the model’s performance on the targeted pair (diagonal), while off-diagonal cells (describing the model’s performance on the same fact for other tasks) and the bottom row (describing the model’s performance on other facts on the targeted task) stay near the baseline. The magnitude of the drops varies by task, with generative tasks (OpenQA, FiTB, Multi-Hop) generally showing the largest drops, while Neg MCQA drops are typically the smallest yet still notable. In §4 we analyze this further.

12.5.0.2 Full sufficiency results

Sufficiency results demonstrate the same task-specific pattern across all three models and five datasets ([fig:sufficiency_gemma,fig:sufficiency_olmo_13b,fig:sufficiency_olmo_7b]). Patching the localized (fact,task) components’ activations into the model’s run on a corrupted prompt recovers performance primarily on the (fact, task) diagonal, with small recovery for the target fact on other tasks, or other facts on the target task.

13 Entanglement analysis: additional details↩︎

In §4 we quantified the degree of cross-task entanglement between parametric encodings of different (fact,task) pairs. Here, we provide the formal metric definitions, complete per-task entanglement scores, and task pairwise entanglement heatmaps.

13.0.0.1 Entanglement metric details

The model performance on a (fact,task) pair is measured via variant-tolerant accuracy (§12.4.0.2), chance-normalized as \(a' = (a - c)/(1 - c)\) with chance levels \(c = 0.25\) for MCQA, \(c = 0.5\) for Verification and Neg MCQA, and \(c = 0\) for all other tasks. We define two types of relative change. The target drop \(\Delta_{\text{target}}(f, t)\) measures how much ablation of the (fact,task) parametric encoding degrades the performance on the targeted pair, clamped so that performance increases receive no credit and drops beyond the baseline are capped: \[\min\!\left(\max\!\left(\frac{|a'_{\text{before}}| - |a'_{\text{after}}|}{|a'_{\text{before}}|},\; 0\right),\; 1\right) \label{eq:delta95target}\tag{8}\] The collateral change \(\Delta_{\text{coll}}(f, t)\) captures any perturbation to a non-targeted pair, capped at \(1\): \[\min\!\left(\frac{|a'_{\text{after}} - a'_{\text{before}}|}{|a'_{\text{before}}|},\; 1\right) \label{eq:delta95coll}\tag{9}\] The sum over other facts on the same task in 6 ranges over the held-out evaluation split (see §3). The resulting per-task scores \(\operatorname{Ent}_{\text{task}}\) for all (model,dataset) pairs are reported in 5.

4pt

13.0.0.2 Pairwise entanglement

To examine whether specific task pairs are more entangled than others, we train separate masks for each directed pair of tasks \((t_A, t_B)\) and compute a pairwise entanglement score \(\operatorname{Ent}(t_A \!\to\! t_B)\), averaged across facts. This is 6 with the cross-task sum reduced to the single term \(t' = t_B\). Since this requires training \(|\mathcal{T}|^2\) masks per fact, compared to \(|\mathcal{T}|\) masks in §3, we limit this analysis to OLMo-2-7B IT on the (country, official language, language) and (country, capital-of, city) datasets. 16 presents the resulting heatmaps. Row means demonstrate another instance of the generation-discrimination split: ablating generation-task encodings causes modest collateral damage across evaluated tasks (\(\mu = 0.06\)–\(0.12\)), while ablating discrimination-task encodings produces broader collateral damage (\(\mu = 0.11\)–\(0.25\)).

14 The role of task-specific encodings in CoT reasoning: additional details↩︎

In §5 we tested whether chain-of-thought (CoT) reasoning engages task-specific encodings beyond those tied to the evaluation task. Here, we provide the prompt construction procedure, filtering criteria, and results for all models and datasets.

14.1 Additional implementation details↩︎

14.1.0.1 CoT prompt construction

For the prompts in the mechanistic analysis (§3), we use an instruction that ends with “Your response should be formatted as: ‘Answer: {your answer}’.” For the CoT evaluation, we replace this with “Before answering, think step by step. Your response should be formatted as: ‘Reasoning: {your reasoning}. Answer: {your final answer}’.” After applying the model’s chat template to the prompt, we augment the prompt with the string “Reasoning:”.

14.1.0.2 Reasoning generation

For each prompt we generate a reasoning trace by greedy decoding with at most 200 new tokens, truncated at the first generated “Answer:” marker. We place the resulting trace in the assistant turn and append the answer prefix. We then score the probability of the first token of the target answer at the end of the prefix, as in the direct-answering condition. This also allows evaluating generations that never produce an answer marker.

14.1.0.3 Ablations

For each (fact, task) pair, we reuse its localized mask and zero-ablate the identified components, as described in §3.

14.1.0.4 Fact filtering

We use facts whose post-CoT accuracy meets or exceeds the per-task threshold from §9 on every task, namely 0.85 for MCQA, Neg MCQA, and Verification, and 0.75 for OpenQA and FiTB. 6 reports the counts.

14.1.0.5 Evaluation

From each ablation we read two quantities, both reported as mean accuracy across facts (using the formatting tolerance from §12.4.0.2), under direct answering and CoT: (i) the accuracy drop on the ablated task itself (same-task effect), which is meant to test whether CoT recovers what direct answering loses; and (ii) for each fact, the accuracy drop caused by the other task’s encoding that most damages each condition (measured via accuracy drop; cross-task effect), testing whether CoT suffers more collateral damage than direct answering. In the cross-task panels, since CoT accuracy without ablation exceeds \(0.99\) for every model and dataset, a separate CoT no-ablation bar is not shown.

14.2 Additional results↩︎

[fig:cot_recovery_olmo_7b,fig:cot_recovery_olmo_13b,fig:cot_recovery_gemma_9b] present the direct-versus-CoT results across all datasets for OLMo-2-7B IT, OLMo-2-13B IT, and Gemma-2-9B IT, and [fig:cot_heatmap_olmo_7b,fig:cot_heatmap_olmo_13b,fig:cot_heatmap_gemma_9b] present the corresponding per-task ablation heatmaps. The pattern holds throughout. Direct answering drops significantly when the evaluation task’s own parameters are ablated while CoT stays close to the unablated baseline, and direct answering is less affected than CoT by another task’s ablation.

15 Resources and packages↩︎

Our experiments use models and code from HuggingFace Transformers [34]. In the (fact, task) interaction analysis (§11) we used SciPy [36] for the \(F\)-test. All experiments requiring GPU were run on a single 256GB AMD MI325X GPU. In the mechanistic experiment (§3), we trained masks for different facts in parallel (up to two facts at a time on a single GPU). Training masks for one fact takes approximately 10 hours, yielding an effective rate of \({\approx}\)​5 hours per fact. Across all 437 target facts, we estimate a total of \({\approx}\)​2,200 GPU hours. The remaining experiments are negligible in comparison.

References↩︎

---

1. We repeated the analysis with \(\theta = 0.4\) and \(\theta = 0.8\) and observed similar trends.↩︎

2. The downsampling in §2 was needed for cross-dataset aggregation; here we analyze each dataset separately. Completion evaluates next-token prediction on plain sentences (e.g., The capital city of France is), incompatible with Instruct models’ chat-template format. Multi-hop tasks are excluded from the behavioral experiment since their bridging entities complicate co-emergence tracking.↩︎