1 Introduction↩︎

Activation-dark source selection. Which source data should adapt a shared-vocabulary LLM for a target string domain such as SMILES, proteins, or genomics? This is the operational role of finite-label, label-aware transferability scores such as LogME/LEEP in small-output settings, but their standard forms are not direct plug-ins at \(K\!\sim\!10^5\): they require label maps or task-specific class statistics rather than a streaming estimate of joint activation–error geometry. The hard regime is activation dark: tasks share tokens and frozen activations, yet label-conditioned errors point in different update directions.

Transfer is geometry in update space. Transfer learning is a counterfactual question: given a pretrained model, how will its parameters move when optimized for a new objective? Despite being counterfactual, transfer prediction often uses representation-similarity scores (e.g., CKA, RSA, and SVCCA) computed from frozen activations on a probe set. These metrics implicitly assume that models (or tasks) that “see” the world similarly will also update similarly. Our results show that this assumption can fail in a head-level setting unless one makes additional assumptions about error geometry or label structure.

FisherSketch: a telescope for update geometry at vocabulary scale. We study transfer within shared-parameter families and quantify update similarity using Fisher alignment. The central observation is that head Fisher alignment is an ordinary cosine after the right lift: Theorem 2 rewrites it as the cosine between kernel mean embeddings of joint \((a,e)\) features. This turns a Fisher-matrix comparison into a distributional embedding problem while retaining activation geometry, error geometry, and their coupling. At \(K{=}128{,}256\), however, even the error second moment \(\Gamma_e := \mathbb{E}[ee^\top]\) costs \(\approx 61\) GiB per task and naive error projections cost \(O(mK)\) per sample. FisherSketch estimates the product-kernel cosine directly in one pass, producing a 16 KB signature (at \(m{=}4096\)), with a 192 KB split-sample streaming state (Appendix 26.1). These task signatures give a portable, training-free source-selection object and a diagnostic probe for studying how activation, error, and coupling geometry organize LLM tasks (Appendix 26). This makes vocabulary-scale comparisons practical in the activation-dark setting: activation-only baselines can be competitive on natural shifts but can also be near-constant under fixed-prefix prompts, while FisherSketch remains informative (Section 6.2).

This head-level focus requires a shared output basis. Head update geometry depends on inner products between logit-space error vectors (e.g., \(e_i^\top e_j\)). These inner products are only well-defined when tasks share an output coordinate system. For LLM families, this is most naturally a shared vocabulary (e.g., BERT [1], GPT [2], T5 [3], Mistral [4]); for disjoint label subsets we use an aligned label taxonomy/masked-softmax embedding (Appendix 14). When tasks have fundamentally different output spaces, we recommend full-network or internal-layer comparisons instead of head error geometry.

Evaluation (source selection). We evaluate transfer as LoRA source selection across domains (Section 6). Formal metric definitions and protocol details are in Appendix 10.11. Crucially, in verbalizer shift (fixed-prefix classification), activation-only collapses to random (20% top-1) while FisherSketch achieves 66.7% top-1 (Section 6.2). On Llama-3.1-8B across 100 domains, FisherSketch is competitive with activation-only baselines on natural shifts (Section 6.1). As a scientific-sequence proof of concept, Appendix 26.4 evaluates nine molecular SMILES domains and finds that FisherSketch correlates with cross-domain perplexity reduction (\(\rho_s{=}0.53\), \(p{=}0.006\)), while activation-only similarity is not significant (\(p{=}0.081\)).

Why representation similarity can fail. Representation-only metrics (CKA, RSA, SVCCA) compare activations on a probe set, while head update geometry depends on the joint structure of activations and label-conditioned errors. As a result, representation similarity alone does not determine head Fisher alignment. Theorem 1 formalizes this: there exist tasks with identical probe representations (\(Z_i = Z_j\)) whose head updates are orthogonal, so \(A^{\mathrm{head}}_F(i,j)=0\).

We provide two concrete instantiations. First, in a shared-output masked-softmax embedding used to compare disjoint label sets, non-overlapping label blocks imply disjoint error support, so head gradients are orthogonal even when activations match (Appendix 14). Second, in a dense-vocabulary LLM setting (verbalizer shift, Section 6.2), the prompt prefix is fixed so activations are identical under this setup, yet transfer varies because the label token changes error geometry. A compact ViT-B/16 orthogonality check under the shared-output embedding appears in Appendix 9 (Table 4), and Appendix 26.6 shows the effect across partial label overlap. These results are specific to head-level comparisons and do not preclude other forms of transfer (e.g., if one retrains the head). We do not claim CKA is uninformative in all settings; when conditions are favorable (Appendix 23), it can correlate with full-network Fisher, but such correlation is not guaranteed.

Contributions. We make four contributions. First, we give a non-identifiability witness showing that representation-only metrics cannot determine head Fisher alignment without assumptions on label-conditioned error geometry (Theorem 1). Second, we prove that shared-output head Fisher alignment is a product-kernel cosine between joint activation-error mean embeddings, yielding an exact activation/error/coupling decomposition (Theorem 2). Third, we introduce FisherSketch, a one-pass streaming random-feature estimator that avoids materializing \(K^2\) error moments and produces portable 16 KB task signatures–comparable in size to a small model hash, but encoding transfer-relevant geometry–via SRHT at vocabulary scale. Fourth, we use these signatures as task-geometry probes, separating activation, error, and coupling effects, and validate them on ViT-B/16 and LLM transfer settings, with shared-parameter stress tests in Appendix 11.

Roadmap. Section 2 defines Fisher alignment and the task setting; Section 3 proves the non-identifiability result; Section 4 derives the product-kernel structure; Section 5 introduces FisherSketch; and Section 6 presents experiments.

2 Preliminaries↩︎

We consider tasks indexed by \(k\), each with a distribution \(\mathcal{D}_k\) over \((x,y)\) and a per-example loss \(\ell^{(k)}_\theta(x,y)\). Tasks share architecture and parameterization, and we evaluate task \(k\) at a checkpoint \(\theta_k\) (in the common-checkpoint setting, \(\theta_k \equiv \theta_0\)). For head-level comparisons, tasks must share an output coordinate system; otherwise \(e_i^\top e_j\) and \(\langle \Gamma_{e,i}, \Gamma_{e,j}\rangle\) are not defined (Appendix 14).

For task \(k\), define the per-example gradient \(g^{(k)}_{\theta}(x,y) := \nabla_{\theta}\ell^{(k)}_{\theta}(x,y)\) and the data/empirical Fisher \[\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].\] Fisher alignment between tasks \(i,j\) is the normalized Frobenius inner product \[A_{\mathrm{F}}(i,j) := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}}(\mathcal{F}_i,\mathcal{F}_j), \label{eq:fisher-alignment}\tag{1}\] where \(S_{\mathrm{cov}}(A,B) := \langle A,B\rangle_F / (\|A\|_F\|B\|_F)\) for nonzero symmetric matrices.

Hilbert-space view. The Frobenius inner product makes finite matrices a Hilbert space (Hilbert–Schmidt), so \(S_{\mathrm{cov}}(\cdot,\cdot)\) is a cosine. Section 4.2 gives an RKHS analogue: head Fisher alignment is a cosine between kernel mean embeddings in the RKHS of a product kernel over \((a,e)\). Fisher alignment compares second moments (not mean gradients); estimator details are in Appendix 12.1, with full preliminaries in Appendix 12.

3 Why Representation Metrics Fail↩︎

Representation similarity metrics (CKA, RSA, SVCCA) are widely used for transfer prediction. Without assumptions on error geometry or label mapping, they are blind to head-level error structure (Appendix 19).

Disjoint-label witness. Under the shared-output embedding, tasks that share an encoder but have disjoint label subsets have identical representations, yet orthogonal head gradients. A full proof is in Appendix 22. Proof sketch. Under the shared-output masked-softmax embedding, disjoint label blocks imply error vectors have disjoint support, so head gradients are orthogonal even when representations match. Representation-only metrics observe only activations, and therefore cannot detect this. Scope. We use masked-softmax to give a clean orthogonality witness, not to claim that every transfer benchmark lies in the zero-overlap regime. More broadly, without additional assumptions linking errors to activations, representation covariances do not identify head Fisher alignment. We do not claim that CKA is uninformative in all regimes. When the diagnostics are favorable (Appendix 23), it can correlate with full-network Fisher. However, non-identifiability means it can still fail without assumptions on error geometry.

Dense-vocabulary instantiation. While the masked-softmax construction above provides a clean orthogonality witness, Section 6.2 demonstrates the same phenomenon in a fully dense-vocabulary LLM setting. Fixing only the prompt prefix makes representations identical across verbalizers, yet changing the label token alters error geometry and transfer. FisherSketch predicts this variation, while representation-only metrics collapse.

This yields a non-identifiability result. Linear CKA admits the closed form (Appendix 12.2) \[\label{eq:cka-linear} \mathrm{CKA}(Z_i,Z_j) = \frac{\|\hat{\Sigma}_{ij}\|_F^2}{\|\hat{\Sigma}_i\|_F\,\|\hat{\Sigma}_j\|_F},\tag{2}\] Thus it depends only on \((\hat{\Sigma}_i,\hat{\Sigma}_j,\hat{\Sigma}_{ij})\) and not on error geometry \(\Gamma_e\).

Implication. CKA depends only on centered representation covariances, while Fisher alignment also depends on the error second moment \(\Gamma_e\). The same non-identifiability holds for RSA and SVCCA (Appendices 19, 19.1, and 19.2). Thus representation-only metrics are fundamentally limited for head-level transfer without assumptions on error geometry or label mapping. A correlation with full-network Fisher in some regimes does not guarantee accurate transfer ranking (Appendix 23). Empirically, on ViT-B/16 same-dataset pairs, CKA correlates only modestly with exact head Fisher (Pearson 0.43, Spearman 0.45; Table 5 in the Appendix).

To understand why error geometry matters, we next use the product-kernel structure of head Fisher alignment (Section 4). Representation metrics see only activations, whereas Fisher is a cosine in the joint (activation, error) space.

4 What Fisher Measures↩︎

We show that Fisher alignment has a product-kernel structure: head-layer Fisher alignment is the cosine similarity in a joint (activation, error) feature space, decomposing into representation geometry, error geometry, and a coupling term.

4.1 Layer-wise Kronecker Structure of Gradients↩︎

Affine layers have an exact Kronecker gradient structure, which underlies K-FAC-style approximations [5]–[7]. Let \(a_{l-1}\) be the input activation and \(\delta_l := \partial \ell / \partial z_l\) be the backpropagated error (with \(z_l = W_l a_{l-1}\)).

Since classification heads are typically affine (linear classifiers), the exact Kronecker structure 3 holds without approximation. For the output layer (head), with uncentered activation \(a := a_{L-1}\) entering the head and error \(e := \delta_L\), we have \[g_{\mathrm{head}} = a \otimes e. \label{eq:head-grad-kron}\tag{4}\] We analyze head-layer Fisher with respect to the output projection parameters only.¹ Crucially, this uses the uncentered activation \(a\), not the centered representation \(z = a - \mu\) (Appendix 21). For shared-parameter affine maps (convolutions, token-wise linear maps), gradients are sums of Kronecker terms across positions; Appendix 11 gives the exact kernel/mean-embedding view and provides an unbiased FisherSketch variant. The exact activation–error product identity below applies to rank-one/no-sharing heads with a common output basis; for shared-parameter layers we sketch the corresponding shared-parameter Fisher kernel directly, and separable activation/error products are diagnostics rather than identities.

4.2 Head Fisher Alignment as a Product-Kernel Mean Embedding↩︎

A product-kernel view. Recall that for a linear head, the per-example gradient factorizes as \(g = a \otimes e\) (Lemma 1). For two independent draws \((a,e)\sim\mathcal{D}_i\) and \((a',e')\sim\mathcal{D}_j\), the head Fisher inner product can be written as a kernel expectation: \[\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F = \mathbb{E}\big[(g^\top g')^2\big] = \mathbb{E}\big[(a^\top a')^2 (e^\top e')^2\big].\] This is a degree-2 polynomial kernel in \(g = a \otimes e\) (equivalently degree-4 in \((a,e)\)) that factorizes. Define \(k_a\), \(k_e\), and \(k_{ae}\) by \[\begin{align} k_a(a,a') &:= (a^\top a')^2, \\ k_e(e,e') &:= (e^\top e')^2, \\ k_{ae}((a,e),(a',e')) &:= k_a(a,a')\,k_e(e,e'). \end{align}\] Both \(k_a\) and \(k_e\) are PSD kernels, hence so is their product \(k_{ae}\). Kernel perspectives also appear in wide-network limits and neural tangent kernels [8], [9]. A convenient explicit feature map is obtained by symmetric vectorization:² \[\begin{align} \phi_a(a) &:= \mathrm{vec}_{\mathrm{sym}}(aa^\top), \\ \phi_e(e) &:= \mathrm{vec}_{\mathrm{sym}}(ee^\top), \\ \phi(a,e) &:= \phi_a(a)\otimes \phi_e(e), \end{align}\] so that \(k_{ae}((a,e),(a',e')) = \langle \phi(a,e), \phi(a',e')\rangle\).

For each task \(k\), define the (uncentered) kernel mean embedding \[\mu_{ae,k} := \mathbb{E}_{(a,e)\sim\mathcal{D}_k}\big[\phi(a,e)\big].\] Assume finite fourth moments and nonzero embedding norms, so the cosines below are well-defined; in practice we use \(\varepsilon\)-regularized norms for degenerate cases. We write \(S_{ae}(i,j) := \langle \mu_{ae,i}, \mu_{ae,j}\rangle\) and define \(S_a, S_e, \chi, \rho\) analogously in Appendix 10.

When the joint distribution is separable (\(\mu_{ae} = \mu_a \otimes \mu_e\)), Fisher alignment factors into activation and error cosines. The coupling term \(\rho\) measures deviation from separability.

Three-factor form: rank-1 proxy plus non-separability correction. Define the marginal embeddings \[\begin{align} \mu_{a,k} &:= \mathbb{E}[\phi_a(a)] = \mathrm{vec}_{\mathrm{sym}}(M_{a,k}), \\ \mu_{e,k} &:= \mathbb{E}[\phi_e(e)] = \mathrm{vec}_{\mathrm{sym}}(\Gamma_{e,k}), \end{align}\] where \(M_{a,k}:=\mathbb{E}[aa^\top]\) and \(\Gamma_{e,k}:=\mathbb{E}[ee^\top]\). We use \(S_{\mathrm{cov}}(A,B) := \langle A,B\rangle_F / (\|A\|_F\|B\|_F)\) for symmetric \(A,B\); here \(A\) and \(B\) are uncentered moments (\(M_a\), \(\Gamma_e\)). If the joint embedding were separable (rank-1) in the product space, then \(\mu_{ae,k}=\mu_{a,k}\otimes \mu_{e,k}\). In that case, \[\begin{align} \cos(\mu_{ae,i},\mu_{ae,j}) &= \cos(\mu_{a,i},\mu_{a,j})\,\cos(\mu_{e,i},\mu_{e,j}) \\ &= S_{\mathrm{cov}}(M_{a,i},M_{a,j})\cdot S_{\mathrm{cov}}(\Gamma_{e,i},\Gamma_{e,j}). \end{align}\] In general, the joint embedding need not be separable, so we define the non-separability ratio \[\begin{align} \rho_{ij} &:= \frac{A_{\mathrm{F}}^{\mathrm{head}}(i,j)}{S_{\mathrm{cov}}(M_{a,i},M_{a,j})\cdot S_{\mathrm{cov}}(\Gamma_{e,i},\Gamma_{e,j})} \\ &\quad(\text{well-defined when the denominator is nonzero}), \end{align}\] If \(S_{\mathrm{cov}}(M_{a,i},M_{a,j})=0\) or \(S_{\mathrm{cov}}(\Gamma_{e,i},\Gamma_{e,j})=0\), then \(A_{\mathrm{F}}^{\mathrm{head}}(i,j)=0\) and we set \(\rho_{ij}=1\) by convention. Empirically, we estimate this via \(\hat{\rho}_{ij}\) using sample averages; see Appendix 18. The following identity holds exactly: \[\boxed{ A_{\mathrm{F}}^{\mathrm{head}}(i,j) = S_{\mathrm{cov}}(M_{a,i}, M_{a,j}) \cdot S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}) \cdot \rho_{ij}. } \label{eq:fisher-kronecker}\tag{5}\]

Practical note. Dropping the coupling term \(\rho\) in Eq. 5 yields the separable/Kronecker proxy; this can mis-rank transfer, and on ViT-B/16 the proxy is negatively correlated with exact head Fisher (Appendix 18, Table 5), so we do not rely on it in evaluation.

The non-separability ratio \(\rho_{ij}\) captures how the joint embedding deviates from the rank-1 proxy. It can be adversarial; see Appendix 18.

= by -

4.3 Beyond the Head: Full-Network Diagnostics↩︎

To relate head/block proxies to full-network Fisher alignment, Appendix 23 partitions parameters into \(L\) blocks (e.g., layers) and decomposes each task Fisher as \(F_k = D_k + O_k\), where \(D_k\) contains within-block terms and \(O_k\) contains cross-block terms. For any tasks \(i,j\), normalized Frobenius alignment decomposes exactly as \[\begin{align} A^{\mathrm{full}}_F(i,j) &= w_{\mathrm{blk}}\,A^{\mathrm{blk}}_F(i,j) + w_{\mathrm{off}}\,A^{\mathrm{off}}_F(i,j), \\ w_{\mathrm{blk}} &:= \frac{\|D_i\|_F\|D_j\|_F}{\|F_i\|_F\|F_j\|_F}, \\ w_{\mathrm{off}} &:= \frac{\|O_i\|_F\|O_j\|_F}{\|F_i\|_F\|F_j\|_F}. \end{align}\] Here \(A^{\mathrm{blk}}_F\) is the alignment computed from block-diagonal Fisher blocks and \(A^{\mathrm{off}}_F\) is the alignment computed from off-diagonal blocks. The profile cosine \(c_r := w_{\mathrm{blk}} + w_{\mathrm{off}} \in (0,1]\) measures how similarly the two tasks distribute Fisher energy between block-diagonal and cross-block terms (equivalently, the cosine between \((\|D_i\|_F,\|O_i\|_F)\) and \((\|D_j\|_F,\|O_j\|_F)\)). The off-diagonal discrepancy \(\Delta_{\mathrm{off}} := \big|A^{\mathrm{off}}_F - A^{\mathrm{blk}}_F\big|\) captures whether cross-block interactions behave adversarially relative to the block-diagonal geometry. These quantities yield the deterministic certificate \[\big|A^{\mathrm{full}}_F - A^{\mathrm{blk}}_F\big| \le (1-c_r) + w_{\mathrm{off}}\,\Delta_{\mathrm{off}}. \label{eq:full-to-block-bound}\tag{6}\] Table 13 (Appendix 23) summarizes these diagnostics and the observed full-to-block gaps.

4.3.0.1 How to read the diagnostics.

The certificate fails with low profile cosine or large off-diagonal discrepancy. In our ViT/LLM validations, profile cosines are high (\(c_r \ge 0.90\)), so proxy tightness is typically limited by the off-diagonal discrepancy term; see Table 13 (Appendix 23).

Computational consequence (Hilbert-space/RKHS view). Theorem 2 can be read as a kernel mean embedding identity in an RKHS (i.e., a Hilbert space). Define the product polynomial kernel \(k_{ae}((a,e),(a',e')) := (a^\top a')^2\,(e^\top e')^2\), and let \(\mathcal{H}_{ae}\) be its RKHS with feature map \(\Phi\). Then \(\mu_{ae,k} := \mathbb{E}_{(a,e)\sim \mathcal{D}_k}[\Phi(a,e)] \in \mathcal{H}_{ae}\) and \(A_{\mathrm{F}}^{\mathrm{head}}(i,j) = \langle \mu_{ae,i}, \mu_{ae,j}\rangle_{\mathcal{H}_{ae}} / (\|\mu_{ae,i}\|_{\mathcal{H}_{ae}}\,\|\mu_{ae,j}\|_{\mathcal{H}_{ae}})\). Identifying \(\Phi\) with our explicit polynomial map \(\phi\) recovers the finite-dimensional product feature space statement. FisherSketch exploits this by approximating each \(\mu_{ae,k}\) with an \(m\)-dimensional random feature map \(\Psi_m\), so all \(T^2\) alignments become dot products between compact task signatures. Crucially, we do not need to compute the three factors separately; instead, we estimate the joint embedding directly in the random feature space.

5 FisherSketch: Scalable Estimation↩︎

The product-kernel view suggests a streaming estimator. We fix a random feature map \(\Psi_m\) and estimate \(\tilde{\mu}_k := \mathbb{E}_{(a,e)\sim \mathcal{D}_k}[\Psi_m(a,e)]\). We then compute alignments as dot products in \(\mathbb{R}^m\). The exact embedding \(\mu_{ae,k}\) has dimension \(O(d^2 K^2)\), which is too large to store, so we use factored Random Maclaurin features that exploit the Kronecker structure \(g = a \otimes e\).

This requires only \(O(m(d{+}K))\) operations per sample (rather than \(O(mdK)\)), because projections onto \(a\) and \(e\) are computed separately.

Algorithm (in words). Sample Rademacher projections for \(a\) and \(e\). For each task, stream samples and compute \(\psi_j(a,e)\) for \(j=1,\ldots,m\). Accumulate and normalize these values to obtain a task signature \(\hat{\mu}_k\). Cosines between signatures estimate head Fisher alignment. Pseudocode and implementation details are in Algorithm 3 (Appendix).

Estimation (informal). For dense Random Maclaurin features, the unnormalized kernel inner product is unbiased with variance \(O(1/m)\); SRHT preserves the same first-moment target but uses dependent coordinates, and the normalized cosine is a plug-in estimate. Monte Carlo sampling error decays as \(O(1/\sqrt{n})\) for \(n\) samples per task. We use split-half log-ratio estimation with shared projections (a low-variance within-split estimator; we also report a cross-fit variant), U-statistic diagonal correction, and small-norm clamping for stability. For robust rankings at \(K\!\approx\!10^5\), we further stabilize the coupling term by empirical-Bayes shrinkage in log space: let \(\log\hat{\rho}_{ij} := \tfrac12(\log\hat{\rho}^{A}_{ij}+\log\hat{\rho}^{B}_{ij})\) and \(\mathrm{SE}^2_{ij}:=\tfrac14(\log\hat{\rho}^{A}_{ij}-\log\hat{\rho}^{B}_{ij})^2\); then \(w_{ij}=\tfrac{\tau^2}{\tau^2+\mathrm{SE}^2_{ij}}\) and \(\log\hat{\rho}^{\mathrm{shr}}_{ij}=w_{ij}\log\hat{\rho}_{ij}\) (details in Appendix 10.5).

Scaling and memory. At \(K=128{,}256\), storing \(\Gamma_e\) requires \(\sim61\) GiB per task, whereas FisherSketch stores a \(16\) KiB float32 per-task signature. During streaming, split-half estimation keeps six float64 accumulators per task (A/B splits for \(S_{ae},S_a,S_e\)), which is \(192\) KiB. Fixed projections fit on a single GPU by (i) storing dense Rademacher sign matrices as int8 (two \(m{\times}d\) matrices for activations; \(32\) MiB at \(m{=}d{=}4096\)) and (ii) using an SRHT for the \(K\)-dimensional error (about \(1\) MiB of sign-vectors/indices), implemented with a batched Walsh–Hadamard butterfly on GPU (Appendices 10.8–10.9). This yields per-sample cost \(O(md + K\log K + m)\) in the vocabulary-scale setting. Dense Rademacher and Gaussian projections are used for controlled small-dimensional estimator comparisons; vocabulary-scale LLM error sketches use the SRHT backend on the \(K\)-dimensional error side.

Estimator validation. We validate the estimator on ViT-B/16 using 20 fine-tuned checkpoints across four datasets. We compare sketched estimates with exact computation at moderate \(K\). Spearman correlations are \(0.97 \pm 0.01\) for \(\hat{S}_{ae}\), \(0.94 \pm 0.01\) for \(\hat{\chi}\), and \(0.95 \pm 0.01\) for \(\hat{\rho}\). \(\hat{A}_F\) is lower (\(0.79 \pm 0.06\)) due to ratio normalization. Figure 1 shows accuracy versus sketch dimension, and Table 6 in Appendix 9 shows linear-time scaling. At \(n=2000\) and \(m=4096\), the method achieves an 89\(\times\) speedup. For shared-parameter layers, we validate the corresponding shared-layer kernel rather than the no-sharing head factorization: at \(m=4096\), FisherSketch preserves exact-Fisher rankings with Spearman \(0.997\)–\(0.998\) on ResNet-18 convolutional layers and \(0.994\)–\(0.997\) on ViT-B/16 MLP/QKV layers (Appendix 11, Table 9).

6 Experiments↩︎

We evaluate FisherSketch along three axes: (i) sketch accuracy relative to exact computation for moderate \(K\) (Fig. 1; Table 6 in Appendix 9), (ii) vocabulary-scale source selection, and (iii) ablations that isolate activation vs.error geometry. This section focuses on (ii) and (iii). See Appendices 13, 17, and 27 for experimental details, estimator diagnostics, and additional calibration checks. We also evaluate training-free task retrieval and open-set addition using task signatures (Appendix 26) and a small SMILES scientific-sequence proof of concept (Appendix 26.4).

Evaluation metrics and protocol. We evaluate source selection using log-perplexity improvements under LoRA fine-tuning. Let \(\ell_t^{\mathrm{base}}\) denote the base model’s log-perplexity on target domain \(t\), \(\ell_t^{\mathrm{self}}\) the log-perplexity after training on \(t\) itself, and \(\ell_{s\to t}\) the log-perplexity after training on source \(s\) and evaluating on \(t\). We define normalized transfer as \[U_{s\to t} = \frac{\ell_t^{\mathrm{base}} - \ell_{s\to t}}{\max(\ell_t^{\mathrm{base}} - \ell_t^{\mathrm{self}}, 0.01)}.\] This quantity measures the fraction of the target’s self-training gain recovered by transferring from \(s\); it is negative when transfer hurts. For each target \(t\), the oracle source is \(s^\star(t)=\arg\max_s U_{s\to t}\), and we report Top-1/Top-3 accuracy for identifying \(s^\star(t)\) from the fixed candidate set. Given a predicted source \(s_{\mathrm{pred}}(t)\), define per-target regret \(r_t := U_{s^\star(t)\to t} - U_{s_{\mathrm{pred}}(t)\to t}\) and report Max Regret (a tail-risk metric) \(:= \max_t r_t\). We also report Percent Oracle, the mean over targets of \(U_{s_{\mathrm{pred}}(t)\to t}/U_{s^\star(t)\to t}\). As a simple baseline, PPL similarity scores sources by \(-|\ell_s^{\mathrm{base}} - \ell_t^{\mathrm{base}}|\).

6.1 Vocabulary-Scale Transfer Prediction with Tail-Risk Metrics↩︎

2pt

Table 1 reports the headline vocabulary-scale source selection results; we summarize and then unpack them below. FisherSketch compresses each task from \(\sim\)​61 GiB to a 16 KB float32 signature at vocabulary scale (\(K{=}128{,}256\)); the streaming estimator maintains a 192 KB per-task state for split-sample estimation (Appendix 26.1). On Llama-3.1-8B with 100 domains drawn from eight HF datasets, FisherSketch (SRHT) achieves 45.7% \(\pm\) 5.0 top-1 source selection (\(\approx\)​10.9\(\times\) the random baseline) with max regret 0.119 \(\pm\) 0.013 (Table 1; Appendix [app:vocab-scale-data]). It reaches 98.44% \(\pm\) 0.30 of oracle normalized transfer. The per-target Spearman correlation with normalized transfer is 0.47 \(\pm\) 0.02 (Appendix 10.11). Holding LoRA outcomes fixed, sketch stability across SRHT seeds is high (off-diagonal Spearman \(0.982 \pm 0.003\); Appendix 10.11). The strongest trivially scalable baseline (PPL similarity) reaches 28% top-1 source selection, suggesting that error geometry adds signal beyond perplexity matching. Activation-only geometry is competitive on natural shifts, but Section 6.2 shows fixed-prefix prompt regimes where it collapses and FisherSketch remains informative. As a controlled sanity check, in a small-\(K\) linear-probe sweep (five overlap levels), \(S_{\mathrm{cov}}(\Gamma_e)\) correlates 0.977 with transfer gain, matching LogME (0.976) and exceeding LEEP (0.964; Appendix 23.7). Other label-aware baselines are similarly high because overlap varies monotonically across the sweep (Table ¿tbl:tab:controlled-transfer?; Appendix 23.7).

Ablation: activation vs.error geometry. At 500 samples, error geometry \(S_{\mathrm{cov}}(\Gamma_e)\) reaches 41.3% \(\pm\) 1.7 top-1 with max regret 0.210 \(\pm\) 0.011. Activation geometry performs better: 46.3% \(\pm\) 3.7 top-1 with lower tail risk (max regret 0.114 \(\pm\) 0.006). The diagonal error proxy matches the error-only tail risk (max regret 0.210 \(\pm\) 0.011). FisherSketch (SRHT) remains competitive (45.7% \(\pm\) 5.0 top-1; max regret 0.119 \(\pm\) 0.013) and retains high oracle performance (98.44% \(\pm\) 0.30). Conditions under which an error-only metric can win are in Appendix 24. We report the activation-only marginal \(S_{\mathrm{cov}}(M_a)\) (not probe-centered CKA), computed in the same streaming pass as FisherSketch. Practically, if \(S_{\mathrm{cov}}(M_a)\) is nearly constant across candidate sources (e.g., fixed-prefix prompt regimes), activation-only scores are uninformative for ranking, whereas FisherSketch remains informative via label-conditioned error geometry (Section 6.2). Table 1 should not be read as uniform dominance over all proxy baselines: on these natural shifts, the activation-only marginal has slightly higher top-1 accuracy than FisherSketch, while FisherSketch has competitive tail risk and oracle-normalized transfer. The intended use case is the regime where a marginal signal is misleading or incomplete, and activation, error, and coupling geometry should be considered jointly.

6.2 Identifiability Stress Test: Verbalizer Shift↩︎

Table 1 shows that activation geometry can be competitive on natural domain shifts. We next study verbalizer shift under fixed-prefix prompting, where activation similarity is constant. For each dataset (BoolQ, RTE, CB), we fix an identical prompt prefix ending with The answer is and vary only the single-token verbalizer (e.g., Yes/No, True/False, Correct/Wrong, No/Yes). By causality, the answer-position hidden state depends only on the shared prefix, so \(S_{\mathrm{cov}}(M_a)\approx 1\) for all verbalizer pairs.

4pt

LoRA transfer varies sharply across verbalizers, and FisherSketch predicts this by sketching errors under the full \(K{=}128{,}256\) softmax used to compute \(U_{s\to t}\) (token loss on the designated verbalizer, not a renormalized two-way softmax). Across 9 runs (3 seeds \(\times\) 3 datasets), FisherSketch achieves 66.7% top-1 source selection and 95.7% of oracle normalized transfer (Table 2; Fig. 2), while \(S_{\mathrm{cov}}(M_a)\) reduces to the random baseline. In this controlled regime, the error-only marginal \(S_{\mathrm{cov}}(\Gamma_e)\) is also strong (72.2% top-1; 100% of oracle), as expected from the three-factor identity: when \(S_{\mathrm{cov}}(M_a)\) is constant, the ranking signal must come from label-conditioned error geometry. This is a dense-vocabulary instantiation of Theorem 1. Holding representations fixed does not determine head update geometry; the missing information is label-conditioned error structure, which FisherSketch measures at vocabulary scale. Full experimental details and per-run breakdowns are in Appendix 29. Here the error-only marginal is especially strong and exceeds FisherSketch on top-1, which is consistent with the decomposition: when activation geometry contributes no ranking signal, the error marginal can be sufficient.

Protocol and auditability. We release the fixed candidate sets and sketch sample indices, and enforce disjoint sketch vs.LoRA splits; correlations use the union of evaluated pairs, while top-1 and regret use the uniform 24-candidate protocol (Appendix 10.11). Additional controlled-regime experiments and full-network validation details are in Appendices 23.7–23.9.

7 Discussion↩︎

Perspective. FisherSketch makes the non-identifiability insight operational: it turns label-conditioned head update geometry into a comparable object at vocabulary scale by embedding each task as a random-feature approximation to the product-kernel mean embedding (Theorem 2). In verbalizer shift, this remains informative precisely when representation similarity is constant under fixed-prefix prompts (Section 6.2).

Byproduct. A separable proxy \(\mathrm{FA\text{-}CKA} := \mathrm{CKA} \cdot S_{\mathrm{cov}}(\Gamma_e)\) follows when the joint embedding is approximately rank-1. However, it requires \(K^2\) storage, so we do not use it at vocabulary scale (Appendix 15). Beyond source selection, the same \(\ell_2\)-normalized signatures serve as a task-geometry probe: their activation marginal, error marginal, and coupling ratio ask whether task neighborhoods are representation-driven, error-driven, or activation-dark (Appendices 26.3 and 27.1). They also support nearest-neighbor task retrieval and open-set addition without retraining (Appendices 26 and 26.2; 55.5% top-1 on 700-domain Natural Instructions retrieval).

7.1 Limitations & Broader Applicability↩︎

Our formal results concern head (final-layer) empirical Fisher alignment, where per-example gradients satisfy \(g=a\otimes e\). For deep networks, Appendix 23 provides a full-network decomposition and diagnostics (profile cosine \(c_r\) and off-diagonal discrepancy \(\Delta_{\mathrm{off}}\)), with the exact depth-coupling form in Appendix 23.6; Appendix 10.7 describes a structured-projection variant for sketching full-network FisherSketch. Our ViT and LLM validations satisfy these diagnostics; for new architectures, we recommend reporting \((c_r,\Delta_{\mathrm{off}})\).

Head-level error geometry further requires an aligned output space (shared vocabulary or label taxonomy); otherwise error alignment needs an explicit output mapping (Appendix 14). For heterogeneous outputs, head Fisher is not directly comparable across tasks. Appendix 28 reports internal-layer directional Fisher descriptors as a gradient diagnostic; they are complementary to CKA rather than uniformly stronger in that validation.

We use empirical (data) Fisher at a fixed checkpoint, so results may vary across checkpoints, population Fisher, and adaptation methods. Signatures are checkpoint-conditional but require only one streaming pass and scale linearly in streamed tokens (Appendix 9; Table 6; e.g., 4.7s for a 200-sample domain signature on an A100; Appendix 26.5). The 16 KB figure refers to the stored fp32 descriptor at \(m=4096\); internal shared-token layers require transient \(O(nm)\) sketch-feature state before pooling, e.g., about 8 MiB per task per layer at \(n=500\), \(m=4096\) in fp32. In our heterogeneous-output ViT-B/16 validation, four internal layers across eight models used about 250 MiB of transient sketch-feature state and peaked at 4.19 GiB GPU memory during Fisher collection. The unnormalized random-feature inner-product estimates are unbiased (variance \(O(1/m)\) for fixed pairs), while normalized cosines and rankings can degrade for small \(n\) or noisy domains; shrinkage, proxy calibration, and sketch-dimension behavior are documented in Appendices 10.5, 27.1, and 27.3.

Our vocabulary-scale evaluation is LoRA transfer on 100 domains under a fixed candidate-set protocol; despite disjoint sampling and released candidate sets/sketch indices for auditability, absolute numbers may shift with domain construction, candidate draws, or transfer hyperparameters (Appendix 10.11). We expect strongest applicability in within-family, aligned-output settings (LLMs, subset classification); heterogeneous outputs likely require an output embedding/transport map or full-network/shared-parameter comparisons.

7.2 Related Work↩︎

CKA [10], RSA [11], and SVCCA [12] compare representations. Related representation-similarity analyses and deconfounded variants have been proposed [13]–[15], and we formalize their blindness to error geometry. Even after controlling for function-preserving symmetries, representation similarity can remain non-identifiable for label-conditioned update geometry.

Label-aware transfer metrics such as LEEP and LogME use finite-label information [16], [17]. Appendix 23.7 compares them with \(S_{\mathrm{cov}}(\Gamma_e)\) in a small-\(K\) overlap sweep, and Appendix 25 formalizes LEEP, in a stylized hard-prediction limit, as a diagonal error-geometry proxy; we do not claim that LogME is identical to this Fisher marginal. Operationally, FisherSketch plays the source-selection role of these scores for shared-vocabulary LLMs, but compares joint update geometry instead of small-output evidence or source-model label transfer. Related checkpoint-ranking and transferability scores can scale poorly at \(K{\sim}10^5\) in their exact form [18], [19].

Task2Vec [20] embeds tasks via diagonal Fisher. Information-geometric task distances offer another Fisher-style view [21]. Neural Fisher Kernels apply Fisher-kernel ideas to neural networks for data-level representation extraction and low-rank approximations [22]; our focus is task-level Fisher alignment under a shared output basis and a streaming activation/error sketch at vocabulary scale. Mean-gradient cosine compares the first moment \(\mathbb{E}[g]\), diagonal-Fisher/task-vector similarities keep only diagonal entries of the second moment \(\mathbb{E}[gg^\top]\), and influence-style scores answer pointwise perturbation questions (Appendix 27.2). FisherSketch instead sketches the full head-gradient second-moment kernel mean \(\mathbb{E}[\phi(a,e)]\), so it can retain off-diagonal error geometry and activation-error coupling while remaining vocabulary-scale. It is therefore best read as a vocabulary-scale source-selection analogue of finite-label transferability scoring, not as a claim that Task2Vec, LogME/LEEP, CKA, or influence baselines are uninformative in their native regimes. K-FAC [6] approximates Fisher for optimization, whereas we study cross-task alignment and the coupling term \(\rho\) (Theorem 2); Appendix 16 details the connection.

Broader work on transfer examines what is transferred and when transfer fails [23]–[26]. Multi-task and continual learning study task grouping and gradient interactions [27]–[32], Fisher-based continual-learning regularization (EWC) [33], and sequential fine-tuning order effects [34]. These order-level analyses are complementary to FisherSketch: they study how updates compose over time, while FisherSketch supplies a vocabulary-scale pairwise update-compatibility signal.

8 Conclusion↩︎

Representation-only metrics cannot recover head Fisher alignment without assumptions on error geometry or label mapping (Corollary 1). Head Fisher alignment is a product-kernel cosine with a three-factor decomposition that includes a coupling term \(\rho\) (Theorem 2). FisherSketch makes this alignment tractable at vocabulary scale, compressing each task from \(\sim\)​61 GiB to a 16 KB float32 signature (with a 192 KB per-task streaming state for split-sample estimation); on Llama-3.1-8B it achieves 45.7% \(\pm\) 5.0 top-1 (98.44% \(\pm\) 0.30 of oracle normalized transfer), competitive with activation-only on natural domain shifts and robust when activation geometry collapses (Section 6.2). It thereby serves both as a LogME/LEEP-like training-free source selector in shared-vocabulary activation-dark regimes and as a compact probe for how activation geometry, error geometry, and their coupling organize LLM tasks. Diagnostics connect head alignment to full-network behavior in our ViT/LLM validations (Appendix 23.8, Appendix 23.9), and the shared-output requirement delineates the scope of head-level comparisons.

Impact Statement↩︎

This work makes update-geometry comparisons and task indexing practical at LLM vocabulary scale by compressing each task into a small signature. That enables new control-plane designs for adaptation—automated fine-tuning source selection, interference-aware continual and multi-task learning, and inference-time routing/open-set expansion—grounded in predicted parameter-update compatibility rather than representation similarity. More broadly, we provide a scalable way to estimate whether parameter updates learned from one dataset/task are likely to help or harm another, supporting better transfer decisions before incurring the cost of full fine-tuning. In practical automated adaptation pipelines, this can reduce negative transfer, avoid wasted training runs, and surface interference patterns and rare but severe regressions earlier, improving reliability while saving compute and energy.

A key risk is that automated source selection or routing policies built on these signals could amplify bias or reduce coverage by systematically deprioritizing underrepresented, safety-critical, or long-tail data that appears “misaligned.” Compatibility estimates are also predictive and can be wrong, potentially excluding beneficial sources or creating blind spots. We therefore recommend using update-compatibility signals as one input among others (coverage, fairness, safety evaluation, and domain requirements), and auditing any automated selection/routing mechanism for disparate impact and robustness across minority and edge-case data.

Appendix Guide↩︎

The appendices serve three purposes: they prove the identities used in the main text, document the estimator and implementation choices, and give the validation/provenance details that cannot fit in the nine-page body. The guide below is exhaustive at the section level and points to subsection labels when a result is meant to support a specific main-text claim.

8.0.0.1 Core Fisher identity and estimator.

Appendix 9 defines exact head empirical Fisher alignment, the separable Kronecker proxy, and the exact kernel computation used for validation tables. Appendix 10 gives the linear-time product-kernel mean-embedding view, factored Random Maclaurin sketch, Algorithm 3, empirical-Bayes shrinkage (Appendix 10.5), structured full-network sketching (Appendix 10.7), SRHT implementation details (Appendices 10.8–10.9), and the vocabulary-scale storage/validation setup (Appendix 10.11). Appendix 11 extends the estimator from no-sharing heads to shared-parameter affine maps such as convolutions and token-wise transformer projections.

8.0.0.2 Assumptions, proofs, and representation-metric limits.

Appendix 12 collects Fisher/CKA preliminaries, while Appendix 13 records training, data, and metric details for the controlled vision experiments. Appendix 14 states the shared-output masked-softmax embedding used by the non-identifiability witness. Appendices 15–18 analyze the activation/error/coupling decomposition, its K-FAC relationship, estimator diagnostics, and empirical coupling distributions. Appendices 19–22 cover RSA/SVCCA non-identifiability, perturbation and mean-correction lemmas, and the full proof of Theorem 1.

8.0.0.3 Full-network scope and transfer diagnostics.

Appendix 23 explains how head Fisher relates to block/full-network Fisher, including the exact depth-coupling decomposition (Appendix 23.6) and the controlled CIFAR, ViT-B/16, and LLM validations (Appendices 23.7–23.9). Appendix 24 explains why activation geometry can screen candidates while error geometry ranks them, and Appendix 25 relates LEEP-style scoring to diagonal error geometry.

8.0.0.4 Task signatures, retrieval, and validation diagnostics.

Appendix 26 documents the 16 KB signature storage claim, retrieval protocol/evaluation, Natural Instructions retrieval, molecular-SMILES proof of concept, dynamic open-set addition, and the label-overlap continuum (Appendices 26.1–26.6). Appendix 27 collects additional validation diagnostics: rho/delta calibration, gradient and diagonal-Fisher baselines, and sketch-dimension sweep (Appendices 27.1–27.3). Appendix 28 covers incompatible output heads and the scoped one-step LoRA/internal-layer diagnostic bridge. Appendix 29 gives the dense-vocabulary verbalizer-shift experiment, including SRHT backend configuration and compute details.

Captions and subsection introductions state whether each quantity is exact head Fisher, a separable proxy, FisherSketch/SRHT, a controlled dense/Gaussian sketch, a parameter-subsampled full-network diagnostic, or an internal-layer directional descriptor.

9 Fisher Alignment: Exact vs.Proxy Computation↩︎

9.1 Definitions↩︎

For head-layer parameters, the per-example gradient can be written in the exact Kronecker form: \[g_i^{\mathrm{head}} = a_i \otimes e_i,\] where \(a_i \in \mathbb{R}^d\) is the penultimate activation and \(e_i \in \mathbb{R}^K\) is the logit-space error (gradient of the loss with respect to the logits). For cross-entropy, \(e_i = \mathrm{softmax}(W a_i) - y_i\). For heads with a bias term, we augment \(a_i\) with a constant \(1\), so \(d\) includes the bias. This keeps \(g_i^{\mathrm{head}} = a_i \otimes e_i\) unchanged. We use column-major vectorization, so \(\mathrm{vec}(e_i a_i^\top) = a_i \otimes e_i\). If a row-major flattening is used in code, it yields a fixed permutation (equivalently \(e_i \otimes a_i\)), which leaves dot products and alignment cosines invariant.

9.1.0.1 Exact head empirical Fisher.

The exact head empirical Fisher is the uncentered second moment of per-example gradients: \[\mathcal{F}_k^{\mathrm{head}} = \mathbb{E}_{i \sim \mathcal{D}_k}\big[g_i^{\mathrm{head}} (g_i^{\mathrm{head}})^\top\big] = \mathbb{E}\big[(a_i \otimes e_i)(a_i \otimes e_i)^\top\big] = \mathbb{E}\big[(a_i a_i^\top) \otimes (e_i e_i^\top)\big].\]

The exact head empirical Fisher alignment (computed via the kernel identity, i.e., without a Kronecker approximation) is: \[A_{\mathrm{F}}^{\mathrm{exact}}(i,j) = \frac{\langle \mathcal{F}_i^{\mathrm{head}}, \mathcal{F}_j^{\mathrm{head}} \rangle_{\mathrm{F}}}{\|\mathcal{F}_i^{\mathrm{head}}\|_{\mathrm{F}} \, \|\mathcal{F}_j^{\mathrm{head}}\|_{\mathrm{F}}}.\] This is the empirical estimator of the population head alignment \(A_{\mathrm{F}}^{\mathrm{head}}\) used in the theory. We use “exact” to emphasize that no Kronecker approximation is used.

Naively computing this requires materializing a \((dK) \times (dK)\) matrix and scales as \(O((dK)^2)\) storage, which motivates the proxy and sketch estimators for large \(d\) and \(K\).

9.1.0.2 Kronecker proxy Fisher.

The Kronecker proxy approximates the expectation of a Kronecker product by a Kronecker product of expectations: \[\mathcal{F}_k^{\mathrm{head}} = \mathbb{E}\big[(a_i a_i^\top) \otimes (e_i e_i^\top)\big] \approx \mathbb{E}[a_i a_i^\top] \otimes \mathbb{E}[e_i e_i^\top] =: M_{a,k} \otimes \Gamma_{e,k}.\]

This approximation equals the exact Fisher when \(a_i\) and \(e_i\) are statistically independent. Independence is generically violated in trained networks (\(\delta \approx 0.34\); Table 11). However, we can still write the exact multiplicative decomposition \(A_{\mathrm{F}}^{\mathrm{head}}(i,j) = A_{\mathrm{proxy}}(i,j) \cdot \rho_{ij}\) (Theorem 2) when the denominator is nonzero (we \(\varepsilon\)-regularize in practice). We define the population coupling coefficient as \(\rho_{ij} := \chi_{ij}/\sqrt{\chi_{ii}\chi_{jj}}\) (see below for \(\chi\)). Under separability, the substantive approximation is \(\rho_{ij} \approx 1\). Empirically, we estimate it as \(\hat{\rho}_{ij} := A_{\mathrm{F}}^{\mathrm{exact}}(i,j) / A_{\mathrm{proxy}}(i,j)\).

The proxy Fisher alignment is: \[A_{\mathrm{F}}^{\mathrm{proxy}}(i,j) = S_{\mathrm{cov}}(M_{a,i}, M_{a,j}) \cdot S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}),\] where \(S_{\mathrm{cov}}(A,B) := \langle A, B \rangle_F / (\|A\|_F \|B\|_F)\) is the Frobenius cosine for uncentered second moments. It requires only \(O(d^2 + K^2)\) storage (for the second-moment matrices) instead of \(O((dK)^2)\) for the full Fisher. However, the proxy loses the coupling factor \(\rho\) and can invert rankings on some architectures (see below).

9.1.0.3 FisherSketch.

FisherSketch (Section 5 of main text) uses factored random features to estimate Fisher alignment in a single streaming pass. Unlike the Kronecker proxy, FisherSketch preserves the coupling factor \(\rho\) by sketching the product kernel \(k(g_s, g_t) = (\langle a_s, a_t \rangle \cdot \langle e_s, e_t \rangle)^2\) directly.

Complexity: \(O(m(d+K))\) operations per sample and \(O(m)\) storage per task.

The key insight is factored projection. Instead of forming \(g = a \otimes e\) (dimension \(dK\)), we compute \[\psi_l(a,e) = (r_{a,l} \cdot a)(r_{e,l} \cdot e)(r'_{a,l} \cdot a)(r'_{e,l} \cdot e),\] exploiting \((r_a \otimes r_e) \cdot (a \otimes e) = (r_a \cdot a)(r_e \cdot e)\). Each of the four dot products is \(O(d)\) or \(O(K)\); with \(m\) features, the total cost is \(O(m(d+K))\).

The main text vocabulary-scale experiments use FisherSketch.

9.2 Which Method is Used Where↩︎

9.2.0.1 Head-level validation tables use exact Fisher.

The correlations in controlled experiments are computed against the exact head empirical Fisher alignment \(A_{\mathrm{F}}^{\mathrm{exact}}\).

Computation method: We exploit the kernel-alignment interpretation. For head Fisher with gradient \(g_i = a_i \otimes e_i\), the Frobenius inner product can be computed as: \[\langle \mathcal{F}_i, \mathcal{F}_j \rangle_{\mathrm{F}} = \mathbb{E}_{s \sim i, t \sim j}\big[\langle g_s, g_t \rangle^2\big] = \mathbb{E}_{s \sim i, t \sim j}\big[(\langle a_s, a_t \rangle \cdot \langle e_s, e_t \rangle)^2\big].\]

This avoids materializing the \((dK) \times (dK)\) matrix and requires \(O(n_i n_j (d + K))\) operations for \(n_i, n_j\) samples.

Implementation: Exact head Fisher alignment is computed without materializing the Fisher matrix or forming \((dK)\)-dimensional gradient vectors, using only inner products: \[\begin{align} &\text{for } s \in \{1,\dots,n_1\},\, t \in \{1,\dots,n_2\}: \\ &\quad \text{fisher\_exact} \mathrel{+}= (\langle a_s^{(1)}, a_t^{(2)} \rangle \cdot \langle e_s^{(1)}, e_t^{(2)} \rangle)^2 \\ &\text{fisher\_exact} \gets \text{fisher\_exact} / (n_1 n_2 \cdot \|\mathcal{F}_1\|_F \cdot \|\mathcal{F}_2\|_F) \end{align}\] We compute \(\|\mathcal{F}_k\|_F\) using the same kernel identity on within-task pairs (\(s,t \sim k\)), which costs \(O(n_k^2(d+K))\) per task. We use V-statistics (including \(s=t\)), so differences from the population identity are \(O(1/n)\). For FisherSketch norms in the main text, we apply a U-statistic correction to remove diagonal bias; this does not affect the exact baseline reported here.

9.2.0.2 Why exact for validation?

Using exact Fisher eliminates concerns about circularity: the FA-CKA correction (which uses \(S_{\mathrm{cov}}(\Gamma_e)\)) is validated against Fisher alignment computed without assuming Kronecker structure.

A frozen-backbone stress test uses a shared-backbone MLP (fixed features; linear heads) and synthetic labels with controlled overlap. We report both exact and proxy values to show the discrepancy. At 75% overlap, the exact alignment is 0.37 while the proxy is 0.65 (77% relative error), and proxy error ranges from 0–77% across the sweep (Table 3).

9.2.0.3 Practical use: Proxy accuracy is architecture-dependent.

Since \(A_F^{\mathrm{head}} = A_{\mathrm{proxy}} \cdot \rho\) exactly (Theorem 2), a positive \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}})\) typically coincides with positive proxy–Fisher correlation, while a negative value can indicate inversion (this is a diagnostic, not a guarantee). Unless stated otherwise, \(\mathrm{Corr}(\cdot,\cdot)\) denotes Pearson correlation. Here \(\hat{\rho}\) denotes the empirical estimate of \(\rho\) from finite samples (defined in Appendix 18). On ResNet-50 (\(n=75\) same-dataset pairs), \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = +0.88\) (Pearson), and the proxy correlates positively with exact Fisher (Pearson \(= 0.98\); Spearman \(= 0.69\)). On ViT-B/16 (\(n=40\) same-dataset pairs), \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = -0.85\) (Pearson), and the proxy is negatively correlated with exact Fisher (Pearson \(= -0.34\); Spearman \(= -0.30\); Table 5). All head-level validation tables use exact Fisher; the proxy is conceptual motivation only. Full-network LLM validation uses parameter-subsampled gradients across all layers for computational tractability.

9.2.0.4 Estimator usage across experiments.

Head-level validation tables use exact Fisher via the kernel identity described above (Appendix 9). Vocabulary-scale experiments use FisherSketch with factored random features (Section 5). Full-network LLM validation uses parameter-subsampled gradients across layers; Appendix 10.7 describes a structured-projection full-network sketch.

9.2.0.5 Why proxy shows negative rank correlation on ViT (and why FA-CKA still works).

On ViT-B/16 (\(n=40\) same-dataset pairs), \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = -0.85\) (Pearson). High-proxy pairs tend to have low \(\hat{\rho}\), so the proxy is negatively correlated with exact Fisher (Pearson \(= -0.34\); Spearman \(= -0.30\); Table 5), indicating rank inversion. FA-CKA remains strongly correlated with exact Fisher (Pearson \(= 0.79\); Spearman \(= 0.90\)) because it is validated directly against exact Fisher—it does not use the proxy. CKA alone is only moderately correlated with exact Fisher (Pearson corr.\(= 0.43\), Spearman corr.\(= 0.45\)), reflecting stable representations but varying error geometry.

The 40 pairs comprise all same-dataset seed pairs across 4 datasets (CIFAR-100, Flowers102, DTD, Oxford-Pets) with 5 training seeds each. Cross-dataset pairs (10 total, subsampled) have exact Fisher \(=0\) by construction because their label spaces are disjoint under the shared-output masked-softmax embedding (Assumption 1). These pairs serve as a sanity check that \(\mathrm{CKA}\cdot S_{\mathrm{cov}}(\Gamma_e)\) outputs zero in this regime (Table 4).

3pt

10 Linear-Time Head Fisher Alignment via Kernel Mean Embeddings↩︎

This appendix develops an \(O(n)\)-in-samples estimator for the head Fisher alignment quantities in Theorem 2. The method applies when the output dimension \(K\) is moderate (classification, multiple-choice, retrieval). Vocabulary-scale validation appears in Section 10.11.

10.1 Quadratic Feature Map Identity↩︎

10.2 Mean Embedding Representation↩︎

Define feature maps: \[\begin{align} \phi_a(a) &:= \mathrm{vec}_{\mathrm{sym}}(aa^\top) \in \mathbb{R}^{d(d+1)/2}, \\ \phi_e(e) &:= \mathrm{vec}_{\mathrm{sym}}(ee^\top) \in \mathbb{R}^{K(K+1)/2}, \end{align}\] \[\phi(a,e) := \phi_a(a) \otimes \phi_e(e) \in \mathbb{R}^{d(d+1)/2 \cdot K(K+1)/2}\]

We assume finite fourth moments so the expectations below exist. When we state explicit bounds, we assume \(\|a\|_2 \le R_a\) and \(\|e\|_2 \le R_e\) almost surely.

For tasks \(i,j\), define the kernel expectations (with independent draws from each task): \[\begin{align} S_{ae}(i,j) &:= \mathbb{E}_{(a,e)\sim \mathcal{D}_i,\;(a',e')\sim \mathcal{D}_j} \big[(a^\top a')^2 (e^\top e')^2\big], \\ S_a(i,j) &:= \mathbb{E}_{a\sim \mathcal{D}_i,\;a'\sim \mathcal{D}_j} \big[(a^\top a')^2\big], \\ S_e(i,j) &:= \mathbb{E}_{e\sim \mathcal{D}_i,\;e'\sim \mathcal{D}_j} \big[(e^\top e')^2\big]. \end{align}\] When the denominators are nonzero, we also define the coupling ratios \[\chi_{ij} := \frac{S_{ae}(i,j)}{S_a(i,j)\,S_e(i,j)}, \qquad \rho_{ij} := \frac{\chi_{ij}}{\sqrt{\chi_{ii}\chi_{jj}}}.\] In practice we \(\varepsilon\)-regularize near-zero denominators (i.e., add \(\varepsilon\) before division).

This is the standard kernel mean embedding identity; see, e.g., [35].

Key insight: The \(n^2\) barrier is not fundamental—it arises from computing the expectation naively. The true barrier is that \(\mu_{ae,i}\) has dimension \(O(d^2 K^2)\), which is too large to store exactly.

10.3 Factored Random Maclaurin Sketching↩︎

We use the Random Maclaurin construction [36], which provides an unbiased estimator for squared inner products.

Crucially, this requires only \(O(m(d+K))\) operations per sample—not \(O(mdK)\)—because we compute \((r_a^\top a)\) and \((r_e^\top e)\) separately.

10.4 Implementation Details↩︎

10.4.0.1 Split-sample estimation.

To control ratio bias when computing \(\chi = S_{ae}/(S_a \cdot S_e)\), we primarily use split-half log-ratio estimation with shared projections (within-split) for lower variance; we also report a cross-fit variant that computes numerator and denominator on disjoint data splits, which reduces finite-sample bias and yields a consistent estimator as \(n \to \infty\); we also report jackknife standard errors as a robustness check.

10.4.0.2 U-statistic correction.

For diagonal entries (\(i=j\)), the mean embedding inner product includes self-terms (\(s=t\)) that overestimate the population quantity. We apply the U-statistic correction to all three matrices \(S_{ae}(i,i)\), \(S_a(i,i)\), \(S_e(i,i)\) (since \(\rho_{ij}\) depends on \(\chi_{ii}\) and \(\chi_{jj}\)): \[\widehat{S}^U = \frac{1}{n(n-1)} \left( \left\| \sum_{s=1}^n \psi(x_s) \right\|^2 - \sum_{s=1}^n \| \psi(x_s) \|^2 \right)\] where \(\psi\) is the feature map for the quantity of interest (\(\psi=\Psi_m\), \(x_s=(a_s,e_s)\) for \(S_{ae}\); \(\psi=\phi_a\) with \(x_s=a_s\) for \(S_a\); and \(\psi=\phi_e\) with \(x_s=e_s\) for \(S_e\)), or their corresponding sketches.

10.4.0.3 Positivity clamping.

True values \(S_a, S_e, S_{ae}\) are nonnegative, but sketch estimates can be slightly negative at modest \(m\) due to variance. We clamp: \(\hat{S} \leftarrow \max(\hat{S}, \epsilon)\) with fixed \(\epsilon = 10^{-10}\) before computing ratios.

10.5 Noise Robustness via Empirical-Bayes Shrinkage↩︎

10.5.0.1 Split-half identifiability and Empirical Bayes shrinkage.

For each task \(k\), we randomly split its samples into two halves \(s \in \{A,B\}\) (50/50 in expectation) and compute \(S_{ae}^s, S_a^s, S_e^s\) on each half using the same sketch projections, U-statistic diagonal correction, and the clamping above. Because the sketch is fixed across halves, the diagnostic below conditions on the projection draws and primarily reflects sampling variability. (This split-half diagnostic uses within-split estimates; the cross-fit ratio in the preceding paragraph is a separate bias-control variant.) Define \[\log \hat{\chi}_{ij}^s := \log S_{ae}^s(i,j) - \log S_a^s(i,j) - \log S_e^s(i,j), \qquad \log \hat{\rho}_{ij}^s := \log \hat{\chi}_{ij}^s - \tfrac{1}{2}\log \hat{\chi}_{ii}^s - \tfrac{1}{2}\log \hat{\chi}_{jj}^s .\] The split-half reliability diagnostic is the off-diagonal correlation \[r := \mathrm{Corr}_{i \neq j}\!\left(\log \hat{\rho}_{ij}^A, \log \hat{\rho}_{ij}^B\right),\] computed over off-diagonal pairs (equivalently \(i < j\) up to duplication). Let \(\bar{\ell}_{ij} = \tfrac{1}{2}(\log \hat{\rho}_{ij}^A + \log \hat{\rho}_{ij}^B)\) and estimate per-pair noise via \[\widehat{\mathrm{SE}}_{ij}^2 = \frac{(\log \hat{\rho}_{ij}^A - \log \hat{\rho}_{ij}^B)^2}{4},\] the usual variance estimate for the mean under independent halves. Define \[V_{\mathrm{total}} = \mathrm{Var}_{i \neq j}(\bar{\ell}_{ij}), \quad V_{\mathrm{noise}} = \mathbb{E}_{i \neq j}[\widehat{\mathrm{SE}}_{ij}^2], \quad \tau^2 = \max(0, V_{\mathrm{total}} - c\,V_{\mathrm{noise}}),\] with conservative \(c=1.5\) (we use \(c \in [1.5, 2.0]\)). This is a conservative method-of-moments estimate of between-pair variance, inflating noise to encourage shrinkage toward \(\rho=1\) when coupling is weak. The empirical Bayes shrinkage map is \[w_{ij} = \frac{\tau^2}{\tau^2 + \widehat{\mathrm{SE}}_{ij}^2 + 10^{-10}}, \qquad \log \hat{\rho}_{ij}^{\mathrm{EB}} = w_{ij}\bar{\ell}_{ij}, \qquad \hat{\rho}_{ij}^{\mathrm{EB}} = \exp(\log \hat{\rho}_{ij}^{\mathrm{EB}}),\] which shrinks toward \(\log \rho = 0\) (i.e., \(\rho = 1\), the separable proxy). Because shrinkage is applied in log space, \(\hat{\rho}_{ij}^{\mathrm{EB}}\) corresponds to the posterior median of \(\rho_{ij}\) under the log-normal model. We report the scalar diagnostic \(\alpha_{\mathrm{EB}} := \mathbb{E}_{i \neq j}[w_{ij}]\) but apply the pairwise weights in the final estimator. Using split-averaged \(S_a = \tfrac{1}{2}(S_a^A + S_a^B)\) and \(S_e = \tfrac{1}{2}(S_e^A + S_e^B)\), the shrunk alignment is \[\hat{A}_F^{\mathrm{EB}}(i,j) = \hat{A}_{\mathrm{proxy}}(i,j)\,\hat{\rho}_{ij}^{\mathrm{EB}}, \qquad \hat{A}_{\mathrm{proxy}}(i,j)=\frac{S_a(i,j)S_e(i,j)}{\sqrt{S_a(i,i)S_a(j,j)S_e(i,i)S_e(j,j)}}.\] All diagnostics are computed after clamping; when clamping is active for some pairs, it can inflate \(\widehat{\mathrm{SE}}_{ij}^2\) and thus increase shrinkage, which is conservative.

10.5.0.2 Complexity.

Per-task: \(O(n_i \cdot m \cdot (d + K))\) for sketch computation. Pairwise: \(O(m)\) per dot product. Total: \(O(\sum_i n_i \cdot m(d+K) + T^2 m)\).

10.5.0.3 Memory and reproducibility.

Storing Rademacher matrices requires \(O(m(d+K))\) memory, plus \(O(Tm)\) for task embeddings. For large \(d\) and \(m=4096\), this can be nontrivial. An alternative is stateless on-the-fly generation using seeded hashing (4-wise independent), which trades compute for memory. Seed sensitivity is reported for multi-run validations; Table 10 summarizes seeds and averaging across experiment blocks.

10.6 Validation↩︎

10.6.0.1 Synthetic data.

On synthetic data with varying activation-error coupling, the sketched estimator achieves Spearman corr.\(> 0.99\) with exact \(S_{ae}\) computation at \(m = 4096\). The diagonal \(\chi_{ii}\) values (within-task coupling) achieve Spearman corr.\(> 0.94\) with exact values. Note that off-diagonal \(\chi_{ij}\) values are typically close to \(1.0\) for settings where \(a\) and \(e\) are approximately independent within each task (so \(S_{ae}(i,j) \approx S_a(i,j)S_e(i,j)\) for any pair), which is expected and correctly captured by the sketched estimator.

10.6.0.2 Real models.

On ViT-B/16 with 20 fine-tuned checkpoints across 4 datasets (CIFAR-100, Flowers102, DTD, Oxford-Pets), we validate all key quantities by comparing sketched estimates to exact computation. In this moderate-\(K\) setting, we compute \(S_a\) and \(S_e\) exactly from their quadratic feature maps; for large \(K\) we sketch any quantity that cannot be materialized. For same-dataset pairs (where Fisher alignment is non-zero), at \(m = 4096\):

• \(\hat{S}_{ae}\): Spearman corr.\(0.97 \pm 0.01\)

• \(\hat{\chi}\): Spearman corr.\(0.94 \pm 0.01\)

• \(\hat{\rho}\): Spearman corr.\(0.95 \pm 0.01\)

• \(\hat{A}_F\): Spearman corr.\(0.79 \pm 0.06\) (lower due to ratio normalization)

Rankings for \(\hat{S}_{ae}\), \(\hat{\chi}\), and \(\hat{\rho}\) remain \(> 0.90\) at \(m = 2048\), while \(\hat{A}_F\) is 0.76. Cross-dataset pairs (disjoint label spaces under the shared-output embedding) have Fisher \(\approx 0\) by construction and are excluded from correlation analysis. Figure 1 shows how correlation improves with sketch dimension.

10.6.0.3 Wall-clock timing.

To demonstrate the practical speedup from linear-time estimation, we benchmark exact \(O(n^2)\) computation against sketched \(O(n)\) computation. Setup: 5 synthetic tasks, \(d=256\) (activation dimension), \(K=100\) (error dimension), \(m=4096\) (sketch dimension), averaged over 3 seeds. Table 6 shows runtime scaling: The sketched estimator achieves 89\(\times\) speedup at \(n=2000\) samples/task with \(m=4096\), with runtime scaling linearly in \(n\) (vs.quadratically for exact computation). Smaller sketch dimensions yield larger speedups (e.g., 437\(\times\) at \(m=1024\)) with slightly reduced accuracy. This makes head Fisher alignment practical at scale. Full-network sketching via structured projections is also linear in hidden sizes and sequence length (independent of parameter count; Appendix 10.7); our LLM experiments instead use parameter-subsampled gradients across all layers for computational tractability, while head-level validation uses the kernel identity.

10.7 Full-Network FisherSketch via Structured Rademacher Projections↩︎

The head-layer sketching above extends naturally to full-network Fisher alignment by exploiting the per-layer Kronecker structure of gradients.

10.7.0.1 The computational challenge.

For a full network with \(L\) layers and total parameters \(p = \sum_{\ell=1}^L d_{\ell-1} d_\ell\), naively computing \(r^\top g\) for a random Rademacher vector \(r \in \{\pm 1\}^p\) requires \(O(p)\) operations per sample—intractable for modern LLMs with billions of parameters.

10.7.0.2 Structured projection.

Because the full gradient \(g = [a_0 \otimes \delta_1; \ldots; a_{L-1} \otimes \delta_L]\) is a concatenation of Kronecker products, we can factor the projection. For each layer \(\ell\), generate independent Rademacher vectors: \[r_\ell^{(a)} \in \{\pm 1\}^{d_{\ell-1}}, \quad r_\ell^{(\delta)} \in \{\pm 1\}^{d_\ell}.\] The block of \(r\) corresponding to layer \(\ell\) is implicitly \(r_\ell = r_\ell^{(a)} \otimes r_\ell^{(\delta)}\), and the projection becomes: \[r^\top g(x) = \sum_{\ell=1}^L (r_\ell^{(a)\top} a_{\ell-1}(x)) \cdot (r_\ell^{(\delta)\top} \delta_\ell(x)). \label{eq:structured-projection}\tag{7}\] This requires only \(O(\sum_\ell (d_{\ell-1} + d_\ell))\) operations—linear in hidden sizes, not in parameter count (with a linear factor in positions for shared-parameter layers). For a shared-parameter affine layer used at positions \(t \in \{1,\dots,S_\ell\}\), the per-layer term becomes \[(r_\ell^{(a)} \otimes r_\ell^{(\delta)})^\top g_\ell(x) = \sum_{t=1}^{S_\ell} (r_\ell^{(a)\top} a_{\ell-1,t}(x))\,(r_\ell^{(\delta)\top} \delta_{\ell,t}(x)),\] so the structured projection remains computable in time linear in hidden sizes and the number of positions.

10.7.0.3 Algorithm: Full FisherSketch.

For task \(k\) with samples \(\{x_t\}_{t=1}^{n_k}\):

1. Sample \(m\) pairs of structured Rademacher vectors \((r^{(j)}, s^{(j)})\) with per-layer factorization.

2. For each sample \(x_t\): run forward pass to get \(\{a_{\ell-1}(x_t)\}_{\ell=1}^L\); run backward pass to get \(\{\delta_\ell(x_t)\}_{\ell=1}^L\).

3. Compute sketch features \(\psi^{(j)}(x_t) = (r^{(j)\top} g(x_t))(s^{(j)\top} g(x_t))\) using 7 .

4. Average to get mean embedding \(\mu_k = \frac{1}{n_k} \sum_t \psi(x_t) \in \mathbb{R}^m\).

Full-network Fisher alignment is then \(A_F^{\mathrm{full}}(i,j) \approx \cos(\mu_i, \mu_j)\).

10.7.0.4 Complexity.

Per sample: \(O(L \cdot d_{\max} \cdot m)\) for \(m\) sketch dimensions. Per task: \(O(n_k \cdot L \cdot d_{\max} \cdot m)\). Cross-task alignment: \(O(T^2 \cdot m)\). This is independent of parameter count \(p\) (linear in hidden sizes and positions), enabling full-network Fisher at 70B scale.

10.7.0.5 Implementation note.

We present this structured-projection estimator for completeness; its per-sample cost is linear in layer widths. Our experiments use parameter-subsampled gradients across all layers for computational tractability; the structured-projection estimator provides a full-network sketching option.

10.7.0.6 Relation to exact theorem.

Full FisherSketch approximates the exact full-network Fisher alignment (Theorem 15), just as head FisherSketch approximates head Fisher. The structured Rademacher projection preserves the polynomial kernel structure: \(\mathbb{E}[\psi(x)\psi(x')] = (g(x)^\top g(x'))^2 = (u^\top v)^2\), where \(u, v\) are the depth-profile vectors from Definition 9.

10.8 Fast FisherSketch via SRHT (Implementation Note)↩︎

For vocab-scale applications (\(K > 10^5\)), even the factored Random Maclaurin projection can become compute-limited: computing \(m\) dot products of the form \(r^\top e\) with \(r \in \{\pm 1\}^K\) requires \(O(mK)\) operations per sample. The Subsampled Randomized Hadamard Transform (SRHT) reduces this \(K\)-side cost to \(O(K \log K + m)\); if activations use dense projections, the total per-sample cost is \(O(md + K \log K + m)\).

10.8.0.1 Key construction.

Walsh–Hadamard transforms are defined for lengths \(N=2^p\). Let \(N := 2^{\lceil \log_2 K \rceil}\) and \(H_N \in \{\pm 1\}^{N \times N}\) be the (unnormalized) Walsh–Hadamard matrix. For \(e \in \mathbb{R}^K\), pad with zeros to \(\tilde{e} \in \mathbb{R}^N\). Let \(D = \mathrm{diag}(s)\) with \(s \in \{\pm 1\}^N\) i.i.d.Rademacher. For a fixed row index \(t \in [N]\), define \(r := H_{t,:} \cdot D\). Then \(r\) has i.i.d.Rademacher entries (each coordinate is a fixed \(\pm 1\) from \(H_N\) times an independent Rademacher sign from \(D\)); in particular, the first \(K\) coordinates are i.i.d.Rademacher and \(r^\top \tilde{e} = \sum_{k=1}^K r_k e_k\).

10.8.0.2 Fast projection.

Given indices \(t_1, \ldots, t_m \in [N]\), all \(m\) projections can be computed as:

1. Pad and sign flip: \(\tilde{e}' \leftarrow D \odot \tilde{e}\) \(O(N)\)

2. Compute Hadamard transform: \(u \leftarrow H_N \tilde{e}'\) \(O(N \log N)\)

3. Gather entries: \(\{u_{t_j}\}_{j=1}^m\) \(O(m)\)

Total: \(O(N \log N + m) = O(K \log K + m)\) (since \(N < 2K\)), instead of \(O(mK)\) on the error side.

10.8.0.3 Memory reduction.

Dense Rademacher matrices require \(O(mK)\) entries (about \(\sim\)​2 GB if stored as float32, or \(\sim\)​0.5 GB if stored as int8 signs). SRHT requires only \(O(N)\) float32 signs for diagonal matrices \((D, D')\) plus \(O(m)\) integers for row indices; since \(N < 2K\) this is still \(O(K)\). At \(K=128{,}256\) (so \(N=131{,}072\)) and \(m=4096\), this reduces memory to \(\sim\)​1 MB.

10.8.0.4 Unbiasedness.

The SRHT construction preserves the polynomial kernel property. For independent \((D, t)\) and \((D', t')\), and zero-padded \(\tilde{x}, \tilde{y} \in \mathbb{R}^N\): \[\mathbb{E}_{D,D',t,t'}\big[(r^\top \tilde{x})(r^\top \tilde{y})(r'^\top \tilde{x})(r'^\top \tilde{y})\big] = (x^\top y)^2,\] exactly as with dense Rademacher. Thus the SRHT backend targets the same unnormalized product-kernel inner product in expectation; the reported Fisher alignment cosine is the corresponding normalized plug-in estimate.

10.8.0.5 Dependence across coordinates.

SRHT uses a shared \((H_N,D)\) and row subsampling, so the \(m\) coordinates are not independent. Unbiasedness still holds, but variance constants can differ from the i.i.d.Rademacher case; we use the \(1/m\) scaling as a practical guide.

10.8.0.6 When to use SRHT.

SRHT is beneficial when \(mK\) is large. For typical LLM settings (\(K \approx 128{,}000\), \(m \approx 4096\)), the theoretical complexity reduction is \(O(mK) / O(K \log K) \approx 255\times\), though empirical speedups depend on hardware (memory bandwidth, cache effects). For smaller vocabularies or moderate \(m\), dense Rademacher may be faster due to simpler cache patterns and better batching. Our implementation automatically selects based on dimension.

10.9 Fast SRHT Error Projections in PyTorch↩︎

For vocabulary-scale errors \(e\in\mathbb{R}^{K}\), FisherSketch uses an SRHT to avoid storing dense \(m{\times}K\) Rademacher matrices. Let \(\tilde{K}\) be the next power of two \(\ge K\) and let \(H_{\tilde{K}}\) denote the \(\tilde{K}{\times}\tilde{K}\) Hadamard matrix. We sample sign vectors \(D,D'\in\{\pm1\}^{\tilde{K}}\) and index sets \(I,I'\subseteq[\tilde{K}]\) with \(|I|=|I'|=m\). Given a batch \(E\in\mathbb{R}^{B\times K}\), we pad to \(\tilde{K}\) and compute \[U = H_{\tilde{K}}\bigl((E_{\mathrm{pad}})\odot D\bigr),\qquad U' = H_{\tilde{K}}\bigl((E_{\mathrm{pad}})\odot D'\bigr),\] then return \(U_{[:,I]}\) and \(U'_{[:,I']}\). This gives \(O(\tilde{K}\log\tilde{K} + m)\) cost per batch for the error-side projections.

10.9.0.1 Batched FWHT (vectorized butterfly).

PyTorch does not provide a native FWHT; we implement the Hadamard transform via an iterative butterfly that uses only reshape/slice/concatenate operations. This avoids Python recursion and runs efficiently on GPU:

10.10 Low-Precision Sign Storage for Fixed Projection Memory↩︎

Dense Rademacher projections for activations (and errors, when dense) are stored as int8 signs on GPU and cast on-the-fly to the compute dtype during matmul. This reduces persistent projection storage by \(4\times\) compared to float32 while leaving the estimator unchanged.

10.11 Vocabulary-Scale Validation↩︎

We validate FisherSketch at vocabulary scale (\(K=128{,}256\)) using Llama-3.1-8B on 100 domains constructed from eight HF datasets (Table ¿tbl:tab:vocab-scale-datasets?). At this scale, storing the error second moment \(\Gamma_e\) per task requires \(K^2 \times 4\) bytes \(\approx 61\) GiB, so 100 tasks would require \(\sim\)​6.0 TiB. FisherSketch stores activation projections as dense int8 sign matrices: \(2dm\) entries (32 MiB for \(d{=}m{=}4096\)). In our PyTorch implementation these signs are cast to the compute dtype (bf16/fp16/fp32) on-the-fly for GEMM; this is an exact re-encoding of the same Rademacher \(\{\pm 1\}\) projections and yields indistinguishable transfer metrics versus storing the same signs in float32. For error projections we use SRHT, which stores only two sign vectors \((D,D')\) and row indices (approximately 1 MB at \(K{=}128{,}256\), \(m{=}4096\)). During streaming, we maintain six float64 accumulators per task for split-sample estimation (192 KB per task), but the stored signature is just the float32 \(\hat{\mu}_{ae}\) vector (16 KB per task; 48 KB if \(\hat{\mu}_a,\hat{\mu}_e\) are also stored). Fixed projection memory is about 33 MiB; for retrieval, the FAISS index stores another copy of \(\hat{\mu}_{ae}\). SRHT on activations can reduce fixed memory further. This is a \(\mathbf{\sim 4.0\times 10^6}\) reduction in per-task storage (61 GiB \(\rightarrow\) 16 KB); the working-memory reduction is \(\sim 3.3\times 10^5\) if counting the streaming state (192 KB per task). Accounting for the fixed projections, the total reduction for 100 tasks is \(\sim 1.8\times 10^5\); with dense error projections, storing the two \(m{\times}K\) sign matrices would require \(\approx 1.0\) GiB fixed if stored as int8 (or \(\approx 3.9\) GiB if stored as float32), and the total reduction would be \(\sim 6\times 10^3\) (or \(\sim 1.6\times 10^3\) for float32).

10.11.0.1 Datasets and domain construction.

We draw from eight HF datasets, using the train split and the specific text fields shown in Table ¿tbl:tab:vocab-scale-datasets?. Each domain is a deterministic subsample; domain names follow the pattern family and family_i (12–13 slices per family), with a fixed seed per domain. We record dataset provenance and sample indices in domain_metadata.json; no fallback or synthetic data were used in the reported runs. We do not redistribute raw text, and dataset licenses follow the respective HF dataset cards.

10.11.0.2 Sampling and splits.

FisherSketch uses 500 samples per domain (\(m{=}4096\)), keeping texts with \(\ge 100\) characters and truncating to max_seq_length\(\times\)​4. LoRA validation draws 300 texts per domain (200 train, 100 eval) using a separate data seed (default: seed+1); eval samples are disjoint from train by construction. Early stopping uses the source-domain eval split.

10.11.0.3 Disjointness and auditability.

Validation explicitly excludes the FisherSketch sample indices for each domain (recorded in domain_metadata.json), preventing overlap between sketching and LoRA train/eval data.

10.11.0.4 Candidate sampling and pair counts.

For each target, we sample 24 candidate sources uniformly from the other 99 domains (uniform protocol), giving \(100 \times 24 = 2400\) off-diagonal pairs. We also sample a stratified set (3 sources per family across 8 families, filled to 24 if needed); we evaluate the union of uniform and stratified pairs. For the reported run this yields 4,188 unique off-diagonal pairs, plus 100 diagonal pairs for self-PPL. Top-1 and regret are computed within the uniform 24-candidate sets; correlation analyses use the union of evaluated pairs. Candidate sets and metadata are saved in candidate_sources_*.json and pair_sampling_metadata.json.

10.11.0.5 Transfer validation.

To verify that FisherSketch at \(K{>}10^5\) produces meaningful alignment, we validate against actual LoRA transfer performance: for each source-target pair, we fine-tune a LoRA adapter on the source domain and evaluate perplexity on the target. The normalized transfer score measures what fraction of possible improvement (relative to self-training) the source provides in loss space (log PPL). On 4,188 off-diagonal pairs (100 domains), FisherSketch with SRHT error projections achieves 45.7% \(\pm\) 5.0 top-1 source selection (\(\approx\)​10.9\(\times\) random) with max regret 0.119 \(\pm\) 0.013 (mean \(\pm\) std over seeds 43–45), confirming that FisherSketch predicts cross-domain transfer at vocabulary scale where baseline methods cannot run.

10.11.0.6 Loss-space regret.

For completeness, Table 8 reports per-target regret in log-PPL units for the uniform 24-candidate protocol. These values are unnormalized and not comparable across targets; normalized-transfer regret in Table 1 remains the primary metric.

= by -

11 Shared-Parameter Affine Layers: Convolutions and Token-Wise Transformer Projections↩︎

Many modern architectures reuse the same affine parameters across multiple positions within an example: convolution kernels are shared across spatial locations, and transformer projection matrices (e.g.,\(W_Q, W_K, W_V, W_O\)) and MLP matrices are shared across tokens. In this regime, the per-example gradient is a sum of outer products over positions rather than a single outer product. This section provides (i) an exact kernel identity for the shared-parameter gradient and (ii) an efficient factored sketch that generalizes FisherSketch. Table 9 gives additional exact-Fisher validation for shared-parameter layers. These results validate the shared-layer FisherSketch estimator itself; they do not assume the separable activation/error proxy that is specific to no-sharing heads. As an implementation check, the shared-layer kernel identity matched materialized per-example gradients to machine precision in the Q1 validation (mean absolute discrepancies at most \(2\times 10^{-16}\) and maxima at most \(6\times 10^{-16}\)).

11.0.0.1 Setup.

Fix a task distribution \(\mathcal{D}_k\) over examples \((x,y)\) and fix model parameters \(\theta\). Consider a shared affine parameter matrix \(W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}\) used at \(S\) positions (possibly varying by example). For an example \((x,y)\), let \(a_t(x)\in\mathbb{R}^{d_{\mathrm{in}}}\) denote the input vector at position \(t\in\{1,\dots,S\}\). Define \[z_t(x)=W\,a_t(x)\in\mathbb{R}^{d_{\mathrm{out}}}.\] Let \(\delta_t(x):=\nabla_{z_t}\ell_\theta(x,y)\in\mathbb{R}^{d_{\mathrm{out}}}\) be the backpropagated gradient with respect to the pre-activation \(z_t\). Stack these as column matrices \[A(x):=[a_1(x),\dots,a_S(x)]\in\mathbb{R}^{d_{\mathrm{in}}\times S},\qquad \Delta(x):=[\delta_1(x),\dots,\delta_S(x)]\in\mathbb{R}^{d_{\mathrm{out}}\times S}.\] We focus on the per-example gradient with respect to \(W\) and its induced Fisher block. If the shared affine map includes a bias term \(b\), we can augment each \(a_t\) with a constant \(1\) and treat \(b\) as an extra column of \(W\). All identities below then apply unchanged.

11.0.0.2 A shared-layer Fisher kernel.

Define the degree-2 polynomial kernel induced by the shared-parameter gradient: \[k_{\mathrm{share}}\bigl((A,\Delta),(A',\Delta')\bigr) := \bigl(g_W^\top g_W'\bigr)^2 = \langle A^\top A',\Delta^\top\Delta'\rangle_F^2. \label{eq:kshare}\tag{10}\] This kernel is positive semidefinite because it is the homogeneous polynomial kernel of degree 2 applied to the feature vector \(g_W\).

11.0.0.3 SharedFisherSketch: an efficient factored sketch.

Although \(g_W\) (as a vector in \(\mathbb{R}^{d_{\mathrm{in}}d_{\mathrm{out}}}\)) can be very large, it admits a structured projection via Lemma 4.

11.0.0.4 Sketch variance and estimator caveats.

Unbiasedness uses only second moments, while the variance depends on higher moments and can be large for polynomial-kernel sketches. See standard analyses of Random Maclaurin features and TensorSketch for variance/accuracy tradeoffs [36], [37]; the shared-parameter sketch inherits the same behavior.

11.0.0.5 Computation and relation to the head.

Computing \(s(A,\Delta;r_a,r_\delta)\) can be done without forming \(g_W\): we first compute \(\alpha:=r_a^\top A\in\mathbb{R}^S\) and \(\beta:=r_\delta^\top \Delta\in\mathbb{R}^S\), then \(s=\alpha^\top\beta\). This costs \(O(S(d_{\mathrm{in}}+d_{\mathrm{out}}))\) per coordinate and is independent of the parameter count \(d_{\mathrm{in}}d_{\mathrm{out}}\). When \(S=1\), so \(A=a\) and \(\Delta=\delta\), Proposition 8 reduces exactly to the head-layer factored sketch (Proposition 5).

12 Preliminaries (Detailed)↩︎

This appendix provides full definitions and derivations for Fisher information and linear CKA, expanding the condensed treatment in Section 2.

12.1 Fisher Information and Fisher Alignment↩︎

The Fisher information matrix is central to optimization and natural-gradient methods and is widely used in continual learning [33], [38]–[40]. For a given task \(k\), define the per-example gradient at \(\theta\) as \[g_\theta^{(k)}(x,y) := \nabla_\theta \ell_\theta^{(k)}(x,y) \in \mathbb{R}^p.\] We work with the data Fisher—also called the empirical Fisher in population form—defined as the uncentered second moment of per-example gradients under \(\mathcal{D}_k\): \[\mathcal{F}_k(\theta) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[ g_\theta^{(k)}(x,y)\,g_\theta^{(k)}(x,y)^\top \big] \in \mathbb{R}^{p\times p}. \label{eq:fisher-def-app}\tag{11}\] This is the gradient outer product Fisher, standard in continual learning [33]; see Remark 9 for Fisher variants. We assume \(\mathcal{F}_k(\theta)\) exists and is positive semidefinite, and we additionally assume positive definiteness where convenient.

In practice, we approximate \(\mathcal{F}_k(\theta)\) from \(n_k\) i.i.d.samples \((x^{(k)}_i,y^{(k)}_i)\sim\mathcal{D}_k\) using the sample average: \[\hat{\mathcal{F}}_k := \frac{1}{n_k} \sum_{i=1}^{n_k} g_\theta^{(k)}(x^{(k)}_i,y^{(k)}_i)\, g_\theta^{(k)}(x^{(k)}_i,y^{(k)}_i)^\top. \label{eq:empirical-fisher-app}\tag{12}\]

To compare the Fisher geometry of two tasks, we use the cosine similarity defined in the main text (Eq. 1 ). High Fisher alignment indicates that tasks \(i\) and \(j\) emphasize similar directions in parameter space. Low Fisher alignment indicates that the tasks are decoupled in the Fisher metric.

In practice, we estimate alignment \(\hat{A}_{\mathrm{F}}(i,j)\) by replacing \(\mathcal{F}_i,\mathcal{F}_j\) in 1 with \(\hat{\mathcal{F}}_i,\hat{\mathcal{F}}_j\).

12.1.0.1 Gradient-space kernel alignment.

It is useful to view Fisher matrices as Gram matrices of gradient features. Assuming \(n_k>0\), define the gradient feature matrix for task \(k\) as \[\Phi_k := \begin{bmatrix} g_\theta^{(k)}(x^{(k)}_1,y^{(k)}_1)^\top \\ \vdots \\ g_\theta^{(k)}(x^{(k)}_{n_k},y^{(k)}_{n_k})^\top \end{bmatrix} \in \mathbb{R}^{n_k \times p}.\] Then \(\hat{\mathcal{F}}_k = \tfrac{1}{n_k}\Phi_k^\top \Phi_k\). Since the cosine normalization cancels constant factors, Fisher alignment equals a kernel alignment between gradient Gram matrices: \[\hat{A}_{\mathrm{F}}(i,j) = \frac{\langle \Phi_i^\top \Phi_i, \Phi_j^\top \Phi_j \rangle_F}{\|\Phi_i^\top \Phi_i\|_F\,\|\Phi_j^\top \Phi_j\|_F}.\] This is formally analogous to CKA, which we now detail for representation features.

12.2 Linear CKA for Representations↩︎

Linear CKA [10] extends kernel alignment [42], [43] to compare neural network representations. Let \(Z_i \in \mathbb{R}^{n\times d_i}\) and \(Z_j \in \mathbb{R}^{n\times d_j}\) be representation matrices with \(n\) samples (rows) and \(d_i,d_j\) features (columns). We assume each column is sample-centered (zero empirical mean). Write \(K_i := Z_i Z_i^\top\) and \(K_j := Z_j Z_j^\top\) for the corresponding Gram matrices.

The linear CKA between \(Z_i\) and \(Z_j\) is defined as: \[\text{CKA}(Z_i, Z_j) := \frac{\|Z_i^\top Z_j\|_F^2}{\|Z_i^\top Z_i\|_F \|Z_j^\top Z_j\|_F} \label{eq:cka-def}\tag{13}\] We assume \(\|Z_i^\top Z_i\|_F,\|Z_j^\top Z_j\|_F>0\) so CKA is well-defined.

Linear CKA is invariant to orthogonal transformations and isotropic rescaling of the feature space. It is typically computed on paired data: the rows of \(Z_i\) and \(Z_j\) correspond to the same inputs \(x_1,\dots,x_n\) fed through two different networks, layers, or heads. SVCCA [12] (with PCA truncation) is invariant to orthogonal rotations and isotropic rescaling, but is not invariant to arbitrary invertible linear transforms in general because the PCA truncation can change the retained subspace. Plain CCA is invariant to invertible linear transforms on the full space. However, practical CCA/SVCCA pipelines [44] can be sensitive to preprocessing choices. CKA is invariant to orthogonal rotations and isotropic rescaling without relying on a truncation step, making it popular for comparing representations across random initializations or architectures.

The key distinction we explore in the main text is between auto-covariance alignment (which governs Fisher geometry) and cross-covariance alignment (which CKA measures). This distinction is made precise in Section 3.

13 Experimental Protocol Details↩︎

This appendix provides complete experimental details for the supplementary CIFAR-100 regime experiments and the synthetic overlap/similarity sweeps. The main paper FisherSketch experiments use ViT-B/16 for sketch accuracy and Llama-3.1-8B for vocab-scale transfer; full-network validation additionally includes Llama-3.1-70B (Appendix 23.9).

13.1 Architectures and Training (Supplementary)↩︎

Supplementary CIFAR-100 regime experiments use torchvision ResNet-18 [45] (ImageNet-style stem: \(7\times7\) conv, stride 2, maxpool; no CIFAR-specific stem) on CIFAR-100; synthetic overlap/similarity sweeps use small shared-backbone MLPs with randomly generated labels. Training uses SGD (momentum 0.9, weight decay \(5 \times 10^{-4}\) applied to all parameters, including biases and BatchNorm parameters) with PyTorch cosine annealing (CosineAnnealingLR with \(T_{\max}=\text{epochs}\), \(\eta_{\min}=0\), stepped once per epoch; no warmup or restarts); source models are trained for 100 epochs with learning rate 0.2 and batch size 1024, and transfer runs use 50 epochs with the same optimizer settings. Data augmentation uses random crop (32, padding 4) and random horizontal flip; test-time uses only normalization (mean \((0.5071, 0.4867, 0.4408)\), std \((0.2675, 0.2565, 0.2761)\)). For class-subset tasks, we filter CIFAR-100 to the specified 50 classes and remap labels to \(\{0,\dots,49\}\) (25,000 train / 5,000 test; \(\approx\)​25 steps/epoch at batch size 1024). Linear-probe heads on frozen backbones are trained with SGD (learning rate 0.1, weight decay \(1 \times 10^{-4}\), cosine annealing) for 100 epochs. For the supplementary CIFAR-100 regime experiments and synthetic overlap/similarity sweeps in this appendix, we use a single fixed seed (42) and do not average over seeds unless explicitly noted; multi-seed validations are called out in Table 10.

13.1.0.1 Random seeds and averaging.

Table 10 summarizes which experiments use multiple seeds and whether results are averaged.

4pt

13.2 Task Pairs, Overlap, and Similarity Sweeps↩︎

13.2.0.1 CIFAR-100 overlap tasks.

We define a fixed source task using classes 0–49 (50 classes). Target tasks are contiguous class ranges with controlled overlap: 0–49 (100% overlap, 50/50 shared), 13–62 (37/50 shared; 0.74 overlap), 25–74 (25/50 shared; 0.50 overlap), 38–87 (12/50 shared; 0.24 overlap), and 50–99 (0% overlap, disjoint). Overlap fraction is \(|\mathcal{C}_{\mathrm{src}} \cap \mathcal{C}_{\mathrm{tgt}}|/50\). Class identities are preserved (no label permutation).

13.2.0.2 Synthetic similarity sweeps.

For class-count/overlap sweeps in the synthetic experiments, label overlap is defined by class-set overlap: overlap \(=1\) uses identical labels, overlap \(=0\) uses disjoint class halves, and intermediate overlap samples labels from the overlapping class ranges. Network similarity is controlled by linearly interpolating two random networks across all layers: \(W^{(2)}_l \leftarrow \alpha W^{(1)}_l + (1-\alpha) W^{(2,\mathrm{rand})}_l\) with \(\alpha \in \{0, 1/(N{-}1), \dots, 1\}\) (default \(N{=}100\)). We do not renormalize weights after interpolation. Because we do not renormalize, interpolation scales weight norms by \(\sqrt{\alpha^2 + (1-\alpha)^2}\), so \(\alpha\) jointly controls similarity and scale.

13.3 Metric Computation↩︎

• CKA: Computed on centered penultimate-layer activations using the full CIFAR-100 test set (10,000 images) shared across models; optional deterministic subsampling is used only for fast runs.

• \(S_{\mathrm{cov}}(\Gamma_e)\): Error second-moment matrices \(\Gamma_e = \frac{1}{n}\sum e_i e_i^\top\) computed on each task’s test split, where \(e = \text{softmax}(Wa) - y\) (predictions minus one-hot labels, with masked softmax: logits set to \(-\infty\) outside the task’s label block; Assumption 1). Errors are embedded into the shared 100-class space via that shared-output embedding, so coordinates outside the block are zero; linear-probe runs use the trained probe head.

• Fisher alignment: Exact head Fisher via the kernel identity (Appendix 9) is used only for head-level validation. The CIFAR-100 regime experiments report FA-CKA \(= \mathrm{CKA} \cdot S_{\mathrm{cov}}(\Gamma_e)\) and the separable proxy \(S_{\mathrm{cov}}(\Sigma_z) \cdot S_{\mathrm{cov}}(\Gamma_e)\) instead, where \(\Sigma_z\) is the centered covariance of penultimate activations (computed on the same test set as CKA).

13.4 Reported Correlations↩︎

For CIFAR-100 regime experiments, correlations (Pearson corr.) are computed across the five target tasks within each regime (five overlap levels). For synthetic similarity sweeps, correlations are computed across the sweep points (\(N\) trials). Because \(N\) is small in the CIFAR-100 regime, correlations are reported for completeness and interpreted qualitatively.

14 Shared Output Space: Details↩︎

14.1 Setup↩︎

To compare error covariances \(\Gamma_{e,i}\) and \(\Gamma_{e,j}\) across tasks with different label vocabularies, we use a common coordinate system.

We assume each task \(k\) predicts over its label subset \(\mathcal{C}_k \subseteq \{1,\ldots,K\}\), either via a subset head or a shared \(K\)-way head with masked softmax (logits set to \(-\infty\) outside \(\mathcal{C}_k\)). If tasks use subset heads, we identify their head parameters with a shared \(K\)-way matrix by zero-padding rows outside \(\mathcal{C}_k\). Predictions still use the masked softmax, so gradients and errors live in the common coordinate system. We assume label identities are aligned across tasks in this shared coordinate system. To compare error covariances across tasks, we embed these predictions into a common \(K\)-dimensional space via the injection \(P_k\) in Assumption 1. The embedded error vector \(e_k \in \mathbb{R}^K\) is zero in coordinates outside \(\mathcal{C}_k\). This setup is standard when tasks are subsets of a larger taxonomy (e.g., CIFAR-100 has \(K{=}100\) classes and each task uses a subset). The disjoint-support implications below rely on masked softmax (logits set to \(-\infty\) outside \(\mathcal{C}_k\)); with an unmasked \(K\)-way softmax, probability mass can appear on non-task classes, so the errors need not be sparse.

14.2 Implications↩︎

Under this embedding: \[\langle \Gamma_{e,i}, \Gamma_{e,j} \rangle_F = \langle \Gamma_{e,i}[\mathcal{C}_{ij}], \Gamma_{e,j}[\mathcal{C}_{ij}] \rangle_F, \qquad S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}) = \frac{\langle \Gamma_{e,i}[\mathcal{C}_{ij}], \Gamma_{e,j}[\mathcal{C}_{ij}] \rangle_F}{\|\Gamma_{e,i}\|_F \|\Gamma_{e,j}\|_F}, \label{eq:overlap-restricted}\tag{14}\] where \(\mathcal{C}_{ij} := \mathcal{C}_i \cap \mathcal{C}_j\) and \(\Gamma_{e,k}[\mathcal{C}_{ij}]\) denotes the submatrix restricted to rows/columns in \(\mathcal{C}_{ij}\). If \(\|\Gamma_{e,i}\|_F\) or \(\|\Gamma_{e,j}\|_F\) is zero, then \(S_{\mathrm{cov}}\) is undefined (we \(\varepsilon\)-regularize in practice).

• Disjoint tasks (no shared classes, \(\mathcal{C}_i \cap \mathcal{C}_j = \emptyset\)): Error covariances have disjoint support, so \(\langle \Gamma_{e,i}, \Gamma_{e,j} \rangle_F = 0\) and \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}) = 0\) when the norms are nonzero.

• Partially overlapping tasks: The overlap-restricted inner product above governs \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\). In a stylized toy model that ignores the simplex constraint \(\mathbf{1}^\top e = 0\), assume \(\Gamma_{e,k} = \sigma_k^2 I_{\mathcal{C}_k}\),³ \[S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}) = \frac{|\mathcal{C}_{ij}|}{\sqrt{|\mathcal{C}_i|\,|\mathcal{C}_j|}}.\]

• Identical tasks: If \(\Gamma_{e,i} = c\,\Gamma_{e,j}\) with \(c>0\), then \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}) = 1\).

14.2.0.1 Implementation sanity checks.

It is helpful to verify that (i) \(e_k\) is zero outside \(\mathcal{C}_k\) under masked softmax (logits set to \(-\infty\) outside \(\mathcal{C}_k\)), (ii) Eq. 14 holds numerically within floating-point tolerance, and (iii) \(\Gamma_{e,k}\mathbf{1} \approx 0\) for softmax errors.

This framework applies when tasks are subsets of a larger taxonomy with a fixed label alignment (e.g., selecting animal vs.vehicle classes from CIFAR-100). The resulting \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\) depends on the chosen embedding. For different label vocabularies, any cross-task embedding is a modeling choice. For tasks with fundamentally different output spaces (e.g., classification vs.regression), error-covariance comparison requires an explicit embedding, while representation-only metrics remain agnostic to gradient compatibility.

15 CKA–Fisher Gap Decomposition (Head Layer)↩︎

This appendix provides a self-contained deviation bound that relates empirical linear CKA (on a paired probe set) to population head Fisher alignment. The bound is primarily diagnostic: it separates (i) finite-sample CKA estimation error, (ii) CKA’s inherent cross-vs.-auto covariance mismatch, (iii) centering (mean) effects, (iv) error-geometry misalignment, and (v) activation–error dependence (i.e., Kronecker-factorization violation).

15.0.0.4 Probe mismatch.

If the probe input marginal \(\mathcal{D}_0\) differs from the task input marginals \(\mathcal{D}_i^x,\mathcal{D}_j^x\), the only change is in the mean-correction step. Insert the additional term \(\bigl|S_{\mathrm{cov}}(\Sigma_i,\Sigma_j) - S_{\mathrm{cov}}(\Sigma_{a,i},\Sigma_{a,j})\bigr|\) to account for replacing probe covariances by task covariances. The error term \(\varepsilon_e\) and coupling residuals \(\delta_k\) are often the dominant obstructions in many-class settings, explaining why CKA can decouple from head Fisher when label/error structures differ.

15.0.0.5 Estimating \(\kappa_{ij}\).

When \(d\) is moderate and we form explicit sample covariances, \(\kappa_{ij}\) can be estimated directly as \(\big|S_{\mathrm{cross}}(\hat{\Sigma}_{ij};\hat{\Sigma}_i,\hat{\Sigma}_j) - S_{\mathrm{cov}}(\hat{\Sigma}_i,\hat{\Sigma}_j)\big|\). For high-dimensional representations where it is undesirable to interpret \(S_{\mathrm{cov}}(\hat{\Sigma}_i,\hat{\Sigma}_j)\) in a fixed coordinate basis, we additionally report the rotation-invariant Procrustes diagnostic \(\hat{\kappa}\) from Appendix 17 as a proxy for cross-vs.-auto mismatch.

16 Relationship to K-FAC and Prior Work↩︎

This appendix clarifies how our work relates to prior work on Kronecker-factored approximations to the Fisher information matrix, particularly K-FAC [6], [7]. Subsequent work has extended K-FAC to recurrent networks, eigenbasis approximations, and distributed training [46]–[48].

16.1 What K-FAC Established↩︎

K-FAC (Kronecker-Factored Approximate Curvature; [6]) introduced a Kronecker-factored approximation of layer-wise Fisher information matrices for efficient second-order optimization. Key components include:

16.1.0.1 Exact gradient structure.

K-FAC observes that for layer \(l\) with weight matrix \(W_l\), the per-example gradient has an exact Kronecker structure: \[\mathrm{vec}(\nabla_{W_l} \ell) = a_{l-1} \otimes \delta_l\] Here \(a_{l-1}\) is the layer input and \(\delta_l\) is the backpropagated error. This identity follows directly from the chain rule.

16.1.0.2 Independence approximation.

K-FAC’s central approximation is that activations and errors are approximately independent, yielding the approximation: \[\mathbb{E}[(a \otimes \delta)(a \otimes \delta)^\top] \approx \mathbb{E}[aa^\top] \otimes \mathbb{E}[\delta\delta^\top]\] This factorization enables efficient Fisher inversion (two small matrix inversions instead of one giant one).

16.1.0.3 Block-diagonal approximation.

K-FAC further approximates the full Fisher as block-diagonal across layers, ignoring cross-layer correlations.

16.2 Our Novel Contributions Beyond K-FAC↩︎

Our work builds on K-FAC’s structural insights but addresses a different question. K-FAC asks: “How can we efficiently approximate the natural gradient for optimization?” We ask: “Why do representation similarity metrics fail to predict gradient alignment?”

16.2.0.1 1. Error covariance alignment as the critical factor.

While K-FAC uses the approximation \(\Sigma_{\psi,k} \approx M_{a,k} \otimes \Gamma_{e,k}\) (Lemma 3) for computational efficiency, we identify error-covariance alignment \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\) as the missing factor that explains why CKA can decouple from Fisher alignment. K-FAC does not study cross-task alignment or representation metrics.

16.2.0.2 2. Formal characterization of CKA’s limitation.

We prove that CKA (and all representation-only metrics) structurally cannot, in general, access error covariance information (Theorem 1). This is a fundamental limitation independent of any approximation quality.

16.2.0.3 3. Quantitative diagnostics for the CKA–Fisher gap.

Our analysis exposes measurable drivers of the gap, including the coupling correction \(\rho\) (Theorem 2), the Kronecker residual \(\delta\) (Remark 3), and the coupling-misalignment term \(\kappa\) (Appendix 17). K-FAC focuses on optimization convergence, not metric reliability.

16.2.0.4 4. Label structure dependence.

We show that \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\) drops to zero for disjoint \(K\)-class tasks under the shared output-space embedding (Appendix 14), illustrating a failure mode for CKA when label structures differ. This is orthogonal to K-FAC’s optimization focus.

16.2.0.5 5. Practical estimator (FA-CKA).

FA-CKA provides a forward-pass-only estimator of head Fisher alignment without computing gradients or Fisher matrices. K-FAC requires gradient computation for natural gradient steps.

16.3 Independence Assumption: Our Analysis vs. K-FAC↩︎

K-FAC relies on approximate independence between activations and backpropagated errors (denoted \(\delta\) in H.1 and \(e\) elsewhere in this paper). Under this assumption, \(\Sigma_{\psi,k} = \mathbb{E}[(a \otimes e)(a \otimes e)^\top] \approx M_{a,k} \otimes \Gamma_{e,k}\) (Lemma 3). Our analysis is unconditional: Theorem 2 is an exact identity for head Fisher alignment with no independence assumption.

16.3.0.1 In K-FAC:

Violating independence reduces optimization efficiency but does not fundamentally break the method; even with imperfect factorization, K-FAC can remain a reasonable preconditioner.

16.3.0.2 In our work:

Head Fisher alignment decomposes exactly as \(A_{\mathrm{F}}^{\mathrm{head}}(i,j) = A_{\mathrm{proxy}}(i,j) \cdot \rho_{ij}\) (Theorem 2). Here \(A_{\mathrm{proxy}} := S_{\mathrm{cov}}(M_a) \cdot S_{\mathrm{cov}}(\Gamma_e)\), and \(\rho_{ij}\) is defined when the denominator is nonzero. Independence implies \(\rho_{ij} = 1\) (Corollary 2), but \(\rho \neq 1\) does not invalidate the theory—it is simply a measurable correction. The Kronecker residual \(\delta\) yields a worst-case alignment bound (Appendix 20); whether \(\rho \approx 1\) is architecture-dependent (on ViT, \(\rho\) varies from 0.35 to 1.11; see Appendix 18).

Crucially, even with perfect independence (\(\rho = 1\)), CKA can fail when error covariances are misaligned (small \(S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\)), e.g., for disjoint label subsets under the shared output-space embedding (Appendix 14).

16.4 Summary of Attribution↩︎

• Gradient Kronecker structure (Eq. 3 ): Exact identity, noted by K-FAC [6]

• Fisher independence factorization (Lemma 3): K-FAC’s foundational approximation

• Error covariance alignment factor: Our contribution

• CKA limitation theorem (Theorem 1): Our contribution

• Quantitative diagnostics for the CKA–Fisher gap (\(\rho\), \(\delta\), \(\kappa\)): Our contribution

• FA-CKA estimator: Our contribution

17 Estimation and Diagnostic Experiments↩︎

This appendix provides implementation details for estimating the structural quantities in our theory and presents diagnostic experiments that validate the theory across different architectures.

17.1 Estimating Structural Quantities↩︎

Each term in our decomposition can be estimated from finite samples.

17.1.0.1 Representation misalignment (\(\kappa\)).

Define the population mismatch \(\kappa_{ij} := \big|S_{\mathrm{cross}}(\Sigma_{ij};\Sigma_i,\Sigma_j) - S_{\mathrm{cov}}(\Sigma_i,\Sigma_j)\big|.\) Direct estimator (when covariances are materialized). When the representation dimension \(d\) is shared across models and we form explicit sample covariances, we estimate \(\kappa_{ij}\) directly via \[\hat{\kappa}^{\mathrm{dir}}_{ij} := \Big|\widehat{S}_{\mathrm{cross}}(\hat{\Sigma}_{ij};\hat{\Sigma}_i,\hat{\Sigma}_j) - S_{\mathrm{cov}}(\hat{\Sigma}_i,\hat{\Sigma}_j)\Big|, \qquad \widehat{S}_{\mathrm{cross}}(\hat{\Sigma}_{ij};\hat{\Sigma}_i,\hat{\Sigma}_j) := \frac{\|\hat{\Sigma}_{ij}\|_F^2}{\|\hat{\Sigma}_i\|_F\|\hat{\Sigma}_j\|_F}.\] This is the plug-in estimator that matches the definition of \(\kappa_{ij}\).

Rotation-invariant diagnostic (Procrustes proxy). For high-dimensional representations, it can be undesirable to interpret \(S_{\mathrm{cov}}(\hat{\Sigma}_i,\hat{\Sigma}_j)\) in a fixed coordinate basis. We therefore also report a rotation-invariant Procrustes diagnostic \(\hat{\kappa}\), computed from the regularized whitened cross-covariance (below). This \(\hat{\kappa}\) is a proxy for cross- and auto-covariance mismatch rather than an unbiased estimator of \(\kappa_{ij}\). Let \(Z_i \in \mathbb{R}^{n_i \times d}\) and \(Z_j \in \mathbb{R}^{n_j \times d}\) be centered representations on probe inputs, with a shared feature dimension \(d\). Unless stated otherwise, we use the same probe set as CKA; if sample counts differ, we form a paired subset of size \(n_{ij}\) with matched inputs. Let \(Z_i^{(ij)}, Z_j^{(ij)} \in \mathbb{R}^{n_{ij} \times d}\) denote those paired rows (in our experiments \(n_i = n_j = n_{ij}\)). Choose \(r_{\mathrm{sub}}\) as the smallest \(r\) such that the top-\(r\) eigenvalues of both \(\hat{\Sigma}_i\) and \(\hat{\Sigma}_j\) explain at least 90% of the trace (equivalently, take the larger of the two 90% ranks). Define \(O(r_{\mathrm{sub}}) := \{Q \in \mathbb{R}^{r_{\mathrm{sub}} \times r_{\mathrm{sub}}} : Q^\top Q = I\}\).

1. Compute sample covariances with regularization: \(\hat{\Sigma}_i^{(\lambda)} = Z_i^\top Z_i / n_i + \lambda I\), \(\hat{\Sigma}_j^{(\lambda)} = Z_j^\top Z_j / n_j + \lambda I\), \(\hat{\Sigma}_{ij} = (Z_i^{(ij)})^\top Z_j^{(ij)} / n_{ij}\). We use \(\lambda \approx 10^{-6}\) for numerical stability.

2. Compute eigendecompositions and extract the leading \(r_{\mathrm{sub}}\)-dimensional subspace bases: take \(\hat{U}_i, \hat{U}_j \in \mathbb{R}^{d \times r_{\mathrm{sub}}}\) from the top-\(r_{\mathrm{sub}}\) eigenvectors of \(\hat{\Sigma}_i\) and \(\hat{\Sigma}_j\).

3. Compute regularized whitened cross-covariance: \(\hat{\tilde{C}} = (\hat{\Sigma}_i^{(\lambda)})^{-1/2} \hat{\Sigma}_{ij} (\hat{\Sigma}_j^{(\lambda)})^{-1/2}\).

4. Project onto the shared subspace: \(\hat{C}_r = \hat{U}_i^\top \hat{\tilde{C}} \hat{U}_j\).

5. Solve the orthogonal Procrustes problem \(Q^\star = \arg\min_{Q \in O(r_{\mathrm{sub}})} \|\hat{C}_r - Q\|_F\) (via SVD). Define \[\hat{\kappa}(i,j) = \frac{\|\hat{C}_r - Q^\star\|_F}{\sqrt{r_{\mathrm{sub}}}}.\]

If the spectrum is simple, sign-aligning the eigenvectors yields \(Q^\star = I\), so \(\hat{\kappa}\) reduces to \(\|\hat{C}_r - I\|_F/\sqrt{r_{\mathrm{sub}}}\). Because \(Q^\star\) solves the Procrustes problem, \(\hat{\kappa}\) depends only on the singular values of \(\hat{C}_r\) and is invariant to orthogonal basis rotations within the chosen subspaces. Small \(\hat{\kappa}\) indicates aligned coupling in the whitened cross-covariance. With the normalization above, uncorrelated representations (whitened cross-covariance near zero in the shared subspace) yield \(\hat{\kappa} \approx 1\).

17.1.0.2 Kronecker independence violation (\(\delta\)).

For head gradients, we use the column-stacking \(\mathrm{vec}\) convention so \(g = a \otimes e\). With row-major vectorization, \(g = e \otimes a\) and the Kronecker factors swap. We estimate the Kronecker approximation error per task \(k\) by comparing the empirical gradient second moment \(\hat{\Sigma}_{\psi,k} = \frac{1}{n_k}\sum_i g_i g_i^\top\) to the Kronecker product \(\hat{M}_{a,k} \otimes \hat{\Gamma}_{e,k}\) (with \(\hat{M}_{a,k} = \frac{1}{n_k}\sum_i a_i a_i^\top\)): \[\hat{\delta}_k = \frac{\|\hat{\Sigma}_{\psi,k} - \hat{M}_{a,k} \otimes \hat{\Gamma}_{e,k}\|_F}{\|\hat{\Sigma}_{\psi,k}\|_F}.\] We set \(\hat{\delta}_k := 0\) when \(\|\hat{\Sigma}_{\psi,k}\|_F = 0\).

17.1.0.3 Block dominance (\(r\)).

We compute layer-wise Fisher blocks \(\hat{\mathcal{F}}_{ll}\) and cross-layer blocks \(\hat{\mathcal{F}}_{lm}\) from gradient samples, then estimate \[\hat{r} = \sqrt{\frac{\sum_{l \neq m} \|\hat{\mathcal{F}}_{lm}\|_F^2}{\sum_l \|\hat{\mathcal{F}}_{ll}\|_F^2}}.\]

17.1.0.4 Error covariance alignment (\(S_{\mathrm{cov}}(\Gamma_e)\)).

For \(K\)-class classification, define the error vector \(e^{(k)} = \hat{p} - \mathbf{1}_y \in \mathbb{R}^K\) and the error second moment \(\Gamma_{e,k} = \mathbb{E}[e^{(k)} e^{(k)\top}]\), a \(K \times K\) matrix. To estimate \(\Gamma_e\): (i) run forward passes on \(n\) samples to obtain softmax outputs \(\hat{p}\) and targets \(y\); (ii) compute \(e = \hat{p} - \mathbf{1}_y\) per sample; (iii) estimate \(\hat{\Gamma}_e = \frac{1}{n}\sum_i e_i e_i^\top\); and (iv) compute \[S_{\mathrm{cov}}(\hat{\Gamma}_{e,i}, \hat{\Gamma}_{e,j}) = \frac{\langle \hat{\Gamma}_{e,i}, \hat{\Gamma}_{e,j} \rangle_F}{\|\hat{\Gamma}_{e,i}\|_F \|\hat{\Gamma}_{e,j}\|_F}.\] Computational cost is \(O(nK^2)\), tractable for ImageNet (\(K{=}1000\)) with \(n \approx 1000\) samples. We set \(S_{\mathrm{cov}}(\hat{\Gamma}_{e,i}, \hat{\Gamma}_{e,j}) := 0\) when \(\|\hat{\Gamma}_{e,i}\|_F = 0\) or \(\|\hat{\Gamma}_{e,j}\|_F = 0\). For very large \(K\), a diagonal proxy \(S_{\mathrm{cov}}(\mathrm{diag}(\Gamma_{e,i}), \mathrm{diag}(\Gamma_{e,j}))\) provides a cheaper \(O(nK)\) approximation, though it is not a general lower bound.

17.2 Diagnostic Experiments↩︎

17.2.0.1 Setup.

We evaluate feedforward networks of varying depth and width on classification tasks. We measure: (i) penultimate-layer CKA, (ii) empirical head Fisher alignment \(A_{\mathcal{F}}^{\text{head}}\), (iii) full-network Fisher alignment \(A_{\mathcal{F}}^{\text{full}}\), and (iv) the structural quantities \(\kappa\) (reported as \(\hat{\kappa}\)), \(\delta\), \(r\), and \(w_{\mathrm{head}}\) (head dominance; see 34 ).

17.2.0.2 CKA–Fisher correlation.

Across 30 task pairs with varying similarity, we observe strong correlation between CKA and full-network Fisher alignment (Pearson corr.\(= 0.90\), Fisher-z 95% CI \([0.80, 0.95]\)) and moderate correlation with head Fisher alignment (Pearson corr.\(= 0.77\), Fisher-z 95% CI \([0.57, 0.88]\)). The correlation between task similarity and CKA (Pearson corr.\(= 0.86\), Fisher-z 95% CI \([0.72, 0.93]\)) confirms that CKA captures meaningful task relationships. When the structural conditions in our theory degrade (e.g., larger \(\hat{\kappa}\) or \(\delta\), or weaker head dominance), the CKA–Fisher gap increases from a minimum of 0.04 to a maximum of 0.51.

17.2.0.3 Coupling strength validation.

We verify that \(\hat{\kappa}\) tracks representation coupling: for highly similar networks \(\hat{\kappa} \approx 0\) (strong coupling), while for uncorrelated representations \(\hat{\kappa} \approx 1\) after normalization (weak coupling). Intermediate task similarities yield intermediate \(\hat{\kappa}\) values as expected.

17.2.0.4 Block dominance across architectures.

We observe that block dominance \(r\) varies dramatically with architecture:

• Shallow-wide (depth 1–2, width \(\geq 32\)): \(r \approx 0.2\)–\(0.5\), \(w_{\mathrm{head}} \approx 0.94\)–\(0.99\).

• Deep networks (depth \(\geq 3\)): \(r > 1\), \(w_{\mathrm{head}} < 0.5\).

These ranges suggest that head dominance (large \(w_{\mathrm{head}}\)) holds reliably only for shallow-wide architectures; deeper networks require the full-network diagnostics in Appendix 23.

17.3 Independence Violation Across Architectures↩︎

The Kronecker independence assumption (\(\delta = 0\)) is violated across all architectures tested (\(\delta \approx 0.34\), Table 11). By Theorem 2, head Fisher alignment satisfies \(A_{\mathrm{F}}^{\mathrm{head}}(i,j) = A_{\mathrm{proxy}}(i,j) \cdot \rho_{ij}\) exactly, where \(\rho_{ij}\) is the coupling correction.

This has two consequences:

1. Empirical validation: Head-level validation uses exact Fisher alignment via the kernel identity (Appendix 9), while full-network LLM validation uses parameter-subsampled gradients across all layers for computational tractability.

2. Negative proxy rank correlation on ViT: On ViT-B/16 (\(n=40\) same-dataset pairs), Pearson \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = -0.85\), and the proxy is negatively correlated with exact Fisher (Pearson corr.\(= -0.34\), Spearman corr.\(= -0.30\); Table 5), indicating an inverted ranking signal. FA-CKA succeeds (Pearson corr.\(= 0.79\), Spearman corr.\(= 0.90\)) because it is validated against exact Fisher.

17.3.0.1 Architecture-dependent coupling.

Independence implies \(\rho_{ij} = 1\) (Corollary 2), but \(\rho \neq 1\) for both CNNs and transformers. The key difference is Pearson \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}})\): positive on ResNet (\(+0.88\), \(n=75\)) and negative on ViT (\(-0.85\), \(n=40\)). Consistent with this, the proxy correlates positively with exact Fisher on ResNet (Pearson corr.\(= 0.98\), Spearman corr.\(= 0.69\); \(n=75\)) and negatively on ViT (Pearson corr.\(= -0.34\), Spearman corr.\(= -0.30\); \(n=40\); Table 5). FA-CKA works regardless because it is validated against exact Fisher, not the proxy.

18 Empirical Distribution of Coupling Correction \(\hat{\rho}\)↩︎

We compute the empirical coupling correction \[\hat{\rho}_{ij} := \frac{A_{\mathrm{F}}^{\mathrm{exact}}(i,j)}{A_{\mathrm{proxy}}(i,j)}, \qquad A_{\mathrm{proxy}}(i,j) = S_{\mathrm{cov}}(M_{a,i}, M_{a,j}) \cdot S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j}),\] using exact head Fisher alignment \(A_{\mathrm{F}}^{\mathrm{exact}}\) (Appendix 9), as a plug-in estimator of \(\rho_{ij}\). We retain pairs satisfying \(A_{\mathrm{F}}^{\mathrm{exact}}(i,j) > \varepsilon\) and \(A_{\mathrm{proxy}}(i,j) > \varepsilon\), using \(\varepsilon = 10^{-6}\). In our ViT/ResNet experiments, this filter keeps same-dataset pairs and excludes cross-dataset pairs under Assumption 1. By Theorem 2, \(\rho_{ij} = 1\) under independence. When coupling does not cancel, \(\rho_{ij}\) deviates from 1; \(\hat{\rho}\) captures that deviation empirically.

For ResNet-50, we use five datasets, three seeds, and two training modes (fine-tuned and linear probe), yielding six tasks per dataset and 15 within-dataset task pairs (total \(n=75\)).

18.0.0.1 Interpretation.

On ViT (\(n=40\)), \(\hat{\rho}\) ranges from 0.35 to 1.11, and Pearson \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = -0.85\). \(\hat{\rho}\) is not constrained to be at most 1; in our runs it reaches 1.11. Empirically, high-proxy pairs tend to have lower \(\hat{\rho}\). Consistent with this, the proxy is negatively correlated with exact Fisher (Pearson corr.\(= -0.34\); Spearman corr.\(= -0.30\); Table 5). On ResNet-50 (\(n=75\)), \(\hat{\rho}\) ranges from 0.15 to 0.99 and Pearson \(\mathrm{Corr}(\hat{\rho}, A_{\mathrm{proxy}}) = +0.88\). The proxy correlates positively with exact Fisher (Pearson corr.\(= +0.98\); Spearman corr.\(= +0.69\)), indicating moderate rank agreement.

18.0.0.2 Why FA-CKA works despite \(\hat{\rho}\) variation.

FA-CKA (defined as \(\mathrm{CKA} \times S_{\mathrm{cov}}(\Gamma_e)\)) correlates well with exact Fisher (Pearson corr.\(= 0.79\); Spearman corr.\(= 0.90\); \(n=40\); Table 5). By contrast, \(A_{\mathrm{proxy}} = S_{\mathrm{cov}}(M_a) \times S_{\mathrm{cov}}(\Gamma_e)\) is negatively correlated (Pearson corr.\(= -0.34\)). Since \(A_{\mathrm{F}}^{\mathrm{exact}} = S_{\mathrm{cov}}(M_a) \cdot S_{\mathrm{cov}}(\Gamma_e) \cdot \hat{\rho}\), we can write \[\frac{\mathrm{FA\text{-}CKA}(i,j)}{A_{\mathrm{F}}^{\mathrm{exact}}(i,j)} = \frac{\mathrm{CKA}(i,j)}{S_{\mathrm{cov}}(M_{a,i}, M_{a,j})} \cdot \frac{1}{\hat{\rho}_{ij}}.\] Two factors explain this. First, on ViT, CKA differs from \(S_{\mathrm{cov}}(M_a)\): centering contributes (Appendix 21), and representation mismatch \(\kappa_{ij}\) can also drive the gap (Theorem 10). Across same-dataset pairs (\(n=40\)), the ratio \(\mathrm{CKA}(i,j)/S_{\mathrm{cov}}(M_{a,i}, M_{a,j})\) averages about 2.28 and is strongly negatively correlated with \(1/\hat{\rho}_{ij}\) (Pearson corr.\(= -0.91\)), which partially offsets coupling effects. Second, \(S_{\mathrm{cov}}(\Gamma_e)\) alone already correlates strongly with \(A_F^{\mathrm{exact}}\) (Pearson corr.\(= 0.78\); \(n=40\)), suggesting that the \(\Gamma_e\) component drives much of the signal. FA-CKA is validated empirically against exact Fisher, rather than justified solely by the \(\rho = 1\) proxy assumption.

18.1 Proof of Remark 4: Proxy Approximation Bound↩︎

We prove that when the coupling tensor \(C_k\) is small relative to the rank-1 part, the separable proxy is guaranteed to be accurate.

Let \(x_k := \mu_{ae,k}\), \(u_k := \mu_{a,k} \otimes \mu_{e,k}\), and \(v_k := C_k\). Then \(x_k = u_k + v_k\), where \(u_k\) is the rank-1 component and \(v_k\) is the coupling residual. Define \(\varepsilon_k := \|v_k\|/\|u_k\|\).

Proof of Remark 4. Combining Lemmas 7 and 8 with \(\eta_k = 2\varepsilon_k/(1-\varepsilon_k)\) yields the stated proxy approximation bound. Note that \(A_{\mathrm{proxy}}(i,j) = \cos(\mu_{a,i} \otimes \mu_{e,i}, \mu_{a,j} \otimes \mu_{e,j}) = \langle \hat{u}_i, \hat{u}_j \rangle\). Also, \(A_F^{\mathrm{head}}(i,j) = \langle \hat{x}_i, \hat{x}_j \rangle\). 0◻

19 RSA and SVCCA are Representation-Only Metrics↩︎

This appendix proves that RSA and SVCCA, like CKA, are representation-only metrics (Definition 1). Therefore, they inherit the same fundamental limitation (Theorem 1).

19.1 RSA (Representational Similarity Analysis)↩︎

This follows directly from Theorem 1 since RSA is a representation-only metric (Proposition 11).

19.2 SVCCA (Singular Vector CCA)↩︎

Implementation details (Appendix K.3). Our experiments use the official SVCCA implementation [12]. It mean-centers activations (via np.cov), prunes low-variance directions below \(\epsilon = 10^{-10}\) (equivalently \(\lambda = 10^{-10}\) in Appendix K.2), and reports the mean canonical correlation up to the default truncation threshold \(0.98\). A standard 99% variance SVD truncation [12] yields the same representation-only conclusion.

19.3 Empirical Verification↩︎

We verify these theoretical results numerically:

19.3.0.1 Implementation details.

Test 1 uses \(n=200\), \(d=64\), \(K=10\), seed \(=100\). Test 2 (table) uses \(n=500\), \(d=128\), seed \(=42\) (reset for each \(K\)) with \(K \in \{5,10,20,50,100\}\). Following the official implementation, SVCCA uses \(\epsilon = 10^{-10}\) (equivalently \(\lambda = 10^{-10}\)) and the default truncation threshold \(0.98\).

19.3.0.2 Test 1: Invariance to error structure.

For two task pairs with identical representations but different error covariances:

• Pair 1: \(S_{\text{cov}}(\Gamma_{e,1}) = 0.999\)

• Pair 2: \(S_{\text{cov}}(\Gamma_{e,2}) = 0.000\) (disjoint classes)

Result: \(|\text{RSA(pair 1)} - \text{RSA(pair 2)}| < 10^{-8}\) and \(|\text{SVCCA(pair 1)} - \text{SVCCA(pair 2)}| < 10^{-8}\).

Both metrics are completely invariant to error structure, as predicted by the theory.

19.3.0.3 Test 2: RSA-Fisher and SVCCA-Fisher gaps.

For disjoint \(K\)-class tasks with shared representations:

Both RSA and SVCCA exhibit the same pathological gap as CKA: perfect similarity despite orthogonal gradients.

20 Explicit Perturbation Bound for Alignment↩︎

This appendix states a unit-normalization perturbation bound (with constant \(2\)) used in later alignment bounds.

If we instead define \(\varepsilon := \|r\|/\|x_0\| < 1\), then \(\|x\| = \|x_0 + r\| \ge \|x_0\| - \|r\| = (1-\varepsilon)\|x_0\|\). Consequently, \[\left\|\frac{x}{\|x\|} - \frac{x_0}{\|x_0\|}\right\| \le \frac{2\varepsilon}{1-\varepsilon}.\]

20.0.0.1 Interpretation.

In the head-Fisher setting, set \(\Sigma_k^{(0)} := M_{a,k} \otimes \Gamma_{e,k}\). Then \(R_k\) is the coupling residual from Remark 3. Under this identification, \(\varepsilon_{R,k}\) matches the reported \(\delta \approx 0.34\). The corollary then yields a worst-case bound of \(2(0.34 + 0.34) = 1.36\). Because \(S_{\mathrm{cov}}(\cdot,\cdot) \in [0,1]\) for PSD matrices, we may clip the bound at \(1\). The worst-case term \(2(\varepsilon_{R,i} + \varepsilon_{R,j})\) can be vacuous. Empirically, observed deviations are typically much smaller (see Appendix 18).

21 Proof of the Mean-Term Correction Lemma↩︎

This appendix proves Lemma 10, which quantifies the gap between covariance alignment computed from centered and uncentered second moments.

21.0.0.1 Remark (tighter denominator).

Lemma 9 directly gives \(\|u-u_0\|_F \le 2\|\Delta_A\|_F/\|A+\Delta_A\|_F\) and similarly for \(v\). Define \(\tilde{\varepsilon}_{\mu,k} := \|\mu_{a,k}\|_2^2 / \|M_{a,k}\|_F\). Then the always-valid bound \(\big|S_{\mathrm{cov}}(M_{a,i}, M_{a,j}) - S_{\mathrm{cov}}(\Sigma_{a,i}, \Sigma_{a,j})\big| \le 2(\tilde{\varepsilon}_{\mu,i} + \tilde{\varepsilon}_{\mu,j})\). Since \(\|M_{a,k}\|_F \ge \|\Sigma_{a,k}\|_F\) under \(\Sigma_{a,k} \succeq 0\), this bound is never looser than the stated bound.

21.0.0.2 Interpretation.

If \(\varepsilon_{\mu,k}\) is large, the bound can be vacuous (i.e., greater than \(1\)), but it still quantifies a worst-case impact of mean terms; reporting \(\varepsilon_{\mu,k}\) keeps the approximation auditable.

21.0.0.3 Empirical mean-term sizes (ViT-B/16).

Using the ViT-B/16 validation data from Appendix 18 (20 tasks; 40 same-dataset pairs), we find \(\varepsilon_{\mu,k} \in [0.78, 1.35]\) with a median of \(0.96\). Accordingly, the worst-case bound \(2(\varepsilon_{\mu,i}+\varepsilon_{\mu,j})\) ranges from \(3.16\) to \(5.29\) across same-dataset pairs (vacuous). The observed gap \(|S_{\mathrm{cov}}(M_{a,i},M_{a,j}) - S_{\mathrm{cov}}(\Sigma_{a,i},\Sigma_{a,j})|\) ranges from \(0.007\) to \(0.147\) (median \(0.085\)).

22 Proof of Theorem 1↩︎

This appendix provides a full proof of Theorem 1, showing that representation-only metrics are fundamentally blind to error structure.

22.1 Full Proofs for CKA Non-Identifiability↩︎

This section provides the complete proofs of the linear CKA identity (Eq. 2 ) and Corollary 1, which appear in the main text.

23 Extension to Full Networks↩︎

The main text establishes a rigorous bridge between CKA and head (final layer) Fisher alignment. This appendix provides exact structural decompositions that relate full-network Fisher alignment to head Fisher alignment, with measurable diagnostics that explain when and why the reduction works.

23.0.0.1 Scope.

The block-diagonal/off-diagonal decompositions below are purely algebraic and hold for any parameter partition. The exact three-factor identity in Section 23.6 additionally assumes that per-layer gradients are rank-1 (affine layers without parameter sharing). For convolutional or attention layers, the identity changes because per-example gradients are sums of outer products across positions. Appendix 11 gives the exact shared-parameter kernel identity and an unbiased sketch. Our empirical validation in this appendix relies on the block-decomposition diagnostics.

3pt

23.1 Full-Network Fisher Block Structure↩︎

Consider an \(L\)-layer network with parameters \(\theta = (\theta_1, \ldots, \theta_L)\) where \(\theta_l = \mathrm{vec}(W_l)\). The full-network Fisher matrix has block structure: \[\mathcal{F} = \begin{bmatrix} \mathcal{F}_{11} & \mathcal{F}_{12} & \cdots & \mathcal{F}_{1L} \\ \mathcal{F}_{21} & \mathcal{F}_{22} & \cdots & \mathcal{F}_{2L} \\ \vdots & \vdots & \ddots & \vdots \\ \mathcal{F}_{L1} & \mathcal{F}_{L2} & \cdots & \mathcal{F}_{LL} \end{bmatrix}, \label{eq:full-fisher-blocks-app}\tag{18}\] where \(\mathcal{F}_{ll} = \mathbb{E}[g_l g_l^\top]\) are the layer-wise Fisher blocks and \(\mathcal{F}_{lm} = \mathbb{E}[g_l g_m^\top]\) for \(l \neq m\) are cross-layer blocks.

23.1.0.1 Exact algebraic split.

Define, for each task \(k\):

• Block-diagonal part: \(D_k := \mathrm{blkdiag}(\mathcal{F}^{(k)}_{11}, \ldots, \mathcal{F}^{(k)}_{LL})\)

• Off-diagonal part: \(O_k := \mathcal{F}^{(k)} - D_k\)

These have disjoint support in block coordinates: \(D_k\) is nonzero only on diagonal blocks \((\ell, \ell)\), while \(O_k\) is nonzero only on off-diagonal blocks \((\ell, m)\) with \(\ell \neq m\). This implies \(\langle D_k, O_k \rangle_F = 0\) and \(\|\mathcal{F}^{(k)}\|_F^2 = \|D_k\|_F^2 + \|O_k\|_F^2\).

23.1.0.2 Degenerate cases.

If any denominator is zero, we use \(\varepsilon\)-regularized norms \(\|X\|_{F,\varepsilon} := \max(\|X\|_F, \varepsilon)\) and set the corresponding alignment to \(\langle X,Y\rangle_F/(\|X\|_{F,\varepsilon}\|Y\|_{F,\varepsilon})\) (which yields \(0\) when \(X=0\) or \(Y=0\)). Weights are computed directly from norms via the first equalities in 23 and 34 ; ratios \(r_k,\tau_k\) are used only when denominators are nonzero.

23.2 Step 1: Exact Decomposition for Full-to-Block-Diagonal↩︎

The key insight is that mixed terms vanish by disjoint support, yielding an exact weighted sum with nonnegative weights.

23.2.0.1 Exact gap identity.

Rewriting the weighted sum using the profile cosine: \[A_F^{\mathrm{full}} = (w_{\mathrm{blk}} + w_{\mathrm{off}}) A_{\mathrm{blk}} + w_{\mathrm{off}}(A_{\mathrm{off}} - A_{\mathrm{blk}}) = c_r \cdot A_{\mathrm{blk}} + w_{\mathrm{off}}(A_{\mathrm{off}} - A_{\mathrm{blk}}).\] Thus the gap has the exact form: \[\boxed{ A_F^{\mathrm{full}} - A_{\mathrm{blk}} = (c_r - 1) A_{\mathrm{blk}} + w_{\mathrm{off}} (A_{\mathrm{off}} - A_{\mathrm{blk}}) } \label{eq:full-gap-exact}\tag{28}\]

23.2.0.2 Why this works for deep nets.

The bound has two measurable terms; each is small when:

• Profile mismatch: \((1 - c_r)\) is small when \(r_i \approx r_j\) (tasks have similar cross-layer energy profiles).

• Adversarial off-diagonal: \(w_{\mathrm{off}} \cdot \Delta_{\mathrm{off}}\) is small when \(\Delta_{\mathrm{off}}\) is small (off-diagonal alignment tracks block-diagonal).

When tasks share similar architectures and training regimes, both conditions typically hold.

23.3 Step 2: Block-Diagonal to Head Fisher↩︎

Now split the block-diagonal part into head vs.rest:

• Head block: \(H_k := \mathcal{F}^{(k)}_{LL}\)

• Non-head blocks: \(R_k := \mathrm{blkdiag}(\mathcal{F}^{(k)}_{11}, \ldots, \mathcal{F}^{(k)}_{L-1,L-1})\)

Again, these have disjoint support: \(\langle H_k, R_k \rangle_F = 0\) and \(\|D_k\|_F^2 = \|H_k\|_F^2 + \|R_k\|_F^2\).

Since \(H_k\) is the head block, \(A_{\mathrm{head}}(i,j)\) coincides with the head Fisher alignment \(A_F^{\mathrm{head}}(i,j)\).

23.3.0.1 Exact gap identity.

\[\boxed{ A_{\mathrm{blk}} - A_{\mathrm{head}} = (c_\tau - 1) A_{\mathrm{head}} + w_{\mathrm{rest}} (A_{\mathrm{rest}} - A_{\mathrm{head}}) } \label{eq:blk-gap-exact}\tag{38}\]

23.4 Combined Full-Network Theorem↩︎

Whenever a head-layer bound of the form \(|\mathrm{CKA} - A_{\mathrm{head}}| \le \varepsilon_{\mathrm{head}}\) is available, it plugs directly into 41 to yield a closed-form CKA-to-full-network bound (e.g., Theorem 10 in Appendix 15 under matched probe/input distributions).

23.4.0.1 Why this theorem “works” for deep nets.

Unlike worst-case bounds that become vacuous when \(r, \tau > 1\), this bound remains informative because it has four measurable terms that are small when:

• Cross-layer profile mismatch: \((1 - c_r)\) is small when \(r_i \approx r_j\).

• Head/rest profile mismatch: \((1 - c_\tau)\) is small when \(\tau_i \approx \tau_j\).

• Adversarial off-diagonal: \(w_{\mathrm{off}} \cdot \Delta_{\mathrm{off}}\) is small when \(\Delta_{\mathrm{off}}\) is small.

• Adversarial non-head: \(w_{\mathrm{rest}} \cdot \Delta_{\mathrm{rest}}\) is small when \(\Delta_{\mathrm{rest}}\) is small.

When tasks share similar architectures/training (so \(r_i \approx r_j\), \(\tau_i \approx \tau_j\)), the profile cosines \(c_r, c_\tau \approx 1\). When the network exhibits consistent alignment patterns across blocks, \(\Delta_{\mathrm{off}}, \Delta_{\mathrm{rest}}\) are small. Both conditions typically hold in practice.

23.5 Estimating Diagnostics from Gradient Samples↩︎

All quantities can be estimated from mini-batch gradient samples:

23.5.0.1 Weights and profile cosines.

For each layer \(l\), collect per-example gradients \(g_l^{(t)}\). Estimate Fisher block norms: \[\|\hat{\mathcal{F}}_{lm}\|_F^2 \approx \left\|\frac{1}{n}\sum_{t=1}^n g_l^{(t)} g_m^{(t)\top}\right\|_F^2.\] Then compute \(w_{\mathrm{blk}}, w_{\mathrm{off}}, c_r\) from 23 –26 and \(w_{\mathrm{head}}, w_{\mathrm{rest}}, c_\tau\) from 34 –36 . This plug-in estimator is biased for finite \(n\); an unbiased alternative is the U-statistic \(\frac{1}{n(n-1)}\sum_{t\neq t'} \langle g_l^{(t)}, g_l^{(t')}\rangle \langle g_m^{(t)}, g_m^{(t')}\rangle\).

23.5.0.2 Discrepancies \(\Delta_{\mathrm{off}}, \Delta_{\mathrm{rest}}\).

Compute cross-task alignments \(A_{\mathrm{off}}, A_{\mathrm{rest}}, A_{\mathrm{blk}}, A_{\mathrm{head}}\) from gradient inner products, then take differences.

23.6 Exact Full-Network Decomposition with Depth Coupling↩︎

While the previous subsections provide bounds relating full-network Fisher to head Fisher, here we establish an exact identity analogous to Theorem 2 but for the full network.

23.6.0.1 Rank-1 per-layer gradients.

For affine layers applied to a single activation vector (no parameter sharing), the per-example gradient satisfies \(g_\ell=\mathrm{vec}(\delta_\ell a_{\ell-1}^\top)\) (Lemma 1). In this rank-1 setting, the full gradient inner product factorizes across depth: \[g^{(i)\top} g^{(j)} = \sum_{\ell=1}^L \langle a_{\ell-1}^{(i)}, a_{\ell-1}^{(j)} \rangle \cdot \langle \delta_\ell^{(i)}, \delta_\ell^{(j)} \rangle = \sum_{\ell=1}^L u_\ell v_\ell = u^\top v. \label{eq:gradient-depth-product}\tag{42}\] Biases can be incorporated by augmenting activations with a constant \(1\), as in the head-layer analysis.

23.6.0.2 Geometric interpretation.

The coupling coefficient \(\rho^{\mathrm{full}}\) measures depth interference: whether forward and backward similarity profiles align in depth space. Since \((u^\top v)^2 = \|u\|^2 \|v\|^2 \cos^2\theta\) where \(\theta\) is the angle between \(u\) and \(v\): \[\chi^{\mathrm{full}}_{ij} = \frac{\mathbb{E}_{i,j}[\|u\|^2 \|v\|^2 \cos^2\theta]}{\mathbb{E}_{i,j}[\|u\|^2] \cdot \mathbb{E}_{i,j}[\|v\|^2]}.\] When forward and backward similarities co-localize in depth (all layers contribute similarly), \(\cos\theta \approx 1\) yields constructive interference. Misalignment across depth yields destructive interference.

23.6.0.3 Relation to profile diagnostics.

The profile cosine \(c_r\) (Definition 4) and discrepancy \(\Delta_{\mathrm{off}}\) (Definition 5) from the bound analysis are indirect measures of depth coupling. Corollary 7 bounds the gap between full and block-diagonal Fisher; the exact theorem above shows the gap is controlled by \(\rho^{\mathrm{full}}\) deviating from the head-layer coupling.

23.6.0.4 Computational note.

The head theorem (Theorem 2, \(L=1\)) is a special case: the depth profiles reduce to scalars \((u, v \in \mathbb{R}^1)\), so \(u^\top v = uv\) and \(\rho^{\mathrm{full}} = \rho\).

23.7 Controlled Transfer Validation (CIFAR-100)↩︎

We validate that \(S_{\mathrm{cov}}(\Gamma_e)\) predicts transfer in a controlled setting where class overlap can be precisely manipulated.

23.7.0.1 Setup.

Using ResNet-18 on CIFAR-100, we train a source task on classes 0–49 (fixed), then measure linear probe transfer to target tasks with varying class overlap (0%, 24%, 50%, 74%, 100%; i.e., 0/50, 12/50, 25/50, 37/50, 50/50 shared). Linear probe transfer—training only the classification head while freezing the backbone—matches the head-layer theory assumptions. Because the backbone is shared, representation similarity is constant (CKA \(\approx 1\)), so any variation in transfer must come from error geometry. We compare \(S_{\mathrm{cov}}(\Gamma_e)\) against standard transferability metrics (LogME, LEEP, H-score, NCE) [16], [17], [50], [51] that require explicit label information.

23.7.0.2 Results.

Table ¿tbl:tab:controlled-transfer? shows that \(S_{\mathrm{cov}}(\Gamma_e)\) tracks class overlap (0.00 at 0% overlap, 1.00 at 100%) and predicts linear probe transfer gain as well as dedicated transferability metrics.

23.7.0.3 Why CKA fails.

With a frozen backbone, CKA is constant by construction (CKA \(\approx 1.00\)), so it cannot discriminate transfer quality: all source-target pairs appear identical to representation-only metrics regardless of label compatibility. \(S_{\mathrm{cov}}(\Gamma_e)\) captures the error structure that CKA misses, enabling meaningful source ranking.

23.7.0.4 Baseline comparison.

All metrics achieve high correlation with transfer gain (\(\rho > 0.96\)) because overlap is monotonically related to transfer. The key advantage of \(S_{\mathrm{cov}}(\Gamma_e)\) is scalability: standard LogME/LEEP-style finite-label implementations require explicit label maps or class statistics and are not designed to estimate full vocabulary-scale error covariance; FisherSketch estimates the corresponding joint update geometry in one pass without materializing \(\Gamma_e\), using \(O(d+K+m)\) working memory plus projection state.

23.7.0.5 Limitation.

With only 5 overlap levels, correlation comparisons have limited statistical power. The vocab-scale experiment (100 domains, 24 candidates/target, 4,188 transfer pairs) provides stronger evidence.

23.8 Vision Transformer Validation↩︎

We validate the full-network decomposition (Corollary 7) on Vision Transformers to address concerns about modern attention-based architectures.

23.8.0.1 Experimental setup.

We fine-tune ViT-B/16 [52] (pretrained on ImageNet-21k) on four downstream classification tasks: CIFAR-100 (100 classes), Flowers102 (102 classes), Oxford Pets (37 classes), and DTD (47 classes). Each task is trained with 5 random seeds, yielding 20 models total. We analyze 50 pairwise task comparisons (40 same-dataset pairs with different seeds, 10 cross-dataset pairs, subsampled). Images are resized to \(224 \times 224\), representing a substantial scale increase from the CIFAR-32 experiments in the main text.

23.8.0.2 Full parameter partition.

Unlike prior work that tracks only the classification head, we collect per-example gradients across the complete parameter partition: patch embedding, all 12 transformer blocks, final layer norm, and classification head (14 parameter groups total). This ensures our validation matches the theorem’s assumptions—no layers are omitted. Gradients are computed in float64 precision with 1000 samples per task to ensure numerical stability.

23.8.0.3 Head gradient comparability across datasets.

To compare head gradients across datasets with different class counts (\(K_{\text{CIFAR}}=100\), \(K_{\text{Flowers}}=102\), \(K_{\text{Pets}}=37\), \(K_{\text{DTD}}=47\)), we embed all error and prediction vectors into a shared \(K_{\mathrm{shared}} = 286\) dimensional space. Each dataset occupies a contiguous, disjoint block: CIFAR-100 uses coordinates \([0, 100)\), Flowers102 uses \([100, 202)\), Oxford Pets uses \([202, 239)\), and DTD uses \([239, 286)\). Predictions use masked softmax (logits set to \(-\infty\) outside their respective blocks) and one-hot labels are embedded via the shared-output injection, yielding 286-dimensional error vectors with zeros outside their respective blocks. Head gradients thus have uniform dimension \(K_{\mathrm{shared}} \times d + K_{\mathrm{shared}} = 286 \times 768 + 286 = 219\,934\) parameters. For cross-dataset pairs, this construction yields exactly zero Fisher alignment: the error vectors have disjoint support, so \(\langle e_i, e_j \rangle = 0\) for all sample pairs, making head gradient inner products zero.

23.8.0.4 Key finding: profile cosines remain near-unity.

Despite the attention-based architecture and larger scale, profile cosines \(c_r\) remain extremely high (Table 14): \[c_r = 0.9995 \pm 0.0011 \quad \text{(mean \pm std, n=50)}.\] This confirms that ViT models trained on related fine-tuning tasks share similar cross-layer energy profiles, even though the absolute energy ratio \(r \approx 2.75\) substantially exceeds unity (off-diagonal energy exceeds diagonal energy by \(\sim\)​2.7\(\times\)). The deterministic bound is satisfied for all 50 pairs (numerical check), and the mean gap (0.077) closely matches the mean bound (0.078; Table 14).

23.8.0.5 Constructed orthogonality under shared-output embedding (sanity check).

Table 4 (main text) provides an illustrative regime on ViT: for cross-dataset pairs (e.g., CIFAR-100 vs.Flowers102), the error structures are orthogonal because disjoint label spaces are embedded via the masked-softmax shared-output embedding (logits set to \(-\infty\) outside each block). CKA remains moderately high (\(\approx 0.57\)) because representations share similar geometry, but \(S_{\mathrm{cov}}(\Gamma_e) = 0\) and FA-CKA drops to zero, as required by the construction. (This orthogonality is with respect to the disjoint-block embedding of label spaces; alternative embeddings would define different cross-task error geometry.)

23.8.0.6 Kronecker proxy vs.exact head Fisher.

On the 40 same-dataset pairs (where exact Fisher is nonzero), we compare the Kronecker proxy with exact head Fisher computed via the kernel trick (Appendix 9). The Kronecker proxy is anti-correlated with exact Fisher (\(r = -0.34\), \(\rho = -0.30\)), while FA-CKA maintains strong correlation (\(r = 0.79\), \(\rho = 0.90\)); see Table 5. This confirms that FA-CKA’s validity does not depend on the Kronecker approximation.

23.9 Large Language Model Validation↩︎

We extend the full-network decomposition validation to Large Language Models, demonstrating that Corollary 7 generalizes beyond vision architectures to modern autoregressive transformers.

23.9.0.1 Experimental setup.

We evaluate on two architectures: Llama-3.1-8B [53] (32 layers, 4096 hidden dim) and Qwen2.5-14B [54] (48 layers, 5120 hidden dim). Using 30 MMLU subjects as tasks provides a natural multi-task setting with shared \(K=4\) output space (4-way multiple choice), eliminating masked-softmax block-embedding confounds. For each model, we sample 100 task pairs from the 435 possible combinations, collecting 100 gradient samples per subject across the full parameter partition: embeddings, all transformer layers, final layer norm, and a 4-way classification head.

23.9.0.2 Empirical tightness of the deterministic bound.

Despite architectural differences (32 vs. layers, different training objectives), both models show \(c_r \approx 0.99\) and mean gaps 0.332/0.243 (Table 15). The inequality itself is deterministic, so satisfaction is a numerical check rather than a discovery: \[|A_F^{\mathrm{full}} - A_{\mathrm{blk}}| \leq (1 - c_r) + w_{\mathrm{off}} \Delta_{\mathrm{off}} \quad \text{for all 100 pairs.}\] High profile cosines indicate consistent depth profiles in residual-stream architectures.

23.9.0.3 Scale validation: 70B parameters.

To validate that the decomposition extends to frontier-scale models, we evaluate on 8 models at 67–72B parameters across three architectures (Table 16). Profile cosines remain high (\(c_r \approx 0.98\)) and gaps range 0.158–0.464 across models; bound satisfaction is a numerical check rather than a discovery. This suggests the residual stream geometry remains consistent even at 10\(\times\) scale.

23.9.0.4 Comparison with ViT.

LLMs exhibit higher \(\Delta_{\mathrm{off}}\) than ViT (0.25–0.75 vs.), reflecting more cross-layer interaction variance in deeper models (32–80 layers vs. layers). However, the bound remains tight: mean gap tracks mean bound closely across all 10 models tested. Bound satisfaction is expected from the inequality; the key insight is that profile cosines \(c_r \approx 0.98\)–\(0.99\) dominate the bound, making the block-diagonal reduction reliable even when \(\Delta_{\mathrm{off}}\) is elevated.

23.9.0.5 Vocabulary-scale transfer validation.

To validate FisherSketch at vocabulary scale, we measure correlation between FisherSketch-estimated Fisher alignment and actual LoRA transfer success. Protocol: (1) Compute base perplexity for each domain; (2) fine-tune LoRA adapter on source domain (rank=8, \(\alpha\)=16, up to 50 steps with early stopping on source-dev loss, patience=3, lr=\(5 \times 10^{-5}\)); (3) evaluate on target domain; (4) compute normalized transfer score in loss space: \((\log \text{base}_{\text{ppl}} - \log \text{transfer}_{\text{ppl}}) / (\log \text{base}_{\text{ppl}} - \log \text{self}_{\text{ppl}})\), with a small denominator floor (0.01) for stability. Negative transfer (transfer PPL \(>\) base PPL) can occur; the normalized score handles this naturally. Setup: 100 domains constructed from eight HF datasets (Appendix [app:vocab-scale-data]), 500 samples/domain for FisherSketch (\(m\)=4096). LoRA transfer uses 200 train / 100 eval samples per domain; we evaluate 4,188 cross-domain pairs (union of uniform + stratified candidates) and report top-1 under the uniform 24-candidate protocol. Sketching and LoRA splits are disjoint by construction; candidate sets and sketch sample indices are saved for audit. Result: FisherSketch with SRHT error projections achieves 45.7% \(\pm\) 5.0 top-1 source selection (\(\approx\)​10.9\(\times\) random) with max regret 0.119 \(\pm\) 0.013 (mean \(\pm\) std over seeds 43–45), confirming that FisherSketch predicts cross-domain transfer at vocabulary scale (\(K\)=128,256) where storing \(\Gamma_e\) scales as \(O(K^2)\). Per-target Spearman corr.\(= 0.47 \pm 0.02\); global off-diagonal Spearman corr.\(= 0.47 \pm 0.01\) is lower because it mixes heterogeneous task families. A diagonal error-geometry proxy (cosine between \(\mathrm{diag}(\Gamma_e)\) vectors; LEEP-style) is weaker: 44.0% \(\pm\) 2.2 top-1 and max regret 0.210 \(\pm\) 0.011 under the same uniform-24 protocol, indicating off-diagonal error covariance carries signal at vocab scale. Sketch-only stability with fixed LoRA outcomes is moderate: across SRHT seeds 43–45, pairwise top-1 agreement is 0.51 \(\pm\) 0.02 and off-diagonal Spearman is 0.982 \(\pm\) 0.003.

23.10 Empirical Findings↩︎

23.10.0.1 What we measure.

• Shallow-wide networks (depth 1–2): \(c_r \approx 0.95\)–\(0.99\), \(c_\tau \approx 0.95\)–\(0.99\). Profile cosines close to 1 mean tasks have similar energy profiles.

• Deep networks (depth \(\geq 3\)): \(c_r, c_\tau\) can be lower when tasks have different \(r, \tau\) values, but this does not imply the bound is vacuous.

• Vision Transformers (ViT-B/16, Table 14): \(c_r = 0.9995\), with mean gap 0.077 vs bound 0.078 across 50 task pairs. Full 14-layer partition (patch embedding through classification head) confirms profile alignment persists in modern attention-based architectures.

• Large Language Models (8B–70B, Tables 15–16): \(c_r \approx 0.95\)–\(1.00\), with mean gaps 0.158–0.464 across 8B–70B models. The residual stream architecture maintains geometric consistency from 32 to 80 transformer layers.

23.10.0.2 The key finding: discrepancies are small.

Even when \(c_r, c_\tau < 1\) (deep nets with varying energy profiles), we observe:

• \(\Delta_{\mathrm{rest}} \approx 0.10\)–\(0.20\): non-head alignment tracks head alignment.

• ViT-B/16: \(\Delta_{\mathrm{off}} \approx 0.05\)–\(0.15\) (mean 0.088), with mean gap 0.077 vs bound 0.078 across the full 14-layer parameter partition (Figure 5).

• LLMs (8B–70B): \(\Delta_{\mathrm{off}} = 0.25\)–\(0.75\) (higher due to deeper architectures), with gaps 0.158–0.464; high profile cosines (\(c_r \approx 0.95\)–\(1.00\)) keep the bound tight.

This explains why, in residual-stream architectures with aligned profile cosines, CKA can correlate strongly with full-network Fisher (\(r > 0.89\)) in our validation suite (Vision Transformers and Large Language Models). This does not contradict non-identifiability for transfer: when error geometry differs, CKA can remain high while head alignment (and transfer) collapse.

23.10.0.3 Key takeaway.

The exact decompositions 27 and 37 show that what matters is not “\(r < 1\)” but “\(c_r \approx 1\) and \(\Delta_{\mathrm{off}}\) small.” Our theory predicts exactly what to measure: the profile cosines \(c_r, c_\tau\) and discrepancies \(\Delta_{\mathrm{off}}, \Delta_{\mathrm{rest}}\). When these are favorable, the head-layer analysis transfers to full-network Fisher—and empirically, they are.

24 Why Error Geometry Ranks While Activation Geometry Screens↩︎

This appendix provides a stylized explanation for the empirical asymmetry observed in Section 6.1: the error-geometry score \(S_{\mathrm{cov}}(\Gamma_e)\) can yield higher top-1 source selection accuracy, while incorporating activation geometry reduces worst-case regret. We model source selection for a fixed target task as follows.

Let \(\mathcal{S}\) be a finite set of candidate sources. For each \(s\in\mathcal{S}\) define a nonnegative error score \(E_s\in(0,1]\) (e.g., \(E_s := S_{\mathrm{cov}}(\Gamma_{e,s},\Gamma_{e,t})\)). Define also a nonnegative compatibility factor \(A_s\ge 0\) (e.g., an activation/coupling term such as \(S_{\mathrm{cov}}(M_{a,s},M_{a,t})\), optionally folding \(\rho\) into \(A_s\)). The product score is \(P_s := A_s E_s\). We compare the error-only selector \(\hat{s}_E := \arg\max_{s\in\mathcal{S}} E_s\) and the product selector \(\hat{s}_P := \arg\max_{s\in\mathcal{S}} P_s\).

25 LEEP as a Diagonal Error-Geometry Proxy↩︎

We formalize the claim that LEEP [16] leverages error geometry. The statement below is deliberately about LEEP, not LogME: LogME is used elsewhere as a finite-label evidence baseline, while this appendix proves only a stylized LEEP–error-geometry connection. Empirically, the diagonal proxy underperforms at vocabulary scale (top-1 accuracy 44.0% \(\pm\) 2.2; max regret 0.210 \(\pm\) 0.011; Appendix 10.11), consistent with the theory below. Let \((X,Y)\sim\mathcal{D}\) be the target distribution with \(Y\in[K]\), and let \(p(\cdot\mid x)\in\Delta^{K-1}\) be a source classifier’s predictive distribution. Let \(Z\mid X \sim \mathrm{Cat}(p(\cdot\mid X))\). Define the soft confusion matrix \(C_{y,z} := \mathbb{E}[\mathbf{1}\{Y=y\}\cdot p(z\mid X)]\). Define the marginal \(q_z := \sum_y C_{y,z}\). Then \(C_{y,z} = \mathbb{P}(Y=y, Z=z)\) and \(q_z = \mathbb{P}(Z=z)\). Assume \(q_z>0\), and define \(p(y\mid z) := C_{y,z}/q_z = \mathbb{P}(Y=y\mid Z=z)\). Assume \(p(y\mid z) > 0\) whenever \(\mathbb{P}(Y=y, Z=z) > 0\) (or define \(\mathrm{LEEP}=-\infty\) otherwise). The population LEEP score is \[\label{eq:leep-def} \mathrm{LEEP}(p;\mathcal{D}) := \mathbb{E}_{X,Y}\log\sum_{z=1}^K p(Y\mid z)\,p(z\mid X) = \mathbb{E}_{X,Y}\log \mathbb{E}_{Z\mid X}[p(Y\mid Z)].\tag{44}\]

This shows the conditional-entropy lower bound (and LEEP in the hard-prediction limit) is a monotone function of a diagonal error second moment under the uniform-confusion model. In this stylized regime, LEEP is a “diagonal proxy for error geometry” in a formal sense. This formal connection should not be read as a derivation of LogME; LogME remains a finite-label evidence baseline in our comparisons.

26 Task Signatures and Retrieval↩︎

Algorithm 3 produces per-task mean embeddings \(\hat{\mu}_{ae,k} \in \mathbb{R}^m\) as a direct byproduct of computing pairwise Fisher alignments. These embeddings, which we call task signatures, enable efficient indexing, retrieval, and dynamic updates without recomputing all pairwise alignments.

26.1 Signature Storage↩︎

26.1.0.1 Storage footprint.

Each task signature consists of the joint embedding \(\hat{\mu}_{ae} \in \mathbb{R}^m\) stored in float32: \[\text{Storage per task} = m \times 4\,\text{bytes} = 16\,\text{KB at } m{=}4096.\] Optionally, the factorized components \((\hat{\mu}_a, \hat{\mu}_e)\) can be stored for \(S_{\mathrm{cov}}(\Gamma_e)\) screening, yielding 48 KB per task.

26.1.0.2 Compute-time footprint.

During streaming, our implementation maintains six float64 accumulators per task for split-sample estimation (Appendix 17); Algorithm 3 shows the simplified single-embedding update. This requires: \[\text{Streaming state per task} = 6 \times m \times 8\,\text{bytes} = 192\,\text{KB at } m{=}4096.\] This is transient during signature construction; once signatures are computed, retrieval only requires the float32 \(\hat{\mu}_{ae}\) vectors (16 KB per task). If component embeddings \((\hat{\mu}_a,\hat{\mu}_e)\) are also stored (as in FisherAtlas by default), persistent per-task storage is 48 KB, and the FAISS index stores another copy of \(\hat{\mu}_{ae}\) for fast search.

26.1.0.3 Comparison to Task2Vec.

Task2Vec [20] stores the diagonal of the Fisher information matrix, requiring \(O(p)\) storage. Here \(p\) is the number of parameters. For a 70B-parameter model, this is approximately 280 GB per task (float32). FisherSketch compresses this to 16 KB—a factor of \({\sim}10^7\) reduction—while preserving pairwise alignment structure via the product-kernel embedding.

26.2 Retrieval Procedure↩︎

26.2.0.1 Prompt signature.

Given a held-out prompt, we compute its signature by streaming over token positions using next-token prediction as the self-supervised objective. Specifically, for a prompt with \(T\) tokens, we extract \((a_t, e_t)\) pairs for \(t = 1, \ldots, T-1\). Here \(a_t\) is the hidden state before the head, and \(e_t = \mathrm{softmax}(W_{\mathrm{head}} a_t) - \mathbf{1}_{y_{t+1}}\) is the prediction error. The prompt signature is: \[\hat{\mu}_{\mathrm{prompt}} = \frac{1}{T-1} \sum_{t=1}^{T-1} \Psi_m(a_t, e_t),\] where \(\Psi_m\) is the factored Random Maclaurin feature map (Section 10).

26.2.0.2 Nearest-neighbor retrieval.

Given an index of \(K\) task signatures \(\{\hat{\mu}_{ae,k}\}_{k=1}^K\), retrieval returns: \[k^* = \arg\max_{k \in [K]} \cos(\hat{\mu}_{\mathrm{prompt}}, \hat{\mu}_{ae,k}) = \arg\max_{k \in [K]} \hat{\mu}_{\mathrm{prompt}}^\top \hat{\mu}_{ae,k}.\] Since all embeddings are \(\ell_2\)-normalized, retrieval reduces to a single matrix-vector product.

26.3 Retrieval Evaluation↩︎

We evaluate training-free task retrieval on 700 domains from Natural Instructions [55] using Llama-3.1-8B signatures at \(m{=}4096\).

26.3.0.1 Protocol.

For each domain, we compute a signature from 200 training samples. Evaluation uses held-out prompts (10 per domain, 6974 total). For each held-out prompt, we compute its signature and retrieve the nearest domain.

26.3.0.2 Results.

FisherSketch achieves 55.5% top-1 accuracy without any training, compared to 51.7% for mean-hidden baselines. Learned probes achieve higher accuracy (62.9%) but require labeled training data and cannot support open-set addition. A pairwise diagnostic over all 244,650 domain pairs shows substantial activation-dark structure: activation similarity and Fisher alignment have Spearman \(0.525\), and one nearest-neighbor pair has \(S_a=0.904\) but Fisher alignment \(0.088\). Under the fixed high-activation/low-Fisher nearest-neighbor diagnostic, 121 of 700 domains have an activation-confusable but update-distinct neighbor, while only 11 show the reverse pattern.

26.4 Molecular SMILES Proof of Concept↩︎

We include a small scientific-sequence check because shared-vocabulary source selection is especially natural for molecular strings. Using Llama-3.1-8B, we form nine molecular SMILES domains (brominated, chlorinated, fluorinated, highly aromatic, large complex, multi-halogen, nitrogen-heavy, simple aliphatic, and sulfur-containing). FisherSketch signatures use \(m=2048\) and 180 molecules per domain; transfer is measured by cross-domain perplexity reduction under a 20-example in-context adaptation protocol, averaged over seeds 42–44 with 50 evaluation molecules per target.

The point of this experiment is not to claim a complete chemistry benchmark. It checks the same activation-dark mechanism in a scientific string setting: activation similarity is high and not significant as a transfer predictor at this scale, while the joint FisherSketch score and the error marginal correlate with held-out cross-domain perplexity reduction.

26.5 Dynamic Addition↩︎

26.5.0.1 Protocol.

To add a new domain to an existing index:

1. Collect 200 samples from the new domain.

2. Compute the domain signature via a single forward pass (4.7 seconds on an A100).

3. Append the signature to the index.

No retraining is required—signatures are simply appended.

26.5.0.2 Evaluation.

We evaluate on four held-out domains (code, medical, legal, math), each added to an existing 674-domain index. Retrieval accuracy on 30 held-out prompts from each new domain:

This is 64\(\times\) faster than retraining a learned classifier on the expanded domain set.

26.6 Error Geometry Tracks Label Overlap↩︎

The non-identifiability witness in the main text uses the zero-overlap endpoint for clarity. To check that the effect is not merely a corner case, we run a fixed-representation label-overlap sweep in which activation geometry is held constant while the shared-output label support varies.

The gap grows continuously as overlap decreases. Thus the theorem isolates a clean endpoint of a graded structural phenomenon: representation metrics see the unchanged activation geometry, whereas Fisher alignment also sees label-conditioned error geometry.

27 Additional Validation Diagnostics↩︎

27.1 Rho/Delta Calibration for Proxy Reliability↩︎

The coupling diagnostic should be used as a warning signal, not as a universal threshold. In the calibration below, lower pairwise \(\rho\) and higher \(\delta\) coincide with larger gaps between exact Fisher alignment and the separable proxy. High \(\rho\) is not a guarantee, but low \(\rho\) is a practical red flag that separable activation/error proxies may be unreliable.

27.2 Gradient and Diagonal-Fisher Baselines↩︎

Table 22 compares FisherSketch with first-moment and diagonal-Fisher baselines in the conditional 100-domain LLM regime. We separate the SRHT row used for the vocabulary-scale backend from a dense-projection check to avoid mixing estimator provenance. FisherSketch improves top-1 over these gradient baselines, while activation-only, error-only, and product proxies remain competitive on rank correlation; this is why the main text avoids unconditional dominance claims.

27.3 Sketch-Dimension Sweep↩︎

Table 23 gives empirical guidance for the 100-domain LLM regime. It should not be overinterpreted as a monotone top-1 law: larger sketches stabilize the estimator, but source-selection accuracy also depends on transfer noise, candidate composition, and near ties. The \(m=4096\) sweep row comes from the SRHT sweep extraction, while Table 22 reports the SRHT EB-shrunk row for the gradient-baseline comparison.

28 Heterogeneous Outputs and Internal-Layer Directional Fisher↩︎

Head-level Fisher alignment requires a shared output basis. When label spaces or heads differ, head Fisher is undefined for cross-dataset pairs rather than merely weak. We therefore treat internal-layer Fisher descriptors as gradient diagnostics rather than replacements for representation metrics. Table 24 reports per-target Spearman correlations from the heterogeneous-output ViT-B/16 validation. We use the target-normalized directional descriptors for the internal Fisher rows; raw symmetric block Fisher was near zero or negative in this experiment and should not be interpreted as the successful signal.

28.0.0.1 One-step LoRA tangent bridge.

For a layer \(W\in\mathbb{R}^{d_{\rm out}\times d_{\rm in}}\) with loss gradient \(G=\nabla_W\ell\), a LoRA parameterization writes the update as \(BA\) with \(B\in\mathbb{R}^{d_{\rm out}\times r}\) and \(A\in\mathbb{R}^{r\times d_{\rm in}}\) [56]. At the common initialization used in many LoRA implementations (\(B=0\), \(A\) fixed), one infinitesimal gradient step gives \(B^+=-\eta G A^\top\) and hence the induced weight update \[\Delta W = B^+A = -\eta G(A^\top A).\] Therefore, for a fixed LoRA tangent subspace, one-step update inner products are \[\langle \Delta W_i,\Delta W_j\rangle_F = \eta^2\,\mathrm{tr}\!\left((A^\top A)G_i^\top G_j(A^\top A)\right),\] i.e., a gradient/Fisher alignment after a fixed low-rank projection. This is the intended bridge between internal Fisher descriptors and LoRA transfer. It does not characterize multi-step nonlinear LoRA training, and Table 24 should be read as diagnostic evidence rather than a dominance claim over CKA.

The stored descriptor is still \(m\) floats per task per layer after pooling, but internal shared-token collection has a larger transient state. At \(n=500\) and \(m=4096\), the transient sketch-feature state is about 8 MiB per task per layer in fp32; in this run, four internal layers across eight models used about 250 MiB of transient feature state and peaked at 4.19 GiB GPU memory during Fisher collection. For the QKV validation, pooled sketch descriptors store about \(432\times\) fewer values than saving all per-example QKV gradients.

29 Verbalizer-Shift Experiment Details↩︎

This section provides full experimental details for the verbalizer-shift controlled experiment (Section 6.2 of the main text). This experiment demonstrates that FisherSketch predicts transfer in a regime where representation-only metrics are maximally uninformative.

29.1 Experimental Design↩︎

29.1.0.1 Core insight.

For prompt-based binary classification with a shared prefix, the hidden state at the answer position depends only on the prefix tokens due to causal attention. Therefore, changing only the label verbalizer tokens (e.g., from Yes/No to True/False) yields:

• Identical activations: \(M_a\) is the same across all verbalizers \(\Rightarrow\) \(S_{\mathrm{cov}}(M_a) \approx 1\) for all pairs.

• Different errors: The error vector \(e = \mathrm{softmax}(\ell) - \mathrm{onehot}(y)\) changes because the target token ID differs \(\Rightarrow\) \(S_{\mathrm{cov}}(\Gamma_e)\) varies across pairs.

This isolates error geometry as the only source of variation, providing a clean test of whether FisherSketch can predict transfer when representation-only metrics are degenerate.

29.1.0.2 Connection to Theorem 1.

This provides a dense-vocabulary instantiation of the non-identifiability result. Even when representations are identical (\(Z_i = Z_j\)), head Fisher alignment is not identified because error geometry can differ. Unlike the masked-softmax witness construction, this uses the full vocabulary softmax (\(K{=}128{,}256\)) with standard cross-entropy loss.

29.2 Verbalizer Configuration↩︎

We use six single-token verbalizers on Llama-3.1-8B (vocabulary size 128,256). All tokens were verified to be single tokens using the Llama-3.1 tokenizer.

29.2.0.1 Flipped verbalizer (no_yes).

The no_yes verbalizer inverts the label mapping: “No” indicates true, “Yes” indicates false. This tests whether FisherSketch captures output-space structure beyond semantic intuition.

29.3 Datasets and Prompt Template↩︎

29.3.0.1 Datasets.

We evaluate on three binary classification datasets:

• BoolQ [57]: Yes/no reading comprehension (9,427 train examples).

• RTE [58]: Textual entailment (2,490 train examples).

• CB [58]: Commitment bank, binarized to entailment vs.not (250 train examples).

29.3.0.2 Prompt template.

All datasets use the same template (prefix only, no label token in input):

Passage: {passage}

Question: {question}

The answer is

For RTE/CB, we map hypothesis/premise to the passage/question fields. The model predicts the next token (the verbalizer) given this prefix.

29.3.0.3 Data splits.

• Sketch: 500 samples for FisherSketch computation (seed 42/43/44).

• Train: 200 samples for LoRA fine-tuning.

• Eval: 100 samples for transfer evaluation (50 early-stopping, 50 final eval).

For CB (250 total examples), splits are scaled proportionally: sketch = 118, train = 94, eval = 38. Sketch and LoRA splits are disjoint.

29.4 FisherSketch Configuration↩︎

• Sketch dimension: \(m = 4096\)

• Sketch samples: \(n = 500\) per verbalizer (scaled for CB)

• Error projection: SRHT on the full \(K=128{,}256\) vocabulary error side; dense Rademacher/Gaussian projections are used only in controlled small-dimensional estimator comparisons

• Activation extraction: Last layer hidden state at answer position

• Error computation: \(e = \mathrm{softmax}(\ell) - \mathrm{onehot}(y)\) over full 128K vocabulary

29.5 LoRA Fine-Tuning Configuration↩︎

• LoRA rank: \(r = 8\)

• LoRA alpha: \(\alpha = 16\)

• Target modules: q_proj, k_proj, v_proj, o_proj

• Training steps: 100

• Learning rate: \(5 \times 10^{-5}\)

• Batch size: 2 (gradient accumulation 4, effective batch 8)

• Loss: Full-vocabulary cross-entropy on the single target token (not restricted to the two verbalizer tokens)

• Early stopping: Patience 5 on held-out 50 samples

29.5.0.1 Critical implementation detail.

The loss is computed over the full vocabulary: \[\mathcal{L} = -\log p(y_{\mathrm{verbalizer}} \mid \mathrm{prefix})\] where \(p\) is the full softmax over 128K tokens. We do not renormalize over only the two verbalizer tokens—this avoids reintroducing masked-softmax artifacts.

29.6 Transfer Metric↩︎

We use the same normalized transfer metric as the main text (Table 1): \[U_{s \to t} = \frac{\ell_t^{\mathrm{base}} - \ell_{s \to t}}{\max(\ell_t^{\mathrm{base}} - \ell_t^{\mathrm{self}}, 0.01)}\] where:

• \(\ell_t^{\mathrm{base}}\): Base model loss on target verbalizer (no fine-tuning)

• \(\ell_t^{\mathrm{self}}\): Loss after fine-tuning on target verbalizer itself

• \(\ell_{s \to t}\): Loss on target after fine-tuning on source verbalizer

29.6.0.1 Source selection metrics.

For each target verbalizer \(t\), we select the best source from the other 5 verbalizers:

• Top-1 accuracy: Does the predicted source match the oracle (argmax transfer)?

• % Oracle: Mean \(U_{\mathrm{predicted}} / U_{\mathrm{oracle}}\) across targets.

• Max regret: \(\max_t (U_{\mathrm{oracle},t} - U_{\mathrm{predicted},t})\).

29.7 Per-Run Results↩︎

29.7.0.1 Key observations.

1. Activation invariance confirmed: \(S_{\mathrm{cov}}(M_a)\) mean is 1.0013 \(\pm\) 0.0006 across all runs, confirming that activations are identical.

2. Error divergence confirmed: \(S_{\mathrm{cov}}(\Gamma_e)\) ranges from near-zero to 0.99 across verbalizer pairs.

3. FisherSketch predicts transfer: 66.7% top-1 (3.3\(\times\) random), 95.7% of oracle.

29.8 Flipped Verbalizer Analysis↩︎

A surprising finding: the flipped verbalizer (no_yes \(\to\) yes_no) shows positive transfer (U \(\approx\) 0.7), not negative transfer as naive semantic intuition would predict.

29.8.0.1 Explanation.

The base Llama-3.1-8B does not concentrate probability on Yes/No tokens—it spreads mass across the full vocabulary. Fine-tuning on no_yes teaches the model that “the answer is one of {Yes, No}”—this output space learning dominates over the semantic mapping.

After fine-tuning on no_yes:

• \(p(\texttt{Yes} \mid \text{true}) \approx 0.30\) (up from \(\approx 0.05\) in base model)

• \(p(\texttt{No} \mid \text{true}) \approx 0.60\)

Even though the mapping is flipped, \(p(\text{correct})\) increased from 5% to 30%, so the loss decreases.

29.8.0.2 FisherSketch correctly predicts this.

\(S_{\mathrm{cov}}(\Gamma_e)\) between yes_no and no_yes is \(\approx 0.9\) (they share the same output tokens), so FisherSketch predicts high transfer—matching the empirical result.

This demonstrates that FisherSketch captures what fine-tuning actually learns (output-space structure), not what we semantically assume it learns.

29.9 Error Similarity Matrix↩︎

Table 27 shows \(S_{\mathrm{cov}}(\Gamma_e)\) for seed 42, BoolQ. The matrix exhibits interpretable structure:

• Same-token pairs (yes_no \(\leftrightarrow\) no_yes) have high similarity (\(\approx 0.9\)).

• Semantically related pairs (correct_wrong \(\leftrightarrow\) right_wrong) have moderate similarity (\(\approx 0.3\)).

• Unrelated pairs have near-zero similarity.

29.10 Compute and Reproducibility↩︎

29.10.0.1 Hardware.

The experiments reported in Appendix 29 were run on a single NVIDIA A100 (40GB).

29.10.0.2 Timing.

• FisherSketch computation (Phase 1): \(\approx\)​30 minutes per seed (3 datasets \(\times\) 6 verbalizers)

• LoRA fine-tuning + transfer evaluation (Phase 2): \(\approx\)​2 hours per seed

• Total: \(\approx\)​7.5 hours for all 3 seeds

29.10.0.3 Reproducibility.

Sketch seeds (42, 43, 44) and LoRA seeds are fixed and released. All indices (sketch/train/eval splits) are deterministic given the seed.

References↩︎

---

1. With tied input/output embeddings, this ignores the additional input-embedding gradient term, but the head gradient for the output matrix still satisfies 4 exactly.↩︎

2. We use upper-triangular vectorization with off-diagonals scaled by \(\sqrt{2}\) (Lemma 2, Appendix 10), so \(\langle \mathrm{vec}_{\mathrm{sym}}(aa^\top), \mathrm{vec}_{\mathrm{sym}}(a'a'^\top) \rangle = (a^\top a')^2\).↩︎

3. For softmax errors, \(\mathbf{1}^\top e = 0\) implies \(\Gamma_{e,k}\mathbf{1} = 0\) (rank at most \(|\mathcal{C}_k|-1\)), so a realizable isotropic model on the task support is \(\Gamma_{e,k} \propto I_{\mathcal{C}_k} - \frac{1}{|\mathcal{C}_k|}\mathbf{1}\mathbf{1}^\top\).↩︎

4. The Kronecker order \(a\otimes e\) matches our column-stacking \(\mathrm{vec}\) convention (so \(\mathrm{vec}(u v^\top)=v\otimes u\)). Using row-major vectorization swaps the Kronecker factors; all alignment scores are unchanged by this fixed permutation, but residuals such as \(\delta_k\) must be computed with the matching order.↩︎