1 Introduction↩︎

Grokking—the delayed onset of generalization long after a network has fit its training data—has become a controlled setting for asking how neural networks transition from memorization to structured solutions [1], [2]. A large body of recent work intervenes on a single training run to probe this transition: freezing, rescaling, or projecting weights along a trajectory and observing the effect on delay or on the emerging circuit [2], [3]. Such within-trajectory interventions, however, cannot answer a distinct question: of everything a partially trained checkpoint contains, which parts are portable—able to determine an outcome when transplanted into a different, independently trained run? Within-trajectory interventions can establish that a property is necessary for an outcome inside one run; they cannot establish whether that property is portable—able to produce the outcome when placed in a different run’s context. Distinguishing these two questions is our organizing concern throughout the paper: we do not ask why direction encodes circuit identity, but whether it is causally portable, and if so, under what conditions.

We study this question through a decomposition of the weight vector into a radial part (its norm) and an angular part (its unit direction), \(\theta = r \cdot u\). This decomposition itself is not new: the interplay of weight norm and direction in grokking has been analyzed within single trajectories [3], [4]. Our contribution is a cross-trajectory causal probe. Given two networks \(A\) and \(B\) trained from different seeds, we build a chimera by combining \(B\)’s norm with \(A\)’s direction (or vice versa), continue training, and ask two questions: does the grokking delay of the chimera follow the run that donated the norm, and does the final circuit identity follow the run that donated the direction?

We find a sharp asymmetry. Circuit identity—measured by a task-appropriate power spectrum of the token embedding—follows the angular donor cleanly, consistently, and specifically to that donor’s content. Grokking delay follows the radial donor only weakly, and this weak effect cannot be localized to any single layer group. Pushing further, we interpolate the implanted direction along the geodesic from \(A\) to \(B\) at fixed norm and find that identity transfer is threshold-like: the final circuit remains recipient-like across a plateau of interpolation values and then switches to donor-like over a narrow range. The location of this threshold is not constant—and, revealingly, it is predicted by the recipient’s weight norm. This last finding is only accessible because the intervention is cross-trajectory: it concerns not merely what transfers, but when transfer is possible.

1.0.0.1 Contributions.

1. Method. Cross-trajectory chimera interventions, a causal probe that recombines the magnitude and direction of independently trained networks, together with an angle-matched random control that isolates donor-specific content from generic angular perturbation (Section 3).

2. Dissociation. The angular component carries a transferable, donor-specific component associated with circuit identity across two tasks (\(40/40\) recombinations sign-correct); the radial component carries only a modest, non-localizable delay effect (Section 4, 6).

3. Threshold and its dependence on state. Identity transfer is threshold-like in the interpolation coordinate, and the threshold location is predicted by the recipient’s norm, separating perfectly by norm class across all \(20\) pairs and both tasks. We contribute an adaptive bisection procedure that localizes the threshold to \(\pm 1/64\) at roughly three evaluations per pair, a reusable measurement strategy for interventions that are stable under their training protocol (Section 5); this separation, and the identity-transfer result above, both survive an optimizer-state ablation that transplants Adam moments between donor and recipient rather than resetting them (Section 8).

4. Supporting characterization. Circuit spectra continue reorganizing for hundreds to thousands of steps after generalization is behaviourally complete, and no two seeds converge to near-identical spectra (Section 7).

2 Related Work↩︎

2.0.0.1 Transferring weights between grokking runs.

Closest to our setting is work that transfers a learned embedding from a weaker, already-generalized network to initialize a stronger one, accelerating or inducing grokking in the target [5]. That line treats transfer as a method for speeding up generalization and measures its effect on delay and accuracy. Our aims differ in three ways. First, we recombine two peer runs by explicitly separating norm from direction, rather than transplanting a whole embedding from a privileged source. Second, our primary outcome is circuit identity, not delay; nothing in the acceleration line measures which of several distinct circuits the recipient adopts. Third, we introduce a donor-content control (an angle-matched random direction) that distinguishes “\(B\)’s specific direction mattered” from “a perturbation of this angular size mattered.” We also find that the identity-carrying signal is present in multiple hidden layer groups, not the embedding alone (Section 7), which situates our result outside an embedding-centric account.

2.0.0.2 Merging independently trained networks.

A parallel literature merges or interpolates independently trained networks after aligning their permutation symmetries, seeking low-loss connectivity or a single merged model [6], [7]. We deliberately do not align permutations: the point of a chimera is to serve as a causal probe, and we measure a circuit-identity signature and the dynamics of continued training rather than a loss barrier between fixed endpoints.

2.0.0.3 Basin selection and mode determinism.

Which solution a trajectory reaches has been studied through linear mode connectivity and the role of the early training phase in determining the eventual basin [8], [9], and more recently through singular-learning-theory and exploration-based accounts of basin selection in grokking [10]. These analyses are largely observational—they characterize which basin is reached under natural training. Our chimera intervention provides a causal complement: by transplanting a direction across trajectories we measure the boundary at which basin membership changes, and we show that the location of that boundary depends on the recipient’s state.

2.0.0.4 Relationship to our prior work.

Two of our earlier studies analyze single-trajectory grokking: one establishes that norm and direction are jointly necessary to explain the delay within a run, and the other calibrates a norm–timescale relationship for the delay. The present paper is neither an extension nor a re-analysis of those datasets. It concerns cross-trajectory transferability and a dissociation between the two components’ portable content, using disjoint experiments; where we note that the modest delay effect is consistent with the earlier norm–timescale relationship, we say only that, and draw no stronger link.

3 Setup and Methods↩︎

3.0.0.1 Tasks and model.

We study two tasks on which a small transformer groks: modular addition \((a+b)\bmod p\) and modular multiplication \((a\cdot b)\bmod p\) on the multiplicative group \(\{1,\dots,p-1\}\), with \(p=59\). Inputs are tokenized as \([a,b,{=}]\) and the model predicts the result token. The architecture is a one-layer transformer (embedding dimension \(128\)) trained full-batch with AdamW (\(\text{lr}=10^{-3}\), weight decay \(1.0\), \(\beta=(0.9,0.98)\)) at a \(50\%\) train/test split. For multiplication we restrict to the multiplicative group because including \(0\) makes every \((0,b)\) pair collapse to \(0\), a degenerate shortcut that both inflates accuracy and pollutes the spectral fingerprint defined below.

3.0.0.2 Radial–angular decomposition and chimeras.

For a flattened weight vector \(\theta\) we write \(\theta = r\,u\) with \(r = \lVert\theta\rVert\) and \(u = \theta/\lVert\theta\rVert\). Given a pre-grokking checkpoint from run \(A\) (recipient) and one from run \(B\) (donor), we form chimera variants and continue training under the recipient’s optimizer state; results are reported for a reset optimizer unless noted, and we verify in Section 8 that this choice does not drive our conclusions. The core variant is radial (\(r_B\,u_A\): recipient direction, donor norm) and its mirror reverse_radial (\(r_A\,u_B\)). We include four controls: mid_norm (\(\tfrac{1}{2}(r_A{+}r_B)\,u_A\), isolating the effect of changing norm alone), random_u (a fully random direction at the recipient’s norm), and—critically—matched_random (a random direction placed at exactly the angular distance that \(u_B\) has from \(u_A\)). In high dimensions a uniformly random direction is nearly orthogonal to \(u_A\), whereas two independently trained runs on the same task are not; the matched-random control equalizes the angular displacement so that any difference from the true donor isolates donor-specific content rather than perturbation magnitude.

3.0.0.3 Circuit-identity metric.

Following the Fourier account of modular-arithmetic circuits [2], we summarize a network’s circuit by the normalized power spectrum of its token-embedding rows. The domain of the transform must match the task: for addition, circuits are periodic in the token index, so we take the FFT over rows \(0,\dots,p-1\); for multiplication, circuits are periodic in the discrete logarithm of the token (multiplication modulo \(p\) is addition modulo \(p-1\) under \(t\mapsto\log_g t\) for a primitive root \(g\)), so we reorder rows as \([g^0,g^1,\dots,g^{p-2}]\) before the transform. Using the index-domain transform for multiplication is blind to genuine circuit differences—two synthetic multiplication circuits at different frequencies register cosine similarity \(0.9998\) under the wrong domain and \(0.0000\) under the correct one. We define circuit similarity as the cosine similarity of two normalized power spectra and, for a chimera, report \(\mathrm{CFS\_lean} = \mathrm{sim}(\text{chimera},B_{\text{final}}) - \mathrm{sim}(\text{chimera},A_{\text{final}})\), so that negative values indicate an \(A\)-like final circuit and positive values a \(B\)-like one.

3.0.0.4 Directional dose-response.

To move continuously from recipient to donor direction we interpolate along the geodesic on the unit sphere, \[\mathrm{slerp}(u_A,u_B,t) = \frac{\sin\big((1{-}t)\phi\big)}{\sin\phi}\,u_A + \frac{\sin\big(t\phi\big)}{\sin\phi}\,u_B, \qquad \phi = \arccos(u_A\!\cdot u_B),\] and implant \(r_A\cdot\mathrm{slerp}(u_A,u_B,t)\) at fixed recipient norm \(r_A\). Spherical interpolation preserves unit norm along the entire path, so the intervention varies only the angular coordinate; linear interpolation of unit vectors would shrink the mixture’s norm (up to \({\sim}29\%\) at \(t=0.5\) for near-orthogonal endpoints), simultaneously changing magnitude and direction and confounding the two axes we seek to dissociate. The endpoint \(t{=}1\) coincides exactly with reverse_radial and is reused rather than recomputed.

3.0.0.5 Statistical protocol.

We select checkpoint pairs so that no seed appears in more than one pair (disjoint matching), yielding \(10\) fully independent pairs per task drawn from \(20\) distinct seeds. This avoids pseudoreplication: reusing a single seed across many pairs would make pooled statistics non-independent [11]. We report cluster-level statistics accordingly [12], use exact sign tests for directional consistency, and bootstrap confidence intervals for pooled means. Decision thresholds separating a usable signal from a confound were fixed before analysis. Throughout, we distinguish “statistically distinguishable from zero” from “large”; the delay effect below is the former but not the latter.

3.0.0.6 Threshold localization by bisection.

Under our protocol (full-batch training, reset optimizer) repeated continuations are sufficiently stable that the interpolation threshold \(t^\star\) per pair is limited by measurement resolution rather than by sampling noise. We therefore localize \(t^\star\) by bisection: the coarse grid brackets the sign change of \(\mathrm{CFS\_lean}\) to an interval of width \(0.25\), and each bisection step halves it, reaching \(\pm 1/64\) in three evaluations per pair—an order of magnitude fewer continuations than a uniform fine grid over all pairs. Because CUDA kernels are not bit-deterministic, we report \(t^\star\) with its bisection half-width as a resolution bound rather than as an exact real number.

4 The angular component carries transferable circuit identity↩︎

4.0.0.1 Endpoint swaps.

Implanting the donor’s direction while keeping the recipient’s norm drives the continued run toward the donor’s eventual circuit. On modular addition, the radial variant (recipient direction \(u_A\), donor norm \(r_B\)) yields a final circuit that is \(A\)-like in every pair (mean \(\mathrm{CFS\_lean}=-0.719\), \(10/10\) negative), while its mirror reverse_radial (donor direction \(u_B\)) is \(B\)-like in every pair (mean \(+0.478\), \(10/10\) positive). Modular multiplication replicates this exactly and more strongly on the \(B\)-side (means \(-0.719\) and \(+0.725\); \(10/10\) and \(10/10\)). Pooling the two variants across both tasks, the final circuit follows the direction donor in \(40/40\) independent recombinations; each per-task, per-variant sign test gives \(p=2\!\times\!10^{-3}\) (Figure 2).

4.0.0.2 The effect is donor-specific, not generic perturbation.

The angle-matched random control—a random direction at the same angular distance from \(u_A\) as \(u_B\)—produces final circuits with essentially no lean (mean \(-0.031\) on addition, \(-0.051\) on multiplication; leans scattered near zero). Formally, the donor variants are distinguishable from the matched control on the circuit-identity axis in \(58/60\) comparisons on addition and \(60/60\) on multiplication. Thus the transfer reflects the donor’s specific direction, not merely a perturbation of that angular magnitude.

4.0.0.3 Changing only the norm does not change identity.

The mid_norm control (recipient direction, averaged norm) leans \(A\)-like exactly as radial does (\(-0.707\) vs.\(-0.719\) on addition; \(-0.736\) vs.\(-0.719\) on multiplication; \(10/10\) negative in both). Altering the implanted norm while holding direction fixed leaves circuit identity unchanged, completing the dissociation from the identity side.

5 Identity transfer is threshold-like, and its threshold tracks recipient norm↩︎

5.0.0.1 A threshold, not a blend.

Interpolating the implanted direction from \(u_A\) to \(u_B\) at fixed norm does not move circuit identity smoothly. Across all \(20\) pairs the endpoints have opposite sign as expected, but the change is concentrated: on average \(82\)–\(88\%\) of the total change in \(\mathrm{CFS\_lean}\) occurs within a single grid step, with plateaus on either side (Figure 3).

A \(16\)-point fine grid on representative pairs confirms the plateau–jump–plateau shape and agrees with the bisection threshold to within one grid step. Identity transfer thus behaves as a switch between two basins rather than a continuous mixture—a graded intervention with a sharply graded response, consistent with a winner-take-all selection between distinct circuits.

5.0.0.2 The threshold location is predicted by recipient norm.

The interpolation value \(t^\star\) at which identity flips is not constant across pairs, and it is organized by the recipient’s weight norm. Localizing \(t^\star\) to \(\pm 1/64\) by bisection, pairs whose recipient has high norm (far from grokking) flip early—only a small move toward \(u_B\) is needed—whereas pairs whose recipient has low norm (near grokking) flip late. On modular addition the high-norm (slow) recipients have \(t^\star\in[0.238,0.297]\) and the low-norm (fast) recipients \(t^\star\in[0.641,0.891]\), a clean gap of \(0.34\); on modular multiplication the corresponding ranges are \([0.235,0.443]\) and \([0.678,0.856]\), a gap of \(0.23\). The two norm classes separate perfectly on every pair in both tasks; pooling all \(20\) pairs preserves perfect separation (gap \(0.20\)). Treating each task’s separation as an exact permutation test under the observed grouping gives probabilities \(2.2\!\times\!10^{-2}\) and \(8.3\!\times\!10^{-3}\), and \(1.9\!\times\!10^{-4}\) jointly (Figure 4).

Because the intervention holds the recipient’s norm fixed and varies only direction, this is a statement about susceptibility: at high norm, circuit identity is easily overwritten by a donor direction; at low norm, it resists.

5.0.0.3 Scope of the norm–threshold relationship.

We report this as a separation between two norm classes, not as a fitted continuous law \(t^\star(r)\). The binary slow/fast grouping coincides with our pair-selection design, which favours large norm differences and therefore produces two well-separated norm groups rather than a dense sweep; establishing a functional form would require pairs sampled across a continuum of recipient norms. What the data support is the ordinal claim—higher recipient norm, earlier threshold—which holds without exception here.

6 The radial component: a modest, distributed delay effect↩︎

The norm carries a genuine but weak influence on grokking delay, and no detectable identity information. Under the radial variant, the chimera’s delay shifts toward the norm donor by a normalized displacement whose per-pair donor-following score averages \(0.28\) on addition (\(9/10\) positive) and \(0.41\) on multiplication (\(10/10\) positive)—reliably nonzero but far from the value that would indicate the chimera fully inherits the donor’s delay. Expressed as a fractional displacement between recipient and donor delay, the chimera moves about \(30\%\) of the way toward the donor, pooled across tasks. The three implanted-norm levels available per pair (recipient, averaged, donor norm) order delay monotonically in \(17/20\) pairs, consistent with a graded but weak norm–delay relationship. Crucially, when the norm swap is restricted to individual layer groups the delay effect vanishes for every group—it is a property of the whole weight vector, not attributable to any single layer.

Relative to the strong, localizable, threshold-structured identity effect, the delay effect is weak, distributed, and carries no identity signal: the two geometric components play different roles.

7 Supporting characterization↩︎

7.0.0.1 Which layers carry the identity signal (summary; full analysis in Appendix 12).

Restricting the directional swap to individual layer groups shows that the identity-carrying signal is not the embedding’s alone: on both tasks, the attention, MLP, and unembedding groups each individually pull the final circuit toward the donor, but combining them gives a weaker lean than the strongest single group—the contributions are non-additive. We regard this as suggestive rather than established, given three pairs per task and a circularity that affects the embedding-based readout specifically; the full numbers, the circularity issue, and the figure are in Appendix 12.

7.0.0.2 Circuits keep reorganizing after grokking.

Tracking each run’s spectral similarity to its own final circuit across training, we find that \(50\%\) of the post-grokking reorganization is reached at a median of \(350\) steps after the test-accuracy transition, and \(90\%\) at a median of \(1250\)–\(1850\) steps, in every one of \(32\) seeds per task. Circuit structure thus continues to consolidate well after generalization is behaviourally complete; our contribution here is to quantify this on independent seeds with a spectral identity measure, complementing qualitative reports of a post-grokking cleanup phase [2].

7.0.0.3 No two seeds share a circuit.

At a similarity threshold of \(0.9\), all \(32\) seeds per task occupy distinct circuits; mean pairwise similarity is \(0.18\)–\(0.20\), near the \(1/\sqrt{d}\) baseline for the fingerprint dimension, and strictly positive for every pair. Independently trained networks converge to distinct but not unrelated spectral structures; whether the shared positive component reflects a common substrate is left open.

8 Optimizer-state ablation↩︎

AdamW carries, in addition to the weights \(\theta\), a per-parameter history of gradient statistics (the first and second moment estimates \(m,v\)). When we build a chimera and continue training, this history could in principle belong to either parent run rather than being a property of \(\theta\) itself. We test this directly: for both the endpoint-swap intervention (Section 4) and the threshold-localization intervention (Section 5), we repeat the experiment under three sources for the continuation optimizer’s moments—reset (freshly initialized, as reported above), recipient (transplant \(A\)’s moments), and donor (transplant \(B\)’s moments)—and ask whether the reported effect survives all three.

8.0.0.1 Circuit-identity transfer (C1) is unaffected.

Table 1 reports \(\mathrm{CFS\_lean}\) for the radial and reverse_radial endpoint swaps under all three optimizer conditions. The sign is consistent in \(10/10\) pairs in every cell but one (\(9/10\) for radial on addition under recipient), and mean magnitudes shift by at most \(0.08\) across conditions. The identity-transfer effect is a property of the weight configuration, not of which run’s gradient history the optimizer happens to carry.

8.0.0.2 The threshold–norm relationship (C5) is unaffected.

Repeating the full bisection procedure under recipient and donor moments, on all \(20\) pairs across both tasks, the slow/fast recipient-norm groups separate perfectly under every one of the three conditions (Figure 6). The separation gap narrows somewhat under moment transplantation (e.g.modular addition: \(0.34\) under reset versus \(0.25\) under recipient and donor) but never closes, and the per-task exact permutation probability is unchanged because the same slow/fast grouping separates in every condition (\(2.2\!\times\!10^{-2}\) for addition, \(8.3\!\times\!10^{-3}\) for multiplication, in each of the three conditions). Treating the six task–condition separations as independent, the joint probability of this pattern under the null is \(6.3\!\times\!10^{-12}\). Per-pair, \(t^\star\) shifts by an average of \(0.03\)–\(0.04\) when moments are transplanted—about twice the bisection resolution (\(\pm 1/64\)), and small relative to the \(0.18\)–\(0.34\) gap that separates the two recipient-norm groups. We conclude that the threshold’s dependence on recipient norm is a property of \(\theta\), with optimizer moments contributing a secondary, non-decisive perturbation.

9 Discussion↩︎

Our results map a division of labour between the two geometric components of a weight vector (Figure 7). Direction indexes which solution a trajectory approaches: transplanting it across independent runs transfers circuit identity, does so specifically to the donor’s content, and does so as a threshold switch between basins. Norm plays a different role: it carries a weak, spatially distributed influence on when generalization occurs and no identity information, but it governs how susceptible an identity is to being overwritten—high-norm recipients flip to a donor’s circuit under a small directional nudge, low-norm recipients resist. Interpreted through the basin picture, a chimera that is far from convergence (high norm) sits in a shallow region of the landscape where a modest directional change suffices to cross into a neighbouring basin, whereas a chimera near convergence (low norm) is already committed. This interpretation is consistent with the norm–timescale relationship we reported previously for single-run delay, but we do not claim it explains that relationship.

The non-additivity of the identity signal across hidden layers is an open puzzle. We considered and ruled out a training-instability explanation—the combined swaps re-grok to full accuracy rather than destabilizing—leaving a genuine interaction among layer groups as the more likely account, which we do not resolve here.

9.0.0.1 What does not follow from our results.

Our findings do not show that direction alone determines circuit identity; that norm is irrelevant to representation; that basin geometry is fully characterized by weight norm; or that the observed dissociation generalizes beyond the two cyclic-group tasks and single architecture studied here. Each of these is a plausible next question, not a conclusion we are entitled to.

10 Limitations↩︎

We study a single architecture and two tasks from the same (cyclic-group) family; whether the dissociation holds for non-group tasks or larger models is untested. We explored a non-cyclic task (sparse parity) and found that it groks cleanly only under a different optimization regime (weight decay \(0.1\), lower learning rate); rather than report a cross-task “replication” confounded by mismatched hyperparameters, we leave the non-cyclic extension to future work with the validated configuration in hand. Our circuit-identity metric is an embedding-spectrum proxy for the full circuit, not a complete circuit equivalence, and the embedding-swap localization result is circular for that reason (Section 7). The threshold \(t^\star\) is localized on a grid of resolution \(\pm1/64\); the norm–threshold relationship is established as a two-class separation rather than a continuous law, because our pair selection produces well-separated norm groups by design. Continued-training hyperparameters are held fixed regardless of donor and recipient origin. The optimizer-state ablation (Section 8) covers all \(20\) pairs for the bisection results but only two representative pairs per task for the underlying fine interpolation grid, matching the scope of the main fine-grid analysis.

11 Conclusion↩︎

Cross-trajectory chimera interventions dissociate the roles of weight magnitude and direction in grokking. Direction carries a transferable, donor-specific component associated with circuit identity and transfers it as a threshold switch between basins; the threshold’s location is predicted by the recipient’s norm, revealing that transferability depends on the recipient’s dynamical state. Norm carries only a modest, distributed delay effect and no identity signal. The adaptive bisection procedure we use to localize the threshold is a reusable tool for interventions that are stable under their training protocol. Beyond the specific geometry of grokking, the chimera construction offers a general template for asking not just what a partially trained network contains, but what of it is portable to another.

Reproducibility Statement↩︎

All experiments use publicly reproducible task definitions, fixed random seeds recorded per run, and a pipeline that regenerates every table and figure from raw logs. The full analysis (pair selection, chimera and interpolation interventions, bisection, and statistical reporting) is released as scripts with per-job resume markers; each reported number is emitted directly by the analysis code from the experiment CSVs.

12 Layer-localization of the identity signal↩︎

This appendix gives the full layer-group analysis summarized in Section 7. Restricting the directional swap to individual layer groups shows that the identity-carrying signal is not the embedding’s alone. On modular addition, the attention, MLP, and unembedding groups each individually pull the final circuit toward the donor (\(\mathrm{CFS\_lean}\approx-0.6\) each when swapped in the \(A\)-direction test), yet swapping all of them together produces a markedly weaker lean (\(-0.145\)): the contributions are non-additive. Modular multiplication reproduces the qualitative pattern—each hidden group carries the correct-sign signal (attention \(-0.29\), MLP \(-0.45\), unembedding \(-0.32\)) and the combination is again non-additive—though per-task magnitudes differ (Figure 8).

We flag one confound explicitly: because the circuit metric is computed on the embedding spectrum, directly swapping the embedding’s direction manipulates the very object being measured, so the embedding and “all-but-embedding” results are circular and we do not interpret them; the non-circular hidden-layer results are the basis for the redundancy claim above, which we regard as suggestive rather than established, given three pairs per task.

References↩︎