1 Introduction↩︎

For a single attention head with norm-processed residual token \(x\in\mathbb{R}^d\) and projections \(W_q,W_k\), the pre-softmax score is a bilinear form in one operator, \[\mathrm{score}(i,j) = q_i^\top k_j = x_i^\top(W_q^\top W_k)x_j = x_i^\top M x_j,\qquad M=W_q^\top W_k\in\mathbb{R}^{d\times d}.\] \(M\) is the natural, gauge-invariant object of study: the reparametrization \(W_q\!\to\!W_qG,\;W_k\!\to\!W_kG^{-\top}\) leaves \(M\) and every observable unchanged, so raw \(W_q/W_k\) entries are gauge artifacts while functions of \(M\) are physical [1]. \(M\) is low rank (\(\mathrm{rank}\le d_k\)) and, crucially, generally non-symmetric—so it has a complex eigenspectrum.

A growing literature reads non-normal/complex spectral structure into transformers: Schur decompositions of the residual-stream Jacobian [2], “non-Hermitian” operator-theoretic framings [3], and symmetric/antisymmetric decompositions of the QK matrix [4]; concurrent work catalogues per-head complex eigenvalues of the attention interaction and ablates its skew/symmetric channels against corpus perplexity [5]. Yet the QK operator’s complex spectrum remains behaviorally unanchored — no random-matrix nulls, no head-function taxonomy, no head-resolved causal tests, no positional-scheme contrasts — and the field’s discipline demands we ask not whether the structure exists but whether it does measurable work. Concretely: does the complex-eigenvalue view of \(M\) predict head behavior that the plain symmetric/antisymmetric split does not already capture?

We answer this at three levels: statics across seven pretrained models — GPT-2 small, OPT-1.3B, and GPT-Neo-1.3B (learned-absolute positions), BLOOM-1b1 (ALiBi: position enters as a score bias, leaving QK positionally unconstrained), Pythia-410m/1.4B (\(25\%\) RoPE), and Llama-3-8B (\(100\%\) RoPE, RMSNorm, GQA); dynamics across the public Pythia training checkpoints; and interventions in constrained-from-scratch training runs. Our contributions:

1. A matched-null spectral framework for the QK operator (§3): four decompositions of the norm-folded \(M\), a primary directionality metric \(D_{\text{head}}\) and its plain-split baseline dir_frac, and—essential—a random-orientation (Ginibre) null against which directionality must be read (a random low-rank \(M\) already has \(\texttt{dir\_frac}\approx1/\sqrt2\)).

2. Descriptive: spectral directionality separates head function across all seven models (§4), extending [4] from training objective to head function.

3. Causal anatomy on GPT-2 (§5): the symmetric part \(M_S\) is \(6.7\times\) more load-bearing than the antisymmetric \(M_A\); \(M_A\) is causally critical only for the canonical induction/previous-token circuit and inert elsewhere. On GPT-2 the complex refinement \(D_{\text{head}}\) does not beat dir_frac.

4. The architecture-conditional signature (§6): the strongest previous-token heads are spectrally rotational under RoPE(top-quintile \(D_{\text{head}}\)) and non-rotational under learned-absolute and ALiBi positions (bottom-quartile, four models); the model-level separation is perfect at every top-\(k\) tested (exact permutation \(p{=}0.029\); head-level \(p{=}2.5{\times}10^{-5}\), descriptive); \(d_k\) held fixed across schemes. Aggregate partials are shown to be bulk-contaminated and unreliable for this contrast.

5. Mechanism (§7): a per-frequency complex decomposition ties \(D_{\text{head}}\) to the RoPE rotational phase \(\mathrm{Im}(M_t)\); adding RoPE-phase features as controls attenuates \(D_{\text{head}}\)’s advantage by \({\sim}47\)–\(61\%\) (\(25\%\) RoPE) and \({\sim}70\)–\(98\%\) (\(100\%\) RoPE), estimator-dependent.

6. Causal confirmation (§8): ablating \(\mathrm{Im}(M_t)\) symmetrizes the relative-position kernel and destroys induction on exactly the flagged heads.

7. A checkpoint natural history (§9): on Pythia-410m/160m (22 public checkpoints each), all heads are born at the Ginibre null; the rotational signature locks in with circuit formation (not before it), and the population-median suppression that produces the static profile follows formation — the profile is a consolidated fingerprint, not a precursor.

8. Constrained-training interventions (§10): a \(\{\)APE, RoPE\(\}\times\{\)free, sym-\(M\), \(\mathrm{Im}(M_t)\)-suppressed\(\}\) grid (\(n{=}5\) seeds) shows no spectral channel is necessary (every arm reroutes) while every constraint carries a significant search cost that reveals the scheme’s default solution, and dissociates the static-antisymmetry and RoPE-phase channels at the weight level.

One sentence unifies the three levels: the positional scheme sets the default spectral algebra of attention’s solutions — a fingerprint sculpted after function, an economy of search rather than a hard constraint. We report the negatives (GPT-2 decorativeness; \(M_A\)’s inertness away from a few heads) as plainly as the positives.

2 Related work↩︎

QK circuits and weight-space interpretability. [1] frame \(M{=}W_q^\topW_k\) and \(W_{OV}\) as the two low-rank circuits and use OV eigenvalue positivity as a copying statistic; they perform no QK spectral analysis. [6] find weight-SVD interpretable but report it fails on QK—motivating an orientation-recovering (Schur/complex) view. Our object and the OV/QK distinction follow this line; our contribution is the QK-side, imaginary-axis, behavior-anchored analog.

Symmetric/antisymmetric QK. [4] decompose \(W_{qk}{=}M_s{+}M_n\) and define a Frobenius symmetry score, anchoring it to the training objective. Our dir_frac is a monotone transform of their score (\(s = 1-2\,\texttt{dir\_frac}^2\)); we anchor the split to head function and add the complex/Schur refinement and its causal test.

Non-normal / non-Hermitian transformers. Concurrently with this work, [5] decompose the per-head attention interaction into skew (“routing”) and symmetric (“filtering”) channels across \(1{,}776\) heads of five pretrained models, report \(\max\mathrm{Re}\,\lambda>0\) for every head, find by all-heads truncation surgery on GPT-2 Large that the symmetric channel is by far the more load-bearing (convergent, at the aggregate level, with our §5), and train a stability-constrained skew-minus-diagonal attention from scratch. Their analysis stops at norms, ranks, and stability — no behavioral taxonomy, no random-matrix null, no head-resolved causal test, no positional-scheme contrast, and their weight-level routing–filtering ratio is (like Saponati’s score) a monotone transform of dir_frac— so it cannot ask whether the complex spectrum does work beyond the plain split; that anchoring is our contribution. [2] Schur-decompose the residual-stream Jacobian and show learned non-normality is functionally central—but on a different operator, without pseudospectra, and anchored to rank/propagation, not head function. [3] labels \(W_q^\topW_k\) non-Hermitian in a Dyson-series theory with no computation. We differ by analyzing the QK operator’s complex spectrum empirically and anchoring it to head behavior.

RoPE and positional heads. RoPE[7] injects \(e^{im\theta_t}\) phase, \(\theta_t=\mathrm{base}^{-2t/d}\). [8] and [9] show positional heads use high RoPE frequencies. We connect a weight-space directionality scalar to per-head RoPE-frequency usage and to head function, which they do not.

Circuit formation over training. [10] established the induction phase change; [11] track circuits across all Pythia checkpoints behaviorally (induction at \({\sim}2{\times}10^9\) tokens); toy-model studies dissect formation [12]–[14]; and [15] regularize an interpretable attention property during MLM pretraining — the template our intervention ports to autoregressive weight space. [4] track their symmetry score during training (per-layer, own encoder/decoder runs). None of these tracks a weight-space spectral quantity per head across public checkpoints against the phase change (§9), and none constrains QK spectral structure during autoregressive training (§10).

Constrained-attention training. Hard symmetric (shared-QK) causal LMs exist — Reformer [16] and recent systematic sharing studies — and symmetric dot-product BERT trains well [17], but none analyzes circuits under the constraint, and the soft, decomposed versions (penalize only \(M_A\), or only \(\mathrm{Im}(M_t)\)) are new here; concurrent observational work crosses positional schemes with induction formation in toy models [18].

Head taxonomy. We use standard detectors—previous-token, duplicate, induction [10]—and IOI [19] head classes for validation.

3 Method↩︎

Object and folding. We use TransformerLens with fold_ln=True to absorb the norm gain into \(W_q,W_k\), and analyze the norm-folded operator \(M_{\text{eff}}=CM C\) for LayerNorm (centering \(C=I-\tfrac1d\mathbf{1}\mathbf{1}^\top\)) and \(M_{\text{eff}}=M\) for RMSNorm. All spectral work is float64. We verify the convention by reconstructing the model’s own pre-softmax scores from \(M\) and matching to \(<10^{-4}\) (learned-absolute) or, for RoPE models where position cancels only at \(i{=}j\), matching the diagonal (bf16 models match to bf16 precision; the fast metric path is exact, \(\texttt{dir\_frac}\;\Delta{=}10^{-16}\)). A random \(G\in GL(d_k)\) leaves \(M\), its eigenvalues, and \(D_{\text{head}}\) invariant to \({\sim}10^{-14}\) while changing raw column norms—confirming gauge hygiene.

Decompositions. Per head we compute the SVD, the symmetric/antisymmetric split \(M{=}M_S{+}M_A\), the real Schur form (\(2\times2\) blocks \(\to\) rotation angles \(\theta\)), and the complex eigendecomposition (with eigenvector condition number as a non-normality flag).

Metrics. \[\begin{align} \texttt{dir\_frac}&= \|M_A\|_F/\|M\|_F && \text{(plain antisymmetric magnitude; Saponati's score)}\\ D_{\text{head}}&= \textstyle\sum_r |\mathrm{Im}\,\lambda_r| \big/ \sum_r |\lambda_r| && \text{(primary complex/Schur directionality)}\\ \texttt{content\_pos\_frac}&= \textstyle\sum \lambda^+ \big/ \sum |\lambda| \;\text{of}\;M_S&& \text{(signed content axis; QK-side echo of OV positivity)} \end{align}\] plus \(\text{henrici}=\sqrt{\|M\|_F^2-\sum|\lambda|^2}/\|M\|_F\) (departure from normality) and self_match(diagonal dominance of \(W_E M W_E^\top\)).

The random-orientation null (essential). A random low-rank \(M\) is already highly “directional”: for independent Gaussian \(W_q,W_k\) the symmetric and antisymmetric parts get equal expected Frobenius mass, so \(\texttt{dir\_frac}_{\text{null}}\approx 1/\sqrt2\approx0.707\) and \(D_{\text{head}}\)’s null is likewise high. We therefore report every metric relative to a matched-\(\Sigma\) random-orientation null (same singular values, random left/right frames), computed exactly in the rank-\(k\) core. Empirically \(\texttt{dir\_frac}_{\text{null}}{=}0.707\), \(D_{\text{head,null}}{=}0.608\); \(141/144\) GPT-2 heads deviate (\(|z|{>}2\) on dir_frac or \(D_{\text{head}}\); dir_frac alone: \(140/144\)). Nulls are computed exhaustively for GPT-2; the cross-model analyses of §6 use raw metrics — on GPT-2, null-referenced metrics give an indistinguishable partial (\(-0.204\) vs \(-0.206\)), so this choice is not load-bearing.

Models & corpus. GPT-2 small (144 heads), OPT-1.3B (768), GPT-Neo-1.3B (384), BLOOM-1b1 (384; ALiBi), Pythia-410m/1.4B (384), Llama-3-8B (1024; GQA expanded, RMSNorm, bf16). Behavioral runs use a \(1\mathrm{k}\times128\) slice of NeelNanda/pile-10k; detectors use repeated-random sequences. All on one A100.

Checkpoint and intervention pipelines. For §9 we load Pythia public checkpoints natively (TransformerLens checkpoint_value); the fused-QKV unpacking and folding conventions were verified against the main-revision pipeline to machine precision (weight metrics at the final checkpoint match the independent pipeline exactly, \(\Delta{=}0\)). For §10 we train 2-layer attention-only models from scratch with soft spectral penalties; definitions and the crucial sym-\(M\) \(\neq\) \(\mathrm{Im}(M_t)\) distinction are given there. Pre-registration: the dynamics questions (Q1–Q3) and intervention predictions (P1–P4) were locked in the project’s intervention plan before any checkpoint was downloaded or any constrained run launched; we report the falsified predictions as findings.

4 Spectral directionality separates head function↩︎

On GPT-2 (\(n{=}144\), Spearman, BH-FDR \(q{<}10^{-3}\)), directionality maps onto head class (Table 1). Previous-token heads are directional (high dir_frac, low content positivity); duplicate/similarity heads are symmetric; induction heads read as content-like. We note plainly that our preregistered hypothesis predicted the opposite (induction heads directional); that prediction failed. The two-head-circuit reading — the induction head’s QK consumes composed content written by a previous-token head, so its single-head \(M\) is symmetric — was recorded in the analysis plan before these correlations were computed, and we verify it directly: by the K-composition statistic (the Frobenius overlap of an induction head’s QK operator with an earlier head’s OV write, ranked over all earlier heads), the canonical previous-token head ranks #1 for every induction head tested — GPT-2’s 4.11 for 5.1/5.5/6.9 (ranks 1/60, 1/60, 1/72) and Pythia-410m’s 5.2 for 8.6/10.9/11.14 (1/128, 1/160, 1/176) — confirming that induction QK is wired to consume the prev head’s composed output. Copying (OV) is orthogonal to every QK metric, a clean control. The content signature is legible: self_match equals content_pos_frac at \(r{=}0.96\); empirical pre-softmax score asymmetry on real inputs tracks dir_frac at \(0.66\). This replicates on Pythia and, attenuated, on Llama (content_pos_frac\(\to\)prev \(-0.30\) vs GPT-2’s \(-0.63\); §6), establishing spectral directionality as an architecture-general head-function signature and extending [4] from objective to function (Fig. 1).

5 Causal anatomy on GPT-2: \(M_S\ggM_A\)↩︎

Symmetrization ablation. Replacing a head’s \(M\) with \(M_S\)(killing \(M_A\)) or \(M_A\)(killing \(M_S\)) and measuring held-out CE, the symmetric part is \(6.7\times\) more load-bearing: \(\sum \Delta\mathrm{CE}_{\text{anti}}{=}2.60\) vs \(\sum\Delta\mathrm{CE}_{\text{sym}}{=}0.39\); killing \(M_A\) costs \(\le0.02\) general CE anywhere. But a targeted induction task exposes that \(M_A\) is causally essential for specific heads: killing it on the canonical induction head 5.1 raises induction loss \(8\times\) (\(0.21\to1.77\); \(15\times\) the next-largest effect) and on the previous-token head 4.11 by \({\sim}50\%\). Canonical induction heads 5.5/6.9 show smaller positive effects (\(+0.03\), ranks 7–8 of 144); the top-5 by effect are 5.1, 4.11, 6.7, 3.0, 8.5, so top-5 overlap with the canonical circuit is 2/5 (hypergeometric \(p{=}0.014\)), with the argmax landing on a canonical induction head — circuit-level enrichment, not an exact circuit match. Crucially this importance is a discrete circuit fact: no spectral scalar (dir_frac, \(D_{\text{head}}\), imag_mass, \(\|M_A\|\)) predicts which heads need \(M_A\)(all \(|\rho|\lesssim0.1\)), so the 144-head aggregate is null.

Incremental validity. For predicting the previous-token score, the complex refinement adds no reliable positive increment on GPT-2: \(\mathrm{partial}\,\rho(D_{\text{head}},\text{prev}\mid\texttt{dir\_frac})=-0.21\) (layer-clustered 95% CI \([-0.51,+0.08]\) — the point estimate is negative but not reliably so; among prev-candidate heads, \(\text{prev}{>}0.05\), the residual association is strongly negative, \(-0.63\)), and in a nested regression on the ablation effect dir_frac explains \(0.001\) of variance rising only to \(0.028\) with \(D_{\text{head}}\) added, the added term wrong-signed. On learned-absolute positions, the plain antisymmetric magnitude captures everything the complex spectrum reliably offers about its strong prev heads; §6 shows this head-profile pattern generalizes to OPT-1.3B and GPT-Neo-1.3B, though their aggregate statistics do not.

6 The architecture-conditional signature (main result)↩︎

We run the incremental-validity test across positional schemes on seven models (Table 2). On every RoPE model, \(D_{\text{head}}\)’s incremental contribution beyond dir_frac is positive and robust (clustered CIs \([+0.21,+0.59]\)). Under learned-absolute positions, however, the aggregate statistic is heterogeneous (GPT-2 \(-0.21\); OPT-1.3B \(+0.35\); GPT-Neo-1.3B \(+0.20\)) — and decomposing it shows why it is the wrong statistic: a positive gradation among near-zero-prev bulk heads is present in most models (including GPT-2: \(+0.25\) within \(\text{prev}{\le}0.05\)), while the signal that differs by architecture lives in the strong previous-token heads themselves. The last column of Table 2 gives each model’s top-5 prev heads’ within-model \(D_{\text{head}}\) percentile: bottom-quartile under learned absolute (medians \(0.21/0.23/0.14\); e.g.GPT-2’s 4.11 at \(0.22\); all with content-like symmetric profiles, content_pos_frac\(0.46\)–\(0.57\)) and ALiBi (\(0.26\)) versus top-quintile under RoPE (medians \(0.89/0.85/0.79\); Fig. 2a). Holding \(d_k{=}64\) fixed (GPT-2, OPT-1.3B, Pythia-410m: \(0.21/0.23\) vs \(0.89\)) ties the contrast to the positional scheme, not head dimension.

Model-level inference and sensitivity. Heads within a model share training, layers, and detectors, so the model is the honest experimental unit. At model level the separation is perfect — every non-RoPE model’s top-5 median sits below every RoPE model’s — giving an exact model-level permutation \(p{=}1/\binom{7}{3}{=}0.029\), the minimum attainable with seven models; the head-level joint Mann–Whitney \(p{=}2.5{\times}10^{-5}\) is reported as descriptive only. The contrast is not an artifact of the \(k{=}5\) choice: model-level separation is perfect at every \(k\in\{1,3,5,10\}\), and leave-one-model-out preserves it for all seven models. The binding statistical limit is the model count itself — one ALiBi and one full-RoPE family — which more model families, not more heads, would address (Limitations). The defensible claim is profile-level: the heads that implement previous-token attention are spectrally rotational under RoPE and spectrally non-rotational (content-like) under learned absolute positions. Mechanistically, this is what the positional algebra dictates: with learned absolute embeddings, prev-token attention can be built by matching adjacent position vectors — a symmetric, content-like operation — whereas RoPE builds it from rotations, imprinting complex eigenstructure. ALiBi provides the cleanest test: position enters as a score bias, so QK needs no positional structure at all — and BLOOM-1b1’s top prev heads are indeed non-rotational (percentile \(0.26\)), patterning with the learned-absolute group exactly as the mechanism predicts. Under RoPE these heads sit at the dir_frac null (\({\sim}0.70\)) with \(D_{\text{head}}\) at the Ginibre value (\({\approx}0.61\)) while the population suppresses below it (medians \(0.40\)–\(0.45\); per-model matched nulls: 410m prev heads’ \(D_{\text{head}}\) \(z\in[-0.1,+5.0]\) vs population median \(z{=}{-}11\); 1.4B prev-head median \(z{=}{-}3.7\) vs population \(z{=}{-}13.9\)): they retain near-random rotational structure that other heads learn to suppress, and the Schur/complex view detects this where the Frobenius magnitude cannot.

Robustness and scope. Three qualifications sharpen the claim. (i) Metric specificity: among complex-spectrum summaries, imag_mass is incrementally positive on all models and Henrici non-normality negative on all; \(D_{\text{head}}\) composites these two channels. No single aggregate scalar separates the schemes — the head-profile statistic does. (ii) Class vs bulk: the RoPE positives are class-level separation (rank-biserial \(+0.44/+0.60\) at \(\text{prev}{>}0.2\) in Pythia; \(+0.29\) at \(\text{prev}{>}0.1\) in Llama) plus bulk gradation; within confirmed prev heads there is no further gradation (subset partials n.s.). Under learned-absolute positions the class-level tests are null-to-negative at the strong end (GPT-2 \(-0.39\) at \(\text{prev}{>}0.2\), significant; OPT \(-0.11\) n.s.; GPT-Neo \(+0.13\) n.s.), though GPT-Neo shows a positive signal at the looser \(\text{prev}{>}0.1\) band (\(+0.34\)) driven by its moderate-prev heads — its top prev heads still sit at percentile \(0.14\). The learned-absolute models are thus heterogeneous in their moderate bands (candidate sources: training corpus, GPT-Neo’s alternating local-attention layers) while uniform at the top of the class. (iii) GPT-2’s negative aggregate concentrates exactly among prev-candidate heads (\(-0.63\) at \(\text{prev}{>}0.05\)); OPT’s positive aggregate is entirely bulk (\(-0.18\) among its prev candidates). Aggregate partials should not be read as the architecture signal in either direction.

3.5pt

7 Mechanism: \(D_{\text{head}}\) tracks the RoPE phase↩︎

RoPE pairs each head’s \(d_k\) dims (rotate-half convention, verified) into frequencies \(\theta_t\); the QK interaction per frequency is a rank-1 complex sub-operator \(M_t=w_q^t\,\overline{w_k^t}^{\top}\), with the static \(M=\sum_t\mathrm{Re}(M_t)\). The directional (relative-position-asymmetric) content is carried by the imaginary part: \[\mathrm{score}(\Delta)-\mathrm{score}(-\Delta)=\sum_t 2\sin(\Delta\theta_t)\,n^\top \mathrm{Im}(M_t)\, n .\] (Full-score reconstruction from \(\{M_t,\theta_t\}\) matches the model to \(10^{-6}/10^{-2}\) in fp32/bf16.) Empirically, previous-token heads have high rotary directional fraction rope_imag_frac(correlation with prev: \(+0.53/+0.42/+0.50\) for 410m/1.4B/Llama) concentrated at high frequencies (\(+0.68\) on full-RoPE Llama—a [8], [9] replication), and \(D_{\text{head}}\) tracks this deployment. Adding per-frequency RoPE-phase summaries as controls attenuates \(D_{\text{head}}\)’s residual association substantially (Table 3) — most completely on full-RoPE Llama, where the default-estimator residual is indistinguishable from zero (\(+0.006\), \(p{=}0.84\); fully rank-based: \(+0.075\), \(p{=}0.016\)). Two qualifications. First, all quantities are co-derived from the same weights, so this is shared-variance accounting, not causal mediation. Second, the attenuating covariate changes identity with RoPE fraction: on partial-RoPE Pythia the attenuation runs through rope_imag_frac(how much rotary content is directional; freq_centroid alone attenuates nothing, \(+0.35\)), whereas on full-RoPE Llama it runs almost entirely through freq_centroid (where in frequency the directional mass sits: alone \(+0.33\to-0.00\), while rope_imag_frac alone leaves \(+0.32\)) — consistent with all directional content being rotary by construction at \(100\%\), shifting the informative axis from amount to location. The pattern is consistent with fuller attenuation at higher RoPE fraction, but with two dose levels and within-dose spread (\(61\%\) vs \(38\%\)) comparable to the dose gap, we do not press a dose–response reading. The relative-position kernel confirms these heads peak at \(\Delta{=}{-}1\) with RoPE-periodic oscillation (Fig. 3, left).

Placebo controls. Two controls calibrate what the attenuation means. (i) RoPE-model specificity: on all three learned-absolute models (no RoPE), computing a pseudo-rope_imag_frac with a fake rotate-half pairing over all head dims yields no prev signal (GPT-2 \(-0.14\), OPT-1.3B \(-0.06\), GPT-Neo-1.3B \(-0.09\); all n.s., vs \(+0.42\)–\(+0.53\) on RoPE models), and adding the pseudo-phase features barely attenuates (\(4\)–\(22\%\)): the recipe finds nothing where RoPE is absent. (ii) Pairing specificity: randomly permuting the rotary columns of \(W_Q,W_K\) — which leaves \(M\)(hence \(D_{\text{head}}\), dir_frac) unchanged and preserves rotary-block membership, but scrambles the pairing and frequency assignment — still attenuates \(31\)–\(38\%\) on average, so much of the attenuation is carried by rotary-block membership alone. The increment specific to the true pairing/frequency assignment is marginal on Pythia-410m (true \(60.7\%\) vs the permutation null, empirical \(p{=}0.039\), \(n{=}50\) permutations) and not significant on Pythia-1.4B (\(37.7\%\), \(p{=}0.12\)). The mechanism is thus RoPE-specific and largely block-level; the finer pairing-level reading should be held lightly.

4pt

8 Causal confirmation: ablating \(\mathrm{Im}(M_t)\)↩︎

We ablate a head’s directional RoPE content by zeroing \(\mathrm{Im}(M_t)\) (dropping the \(\sin(\Delta\theta)\) component), which symmetrizes its relative-position kernel; the decomposition \(\text{full}=\text{symmetric}+\text{directional}\) is verified to \(6\times10^{-6}\) (fp32; \(2\times10^{-5}\) on bf16 Llama). Sweeping the ablation over all heads of all three RoPE models, the largest induction-loss effect lands on a top-2 prev-scoring head every time — an identity predicted a priori: Pythia-410m head 5.2 (prev rank 1/384; prev \(0.95\), \(D_{\text{head}}\,0.61\)): induction CE \(0.52\to2.24\) (\({\sim}4\times\); exact permutation \(p{=}1/384\)); Pythia-1.4B head 1.11 (rank 1/384): \(0.56\to1.11\) (\({\sim}2\times\); \(p{=}1/384\)); Llama-3-8B head L14.H26 (rank 2/1024; prev \(0.65\), \(D_{\text{head}}\,0.61\)): \(0.30\to0.91\) (\({\sim}3\times\); \(p{=}2/1024\)) — jointly \({\approx}10^{-8}\) under random assignment. (Llama’s #1 prev head sits in layer 0, too early to feed induction; the causally-critical head is the mid-depth prev head.) Effects are highly concentrated: \(92\)–\(99\%\) of heads move by \(|\Delta\text{Ind}|<0.01\); the other \(\text{prev}{>}0.5\) heads show \({\approx}0\) effects (consistent with redundancy across a multi-head circuit), and a few low-prev heads show small effects (\(\le0.17\)), suggesting downstream circuit participation the prev detector does not capture; head-level aggregate correlations are correspondingly null. The before/after kernel shows the \(\Delta{=}{-}1\) peak flattening (Fig. 3, right). As in §5, causal importance is concentrated, not a smooth spectral gradient; what the ablation establishes is an existence-with-predicted-identity claim: the rotational phase \(D_{\text{head}}\) reads is causally load-bearing for positional routing in the specific heads that carry it.

9 When the imprint arises: a checkpoint natural history↩︎

The static profile (§6) is an end-state. To see how it arises we track 22 public checkpoints each of Pythia-410m and Pythia-160m (log-spaced steps \(0\ldots143{,}000\); the dense early grid straddles the induction onset at \({\sim}2{\times}10^9\) tokens [11]), measuring per head: the audited weight metrics (dir_frac, \(D_{\text{head}}\), rope_imag_frac), behavioral previous-token and prefix-matching scores, an in-context-learning proxy (second-copy minus first-copy NLL on repeated-random sequences), and the K-composition wiring between each model’s eventual prev head and its induction heads. (The “eventual prev head” is identified retrospectively at the final checkpoint; Q3 below tests — and rejects — its prospective identifiability.)

A three-act history at half-doubling checkpoint resolution, replicated across both models (Fig. 4). Act I (0–0.5B tokens): silence at the null. Every head sits at the Ginibre null (behavior \(0.01\); rope_imag_frac\(0.500\); population median \(D_{\text{head}}\) \(0.61\); K-composition at baseline) — step 0 doubles as an in-vivo verification of the paper’s random-orientation null. Act II (1–4B tokens): sharp formation. Previous-token behavior jumps first (\(0.37\) at step 512 \(\to\) \(0.95\) at step 1000 — the same steps, and even the same \(0.37\) waypoint, in both models), with induction behavior, the ICL proxy, and the K-composition wiring following within the same window. The prev head’s within-model rotary-phase (rope_imag_frac) percentile snaps from at-or-below chance (\(0.03\)–\(0.04\) at step 512, mid-formation, in both models) to the top decile (\(0.90\)–\(0.99\)) by step 1000: the signature locks in with the behavior — at our half-doubling resolution, neither leads. Act III (4–300B tokens): slow differential sculpting. The population median \(D_{\text{head}}\) is suppressed well below the null (\(0.61\to0.395\) / \(0.453\)) over the next \(100\times\) of training while the previous-token heads retain it, and the absolute rotary-phase content of the prev head consolidates slowly (\(0.500\to0.578\), same endpoint in both models). The static “retain-vs-suppress” profile of §6 is therefore the end state of a post-formation differentiation process.

Pre-registered answers. Q1 (lead/lag): the percentile signature is simultaneous with formation; the absolute consolidation lags. Q2: population suppression is a post-formation process. Q3 (predictive signature): no — before formation the eventual prev head is not reliably identifiable from its spectrum (its percentile even sits low mid-formation); we report this pre-registered negative plainly. The natural history thus bounds what statics can claim: the rotational imprint is the fingerprint the algorithm leaves in weight space, consolidated after function — and because observation cannot order structure and function within the formation window, necessity can only be settled by intervention.

10 Is the rotational channel necessary? Constrained-training interventions↩︎

We train 2-layer attention-only models (d\({=}128\), 4 heads, \(d_k{=}32\)) from scratch on a per-sequence random-map task — \(x_{t+1}=f_{\mathrm{seq}}(x_t)\) w.p.\(0.9\), uniform noise w.p.\(0.1\), with \(f_{\mathrm{seq}}\) resampled every sequence — which forces induction (no global memorization; no fixed-offset shortcut) and yields a crisp formation time. Grid: \(\{\)APE, RoPE\(\}\times\{\)free; sym-\(M\): penalize \(\|M_A\|_F^2/\|M\|_F^2\); \(\mathrm{Im}(M_t)\)-suppressed (RoPE only): penalize \(\sum_t\|\mathrm{Im}(M_t)\|_F^2/\sum_t(\|\mathrm{Re}\|^2{+} \|\mathrm{Im}\|^2)\}\), \(n{=}5\) seeds. A load-bearing algebraic point shapes the grid: symmetrizing the static \(M\) does not zero \(\mathrm{Im}(M_t)\) (the static operator contains only the \(\mathrm{Re}\) parts, \(M=\sum_t\mathrm{Re}(M_t)+M_{\text{non-rot}}\)), so the two constraints dissociate the static-antisymmetry and phase channels. Both penalties enforce hard (final \(\mathrm{Im}\) share \(0.000\); dir_frac\(0.004\)–\(0.006\)) and no arm loses final capability (all reach the task floor with strong prev and induction heads).

Results against the pre-registered predictions (Fig. 5; Table 4). P1 (free base rates) holds: APE forms fastest (\(600\pm0\) steps), RoPE free at \(940\pm55\). P2’s strong form is falsified: suppressing the phase to zero delays formation only \({\sim}1.4\times\) and the circuit reroutes — the prev head re-forms with \(\texttt{rope\_imag\_frac}{=}0.005\) and a qualitatively different (cos-only) relative-position kernel whose peak still lands at \(\Delta{=}{-}1\). P3 is falsified in the opposite direction, our largest surprise: forcing symmetry on APE is the costliest constraint in the grid (\(600\to1740\pm241\), \(2.9\times\)) — the fast APE solution is antisymmetric embedding-matching, and the symmetric variant (nearby-kernel \(+\) causal mask), while reachable, is much harder to find; trained learned-absolute LLMs end at symmetric profiles (§6) with \(1000\times\) more data, and the toy exposes the search cost of that end state. P4’s formation-time version fails (RoPE+sym delays \(1.66\times\)) but its mechanistic dissociation lands exactly: the sym-arm’s prev head carries full phase content (\(\texttt{rope\_imag\_frac}\;0.52\)) through a fully symmetric static operator (\(\texttt{dir\_frac}\;0.004\)–\(0.005\)), and its positional kernel is indistinguishable from the free arm’s — directional attention through a symmetric \(M\), which is impossible under absolute positions.

Verdict. No spectral channel is necessary — the solution space is degenerate and training reroutes around every constraint we imposed — but each constraint carries a significant, quantifiable search cost (Table 4), and the cost structure identifies each scheme’s default solution. With §5 and §8 this completes a three-way distinction the field often conflates: trained-circuit dependence (post-hoc ablations are catastrophic), developmental preference (defaults are found much faster), and necessity (nothing here is strictly necessary). A companion note [20] tests the constructive converse — ban-free assistance — and finds it selects the same implementations far more cheaply than banning them (\(1.3\times\) vs.\(2.9\times\) for the antisymmetric-to-symmetric flip) while solution-specific initialization accelerates formation outright, evidence that this spectral economy is not only priced but steerable.

11 Discussion↩︎

What earns its keep, and where. The plain symmetric/antisymmetric split of \(M\) is an architecture-general descriptor of head function (§4) and, causally, the symmetric part is the workhorse (§5). The complex-eigenvalue refinement is architecture-conditional at head level: the previous-token solution is spectrally non-rotational (content-like) under learned-absolute positions and rotational under RoPE, where \(D_{\text{head}}\) tracks the rotational phase whose causal load §8 establishes head-locally (§6–8). This gives the recent “non-Hermitian transformer” program a concrete, controlled home—the complex structure of the QK operator does measurable, causal work precisely when position is encoded as complex phase—while honestly bounding it: on GPT-2 the non-Hermitian label is decorative.

Practical reading. For weight-only head triage: under RoPE, rope_imag_frac(or the \(D_{\text{head}}\) percentile) flags positional-routing heads without a forward pass; under absolute/ALiBi schemes the same statistics are misleading in aggregate (bulk gradients) and only the strong-head profile is informative. For interpretability methodology, the dynamics result is a caution: a clean static weight–function correspondence need not be predictive during training, and post-hoc ablation severity must not be read as developmental necessity.

Companion work (context only). Companion work in preparation analyzes the iterated attention propagator with pseudospectral tools under mask-structure nulls; none of this paper’s claims relies on it. We flag two of its directions as context: a division of labor consistent with our tool choice (eigenvalue-level summaries suffice for the static scoring form; resolvent-level tools are needed for the depth-iterated propagator), and a state-dependent character of induction heads that static weight fingerprints cannot capture — consistent with their content-like static profile in §4.

Limitations. (i) The RoPE evidence in §6–7 is correlational (shared-variance) over heads; §8 supplies causation but as a concentrated per-head effect (now on all three RoPE models). (ii) Three learned-absolute models are tested with a consistent strong-head profile but heterogeneous aggregate and moderate-band statistics, whose sources (training corpus; GPT-Neo’s alternating local attention) are unresolved; the ALiBi scheme is covered by a single model (BLOOM-1b1). (iii) Inference treats heads as units: heads share layers and training (layer-clustered design effects \(3.6\)–\(6.9\); we report clustered CIs), and Llama heads share K-projections within GQA groups of 4 (between-group variance \(0.55\)–\(0.79\)) — group-level dependence is only partially resolved. (iv) Llama runs in bf16 (float64 metrics from bf16 weights are exact w.r.t.the model; the diagonal convention check matches to bf16 precision only). (v) Single-head ablations miss composition; the prev\(\to\)induction wiring is verified by K-composition (§4) but full path patching is future work. (vi) QK biases lie outside \(M\) and are held fixed in ablations; the prev-detector’s bulk-ordering reliability is unquantified. (vii) A non-Llama full-RoPE family (Qwen/Mistral) would broaden the third architecture point.

12 Conclusion↩︎

For the attention QK operator, the plain antisymmetric split describes head function architecture-generally; the complex-eigenvalue refinement earns its keep only under RoPE, where it tracks the rotational phase whose causal load head-level ablations establish. Tracking training shows this spectral signature is born at the random-matrix null, locks in with circuit formation, and is sculpted into today’s profiles after function arrives; constrained training shows it is a default, not a necessity — every spectral channel we blocked was rerouted around, at a significant and interpretable search cost. The non-Hermitian view of attention is neither uniformly profound nor uniformly decorative: the positional scheme sets the default spectral algebra of attention’s solutions, and we measure that claim statically, dynamically, and causally in the settings analyzed. Scaled constrained pretraining is its direct falsification path.

12.0.0.1 Reproducibility & pre-registration.

Code in src/: extract, decompose, metrics, observables, semantics, p4_rope, rope, p5_rope_ablation, p9_checkpoint_dynamics, p10_training_intervention. Per-head tables: results/cache/*_head_full.parquet; checkpoint trajectories and training histories under results/cache/. The dynamics questions (Q1–Q3) and intervention predictions (P1–P4) were pre-registered in the project plan before data collection; falsified predictions (Q3, P2-strong, P3, P4-timing) are reported as such. The full artifact bundle — code, per-head tables, checkpoint trajectories, training histories, figure scripts, and the pre-registration documents — is released at https://github.com/HengyuLi-Ozaki-lab/qk-spectral-fingerprint; the repository’s version history stamps which analyses were specified before their data existed.

References↩︎