1 Introduction↩︎

Understanding how neural networks internally represent structured information is a central goal of mechanistic interpretability [1], [2]. While much work has focused on identifying circuits and features [3]–[5], recent studies have begun characterizing the geometry of learned representations.

[6] demonstrated that Claude 3.5 Haiku represents character counts on curved 1D manifolds in the residual stream, discretized by sparse autoencoder (SAE) features that tile the manifold like place cells tile physical space [7]. Attention heads perform geometric transformations (twisting and aligning the manifold) to compute distance to line boundaries. This raises a natural question: do these geometric motifs generalize to other tasks requiring ordinal or counting representations, and do different model architectures learn the same geometry?

We investigate four ordinal tasks across three open-weight models: Gemma-2-2B [8], Gemma-2-9B [8], and Qwen3-4B [9]. The tasks are bracket nesting depth, Python indentation level, markdown table position (row and column), and numeric magnitude. For each task and model, we (1) train linear probes to verify decodability, (2) discover manifold subspaces via PCA on per-value mean activations, (3) find SAE feature families using Gemma Scope [10] for the Gemma models, (4) analyze attention heads for geometric computation (twisting and alignment), and (5) test the necessity of manifold subspaces via activation patching.

Our main findings:

• Manifold hypothesis partially generalizes: Bracket depth and table column index produce 1D manifolds with place-cell feature tiling across all models. Indentation uses a 3D subspace. Numeric magnitude shows no coherent manifold despite strong linear decodability.

• Geometric computation is architecture-dependent: Bracket depth manifolds are pre-organized from the embedding layer in all three models. Indentation and numeric magnitude show dramatically more twisting in Qwen3-4B than in either Gemma model, despite similar probing accuracy.

• Coherent vs.incoherent twisting: High twist alone does not imply ordinal geometric computation. Qwen3-4B’s indentation twisters preserve ordinal order, while its numeric twisters do not (\(\rho \leq 0.23\)). This distinguishes heads that geometrically construct an ordinal manifold from heads that merely rearrange representations without structure.

• Subspace ablation confirms informational concentration: Activation patching of the top-\(k\) manifold directions causes dramatically larger probe accuracy drops than random-direction controls across all tasks and all three models, establishing that the identified subspaces are necessary for linear decodability of the ordinal variable.

• A key factor is local computability. Tasks where the ordinal variable is directly computable from token identity (brackets, table columns) produce clean 1D manifolds that require no geometric construction. Tasks requiring cross-position integration or semantic extraction produce higher-dimensional or incoherent representations.

2 Related Work↩︎

2.0.0.1 Representation geometry.

The study of geometric structure in neural network representations has a rich history. [11] introduced structural probing to find syntax in word representations, establishing the methodology of projecting representations into task-relevant subspaces. The logit lens [12] and tuned lens [13] revealed that intermediate representations progressively converge toward output predictions. [14] found emergent linear structure in LLM representations of truth values. [15] showed that LLMs represent continuous variables (space and time) as linear directions. [16] formalized the linear representation hypothesis, and [17] extended it, showing that some features require multi-dimensional subspaces rather than single directions. [18] argued that features are represented as manifolds and that cosine similarity encodes intrinsic manifold geometry. [19] showed that transformers represent belief state geometry in their residual stream, including non-trivial fractal structures. [6] made this concrete for counting tasks, showing that character counts trace curved 1D manifolds with SAE features acting as place-cell analogs. Their analysis of attention heads operates in Q-space, i.e. the subspace obtained by projecting residual stream activations through each head’s query weight matrix, which we adopt in Section 3. Our work directly extends this to multiple tasks and multiple open-weight models.

2.0.0.2 Theoretical grounding.

[20] proved that transformers learn factored representations in orthogonal subspaces, with dimensionality growing linearly when factors are conditionally independent. This predicts the orthogonal arrangement observed by [6] and suggests it should generalize across tasks. [21] argued that representation geometry should be understood information-geometrically rather than in Euclidean terms, which is relevant for interpreting our curved manifold findings and for the causal steering experiments we perform.

2.0.0.3 Sparse autoencoders and features.

Sparse autoencoders have emerged as a primary tool for decomposing neural network representations into interpretable features [4], [22]. [5] scaled this approach to Claude 3 Sonnet, finding interpretable features at scale. Gemma Scope [10] provides pretrained SAEs for Gemma-2 models, which we use for feature family discovery. [23] studied how SAE scaling interacts with feature manifolds, finding that manifold structure affects the number of features needed; our results on numeric magnitude (high selectivity but zero monotonic features) are consistent with their finding that SAEs struggle to decompose manifold-structured features into clean families. [24] noted that SAEs assume temporal independence, potentially missing sequential structure in how manifolds evolve across positions, which is a relevant caveat for our ordinal tasks.

2.0.0.4 Mechanistic interpretability of structured tasks.

Prior work has studied how transformers represent structured information including modular arithmetic [25] and algorithmic tasks [2]. [26] proposed a tiered framework for mechanistic understanding; if multiple tasks and multiple architectures use the same geometric motifs, this would constitute evidence for principled rather than task-specific or architecture-specific representations. Our cross-model comparison directly addresses this question.

2.0.0.5 Cross-architecture comparison.

Gemma-2 [8] uses grouped-query attention, SwiGLU activations, and alternating local/global attention. Qwen3-4B [9] uses grouped-query attention, RoPE positional encoding, and SwiGLU activations, with a substantially larger head count (32 heads vs. in Gemma-2-2B). These architectural differences, particularly the head count and positional encoding scheme, may explain the differences in geometric computation we observe.

3 Methods↩︎

3.1 Models and Tasks↩︎

We study three open-weight models: Gemma-2-2B (2.6B parameters, 26 layers, 8 heads), Gemma-2-9B (9.2B parameters, 42 layers, 16 heads), and Qwen3-4B (4.0B parameters, 36 layers, 32 heads). Qwen3-4B uses grouped-query attention and RoPE positional encoding. We use Gemma Scope [10] residual stream SAEs (16K dictionary) for the Gemma models; no pretrained SAEs exist for Qwen3-4B.

We study four ordinal tasks. Bracket nesting depth: sequences of mixed brackets with per-token depth labels (0–10); 3,000 samples, half synthetic and half from The Stack. Python indentation level: per-token indent level (0–5 in units of 4 spaces); 3,000 samples from The Stack. Markdown table position: synthetic tables with per-token row (0–14) and column (0–8) indices; 2,000 tables. Numeric magnitude: text with numbers labeled by \(\lfloor\log_{10}(\text{value})\rfloor\) (0–8); 5,000 samples from OpenWebText. All sequences are tokenized with each model’s native tokenizer (max length 256).

3.2 Probing, Manifold Discovery, and Feature Families↩︎

For each task, model, and layer \(\ell\), we collect residual stream activations at labeled token positions (4,000 train / 1,000 test). We train logistic regression probes (classification) or Ridge regression probes (numeric magnitude) and report test accuracy or \(R^2\).

For manifold discovery, we compute the mean activation \(\bar{\mathbf{h}}_v = \frac{1}{|S_v|}\sum_{i \in S_v} \mathbf{h}_i\) for each ordinal value \(v\) and apply PCA to the matrix of per-value means. We report the number of Principal Components (PCs) needed to explain 90% of variance and pairwise cosine similarities between per-value means. We note that this characterizes the geometry of class centroids: high PC1 variance among centroids is necessary but not sufficient for individual tokens to concentrate near a 1D manifold, as it is consistent with heterogeneous or multimodal within-class distributions. We use this centroid geometry because it captures how the model distinguishes ordinal values; the subspace ablation results (Section 4) provide complementary evidence that these directions are informationally concentrated.

For the Gemma models, we use Gemma Scope SAEs at four evenly spaced layers to find feature families. For each feature \(j\), the mean activation per ordinal value gives a tuning curve \(\tau_j(v)\). We rank features by between-value variance, keeping those active on \(>5\%\) of tokens. A feature is selective if its peak exceeds \(3\times\) its mean, and monotonic if its tuning curve is approximately non-decreasing or non-increasing.

3.3 Geometric Computation and Subspace Ablation↩︎

Following [6], we measure how much each attention head rearranges the manifold geometry by projecting the per-value activation means into the query (Q) space. Formally, for an attention head \(h\) in layer \(\ell\) with query weight matrix \(\mathbf{W}_Q^h \in \mathbb{R}^{d_{\text{query}} \times d_{\text{model}}}\), the projected mean for ordinal value \(v\) is computed as: \[\bar{\mathbf{q}}_v^h = \mathbf{W}_Q^h \bar{\mathbf{h}}_v\] where \(\bar{\mathbf{h}}_v \in \mathbb{R}^{d_{\text{model}}}\) is the mean residual stream activation vector for value \(v\).

Using these projections, we compute three distinct diagnostic scores for each attention head:

• Twist Score: Defined as \(1 - \text{corr}(\mathbf{d}_{\text{resid}}, \mathbf{d}_Q)\), where \(\mathbf{d}_{\text{resid}}\) is the vector of all pairwise Euclidean distances between the per-value means in the original residual stream (\(\|\bar{\mathbf{h}}_u - \bar{\mathbf{h}}_v\|_2\) for all pairs \(u, v\)), \(\mathbf{d}_Q\) is the vector of corresponding pairwise distances in the projected Q-space (\(\|\bar{\mathbf{q}}_u^h - \bar{\mathbf{q}}_v^h\|_2\)), and \(\text{corr}\) denotes the Pearson correlation coefficient. A high twist score indicates that the attention head’s projection significantly rearranges or distorts the relative geometric distances of the original manifold.

• Alignment Score: Defined as the fraction of total variance explained by the first principal component (\(\text{PC1}\)) of the Q-projected manifold centroids \(\{\bar{\mathbf{q}}_v^h\}_v\). This measures the extent to which the head compresses the projected manifold into a dominant one-dimensional axis.

• Ordinality Score: Defined as \(|\rho_{\text{Spearman}}(v, \text{PC1}(v))|\), where \(\rho_{\text{Spearman}}\) is the Spearman rank correlation coefficient calculated between the true, ordered ordinal labels \(v\) and the coordinates of their corresponding projected means along the first principal component \(\text{PC1}\) of the Q-space. A high ordinality score indicates that the head preserves the strict monotonic progression of the ordinal sequence along its primary axis.

To test whether the manifold subspaces are necessary for linear decodability, we patch the top-\(k\) PCA directions of the per-value means by replacing a token’s projection onto this subspace with that of a token with a different ordinal label, leaving the orthogonal complement unchanged. We compare the resulting probe accuracy drop against patching \(k\) random directions as a control. We report results at \(k=3\) for the layer with the largest manifold drop. We follow best practices from [27] and note the caveat of [28] that subspace patching can activate parallel pathways; our random-direction control is designed to mitigate this concern. We note that this methodology tests informational concentration in the subspace, not behavioral causality (i.e., whether the model’s downstream generation depends on this subspace).

4 Results↩︎

4.1 Manifold Structure and Feature Tiling↩︎

Linear probes achieve high test accuracy for all tasks across all three models (1). Indentation and table columns reach more than \(94\%\) test accuracy by mid-network in every model. Bracket depth peaks at 53–57% test accuracy, which is low in absolute terms but well above chance (1/11 \(\approx\) 9%), and with near-perfect training accuracy (\(\geq 99\%\)), indicating substantial overfitting. This gap likely arises because bracket-depth labels are highly correlated with token identity (specific bracket characters), creating spurious shortcuts the probe exploits on training tokens but that do not generalize to held-out tokens with different token-depth correlations. The manifold structure findings for brackets should be interpreted with this caveat: the subspace cleanly separates depth values in the training distribution, but a probe trained on it does not robustly generalize. Table rows reach 74–82% test accuracy. Numeric magnitude achieves train \(R^2 > 0.93\) in all models but test \(R^2\) varies substantially: 0.54 for Gemma-2-2B, 0.55 for Gemma-2-9B at layer 0 (degrading to negative at deeper layers due to overfitting), and 0.69 for Qwen3-4B. The ordinal variable is linearly accessible in the residual stream for every task and model studied.

4.1.0.1 Manifold dimensionality.

The tasks separate into consistent tiers across all three models (4). Bracket depth and table column index concentrate more than \(90\%\) of variance in PC1 at mid-to-late layers, replicating the curved 1D manifold found by [6] for character counts. Indentation requires 2–3 PCs for 90% variance (PC1 \(\approx\) 47–54%), likely because indent level interacts with code structure. Table rows require 1–4 PCs depending on model, with Qwen3-4B collapsing to 1D at mid-layers while Gemma models use 3–4D. Numeric magnitude requires 4–5 PCs across all models (PC1 only 40–62%), with no clear ordinal progression in the PCA projection. The existence of multi-dimensional manifolds is consistent with [17], who showed that some LLM features require multi-dimensional subspaces.

4.1.0.2 Feature families (Gemma models).

SAE features tile the bracket-depth manifold cleanly (2, 2): at layer 8 in Gemma-2-2B, more than two-thirds of the top features are selective and include a sharp depth\(=\)​0 detector, a monotonically increasing feature, and features peaking at specific depths. In Gemma-2-9B, roughly a third to half of the top features show similar place-cell structure. For indentation and table rows, selectivity (the fraction of top features with a peak exceeding three times their mean activation) is weaker (2–10 of the top 20) and features preferentially encode boundary values (indent\(=\)​0, row\(=\)​0) over fine-grained position. Numeric magnitude is the clear outlier: despite 6–12 out of 20 features being selective, zero are monotonic at any layer in either model. Tuning curves are erratic, and the SAE cannot decompose numeric magnitude into an orderly feature family. This contrasts with [29], who found monotonic representations in encoder models, and is consistent with [30]’s observation that log-scale compression makes fine ordinal distinctions harder to encode discretely.

4.2 Geometric Computation and Subspace Ablation↩︎

We analyzed all attention heads per model per task (3). The degree of geometric computation varies substantially across architectures, even when probing accuracy is similar.

4.2.0.1 Pre-organized vs.constructed manifolds.

Bracket depth manifolds are pre-organized in all models: max twist \(\leq 0.13\), residual alignment starting at 0.65–0.78 and climbing to near-perfect (3, left). We attribute this to bracket depth being directly encoded in token identity; the embedding layer already places tokens at the correct manifold position, so heads need only read the manifold, not construct it. This contrasts with [6]’s character counting, where the count depends on all preceding characters. For indentation and table rows, residual alignment starts low and climbs through the network, indicating active manifold construction by attention heads.

4.2.0.2 Coherent vs.incoherent twisting.

The most notable cross-architecture difference is in twisting magnitude. Qwen3-4B shows 4–10\(\times\) higher maximum twist than Gemma for indentation (0.803 vs.–0.331) and numeric magnitude (0.724 vs.–0.288). Crucially, Qwen3-4B’s indentation twisters preserve ordinal order (\(\rho \geq 0.77\)), they rearrange the manifold while maintaining the ordinal progression, exactly the behavior [6] found for character counting. Its numeric twisters do not (\(\rho \leq 0.23\)), indicating incoherent rearrangement rather than structured geometric computation. The twist concentration ratio (max/mean) is similar across models (\(\approx\)​5–8), meaning the difference is not specialization but rather that all Qwen3-4B heads twist more strongly in absolute terms, possibly reflecting its RoPE positional encoding or attention architecture (3, right).

4.2.0.3 Subspace ablation.

Activation patching confirms that the manifold subspaces are necessary for linear decodability (4). Patching the top-3 manifold directions causes large drops in probe accuracy, while random directions cause negligible drops (\(<\)​0.01 except Qwen3-4B numeric). The ratio exceeds more than 28 times for every task–model combination. This is consistent with informational concentration in the identified subspace, not behavioral causality, whether patching shifts downstream generation remains an open question. Notably, even numeric magnitude, which lacks clean manifold structure, shows large manifold drops, indicating that high-dimensional representations can still concentrate task-relevant information in a small subspace. For Qwen3-4B numeric, the elevated random drop suggests this representation is less geometrically organized.

\(^\dagger\) Gemma-2-9B numeric probe overfits severely (train \(R^2=0.99\), test \(R^2=-2.92\)). The large manifold drop (20.0) reflects that patching disrupts the representation, but the absolute value should be interpreted cautiously given the broken baseline.

4.2.0.4 Cross-task sharing.

Features #9213, #15567, #7214, and #6631 appear among the top-20 features for multiple tasks in Gemma-2-2B, but all are primarily value\(=\)​0 detectors; better characterized as “beginning of structure” features than general ordinal position detectors. Cross-task overlap of top-20 twisting heads is modest (0–8 shared heads per pair) and model-dependent. No head appears in the top-20 twisters for all four tasks in any model.

5 Discussion and Conclusion↩︎

Our results suggest a tentative taxonomy of ordinal representations based on how the variable relates to token identity, though it is derived from only four tasks and awaits validation on additional domains. Tasks where the variable is locally computable from token identity (bracket depth, table column index) produce clean 1D manifolds pre-organized from the embedding layer. Tasks requiring cross-position integration (table rows, indentation) produce higher-dimensional manifolds (2–4D) that are actively constructed by attention heads. Tasks requiring semantic extraction (numeric magnitude) produce diffuse, high-dimensional representations (4–5D) without clean manifold structure, despite strong linear decodability. This distinction between decodability and geometric structure is important: our “manifold dimensionality” refers to the centroid manifold (PCA on per-value means), and low centroid dimensionality does not guarantee that token-level representations concentrate near a low-dimensional curve, though the subspace ablation results indicate these directions are informationally privileged.

The degree of geometric computation is architecture-dependent: Qwen3-4B shows 4–10\(\times\) stronger twisting than Gemma for tasks requiring manifold construction, with its indentation twisters preserving ordinal order while its numeric twisters do not. This may reflect RoPE positional encoding or architectural differences rather than head specialization, since the twist concentration ratio is similar across models.

5.0.0.1 Limitations.

The bracket depth probe overfits substantially (99% train, 53–57% test). The synthetic bracket data has a minor labeling inconsistency (always closing with ‘)’ regardless of opener). No pretrained SAEs exist for Qwen3-4B, so the two central findings, place-cell feature tiling and strong geometric twisting, are demonstrated on different models; training an SAE on Qwen3-4B to bridge this gap is an important next step. The twist score operates on per-value mean activations and may miss transformations on individual tokens. Our subspace ablation establishes informational concentration, not behavioral causality.

5.0.0.2 Future directions.

Training SAEs on Qwen3-4B would test whether stronger geometric computation corresponds to cleaner feature tiling. The information-geometric framework of [21] may better characterize curved manifolds than Euclidean PCA. Extending to larger models (Gemma-2-27B, Qwen3-14B) would test whether architecture-dependent differences scale with model size. Behavioral causality experiments, testing whether patching subspaces shifts downstream generation, would strengthen the causal interpretation of our findings.

References↩︎