Keywords: Neural Representations; Metric Learning; Large Language Models; Interpretability; Symbolic and non-Symbolic approaches

1 Introduction↩︎

Marr’s hierarchy describes information-processing systems using three levels (1; [1]): computational, algorithmic, and implementational. The computational level defines the system’s goal. Many algorithms can solve the same computational goal. Finally, the implementational level concerns how an algorithm is realized. In biological systems, this includes physical hardware such as brains. In artificial systems, it also includes network topology and architecture. Many implementations can run the same algorithm.

Large Language Models (LLMs) fit into Marr’s hierarchy. At the computational level, most language models share the same goal: next-token prediction (but see [2] for multi-token prediction). At the implementational level, models can use very different architectures, such as Transformers, State-Space Models (SSMs), or Recurrent Neural Networks (RNNs). At the algorithmic level, the Platonic Representation Hypothesis [3] predicts increasing convergence: as models become larger and more competent, their representations should align across architectures, objectives, and even data modalities. From Marr’s perspective, different implementational choices can still lead to shared algorithmic organization. Yet differences in architecture and training recipes may also make models rely on distinct linguistic representations. Empirical work on this hypothesis has primarily used geometric alignment measures, such as mutual nearest-neighbor overlap between representational kernels, leaving open whether convergent representations are organized by the same explicit linguistic features. In this work, we ask whether language models converge at this algorithmic level, and if so which linguistic features explain that convergence.

Standard model-similarity measures compare raw representational geometry, including second-order isomorphism/RSA [4]–[8], linear regression [9], CCA [10]–[13], statistical shape analysis [14], and DSA [15]. These measures tell us whether two models share a similar raw geometric space. However, they act as a black box. They do not explain why the representations align. They cannot tell us if two models are similar because, e.g., they both heavily encode the same syntactic structure, or simply because they share superficial lexical biases.

To address this, we shift from purely geometric comparisons to feature-based representational similarity. We use metric-learning encoding models (MLEMs; [16]). MLEMs learn a metric over interpretable features of stimuli to approximate the distances between the neural representations of those stimuli. Applied to embeddings of language models on sentences with specific linguistic properties, this produces a linguistic signature, a profile that quantifies how much each linguistic feature contributes to predicting neural distances among stimuli for a specific layer or model. By comparing these signatures, we measure similarity based on explicit linguistic strategies, rather than raw geometry.

We test this approach on 43 models across three different classes of neural architectures: decoder Transformers, State-Space Models (SSMs), and Recurrent Neural Networks (RNNs). We find that the similarity between model linguistic signatures is driven most strongly by their family and release date. The latter acts as a proxy for general LLM development, summarizing correlated changes in architecture and training recipes. Parameter count, training token count, and depth make smaller contributions. This suggests that the way language models solve language processing is shaped more by model family and model generation than by scale or specific architectural differences.

More broadly, the same feature-based comparison can extend to speech, vision, or brain recordings whenever stimuli can be paired with explicit theoretical features.

2 Methodology: Linguistic Signature Similarity↩︎

2.1 Measuring Similarity through Interpretable Features↩︎

Two standard approaches relate theoretical features to neural representations: decoding and encoding models [17]. Decoding models, or probes, predict features from neural activity [18]–[20]. High decoding performance shows that feature-related information is available in the representations, but it does not show that the feature itself organizes the neural space. A feature can be decodable simply because it correlates with another variable actually used by the model [21]–[23]. Encoding models instead predict neural activity from features and can include several features or confounds at once, disentangling their contributions. However, standard encoding models are often univariate: they fit each neuron, voxel, or embedding dimension independently. They can therefore miss information encoded only in distributed geometry across a population of units [24], [25]. MLEMs keep the encoding logic but move the target from individual activations to pairwise neural distances among stimuli. They learn which feature distances best predict those neural distances, yielding feature importances for the geometry of a layer’s representations. In our approach, these importances define the layer’s linguistic signature. We then compare linguistic signatures across layers and models. If two models or layers have similar signatures, their representations are organized by similar linguistic variables, even when their raw geometries differ.

2.2 Metric-Learning Encoding Models (MLEMs)↩︎

MLEMs [16] learn a distance function over input features (e.g., tense, subject number) to match the actual distances between neural representations (2).

Formally, consider a set of \(n\) stimuli (e.g. sentences), each described by \(m\) interpretable features \(F\) (e.g., tense, gender). First, we consider the pairwise neural distances \(D^N\) between the neural embeddings \(Y_i\) and \(Y_j\) corresponding to stimuli \(s_i\) and \(s_j\) (for instance with the Euclidean distance)¹: \(D_{ij}^N = \left\|Y_i - Y_j\right\|_2\). Second, we consider the pairwise feature distances. Let \(D_{ij}^F\) be the \(m\)-dimensional vector of feature distances for a pair of stimuli: \(D_{ij}^F=\left( \;D_{ij}^{f_1} \;, \;D_{ij}^{f_2} \;, \;\ldots \;, \;D_{ij}^{f_m} \;\right)\), where \(D_{ij}^{f_k}\) is the distance with respect to feature \(f_k\). For example, for a categorical feature such as sentence tense, \(D_{ij}^{f_k}=0\) if two sentences have the same tense and \(D_{ij}^{f_k}=1\) otherwise. For an ordered feature such as word frequency, \(D_{ij}^{f_k}\) can instead be the absolute difference between the two feature values. To evaluate which input features best explain the neural distances, MLEMs learn a distance over the feature-distance vectors \(D_{ij}^F\) that best models the neural distances \(D_{ij}^N\). Formally, they learn a symmetric positive definite matrix \(W \in \mathbb{S}^{++}_m\) such that \(D_{ij}^N\) is modeled as \(\widehat{D}_{ij}^N=\left\|D_{ij}^F\right\|_W = \sqrt{{D_{ij}^F}^T W D_{ij}^F}\).

The diagonal entries of \(W\) weigh individual features, while its off-diagonal entries weigh feature interactions. MLEMs optimize \(W\) to maximize the Spearman correlation \(\rho\) between the empirical neural distances \(D_{ij}^N\) and the modeled distances \(\widehat{D}_{ij}^N\): \(\sup_{W\in\mathbb{S}^{++}_m} \rho_{i<j}\left( \;\widehat{D}_{ij}^N \;, \;D_{ij}^N \;\right)\).

The Spearman correlation focuses on preserving the relative arrangement (topology) of stimuli, rather than the absolute scale which is arbitrary in neural spaces. Because exact ranks are not differentiable, the model optimizes a differentiable relaxation of \(\rho\) [27]. The model is then trained using stochastic gradient descent on batches of stimulus pairs. Positive definiteness of \(W\) is enforced via Cholesky parameterization.

2.3 Extracting Linguistic Signatures (Feature Importance Profiles)↩︎

Feature Importances (FI) for each interpretable feature can be computed from an MLEM. Feature Importance is defined as the average decrease in the model’s Spearman correlation score on a test set when randomly permuting a specific feature’s distance matrix or that of an interaction between two features. This approach is more robust than looking at the raw weights in \(W\), which can be more affected by correlations between features and scaling [28]. The scale of FIs is more interpretable as they are differences of Spearman scores and in particular it allows for comparison between different MLEMs (e.g. trained on neural representations from different language models).

For each layer of a language model, we collect sentence-level embeddings from the last-token representation of each sentence in a synthetic dataset balanced to mitigate correlations among linguistic features of interest (described later), and compute the FIs for those features using MLEMs. We refer to the resulting FI vector as the layer’s linguistic signature. A high FI means that differences along that feature predict relatively large distances between sentence embeddings in that layer. For example, if tense has high FI, sentence pairs that differ in tense tend to be farther apart in the layer’s representation space than sentence pairs with the same tense. For an entire model, we consider the layer-wise profile formed by the linguistic signatures of its different layers and refer to this profile as the model’s linguistic signature. It tracks the development of linguistic features along its layers.

The FIs are computed on held-out sentences using 5-fold cross-validation. Throughout the different analyses, we report the average of the metric of interest across those folds and display the variability (standard deviation) as error bars or shaded areas to help reliably compare data points.

2.4 Linguistic Signature Similarity↩︎

Because a linguistic signature summarizes how strongly a layer relies on each linguistic feature, comparing signatures lets us move beyond standard geometric similarity. The question is not only whether two representations are close, but whether they prioritize the same linguistic information. We analyze linguistic-signature similarity at three levels: model-level, layer-level, and feature-level.

At the model level, we want to compare how linguistic signatures evolve across layers, but models often have different numbers of layers. A direct layer-by-layer comparison is therefore not well defined, and even models with the same number of layers may express similar feature-importance patterns at shifted relative layer positions. To address this, for each pair of models we compute normalized multi-dimensional Dynamic Time Warping (DTW; [29]) between their layer-wise FI profiles (2 (b)). DTW aligns similar feature-importance vectors along the depth axis, allowing us to test whether linguistic features follow similar profiles across layers even when models differ in layer count or features emerge at different depths. The normalization further mitigates the confounding effect of large differences in numbers of layers between models. Implementation details can be found above 11 in Appendix.

We then fit a second MLEM on the resulting model-level similarity matrix to explain its structure from model properties. The predictors are family membership, architecture class (Transformer, SSM, RNN), parameter count, release date, depth, depth-to-width ratio², and training token count. Numerical properties that span multiple orders of magnitude (parameter count, depth, and training token count) are log-scaled We included release date as a proxy for general LLM development. It summarizes many architectural choices (e.g., activation type, normalization type, positional encoding, attention type) which are highly correlated with one another, as well as factors we cannot easily retrieve or estimate (e.g., training FLOPs, training data types). We also excluded width and vocabulary size because they were highly correlated with the other variables. The remaining model properties show only moderate pairwise correlations (7 in Appendix), making the resulting FI profile more reliable.

At the layer level, we compare individual FI profiles using Euclidean distance (2 (c)). Two layers are close only if they rely on the same linguistic features with the same intensity.

At the feature level, we inspect why particular layer pairs are close or far. We compare their linguistic signatures in scatter plots to find which features lead the similarity or difference. We visualize the neural representations at those layers using Multi-Dimensional Scaling (MDS; [30]) to study whether in fact the models organize their neural space with respect to the leading features in a similar way or not. This provides a qualitative validation that linguistic-signature similarity reflects actual neural representations similarity with respect to the identified linguistic features.

Finally, we compare this feature-based approach at the layer level to a feature-agnostic baseline. We use Representational Similarity Analysis (RSA; [7]) directly on the pairwise neural distance matrices \(D^N\) for each layer. Contrary to RSA, linguistic-signature similarity is more interpretable and naturally supports the three complementary levels of analysis above.

3 Experimental Setup↩︎

We evaluated 43 autoregressive language models from 10 families, spanning 100M to 14B parameters and 3 architecture classes: decoder Transformers, State-Space Models, and Recurrent Neural Networks. The model set includes older Transformer baselines (GPT-2, OPT, Pythia), newer Transformer families (OLMo-2, Llama-3.2, Ministral-3, Qwen3), State-Space Models (Mamba, Mamba-2), and a Recurrent Neural Network family (RWKV-7). All models share the next-token prediction objective but differ in architecture, scale, training data, and release date, allowing us to test which factors explain linguistic-signature similarity. For each model and layer, we collected the hidden state of the last token of each sentence, as it is the position that has access to the full sentence in autoregressive models.

We used the Relative Clause dataset from [16] which contains 7.7k sentences generated using Context-Free Grammars. The dataset crosses relative clause attachment site (center-embedded vs. peripheral) with relative clause type (subject vs. object), and provides 12 linguistic features including syntactic properties, and lexical frequency and morphological attributes of particular elements of the sentence. Its balanced design mitigates feature correlations (6), which is important for reliable Feature Importances [28].

More details on the included models, embedding extraction procedure, and dataset design are provided in [sec:model_inventory] [sec:relative_clause_dataset] in Appendix.

4 Results↩︎

We trained an MLEM for each layer of all the 43 models. 8 in Appendix shows the performance of these MLEMs measured with Spearman correlation on the held-out splits. Across models, encoding performance follows the same broad profile: Spearman correlation is low in shallow and deep layers (\(\approx 0.2\)) and peaks in middle layers (0.5-0.7). This is consistent with prior work finding that syntactic information is primarily processed in middle layers of large language models [31], [32]. Within each family, the peak tends to occur at a similar relative layer position across model sizes, but this position differs across families.

4.1 Linguistic Signatures↩︎

We first examined the linguistic signatures themselves and how they evolve across layers (9 in Appendix). Three main patterns emerge from this analysis.

Linguistic features show different importance dynamics across model layers. This validates the sanity of the method and shows that features do not organize the sentence-level representations at the same layer depths.

In many models, the main syntactic features of the dataset, namely Relative Clause type and Attachment site, have their highest importance in middle layers. This pattern mirrors the global encoding-performance profile and recovers again prior findings that syntactic information is primarily represented in middle layers of language models.

Models within the same family show visually similar linguistic signatures, even when they differ substantially in parameter count and number of layers. This suggests that linguistic signatures are shaped by model family rather than scale.

4.2 What Makes Two Models Think Alike?↩︎

We asked which language models have the most similar linguistic signatures. To do so, we compared their layer-wise linguistic-signature profiles using Dynamic Time Warping (DTW). The resulting similarity structure is summarized with an MDS projection in 3a and the full DTW distance matrix is shown in 11 in Appendix. We see a broad axis corresponding to release date and again models visually cluster primarily by family, indicating that models from the same family tend to develop similar linguistic signatures across layers. To explain this structure quantitatively, we then treated the pairwise DTW matrix itself as the target of a second MLEM defined over model properties. The right panel of 3 shows that release date, which acts as a proxy to summarize general LLM development, is the strongest predictor of model-level linguistic signature similarity, followed by family membership, with smaller contributions from parameter count, training token count, and depth. By contrast, architecture class and depth-to-width ratio contribute little once the other model properties are included. These results suggest that similarity in model-level linguistic signatures is not primarily shaped by scale or architecture class. This pattern is consistent with the Platonic Representation Hypothesis in showing that different architectures can share representational profiles across layers. However we find that convergence is partial and structured by model family and model generation rather than collapsing to a single endpoint when scale increases.

Do models from the same family follow similar layer-wise trajectories of linguistic signatures? To address this question, we examined layer-level linguistic signatures (FI vectors) with a PCA computed over all layers from all models (4). We also show a Euclidean distance matrix between layer-level linguistic signatures in 10 in Appendix. In this shared PCA space, models from the same family trace remarkably similar trajectories, indicating that their linguistic signatures evolve similarly across layers. For many families, these trajectories form loops, with the last layers returning close to the first layers. This loop-like organization is visible even in models without tied input and output embeddings, where weight tying means sharing the input token embedding matrix with the output softmax matrix [33]. Because models must map their internal representations back to token space for next-token prediction, this return toward early-layer signatures may reflect constraints imposed by the output objective.

We also compare our approach to a feature-agnostic RSA baseline applied directly to the neural representations of the models from the GPT-2, OPT and Pythia families. 12 13 in Appendix show the RSA similarity matrix and its MDS projection. The latter shows that models from the same family tend to follow similar trajectories, much like the PCA of linguistic signatures. However, it does not clearly show the loop-like organization, even for GPT-2, whose input and output embeddings are tied. More importantly, RSA remains purely geometric: it does not identify which linguistic features drive similarity and it does not support the model-level, layer-level, and feature-level analyses enabled by linguistic signatures.

4.3 What Makes Two Linguistic Signatures Similar?↩︎

Why do two model layers represent linguistic information similarly or differently? We addressed this question by zooming in on specific Feature Importance (FI) differences between layers. For a given pair of layers, we asked which linguistic features account for their similarity or discrepancy, and whether the same features visibly organize their neural spaces. To choose informative pairs, we turned to the layer-level linguistic-signature distance matrix (10 in Appendix). Inspecting all layer pairs was impractical so for all models we kept only one layer per eighth of relative layer position. For each of them, we considered other retained layers from models in other families within the same eighth of relative layer position, then selected the ones with the closest and farthest linguistic signatures. 5 shows two illustrative comparisons from this systematic search.

The first row of 5a shows the FIs for Pythia-6.9B layer 14 against those of its closest counterpart, Mamba-790M layer 21, and its farthest counterpart, Qwen3-4B layer 16. This comparison dissociates architecture and scale from linguistic signature, since Qwen3-4B is closer to Pythia-6.9B in architecture and parameter count but much farther in linguistic-signature space. The most important feature for the similar layers is Relative Clause type and we see clear clustering in their neural spaces whereas the dissimilar Qwen3-4B layer shows little such organization (5b top row). In the second row, Pythia-1.4B layer 8 and Mamba2-780M layer 14 are again close, with Verb lemma moderately important in both cases, while Qwen3-4B layer 11 assigns much higher importance to this feature and shows correspondingly sharper clustering in neural space. This analysis shows that the feature-based approach provides a clear, interpretable explanation for why two layers are similar or different in their linguistic processing.

5 Summary and Conclusions↩︎

Language models share the same computational task of next-token prediction. However, architectural differences can drastically change the algorithms and representations they use to process text. In this paper, we introduced a framework to test whether different models converge on similar linguistic representations or instead process text in different ways. While traditional approaches treat similarity as a purely geometric property, we introduced a feature-based notion of representational similarity based on explicit linguistic signatures derived from Metric-Learning Encoding Models.

We tested this framework on decoder Transformers, State-Space Models, and Recurrent Neural Networks. At the model level, profiles of linguistic signatures are broadly organized by release date, serving as a proxy for general LLM development while architecture class seems to not be a predictive feature. Analyses at the model and layer levels comparing these profiles show that models within the same family tend to organize linguistic information similarly. Finally, at the feature level, the method makes it possible to explain particular similarities and differences by identifying which linguistic features dominate which translates into organization differences of the corresponding neural spaces.

These findings refine the kind of convergence hypothesized by the Platonic Representation Hypothesis. They support that different architecture classes, including Transformers, SSMs, and RNNs, can exhibit similar linguistic signatures. However, they do not support a simple picture in which scale alone drives all models toward one undifferentiated representational endpoint. Instead, linguistic convergence is structured by model family and model generation.

Together, these results illustrate the value of feature-based approaches. By decomposing neural spaces into symbolic dimensions, we obtain an interpretable account of how language models represent and process language. This comparison based on linguistic signatures can naturally generalize to domains like speech and vision, and to comparing artificial networks with biological brains.

6 Limitations↩︎

While our approach explains model similarity using clear linguistic features, it has limitations. First, we use a synthetic dataset where the linguistic features are largely independent, but it limits the analysis to a specific set of syntactic constructions. Future work should test whether the same conclusions hold on natural, unconstrained text.

Second, linguistic signatures are defined relative to the features included in the symbolic description of the stimuli. A model might rely on variables outside this feature set. In that case, MLEMs would still explain which of the specified features account for pairwise neural distances, but the resulting signature would only capture part of the model’s representational organization.

Third, our analyses operate at the level of whole layers. This is appropriate for comparing broad representational geometries, but it does not identify the specific units, circuits, or algorithms that produce those geometries. A feature could be perfectly computed by a localized mechanism and therefore have a weak layer-level importance. MLEMs should therefore be viewed as complementary to mechanistic, causal, and unit-level analyses: they add a feature-level account of neural distances, but they do not replace circuit-level explanations.

Our results also do not directly test the cross-modal part of the Platonic Representation Hypothesis. Our analysis is restricted to language models and to a syntactic dataset. Future work could apply MLEMs to paired text, speech, vision, and brain data using shared interpretable features, testing whether cross-modal alignment is driven by the same explicit variables rather than by raw geometric overlap alone.

7 Broader Impact↩︎

This work introduces a framework for understanding the linguistic representations of large language models. By shifting from purely geometric comparisons to feature-based analyses, our approach contributes to the growing field of representational interpretability. Improving our understanding of how different architectures process language is a critical step toward making these systems more transparent, predictable, and safe. Furthermore, explicitly identifying the linguistic features that drive model representations can eventually help researchers uncover and mitigate implicit biases encoded in neural spaces.

8 Model Inventory and Experimental Setup↩︎

We included the GPT-2 [34], OPT [35], and Pythia [36] families as older decoder-only Transformer baselines, because they represent different stages of open next-token prediction models, from early models trained on relatively modest corpora to systematic scaling suites. We also included newer families trained with more recent methods and larger datasets, namely OLMo-2 [37], Llama-3.2 [38], Ministral-3 [39], and Qwen3 [40]. These newer families were chosen for complementary reasons: OLMo-2 for its transparent open training recipe [37], Llama-3.2 and Ministral-3 as recent distilled families [38], [39], and Qwen3 for very large-scale multilingual pretraining. Finally, we included Mamba and Mamba-2 [41], [42] and RWKV-7 [43] to test whether similarities in linguistic structure persist across state-space and recurrent architectures.

We collected sentence representations as the hidden state from each layer on the last token (the period). Due to causal attention in autoregressive Transformers and the recurrent nature of SSMs and RNNs, the last token is the only token that has seen the full sentence. Therefore, only the last-token embedding can encode information from the whole sentence, including the linguistic features of interest. We also tested representations as the concatenation of all token embeddings. This richer representation was not retained because pairwise distances were driven by specific token identities rather than by linguistic features.

OPT: We excluded facebook/opt-350m because its architecture uses heterogeneous hidden state dimensions (1024 to 512 in the final layer), which prevents the uniform layer-wise analysis applied to all other models.

Pythia: We excluded EleutherAI/pythia-14m-deduped, EleutherAI/pythia-70m-deduped, and EleutherAI/pythia-160m-deduped because exploratory MLEM fits stayed near floor across layers and did not reliably recover the Relative Clause type and Attachment site contrasts. These models therefore appeared too small to produce stable sentence-level syntactic structure for the present analysis. We also excluded EleutherAI/pythia-2.8b-deduped because its FI profiles were degenerate and inconsistent with neighboring Pythia checkpoints, so we suspect a corrupted checkpoint or failed training.

9 Relative Clause Dataset↩︎

This section details the synthetic Relative Clause dataset used throughout the study. We detail its 2x2 design in [tab:rc95design], its linguistic features in 1, and the correlations between them in 6. This dataset is a derived version of the Relative Clause dataset introduced by [16]. Relative to the original release, we removed the intervener-number and intervener-gender columns, used a simplified noun lexicon and the verbs see and like to instantiate the templates, and recomputed Zipf-frequency values from those lexical items.

10 Model Properties Analysis↩︎

11 Additional Results↩︎

The remaining figures provide full visual outputs for layer-wise signatures, model-level similarity, baseline geometric similarity, and MLEM fit quality.

To compute DTW distances, we use the implementation available in the Python package dtw-python (cf. corresponding GitHub repository; [44]). We used the returned normalizedDistance attribute.

12 Compute Resources↩︎

The project, including exploration and final experiments, required roughly 2,500 CPU hours and 500 H100 GPU hours on an HPC cluster. The extraction of LLM embeddings was performed on H100 GPUs and the MLEM pipeline can leverage GPUs and was run on a mix of GPU and CPU workers.

13 Asset Licenses↩︎

The Relative Clause dataset used in this paper is a slight modification of the dataset introduced by [16]. The original dataset does not currently have a public license. We obtained authorization from the original authors to modify and redistribute this version in the supplementary material code. The public models used in the paper are released under the following terms: GPT-2 under MIT; OPT under the Meta OPT non-commercial research license; Pythia, OLMo-2, Ministral-3, Qwen3, Mamba, and RWKV-7 under Apache-2.0; Mamba-2 converted checkpoints under MIT; and Llama 3.1/3.2 under the Llama Community License and Acceptable Use Policy. The main software dependencies are under permissive open-source licenses, except dtw-python, an external GPLv3 package used to compute the Dynamic Time Warping distance. We use it as an external dependency without modifying its source code.

References↩︎

---

1. Many neural dissimilarities could be used such as ones based on cosine similarity, correlation, or Mahalanobis distance depending on the representation type and noise structure [7], [26]. We choose the classical Euclidean distance for its simplicity and interpretability.↩︎

2. We call depth the number of layers and width the number of hidden units per layer.↩︎