1 Introduction↩︎

Publicly available large language models (LLMs) have been growing rapidly in recent years, with hundreds of thousands of models now available through open-source platforms such as Hugging Face (HF) [1]. These models exhibit significant diversity in their training trajectories, including variations in model architecture, parameter scale, training data, and empirical performance [2]. As the ecosystem grows, there is an increasing need for reliable and efficient methods to characterize, compare, and manage LLMs beyond task-level benchmarks. Representative examples include identifying whether deployed LLM systems can be traced to the same backbone foundation model for intellectual property and safety purposes [3], as well as tracking and organizing model variants and versions in large-scale LLM databases [4]. Moreover, LLMs present distinct idiosyncrasies across downstream tasks, with different models specializing in different areas such as code generation, mathematical reasoning, long-context processing, or interactive dialogue. These considerations point to the need for a principled and data-light framework to trace model sources, capture model idiosyncrasies, and quantify model performances for large-scale LLM collections.

In this work, we introduce a spectral shape-based perspective for LLM quantification that builds on insights from Heavy-Tailed Self-Regularization (HT-SR) theory [5]–[7]. HT-SR theory analyzes the eigenspectrum of weight matrices, and prior studies [6], [8] have shown that the shape of the empirical spectral density (ESD) encodes informative signals about model capacity, quality, and training dynamics that are not well captured by scale-based measures. Moreover, ESD shape metrics are data-independent, computationally lightweight, and remain robust under noise and scaling. These properties make them particularly well-suited to large-scale LLM settings, where task-specific evaluation can be expensive. An overview of different quantification metrics is shown in Tab. 1.

We use the shape metric of the ESD, namely PL_Alpha_Hill, as a spectral signature of each LLM. The PL_Alpha_Hill metric summarizes the global structure of layer-wise weight eigenspectra into a compact representation, enabling efficient storage and comparison across large model collections. We adopt this spectral signature to support three downstream tasks: supervised model classification, unsupervised clustering of LLMs, and quantifying LLM performance. Our contributions are summarized below.

• We introduce a shape-based metric for reliably and efficiently characterizing, comparing, and quantifying LLMs beyond task-level benchmarks. Our metric is scale-invariant, data-free, theory-driven, and easy-to-compute.

• We curate a diverse and representative LLM corpus of up to 499 models and use it to benchmark three tasks: LLM similarity measurement, model classification and clustering, and performance prediction. We further conduct a systematic comparison of our spectral signature against a range of baselines across various experimental settings, demonstrating consistent effectiveness across tasks and conditions.

• We compare shape- and scale-based metrics under output-invariant reparameterizations and noise perturbations. ESD shape metrics remain stable, while scale-based baselines degrade, indicating that spectral shape captures intrinsic weight structure for reliable LLM quantification.

2 Related Work↩︎

2.0.0.1 Model Lineage and Independence Testing.

The growing population of open-source LLMs has spurred research into determining LLM lineage—whether two models are independently trained or one is trained from another—which is crucial for intellectual property and model safety. Recent works use distinct approaches: output-based methods use output comparison [12], fingerprint queries [13], and lexical cues [14], [15] to compare models without requiring parameter access; watermarking techniques inject provenance information during training or decoding [16]–[19]; whereas white-box methods perform hypothesis testing [3] or fingerprinting using metrics derived from model weight structures [11], [20]–[22].

2.0.0.2 Model Zoos and LLM Embeddings.

There has been growing interest in exploring representation-based approaches to capture model dependence, properties, and relationships, enabling efficient organization and management of large model zoos. Relevant directions include LLM embeddings [1], weight-space learning [2], [4], [23], and loss-landscape-based grouping [23], [24]. LLM embedding methods [1] learn compact and model-level representations that can be used to predict downstream performance across tasks without additional inference. In contrast, weight-space learning approaches operate directly on model parameters. For example, Horwitz et al. [2] combine weight-space distances with model metadata to construct an “atlas’’ of open models, enabling similarity search and provenance analysis within large repositories. A complementary line of work [24] considers loss-landscape and phase-based information [25], such as mode connectivity and CKA similarity, to partition models into distinct regimes and assess their similarity.

2.0.0.3 HT-SR Theory.

HT-SR theory is a theoretical framework based on random matrix theory for analyzing weight matrices geometry of neural networks (NNs) [5], [26]–[28]. HT-SR theory was proposed based on the observation that well-trained, state-of-the-art NNs often exhibit HT structures in the ESD of each layer. In the meantime, several rigorous theories in stochastic gradient descent relating HT phenomena to generalization performance were established, providing further theoretical support for HT-SR theory [29]–[33]. Recent works extend theoretical foundations and show that HT-SR Theory can be used to assess the training quality of modern deep NN models across various domains (e.g., computer vision and natural language processing) without accessing any training or test data [5], [8]. Based on this insight, recent studies have further proposed HT-SR theory-based methods for practical NN applications, including hyperparameter tuning [34]–[36] and model pruning [36], [37]. In this work, we use the HT-SR theory to analyze the geometric similarities of weight matrices of LLMs.

3 Preliminaries↩︎

3.0.0.1 Problem Setup.

In large LLM repositories, many checkpoints share a common pretrained backbone but differ through post-training, distillation, pruning/quantization, or model merging. Our goal is to perform data-free lineage analysis: comparing models and tracing backbone relationships directly from their weights. This requires robustness to output-invariant transformations, such as uniform scaling, hidden-unit permutations, and linear-chain reparameterizations, which can make naive weight-space similarities unreliable. We therefore focus on spectral shape statistics of weight matrices, which capture intrinsic weight structures while being less sensitive to transformations that change the parameterization but preserve the model’s input-output function.

3.0.0.2 Weight Analysis with HT-SR Theory.

HT-SR theory [5] demonstrates the empirical fact that well-trained NNs tend to exhibit strong correlations in weight matrices, resulting in a heavy-tailed ESD of each weight matrix. ESD characterizes the eigenvalue distribution of the correlation matrix \(\mathbf{W}^{\top}\mathbf{W}\) of the weight matrix \(\mathbf{W}\). HT-SR theory can be used to analyze the dynamics of weight matrix ESDs during training, by modeling the tail component of the ESD with a power-law (PL) distribution within the interval \((\lambda_{min}, \lambda_{max})\), and its tail index \(\alpha\): \(p(\lambda) \propto \lambda^{-\alpha}, \quad \lambda_{min} < \lambda < \lambda_{max}\).

One commonly used method to approximate the tail index \(\alpha\) is with the Hill Estimator¹ [41], which computes an estimate PL_Alpha_Hill: \[\text{PL\_ Alpha\_ Hill} = 1 + \frac{k}{\sum_{i=1}^{k}\operatorname{ln}\frac{\lambda_{n-i+1}}{\lambda_{n-k}}},\] where \(\{\lambda_i\}_{i=1}^n\) is sorted in ascending order. For the selection of \(k\), we adopt the approach from previous works [8], [34], where we set \(k=\frac{n}{2}\). PL_Alpha_Hill measures the shape of the eigenspectrum of each layer individually. Unlike norm-based metrics such as the spectral norm or RMS norm, PL_Alpha_Hill is robust under scaling. Moreover, since permutations are orthogonal changes of basis, the spectrum of \(\mathbf{W}^{\top}\mathbf{W}\) (and hence its ESD shape) is invariant to hidden-unit permutations, motivating PL_Alpha_Hill as a stable building block for our spectral signatures (Sec. 4.1). PL_Alpha_Hill also measures the dynamics of the ESD shape, and previous works [8], [35] have shown that PL_Alpha_Hill reflects the training quality of different layers. Layers with larger PL_Alpha_Hill tend to be “over-trained,” while layers with smaller PL_Alpha_Hill tend to be relatively “under-trained.”

3.0.0.3 Robustness of ESD Shape Metrics in Post-training.

A critical prerequisite for a spectral signature to serve as a reliable lineage identifier is its robustness across downstream adaptations. We validate the robustness of the layer-wise PL_Alpha_Hill as a spectral signature by examining its stability under post-training transformations. We observe that the impact of post-training on the weight ESD is minimal; the spectral signature is predominantly shaped during the pre-training phase. As shown in Fig.  1, the spectral signature exhibits distinct behaviors for intra-family and inter-family comparisons. We observe a near-perfect spectral overlap between Llama-3.1 and its post-trained Tulu-3 variant. Simultaneously, the separation from the Llama-2 baseline is clear. This distinction supports that the spectral signature is primarily determined during pre-training and remains robust to post-training adjustments. We provide additional results in Appendix 7.1 to further corroborate these findings.

4 Method↩︎

Using robustness properties of PL_Alpha_Hill established in Sec. 3, we present our methodology for applying PL_Alpha_Hill-based spectral signatures to model quantification. As illustrated in Fig. 2, for a given collection of LLMs, we extract the PL_Alpha_Hill statistic from each weight matrix and aggregate these measurements to construct a model-level spectral signature, as formalized in Sec. 4.1. We then use these spectral signatures to address three model quantification tasks: (1) measuring similarity between LLMs, (2) model classification and clustering, and (3) predicting downstream LLM performance. The detailed methodology is shown in Sec. 4.2.

4.1 Formulation of Spectral Signatures↩︎

For a model \(f_{\theta}\) in parameter space \(\Theta\) with \(L\) transformer blocks and \(N_{\text{mod}}\) modules, we quantify the model by constructing a spectral signature: \(\mathbf{Z}\in\mathbb{R}^{N_{\text{mod}}\times L}\), which is a tensor consisting of the PL_Alpha_Hill measure of each module. Each row \(\mathbf{Z}_{i, :}\in\mathbb{R}^{L}\) is a vector of PL_Alpha_Hill of modules of the same type. For a standard transformer architecture, we have \(N_{\text{mod}} = 7\). The tensor representation of spectral signature creates a mapping \(\Theta \to \mathbb{R}^{N_{\text{mod}}\times L}\) that extracts spectral information of each weight matrix in the model. It allows module-level analysis, as different types of transformer layers often undergo different degrees of structural changes during post-training. Our empirical observation shows that Multi-Layer Perceptron (MLP) projection layers present larger separation in independently trained models compared to attention layers.

4.2 Spectral Signatures for Model Quantification↩︎

The spectral signature encodes robust spectral information of each weight matrix in an LLM, which can be used for systematic management and quantification of large collections of LLMs. In this section, we describe how spectral signatures can be systematically applied to model quantification tasks, and we demonstrate their use in the following representative scenarios.

4.2.0.1 Measuring LLM Similarities.

To quantify the relations between models in a model collection \(\mathcal{F}\), we define a similarity measure based on their spectral signatures. Let \(\mathbf{Z}^{(k)} \in \mathbb{R}^{N_{\text{mod}} \times L}\) denote the spectral signature of model \(k\), where the \(i\)-th row \(\mathbf{Z}^{(k)}_{i, :}\in\mathbb{R}^{L}\) corresponds to the vector of PL_Alpha_Hill of all occurrences of the \(i\)-th module type (e.g., Attention Q/K/V, MLP). Given two models \(f_{\theta_1}, f_{\theta_2}\) with signatures \(\mathbf{Z}^{(1)}, \mathbf{Z}^{(2)}\), we compute the pairwise similarity by averaging the Spearman rank correlations across aligned modules: \[\label{eq:similarity} \mathrm{Sim}(\mathbf{Z}^{(1)}, \mathbf{Z}^{(2)}) = \frac{1}{N_{\text{mod}}} \sum_{i=1}^{N_{\text{mod}}} \rho_s(\mathbf{Z}^{(1)}_{i, :}, \mathbf{Z}^{(2)}_{i, :}),\tag{1}\] where \(\rho_s(\cdot, \cdot)\) is the Spearman’s rank correlation coefficient: \[\label{eq:sr} \rho_s(\mathbf{u}, \mathbf{v}) = \mathrm{corr}\!\left(\mathrm{rank}(\mathbf{u}),~\mathrm{rank}(\mathbf{v})\right).\tag{2}\] We use Spearman correlation in Eqs. 1 –2 because it depends only on the relative depth-wise ordering of PL_Alpha_Hill values, making the similarity less sensitive to module-specific affine distortions of the statistics and improving comparability across scales. More importantly, PL_Alpha_Hill itself provides exact invariances that directly match our output-invariant stress tests. For any weight matrix \(\mathbf{W}\), let \(\mathbf{C}=\mathbf{W}^{\top}\mathbf{W}\) with eigenvalues \(\{\lambda_j\}\). If \(\mathbf{W}' = c\,\mathbf{P}\mathbf{W}\mathbf{Q}\) where \(c>0\) and \(\mathbf{P},\mathbf{Q}\) are orthogonal (including permutations), then \({\mathbf{W}'}^{\!\top}\mathbf{W}' = c^2\,\mathbf{Q}^{\top}\mathbf{C}\mathbf{Q}\) has eigenvalues \(\lambda'_j=c^2\lambda_j\). Since the Hill estimator used for PL_Alpha_Hill depends on log-ratios of tail eigenvalues, these ratios are unchanged and thus \(PL\_ Alpha\_ Hill(\mathbf{W}')=PL\_ Alpha\_ Hill(\mathbf{W})\). This yields permutation- and scale-robust similarity without solving explicit neuron-matching problems that arise in matrix-level fingerprints such as AWM [20].

4.2.0.2 Model Classification

The properties of the spectral signature further enable its use for identifying the lineage of an LLM among candidate base models, as models derived from the same base model exhibit highly similar eigenspectrum shapes, whereas models with different origins show substantial divergence in shape. We approach the lineage identification as an unsupervised classification problem in the spectral signature space, and we use a nearest-neighbor classifier based on the spectral similarity from Eq. 1 . Let \(\{\mathbf{Z}^{(k)}\}_{k=1}^{K}\) denote the set of spectral signatures for \(K\) known base foundation models. For a given query model with spectral signature \(Z\), we compute its similarity to each base model via the spectral signature similarity formula defined in Eq. 1 . The predicted lineage \(\hat{k}\) is the base model that maximizes this similarity: \[\label{eq:classification} \hat{k} = \operatorname*{arg\,max}_{k \in \{1, \dots, K\}} \operatorname{Sim}(\mathbf{Z}, \mathbf{Z}^{(k)})\tag{3}\]

4.2.0.3 Model Clustering.

Spectral signature of LLMs further enables the discovery of underlying genealogical structures within large model collections without supervision. We formulate this problem as a clustering problem in the space of spectral signatures. Given a model collection \(\mathcal{F} = \{f_{\theta_1}, \dots, f_{\theta_N}\}\), we extract the spectral signature \(\mathbf{Z}^{(k)} \in \mathbb{R}^{N_{\text{mod}} \times L}\) of each model \(f_{\theta_k}\). For vector-based clustering algorithms, we flatten \(\mathbf{Z}^{(k)}\) into a feature vector \(\mathbf{z}^{(k)} \in \mathbb{R}^{N_{\text{mod}} \cdot L}\). We use spectral signatures for four unsupervised pipelines: HDBSCAN, Bayesian Gaussian Mixture Models (BGMM), Spectral Clustering, and K-Means. Due to the high dimensionality of the spectral signature representations, we apply Uniform Manifold Approximation and Projection (UMAP) as a preprocessing step for HDBSCAN, BGMM, and K-Means. In contrast, Spectral Clustering operates directly on the original high-dimensional pairwise distance matrix, where dimensionality reduction is implicitly handled through the graph Laplacian eigenstructure. Hyperparameters for clustering algorithms (i.e., number of clusters) are selected by maximizing the silhouette score. To ensure that clustering quality reflects intrinsic structure in the weight space rather than artifacts introduced by the embedding process, all internal evaluation metrics (e.g., silhouette scores) are computed using distances in the original spectral signature space. We assess the robustness and generalizability of each clustering pipeline by repeating experiments under random subsampling at varying sample sizes, verifying consistent clustering behavior across regimes.

4.2.0.4 Predicting LLM Performance.

To quantify the predictive power of the proposed spectral signature, we treat the performance estimation as a non-parametric regression task over the model manifold. Given a query model \(f_{\theta}\) with spectral signature \(\mathbf{Z}\) and a reference pool of models \(\mathcal{P} = \{(\mathbf{Z}^{(i)}, \mathbf{b}_i)\}_{i=1}^N\), where \(\mathbf{Z}^{(i)}\) is the spectral signature and \(\mathbf{b}_i\) is the vector of benchmark scores, we estimate the performance of \(f_{\theta}\) using distance-weighted \(k\)-nearest neighbor (\(k\)-NN) Interpolation. Specifically, we first derive a distance metric \(d(\mathbf{Z}, \mathbf{Z}^{(i)})\) from the spectral similarity \(\mathrm{Sim}(\mathbf{Z}, \mathbf{Z}^{(i)})\). We employ the transformation \(d(\mathbf{Z}, \mathbf{Z}^{(i)})= 1 - \mathrm{Sim}(\mathbf{Z}, \mathbf{Z}^{(i)})\). This ensures that models with highly correlated spectral shapes are positioned closer in the embedding space. The predicted benchmark scores \(\tilde{\mathbf{b}}\) are computed by aggregating the top-\(k\) most similar neighbors in \(\mathcal{P}\): \[\tilde{\mathbf{b}} = \frac{\sum_{i \in \mathcal{K}} w_i \cdot \mathbf{b}_i}{\sum_{i \in \mathcal{K}} w_i},\], where \(\mathcal{K}\) is the set of \(k\) nearest neighbors, and \(w_i = 1 / (d(\mathbf{Z}, \mathbf{Z}^{(i)}) + \epsilon)\) is the assigned weight. This interpolation relies on the assumption that models that exhibit similar HT-SR patterns in their weight matrices are likely to possess comparable generalization capabilities. We then quantify the prediction accuracy with: \[D_{perf}=\|\tilde{\mathbf{b}}-\mathbf{b}\|, \label{eq:Perfdist}\tag{4}\] where \(\mathbf{b}\) represents the ground truth benchmark scores. Typically, \(\|\cdot\|\) in Eq. 4 is defined by mean absolute error (MAE).

5 Experiments↩︎

In this section, we demonstrate that spectral signatures provide an efficient and accurate tool for a range of downstream tasks. Specifically, we present details of the experimental setup as well as the results of (i) measuring LLM similarities, (ii) model classification and clustering, and (iii) predicting and ranking model performance.

5.1 Measuring LLM Similarities↩︎

5.1.0.1 Model Corpus.

We select Llama-2-7b [42] as the base model and compare it with four types of related variants that share the same architecture: (i) post-trained models (SFT and RL), (ii) merged models and pruned models, (iii) noise-perturbed models obtained by adding i.i.d. Gaussian perturbations to all weight entries with noise strength \(\gamma \in \{0.05,0.30,1.00\}\), as defined in Appendix 7.6, (iv) output-invariant transformations on adjacent matrices and output-invariant permutations of hidden units, as defined in Appendix 7.6. This corpus isolates a key practical goal: a lineage/similarity signal should remain stable under post-training and output-invariant reparameterizations, while being informative under destructive edits (e.g., aggressive pruning). We further validate the generalizability of our spectral signature to Mixture-of-Experts architectures and cross-depth model pairs in Appendix  7.5

5.1.0.2 Setup.

We compare against both data-aware and data-free baselines. For data-aware methods, we use REEF [10], which computes similarity from activation features on a fixed prompt set (300 prompts for TruthfulQA [43]), and Logits [9], which measures similarity from model outputs on the same prompts. For data-free approaches, we apply PCS [11] that computes cosine similarity between flattened weight tensors under a fixed alignment, and discuss alignment-based weight signature (e.g., AWM [20]) as a complementary point of comparison: these methods operate at the matrix level and often require per-pair matching/alignment, which makes them less convenient as a drop-in kernel when building dense similarity matrices for zoo-scale clustering and retrieval.

5.1.0.3 Results.

Tab. 2 summarizes the overall similarity performance of our method compared to baseline models. It demonstrates the robustness of our spectral signature under diverse scenarios, including fine-tuning, pruning, model merging, noise injection, and output-invariant attacks. First, the similarity scores of our method remain consistently high among Llama-2-7b related variants, such as multiple fine-tuned models, with scores in the \([0.97,1.0]\) range across instruction tuning and domain shifts. In contrast, data-aware baselines can fail under prompt- or language-induced behavior changes. For example, Logits drops sharply for the Chinese variants, and REEF degrades for the LoRA-unloaded model, whereas our scores remain stable. Under weight noise injection, our similarity decreases mildly as \(\sigma\) increases from \(\approx 0.999\) at \(\gamma=0.05\) to \(\approx 0.979\) at \(\gamma=1.00\), outperforming PCS and REEF at the strongest noise perturbation. Moreover, our method is robust to hidden-unit permutations and random scaling under output-invariant transformations, while PCS is sensitive to both. For unstructured pruning, the similarity score using spectral signature decreases gradually at 30–50% sparsity and drops substantially at 70% sparsity, indicating that the spectral signature is informative of the structural changes under more aggressive parameter removal.

Quantifying similarity among large model corpora typically requires constructing a stable and efficient pairwise similarity matrix. However, existing methods often fall short computationally: data-dependent methods require prompts and incur inference costs for each comparison, while pair-specific methods such as AWM incur cumulative overhead when constructing dense similarity matrices over large model collections. Our approach computes a compact signature once per model and yields a computationally-cheap similarity that remains robust under the varying transformations that break coordinate-dependent weight distances, making it practical for large-scale lineage analysis.

5pt

5.2 Model Classification and Clustering↩︎

5.2.0.1 Model Corpus.

We construct a corpus of 128 publicly available models derived from 8 model families: llama-7b [44], llama-2-7b [42], llama-3.1-8b [45], mistral-7b-v0.1, mistral-7b-v0.2 [46], open-llama-7b, open-llama-7b-v2 [47], and qwen-1.5-7b [48]. For each derived model, we assign a training-lineage label corresponding to one of the eight base models, using the model’s publicly documented training provenance. For the classification task, we classify each of the 128 derived models into one of the 8 base families by selecting the base with the highest spectral signature similarity score to the model. For the clustering task, we perform unsupervised clustering on the signatures of all 128 derived models and 8 base models.

5.2.0.2 Setup.

For the classification task, we compute the similarity score using three different kernels on spectral signatures: Spearman rank correlation, radial basis function (RBF) kernel, and cosine similarity (detailed in Appendix Sec. 7.2). This allows us to assess the robustness of classification performance under different similarity measures. For the clustering task, we conduct two complementary studies. First, we assess clustering performance using spectral signatures constructed with different ESD-based metrics, including shape metrics (PL_Alpha_Hill), scale metrics (Log_Norm and Spectral_Norm), and a combined shape–scale metric (Log_Alpha_Norm). Second, we evaluate the robustness of clustering by applying four clustering algorithms, as described in Sec. 4, to these spectral signatures. We assess clustering quality using the Silhouette Coefficient [49] and the Davies–Bouldin Index [50], as detailed in Appendix 7.3.

5.2.0.3 Classification Results.

Fig. 3 shows that the proposed spectral signature enables accurate lineage classification across all eight model families with an overall accuracy of 98.44%. Across similarity kernels, the classifier consistently achieves near-perfect accuracy, indicating that lineage information is strongly preserved in the spectral signature. Performance is stable across Spearman rank, RBF, and cosine similarity, suggesting that the results are not sensitive to the choice of similarity measure. Minor accuracy variations are mainly observed for closely related families, such as mistral-7b-v0.1 and mistral-7b-v0.2, which correspond to different versions within the same model line and share highly similar architectures and training procedures. Overall, these results demonstrate that spectral signatures provide a reliable and discriminative basis for model lineage classification.

5.2.0.4 Clustering Results.

Tab. 3 compares the clustering performance of spectral signatures constructed from scale- and shape-based metrics. The proposed shape-based PL_Alpha_Hill achieves the best performance, with the highest Silhouette score of 0.91 and the lowest Davies-Bouldin index of 0.22, consistently outperforming the scale-based alternatives. To further evaluate the robustness of our proposed spectral signature, in Tab. 4 we report the result of different clustering algorithms. All clustering algorithms that use the spectral signature achieve high silhouette scores, ranging from 0.73 to 0.91. This shows that the proposed spectral signature provides a robust and effective embedding for different classes of clustering algorithms. As an illustrative example, in Fig. 9 we provide the clustering result of 2D t-SNE using the spectral clustering algorithm with spectral signatures constructed by PL_Alpha_Hill. As we can see, the proposed spectral signature clearly separates different model families, resulting in distinct clusters.

5.3 Quantifying and Ranking LLM Performance↩︎

5.3.0.1 Model Corpus.

For model ranking and prediction tasks, we curate 499 open-source models from the Open LLM Leaderboard [51]. This corpus spans a broad spectrum of architectures, focusing on the Llama, Mistral, GPT, and Pythia families, with parameter scales ranging from 19 million to 70 billion. Further validation on additional scales and intra-family prediction is provided in Appendix 7.5.

5.3.0.2 Setup.

To evaluate whether spectral signatures can predict downstream performance, we consider four representative benchmarks: ARC-Challenge (reasoning) [52], HellaSwag (commonsense) [53], MMLU (world knowledge) [54], and TruthfulQA (factuality) [43]. The models span a wide performance range (e.g., MMLU scores from 23.0 to 70.67), covering both low- and high-performance regimes. For each model, we estimate performance using distance-weighted \(k\)-NN interpolation (\(k=3\)) based on spectral-signature similarity. We use 5-fold cross-validation to assess generalization.

We compare against two activation-based baselines. First of all, EmbedLLM [1] encodes each model by extracting the hidden-state embedding at the final token position from the last transformer layer on a fixed prompt set \(\mathcal{P}\), and trains a linear regressor to predict benchmark scores under leave-one-out evaluation. Second, LLMDNA [55] extracts a \(128 \times 128\) model representation from 100 random samples, which is nearly invariant to the specific dataset or questions used for extraction. We use the same regression protocol for LLMDNA as for EmbedLLM. Prediction quality is evaluated by MAE and Kendall’s Tau rank correlation [56]. We define “Significance” as the number of repeated train/test splits (out of 100), where Kendall’s Tau between predicted and ground-truth rankings is statistically significant at the 5% level.

We further discuss why our spectral signatures are well suited for this setting. Activation-based performance quantification requires task data at inference, and does not scale well to frequent re-ranking across large model collections. Conversely, other weight-based methods such as PCS [11] and AWM [20] are primarily designed for pair-wise provenance checks and do not naturally provide a compact, geometry-aware representation amenable to regression across heterogeneous architectures. Our spectral signatures provide a lightweight, task-agnostic representation that supports nearest-neighbor interpolation without any prompts.

5.3.0.3 Results.

We run 100 repeated trials with random train/test splits (249 models for training and 250 models for testing in each trial). Fig. 4 compares predicted benchmark scores with their ground-truth values. The points cluster closely around the diagonal (\(y=x\)), indicating good agreement between predicted and actual performance. We report (i) the average MAE over the 100 trials, which measures absolute prediction error, and (ii) the number of trials in which Kendall’s Tau correlation between predicted and ground-truth rankings on the test set is statistically significant. As shown in Tab. [tbl:tab:significance], spectral signatures achieve consistently low MAE and statistically significant rank correlation in all trials across all benchmarks. While EmbedLLM achieves slightly lower MAE on ARC and TruthfulQA, spectral signatures outperform EmbedLLM on MMLU and remain competitive on HellaSwag. This gap reflects that benchmarks targeting specific model capabilities (TruthfulQA, ARC) are more directly reflected in task-specific activations, while benchmarks covering general knowledge (MMLU, HellaSwag) align more closely with the pretraining capacity encoded in weight geometry. LLMDNA [55], despite requiring comparable inference cost, yields substantially higher MAE across all benchmarks. Unlike EmbedLLM, which extracts task-conditioned activations, LLMDNA produces a single task-agnostic embedding per model. The compression loses the benchmark-specific variability needed for accurate performance prediction.

5.3.0.4 Computational Cost.

We report a runtime comparison between our spectral signature and baselines under the same model and hardware settings. We use Llama-2-7b as the representative backbone and run all methods on a single NVIDIA L40 GPU with FP16 precision. For EmbedLLM, we follow the original pipeline and extract last-token hidden-state embeddings on a fixed prompt set. For LLMDNA, we largely follow the original pipeline, with slight modifications to adjust for our hardware and specific models in our dataset. For our method, we compute the spectral signatures directly from model weights without any forward passes. We measure Run Time from the end of model loading to the end of signature/embedding extraction. One iteration of EmbedLLM embeddings takes 2,633 seconds, and one iteration of LLMDNA embeddings takes 5,393 seconds, while the Runtime of Spectral Signature is 337 seconds, significantly more time-efficient than the baselines.

5pt

5.4 Generalization to Diverse Architectures↩︎

Here, we test our spectral signature on more diverse architectures.

5.4.0.1 Model Lineage on MoE models

We evaluate our spectral signature on Pangu-Pro-MoE [57], an MoE architecture derived from the Qwen2.5-14B family. To construct spectral signatures for MoE modules, we average the spectral metrics across experts. Tab. 6 reports the similarity between Pangu-Pro-MoE and candidate source models. Our method identifies Qwen2.5-14B, Qwen2.5-Coder-14B, and OpenMath-Nemotron-14B as the most likely sources, demonstrating that spectral signatures generalize to MoE architectures with minimal adaptation.

5.4.0.2 Similarity Between Depth-Mismatched Models.

Models derived from the same source may differ in depth. We use an order-preserving layer matching via dynamic programming (DP)[58] on spectral signatures. We build a cost matrix \(D[i,j] = mean|Z_1[i] - Z_2[j]|\) over 7 z-score normalized module columns, then find the injective order-preserving mapping minimizing total cost via DP. We evaluate on the model pair (Llama-3.1-8B vs. Llama-3.2-3B) to enable direct comparison with their activation-based ground truth. Tab. 7 shows a \(3.7\times\) lower matching cost for the related pair. Moreover, the optimal matching skips layers {11, 17, 23, 29} of Llama-3.1-8B, consistent with prior activation-based matching results [3]. This agreement suggests that our data-free spectral signatures capture structural information for cross-depth layer alignment.

5.5 Ablation Study↩︎

PL_Alpha_Hill remains robust under output-invariant reparameterizations. We evaluate whether PL_Alpha_Hill is a robust metric for constructing a spectral signature by performing output-invariant reparameterizations on weight matrices before measuring model similarity with Eq. 1 . We compare the spectral signature constructed by PL_Alpha_Hill with other ESD scale metrics (Log Norm and Spectral Norm). Concretely, we apply two reparameterizations on LLaMA-2-7B model: (i) using output-invariant scaling to consecutive attention modules (Q/K and V/O pairs), and (ii) output-invariant change of basis between two consecutive linear layers in the feed-forward network (FFN) module (more details can be found in Appendix 7.6). From Fig. 5, we can see that applying output-invariant weight scaling causes PCS-based and ESD scale-based similarity to degrade substantially as we use larger scaling factors, while the spectral signature with PL_Alpha_Hill remains stable, correctly recognizing the transformed weights as functionally identical. Furthermore, under MLP hidden-unit permutations, PCS collapses to near zero, but PL_Alpha_Hill remains near-perfect (Tab. 8).

Here, we discuss why this ablation is important. Data-aware similarity (e.g., Logits and REEF) is expected to remain high under strictly output-invariant transforms, but it requires a prompt distribution and \(O(\#\text{prompts})\) inference per comparison, making it unsuitable as a zoo-scale kernel. Weight-matrix fingerprints such as AWM [20] reduce some reparameterization sensitivity but still require matrix-level comparisons (and often assignment/matching) per pair. In contrast, our ESD shape signature is computed once per model and yields invariance to common output-preserving symmetries without explicit matching. Note that pure scale metrics (Log Norm, Spectral Norm) are also invariant here, but precisely because they track magnitude, they tend to saturate and are less discriminative across independently trained families (see Sec. 5.1).

PL_Alpha_Hill remains robust under random perturbations. To simulate post-training weight changes, we inject Gaussian noise of increasing magnitude into model weight entries and evaluate the stability of different similarity metrics. We follow the noise strength \(\gamma\) definition in Appendix 7.6 and report full results in Tab. 9. Across all noise levels, our PL_Alpha_Hill–based similarity remains close to the unperturbed model, but PCS degrades substantially as noise magnitude increases. These results indicate that spectral shape features provide a robust measure of model similarity under realistic weight-space perturbations. Moreover, spectral scale metrics also perform well in this scenario since additive Gaussian noise perturbs weights without strongly changing their overall magnitude. Beyond global averages, we also analyze the layer-wise response to perturbation. Fig. 6 demonstrates that larger perturbations generally increase PL_Alpha_Hill while preserving similar depth-wise trends, showing that spectral signatures respond systematically to weight-space noise.

6 Conclusion↩︎

This work presents a spectral shape–based framework for large-scale analysis of LLMs that is data-free, computationally efficient, and robust to post-training transformations. We curate and benchmark a large and diverse LLM corpus and conduct extensive experiments across similarity measurement, classification, clustering, and performance prediction. Our results have shown the consistent effectiveness of our spectral signature across various experimental setups. Furthermore, we validate that ESD shape metrics, particularly PL_Alpha_Hill, capture intrinsic structural properties of LLMs that are stable under output-invariant reparameterizations and noise perturbations. Therefore, our spectral signature can provide a practical tool for managing modern LLM ecosystems, with implications for model provenance, organization, and scalable evaluation.

7 Appendix↩︎

7.1 Robustness of ESD Shape Metrics↩︎

Pre-training and post-training reshape the correlations within weight matrices, as reflected in variations in the heavy-tailed index across layers. To validate the reliability of our method, we investigate whether the spectral structure preserves sufficient lineage information under post-training perturbations.

7.1.0.1 PL_Alpha_Hill is robust under perturbation.

Fig. 7 shows that ESD shape metrics can encode lineage information beyond pretraining. Using Spearman correlation between layer-wise PL_Alpha_Hill profiles, models derived from the same 7B base model exhibit high similarities, whereas similarities across base families are substantially lower. Moreover, within the Llama-2 lineage, we observe finer sub-structure: distinct clusters align with different post-training trajectories (e.g., instruction-tuned vs. CodeLlama-style derivatives), suggesting that post-training alters spectral signatures in a structured manner. These observations motivate our use of spectral signatures for downstream tasks such as lineage attribution, clustering, and performance prediction (Sec.5).

7.1.0.2 The Forging of Spectral Signature

As illustrated in Fig.  8, we observe three distinct phases of spectral evolution:

1. Initialization. At Step 0, all layers exhibit large, uniform PL_Alpha_Hill values (metrics for attention and MLP weight matrices differ due to the aspect ratio difference [36]), resembling MP distribution for random matrices [59].

2. Formulation. As training progresses, PL_Alpha_Hill decreases globally, signifying feature learning. Crucially, PL_Alpha_Hill drops in early layers, flattens in middle layers, dips in mid-deep layers, and rises in the final layers.

3. Stabilization As shown in Fig.  8, the spectral signature stabilizes rapidly. While the early phase (\(<16k\) steps) shows high volatility, the signature “locks in” after step 16,000. Subsequent updates refine the weights but do not alter the fundamental topological shape.

7.1.0.3 PL_Alpha_Hill undergo minimal change during post-training.

Because post-training updates (e.g., SFT, RLVR) introduce relatively small changes in parameter space, they do not erase the heavy-tailed structure established during pre-training. Based on that, PL_Alpha_Hillworks as a stable and lineage-preserving signature that remains robust to common downstream adaptations. This motivates us to use it for performance prediction and model attribution.

7.2 Metrics for Classification Quality↩︎

Given two signatures \(\mathbf{Z}_1,\mathbf{Z}_2\), we consider: (i) Spearman rank correlation \(\rho_s(\mathbf{Z}_1,\mathbf{Z}_2)\) as our default similarity (Eq. 2 ), (ii) an RBF kernel applied to the signature distance, \[k_{\text{RBF}}(\mathbf{Z}_1,\mathbf{Z}_2)=\exp\!\left(-\frac{\|\mathbf{Z}_1-\mathbf{Z}_2\|_2^2}{2\sigma^2}\right) \label{eq:rbf}\tag{5}\] , and (iii) cosine similarity, \[k_{\cos}\mathbf{(}{Z}_1,\mathbf{Z}_2)=\frac{\langle \mathbf{Z}_1,\mathbf{Z}_2\rangle}{\|\mathbf{Z}_1\|_2\|\mathbf{Z}_2\|_2}. \label{eq:cos}\tag{6}\]

7.3 Metrics for Internal Clustering Validation↩︎

For unsupervised model-family discovery, we evaluate spectral-signature clusters using two standard internal validity indices: the Silhouette Coefficient and the Davies-Bouldin Index.

7.3.0.1 Silhouette Coefficient.

The Silhouette Coefficient measures clustering quality by comparing intra-cluster cohesion and inter-cluster separation. For a point \(i \in C_k\), let \(a(i)\) denote its average distance to other points in \(C_k\), and let \(b(i)\) denote the smallest average distance to points in any other cluster. The silhouette coefficient is \[s(i) = \frac{b(i) - a(i)}{\max{a(i), b(i)}} .\] The overall Silhouette Score is the average of \(s(i)\) over all samples.

7.3.0.2 Davies-Bouldin Index.

The Davies-Bouldin Index (DBI) compares intra-cluster dispersion with inter-cluster separation. Let \(S_i\) denote the dispersion of cluster \(i\), and \(M_{ij}\) denote the distance between the centroids of clusters \(i\) and \(j\). Define \[R_{ij} = \frac{S_i + S_j}{M_{ij}}.\] Then \[DB = \frac{1}{k} \sum_{i=1}^{k} \max_{j \neq i} R_{ij}.\]

7.4 Additional Results on Model Clustering↩︎

In Fig. 9, we show a t-SNE visualization of a 2D embedding for spectral clustering on PL_Alpha_Hill with ground truth labels.

7.5 Additional Generalization Tests↩︎

7.5.0.1 Model Scale Generalization.

As shown in Tab. 10, we evaluate additional models at 1.3B (GPT-Neo family) and 70B (Llama-3.1 family); related pairs achieve similarity \(\geq\) 0.99 while unrelated pairs score 0.55–0.69, consistent with the main results.

7.5.0.2 Intra-family Performance Prediction.

To further validate that spectral signatures capture within-family performance variation, we evaluate fine-tuned models with substantial benchmark gains over their base models (\(>5\) points). As shown in Tab. 11, the predicted scores track the true scores rather than collapsing to the base-model prediction. For the Llama-13B family, our method achieves MAEs of 1.94 on TruthfulQA and 1.55 on MMLU, with significant rank correlation in 7 out of 8 settings.

7.6 Additional Details on Ablation Studies↩︎

7.6.0.1 Output-Invariant Reparameterization

We apply scale-preserving reparameterization to consecutive linear maps while keeping the composed operator unchanged. We define a scaling factor \(a = \exp(\gamma z)\) with strength \(\gamma\) and \(z \sim \mathcal{N}(0, 1)\). The weights are updated as \(Q' = a Q, K' = K/a\) (and similarly for \(V, O\)). For Permutation on Hidden Units, we generate a random permutation matrix \(P\) and construct a transformed weight set: \(W' = \{P W_{gate}, P W_{up}, W_{down} P^{-1}\}\) for each layer. The similarity between reparameterized models and their origins remains 1 for Llama-2-7b, Qwen-1.5-7b, and Mistral-7b, suggesting robustness of our method. In contrast, PCS degrades substantially under both transformations.

7.6.0.2 Noise Perturbation

For each weight matrix \(W_\ell\), we sample \(Z_\ell \sim \mathcal{N}(0,I)\) with the same shape and define \[\Delta W_\ell = \mathrm{std}(W_\ell) Z_\ell, \qquad W_\ell^{(\gamma)} = W_\ell + \gamma \Delta W_\ell,\] where \(\gamma\) controls the noise strength. We pre-generate \({\Delta W_\ell}_\ell\) once and reuse it across all \(\gamma\) values for controlled comparisons.

References↩︎

---

1. It is known that more mathematically principled estimators exist, such as those based on MLE [38], [39] or eigenvalue repulsion [40]. However, in our experiments, the Hill estimator demonstrates more stable performance and is easier to scale to large settings.↩︎